Optimizing Neural Network for Computer Vision task in Edge Device
Extracted source text · arXiv 2110.00791
%\pdfoutput=1
\documentclass{article}
\pdfoutput=1
\usepackage{arxiv}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{lipsum} % Can be removed after putting your text content
\usepackage{graphicx}
\usepackage{natbib}
\usepackage{doi}
\usepackage{etoolbox}
\patchcmd{\thebibliography}{\section*}{\section}{}{}
\title{Optimizing Neural Network for Computer Vision task in Edge Device }
%\date{September 9, 1985} % Here you can change the date presented in the paper title
%\date{} % Or removing it
\author{ \href{https://orcid.org/0000-0001-6214-4729}{\includegraphics[scale=0.06]{orcid.pdf}\hspace{1mm}Ranjith M S}\\%\thanks{Use footnote for providing further
%information about author (webpage, alternative
% address)---\emph{not} for acknowledging funding agencies.} \\
%Department of Electronics and Communication\\
%The National Institute of Engineering\\
%Mysore, PA 15213 \\
% \texttt{ranjithms523@} \\
%% examples of more authors
\And
%\href{https://orcid.org/0000-0000-0000-0000}{\includegraphics[scale=0.06]{orcid.pdf}
%\hspace{1mm}
Dr.~ S Parameshwara \\
% Department of Electronics and Communication\\
%The National Institute of Engineering\\
%\texttt{stariate@ee.mount-sheikh.edu} \\
\And
%\href{https://orcid.org/0000-0000-0000-0000}{\includegraphics[scale=0.06]{orcid.pdf}
%\hspace{1mm}
Pavan Yadav A \\
%Department of Electronics and Communication\\
%The National Institute of Engineering\\
%% \AND
\And
%\href{https://orcid.org/0000-0000-0000-0000}{\includegraphics[scale=0.06]{orcid.pdf}
%\hspace{1mm}
Shriganesh Hegde \\
\And
Department of Electronics and Communication\\
The National Institute of Engineering\\
%% Coauthor \\
%% Affiliation \\
%% Address \\
%% \texttt{email} \\
%% \And
%% Coauthor \\
%% Affiliation \\
%% Address \\
%% \texttt{email} \\
%% \And
%% Coauthor \\
%% Affiliation \\
%% Address \\
%% \texttt{email} \\
}
% Uncomment to remove the date
%\date{}
% Uncomment to override the `A preprint' in the header
\renewcommand{\headeright}{Research Paper}
\renewcommand{\undertitle}{Research Paper}
%\renewcommand{\shorttitle}{\textit{arXiv} Template}
%%% Add PDF metadata to help others organize their library
%%% Once the PDF is generated, you can check the metadata with
%%% $ pdfinfo template.pdf
\hypersetup{
pdftitle={Optimizing Neural Network for Computer Vision task in Edge Device},
pdfsubject={q-bio.NC, q-bio.QM},
pdfauthor={Ranjith M S},
pdfkeywords={Embedded System, Edge Computing, Optimization, Deep learning, Computer Vision, Pruning, Quantization, ARM, convolution neural network},
}
\begin{document}
\maketitle
\begin{abstract}
The field of computer vision has grown very rapidly in the past few years due to networks like convolution neural networks and their variants. The memory required to store the model and computational expense are very high for such a network limiting it to deploy on the edge device. Many times, applications rely on the cloud but that makes it hard for working in real-time due to round-trip delays. We overcome these problems by deploying the neural network on the edge device itself. The computational expense for edge devices is reduced by reducing the floating-point precision of the parameters in the model. After this the memory required for the model decreases and the speed of the computation increases where the performance of the model is least affected. This makes an edge device to predict from the neural network all by itself.
\end{abstract}
% keywords can be removed
\keywords{Embedded System \and Edge Computing \and Optimization \and Deep learning\and Computer Vision \and Pruning \and Quantization \and ARM \and convolution neural network}
\section{Introduction}
With the rise of Artificial intelligence, it is realized that deep-learning-based approaches give satisfactory results compared to numerous other state-of-the-art schemes which are hand-engineered to all the computer vision tasks. The features learned from Convolutional neural networks outperform other hand-engineered feature-based methods like SIFT [\cite{r1}] and HoG [\cite{r2}] in computer vision tasks like image classification and object detection.\\
The availability of large datasets and powerful computation devices made it possible to train the large and complex neural networks to obtain the desired performance on many computer vision tasks. The large amount of open-source pre-trained models trained on large datasets like ImageNet [\cite{r3}], MS-COCO [\cite{r4}], SHVN [\cite{r5}] created a large number of useful filters especially the features learned from the initial layer helps in transfer learning a lot. In transfer learning, most of the time only the last few layers of pre-trained models are modified and trained which counters the problem of having fewer data to a certain extent.\\
Currently, a variety of embedded systems are deployed but the usage of neural networks is limited in edge devices like microcontrollers, Raspberry Pi. Household devices like Refrigerators, washing machines use a set of logic, rules for their automatic operations. By optimizing the network trained on a dataset traditional way of controlling can be replaced by intelligently monitoring the systems with the power of AI and neural networks [\cite{r6}]. Optimizing the convolutional neural network architectures like ResNet [~\cite{r7}], DenseNet [~\cite{r8}], AlexNet [~\cite{r9}] which are generally used in computer vision tasks allows creating many useful applications. for example, using gestures recognized from the neural network to perform a specific task based on the requirement, where the task and the gesture recognition are performed by the edge device itself. \\
\subsection{Literature Survey for implementing and training the neural network}
Since we use dense layers along with convolutional layers for our task we prefer He initialization [~\cite{r10}] as compared with Xavier Initialization ~\cite{r11} for initializing the weights while training the network. But using LSUV
initialization [~\cite{r12}] leads to the learning of very deep nets that produces networks with test accuracy better or equal to standard methods.\\
“overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate.” [~\cite{r13}].\\
The optimizers such as Adam [~\cite{r14}], RMS [~\cite{r15}] lead for faster convergence. And adaptive optimizers like LARS [~\cite{r16}], AdamW [~\cite{r17}] lead to better generalization.\\
“One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate” [~\cite{r18}].\\
To speed up training we use Batch Normalization [~\cite{r19}].\\
We evaluate the performance of the optimized model for recognizing hand gestures in raspberry pi. We collect the dataset for our task, in order to obtain a good dataset, we take images in different lightings, different cameras, and angles which helps in creating a robust good accurate model. The neural network is implemented and trained using a device like GPU which has strong computational power. Then the model is optimized by reducing the floating-point precision of trained parameters and by pruning the parameters. The model is then evaluated and deployed on edge device like raspberry pi.
\section{Experimental Setup}
\label{sec:Experimental Setup}
\begin{figure}[h]
%\left
{\includegraphics[width=14cm, height=7cm]{images/nn_archi_32_final.jpg}}
\caption{Neural Network Architecture}
\label{fig:fig1}
\end{figure}
%\lipsum[4] See Section \ref{sec:headings}.
\subsection{Preparation of dataset}
Dataset is prepared by collecting images for each different class. i.e., for each unique gesture to be recognized. To obtain a robust model many numbers of images are captured in different light intensity, angles, background, distance, camera for every gesture to be recognized. Then images belonging to a unique class are stored in a separate folder. Different sized datasets of images are created i.e., captured images are scaled to 64×64, 96×96, 128×128, 256×256 hence each size dataset we will have the images for each class of that size. This is done to find the optimal size for input images such that the images with less resolution are trained to the desired accuracy by which we can reduce the model size and hence computations required for prediction. \\
\subsubsection{Steps}
\begin{itemize}
\item Collect the images for each unique gesture under different lighting conditions, background, camera, angle and save each gesture image in their respective folders.
\item Create a unique folder for all the required sizes of images (E.g., folder\_64, folder\_96 …etc)
\item Under each different folder (folders created for the required size dataset) create the folders for each unique class (gesture) (E.g., one, two, three …. etc).
\item Read the images from the original folder resize them and save accordingly. We have used OpenCV to read images, resize them and store them accordingly. E.g. read an image of class ‘one’
\item resize it to 64×64, 96×96, 128×128, 256×256 and store them in folder ‘one of ‘folder\_64’, ‘folder\_96’, ‘folder\_128’, ‘folder\_256’ respectively
\end{itemize}
\begin{figure}[h]
\centering
{\includegraphics[width=15cm, height=7cm]{images/sample_images.png}}
\caption{Sample images in dataset}
\label{fig:fig01}
\end{figure}
\subsection{Training the Network}
TensorFlow [~\cite{r20}] is used to create the network and load the dataset during training. The images are normalized during every iteration before feeding into the network and a few augmentation’s like horizontal flips were performed in order to generalize the model well. Dataset is split into 85\% for training and 15\% for validation. Fig-2 shows the network architecture for the training image dataset of size 256×256 with 3 channels (RGB). The normalized input images are 2D-convolved with a kernel size of 3×3 and 32 such filters are used, and the output is passed through the non-Linearity activation function ReLU [21]. Then the 2D-max-pooling (kernel size 2×2) is applied followed by Dropout. Then the output is 2D-convolved with a kernel size of 3×3 (64 filters) and passing through ReLU activation followed by 2D-max-pooling and Dropout. Then the output is flattened followed by Dropout and 2 Dense layers having 128 and 5 (as the number of unique gestures is 5) hidden units respectively. The first hidden layer has the ReLU activation function and the latter has SoftMax activation.
\begin{table}[h]
\caption{Network structure}
\centering
\begin{tabular}{lll}
\toprule
Layer & Output shape & No. of parameters \\
\toprule
Input & (None, 256,256,3) & 0 \\
Conv2D & (None, 254,254,32) & 896 \\
Max-pooling2D & (None, 127,127,32) & 0 \\
Dropout & (None, 127,127,32) & 0 \\
Conv2D & (None, 125,125,64) & 18496 \\
Max-pooling2D & (None, 62,62,64) & 0 \\
Dropout & (None, 62,62,64) & 0 \\
Flatten & (None,246016) & 0 \\
Dropout & (None,246016) & 0 \\
Dense & (None,128) & 31490176 \\
Dense & (None,5) & 645 \\
\bottomrule
Total params & 31510213 \\
\bottomrule
\end{tabular}
\end{table}
The model is trained using Tesla-K80 GPU which can do up to 8.73 Teraflops single-precision and up to 2.91 Teraflops double-precision performance with NVIDIA GPU Boost. The model is set to train for a maximum of 25 epochs provided the validation loss will not increase continuously for 3 epochs in that case the training is stopped, and the model weights are saved. \\
The model is trained using Adam optimizer with a batch size of 32. We used Categorical cross-entropy as the loss function to train the model. categorical cross-entropy is given by,
\begin{equation}
C E=-\sum_{i}^{c} t_{i} \log (f(s) i)
\end{equation}
Where f(s)i represents SoftMax function and is given by,
\begin{equation}
f(s){i}=\frac{e^{S{i}}}{\sum_{j}^{C} e^{s_{j}}}
\end{equation}
Where C represents the total number of class and sj are the score inferred by the neural network for each class in C.\\
\begin{figure}[!htbp]
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/accuracy_256.png}
\caption{variation of accuracy for 256×256}
\label{fig4}
\end{minipage}%
\hfill%
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/loss_256.png}
\caption{variation of loss for 256×256}
\label{fig3}
\end{minipage}
\end{figure}
Figure~\ref{fig3} indicates how training and validation loss varied during training in each epoch and we see that both training and validation loss reach a very low value by 14 epochs. And the training is stopped after 14 epochs as the validation loss keeps increasing by a small value. The best weights are saved.
Figure ~\ref{fig4} indicates how accuracy varied during training after each epoch. Training and validation accuracy reach desired values by the end of 14 epochs.
\begin{figure}[!htbp]
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/accuracy_64.png}
\caption{variation of accuracy for 64×64}
\label{f5}
\end{minipage}%
\hfill%
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/loss_64.png}
\caption{variation of loss for 64×64}
\label{f6}
\end{minipage}
\end{figure}
\begin{figure}[!htbp]
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/accuracy_96.png}
\caption{variation of accuracy for 96×96}
\label{f7}
\end{minipage}%
\hfill%
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/loss_96.png}
\caption{variation of loss for 96×96}
\label{f8}
\end{minipage}
\end{figure}
\begin{figure}[!htbp]
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/accuracy_128.png}
\caption{variation of accuracy for 128×128}
\label{f10}
\end{minipage}%
\hfill%
\begin{minipage}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{images/loss_128.png}
\caption{variation of loss for 128×128}
\label{f9}
\end{minipage}
\end{figure}
Similar results were obtained when tried with different image sizes and found that image size with 96x96 works best among all during the test time. Figures ~\ref{f5} - ~\ref{f9} indicates the variation of accuracy and loss for different input sizes of the images during training. Since our aim is to fit the model in edge device memory, it is suitable to choose the model which gives desirable accuracy keeping image size as low as possible. The model trained for image size with 96 × 96 was obtained over 98\% accuracy on the validation set indicating the best input size. The model trained with 128×128 and 256×256 images gave very good training accuracy (nearly 100\%) but were able to give only around 92\% accuracy on the validation set indicating high variance of the model. This may be due to fewer data and the usage of large neural networks.
\begin{figure}[h]
\centering
{\includegraphics[width=15cm, height=5cm]{images/miscl.png}}
\caption{Misclassifications of the network}
\label{f11}
\end{figure}
In many a case one gesture will be similar to the others which usually makes models to perform badly on them. To reduce such misclassifications higher weight is assigned for the loss of such classes. Here slightly higher weight is applied to gestures ‘one’, ‘three’, ‘four’ hence we see a smaller number of misclassifications (Figure- ~\ref{f11}). Else the misclassification for those gesture will be slightly high.
\section{Optimization}
\label{sec:optimization}
The trained model able to perform classifications has the parameters with the floating-point precision of 64-bit occupies more memory while performing the inference. Since floating-point arithmetic operations need more machine cycles and the machine cycles required for an operation increases with increase in floating-point precision. By reducing the floating-point precision machine cycles required to perform the operation reduces along with the memory. The reduction in the floating-point precision does not affect the accuracy more and in rare cases, certain models may gain some accuracy as a result of the optimization process . Since deep neural networks require more floating-point operations for its prediction reducing floating-point precision makes it suitable for edge devices.\\
Pruning of parameters is done properly using the representative dataset during the model optimization i.e., reduce parameter count by structured pruning. We use TensorFlow lite for reducing the floating-point precision of our trained TensorFlow model which leads to optimization. \\
\begin{figure}[h]
\centering
{\includegraphics[width=12cm, height=6cm]{images/acc_opt.png}}
\caption{Variation of accuracy with optimization}
\label{f12}
\end{figure}
Figure- ~\ref{f12} indicates the variation of accuracy with optimization for models trained on different image sizes. For a non-optimized models accuracy indicates the accuracy on validation set (100 examples) and for optimized models’ accuracy is calculated on test set of 32 examples. We observe that the model optimized for the Neural network trained on 96×96 images has good accuracy and performance during inference. Since all the models have similar accuracy, with models being trained on 96×96 and 64×64 images are slightly better.
\begin{figure}[h]
\centering
{\includegraphics[width=12cm, height=6cm]{images/loss_opt.png}}
\caption{Variation of loss with optimization}
\label{f13}
\end{figure}
Although we have similar accuracy between optimized and non-optimized models the loss of optimized model is comparatively high. Figure ~\ref{f13} shows the variation of loss with optimization for neural networks trained on different image size. The loss for non-optimized models is computed on validation set of 100 examples and loss for optimized models is computed on test set of 32 examples.
\begin{figure}[h]
\centering
{\includegraphics[width=12cm, height=6cm]{images/siz_opt.png}}
\caption{Variation of model size with optimization.}
\label{f14}
\end{figure}
Variation of model size with optimization for different input image sizes is shown in the Figure ~\ref{f14}. The size of the model reduces nearly to 33\% of initial size after optimization. We observe that the model with input image shapes of 96×96 and 64×64 after optimization reduces to 15.9MB and 6.5MB respectively which could be deployed in edge devices.
\begin{table}[h]
\caption{Network structure}
\centering
\label{t2}
\begin{tabular}{lllll}
\toprule
Image Size & Optimized & Loss & Accuracy & Model Size \\
\toprule
128 x 128 & No & 0.206300 & 0.92370 & 88759408 \\ \toprule
128 x 128 & Yes & 1.063362 & 0.93750 & 29574240 \\ \toprule
64 x 64 & No & 0.073400 & 0.97730 & 19553392 \\ \toprule
64 x 64 & Yes & 0.533179 & 0.96875 & 6505568 \\ \toprule
96 x 96 & No & 0.089700 & 0.98110 & 47864936 \\ \toprule
96 x 96 & Yes & 0.566464 & 0.96875 & 15942712 \\ \toprule
256 x 256 & No & 0.543200 & 0.92800 & 378166376 \\ \toprule
256 x 256 & Yes & 1.756660 & 0.90625 & 126043192 \\
\bottomrule
\end{tabular}
\end{table}
Table- ~\ref{t2} shows the accuracy and loss of the model before and after the optimization and the respective model size in bytes. Loss and accuracy for the optimized model is computed on the test set containing 32 examples and for non-optimized models they are computed on validation set of 100 examples.
\section{Conclusion}
The optimized models for this use case could be easily deployed on edge device like raspberry pi. Since the memory required for the models with input image shape 96×96 and 64×64 is few MB it could be fulfilled by edge devices easily. In the cases if the edge computing is hard then the aid from neural compute stick could be obtained for the deployment of the model.
{
\small
\nocite{*}
\newcounter{bibcount}
\bibliographystyle{ieee_fullname}
%\bibliography{latex}
\begin{thebibliography}{29}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Mikolajczyk and Schmid(2004)]{r1}
Krystian Mikolajczyk and Cordelia Schmid.
\newblock Scale \& affine invariant interest point detectors.
\newblock \emph{International journal of computer vision}, 60\penalty0
(1):\penalty0 63--86, 2004.
\bibitem[Dalal and Triggs(2005)]{r2}
Navneet Dalal and Bill Triggs.
\newblock Histograms of oriented gradients for human detection.
\newblock In \emph{2005 IEEE computer society conference on computer vision and
pattern recognition (CVPR'05)}, volume~1, pages 886--893. Ieee, 2005.
\bibitem[Challenge()]{r3}
ImageNet Large Scale Visual~Recognition Challenge.
\newblock Olga russakovsky, jia deng, hao su, jonathan krause, sanjeev
satheesh, sean ma, zhiheng huang, andrej karpathy, aditya khosla, michael
bernstein, alexander c. berg, li fei-fei. 2014.
\newblock \emph{Computing Research Repository, Vol. abs/1409.0575}.
\bibitem[Lin et~al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan,
Doll{\'a}r, and Zitnick]{r4}
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Doll{\'a}r, and C~Lawrence Zitnick.
\newblock Microsoft coco: Common objects in context.
\newblock In \emph{European conference on computer vision}, pages 740--755.
Springer, 2014.
\bibitem[Netzer et~al.(2011)Netzer, Wang, Coates, Bissacco, Wu, and Ng]{r5}
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo~Wu, and Andrew~Y
Ng.
\newblock Reading digits in natural images with unsupervised feature learning.
\newblock 2011.
\bibitem[{Ranjith M S} and Parameshwara(2020)]{r6}
{Ranjith M S} and S~Parameshwara.
\newblock Optimizing neural networks for embedded systems.
\newblock \emph{IRJET}, 7\penalty0 (211), 2020.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{r7}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 770--778, 2016.
\bibitem[Huang et~al.(2017)Huang, Liu, Van Der~Maaten, and Weinberger]{r8}
Gao Huang, Zhuang Liu, Laurens Van Der~Maaten, and Kilian~Q Weinberger.
\newblock Densely connected convolutional networks.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 4700--4708, 2017.
\bibitem[Krizhevsky et~al.(2012)Krizhevsky, Sutskever, and Hinton]{r9}
Alex Krizhevsky, Ilya Sutskever, and Geoffrey~E Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock \emph{Advances in neural information processing systems},
25:\penalty0 1097--1105, 2012.
\bibitem[He et~al.(2015)He, Zhang, Ren, and Sun]{r10}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Delving deep into rectifiers: Surpassing human-level performance on
imagenet classification.
\newblock In \emph{Proceedings of the IEEE international conference on computer
vision}, pages 1026--1034, 2015.
\bibitem[Glorot and Bengio(2010)]{r11}
Xavier Glorot and Yoshua Bengio.
\newblock Understanding the difficulty of training deep feedforward neural
networks.
\newblock In \emph{Proceedings of the thirteenth international conference on
artificial intelligence and statistics}, pages 249--256. JMLR Workshop and
Conference Proceedings, 2010.
\bibitem[Mishkin and Matas(2015)]{r12}
Dmytro Mishkin and Jiri Matas.
\newblock All you need is a good init.
\newblock \emph{arXiv preprint arXiv:1511.06422}, 2015.
\bibitem[Hinton et~al.(2012{\natexlab{a}})Hinton, Srivastava, Krizhevsky,
Sutskever, and Salakhutdinov]{r13}
Geoffrey~E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and
Ruslan~R Salakhutdinov.
\newblock Improving neural networks by preventing co-adaptation of feature
detectors.
\newblock \emph{arXiv preprint arXiv:1207.0580}, 2012{\natexlab{a}}.
\bibitem[Kingma and Ba(2014)]{r14}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{arXiv preprint arXiv:1412.6980}, 2014.
\bibitem[Hinton et~al.(2012{\natexlab{b}})Hinton, Srivastava, and Swersky]{r15}
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky.
\newblock Neural networks for machine learning lecture 6a overview of
mini-batch gradient descent.
\newblock \emph{Cited on}, 14\penalty0 (8):\penalty0 2, 2012{\natexlab{b}}.
\bibitem[You et~al.(2019)You, Hseu, Ying, Demmel, Keutzer, and Hsieh]{r16}
Yang You, Jonathan Hseu, Chris Ying, James Demmel, Kurt Keutzer, and Cho-Jui
Hsieh.
\newblock Large-batch training for lstm and beyond.
\newblock In \emph{Proceedings of the International Conference for High
Performance Computing, Networking, Storage and Analysis}, pages 1--16, 2019.
\bibitem[Loshchilov and Hutter(2017)]{r17}
Ilya Loshchilov and Frank Hutter.
\newblock Decoupled weight decay regularization.
\newblock \emph{arXiv preprint arXiv:1711.05101}, 2017.
\bibitem[Smith and Topin(2019)]{r18}
Leslie~N Smith and Nicholay Topin.
\newblock Super-convergence: Very fast training of neural networks using large
learning rates.
\newblock In \emph{Artificial Intelligence and Machine Learning for
Multi-Domain Operations Applications}, volume 11006, page 1100612.
International Society for Optics and Photonics, 2019.
\bibitem[Ioffe and Szegedy(2015)]{r19}
Sergey Ioffe and Christian Szegedy.
\newblock Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
\newblock In \emph{International conference on machine learning}, pages
448--456. PMLR, 2015.
\bibitem[Abadi et~al.(2016)Abadi, Agarwal, Barham, Brevdo, Chen, Citro,
Corrado, Davis, Dean, Devin, et~al.]{r20}
Mart{\'\i}n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen,
Craig Citro, Greg~S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et~al.
\newblock Tensorflow: Large-scale machine learning on heterogeneous distributed
systems.
\newblock \emph{arXiv preprint arXiv:1603.04467}, 2016.
\bibitem[r22()]{r22}
Model optimization | tensorflow lite.
\newblock URL
\url{https://www.tensorflow.org/lite/performance/model_optimizationon}.
\newblock Retrieved from
https://www.tensorflow.org/lite/performance/model\_optimization.
\bibitem[Nair and Hinton(2010)]{r21}
Vinod Nair and Geoffrey~E Hinton.
\newblock Rectified linear units improve restricted boltzmann machines.
\newblock In \emph{Icml}, 2010.
\bibitem[Hunter(2007)]{r23}
John~D Hunter.
\newblock Matplotlib: A 2d graphics environment.
\newblock \emph{Computing in science \& engineering}, 9\penalty0 (03):\penalty0
90--95, 2007.
\bibitem[{Ranjith M S} et~al.(2021)]{r29}
{Ranjith M S} and Santhosh, HS and Swamy, MS.
\newblock Machine Learning Algorithms for the Detection of Diabetes.
\newblock \emph{IRJET}, 8\penalty0 (27), 2021.
\bibitem[Inc()]{r24}
Plotly~Technologies Inc.
\newblock \emph{Collaborative data science}.
\newblock Plotly Technologies Inc, Montreal, QC.
\bibitem[r25()]{r25}
Tensorflow lite inference.
\newblock URL \url{https://www.tensorflow.org/lite/guide/inference}.
\newblock Retrieved from https://www.tensorflow.org/lite/guide/inference.
\bibitem[r26()]{r26}
Tensorflow model optimization.
\newblock URL \url{https://www.tensorflow.org/model_optimization/guide}.
\newblock Retrieved from https://www.tensorflow.org/model\_optimization/guide.
\bibitem[r27()]{r27}
Tensorflow model optimization toolkit — float16 quantization halves model
size — the tensorflow blog.
\newblock URL
\url{https://blog.tensorflow.org/2019/08/tensorflow-model-optimization-toolkit_5.html}.
\newblock Retrieved from
https://blog.tensorflow.org/2019/08/tensorflow-model-optimization-toolkit\_5.html.
\bibitem[r28()]{r28}
Tensorflow lite for microcontrollers.
\newblock URL \url{https://www.tensorflow.org/lite/microcontrollers}.
\newblock Retrieved from https://www.tensorflow.org/lite/microcontrollers.
\end{thebibliography}
} %%% Uncomment this line and comment out the ``thebibliography'' section below to use the external .bib file (using bibtex) .
%%% Uncomment this section and comment out the \bibliography{references} line above to use inline references.
%\begin{thebibliography}{1}
% \bibitem{kour2014real}
% George Kour and Raid Saabne.
% \newblock Real-time segmentation of on-line handwritten arabic script.
% \newblock In {\em Frontiers in Handwriting Recognition (ICFHR), 2014 14th
% International Conference on}, pages 417--422. IEEE, 2014.
% \bibitem{kour2014fast}
% George Kour and Raid Saabne.
% \newblock Fast classification of handwritten on-line arabic characters.
% \newblock In {\em Soft Computing and Pattern Recognition (SoCPaR), 2014 6th
% International Conference of}, pages 312--318. IEEE, 2014.
% \bibitem{hadash2018estimate}
% Guy Hadash, Einat Kermany, Boaz Carmeli, Ofer Lavi, George Kour, and Alon
% Jacovi.
% \newblock Estimate and replace: A novel approach to integrating deep neural
% networks with existing applications.
% \newblock {\em arXiv preprint arXiv:1804.09028}, 2018.
%\end{thebibliography}
\end{document}