Rethinking Convolutional Semantic Segmentation Learning
Extracted source text · arXiv 1710.07991
%% bare_conf.tex
%% V1.4b
%% 2015/08/26
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.8b or later) with an IEEE
%% conference paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/pkg/ieeetran
%% and
%% http://www.ieee.org/
%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE!
%% User assumes all risk.
%% In no event shall the IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including **
%% ** renaming them and changing author support contact information. **
%%*************************************************************************
% *** Authors should verify (and, if needed, correct) their LaTeX system ***
% *** with the testflow diagnostic prior to trusting their LaTeX platform ***
% *** with production work. The IEEE's font choices and paper sizes can ***
% *** trigger bugs that do not appear when using other class files. *** ***
% The testflow support page is at:
% http://www.michaelshell.org/tex/testflow/
\documentclass[conference]{IEEEtran}
% Some Computer Society conferences also require the compsoc mode option,
% but others use the standard conference format.
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[conference]{../sty/IEEEtran}
% Some very useful LaTeX packages include:
% (uncomment the ones you want to load)
% *** MISC UTILITY PACKAGES ***
%
%\usepackage{ifpdf}
% Heiko Oberdiek's ifpdf.sty is very useful if you need conditional
% compilation based on whether the output is pdf or dvi.
% usage:
% \ifpdf
% % pdf code
% \else
% % dvi code
% \fi
% The latest version of ifpdf.sty can be obtained from:
% http://www.ctan.org/pkg/ifpdf
% Also, note that IEEEtran.cls V1.7 and later provides a builtin
% \ifCLASSINFOpdf conditional that works the same way.
% When switching from latex to pdflatex and vice-versa, the compiler may
% have to be run twice to clear warning/error messages.
% *** CITATION PACKAGES ***
%
%\usepackage{cite}
% cite.sty was written by Donald Arseneau
% V1.6 and later of IEEEtran pre-defines the format of the cite.sty package
% \cite{} output to follow that of the IEEE. Loading the cite package will
% result in citation numbers being automatically sorted and properly
% "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using
% cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's
% \cite will automatically add leading space, if needed. Use cite.sty's
% noadjust option (cite.sty V3.8 and later) if you want to turn this off
% such as if a citation ever needs to be enclosed in parenthesis.
% cite.sty is already installed on most LaTeX systems. Be sure and use
% version 5.0 (2009-03-20) and later if using hyperref.sty.
% The latest version can be obtained at:
% http://www.ctan.org/pkg/cite
% The documentation is contained in the cite.sty file itself.
% *** GRAPHICS RELATED PACKAGES ***
%
\ifCLASSINFOpdf
\usepackage[pdftex]{graphicx}
% declare the path(s) where your graphic files are
% \graphicspath{{../pdf/}{../jpeg/}}
% and their extensions so you won't have to specify these with
% every instance of \includegraphics
% \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
\else
% or other class option (dvipsone, dvipdf, if not using dvips). graphicx
% will default to the driver specified in the system graphics.cfg if no
% driver is specified.
% \usepackage[dvips]{graphicx}
% declare the path(s) where your graphic files are
% \graphicspath{{../eps/}}
% and their extensions so you won't have to specify these with
% every instance of \includegraphics
% \DeclareGraphicsExtensions{.eps}
\fi
% graphicx was written by David Carlisle and Sebastian Rahtz. It is
% required if you want graphics, photos, etc. graphicx.sty is already
% installed on most LaTeX systems. The latest version and documentation
% can be obtained at:
% http://www.ctan.org/pkg/graphicx
% Another good source of documentation is "Using Imported Graphics in
% LaTeX2e" by Keith Reckdahl which can be found at:
% http://www.ctan.org/pkg/epslatex
%
% latex, and pdflatex in dvi mode, support graphics in encapsulated
% postscript (.eps) format. pdflatex in pdf mode supports graphics
% in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure
% that all non-photo figures use a vector format (.eps, .pdf, .mps) and
% not a bitmapped formats (.jpeg, .png). The IEEE frowns on bitmapped formats
% which can result in "jaggedy"/blurry rendering of lines and letters as
% well as large increases in file sizes.
%
% You can find documentation about the pdfTeX application at:
% http://www.tug.org/applications/pdftex
\usepackage{subfig}
\usepackage[nocompress]{cite}
\def\citepunct{,\,}
% *** MATH PACKAGES ***
%
\usepackage{amsmath}
% A popular package from the American Mathematical Society that provides
% many useful and powerful commands for dealing with mathematics.
%
% Note that the amsmath package sets \interdisplaylinepenalty to 10000
% thus preventing page breaks from occurring within multiline equations. Use:
%\interdisplaylinepenalty=2500
% after loading amsmath to restore such page breaks as IEEEtran.cls normally
% does. amsmath.sty is already installed on most LaTeX systems. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/pkg/amsmath
\usepackage{amsfonts}
\usepackage{mathtools,xparse}
\usepackage[section]{placeins}
\DeclarePairedDelimiter{\abs}{\lvert}{\rvert}
\DeclarePairedDelimiter{\norm}{\lVert}{\rVert}
\NewDocumentCommand{\normL}{ s O{} m }{%
\IfBooleanTF{#1}{\norm*{#3}}{\norm[#2]{#3}}_{L_2(\Omega)}%
}
% *** SPECIALIZED LIST PACKAGES ***
%
%\usepackage{algorithmic}
% algorithmic.sty was written by Peter Williams and Rogerio Brito.
% This package provides an algorithmic environment fo describing algorithms.
% You can use the algorithmic environment in-text or within a figure
% environment to provide for a floating algorithm. Do NOT use the algorithm
% floating environment provided by algorithm.sty (by the same authors) or
% algorithm2e.sty (by Christophe Fiorio) as the IEEE does not use dedicated
% algorithm float types and packages that provide these will not provide
% correct IEEE style captions. The latest version and documentation of
% algorithmic.sty can be obtained at:
% http://www.ctan.org/pkg/algorithms
% Also of interest may be the (relatively newer and more customizable)
% algorithmicx.sty package by Szasz Janos:
% http://www.ctan.org/pkg/algorithmicx
% *** ALIGNMENT PACKAGES ***
%
%\usepackage{array}
% Frank Mittelbach's and David Carlisle's array.sty patches and improves
% the standard LaTeX2e array and tabular environments to provide better
% appearance and additional user controls. As the default LaTeX2e table
% generation code is lacking to the point of almost being broken with
% respect to the quality of the end results, all users are strongly
% advised to use an enhanced (at the very least that provided by array.sty)
% set of table tools. array.sty is already installed on most systems. The
% latest version and documentation can be obtained at:
% http://www.ctan.org/pkg/array
% IEEEtran contains the IEEEeqnarray family of commands that can be used to
% generate multiline equations as well as matrices, tables, etc., of high
% quality.
% *** SUBFIGURE PACKAGES ***
%\ifCLASSOPTIONcompsoc
% \usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
%\else
% \usepackage[caption=false,font=footnotesize]{subfig}
%\fi
% subfig.sty, written by Steven Douglas Cochran, is the modern replacement
% for subfigure.sty, the latter of which is no longer maintained and is
% incompatible with some LaTeX packages including fixltx2e. However,
% subfig.sty requires and automatically loads Axel Sommerfeldt's caption.sty
% which will override IEEEtran.cls' handling of captions and this will result
% in non-IEEE style figure/table captions. To prevent this problem, be sure
% and invoke subfig.sty's "caption=false" package option (available since
% subfig.sty version 1.3, 2005/06/28) as this is will preserve IEEEtran.cls
% handling of captions.
% Note that the Computer Society format requires a larger sans serif font
% than the serif footnote size font used in traditional IEEE formatting
% and thus the need to invoke different subfig.sty package options depending
% on whether compsoc mode has been enabled.
%
% The latest version and documentation of subfig.sty can be obtained at:
% http://www.ctan.org/pkg/subfig
% *** FLOAT PACKAGES ***
%
%\usepackage{fixltx2e}
% fixltx2e, the successor to the earlier fix2col.sty, was written by
% Frank Mittelbach and David Carlisle. This package corrects a few problems
% in the LaTeX2e kernel, the most notable of which is that in current
% LaTeX2e releases, the ordering of single and double column floats is not
% guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a
% single column figure to be placed prior to an earlier double column
% figure.
% Be aware that LaTeX2e kernels dated 2015 and later have fixltx2e.sty's
% corrections already built into the system in which case a warning will
% be issued if an attempt is made to load fixltx2e.sty as it is no longer
% needed.
% The latest version and documentation can be found at:
% http://www.ctan.org/pkg/fixltx2e
%\usepackage{stfloats}
% stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e
% the ability to do double column floats at the bottom of the page as well
% as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in
% LaTeX2e). It also provides a command:
%\fnbelowfloat
% to enable the placement of footnotes below bottom floats (the standard
% LaTeX2e kernel puts them above bottom floats). This is an invasive package
% which rewrites many portions of the LaTeX2e float routines. It may not work
% with other packages that modify the LaTeX2e float routines. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/pkg/stfloats
% Do not use the stfloats baselinefloat ability as the IEEE does not allow
% \baselineskip to stretch. Authors submitting work to the IEEE should note
% that the IEEE rarely uses double column equations and that authors should try
% to avoid such use. Do not be tempted to use the cuted.sty or midfloat.sty
% packages (also by Sigitas Tolusis) as the IEEE does not format its papers in
% such ways.
% Do not attempt to use stfloats with fixltx2e as they are incompatible.
% Instead, use Morten Hogholm'a dblfloatfix which combines the features
% of both fixltx2e and stfloats:
%
% \usepackage{dblfloatfix}
% The latest version can be found at:
% http://www.ctan.org/pkg/dblfloatfix
% *** PDF, URL AND HYPERLINK PACKAGES ***
%
%\usepackage{url}
% url.sty was written by Donald Arseneau. It provides better support for
% handling and breaking URLs. url.sty is already installed on most LaTeX
% systems. The latest version and documentation can be obtained at:
% http://www.ctan.org/pkg/url
% Basically, \url{my_url_here}.
% *** Do not adjust lengths that control margins, column widths, etc. ***
% *** Do not use packages that alter fonts (such as pslatex). ***
% There should be no need to do such things with IEEEtran.cls V1.6 and later.
% (Unless specifically asked to do so by the journal or conference you plan
% to submit to, of course. )
% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}
\begin{document}
%
% paper title
% Titles are generally capitalized except for words such as a, an, and, as,
% at, but, by, for, in, nor, of, on, or, the, to and up, which are usually
% not capitalized unless they are the first or last word of the title.
% Linebreaks \\ can be used within to get better formatting as desired.
% Do not put math or special symbols in the title.
\title{Rethinking Convolutional Semantic Segmentation Learning}
% author names and affiliations
% use a multiple column layout for up to three different
% affiliations
\author{\IEEEauthorblockN{Mrinal Haloi}
\IEEEauthorblockA{IIT Guwahati, ARTELUS\\
Email: h.mrinal@iitg.ernet.in}
}
% conference papers do not typically use \thanks and this command
% is locked out in conference mode. If really needed, such as for
% the acknowledgment of grants, issue a \IEEEoverridecommandlockouts
% after \documentclass
% for over three affiliations, or if they all won't fit within the width
% of the page, use this alternative format:
%
%\author{\IEEEauthorblockN{Michael Shell\IEEEauthorrefmark{1},
%Homer Simpson\IEEEauthorrefmark{2},
%James Kirk\IEEEauthorrefmark{3},
%Montgomery Scott\IEEEauthorrefmark{3} and
%Eldon Tyrell\IEEEauthorrefmark{4}}
%\IEEEauthorblockA{\IEEEauthorrefmark{1}School of Electrical and Computer Engineering\\
%Georgia Institute of Technology,
%Atlanta, Georgia 30332--0250\\ Email: see http://www.michaelshell.org/contact.html}
%\IEEEauthorblockA{\IEEEauthorrefmark{2}Twentieth Century Fox, Springfield, USA\\
%Email: homer@thesimpsons.com}
%\IEEEauthorblockA{\IEEEauthorrefmark{3}Starfleet Academy, San Francisco, California 96678-2391\\
%Telephone: (800) 555--1212, Fax: (888) 555--1212}
%\IEEEauthorblockA{\IEEEauthorrefmark{4}Tyrell Inc., 123 Replicant Street, Los Angeles, California 90210--4321}}
% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}
% make the title area
\maketitle
% As a general rule, do not put math, special symbols or citations
% in the abstract
\begin{abstract}
Deep convolutional semantic segmentation (DCSS) learning doesn't converge to an optimal local minimum with random parameters initializations; a pre-trained model on the same domain becomes necessary to achieve convergence.In this work, we propose a joint cooperative end-to-end learning method for DCSS. It addresses many drawbacks with existing deep semantic segmentation learning; the proposed approach simultaneously learn both segmentation and classification; taking away the essential need of the pre-trained model for learning convergence. We present an improved inception based architecture with partial attention gating (PAG) over encoder information. The PAG also adds to achieve faster convergence and better accuracy for segmentation task. We will show the effectiveness of this learning on a diabetic retinopathy classification and segmentation dataset.
\end{abstract}
% no keywords
% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle
\section{Introduction}
Semantic segmentation task using deep learning is formulated as a pixel-wise classification problem \cite{segnet, pspnet, mscoco, gated, largekernel}. Generally, it's an encoder-decoder structure, one encoder to encode the features to lower dimensional features map and one decoder to remap the features map into several probabilistic maps (number of classes) of same width and height as input using bilinear interpolation or transposed convolutions \cite{zdeconv, deconv}. End-to-end learning of the encoder-decoder model best achieved using a pre-trained classification model for the encoder on the same domain. However, for a simple problem with low spatial resolutions input size and fewer number of classes pre-trained model isn't necessary, though it helps speed up the learning process. For complex problems with many numbers of classes and higher resolutions, learning step doesn't converge to the best local optimum without a pre-trained model; leading to inferior validation mean IOU(mean pixel intersection over union). For problems with completely unrelated domain such as medical images, satellite images etc. segmentation becomes two unrelated step; first, define and train a classification model and then use the same for the encoder to train the encoder-decoder model. \\
How do we simplify the segmentation learning for any domains and complexity? Can we formulate the two-step processes into one step; at the same time learning both steps?
We can formulate this as high-level multi-tasks learning concept to train the segmentation system. The idea of pure multi-tasks learning deviates from the main principle behind the simplification to learn one task from the feedback of another task. Here in our case segmentation is dependent on the pre-trained model, achieved using the classification step, which is completely independent of segmentation. In general, all tasks of a usual multi-tasks learning model can be learned independently; but here segmentation can't be learned independently. Another aspect of this attempt is that the classification step should be trainable on loose labels (Section~\ref{randomlabel}) of the dataset.
Addressing the above concerns, in this work, we define a fully end-to-end learning system, that learns classification and segmentation jointly. We will show the model performance on a diabetic retinopathy (DR) classification and features segmentation dataset.\\
\subsection{Contribution}
\textbf{End-to-end joint segmentation and classification learning:} In contrast to the existing semantic segmentation models, the proposed learning system doesn't require a pre-trained model; it learns from the feedback of the classification node.
Advantages of a joint segmentation learning system?
a. The scarcity of labeled segmentation data; this method can achieve better accuracy with the fewer number of segmentation ground-truths.
b. Eliminating the need for pre-trained model requirements for convergence.
c. Training complex model for any domains with larger input size.
d. Learning a classification model as by-product alongside.
\textbf{Improved Residual Inception} We have proposed a new improved residual inception block.
\textbf{Partial Attention:} A partial attention mechanism to use the mid-level encoder layer features for decoder layers.
\textbf{Training with partial random labels:} Learning method that can learn segmentation labeling while cooperatively training a classifier agent on partial random labels.
\subsection{Related Work}
In this section, we will quickly go through some of the early and recent works on semantic segmentation using deep learning. Several recent works using encoder-decoder structure focus on improving the mean IOU for semantic segmentation benchmarks such as PASACAL-VOC\cite{voc_extra}, MS-COCO\cite{mscoco} and biomedical image datasets. One of the first success using the convolutional net for semantic segmentation was achieved in FCN\cite{fcn}; where fully connected layers were replaced using unit strided convolution and upsampling layers initialized with simple bilinear interpolation were used to reconstruct the feature map. In DeconvNet\cite{deconv}, a multilayer deconvolution and unpooling network were used to predict the segmentation masks. Segnet\cite{segnet} decoder uses encoder max-pooling indices to upsample low-resolution feature maps without learning the upsampling layers. Also, they have used convolution layer for each upsampling layer to get dense feature map. Recently, object detection and semantic segmentation were jointly learned in Mask R-CNN\cite{maskrcnn}, they have used FCN to predict segmentation mask for each region of interest. Also several other recent methods such as PSPNet\cite{pspnet}, RefineNet\cite{refinenet}, ENet\cite{enet}, Sharpmask\cite{sharpmask} etc. demonstrated the advantages of encoder-decoder model using convolution and transposed convolution for the task of semantic segmentation. In general, the predicted segmentation mask is coarse and requires post-processing to remove outliers. DenseCRF\cite{crf} is one effective post-processing layer for semantic segmentation \cite{crf1,crf2, crf3}, it refines the segmentation masks exploiting the pixel-level pairwise closeness.
For biomedical image segmentation, deep learning based approach was proposed in Unet\cite{unet}. For diabetic retinopathy detection and microaneurysms segmentation \cite{drhaloi} using deep learning was proposed, this method adopted patch-wise classification for segmentation.
\section{Method}
\subsection{Cooperative Learning}
We formulate the segmentation learning as a cooperative learning problem, where two agents operate on two overlapping domains and interact with each other to achieve their respective goals. The terms overlapping domains implies that the datasets for classification and segmentation are sampled from the same distribution; it's not necessary to have an identical sample for both agents. One agent learns to generate segmentation masks for images with the help of the agent responsible for classification. Even though the classifier agent is capable of self-learning, the interaction with the segmentation agent results in improved generalizability. Each of the agents shares a chunk of parameters but uses separate optimizers to learn their parameters. The shared parameters of the agents are learned by both of the agents at a different rate; this is where the agents share knowledge and learn cooperatively. Learning rate for classifier agent is higher than that of segmentation agent as it possesses class specific discriminative knowledge about the domain. Outputs of the two agents Eq~\ref{eg:agents} for some input $x$ is calculated using the shared parameters $M$; $y_{common}$ denotes the output from the shared model part, $y_{clf}$ denotes the output of the classifier agent with private parameters $C$ and $y_{seg}$ denotes the output of the segmentation agent with private parameters $S$.
\begin{equation}
\label{eg:agents}
\begin{aligned}
y_{common} = M(x) \\
y_{clf} = C(M(x)) \\
y_{seg} = S(M(x))
\end{aligned}
\end{equation}
\begin{figure}[h]
\centering
\includegraphics[width=3.1in,height=1.7in]{agent}
\caption{Two agents and their shared parameters net M}
\label{fig:conv_pool}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=3.1in,height=2.7in]{incept}
\caption{Residual Inception}
\label{fig:incept}
\end{figure}
\begin{equation}
\label{eq:loss}
\begin{aligned}
J_{clf} = \min\limits_{\theta_{1} \cup \theta_{2}} Cost(C(M(x)), y_{clf}^{label}) \\
J_{seg} = \min\limits_{\theta_{1} \cup \theta_{3}} Cost(S(M(x)), y_{seg}^{label}) \\
\theta_{1} \in M, \theta_{2} \in C, \theta_3 \in S
\end{aligned}
\end{equation}
Softmax cross-entropy cost Eq~\ref{eq:loss} was used to learn both agents. For the segmentation agent design and learning context, $M$ is refereed as the encoder and $S$ as the decoder. Fig~\ref{fig:conv_pool} shows the interaction between the two agents; sharing a common set of parameters $M$.
\subsection{Network Architecture}
\textbf{Residual Inception module} We have used a modified version of the base inception block proposed in \cite{incept1}, where an additional convolution was used before max-pooling to ensure rich feature representation. We have added a residual connection from the block input to output as shown in Fig~\ref{fig:incept}. The concatenated features were convolved with another $3\times3$ convolution to reduce aliasing and added with $1\times1$ convolved input features; both operations have the same number of output filters. Apart from that, we have used a convolution factorization version Fig~\ref{fig:incept2} of the same module for the classifier node. The idea of convolution factorization was proposed in \cite{incept_google}, it reduces the number of parameters without effecting the generalization performance.
\textbf{Partial attention}
We use partial attention over the selected encoder feature maps. Feeding the encoder information with upsampling boost the reconstructed feature map representation power. The attention mechanism here only focuses on the encoder feature maps with spatial resolutions equal or larger than that of the upsampling outputs. Max-pooling to reduce spatial resolution of encoder feature maps to the upsampling output size; max-pooling ensures maximally activated features. Fig~\ref{fig:attention} shows an overview of the partial attention mechanism used, where $E_i$ are the last feature maps (before reducing the feature map width and height for each block) of each convolution block of the encoder. The switch $s$ is on if the corresponding $a_i > 0$; $op$ is a max-pooling operation for all $a_i >1$ with stride $a_{i}$ and identity operation for $a_i=1$.
\begin{equation}
a_i = \frac{min(height_{E_{i}}, width_{E_{i}})}{min(height_{U_{i}}, width_{U_{i}})}
\end{equation}
\begin{figure}[h]
\centering
\includegraphics[width=3.1in,height=2.7in]{incept2}
\caption{Residual Inception with convolution factorization for the classifier node}
\label{fig:incept2}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=3.2in,height=2.3in]{attention}
\caption{Partial attention over encoder}
\label{fig:attention}
\end{figure}
\textbf{Cooperative learning model}
follows the compositional design principles stated in \cite{deepcomp}; strided convolution is used to down-sample the encoder spatial resolution. To achieve richer feature representation, residual inception block as shown in Fig~\ref{fig:incept} is used for the encoder.
The encoder last layer output is used as the input of both the classifier and the segmentation agents. Classifier agent private node $C$ is designed using the residual factorized inception block as shown in Fig~\ref{fig:incept2}, followed by a global pool layer and a fully connected layer with a number of classes for the classifier. For each spatial dimension level of the encoder and the classifier private node, multiples cascaded residual inception blocks were used. Fig~\ref{fig:cooperative_model} shows the complete design of the joint segmentation and classification learning.
The decoder is designed using consecutive transposed convolutions with $2\times2$ stride and $2\times2$ kernel size followed by a partial attention block to incorporate encoder information for recovering object information. The partial attention block $G$ attend over few selective encoder feature maps with a spatial dimension larger or equal than that of the upsampled decoder output. Encoder feature maps learn input object information at multiple abstractions; the attention mechanism enables faster convergence of decoder parameters with improved generalization performance.
\subsection{Training on partial random labels}
\label{randomlabel}
The cooperative learning model can also be used when the number of labeled samples for the classifier agent is small. The classifier agent trained on partial random labels doesn't downgrade the segmentation agent performance. When we use around 20\% of labeled samples for classifier and around 80\% samples with random labels accuracy of classifier agent are lowered without effecting the performance of the segmentation agent. This observation indicates the ability of both of the agents to learn cooperatively. Also, this helps towards the goal of training on a dataset with the very low number of labeled images; especially relevant on medical image domains.
\section{Experiment Setup}
\textbf{Dataset:} We train the classification agent on the Eyepacs diabetic retinopathy classification dataset \cite{eyepacs} with 5 classes. The 5 classes of this dataset indicate DR severity level; level 0 for no DR, level 1 for mild DR, level 2 for moderate DR, level 3 for severe DR and 4 for proliferative DR. For training 25K images were used and validation performance was calculated on a set of 10k images. For testing, we have used 55K test set provided by eyepacs. The test set was labeled internally by ophthalmologists. Each image was resized to $224 \times 224 \times 3$.
\begin{figure*}
\centering
\includegraphics[width=5.3in,height=2.7in]{main_c}
\caption{Cooperative learning model with partial attention}
\label{fig:cooperative_model}
\end{figure*}
The segmentation agent is trained on a dataset prepared with the help of ophthalmologists; We selected a subset of around 5K images from Eyepacs dataset and annotated retinal pathological features with mask-based annotation. This dataset contains the annotation of around 5 pathological features (hard exudates, soft exudates, hemorrhage, microaneurysms, drusen) with one additional mask annotation for retinal artifacts. The set is split into training and validation set; 3.5K images for training and 1.5K images were used for validation.
\textbf{Preprocessing:}
For both the classifier and the segmentation, $224\times224\times3$ is used as the input size. All input images were resized into $256\times256\times3$ and a randomly cropped patch of size $224\times224\times3$ is used as input. Extensive data augmentation is used such as random flip left/right, up/down and random padding was used for both classification and segmentation. Segmentation masks were also processed with same augmentation its corresponding image. Additionally, for classification additional augmentation such as for as changing the image pixel values randomly using hue, contrast and saturation were used. Also, each image is standardized by it's mean and dividing its standard deviation.
\textbf{Framework:} We have used TEFLA\cite{tefla}, a python framework developed on the top of TENSORFLOW\cite{tf}, for all experiments described in this work.
\textbf{Training procedure:} Batch normalization\cite{bn} is used to reduce covariate shift and achieve faster convergence of both agents. Also, we have used Nesterov momentum optimizer with polynomial learning rate policy. For the classifier agent, the learning rate is 30 times than that of the segmentation agent. The batch size used for the classifier agent is 32 times than that of the segmentation agent. The use of higher batch size is to facilitate faster learning and information transfer from the classifier agent to the segmentation agent. We use lower learning rate for the segmentation agent to make sure that the information it passes to the classifier agent is minimal, but aid to generalization.
\begin{figure*}
\centering
\begin{tabular}{cccc}
\includegraphics[width=42mm]{2532_right.jpg} & \includegraphics[width=42mm]{2532_right_gt.jpg} & \includegraphics[width=42mm]{2532_right_1.jpg} & \includegraphics[width=42mm]{2532_right_2.jpg}\\
\includegraphics[width=42mm]{11896_right.jpg} & \includegraphics[width=42mm]{11896_right_gt.jpg} & \includegraphics[width=42mm]{11896_right_1.jpg} & \includegraphics[width=42mm]{11896_right_2.jpg}\\
\includegraphics[width=42mm]{15414_right.jpg} & \includegraphics[width=42mm]{15414_right_gt.jpg} & \includegraphics[width=42mm]{15414_right_1.jpg} & \includegraphics[width=42mm]{15414_right_2.jpg}\\
Original Image & Ground truth & Unet\_inception & JointSeg[This work] \\
\end{tabular}
\caption{Results comparison of segmentations}.
\label{fig:seg_result}
\end{figure*}
\subsection{Results Analysis}
We have evaluated out model performance on the eyepacs test set of 55k images. We have also trained VGG-19\cite{vgg} and inceptionv3\cite{incept_google} to compare classification agent performance with this proposed approach. Also for the segmentation agent performance comparisons we have trained FCN-8s\cite{fcn} and UNet\cite{unet} on the segmentation set. Here we use the improved inception block for UNet training; which is an improved version of the original UNet design.
Table~\ref{table:iou} shows the mean-IOU comparisons with the proposed approach and the existing algorithms. A significant performance gain is obtained with join segmentation learning. Here fcn-8s and Unet models parameters were initialized with a pre-trained model; trained on eyepacs classification set.
\begin{table}[h]
\begin{center}
\begin{tabular}{ | p{3cm} | c | }
\hline
Method & mIOU \\ \hline
FCN-8s & 55.3 \\ \hline
UNet & 64.4 \\ \hline
JointSeg [This work] & 66.1 \\
\hline
\end{tabular}
\end{center}
\caption{mIOU comparison on segmentation validation set using different methods}
\label{table:iou}
\end{table}
If we randomly initialize the FCN-8s and UNet model parameters; results become non-informatory as shown in Table~\ref{table:noinit}; both FCN-8s and UNet could not achieve convergence to an optimal local minimum. From Table~\ref{table:iou} and Table~\ref{table:noinit} we can understand the limitations and the importance of pre-trained model for traditional segmentation learning. But the joint learning achieves convergence to an optimal local minimum with random initializations.
\begin{table}[h]
\begin{center}
\begin{tabular}{ | p{3cm} | c | }
\hline
Method & mIOU \\ \hline
fcn-8s & 15.3 \\ \hline
Unet & 25.4 \\ \hline
JointSeg [This work] & 66.1 \\
\hline
\end{tabular}
\end{center}
\caption{mIOU comparison with random parameters initializations}
\label{table:noinit}
\end{table}
Table~\ref{table:topk} shows the performance of the classifier agent with standard deep learning classifier model.
\begin{table}[h]
\begin{center}
\begin{tabular}{ | p{3cm} | c | c|}
\hline
Method & Top-1 (\%) & Top-2 (\%) \\ \hline
VGG & 82.5 & 85.3 \\ \hline
InceptionV3 & 84.3 & 87.2 \\ \hline
JointSeg [This work] & 85.1 & 88.1\\
\hline
\end{tabular}
\end{center}
\caption{Accuracy comparison on classification test set using different methods}
\label{table:topk}
\end{table}
Table~\ref{table:bin} shows the classifier agent performance on a binary classification scenario. For a DR screening system to understand whether a person has DR, a binary label is enough. For this test, we combine all images with DR level higher than or equal to 1 as abnormal and DR level 0 as normal. The agent outperforms a normal human ophthalmologist for binary classification.
\begin{table}[h]
\begin{center}
\begin{tabular}{ | p{3cm} | c |}
\hline
Method & Binary Accuracy (\%) \\ \hline
VGG & 89.5 \\ \hline
InceptionV3 & 94.7 \\ \hline
JointSeg [This work] & 95.5\\ \hline
Human Performance & 95.1\\
\hline
\end{tabular}
\end{center}
\caption{Binary Accuracy comparison on classification test set using different methods}
\label{table:bin}
\end{table}
Figure~\ref{fig:seg_result} shows the qualitative performance of the segmentation agent with respect to the ground truths and modified UNet design. Joint Segmentation performs better than improved UNet design. Considering the complexity of this problem to identify very small lesions; the proposed architecture was able to learn meaningful features without a pre-trained model.
% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation results for the network.}
%\label{fig_sim}
%\end{figure}
% Note that the IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.
% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command,
% and the \label for the overall figure must come after \caption.
% \hfil is used as a separator to get equal spacing.
% Watch out that the combined width of all the subfigures on a
% line do not exceed the text width or a line break will occur.
%
%\begin{figure*}[!t]
%\centering
%\subfloat[Case I]{\includegraphics[width=2.5in]{box}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{box}%
%\label{fig_second_case}}
%\caption{Simulation results for the network.}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat[]), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.
% Be aware that for subfig.sty to generate the (a), (b), etc., subfigure
% labels, the optional argument to \subfloat must be present. If a
% subcaption is not desired, just leave its contents blank,
% e.g., \subfloat[].
% An example of a floating table. Note that, for IEEE style tables, the
% \caption command should come BEFORE the table and, given that table
% captions serve much like titles, are usually capitalized except for words
% such as a, an, and, as, at, but, by, for, in, nor, of, on, or, the, to
% and up, which are usually not capitalized unless they are the first or
% last word of the caption. Table text will default to \footnotesize as
% the IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}
% Note that the IEEE does not put floats in the very first column
% - or typically anywhere on the first page for that matter. Also,
% in-text middle ("here") positioning is typically not used, but it
% is allowed and encouraged for Computer Society conferences (but
% not Computer Society journals). Most IEEE journals/conferences use
% top floats exclusively.
% Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the
% \fnbelowfloat command of the stfloats package.
\section{Conclusion}
We have presented an end-to-end deep convolutional semantic segmentation learning method that achieves faster convergence and better accuracy with random parameters initializations. Also, the network learns both classification and segmentation on the same domain. We also showed the performance of our methods on biomedical image processing dataset eyepacs to detect and classify diabetic retinopathy. One of the important aspects of this work is that the classifier agent can be trained on lose/ partial random labels without degrading segmentation agent performance; this helps towards solving the label scarcity problems of medical image domain.
% conference papers do not normally have an appendix
% use section* for acknowledgment
\section*{Acknowledgment}
We would like to thank the data team leads RajaRajalakshmi Kodhandapani and Ophthalmologists of Artelus; who have helped to prepare the segmentation dataset. Also, I would like to thank colleagues Anurag Sahay and Lalit Pant for their valuable advice during this work.
% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}
% references section
% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://mirror.ctan.org/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
%\bibliographystyle{IEEEtran}
% argument is your BibTeX string definitions and bibliography database(s)
%\bibliography{IEEEabrv,../bib/paper}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
\begin{thebibliography}{1}
\bibitem{fcn}
J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015
\bibitem{gated}
M. A. Islam, M. Rochan, N. D. Bruce, and Y. Wang. Gated feedback refinement network for dense image labeling. In CVPR, 2017
\bibitem{refinenet}
G. Lin, A. Milan, C. Shen, and I. Reid. Refinenet: Multipath refinement networks with identity mappings for high-resolution semantic segmentation. arXiv:1611.06612, 2016
\bibitem{largekernel}
C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun. Large kernel matters–improve semantic segmentation by global convolutional network.arXiv:1703.02719, 2017.
\bibitem{zdeconv}
M. D. Zeiler, G. W. Taylor, and R. Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In ICCV, 2011
\bibitem{deconv}
H. Noh, S. Hong, and B. Han. Learning deconvolution network for semantic segmentation.
In ICCV, 2015.
\bibitem{incept1}
Haloi, M. Traffic sign classification using deep inception based convolutional networks. arXiv preprint arXiv:1511.02992, 2015.
\bibitem{deepcomp}
Haloi, M. Deep Learning: Generalization Requires Deep Compositional Feature Space Design. arXiv preprint arXiv:1706.01983, 2017.
\bibitem{incept_google}
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2818-2826), 2016.
\bibitem{pspnet}
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. arXiv:1612.01105, 2016.
\bibitem{enet}
A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. arXiv preprint arXiv:1606.02147, 2016.
\bibitem{maskrcnn}
He, K., Gkioxari, G., Dollár, P., Girshick, R. (2017). Mask r-cnn. arXiv preprint arXiv:1703.06870.
\bibitem{sharpmask}
Pinheiro, P. O., Lin, T. Y., Collobert, R., Dollár, P. (2016). Learning to refine object segments. In European Conference on Computer Vision (pp. 75-91). Springer International Publishing.
\bibitem{crf}
P. Krahenbuhl and V. Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NIPS, 2011.
\bibitem{segnet}
V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv:1511.00561, 2015.
\bibitem{mscoco}
Z. Wu, C. Shen, and A. van den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. arXiv:1611.10080, 2016.
\bibitem{crf1}
L.C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015
\bibitem{crf2}
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L.Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv:1606.00915, 2016.
\bibitem{crf3}
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Torr, P. H. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1529-1537), 2015.
\bibitem{drhaloi}Haloi, M. Improved microaneurysm detection using deep neural networks. arXiv preprint arXiv:1505.04424, 2015.
\bibitem{unet}
Ronneberger, O., Fischer, P., Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 234-241). Springer, Cham.
\bibitem{bn}
S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.
\bibitem{vgg}
Simonyan, K., Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
\bibitem{tefla} https://github.com/openagi/tefla
\bibitem{tf} Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Ghemawat, S. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.0446, 2016.
\bibitem{imagenet}
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet
large scale visual recognition challenge. arXiv:1409.0575, 2014.
\bibitem{eyepacs} https://www.kaggle.com/c/diabetic-retinopathy-detection
\bibitem{voc_extra}
B. Hariharan, P. Arbelaez, L. D. Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In ICCV, 2011.
\end{thebibliography}
% that's all folks
\end{document}