Fast and Accurate Semi-Automatic Segmentation Tool for Brain Tumor MRIs
Extracted source text · arXiv 1705.06823
\documentclass[review,authoryear]{elsarticle}
\graphicspath{ {../Figures/} }
\usepackage[labelfont=bf]{caption}
\captionsetup{labelfont=bf}
\usepackage{lineno,hyperref,natbib,multirow}
%\modulolinenumbers[0]
\journal{arXiv}
\bibliographystyle{model2-names}
%% Highlights
% A software for semi-automatic tumor segmentation based on MITK is presented
% The accuracy of segmentations is validated against reference standards
% The speed of the protocol is 4-11 times faster than similar algorithms
%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
\begin{frontmatter}
\title{Fast and Accurate Semi-Automatic Segmentation Tool for Brain Tumor MRIs}
\author[systems]{Andrew X. Chen}
\author[systems,dbmi]{Ra\'{u}l Rabad\'{a}n\corref{mycorrespondingauthor}}
\cortext[mycorrespondingauthor]{Corresponding author}
\ead{rr2579@cumc.columbia.edu}
\address[systems]{Department of Systems Biology, Columbia University Medical Center, 1130 St. Nicholas Avenue, New York, NY 10032, USA}
\address[dbmi]{Department of Biomedical Informatics, Columbia University Medical Center, 622 W. 168th Street, New York, NY 10032, USA}
\begin{abstract}
Segmentation, the process of delineating tumor apart from healthy tissue, is a vital part of both the clinical assessment and the quantitative analysis of brain cancers. Here, we provide an open-source algorithm (MITKats), built on the Medical Imaging Interaction Toolkit, to provide user-friendly and expedient tools for semi-automatic segmentation. To evaluate its performance against competing algorithms, we applied MITKats to 38 high-grade glioma cases from publicly available benchmarks. The similarity of the segmentations to expert-delineated ground truths approached the discrepancies among different manual raters, the theoretically maximal precision. The average time spent on each segmentation was 5 minutes, making MITKats between 4 and 11 times faster than competing semi-automatic algorithms, while retaining similar accuracy.\\
\end{abstract}
\begin{keyword}
Semi-automatic segmentation; Brain tumor; Glioblastoma; MRI; MITK
\end{keyword}
\end{frontmatter}
\newpage
%\linenumbers
\section{Introduction}
Glioblastoma (GBM) is the most common primary brain malignancy, and the one with the most dismal prognosis \citep{DeAngelis2001}. Imaging, particularly magnetic resonance imaging (MRI), is the standard for diagnosing and assessing the disease \citep{Mabray2015}. The delineation of tumor volumes within images, or segmentation, is important for guiding therapy \citep{Dupont2016, Bauer2013a}, determining prognosis \citep{Kickingereder2016, Cui2015}, and assessing response\citep{Chow2014, Clarke1998}. Related quantitative spatial analyses have demonstrated utility in predicting molecular subtypes \citep{Yang2016} and survival \citep{Czarnek2017, Mazurowski2014, Zhang2014a} of glioblastoma patients. Given the availability of public imaging datasets such as The Cancer Imaging Archive \citep{Clark2013}, there is a clear need for the development of efficient and accurate segmentation utilities, which will allow for the systematic quantification of images in association with clinical and molecular characteristics.
\subsection{Related Work}
Manual segmentation is typically performed with 2D tools to delineate edges on each image slice. While manual segmentation is the gold standard \citep{Porz2014}, it is too time-consuming for large analyses, sometimes taking upwards of an hour per image series \citep{Kaus2001}. Because of this, many fully automatic and semi-automatic algorithms have been created to expedite the segmentation process. Reviews of these methods can be found in \citet{Bauer2013} and \citet{Wang2014}. Benchmarks of fully automatic algorithms have demonstrated encouraging accuracies \citep{Menze2015}, but acceptance of these methods in the clinic is limited due to concerns about errors and transparency \citep{Gordillo2013}. Semi-automatic algorithms draw a balance by unifying the power of computer processing with the intuition of the human operator. However, existing semi-automatic programs still need improvement with regards to operator time \citep{Fyllingen2016} and user-friendliness \citep{Ramkumar2016}.
\subsection{Contribution}
To address these drawbacks, this work provides and validates an accessible semi-automatic protocol for the fast segmentation of glioblastomas. Specific goals included the creation of an intuitive computer-assisted segmentation utility, addition of 3D editing tools for manual correction, and integration within a user-friendly environment. These aims were implemented in the Medical Imaging Interaction Toolkit (MITK) as an extension of its existing Segmentation plugin \citep{Wolf2005, Maleike2009}. This modified software, MITK with augmented tools for segmentation (MITKats), is also a free and open-source program. By addressing the aforementioned goals with MITKats, we expedite semi-automatic segmentation by introducing validated, easy-to-use 3D tools.
\section{Methods}
\subsection{Data sources}
A total of 38 3D MRIs of brain tumors were obtained from two publicly available datasets. 20 cases of high-grade gliomas (including glioblastomas and anaplastic astrocytomas) were obtained from the Multimodal Brain Tumor Segmentation Challenge (BRATS) 2012 \citep{Menze2015}, a notable accuracy benchmark. Four modalities were available for the BRATS dataset: T1, T1 with contrast (T1c), T2, and T2 FLAIR, though MITKats only used T1c and FLAIR images. The ground truths were derived from all four modalities and designated into 3 regions:
\begin{itemize}
\item Active: The contrast-enhancing parts of the tumor, seen on T1c
\item Core: The Active component plus non-enhancing features seen on T1, including necrosis
\item Whole: The hyper-intense region on T2 and FLAIR, corresponding to edema
\end{itemize}
18 cases of glioblastoma were obtained from a previously published study performed at St. Olav's University Hospital \citep{Fyllingen2016}. Only T1c MRIs were used to create segmentations, which were generated in BrainVoyager \citep{Goebel2006}, 3D Slicer \citep{Egger2013}, and ITK-SNAP \citep{Yushkevich2006}. A single tumor region was labeled, comprising enhancing and necrotic regions. Given that the authors did not observe non-enhancing tumor regions, their definition of tumor was closest to that of the Core component from BRATS. While no ground truths were designated in this dataset, each segmentation was timed, serving as a speed benchmark.
Both datasets had originally been pre-processed, resulting in interpolation of image resolutions to a 1mm isotropic voxel size. Images from BRATS had been skull-stripped, while those from St. Olav's had not.
\subsection{Algorithmic Development}
Two modifications were made to the MITK framework in order to expedite the segmentation process in MITKats. First, the Threshold Components tool was added, which expanded connected threshold segmentation to accept multiple seed points (and thus separated regions). It also allowed for the manipulation of seed points independently of the thresholds, as well as streamlining out unnecessary user input. The second modification was adding in segmentation capability to the Clipping Plane View, which was originally intended for volume measurements only. Therefore, a segmentation could be graphically adjusted in three dimensions through extraction of a clipped piece.
MITKats can be found here: https://github.com/RabadanLab/MITKats, and is in the process of being merged onto the main branch of MITK.
\subsection{Segmentation Protocol}
Segmentations were performed in MITKats by A.X.C., a medical student who had used similar software to segment 93 glioblastoma cases as part of a previous project \citep{Crawford}. Segmentation time, as measured by stopwatch, was started after loading the original image(s) and stopped after opening the save segmentation dialog, thus ignoring the time for file operations.
\begin{figure}[htp!]
\includegraphics[width=\textwidth]{Figure1labeled}
\caption{\textbf{Demonstration of a user-friendly semiautomatic segmentation protocol.} a) The Threshold Components tool allows the user to set an intensity threshold and multiple seedpoints (yellow crosses). Regions within the threshold and connected to the seedpoints are selected. b) The deformable clipping plane allows 3D correction of the segmentation, typically useful for removing leaked regions. c) Original image, clipped image, and final morphological operations such as Closing and Fill Holes smooth the enhancing segmentation into a Core segmentation.}
\label{fig:protocol}
\end{figure}
\subsubsection{Thresholding}
A new segmentation label was created for the T1c image, and the Segmentation plugin view is opened. Selecting the newly implemented Threshold Components tool, seed point(s) were placed in enhancing region(s), and the lower threshold adjusted until the apparent hyper-intensities were all included (Fig. 1a). If the tumor was grossly non-enhancing, another label was created where seed point(s) were placed in hypo-intense regions, and the upper threshold adjusted.
\subsubsection{Cropping}
Regions of normal brain tissue that were erroneously included as part of the Thresholding process were removed in two ways. Precise exclusion of regions was done via the Clipping Plane tool, where up to 6 deformable 3D surfaces were superimposed on the segmentation. This allowed the generalized separation of erroneous regions from the tumor body (Fig. 1b), a new feature added in MITKats. For gross corrections, the existing Image Cropping tool could be used instead. A 3D rectangular bounding box was graphically defined and used to mask and overwrite the original segmentation. Finally, if the true tumor contained only one connected component, either of these methods could be followed up with the Picking tool to exclude erroneous regions that were severed from the tumor.
\subsubsection{Smoothing and Filling}
The segmentation was smoothed via the Closing tool, a part of the existing set of Morphological Operations, typically with a radius of 2. This label was saved as the Active component. For the Core volume, the non-enhancing label could be joined to this component via the Union tool, if applicable. To include areas of necrosis, the Closing and Fill Holes tools were used (both typically with radius of 10), and this new label was saved as the Core component (Fig. 1c).
\subsubsection{Whole Tumor}
For segmenting the Whole tumor component of the BRATS dataset, the above protocol for obtaining the Core region was repeated for the FLAIR image.
\subsection{Accuracy and Speed Analysis}
The accuracy of MITKats segmentations was assessed by comparison to the reference segmentations provided by BRATS and St. Olav's datasets. In BRATS, the references were ground truths fused from 4 expert manual annotations. Each of the Whole, Core, and Active components were compared to their reference counterparts via Dice score \citep{Dice1945} as well as calculated tumor volume. The mean inter-rater Dice scores from BRATS were used as controls. This was because the fused ground truths were derived from the individual expert annotations, artificially bolstering the Dice score between any given expert and the fused standard.
For comparing data in the St. Olav's cohort, the MITKats segmentations were assessed via Dice score and volume to all 12 reference segmentations (3 softwares $ \times $ 2 raters $ \times $ 2 repetitions). As a control, Dice scores were calculated only for pairs of reference segmentations created by different raters. The time to create the Core component of the MITKats segmentation was compared against the times reported in the dataset.
\section{Results}
Aggregate Dice scores and volume estimates of MITKats segmentations with respect to BRATS ground truths for each tumor region are shown in Table \ref{tab:brats}. The Dice similarity of MITKats segmentations compared to the ground truth was equivalent to inter-rater variability for the Whole and Active tumor regions, but was worse for the Core region. Volume measurements averaged over all regions were 94\% of those estimated by the ground truth, with a mean fractional error of 18\%.
\begin{table}[h]
\begin{center}
\begin{tabular}{|c|c|c|c|c|}
\hline
\multicolumn{2}{|c|}{Tumor Compartment} & Whole & Core & Active \\
\hline
\multirow{2}{*}{Dice Score (\%)} & MITKats v. BRATS & 88$\pm $5 & 84$\pm $10 & 77$\pm $14 \\
\cline{2-5}
& BRATS Inter-rater & 88$\pm $2 & 93$\pm $3 & 74$\pm $13 \\
\hline
\multicolumn{2}{|c|}{Volume Ratio} & 0.96 & 1.06 & 0.81 \\
\hline
\multicolumn{2}{|c|}{Volume Relative Error} & 10\% & 25\% & 20\% \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Segmentation accuracy approaches inter-rater agreement in the Brain Tumor Segmentation Challenge.} Twenty high-grade glioma cases were segmented using MITKats and compared against ground truths from BRATS. Three regions (Whole, Core, and Active) were segmented for each patient, and the mean Dice scores ($\pm$ standard deviation) are shown. The volumes of each region were also compared to the ground truth.}
\label{tab:brats}
\end{table}
A comparison of estimated volumes for each case across both datasets is shown in Figure \ref{fig:volume}, including a logarithmic Bland-Altman analysis. While the volumes segmented by MITKats were similar to reference segmentations, they tended to be underestimated, particularly for the Active tumor region of the BRATS dataset.
The Core component of the MITKats segmentation was pairwise compared to segmentations performed by other softwares in the St. Olav dataset. The average Dice scores of MITKats segmentations compared to each of the reference segmentations was 0.88, compared against their inter-rater agreement of 0.94 (Figure \ref{fig:speed}a).
The time for segmentation is also compared to the St. Olav's dataset in Figure \ref{fig:speed}b. The speed of MITKats as compared to those reported was an average of 4, 5, and 11 times faster than ITK-SNAP, 3D Slicer, and BrainVoyager, respectively. The typical Core segmentation using MITKats was 4.2 $\pm $ 2.0 minutes on the St. Olav's dataset and 4.0 $\pm $ 3.1 minutes on the BRATS dataset, where uncertainties represent standard deviation.
\begin{figure}[htp!]
\includegraphics[width=\textwidth]{Figure2}
\caption{\textbf{Comparison of segmented volumes by tumor region.} a) Volumes calculated using MITKats are compared against their reference standards for each tumor component. (St. Olav segmentations designated only a single Core-like component.) b) A logarithmic Bland-Altman plot is shown for different regions of segmentations, comparing the ratio of segmented volumes to the averages of the base 10 logarithms of tumor volume (in mL). There is a tendency for MITKats to underestimate tumor volume at low sizes, particularly in the Active tumor component.}
\label{fig:volume}
\end{figure}
\begin{figure}[htp!]
\includegraphics[width=\textwidth]{Figure3}
\caption{\textbf{MITKats is faster than other segmentation softwares, while approaching optimal accuracy.} a) Eighteen glioblastoma cases were segmented using MITKats and compared against the segmentations performed in \citet{Fyllingen2016}. Points represent the average Dice score when compared to all other segmentations performed by different raters. Error bars represent standard deviation. b) The time required for segmenting via MITKats is compared to those using BrainVoyager, 3DSlicer, and ITK-SNAP (mean of 2 trials each).
}
\label{fig:speed}
\end{figure}
\section{Discussion and Conclusion}
To our knowledge, this work is the first to validate the use of the MITK environment for the segmentation of brain tumors. Our modification MITKats provides fast and accurate segmentation, combining a semiautomatic tool with flexible 3D editing, all wrapped in a user-friendly GUI. Benchmarking its accuracy against BRATS, it achieved a performance equal to inter-rater variability across Whole and Active tumor regions. The Core tumor region was somewhat lacking, perhaps due to our protocol not using T1 images, thus having different definitions for the component. Benchmarking its speed against the St. Olav's dataset, MITKats was over 4 times faster than its quickest competitor. While its accuracy was somewhat lower than inter-rater variability, the reference segmentations were also semi-automatically derived, and therefore not necessarily the ground truth. Finally, MITK is an open source toolkit which encourages the free use and continued development of this software.
The scope of this work has some limitations with the protocol and datasets used. This pipeline works best for T1c and FLAIR imagery, because T1 without contrast has limited enhancement, and T2 enhancements are often the same intensity as cerebrospinal fluid. We hope to provide implementation for other modalities in the future as well as simultaneous multimodal analysis, which may benefit the accuracy of segmentations \citep{Dupont2016}. Providing support for perfusion and diffusion weighted images may also be particularly important for response assessment \citep{Huang2015}.
One note is that the BRATS dataset contained not just glioblastomas but also anaplastic astrocytomas, which are intermediates between GBMs and lower grade gliomas (LGGs). Given the satisfactory accuracy of segmenting this mixed dataset, this provides encouragement that MITKats could be extended to LGGs, which are typically harder to segment \citep{Akkus2015}. Eventual implementation and validation for LGGs and tumors outside the brain would be useful. Another point about the BRATS dataset is that the images were already skull-stripped, making the segmentation process somewhat easier. Integrating skull-stripping into our pipeline would be helpful for the user.
It is our belief that speed is not sufficiently emphasized in the current benchmarking of segmentation algorithms. While many studies report average segmentation times, the variability among different cases can be large. Within the datasets examined in this work, the fastest segmentation took less than a minute, while the slowest took almost 11 minutes. Therefore, the individual listing of segmentation times is helpful for the direct comparison of algorithm speeds. We encourage future investigators to record both raw segmentations and operator time for each case to further advance this field.
%\section*{References}
\bibliography{mybibfile}
\section*{Acknowledgments}
We would like to thank Anthea Monod and the rest of the Rabadan lab for their helpful comments and feedback. A.X.C. is funded by NIH grant T32GM007367 as part of the Medical Scientist Training Program (MSTP). R.R. acknowledges funding from the NIH (U54 CA193313, R01 CA185486, R01 CA179044).
%\section*{Vitae}
%Andrew Chen (A.X.C.) is an MD/PhD candidate at Columbia University. Previously, he was an undergraduate at MIT, where he obtained degrees in Physics and Biology. He is interested in applying computational methods to medical image processing and cancer evolution.
%
%Ra\'{u}l Rabad\'{a}n (R.R.) is an Associate Professor in the Department of Systems Biology and Biomedical Informatics at Columbia University. He is the director of the Center for Topology of Cancer Evolution and Heterogeneity. Previously, he was the Martin A. and Helen Chooljian Member at The Simons Center for Systems Biology at the Institute for Advanced Study in Princeton, New Jersey. His current interest focuses on uncovering patterns of evolution in biological systems—in particular, RNA viruses and cancer.
\end{document}
\begin{thebibliography}{30}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\DOIprefix}{doi:}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{http://dx.doi.org/#1}{\path{#1}}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Article
\bibitem[{Akkus et~al.(2015)Akkus, Sedlar, Coufalova, Korfiatis, Kline, Warner,
Agrawal and Erickson}]{Akkus2015}
\bibinfo{author}{Akkus, Z.}, \bibinfo{author}{Sedlar, J.},
\bibinfo{author}{Coufalova, L.}, \bibinfo{author}{Korfiatis, P.},
\bibinfo{author}{Kline, T.L.}, \bibinfo{author}{Warner, J.D.},
\bibinfo{author}{Agrawal, J.}, \bibinfo{author}{Erickson, B.J.},
\bibinfo{year}{2015}.
\newblock \bibinfo{title}{{Semi-automated segmentation of pre-operative low
grade gliomas in magnetic resonance imaging}}.
\newblock \bibinfo{journal}{Cancer Imaging} \bibinfo{volume}{15},
\bibinfo{pages}{1--10}.
\newblock \DOIprefix\doi{10.1186/s40644-015-0047-z}.
%Type = Article
\bibitem[{Bauer et~al.(2013a)Bauer, Lu, May, Nolte, B{\"{u}}chler and
Reyes}]{Bauer2013a}
\bibinfo{author}{Bauer, S.}, \bibinfo{author}{Lu, H.}, \bibinfo{author}{May,
C.P.}, \bibinfo{author}{Nolte, L.P.}, \bibinfo{author}{B{\"{u}}chler, P.},
\bibinfo{author}{Reyes, M.}, \bibinfo{year}{2013}a.
\newblock \bibinfo{title}{{Integrated segmentation of brain tumor images for
radiotherapy and neurosurgery}}.
\newblock \bibinfo{journal}{International Journal of Imaging Systems and
Technology} \bibinfo{volume}{23}, \bibinfo{pages}{59--63}.
\newblock \DOIprefix\doi{10.1002/ima.22037}.
%Type = Article
\bibitem[{Bauer et~al.(2013b)Bauer, Wiest, Nolte and Reyes}]{Bauer2013}
\bibinfo{author}{Bauer, S.}, \bibinfo{author}{Wiest, R.},
\bibinfo{author}{Nolte, L.P.}, \bibinfo{author}{Reyes, M.},
\bibinfo{year}{2013}b.
\newblock \bibinfo{title}{{A survey of MRI-based medical image analysis for
brain tumor studies}}.
\newblock \bibinfo{journal}{Physics in Medicine and Biology}
\bibinfo{volume}{58}, \bibinfo{pages}{R97--R129}.
\newblock \DOIprefix\doi{10.1088/0031-9155/58/13/R97}.
%Type = Article
\bibitem[{Chow et~al.(2014)Chow, Qi, Guo, Miloushev, Iwamoto, Bruce, Lassman,
Schwartz, Lignelli, Zhao and Filippi}]{Chow2014}
\bibinfo{author}{Chow, D.S.}, \bibinfo{author}{Qi, J.}, \bibinfo{author}{Guo,
X.}, \bibinfo{author}{Miloushev, V.Z.}, \bibinfo{author}{Iwamoto, F.M.},
\bibinfo{author}{Bruce, J.N.}, \bibinfo{author}{Lassman, A.B.},
\bibinfo{author}{Schwartz, L.H.}, \bibinfo{author}{Lignelli, A.},
\bibinfo{author}{Zhao, B.}, \bibinfo{author}{Filippi, C.G.},
\bibinfo{year}{2014}.
\newblock \bibinfo{title}{{Semiautomated volumetric measurement on postcontrast
MR imaging for analysis of recurrent and residual disease in glioblastoma
multiforme}}.
\newblock \bibinfo{journal}{American Journal of Neuroradiology}
\bibinfo{volume}{35}, \bibinfo{pages}{498--503}.
\newblock \DOIprefix\doi{10.3174/ajnr.A3724}.
%Type = Article
\bibitem[{Clark et~al.(2013)Clark, Vendt, Smith, Freymann, Kirby, Koppel,
Moore, Phillips, Maffitt, Pringle, Tarbox and Prior}]{Clark2013}
\bibinfo{author}{Clark, K.}, \bibinfo{author}{Vendt, B.},
\bibinfo{author}{Smith, K.}, \bibinfo{author}{Freymann, J.},
\bibinfo{author}{Kirby, J.}, \bibinfo{author}{Koppel, P.},
\bibinfo{author}{Moore, S.}, \bibinfo{author}{Phillips, S.},
\bibinfo{author}{Maffitt, D.}, \bibinfo{author}{Pringle, M.},
\bibinfo{author}{Tarbox, L.}, \bibinfo{author}{Prior, F.},
\bibinfo{year}{2013}.
\newblock \bibinfo{title}{{The Cancer Imaging Archive (TCIA): Maintaining and
Operating a Public Information Repository}}.
\newblock \bibinfo{journal}{Journal of Digital Imaging} \bibinfo{volume}{26},
\bibinfo{pages}{1045--1057}.
\newblock \DOIprefix\doi{10.1007/s10278-013-9622-7}.
%Type = Article
\bibitem[{Clarke et~al.(1998)Clarke, Velthuizen, Clark, Gaviria, Hall, Goldgof,
Murtagh, Phuphanich and Brem}]{Clarke1998}
\bibinfo{author}{Clarke, L.P.}, \bibinfo{author}{Velthuizen, R.P.},
\bibinfo{author}{Clark, M.}, \bibinfo{author}{Gaviria, J.},
\bibinfo{author}{Hall, L.}, \bibinfo{author}{Goldgof, D.},
\bibinfo{author}{Murtagh, R.}, \bibinfo{author}{Phuphanich, S.},
\bibinfo{author}{Brem, S.}, \bibinfo{year}{1998}.
\newblock \bibinfo{title}{{MRI measurement of brain tumor response: comparison
of visual metric and automatic segmentation}}.
\newblock \bibinfo{journal}{Magnetic resonance imaging} \bibinfo{volume}{16},
\bibinfo{pages}{271--279}.
\newblock \DOIprefix\doi{10.1016/S0730-725X(97)00302-0}.
%Type = Article
\bibitem[{Crawford et~al.()Crawford, Monod, Chen, Mukherjee and
Rabad{\'{a}}n}]{Crawford}
\bibinfo{author}{Crawford, L.}, \bibinfo{author}{Monod, A.},
\bibinfo{author}{Chen, A.X.}, \bibinfo{author}{Mukherjee, S.},
\bibinfo{author}{Rabad{\'{a}}n, R.}, .
\newblock \bibinfo{title}{{Topological Summaries of Tumor Images Improve
Prediction of Disease Free Survival in Glioblastoma Multiforme}}.
\newblock \bibinfo{journal}{Arxiv pre-print} \URLprefix
\url{http://arxiv.org/abs/1611.06818},
\href{http://arxiv.org/abs/1611.06818}{\tt arXiv:1611.06818}.
%Type = Article
\bibitem[{Cui et~al.(2015)Cui, Tha, Terasaka, Yamaguchi, Wang, Kudo, Xing,
Shirato and Li}]{Cui2015}
\bibinfo{author}{Cui, Y.}, \bibinfo{author}{Tha, K.K.},
\bibinfo{author}{Terasaka, S.}, \bibinfo{author}{Yamaguchi, S.},
\bibinfo{author}{Wang, J.}, \bibinfo{author}{Kudo, K.},
\bibinfo{author}{Xing, L.}, \bibinfo{author}{Shirato, H.},
\bibinfo{author}{Li, R.}, \bibinfo{year}{2015}.
\newblock \bibinfo{title}{{Prognostic Imaging Biomarkers in Glioblastoma:
Development and Independent Validation on the Basis of Multiregion and
Quantitative Analysis of MR Images.}}
\newblock \bibinfo{journal}{Radiology} \bibinfo{volume}{000},
\bibinfo{pages}{150358}.
\newblock \DOIprefix\doi{10.1148/radiol.2015150358}.
%Type = Article
\bibitem[{Czarnek et~al.(2017)Czarnek, Clark, Peters and
Mazurowski}]{Czarnek2017}
\bibinfo{author}{Czarnek, N.}, \bibinfo{author}{Clark, K.},
\bibinfo{author}{Peters, K.B.}, \bibinfo{author}{Mazurowski, M.A.},
\bibinfo{year}{2017}.
\newblock \bibinfo{title}{{Algorithmic three-dimensional analysis of tumor
shape in MRI improves prognosis of survival in glioblastoma: a
multi-institutional study}}.
\newblock \bibinfo{journal}{Journal of Neuro-Oncology} \bibinfo{volume}{132},
\bibinfo{pages}{55--62}.
\newblock \DOIprefix\doi{10.1007/s11060-016-2359-7}.
%Type = Article
\bibitem[{DeAngelis(2001)}]{DeAngelis2001}
\bibinfo{author}{DeAngelis, L.M.}, \bibinfo{year}{2001}.
\newblock \bibinfo{title}{{Brain Tumors}}.
\newblock \bibinfo{journal}{New England Journal of Medicine}
\bibinfo{volume}{344}, \bibinfo{pages}{114--123}.
\newblock \DOIprefix\doi{10.1056/NEJM200101113440207}.
%Type = Article
\bibitem[{Dice(1945)}]{Dice1945}
\bibinfo{author}{Dice, L.R.}, \bibinfo{year}{1945}.
\newblock \bibinfo{title}{{Measures of the Amount of Ecologic Association
Between Species}}.
\newblock \bibinfo{journal}{Ecology} \bibinfo{volume}{26},
\bibinfo{pages}{297--302}.
\newblock \DOIprefix\doi{10.2307/1932409}.
%Type = Article
\bibitem[{Dupont et~al.(2016)Dupont, Betrouni, Reyns and
Vermandel}]{Dupont2016}
\bibinfo{author}{Dupont, C.}, \bibinfo{author}{Betrouni, N.},
\bibinfo{author}{Reyns, N.}, \bibinfo{author}{Vermandel, M.},
\bibinfo{year}{2016}.
\newblock \bibinfo{title}{{On image segmentation methods applied to
glioblastoma : State of art and new trends}}.
\newblock \bibinfo{journal}{IRBM} \bibinfo{volume}{1}, \bibinfo{pages}{1--13}.
\newblock \DOIprefix\doi{10.1016/j.irbm.2015.12.004}.
%Type = Article
\bibitem[{Egger et~al.(2013)Egger, Kapur, Fedorov, Pieper, Miller,
Veeraraghavan, Freisleben, Golby, Nimsky and Kikinis}]{Egger2013}
\bibinfo{author}{Egger, J.}, \bibinfo{author}{Kapur, T.},
\bibinfo{author}{Fedorov, A.}, \bibinfo{author}{Pieper, S.},
\bibinfo{author}{Miller, J.V.}, \bibinfo{author}{Veeraraghavan, H.},
\bibinfo{author}{Freisleben, B.}, \bibinfo{author}{Golby, A.J.},
\bibinfo{author}{Nimsky, C.}, \bibinfo{author}{Kikinis, R.},
\bibinfo{year}{2013}.
\newblock \bibinfo{title}{{GBM volumetry using the 3D Slicer medical image
computing platform}}.
\newblock \bibinfo{journal}{Sci Rep} \bibinfo{volume}{3},
\bibinfo{pages}{1364}.
\newblock \DOIprefix\doi{10.1038/srep01364}.
%Type = Article
\bibitem[{Fyllingen et~al.(2016)Fyllingen, Stensjoen, Berntsen, Solheim and
Reinertsen}]{Fyllingen2016}
\bibinfo{author}{Fyllingen, E.H.}, \bibinfo{author}{Stensjoen, A.L.},
\bibinfo{author}{Berntsen, E.M.}, \bibinfo{author}{Solheim, O.},
\bibinfo{author}{Reinertsen, I.}, \bibinfo{year}{2016}.
\newblock \bibinfo{title}{{Glioblastoma segmentation: Comparison of three
different software packages}}.
\newblock \bibinfo{journal}{PLoS ONE} \bibinfo{volume}{11}.
\newblock \DOIprefix\doi{10.1371/journal.pone.0164891}.
%Type = Article
\bibitem[{Goebel et~al.(2006)Goebel, Esposito and Formisano}]{Goebel2006}
\bibinfo{author}{Goebel, R.}, \bibinfo{author}{Esposito, F.},
\bibinfo{author}{Formisano, E.}, \bibinfo{year}{2006}.
\newblock \bibinfo{title}{{Analysis of Functional Image Analysis Contest (FIAC)
data with BrainVoyager QX: From single-subject to cortically aligned group
General Linear Model analysis and self-organizing group Independent Component
Analysis}}.
\newblock \bibinfo{journal}{Human Brain Mapping} \bibinfo{volume}{27},
\bibinfo{pages}{392--401}.
\newblock \DOIprefix\doi{10.1002/hbm.20249}.
%Type = Article
\bibitem[{Gordillo et~al.(2013)Gordillo, Montseny and
Sobrevilla}]{Gordillo2013}
\bibinfo{author}{Gordillo, N.}, \bibinfo{author}{Montseny, E.},
\bibinfo{author}{Sobrevilla, P.}, \bibinfo{year}{2013}.
\newblock \bibinfo{title}{{State of the art survey on MRI brain tumor
segmentation}}.
\newblock \bibinfo{journal}{Magnetic Resonance Imaging} \bibinfo{volume}{31},
\bibinfo{pages}{1426--1438}.
\newblock \DOIprefix\doi{10.1016/j.mri.2013.05.002}.
%Type = Article
\bibitem[{Huang et~al.(2015)Huang, Neagu, Reardon and Wen}]{Huang2015}
\bibinfo{author}{Huang, R.Y.}, \bibinfo{author}{Neagu, M.R.},
\bibinfo{author}{Reardon, D.A.}, \bibinfo{author}{Wen, P.Y.},
\bibinfo{year}{2015}.
\newblock \bibinfo{title}{{Pitfalls in the neuroimaging of glioblastoma in the
era of antiangiogenic and immuno/targeted therapy - detecting illusive
disease, defining response}}.
\newblock \bibinfo{journal}{Frontiers in Neurology} \bibinfo{volume}{6},
\bibinfo{pages}{1--16}.
\newblock \DOIprefix\doi{10.3389/fneur.2015.00033}.
%Type = Article
\bibitem[{Kaus et~al.(2001)Kaus, Warfield, Nabavi, Black, Jolesz and
Kikinis}]{Kaus2001}
\bibinfo{author}{Kaus, M.R.}, \bibinfo{author}{Warfield, S.K.},
\bibinfo{author}{Nabavi, A.}, \bibinfo{author}{Black, P.M.},
\bibinfo{author}{Jolesz, F.A.}, \bibinfo{author}{Kikinis, R.},
\bibinfo{year}{2001}.
\newblock \bibinfo{title}{{Automated segmentation of MR images of brain
tumors.}}
\newblock \bibinfo{journal}{Radiology} \bibinfo{volume}{218},
\bibinfo{pages}{586--591}.
\newblock \DOIprefix\doi{10.1148/radiology.218.2.r01fe44586}.
%Type = Article
\bibitem[{Kickingereder et~al.(2016)Kickingereder, Burth, Wick, G{\"{o}}tz,
Eidel, Schlemmer, Maier-Hein, Wick, Bendszus, Radbruch and
Bonekamp}]{Kickingereder2016}
\bibinfo{author}{Kickingereder, P.}, \bibinfo{author}{Burth, S.},
\bibinfo{author}{Wick, A.}, \bibinfo{author}{G{\"{o}}tz, M.},
\bibinfo{author}{Eidel, O.}, \bibinfo{author}{Schlemmer, H.P.},
\bibinfo{author}{Maier-Hein, K.H.}, \bibinfo{author}{Wick, W.},
\bibinfo{author}{Bendszus, M.}, \bibinfo{author}{Radbruch, A.},
\bibinfo{author}{Bonekamp, D.}, \bibinfo{year}{2016}.
\newblock \bibinfo{title}{{Radiomic Profiling of Glioblastoma: Identifying an
Imaging Predictor of Patient Survival with Improved Performance over
Established Clinical and Radiologic Risk Models}}.
\newblock \bibinfo{journal}{Radiology} \bibinfo{volume}{280},
\bibinfo{pages}{880--889}.
\newblock \DOIprefix\doi{10.1148/radiol.2016160845}.
%Type = Article
\bibitem[{Mabray et~al.(2015)Mabray, Barajas and Cha}]{Mabray2015}
\bibinfo{author}{Mabray, M.C.}, \bibinfo{author}{Barajas, R.F.},
\bibinfo{author}{Cha, S.}, \bibinfo{year}{2015}.
\newblock \bibinfo{title}{{Modern brain tumor imaging.}}
\newblock \bibinfo{journal}{Brain tumor research and treatment}
\bibinfo{volume}{3}, \bibinfo{pages}{8--23}.
\newblock \DOIprefix\doi{10.14791/btrt.2015.3.1.8}.
%Type = Article
\bibitem[{Maleike et~al.(2009)Maleike, Nolden, Meinzer and Wolf}]{Maleike2009}
\bibinfo{author}{Maleike, D.}, \bibinfo{author}{Nolden, M.},
\bibinfo{author}{Meinzer, H.P.}, \bibinfo{author}{Wolf, I.},
\bibinfo{year}{2009}.
\newblock \bibinfo{title}{{Interactive segmentation framework of the Medical
Imaging Interaction Toolkit}}.
\newblock \bibinfo{journal}{Computer Methods and Programs in Biomedicine}
\bibinfo{volume}{96}, \bibinfo{pages}{72--83}.
\newblock \DOIprefix\doi{10.1016/j.cmpb.2009.04.004}.
%Type = Article
\bibitem[{Mazurowski et~al.(2014)Mazurowski, Zhang, Peters and
Hobbs}]{Mazurowski2014}
\bibinfo{author}{Mazurowski, M.A.}, \bibinfo{author}{Zhang, J.},
\bibinfo{author}{Peters, K.B.}, \bibinfo{author}{Hobbs, H.},
\bibinfo{year}{2014}.
\newblock \bibinfo{title}{{Computer-extracted MR imaging features are
associated with survival in glioblastoma patients}}.
\newblock \bibinfo{journal}{Journal of Neuro-Oncology} \bibinfo{volume}{120},
\bibinfo{pages}{483--488}.
\newblock \DOIprefix\doi{10.1007/s11060-014-1580-5}.
%Type = Article
\bibitem[{Menze et~al.(2015)Menze, Jakab, Bauer, Kalpathy-Cramer, Farahani,
Kirby, Burren, Porz, Slotboom, Wiest, Lanczi, Gerstner, Weber, Arbel, Avants,
Ayache, Buendia, Collins, Cordier, Corso, Criminisi, Das, Delingette,
Demiralp, Durst, Dojat, Doyle, Festa, Forbes, Geremia, Glocker, Golland, Guo,
Hamamci, Iftekharuddin, Jena, John, Konukoglu, Lashkari, Mariz, Meier,
Pereira, Precup, Price, Raviv, Reza, Ryan, Sarikaya, Schwartz, Shin, Shotton,
Silva, Sousa, Subbanna, Szekely, Taylor, Thomas, Tustison, Unal, Vasseur,
Wintermark, Ye, Zhao, Zhao, Zikic, Prastawa, Reyes and {Van
Leemput}}]{Menze2015}
\bibinfo{author}{Menze, B.H.}, \bibinfo{author}{Jakab, A.},
\bibinfo{author}{Bauer, S.}, \bibinfo{author}{Kalpathy-Cramer, J.},
\bibinfo{author}{Farahani, K.}, \bibinfo{author}{Kirby, J.},
\bibinfo{author}{Burren, Y.}, \bibinfo{author}{Porz, N.},
\bibinfo{author}{Slotboom, J.}, \bibinfo{author}{Wiest, R.},
\bibinfo{author}{Lanczi, L.}, \bibinfo{author}{Gerstner, E.},
\bibinfo{author}{Weber, M.A.}, \bibinfo{author}{Arbel, T.},
\bibinfo{author}{Avants, B.B.}, \bibinfo{author}{Ayache, N.},
\bibinfo{author}{Buendia, P.}, \bibinfo{author}{Collins, D.L.},
\bibinfo{author}{Cordier, N.}, \bibinfo{author}{Corso, J.J.},
\bibinfo{author}{Criminisi, A.}, \bibinfo{author}{Das, T.},
\bibinfo{author}{Delingette, H.}, \bibinfo{author}{Demiralp, a.},
\bibinfo{author}{Durst, C.R.}, \bibinfo{author}{Dojat, M.},
\bibinfo{author}{Doyle, S.}, \bibinfo{author}{Festa, J.},
\bibinfo{author}{Forbes, F.}, \bibinfo{author}{Geremia, E.},
\bibinfo{author}{Glocker, B.}, \bibinfo{author}{Golland, P.},
\bibinfo{author}{Guo, X.}, \bibinfo{author}{Hamamci, A.},
\bibinfo{author}{Iftekharuddin, K.M.}, \bibinfo{author}{Jena, R.},
\bibinfo{author}{John, N.M.}, \bibinfo{author}{Konukoglu, E.},
\bibinfo{author}{Lashkari, D.}, \bibinfo{author}{Mariz, J.A.},
\bibinfo{author}{Meier, R.}, \bibinfo{author}{Pereira, S.},
\bibinfo{author}{Precup, D.}, \bibinfo{author}{Price, S.J.},
\bibinfo{author}{Raviv, T.R.}, \bibinfo{author}{Reza, S.M.S.},
\bibinfo{author}{Ryan, M.}, \bibinfo{author}{Sarikaya, D.},
\bibinfo{author}{Schwartz, L.}, \bibinfo{author}{Shin, H.C.},
\bibinfo{author}{Shotton, J.}, \bibinfo{author}{Silva, C.A.},
\bibinfo{author}{Sousa, N.}, \bibinfo{author}{Subbanna, N.K.},
\bibinfo{author}{Szekely, G.}, \bibinfo{author}{Taylor, T.J.},
\bibinfo{author}{Thomas, O.M.}, \bibinfo{author}{Tustison, N.J.},
\bibinfo{author}{Unal, G.}, \bibinfo{author}{Vasseur, F.},
\bibinfo{author}{Wintermark, M.}, \bibinfo{author}{Ye, D.H.},
\bibinfo{author}{Zhao, L.}, \bibinfo{author}{Zhao, B.},
\bibinfo{author}{Zikic, D.}, \bibinfo{author}{Prastawa, M.},
\bibinfo{author}{Reyes, M.}, \bibinfo{author}{{Van Leemput}, K.},
\bibinfo{year}{2015}.
\newblock \bibinfo{title}{{The Multimodal Brain Tumor Image Segmentation
Benchmark (BRATS)}}.
\newblock \bibinfo{journal}{IEEE Transactions on Medical Imaging}
\bibinfo{volume}{34}, \bibinfo{pages}{1993--2024}.
\newblock \DOIprefix\doi{10.1109/TMI.2014.2377694}.
%Type = Article
\bibitem[{Porz et~al.(2014)Porz, Bauer, Pica, Schucht, Beck, Verma, Slotboom,
Reyes and Wiest}]{Porz2014}
\bibinfo{author}{Porz, N.}, \bibinfo{author}{Bauer, S.}, \bibinfo{author}{Pica,
A.}, \bibinfo{author}{Schucht, P.}, \bibinfo{author}{Beck, J.},
\bibinfo{author}{Verma, R.K.}, \bibinfo{author}{Slotboom, J.},
\bibinfo{author}{Reyes, M.}, \bibinfo{author}{Wiest, R.},
\bibinfo{year}{2014}.
\newblock \bibinfo{title}{{Multi-modal glioblastoma segmentation: Man versus
machine}}.
\newblock \bibinfo{journal}{PLoS ONE} \bibinfo{volume}{9},
\bibinfo{pages}{1--9}.
\newblock \DOIprefix\doi{10.1371/journal.pone.0096873}.
%Type = Article
\bibitem[{Ramkumar et~al.(2016)Ramkumar, Dolz, Kirisli, Adebahr, Schimek-Jasch,
Nestle, Massoptier, Varga, Stappers, Niessen and Song}]{Ramkumar2016}
\bibinfo{author}{Ramkumar, A.}, \bibinfo{author}{Dolz, J.},
\bibinfo{author}{Kirisli, H.A.}, \bibinfo{author}{Adebahr, S.},
\bibinfo{author}{Schimek-Jasch, T.}, \bibinfo{author}{Nestle, U.},
\bibinfo{author}{Massoptier, L.}, \bibinfo{author}{Varga, E.},
\bibinfo{author}{Stappers, P.J.}, \bibinfo{author}{Niessen, W.J.},
\bibinfo{author}{Song, Y.}, \bibinfo{year}{2016}.
\newblock \bibinfo{title}{{User Interaction in Semi-Automatic Segmentation of
Organs at Risk: a Case Study in Radiotherapy}}.
\newblock \bibinfo{journal}{Journal of Digital Imaging} \bibinfo{volume}{29},
\bibinfo{pages}{264--277}.
\newblock \DOIprefix\doi{10.1007/s10278-015-9839-8}.
%Type = Article
\bibitem[{Wang and Liu(2014)}]{Wang2014}
\bibinfo{author}{Wang, J.}, \bibinfo{author}{Liu, T.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{{A survey of MRI-based brain tumor segmentation
methods}}.
\newblock \bibinfo{journal}{Tsinghua Science and Technology}
\bibinfo{volume}{19}, \bibinfo{pages}{578--595}.
\newblock \DOIprefix\doi{10.1109/TST.2014.6961028}.
%Type = Article
\bibitem[{Wolf et~al.(2005)Wolf, Vetter, Wegner, Bottger, Nolden, Schobinger,
Hastenteufel, Kunert and Meinzer}]{Wolf2005}
\bibinfo{author}{Wolf, I.}, \bibinfo{author}{Vetter, M.},
\bibinfo{author}{Wegner, I.}, \bibinfo{author}{Bottger, T.},
\bibinfo{author}{Nolden, M.}, \bibinfo{author}{Schobinger, M.},
\bibinfo{author}{Hastenteufel, M.}, \bibinfo{author}{Kunert, T.},
\bibinfo{author}{Meinzer, H.P.}, \bibinfo{year}{2005}.
\newblock \bibinfo{title}{{The medical imaging interaction toolkit}}.
\newblock \bibinfo{journal}{Medical Image Analysis} \bibinfo{volume}{9},
\bibinfo{pages}{594--604}.
\newblock \DOIprefix\doi{10.1016/j.media.2005.04.005}.
%Type = Article
\bibitem[{Yang et~al.(2016)Yang, Rao, Martinez, Veeraraghavan, Rao and
Yang}]{Yang2016}
\bibinfo{author}{Yang, D.}, \bibinfo{author}{Rao, G.},
\bibinfo{author}{Martinez, J.}, \bibinfo{author}{Veeraraghavan, A.},
\bibinfo{author}{Rao, A.}, \bibinfo{author}{Yang, D.}, \bibinfo{year}{2016}.
\newblock \bibinfo{title}{{Evaluation of tumor-derived MRI-texture features for
discrimination of molecular subtypes and prediction of 12-month survival
status in glioblastoma Evaluation of tumor-derived MRI-texture features for
discrimination of molecular subtypes and prediction o}}
\bibinfo{volume}{6725}.
\newblock \DOIprefix\doi{10.1118/1.4934373}.
%Type = Article
\bibitem[{Yushkevich et~al.(2006)Yushkevich, Piven, Hazlett, Smith, Ho, Gee and
Gerig}]{Yushkevich2006}
\bibinfo{author}{Yushkevich, P.A.}, \bibinfo{author}{Piven, J.},
\bibinfo{author}{Hazlett, H.C.}, \bibinfo{author}{Smith, R.G.},
\bibinfo{author}{Ho, S.}, \bibinfo{author}{Gee, J.C.},
\bibinfo{author}{Gerig, G.}, \bibinfo{year}{2006}.
\newblock \bibinfo{title}{{User-guided 3D active contour segmentation of
anatomical structures: Significantly improved efficiency and reliability}}.
\newblock \bibinfo{journal}{NeuroImage} \bibinfo{volume}{31},
\bibinfo{pages}{1116--1128}.
\newblock \DOIprefix\doi{10.1016/j.neuroimage.2006.01.015}.
%Type = Article
\bibitem[{Zhang et~al.(2014)Zhang, Jiang, Chen, Bai, Cui, Ren, Chen, Wang, Zeng
and Lin}]{Zhang2014a}
\bibinfo{author}{Zhang, Z.}, \bibinfo{author}{Jiang, H.},
\bibinfo{author}{Chen, X.}, \bibinfo{author}{Bai, J.}, \bibinfo{author}{Cui,
Y.}, \bibinfo{author}{Ren, X.}, \bibinfo{author}{Chen, X.},
\bibinfo{author}{Wang, J.}, \bibinfo{author}{Zeng, W.}, \bibinfo{author}{Lin,
S.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{{Identifying the survival subtypes of glioblastoma by
quantitative volumetric analysis of MRI}}.
\newblock \bibinfo{journal}{Journal of Neuro-Oncology} \bibinfo{volume}{119},
\bibinfo{pages}{207--214}.
\newblock \DOIprefix\doi{10.1007/s11060-014-1478-2}.
\end{thebibliography}