Real-time Instance Segmentation of Surgical Instruments using Attention and Multi-scale Feature Fusion
Extracted source text · arXiv 2111.04911
%%
%% Copyright 2019-2020 Elsevier Ltd
%%
%% This file is part of the 'CAS Bundle'.
%% --------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version. The latest version of this license is in
%% http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'CAS Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for cas-dc documentclass for
%% double column output.
%\documentclass[a4paper,fleqn,longmktitle]{cas-dc}
\documentclass[a4paper,fleqn]{cas-dc}
%\usepackage[numbers]{natbib}
%\usepackage[authoryear]{natbib}
\usepackage[compress]{natbib}
\usepackage{pifont}
\usepackage{caption}
\usepackage{subfig}
%%%Author definitions
\def\tsc#1{\csdef{#1}{\textsc{\lowercase{#1}}\xspace}}
\tsc{WGM}
\tsc{QE}
\tsc{EP}
\tsc{PMS}
\tsc{BEC}
\tsc{DE}
%%%
% Uncomment and use as if needed
%\newtheorem{theorem}{Theorem}
%\newtheorem{lemma}[theorem]{Lemma}
%\newdefinition{rmk}{Remark}
%\newproof{pf}{Proof}
%\newproof{pot}{Proof of Theorem \ref{thm}}
\begin{document}
\let\WriteBookmarks\relax
\def\floatpagepagefraction{1}
\def\textpagefraction{.001}
% Short title
\shorttitle{Real-time Instance Segmentation of Surgical Instruments}
% Short author
\shortauthors{J.C. Angeles-Ceron et~al.}
% Main title of the paper
\title [mode = title]{Real-time Instance Segmentation of Surgical Instruments using Attention and Multi-scale Feature Fusion}
\author[1]{Juan Carlos Ángeles-Cerón}
\author[1]{Gilberto Ochoa-Ruiz}[type=author,
auid=000,bioid=1, orcid=0000-0002-9896-8727]\ead{gilberto.ochoa@tec.mx}
\author[1]{Leonardo Chang}
\author[2,3]{Sharib Ali}[type=author,
auid=000,bioid=1, orcid=0000-0003-1313-3542]\ead{sharib.ali@eng.ox.ac.uk}
\cortext[cor1]{Corresponding authors}
% \ead{gilberto.ochoa@tec.mx}
%
\address[1]{Tecnologico de Monterrey, Escuela de Ingeniería y Ciencias, México}
%
\address[2]{Institute of Biomedical Engineering (IBME), Department of Engineering Science, University of Oxford, Oxford, UK}
% \address[2]{Big Data Institute, University of Oxford, Li Ka Shing Centre for Health Information and Discovery, Oxford, UK}
\address[3]{Oxford NIHR Biomedical Research Centre, University of Oxford, Oxford, UK}
% Here goes the abstract
\begin{abstract}
Precise instrument segmentation aid surgeons to navigate the body more easily and increase patient safety. While accurate tracking of surgical instruments in real-time plays a crucial role in minimally invasive computer-assisted surgeries, it is a challenging task to achieve, mainly due to 1) complex surgical environment, and 2) model design with both optimal accuracy and speed. Deep learning gives us the opportunity to learn complex environment from large surgery scene environments and placements of these instruments in real world scenarios. The Robust Medical Instrument Segmentation 2019 challenge (ROBUST-MIS) provides more than 10,000 frames with surgical tools in different clinical settings. In this paper, we use a light-weight single stage instance segmentation model complemented with a convolutional block attention module for achieving both faster and accurate inference. We further improve accuracy through data augmentation and optimal anchor localisation strategies. To our knowledge, this is the first work that explicitly focuses on both real-time performance and improved accuracy. Our approach out-performed top team performances in the ROBUST-MIS challenge with over 44\% improvement on area-based multi-instance dice metric MI\_DSC and 39\% on distance-based multi-instance normalized surface dice MI\_NSD. We also demonstrate real-time performance ($> 60$ frames-per-second) with different but competitive variants of our final approach.
\end{abstract}
% Use if graphical abstract is present
% \begin{graphicalabstract}
% \includegraphics{figs/grabs.pdf}
% \end{graphicalabstract}
% Keywords
% Each keyword is seperated by \sep
\begin{keywords}
Deep learning\\ MIS instance segmentation\\ Real-time instance segmentation\\ Single-stage segmentation\\ Attention Modules\\ Multi-scale feature fusion
\end{keywords}
\maketitle
%\linenumbers
%% main text
\section{Introduction}
\label{sec:introduction}
%%% Give overview of dataset and challenges
%
% \begin{figure*}[t!]
% \centering
% \includegraphics[width=0.6\textwidth]{dataset.pdf}
% \caption{Exemplary images for \textit{in vitro} and \textit{in vivo} clip sequences with corresponding ground truth masks showing stone and laser fiber. Stone debris and blood are highlighted in yellow and red rectangles (on left), respectively.}
% \label{dataset}
% \end{figure*}
%
%
%% a) Clinical motivation (reference to 2-3 clinical papers)
% Know why automated will assist and in what ways (refernce them, 3-5)
Surgical site infection (SSI) has been the most common cause of hospital-acquired infection and the most common way of infection transmission in patients undergoing surgery~\cite{Giorgio2004:JAMASurg,Daniel2019:JAMANet}. It is therefore imminently important to develop strategies for reducing such infection rates. Minimally invasive surgical (MIS) procedures compared to open surgery lowers such risks ~\cite{Daniel2019:JAMANet}. For these reasons and due to the growth the data science in the operating room applications, there exists an increasing demand of computer assisted/robotic surgery to improve the efficacy of MIS~\cite{Bartoli_2012, Sheetz2019:JAMANet}.
Computer-assisted minimally invasive surgery such as endoscopy has grown in popularity in recent years. However, due to the nature of these procedures, issues like limited field-of-view, extreme lighting conditions, lack of depth information and difficulty in manipulating operating instruments demand strenuous amounts of effort from the surgeons~\cite{Ross_2021}. Surgical data science applications \cite{maierhein2021surgical} could provide physicians with context-aware assistance during minimally invasive surgery in order to overcome these limitations and increase patient safety.
One of the main forms of assistance is providing accurate tracking of medical instruments using computer vision techniques, such as object detection and localisation or instance segmentation methods \cite{Ward2021}. These systems are expected to be a crucial component in tasks ranging from surgical navigation, skill analysis and complication prediction during surgeries, as well as other computer integrated surgery (CIS) applications \cite{Maierhein_2021, fu2021}. Nonetheless, methods for accurate tracking of instruments are often deployed in difficult operational scenarios in which the presence of bleeding, over or under exposed frames, smoke, reflection and other types of artifacts are oftentimes unavoidable \cite{Bodenstedt_2018}. The net effect of these issues increases the missed detection rates in endoscopic surveillance and in general, it limits the overall robustness of computer vision algorithms, hampering the adoption of AI-based tools in this context \cite{Ali_2021}. Moreover, real-time deployment of such tools is of tremendous value and one of the major requirement for it to be applied in clinical setting.
Therefore, the development of robust and real-time techniques that can be effectively deployed during real endoscopic interventions is of utmost importance. In this regard, recent years have seen a significant increase in the number of computer vision contests geared towards endoscopy. More specifically, the Robust Medical Instrument Segmentation (ROBUST-MIS) Challenge \cite{Ross_2021} at the International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI) has sought to address some of the issues discussed above. This challenge represents an important and necessary effort to encourage the development of robust models for surgical instrument segmentation, integrating the developments in computer-assisted surgeries, and benchmark the generalization capabilities of the developed methods on different clinical scenarios.
Furthermore, the challenge organizers provide a large high-quality dataset in an effort to overcome one of the major bottlenecks of the development of robust methodologies, i.e., lack of annotated multi-instance instrument segmentation data. The development of surgical tool navigation and tracking methods in a complex environment will enable improved patient care during surgery by maximising the focus of surgeons. It will also accelerate research in the direction of robotic surgery.
Previous approaches for instance segmentation submitted to the ROBUST-MIS challenge have been mostly based on two-stage detectors such as Mask R-CNN \cite{He_2018}. While these models exhibited decent performances in terms of robustness, they suffer from high inference times due to well-known architectural limitations of such models, preventing them for achieving real-time performance (i.e. mean average test time of only around 5 frames-per-second, fps). However, real-time performance is mandatory in order to fully exploit the capabilities of tracking applications in surgeries. While deep learning methods using lightweight models are also available, they fail to robustly segment objects in endoscopy imaging~\cite{Ali2020}. Similarly, there is a trend in using ensemble models to improve the overall segmentation accuracy in these images. However, combining few models together drastically increases the inference time, thereby making it less feasible to deploy such models in clinical settings~\cite{xu2020}.
%{Motivation for proposed approach}
In order to overcome the current inference limitations, while maintaining a robust performance in terms of tool segmentation results, we propose a new approach based on the single-stage model for instance segmentation. Although recent years have seen an steady increase in the search for more capable one stage detectors and instance segmentation architectures (for example PolyYolo~\cite{polyyolo}, BlendMask~\cite{blendmask} and Solov2~\cite{solov2}), the majority of these models have not been used in the context of endoscopic computer vision due to the performance gap they still present. The YOLACT++ \cite{Bolya_2020} architecture is one of the most recent methods for real-time instance segmentation and it is particularly appealing due to its simplified architecture capable of learning to localize instance masks automatically with minimum computational overhead. It does so by generating a dictionary of non-local prototype masks over the entire image and predicting a set of linear combination coefficients per instance. Thus, for this contribution we have used YOLACT++ as a baseline architecture upon which we have developed several improvements to make it suitable for robust surgical instrument instance segmentation. To this extend, we have explored the use of attention modules on the outputs of the network's backbone and feature pyramid network (FPN) at multiple scales. Moreover, we have additionally carried out a series of optimization techniques by analysing the worst-performing frames of the best model in our experiments. The optimization techniques includes stronger data augmentation, anchor optimization, and multi-scale feature fusion.
Our main contributions include:
\begin{itemize}
\item A real-time single-stage instance segmentation framework with attention mechanisms
\item Use of domain-targeted data augmentation
\item Anchor box optimization via a differential evolution search algorithm~\cite{Zlocha_2019}
% based on \cite{Zlocha_2019}.
\item Exploiting global contextual features by integrating multi-scale fusion in the network's backbone
\item Thorough analysis of the worst-case samples to evaluate each of the tested model
\end{itemize}
% THis is introduction, it should not detail YOLOACT, additing
% YOLACT produces full-image instance segmentation by linearly combining the extracted prototypes using a set of predicted coefficients which are used for generating the cropping mask. This process is achieved with minimum computational overhead due to its parallel structure and extremely lightweight assembly process, which we believe make it a strong candidate for the problem we are tackling.
%Deescribe yolact
% Our best model achieves superior performance compared to the winner of the 2019 ROBUST-MIS challenge in terms of robustness scores, obtaining 0.447 MI\_DSC and 0.489 MI\_NSD while reaching real-time performance at $\mathbf{>}$24 fps, although other models attain higher frame rates at a minimum cost in terms of robustness.
The rest of the paper is organized as follows. In Section \ref{sec:related_work} we present previously published work related to medical instrument segmentation, instance segmentation methods, attention mechanisms and multi-scale feature fusion network. In Section~\ref{sec:materials_and_method}, we present the details of the ROBUST-MIS dataset and our proposed approach for surgical instrument segmentation. Section \ref{sec:resultsandExperiments} presents our data preparation, experimental setup and results. Next, in Section \ref{sec:discussion} we discuss the effects of the different types of network configurations and future directions. Finally, Section \ref{sec:conclusion} concludes the paper.
%and its role in our proposal - comment: This is related work its not the place to write role in proposal
\section{Related work}
\label{sec:related_work}
In this section, we will discuss some of the most important aspects to understand the proposed contribution, namely: instance segmentation and its current limitations, recent works in attention mechanisms, anchor box optimization techniques specifically tailored to the addressed problem and finally, multi-scale fusion networks that followed to make our extended instance segmentation model more robust.
\subsection{Deep learning for instrument segmentation}
Deep learning has accelerated research for surgical instrument segmentation and the public access of labelled data via instrument segmentation challenges mostly at EndoVis (refer to \cite{Allan20192017RI},~\cite{Allan20202018RS},~\cite{Ross_2021}) have contributed to these developments over recent years. Built upon the UNet~\cite{Ronneberger_2015}, LinkNet and TernausNet was developed for instrument segmentation on robotic surgery dataset~\cite{shvets2018}, acquired by da Vinci Xi surgical system of several different porcine procedures made available in EndoVis17~\cite{Allan20192017RI}. \cite{milletari2018} proposed a convolutional long short term memory (LSTM) with deep residual networks in coarse-to-fine strategy showing greater improvements over other state-of-the-art approaches including UNet~\cite{Ronneberger_2015} and FCN~\cite{Long:CVPR15} on EndoVis 2015 instrument segmentation challenge dataset{\footnote{https://endovissub-instrument.grand-challenge.org/EndoVisSub-Instrument/}} focused on laparoscopic and robotic surgery.
\newpage
The recent “Robotic Instrument Segmentation Sub-Challenge” introduced also at EndoVis was oriented towards binary segmentation, sub-component segmentation and instrument identification and instance segmentation tasks~\cite{Ross_2021}. The challenge was focused to assess the robustness and generalization capabilities of the deep learning models. Most of the competing methods in this challenge have been mostly based on Mask-RCNN \cite{mask-rcnn} implementations and its variants. Participants also explored methods such as OR-UNet \cite{orunet}, DeepLabV3+ \cite{deeplab}, U-Net \cite{unet} and RASNet \cite{rasnet}. The best performing methods for the binary segmentation task were OR-UNet and DeepLabv3+ with pre-trained ImageNet encoders. Some other contestants also explored the use of ensemble methods, but they were typically limited in speed, and thus some of the most robust methods are incapable of attaining the real-time performances required for realistically segmenting and tracking objects on endoscopic video data. A double decoder-encoder network was explored for faster binary mask segmentation on ROBUST-MIS'19 daataset~\cite{Jha:BHI2021} that outperformed several state-of-the-art methods. Recently, a one-shot instrument segmentation method~\cite{ZHAO2021102240} using anchor guided meta-learning approach was proposed and validated on several publicly available datasets.
% Check more literature -- current methods therre are few more!!!
% General segmentation (semantic) and instance segmentation
% Recently, deep learning based studies have demonstrated great potential in instrument segmentation from robotic surgical video~\cite{}, clinical surgical videos~\cite{}, and still images in endoscopy~\cite{}.
% Fully-supervised instrument segmentation methods; self supervised
% Most of them ( Milletari et al., 2018; Shvets et al., 2018; Yu et al., 2019; Islam et al., 2019; Ni et al., 2019 ) rely on large amounts of labeled surgical videos during training to achieve satisfying per- formance on the testing surgical videos sampled from similar dis- tribution.
% EndoVis 2015 Instrument sub-challenge\footnote{https://endovissub-instrument.grand-challenge.org/EndoVisSub-Instrument/} was first proposed as part of Endoscopic
% vision (EndoVis) challenge, focusing on segmentation and tracking for rigid and articulated instruments in laparoscopic and robotic surgery.
% Recent “Robotic Instrument Segmentation Sub-Challenge” introduced also at EndoVis was oriented towards binary segmentation, sub-component segmentation and instrument identification and instance segmentation tasks~\cite{Ross_2021}. The challenge was focused to assess the robustness and generalization capabilities of the deep learning models. Most of the competing methods in this challenge have been mostly based on Mask-RCNN \cite{mask-rcnn} implementations and its variants for the multi-class instance segmentation andbinary segmentation segmentation tasks. Participants also explored methods such as OR-UNet \cite{orunet}, DeepLabV3+ \cite{deeplab}, U-Net \cite{unet} and
% RASNet \cite{rasnet}. The best performing methods for the binary segmentation task were OR-UNet and DeepLabv3+ with pre-trained ImageNet encoders. Some other contestants also explored the use of ensemble methods, but they were typically limited in speed, and thus some of the most robust methods are incapable of attaining the real-time performances required for realistically segmenting and tracking objects on endoscopic video data.
% Check more literature -- current methods therre are few more!!!
%
\subsection{Real-time instance segmentation methods}
While extensive research has been conducted for the development of real-time object detection and semantic segmentation models, few works have tackled the problem of real-time instance segmentation~\cite{mask-rcnn,yolact}. This is due to the increased complexity in the instance segmentation task that requires prediction of instance labels and pixel-level segmentation simultaneously.
% Question: which One stage methods are you talking about ---> cite them please!!!
One-stage methods~\cite{polyyolo,center-mask, blendmask} though conceptually faster than two-stage methods (e.g., Mask RCNN~\cite{mask-rcnn}), still require many non-trivial computations (e.g., mask voting). This severely limits their speed making them not unsuitable for real-time applications. In contrast, recent methods~\cite{yolact,Bolya_2020} make use of lightweight assembly of masks (only a linear combination is used), making the approach very efficient. Although YOLACT~\cite{yolact} was one of the first real-time one-stage instance segmentation approach, the accuracy gap compared to Mask R-CNN \cite{mask-rcnn} was still significant. While Mask R-CNN is based on a two-stage object detector (e.g., Faster R-CNN \cite{faster-rcnn}), YOLACT \cite{yolact} is built on one-stage detector (RetinaNet \cite{retina-net}) that directly predicts boxes without proposal step, limiting its accuracy. This was partially addressed with the introduction of Yolact++~\cite{Bolya_2020} incorporates deformable convolutions into the backbone network, improving the feature sampling and yielding an improved accuracy. Furthermore, the prediction head was optimized with better anchor scale and aspect ratio choices for an increased object recall.
%
\subsection{Attention mechanisms}
Attention has been able to boost model performance across a wide range of computer vision tasks, such as image captioning \cite{You_2016}, visual question answering \cite{Xu_2016}, and visual attribute prediction \cite{Seo_2018}. Attention allows the network to focus on the most relevant features without the need of additional supervision, preventing redundant use of information and extracting salient features that are useful for a given task. Attention mechanisms enable convolutional neural networks to overcome the size limitations of its receptive field, as it has proven to be excellent at extracting global dependencies between inputs and outputs, thus, improving the modeling of long-range dependencies even at opposite ends of and image \cite{Sinha_2021}. For example, similar textures may appear in different parts of an image, several disjoint semantic cues may provide insight to the general classification of an image, and an object might present complex and occluded parts throughout an image \cite{Chaudhari2019AnAS}. In the context of medical instance segmentation, multiple attention-based models \cite{Kaul_2019, Gu_2020, Sinha_2021} have obtained state-of-the-art performance in fields like brain tumor, skin cancer and lung lesion segmentation on CT scans and X-rays. Until now, instance segmentation of medical instruments in laparoscopic surgeries using attention mechanisms has not been fully explored.
\subsection{Multi-scale feature fusion}
Due to the wide range of scale variation of objects found in instance segmentation, multi-scale features are essential for robust performance \cite{Wang_2020}. Therefore, multi-scale feature aggregation is an adequate strategy to create detailed parsing maps \cite{Ding_2018}. Current methods address this issue by using encoder-decoder architectures \cite{Ronneberger_2015, Milan_2017} that combine high level and low level features at a single scale \cite{Long_2015} or multiple scales \cite{Ronneberger_2015}. However, these approaches suffer from redundant use of information \cite{Sinha_2021}. In the field of medical instrument segmentation, the use of multi-scale aggregation has not been fully explored, especially combined with attention for improved robustness.
\section{Materials and Method}
\label{sec:materials_and_method}
In this section we present details on the dataset used in our study and describe our proposed framework for multi-instance surgical instrument segmentation.
%
\subsection{The ROBUST-MIS challenge dataset}
\label{sec:robust_mis}
\begin{table}[t!]
\fontsize{7}{8}\selectfont
\centering
\caption{Training and test sample distribution for each both training and test stages of the ROUBUST-MIS challenge~\cite{Ross_2021}. The quantities in parenthesis represent the \% of frames with no instrument instance.}\label{tab:data_distribution}
\begin{tabular}{l|l|lll}
\toprule
\multirow{2}{*}{\textbf{Procedure}}
% \multicolumn{1}{c|}{\textbf{Procedure}} &
& \multirow{2}{*}{\textbf{Training}} & \multicolumn{3}{c}{\textbf{Testing}} \\
& \multicolumn{1}{l}{} & \multicolumn{1}{|l}{Stage 1} & \multicolumn{1}{l}{Stage 2} & \multicolumn{1}{l}{Stage 3} \\ \hline
{{\begin{tabular}[l]{@{}c@{}}Procto- \\ colectomy \end{tabular}}} & 2,943 (2\%) & 325 (11\%) & 255 (11\%) & 0 \\
{{\begin{tabular}[l]{@{}c@{}}Rectal\\ resection \end{tabular}}} & 3,040 (20\%) & 338 (20\%) & 289 (15\%) & 0 \\
{{\begin{tabular}[l]{@{}c@{}}Sigmoid\\ resection* \end{tabular}}}
& 0 & 0 & 0 & {{\begin{tabular}[l]{@{}c@{}} 2,880 \\(23\%) \end{tabular}}} \\ \hline
TOTAL & 5,983 (17\%) & 663 (15\%) & 514 (13\%) & {{\begin{tabular}[l]{@{}c@{}}2,880 (23\%) \end{tabular}}} \\ \hline
\multicolumn{5}{l}{*Unknown surgery}
\end{tabular}
\end{table}
%
For our experiments we made use of the Robust Medical Instrument Segmentation (ROBUST-MIS) \cite{Maierhein_2021} challenge dataset which is the first large-scale annotated MIS dataset. The dataset is comprised of a total of 10,040 annotated video frames extracted from 30 minimally invasive daily-routine surgical procedures and includes detailed segmentation ground truth masks for the surgical instruments present in these frames.
The surgical procedures include 10 rectal resection procedures, 10 proctocolectomy procedures, and 10 sigmoid resection procedures. The image resolution of all the provided frames is $960 \times 540$ pixels. In order to measure the robustness test, the dataset is comprised of three unique test sets and divided into different stages:
% To emphasize on model robustness, a ranking scheme of the challenge was particularly devised to specifically measure the worst-case performance on this dataset~\cite{Maierhein_2021} as below:
\begin{enumerate}
\item[] \textbf{Stage 1:} Test data taken from the same procedures from which the training data were extracted\\
\item[]\textbf{Stage 2:} Test data taken from the exact same type of surgery as the training data but from procedures (patients) not included in the training\\
\item[] \textbf{Stage 3:} Test data taken from a different (unseen) but similar type of surgery and different (unseen) patients
\end{enumerate}
The detailed training and test distribution is summarised in Table \ref{tab:data_distribution}. However, for testing we have used only test set 3, \textit{i.e. Stage 3}, which is from an unseen Sigmoid resection procedure and allows us to validate on the generalisability of the proposed framework directly. Sample images for challenging frames provided in the test set 3 is shown in Figure~\ref{fig:sampleimages}.
% This allows us to robustly validate our method and provide comparisons against
% Regarding the partition of the data into training and test sets, all sigmoid resection frames were reserved for stage 3. For stage 2, out of the remaining 20 videos, the two shortest videos per procedure were taken for test set in order to obtain as much training data as possible. Lastly, from the remaining videos, every 10th annotated frame was used for stage 1 test set. All the remaining frames were selected as training data. The detailed training and test distribution is shown in Table \ref{tab:data_distribution}.
%
\begin{figure}[t!]
\centering
\includegraphics[width=0.98\linewidth]{images/mosaic-images.pdf}
\caption{\textbf{Challenging images present in test dataset.} Stage 3 test data consisted of several frames that consisted of a) instrument flare, b) parital occlusions due to blood, c) occlusion due to smoke, d) underexposed regions with instrument, e) motion blur, f) multiple different instruemnts in the scene, g) partial occlusion due to organ, h) transparent instrument and i) different instruments crossing.}
\label{fig:sampleimages}
\end{figure}
\subsection{Method}
%TODO: introduce what the readers will find here!
%
\begin{figure*}[t!]
\centering
% \includegraphics[width=\textwidth]{images/Architecture.pdf}
\includegraphics[width=\textwidth]{images/YOLOAtt-MSFF.pdf}
\caption{%Proposed Attention YOLACT++ architecture.
\textbf{Overview of our proposed framework.} Our architecture is built upon popular single-stage YOLACT++ and comprises of additional multi-scale feature fusion and attention modules. It is to be noted that the attention modules is easily interchangeable from criss-cross attention (CCAM) to convolutional block attention module (CBAM) or none.}
\label{fig:architecture}
%I have removed your dotted line because your contribution is not only placing attention but also multi-scale feature fusion
\end{figure*}
% \subsubsection{Proposed framework}
This section describes our proposed approach for multi-instance segmentation of medical instruments in the ROBUST-MIS dataset. Our proposed framework (see Figure \ref{fig:architecture}) is built upon the single-stage YOLACT++~\cite{Bolya_2020} instance segmentation architecture. For our framework we employ ResNet-101~\cite{He_2015} as the backbone network, followed by a multi-scale feature fusion (MSFF) module used to aggregate contextual information from the feature maps across all scales (i.e. high-to-low resolution feature representations). Each of these contextually rich fused features are then passed through attention modules to further refine these representations before being forwarded to the feature pyramid network (FPN)~\cite{Lin_2017}. A second set of attention modules are then applied to further enhance the FPN output features allowing for an improved performance of the prototype network and our anchor-optimized prediction head. Finally, we perform classical non-maximum suppression for final mask instance prediction which is then combined with the prototype mask and the cropping provides the predicted bounding box.
Below we describe our MSFF module, attention mechanisms, and the anchor optimization used our framework.
% TODO: update figure with : anchor-optimized prediction head
\subsubsection{Multi-scale feature fusion}
\label{sec:ms_fusion}
To aggregate multi-scale features, while maintaining a high-resolution representation, we integrated a fusion module inspired by the method proposed in \citet{Deng_2018}. Considering the features at different scales indicated as $F_s$ where $s$ denote the scale level in the architecture (see Figure \ref{fig:multiscale_fusion}), features from each level $s$ are up-sampled through transposed convolution to the size of the highest resolution feature maps in the architecture, leading to enlarged feature maps $F'_s$. Next, all $F'_s$ are concatenated into a single tensor which is passed through a convolutional layer to integrate context from all scales into a single feature map $F_{MS} = conv([F'_0, F'_1, F'_2, F'_3, F'_4])$. In this manner, $F_{MS}$ encodes both low-level and high-level semantics learned at different stages. Finally, $F_{MS}$ is concatenated with each of the $F'_s$ feature maps and convolved to aggregate multi-scale information, creating multi-scale fused feature maps $F_A$.
%
\begin{figure}[t!]
\centering
% \includegraphics[width=0.98\linewidth]{images/multiscale_transposed_conv.pdf}
\includegraphics[width=0.98\linewidth]{images/MSFF.pdf}
\caption{\textbf{Multi-scale feature fusion (MSFF) module.} Feature maps from the different scale feature maps of the backbone are combined to create aggregated features from all scales.}
\label{fig:multiscale_fusion}
\end{figure}
Note that multi-scale feature fusion can be applied in the same way in both the backbone features and the FPN features. We opted to attach it on the backbone features as we believe it would create stronger representation which will get even further refined on the FPN.
% Sharib
\subsubsection{Attention mechanisms}
\label{sec:attention_mechanisms}
% TODO: only discuss your network: nothing about result or any configuration in your experiments. this will come later. I have added according later!!
We employed Criss-cross Attention Modules (CCAM) \cite{Huang_2020} and Convolutional Block Attention Modules (CBAM) \cite{Woo_2018}, specifically because of their fast and computationally efficient performance, which is paramount to introduce as less computational overhead as possible into the model, and therefore maintain low inference times. We attach the attention modules between the backbone and neck structures, as well as the neck and head of our network (see Figure \ref{fig:architecture}). The rationale behind the selection of these locations is that the addition of attention allows the model to extract richer context by aggregating local information with its corresponding global dependencies \cite{Huang_2020}. Additionally, attention aids to emphasize interdependent relationships between channel maps and between spatial regions without additional supervision \cite{Woo_2018}. Since the backbone network and the FPN are where most of the semantic context is distilled, it is a natural choice to attempt to refine their feature representations using attention, especially since mask prototypes and the prediction head benefit from better features.
% TODO: Add how you combined this in your network (missing), where can we see these attentions and why you decided this!!
\subsubsection{Loss function and anchor optimisation}
% TODO: please write about loss functions in the model
We use a combination of four losses: classification loss $L_{cls}$, bounding box regression loss $L_{box}$, mask loss $L_{mask}$, and semantic segmentation loss $L_{seg}$ with weights of 1, 1.5, 6.125 and 1, respectively. We employ softmax cross entropy for $L_{cls}$ with 1 positive label (instrument) and 1 background label. In the case of $L_{box}$ we used smooth-$L_1$ loss in the same way as \citet{Liu_2016}. $L_{mask}$ is defined as the pixel-wise binary cross entropy between the predicted masks and the ground truth. $L_{seg}$ is applied on layers that are only evaluated during training for additional feature richness~\citet{Bolya_2020}.
% TODO: then go into how you have optimised anchors (please do not write literature rather what you have done)
We further optimized the anchor boxes~\cite{Zlocha_2019} by defining 5 target scales and aspect ratios considering the size of our rescaled images ($600 \times 600$ pixels). The resulting scales were [0.435, 0.502, 0.578, 0.664, 0.762] and the resulting aspect ratios [0.267, 0.554, 1.0, 1.804, 3.746].
%TODO: This should go in experiments/discussion
% Since we noticed adequate detection rates across all scales, we decided to keep the default scales rather than the optimized ones. The intention of doing so is to avoid an extreme adjustment to the training set, and thus allow greater generalization to other types of surgery. In the case of the aspect ratios, we noticed an interesting behavior; we observed that three of the optimized ratios ([0.554, 1, 1.804]) are similar to the default ones ([0.5, 1, 2]), therefore we opted to keep the default ratios. The remaining two ratios [0.267, 3.746] correspond to long horizontal and long vertical objects respectively, which indicates that the previous anchor box configuration was indeed incapable of detecting such instances. We decided to approximate the aspect ratios to [0.25, 4] following the same rationale as with the scales. The idea is that the anchor box regressors will be capable of compensating such small differences.
\section{Experiments and Results}{\label{sec:resultsandExperiments}}
%
\subsection{Data preparation}
As shown in Table \ref{tab:data_distribution}, the ROBUST-MIS dataset contains about 17\% empty frames (ef) on its training set. These frames do not have any visible instruments in them, and although we could have left them as negative examples for training, we opted to remove them from the training set. This decision was taken considering that the data already has plenty of negative examples in the frames’ background for the model to learn from; additionally, removing such frames speeds up the training. In the end, a total of 996 images with no visible instruments were discarded, leaving 4,987 frames in our training set. We then applied an 85-15\% split to the curated training set with 15\% for validation purposes. The training set was then randomly shuffled before creating the train and validation splits. We finally obtained a training set composed of 4,239 frames and a validation set comprised of 748 frames. As a final step, the training and validation datasets were converted to COCO-style for our instance segmentation framework, which involved extracting mask contours and generating bounding box coordinates from the provided annotation images and translating them to the target JSON format.
\subsection{Training setup}
\label{sec:training_setup}
The training was performed on an NVIDIA DGX-1 system consisting of 8 NVIDIA Volta-based GPUs; however, each model was trained on \emph{a single} GPU. The models were trained for up to 400,000 iterations with a learning rate of 0.001, momentum of 0.9, weight decay of $5e^{-4}$, and a batch size of 16. Input images were resized to $600 \times 600$ pixels during training, as this configuration constituted the best trade-off between GPU memory usage and segmentation performance.
% Remove redundant writing
% Additional resizing configurations were attempted with both smaller and larger images; however, no performance benefit was observed.
We applied data augmentation techniques to increase the model’s performance. These included random photometric distortions (i.e., changes in contrast, color-space, saturation, hue, brightness, and noise transformations) and affine transformations (i.e., random scaling, random sample crop, and random mirror).
% We applied data augmentation techniques to increase the model’s performance by introducing as much variability as possible into the data and reduce overfitting. Transformation techniques were applied on the fly on each batch during training. The base set of augmentations includes random photometric distortions (i.e., changes in contrast, color-space, saturation, hue, brightness, and noise transformations) and affine transformations (i.e., random scaling, random sample crop, and random mirror).
%
\subsection{Ablation study setup}
%
\begin{table}[]
\fontsize{8}{9}\selectfont
\centering
\begin{tabular}{l|c|c|c}
\toprule
\multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}\bf{Model}\\ \bf{ Identifier}\end{tabular}} & \begin{tabular}[c]{@{}c@{}}\bf{ Attention}\\ \bf{Type}\end{tabular} & \begin{tabular}[c]{@{}c@{}}\bf{ Backbone}\\ \bf{Attention}\end{tabular} & \begin{tabular}[c]{@{}c@{}}\bf{FPN}\\ \bf{Attention}\end{tabular} \\ \hline
Base YOLACT++ & & & \\ \hline
CCAM-Backbone & CCAM & \ding{51} & \\
CCAM-FPN & CCAM & & \ding{51} \\
CCAM-Full & CCAM & \ding{51} & \ding{51} \\ \hline
CBAM-Backbone & CBAM & \ding{51} & \ding{51} \\
CBAM-FPN & CBAM & & \ding{51} \\
CBAM-Full & CBAM & \ding{51} & \ding{51} \\ \hline
\begin{tabular}[c]{@{}l@{}}CBAM-Full\\ + Aug\end{tabular} & CBAM & \ding{51} & \ding{51} \\ \hline
\begin{tabular}[c]{@{}l@{}}CBAM-Full\\ + Aug + Anch\end{tabular} & CBAM & \ding{51} & \ding{51} \\ \hline
\begin{tabular}[c]{@{}l@{}}CBAM-Full\\ + Aug + Anch + MSFF\end{tabular} & CBAM & \ding{51} & \ding{51} \\
\bottomrule
\end{tabular}
\caption{Model configurations for our experiments with integration of different attention mechanisms in unique settings.}
\label{tab:attn_experiments}
\end{table}
%
Our experiments systematically integrate attention mechanisms in two strategic locations of the baseline YOLACT framework: 1) at the output of each convolutional block of the ResNet-101 backbone~\cite{He_2015}, and 2) at the multi-scale output features of the FPN~\cite{Lin_2017}. The incorporation of attention in these locations was alternated throughout experiments leading to three different network configurations:
\begin{enumerate}
\item Exclusively incorporated in the backbone
\item Exclusively incorporated in the FPN
\item Integrated in the backbone and FPN simultaneously (which we refer to as a \emph{Full} configuration)
\end{enumerate}
\noindent At the end, six attention-based models were created by following this strategy, plus a baseline network without attention. Next, we selected the top performing configuration from the six mentioned network configurations. We then applied other optimization techniques to understand the network performance that included stronger data augmentation and anchor optimization. Finally, we added our multi-scale MSFF module. Table \ref{tab:attn_experiments} summarizes all the different model configurations.
% TODO: In table with attention also include the final proposed model with attention with your proposed CBAM attention? is it full?
%
%TODO====== done till here 5oct2021
\subsection{Metrics and assessment}
%
% TODO: More details needed!!! I will have to read the paper or alternatively JC you can add them? e.g. formulas
The algorithms' performance was evaluated following the guidelines defined by the ROBUST-MIS Challenge. Robustness performance was assessed considering the area-based metric multi-instance dice MI\_DSC and the distance-based multi-instance normalized surface dice MI\_NSD using the code implementations employed in the challenge which are provided in \citep{Ross_2019}. Furthermore, our reported model rankings were computed using the publicly available~\emph{challengeR} \cite{Wiesenfarth_2021} R package developed by the challenge organizers to accurately evaluate competitors. The ranking stability was investigated using bootstrapping for quantifying ranking variability using 1000 samples.
The robustness rankings are particularly focused on models’ capabilities in stage 3 of the challenge and pay particular attention on the worst-case performance of methods. For this reason, the robustness rankings are computed by aggregating the resulting scores for all the test cases by the 5\% percentile instead of by the mean or median.
We performed inference speed assessments by running inference tests on a 10 second video snippet from the ROBUST-MIS dataset a total of ten times per model. The reported frame rates were then aggregated by the mean. Inference was tested on a \emph{single} Tesla P100 GPU from the DGX-1 cluster with video multi-frame enabled.
\begin{table*}[t!]
\centering
\begin{tabular}{llccc}
\toprule
\multicolumn{1}{l|}{\bf Team/Model} & \multicolumn{1}{c|}{\bf Base Method} & \multicolumn{1}{c|}{\bf Aggr. MI\_DSC} & \multicolumn{1}{c|}{\bf Aggr. MI\_NSD} & \multicolumn{1}{c}{\bf FPS} \\ \hline
\hline
\multicolumn{1}{l|}{\textit{www}} & \multicolumn{1}{l|}{Mask R-CNN} & \multicolumn{1}{c|}{0.31} & \multicolumn{1}{c|}{0.35} & \multicolumn{1}{c}{5*} \\
\multicolumn{1}{l|}{\textit{Uniandes}} & \multicolumn{1}{l|}{Mask R-CNN} & \multicolumn{1}{c|}{0.26} & \multicolumn{1}{c|}{0.29} & \multicolumn{1}{c}{5*} \\
\multicolumn{1}{l|}{\textit{SQUASH}} & \multicolumn{1}{l|}{Mask R-CNN} & \multicolumn{1}{c|}{0.22} & \multicolumn{1}{c|}{0.26} & \multicolumn{1}{c}{5*} \\
\multicolumn{1}{l|}{\textit{CASIA\_SRL}} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}Dense Pyramid \\ Attention Network\end{tabular}} & \multicolumn{1}{c|}{0.19} & \multicolumn{1}{c|}{0.27} & \multicolumn{1}{c}{5*} \\
\multicolumn{1}{l|}{\textit{fisensee}} & \multicolumn{1}{l|}{2D U-Net} & \multicolumn{1}{c|}{0.17} & \multicolumn{1}{c|}{0.16} & \multicolumn{1}{c}{12*} \\
\multicolumn{1}{l|}{\textit{caresyntax}} & \multicolumn{1}{l|}{Mask R-CNN} & \multicolumn{1}{c|}{0.00} & \multicolumn{1}{c|}{0.00} & \multicolumn{1}{c}{5*} \\
\multicolumn{1}{l|}{\textit{VIE}} & \multicolumn{1}{l|}{Mask R-CNN} & \multicolumn{1}{c|}{0.00} & \multicolumn{1}{c|}{0.00} & \multicolumn{1}{c}{5*} \\ \hline
\multicolumn{1}{l|}{Base YOLACT++} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.000} & \multicolumn{1}{c|}{0.000} & \multicolumn{1}{c}{75} \\
\multicolumn{1}{l|}{CCAM-FPN} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.000} & \multicolumn{1}{c|}{0.000} & \multicolumn{1}{c}{60} \\
\multicolumn{1}{l|}{CBAM-Backbone} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.245} & \multicolumn{1}{c|}{0.285} & \multicolumn{1}{c}{65} \\
\multicolumn{1}{l|}{CCAM-Full} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.308} & \multicolumn{1}{c|}{0.333} & \multicolumn{1}{c}{45} \\
\multicolumn{1}{l|}{CCAM-Backbone} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.313} & \multicolumn{1}{c|}{0.338} & \multicolumn{1}{c}{49} \\
\multicolumn{1}{l|}{CBAM-FPN} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.315} & \multicolumn{1}{c|}{0.333} & \multicolumn{1}{c}{66} \\
\multicolumn{1}{l|}{CBAM-Full} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.338} & \multicolumn{1}{c|}{0.383} & \multicolumn{1}{c}{65} \\
\multicolumn{1}{l|}{CBAM-Full + Aug} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.382} & \multicolumn{1}{c|}{0.429} & \multicolumn{1}{c}{63} \\
\multicolumn{1}{l|}{CBAM-Full + Aug + Anch} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{0.425} & \multicolumn{1}{c|}{0.471} & \multicolumn{1}{c}{\textbf{69}} \\
\multicolumn{1}{l|}{CBAM-Full + Aug + Anch + MS} & \multicolumn{1}{l|}{YOLACT++} & \multicolumn{1}{c|}{\textbf{0.447}} & \multicolumn{1}{c|}{\textbf{0.489}} & \multicolumn{1}{c}{24} \\ \bottomrule
\multicolumn{5}{l}{* Approximated from base method. Original measurement was not reported.}
%
\end{tabular}
\caption{Evaluation results for stage 3 of the challenge. The upper part of the table shows the aggregated metrics for the competitors of the 2019 ROBUST-MIS challenge. The lower part of the table shows the results of the our developed models. MI\_DSC and MI\_NSD metrics are reported along with the base models and frame rates.}
\label{tab:metric_scores}
\end{table*}
\subsection{Results}
\label{sec:results}
% TODO write brief what readers will find here!
In this section, we compare our results to the ROBUST-MIS challenge methods and present both quantitative and qualitative results.
%
\subsubsection{Quantitative results}
% Remember: Always tables first then plots -- present the results in the same way!!!
%
\begin{table*}[t!]
\centering
\begin{tabular}{l|c|l|l|c|l|c}
\toprule
\multirow{2}{*}{\bf Model}& \multicolumn{2}{c|}{\textbf{Stage 1}} & \multicolumn{2}{c|}{\textbf{Stage 2}} & \multicolumn{2}{c}{\textbf{Stage 3}} \\
& MI\_DSC & \multicolumn{1}{c|}{MI\_NSD} & \multicolumn{1}{c|}{MI\_DSC} & MI\_NSD & \multicolumn{1}{c|}{MI\_DSC} & MI\_NSD \\ \hline \hline
Base YOLACT++ & 0.307 & 0.332 & 0.000 & 0.000 & 0.000 & 0.000 \\ \hline
CCAM-Backbone & 0.399 & 0.468 & 0.317 & 0.333 & 0.313 & 0.338 \\ \hline
CCAM-FPN & 0.230 & 0.334 & 0.000 & 0.000 & 0.000 & 0.000 \\ \hline
CCAM-Full & 0.403 & 0.447 & 0.282 & 0.333 & 0.308 & 0.333 \\ \hline
CBAM-Backbone & 0.425 & 0.462 & 0.318 & 0.350 & 0.245 & 0.285 \\ \hline
CBAM-FPN & 0.433 & 0.485 & 0.329 & 0.365 & 0.315 & 0.333 \\ \hline
CBAM-Full & 0.402 & 0.465 & 0.331 & 0.381 & 0.338 & 0.383 \\ \hline
CBAM-Full + Aug & 0.454 & 0.498 & 0.313 & 0.380 & 0.383 & 0.430 \\ \hline
CBAM-Full + Aug + Anch & {0.457} & \bf{0.500} & \bf{0.439} &{0.468} & 0.425 & 0.471 \\ \hline
CBAM-Full + Aug + Anch + MS & \bf{0.460} & {0.499} & {0.427} & \bf{0.478} & \bf{0.447} & \textbf{0.487} \\ \bottomrule
\end{tabular}
\caption{Metric comparison for all three test stages of ROBUST-MIS dataset. Best metric values are shown in bold for each test dataset.}
\label{tab:suppTab1}
\end{table*}
%
\begin{figure*}[t!]
\centering
\includegraphics[width=0.95\linewidth]{images/boxplots_final_2.pdf}
\caption{Dot-and-boxplots for the MI\_DSC and MI\_NSD showing the individual performances of algorithms on stage 3 of the challenge. The red lines indicate the value of the aggregated metric (by 5\% percentile) for each algorithm.}
\label{fig:boxplots_stage1}
\end{figure*}
Table \ref{tab:metric_scores} shows the detailed result for the multi-instance segmentation task on the ROBUST-MIS dataset. It can be observed that overall, attention-based models show improvement over previous approaches for the aggregated MI\_DSC and MI\_NSD, and most notably FPS.
\emph{CCAM-Full}, \emph{CCAM-Backbone}, and \emph{CBAM-FPN} achieved competitive results in terms of MI\_DSC compared to the the top contestant \emph{www}, scoring 0.308, 0.313, and 0.315, respectively. On the other hand, the three models fall behind by 0.015 in average regarding MI\_NDS. Nonetheless, such a small difference in performance is out-weighted by the dramatic increase of inference speed of at least $9\times$ from all the models.
The highest metric scores from the initial ablation experiments correspond to \emph{CBAM-Full}, which resulted in 0.338 MI\_DSC and 0.383 MI\_NSD. \emph{CBAM-Full} presents a 2.8\% on MI\_DSC and 3.3\% on MI\_NSD improvement compared to the previously best model while attaining real-time inference speed of 65 FPS.
The additional data augmentation efforts applied in \emph{CBAM-Full + Aug} resulted in an improvement of 4.4\% and 4.6\% on MI\_DSC and on MI\_NSD, respectively, with respect to \emph{CBAM-Full}. \emph{CBAM-Full + Aug + Anch} greatly benefited from anchor optimization, which resulted in the model with the best balance between robustness and speed. It achieved scores of 0.425 MI\_DSC, 0.471 MI\_NSD, and runs at 69 FPS. For comparison, this model outperforms team \emph{www}'s by 11.5\% on MI\_DSC and 12.1\% on MI\_NSD while running $13.8\times$ faster.
Our most robust network and proposed architecture, namely \emph{CBAM-Full + Aug + Anch + MS}, reached 13.7\% MI\_DSC and 13.9\% MI\_NSD scores higher, compared to the top contestant of the challenge. It also outperforms \emph{CBAM-Full + Aug + Anch} by 2.2\% and 1.8\% margin. Nonetheless, \emph{CBAM-Full + Aug + Anch + MS}'s increased complexity has an impact on its inference speed which becomes evident when looking at its running speed of 24 FPS. Nevertheless, the model is still $4.8\times$ faster than the previous state-of-the-art. Table~\ref{tab:suppTab1} shows the results for each development stage of our final network on all three test datasets. It can be observed that for other Stage 1 and Stage 2 as well our proposed architecture with CBAM-Full + Aug + Anch showed improved performance over most combinations. However, in these cases the addition of the multi-scale feature fusion network (MS) only provided competitive result (e.g., 0.46 and 0.427 on MI\_DSC for Stage 1 and Stage 2, respectively).
Figure \ref{fig:boxplots_stage1} shows the dot-and-boxplots of the MI\_DSC and MI\_NSD metric values obtained by each of our algorithms on Stage 3 test set used in the challenge. We can observe a large difference between the top model and the baseline model, as well as the progressive improvement from experiment to experiment. Despite most of the models are similar in terms of their median, the improvement is evident when looking at the aggregated metric values, as well as the first and third quartiles of the top model with our final model having the least deviations.
\begin{figure*}[t]
\centering
\subfloat[Qualitative improvements of worst performance frames from \emph{Base YOLACT++} to \emph{CBAM-Full}. Base model faults in alphabetical order: a) Missed instrument detection due to smoke overlay. b) False positive segmentation of bandage as instrument. c) Missed detection due to blood overlay. d) False positive segmentation of blood pool and shadow. e) Missed detection due to motion blur. f) Missed detection of instrument covered in tissue.\label{fig:1a}]{\includegraphics[width=0.47\textwidth]{images/baseline-cbam-full_improvements.pdf}}\hfill
\subfloat[Qualitative improvements of worst performing frames from \emph{CBAM-Full} to \emph{CBAM-Full + Aug}. \emph{CBAM-Full} faults in alphabetical order: a) Missed vertical instrument. b) Single (split) instrument mistakenly regarded as two instances. c) Small instrument on the edge of the field of view. d) Missed vertical instrument and missed instrument on the edge of the field of view. e) Missed instruments on the edge of FOV. f) Missed transparent instrument. \label{fig:1b}] {\includegraphics[width=0.47\textwidth]{images/cbam-full-aug_improvements.pdf}} \vfill
\subfloat[Qualitative improvements of worst performing frames from \emph{CBAM-Full + Aug} to \emph{CBAM-Full + Aug + Anch}. \emph{CBAM-Full + Aug} faults in alphabetical order: a) Missed vertical instrument. b) Missed transparent instrument. c) Missed vertical instrument on the edge of the field of view. d) Missed long horizontal instrument on the bottom of the image. e) Single instrument mistaken as two instruments and overlapped masks. f) Two missed detections of long, horizontal instruments.\label{fig:1c}]{\includegraphics[width=0.47\textwidth]{images/cbam-full-aug-anchors_improvements.pdf}} \hfill
\subfloat[Qualitative improvements of worst performing frames from \emph{CBAM-Full + Aug + Anch} to \emph{CBAM-Full + Aug + Anch + MS}. \emph{CBAM-Full + Aug + Anch} faults in alphabetical order: a) Missed underexposed instrument. b) Missed transparent instrument. c) Missed blurred instruments. d) Missed blurred instrument due to reflection. e) Missed underexposed instrument. f) Missed detection due to motion artifacts.\label{fig:1d}]{\includegraphics[width=0.47\textwidth]{images/full-config.pdf}}
\caption{Qualitative comparison of several challenges instances by incremental improvements in our proposed model} \label{fig:total}
\end{figure*}
\subsubsection{Qualitative results}
To better understand the effects of the different components of our models, we performed a comparative analysis of the 100 frames with the worst performance of each network, as show in Figure \ref{fig:total}. \\ \\
Figure \ref{fig:1a} shows the qualitative comparison between worst frames of \emph{Base YOLACT++} and their corresponding frames from \emph{CBAM-Full}. We can observe from the images on the left side that the baseline model often presents missed detections, which hinders the model’s recall. Similarly, the model confuses tissue and other objects like bandages as instruments, evidencing its lack of robustness. On the other hand, \emph{CBAM-Full} overcomes some of these problems by attending to the important features, leading to better localization and segmentation results. We identified four different conditions that seemed particularly challenging for \emph{CBAM-Full}: transparent instruments, vertical instruments, small instruments on the edge of the field of view, and partially occluded instruments. According to \cite{Ross_2021}, most of these issues have also been challenging to previous participants. Figure \ref{fig:1b} illustrates the comparison of worst frames of \emph{CBAM-Full} and the improvements obtained after training with stronger data augmentation.
In contrast to the baseline model with attention, we can observe that \emph{CBAM-Full + Aug} is capable of addressing problematic instances such as small instruments on the edge of the field of view. Similarly, transparent, partially occluded, and vertical instruments are now detected and segmented to a larger extent.
Howver, \emph{CBAM-Full + Aug} still presented recurrent missed detections on long vertical and transparent instruments. Figure \ref{fig:1c} illustrates the comparison of worst frames of \emph{CBAM-Full + Aug} and the improvements obtained after training with an optimized set of anchors. We can observe that the anchor optimization in \emph{CBAM-Full + Aug + Anch} led to additional detection and segmentation improvements not only on previously undetected objects but across all instruments. Figure \ref{fig:1d} illustrates the comparison of worst frames of \emph{CBAM-Full + Aug + Anch} and the improvements obtained after training with MSFF.
We can observe that \emph{CBAM-Full + Aug + Anch + MS} is better at recognizing more challenging instances, like small under-exposed instruments, transparent instruments, and reflections.
%
\begin{figure}[t!]
\centering
{\includegraphics[width=0.45\textwidth]{images/MIS-Attention.pdf}}
\caption{\textbf{Network output with and without attention.} (top) Large part of image covered with instrument in purple and in blue (left). (bottom) Three instruments two at the bottom of image and one on the top of image (see left). \label{fig:attention}}
\end{figure}
\section{Discussion}
\label{sec:discussion}
%
While deep learning has allowed us to design data driven solutions, generalisation remains a key issue that can cause significant performance degradation, especially in instrument segmentation applications where the intervention objects are of variable shapes and exposed to dynamic environments (e.g., smoke, flares, specularity etc, see Figure~\ref{fig:sampleimages}). In surgery, real-time performance of tool segmentation is of enormous importance for clinical utility. However, most methods build in the past relied on two stage networks that are not computationally efficient.
To this end, we built over single stage YOLACT~\cite{yolact} and added several modifications to improve both the accuracy and robustness of our final surgical instrument segmentation algorithm. Among our models, we observed that those based on CBAM achieved slightly better performance that the ones based on CCAM. Regardless, attention-integrated models always outperformed the attention-less baseline in terms of robustness. Figure~\ref{fig:boxplots_stage1} shows dot-and-boxplots of the metric values obtained by each algorithm over all test cases in stage 3 dataset (unseen test dataset in the MIS challenge).
We can observe that adding attention mechanisms boosts the performance compared to the baseline model used in our architecture design~\cite{yolact}. This especially true for instances below the third quartile, which are the most important for our performance metrics. Among the three model variations to which we added CCAM attention modules, CCAM-Backbone achieved the best results in terms of robustness (0.313 MI\_DSC and 0.338 MI\_NSD, see Table~\ref{tab:metric_scores}). This indicates that the contextually enriched feature maps from the ResNet-101 backbone are powerful enough to generate more accurate mask prototypes and coefficients in the YOLACT architecture, and ultimately better segmentation outputs. However, the CBAM-Full model outperformed all other attention modules by nearly 9\% on MI\_DSC over the best performing CCAM-Backbone.
The superiority of CBAM when integrated together with backbone and FPN all together enhances both channel-wise and block attentions better to represent local and global features well. The FPN layer allows to capture size variability present in the dataset for which CCAM yielded zero on aggregated MI\_DSC score (Figure~\ref{fig:boxplots_stage1}).
We thoroughly investigated on model improvement for generalisability through different mechanisms such as data augmentation and optimisation of anchor weights. These steps were experimentally proven to be right directions giving subsequent increase in both aggregated MI\_DSC and MI\_NSD (see Table~\ref{tab:metric_scores}, Figure~\ref{fig:1a} and Figure~\ref{fig:1b}). For example, by using augmentation, we observed that the images with different view points (mostly long sized instruments either straight or slightly oblique) performed better; this can be because of low number of such samples in the dataset (Figure~\ref{fig:1b}). Similarly, for the variable size instruments optimising anchors provided substantial improvements (Figure~\ref{fig:1c}). We also noticed issues with the small instrument appearing either on the bottom or sides of the frame, and also for those which appeared as background, due to specularity or covering large tissue area with similar color (see Figure~\ref{fig:1c}, left). The use of the Multi-scale feature fusion (MSFF) network allowed us to fuse features at different scales and layers (Figure~\ref{fig:1d}).
Our architecture with MSFF network integrated together with attention maps allowed to transfer both local and global context fusing high and low-level feature representations. As a result, we observed further improvement of nearly 2\% on aggregated MI\_DSC above the CBAM-Full with augmentation and anchor approach (Table~\ref{tab:metric_scores}). Our final best approach has near real-time performance of 24 FPS on NVIDIA Volta GPU. However, our second best performing method can run at 69 FPS which is above the required real-time performance in most cases (i.e., 45 FPS). A similar, performance improvement was observed for other two test datasets (Table~\ref{tab:suppTab1}).
To better understand the behavior of the two types of attention modules implemented in our experiments, we visualize the attention maps generated by 3 different models, in addition to the feature maps of the baseline model (see Figure \ref{fig:attention}). We chose the \emph{CBAM-Full}, \emph{CCAM-Full}, and \emph{CBAM-Full + Aug + Anch + MS} configurations for visualization purposes as they show the refined feature maps using the two different attentions modules. %\emph{CBAM-Full} and \emph{CCAM-Full} provide insights about the differences between both attention modules, and since \emph{CBAM-Full + Aug + Anch + MS} is our top model, we are able to compare on both ends of the performance space.}
Figure \ref{fig:attention} illustrate the activation maps taken before and after the attention modules. We visualize the effects of attention on this specific point of the network since mask prototypes and coefficients are extracted from FPN features and backbone network (Full). We can observe that over-mixing problem of \emph{CCAM-Full}. In fact, activation maps before CCAM in the Full model are cleaner and more discriminative than the actual attention output. This confirms that CCAM excels at refining features that have not been mixed before.
% In some cases, the quality of the feature maps generated before CCAM in the FPN even rivals the refined features generated by \emph{CBAM-Full} and \emph{CBAM-Full + Aug + Anch + MS}.
On the other hand, CBAM-based models produce slightly lower quality feature maps from attention modules in the backbone, as it can be observed by the clouds and blobs on the top maps of the figure. Nonetheless, the features are drastically refined after passing the attention modules in the Full, leading to clean and discriminative activations. This corroborates our hypothesis that CBAM is better at refining previously aggregated data, yielding a superior overall performance.
Furthermore, our current work is built upon publicly available retrospective dataset providing a strong evidence of robustness ability compared to presented methods at the MIS challenge and current literature. In future work, we will validate on prospective data and benchmark it in clinical settings.
% These discoveries provide useful insights about the potential of combining both types of attention mechanisms for future experiments. A promising approach would be to combine CCAM in the backbone to exploit the features that have not yet been mixed, and CBAM in the FPN to further refined the aggregated context from CCAM and FPN. This approach has the potential to combine the strengths of both attention modules and reduce their individual weaknesses.
\section{Conclusion}
\label{sec:conclusion}
%
We have developed a real-time novel architecture that builds upon a single-stage instance segmentation method. Through our step-wise solution to different problems in surgical data, we have identified and integrated components that can be deal with existing challenges for robust segmentation of surgical instruments. We have provided comprehensive experiments and analysis that supports our final architecture development and its impact on surgical tool segmentation in clinic. Our method outperformed all methods reported in recently conducted ROBUST-MIS challenge.
% \section*{Acknowledgments}
% SA is supported by NIHR Oxford Biomedical Research center
\section*{Acknowledgements}
\noindent
The authors would like to thank the AI Hub and the CIIOT at Tecnologico de Monterrey for their support for carrying the experiments on their NVIDIA's DGX computer.
%% Loading bibliography style file
%\bibliographystyle{model1-num-names}
\bibliographystyle{cas-model2-names}
% Loading bibliography database
\bibliography{cas-refs}
\end{document}
\begin{thebibliography}{55}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\DOIprefix}{doi:}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{http://dx.doi.org/#1}{\path{#1}}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Article
\bibitem[{Ali et~al.(2021)Ali, Dmitrieva, Ghatwary, Bano, Polat, Temizel,
Krenzer, Hekalo, Guo, Matuszewski, Gridach, Voiculescu, Yoganand, Chavan,
Raj, Nguyen, Tran, Huynh, Boutry, Rezvy, Chen, Choi, Subramanian,
Balasubramanian, Gao, Hu, Liao, Stoyanov, Daul, Realdon, Cannizzaro,
Lamarque, Tran-Nguyen, Bailey, Braden, East and Rittscher}]{Ali_2021}
\bibinfo{author}{Ali, S.}, \bibinfo{author}{Dmitrieva, M.},
\bibinfo{author}{Ghatwary, N.}, \bibinfo{author}{Bano, S.},
\bibinfo{author}{Polat, G.}, \bibinfo{author}{Temizel, A.},
\bibinfo{author}{Krenzer, A.}, \bibinfo{author}{Hekalo, A.},
\bibinfo{author}{Guo, Y.B.}, \bibinfo{author}{Matuszewski, B.},
\bibinfo{author}{Gridach, M.}, \bibinfo{author}{Voiculescu, I.},
\bibinfo{author}{Yoganand, V.}, \bibinfo{author}{Chavan, A.},
\bibinfo{author}{Raj, A.}, \bibinfo{author}{Nguyen, N.T.},
\bibinfo{author}{Tran, D.Q.}, \bibinfo{author}{Huynh, L.D.},
\bibinfo{author}{Boutry, N.}, \bibinfo{author}{Rezvy, S.},
\bibinfo{author}{Chen, H.}, \bibinfo{author}{Choi, Y.H.},
\bibinfo{author}{Subramanian, A.}, \bibinfo{author}{Balasubramanian, V.},
\bibinfo{author}{Gao, X.W.}, \bibinfo{author}{Hu, H.}, \bibinfo{author}{Liao,
Y.}, \bibinfo{author}{Stoyanov, D.}, \bibinfo{author}{Daul, C.},
\bibinfo{author}{Realdon, S.}, \bibinfo{author}{Cannizzaro, R.},
\bibinfo{author}{Lamarque, D.}, \bibinfo{author}{Tran-Nguyen, T.},
\bibinfo{author}{Bailey, A.}, \bibinfo{author}{Braden, B.},
\bibinfo{author}{East, J.E.}, \bibinfo{author}{Rittscher, J.},
\bibinfo{year}{2021}.
\newblock \bibinfo{title}{Deep learning for detection and segmentation of
artefact and disease instances in gastrointestinal endoscopy}.
\newblock \bibinfo{journal}{Medical Image Analysis} \bibinfo{volume}{70},
\bibinfo{pages}{102002}.
\newblock \DOIprefix\doi{https://doi.org/10.1016/j.media.2021.102002}.
%Type = Article
\bibitem[{Ali et~al.(2020)Ali, Zhou, Braden, Bailey, Yang, Cheng, Zhang, Li,
Kayser, Soberanis-Mukul, Albarqouni, Wang, Wang, Watanabe, Oksuz, Ning, Yang,
Khan, Gao, Realdon, Loshchenov, Schnabel, East, Wagnieres, Loschenov, Grisan,
Daul, Blondel and Rittscher}]{Ali2020}
\bibinfo{author}{Ali, S.}, \bibinfo{author}{Zhou, F.}, \bibinfo{author}{Braden,
B.}, \bibinfo{author}{Bailey, A.}, \bibinfo{author}{Yang, S.},
\bibinfo{author}{Cheng, G.}, \bibinfo{author}{Zhang, P.},
\bibinfo{author}{Li, X.}, \bibinfo{author}{Kayser, M.},
\bibinfo{author}{Soberanis-Mukul, R.D.}, \bibinfo{author}{Albarqouni, S.},
\bibinfo{author}{Wang, X.}, \bibinfo{author}{Wang, C.},
\bibinfo{author}{Watanabe, S.}, \bibinfo{author}{Oksuz, I.},
\bibinfo{author}{Ning, Q.}, \bibinfo{author}{Yang, S.},
\bibinfo{author}{Khan, M.A.}, \bibinfo{author}{Gao, X.W.},
\bibinfo{author}{Realdon, S.}, \bibinfo{author}{Loshchenov, M.},
\bibinfo{author}{Schnabel, J.A.}, \bibinfo{author}{East, J.E.},
\bibinfo{author}{Wagnieres, G.}, \bibinfo{author}{Loschenov, V.B.},
\bibinfo{author}{Grisan, E.}, \bibinfo{author}{Daul, C.},
\bibinfo{author}{Blondel, W.}, \bibinfo{author}{Rittscher, J.},
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{An objective comparison of detection and segmentation
algorithms for artefacts in clinical endoscopy}.
\newblock \bibinfo{journal}{Scientific Reports} \bibinfo{volume}{10},
\bibinfo{pages}{2748}.
\newblock \DOIprefix\doi{10.1038/s41598-020-59413-5}.
%Type = Article
\bibitem[{Allan et~al.(2020)Allan, Kondo, Bodenstedt, Leger, Kadkhodamohammadi,
Luengo, Fuentes-Hurtado, Flouty, Mohammed, Pedersen, Kori, Varghese,
Krishnamurthi, Rauber, Mendel, Palm, Bano, Saibro, Shih, Chiang, Zhuang,
Yang, Iglovikov, Dobrenkii, Reddiboina, Reddy, Liu, Gao, Unberath, Azizian,
Stoyanov, Maier-Hein and Speidel}]{Allan20202018RS}
\bibinfo{author}{Allan, M.}, \bibinfo{author}{Kondo, S.},
\bibinfo{author}{Bodenstedt, S.}, \bibinfo{author}{Leger, S.},
\bibinfo{author}{Kadkhodamohammadi, R.}, \bibinfo{author}{Luengo, I.},
\bibinfo{author}{Fuentes-Hurtado, F.}, \bibinfo{author}{Flouty, E.},
\bibinfo{author}{Mohammed, A.K.}, \bibinfo{author}{Pedersen, M.},
\bibinfo{author}{Kori, A.}, \bibinfo{author}{Varghese, A.},
\bibinfo{author}{Krishnamurthi, G.}, \bibinfo{author}{Rauber, D.},
\bibinfo{author}{Mendel, R.}, \bibinfo{author}{Palm, C.},
\bibinfo{author}{Bano, S.}, \bibinfo{author}{Saibro, G.},
\bibinfo{author}{Shih, C.S.}, \bibinfo{author}{Chiang, H.A.},
\bibinfo{author}{Zhuang, J.}, \bibinfo{author}{Yang, J.},
\bibinfo{author}{Iglovikov, V.I.}, \bibinfo{author}{Dobrenkii, A.},
\bibinfo{author}{Reddiboina, M.}, \bibinfo{author}{Reddy, A.},
\bibinfo{author}{Liu, X.}, \bibinfo{author}{Gao, C.},
\bibinfo{author}{Unberath, M.}, \bibinfo{author}{Azizian, M.},
\bibinfo{author}{Stoyanov, D.}, \bibinfo{author}{Maier-Hein, L.},
\bibinfo{author}{Speidel, S.}, \bibinfo{year}{2020}.
\newblock \bibinfo{title}{2018 robotic scene segmentation challenge}.
\newblock \bibinfo{journal}{ArXiv} \bibinfo{volume}{abs/2001.11190}.
%Type = Article
\bibitem[{Allan et~al.(2019)Allan, Shvets, Kurmann, Zhang, Duggal, Su, Rieke,
Laina, Kalavakonda, Bodenstedt, Garc{\'i}a-Peraza, Li, Iglovikov, Luo, Yang,
Stoyanov, Maier-Hein, Speidel and Azizian}]{Allan20192017RI}
\bibinfo{author}{Allan, M.}, \bibinfo{author}{Shvets, A.A.},
\bibinfo{author}{Kurmann, T.}, \bibinfo{author}{Zhang, Z.V.},
\bibinfo{author}{Duggal, R.}, \bibinfo{author}{Su, Y.H.},
\bibinfo{author}{Rieke, N.}, \bibinfo{author}{Laina, I.},
\bibinfo{author}{Kalavakonda, N.}, \bibinfo{author}{Bodenstedt, S.},
\bibinfo{author}{Garc{\'i}a-Peraza, L.C.}, \bibinfo{author}{Li, W.},
\bibinfo{author}{Iglovikov, V.I.}, \bibinfo{author}{Luo, H.},
\bibinfo{author}{Yang, J.}, \bibinfo{author}{Stoyanov, D.},
\bibinfo{author}{Maier-Hein, L.}, \bibinfo{author}{Speidel, S.},
\bibinfo{author}{Azizian, M.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{2017 robotic instrument segmentation challenge}.
\newblock \bibinfo{journal}{ArXiv} \bibinfo{volume}{abs/1902.06426}.
%Type = Article
\bibitem[{Bartoli et~al.(2012)Bartoli, Collins, Bourdel and
Canis}]{Bartoli_2012}
\bibinfo{author}{Bartoli, A.}, \bibinfo{author}{Collins, T.},
\bibinfo{author}{Bourdel, N.}, \bibinfo{author}{Canis, M.},
\bibinfo{year}{2012}.
\newblock \bibinfo{title}{Computer assisted minimally invasive surgery: is
medical computer vision the answer to improving laparosurgery?}
\newblock \bibinfo{journal}{Med Hypotheses} \bibinfo{volume}{79},
\bibinfo{pages}{858--863}.
%Type = Misc
\bibitem[{Bodenstedt et~al.(2018)Bodenstedt, Allan, Agustinos, Du,
Garcia-Peraza-Herrera, Kenngott, Kurmann, Müller-Stich, Ourselin, Pakhomov,
Sznitman, Teichmann, Thoma, Vercauteren, Voros, Wagner, Wochner, Maier-Hein,
Stoyanov and Speidel}]{Bodenstedt_2018}
\bibinfo{author}{Bodenstedt, S.}, \bibinfo{author}{Allan, M.},
\bibinfo{author}{Agustinos, A.}, \bibinfo{author}{Du, X.},
\bibinfo{author}{Garcia-Peraza-Herrera, L.}, \bibinfo{author}{Kenngott, H.},
\bibinfo{author}{Kurmann, T.}, \bibinfo{author}{Müller-Stich, B.},
\bibinfo{author}{Ourselin, S.}, \bibinfo{author}{Pakhomov, D.},
\bibinfo{author}{Sznitman, R.}, \bibinfo{author}{Teichmann, M.},
\bibinfo{author}{Thoma, M.}, \bibinfo{author}{Vercauteren, T.},
\bibinfo{author}{Voros, S.}, \bibinfo{author}{Wagner, M.},
\bibinfo{author}{Wochner, P.}, \bibinfo{author}{Maier-Hein, L.},
\bibinfo{author}{Stoyanov, D.}, \bibinfo{author}{Speidel, S.},
\bibinfo{year}{2018}.
\newblock \bibinfo{title}{Comparative evaluation of instrument segmentation and
tracking methods in minimally invasive surgery}.
\newblock \href{http://arxiv.org/abs/1805.02475}{\tt arXiv:1805.02475}.
%Type = Article
\bibitem[{Bolya et~al.(2019)Bolya, Zhou, Xiao and Lee}]{yolact}
\bibinfo{author}{Bolya, D.}, \bibinfo{author}{Zhou, C.}, \bibinfo{author}{Xiao,
F.}, \bibinfo{author}{Lee, Y.J.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{{YOLACT:} real-time instance segmentation}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1904.02689}.
%Type = Article
\bibitem[{Bolya et~al.(2020)Bolya, Zhou, Xiao and Lee}]{Bolya_2020}
\bibinfo{author}{Bolya, D.}, \bibinfo{author}{Zhou, C.}, \bibinfo{author}{Xiao,
F.}, \bibinfo{author}{Lee, Y.J.}, \bibinfo{year}{2020}.
\newblock \bibinfo{title}{{YOLACT++:} better real-time instance segmentation}.
\newblock \bibinfo{journal}{IEEE Transactions on Pattern Analysis and Machine
Intelligence} ,
\bibinfo{pages}{1–1}\DOIprefix\doi{10.1109/tpami.2020.3014297}.
%Type = Article
\bibitem[{Caroff et~al.(2019)Caroff, Chan, Kleinman, Calderwood, Wolf, Wick,
Platt and Huang}]{Daniel2019:JAMANet}
\bibinfo{author}{Caroff, D.A.}, \bibinfo{author}{Chan, C.},
\bibinfo{author}{Kleinman, K.}, \bibinfo{author}{Calderwood, M.S.},
\bibinfo{author}{Wolf, R.}, \bibinfo{author}{Wick, E.C.},
\bibinfo{author}{Platt, R.}, \bibinfo{author}{Huang, S.},
\bibinfo{year}{2019}.
\newblock \bibinfo{title}{{Association of Open Approach vs Laparoscopic
Approach With Risk of Surgical Site Infection After Colon Surgery}}.
\newblock \bibinfo{journal}{JAMA Network Open} \bibinfo{volume}{2},
\bibinfo{pages}{e1913570--e1913570}.
%Type = Article
\bibitem[{Chaudhari et~al.(2019)Chaudhari, Polatkan, Ramanath and
Mithal}]{Chaudhari2019AnAS}
\bibinfo{author}{Chaudhari, S.}, \bibinfo{author}{Polatkan, G.},
\bibinfo{author}{Ramanath, R.}, \bibinfo{author}{Mithal, V.},
\bibinfo{year}{2019}.
\newblock \bibinfo{title}{An attentive survey of attention models}.
\newblock \bibinfo{journal}{ArXiv} \bibinfo{volume}{abs/1904.02874}.
%Type = Misc
\bibitem[{Chen et~al.(2020)Chen, Sun, Tian, Shen, Huang and Yan}]{blendmask}
\bibinfo{author}{Chen, H.}, \bibinfo{author}{Sun, K.}, \bibinfo{author}{Tian,
Z.}, \bibinfo{author}{Shen, C.}, \bibinfo{author}{Huang, Y.},
\bibinfo{author}{Yan, Y.}, \bibinfo{year}{2020}.
\newblock \bibinfo{title}{Blendmask: Top-down meets bottom-up for instance
segmentation}.
\newblock \href{http://arxiv.org/abs/2001.00309}{\tt arXiv:2001.00309}.
%Type = Article
\bibitem[{Chen et~al.(2018)Chen, Zhu, Papandreou, Schroff and Adam}]{deeplab}
\bibinfo{author}{Chen, L.}, \bibinfo{author}{Zhu, Y.},
\bibinfo{author}{Papandreou, G.}, \bibinfo{author}{Schroff, F.},
\bibinfo{author}{Adam, H.}, \bibinfo{year}{2018}.
\newblock \bibinfo{title}{Encoder-decoder with atrous separable convolution for
semantic image segmentation}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1802.02611}.
%Type = Inproceedings
\bibitem[{{Ding} et~al.(2018){Ding}, {Jiang}, {Shuai}, {Liu} and
{Wang}}]{Ding_2018}
\bibinfo{author}{{Ding}, H.}, \bibinfo{author}{{Jiang}, X.},
\bibinfo{author}{{Shuai}, B.}, \bibinfo{author}{{Liu}, A.Q.},
\bibinfo{author}{{Wang}, G.}, \bibinfo{year}{2018}.
\newblock \bibinfo{title}{Context contrasted feature and gated multi-scale
aggregation for scene segmentation}, in: \bibinfo{booktitle}{2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition}, pp.
\bibinfo{pages}{2393--2402}.
\newblock \DOIprefix\doi{10.1109/CVPR.2018.00254}.
%Type = Article
\bibitem[{Fu et~al.(2021)Fu, Jin, Zhang, He, Zha, Hu, Gan, Yan, Wang and
Ye}]{fu2021}
\bibinfo{author}{Fu, Z.}, \bibinfo{author}{Jin, Z.}, \bibinfo{author}{Zhang,
C.}, \bibinfo{author}{He, Z.}, \bibinfo{author}{Zha, Z.},
\bibinfo{author}{Hu, C.}, \bibinfo{author}{Gan, T.}, \bibinfo{author}{Yan,
Q.}, \bibinfo{author}{Wang, P.}, \bibinfo{author}{Ye, X.},
\bibinfo{year}{2021}.
\newblock \bibinfo{title}{The future of endoscopic navigation: A review of
advanced endoscopic vision technology}.
\newblock \bibinfo{journal}{IEEE Access} \bibinfo{volume}{9},
\bibinfo{pages}{41144--41167}.
\newblock \DOIprefix\doi{10.1109/ACCESS.2021.3065104}.
%Type = Article
\bibitem[{Gandaglia et~al.(2014)Gandaglia, Ghani, Sood, Meyers, Sammon, Schmid,
Varda, Briganti, Montorsi, Sun, Menon, Kibel and
Trinh}]{Giorgio2004:JAMASurg}
\bibinfo{author}{Gandaglia, G.}, \bibinfo{author}{Ghani, K.R.},
\bibinfo{author}{Sood, A.}, \bibinfo{author}{Meyers, J.R.},
\bibinfo{author}{Sammon, J.D.}, \bibinfo{author}{Schmid, M.},
\bibinfo{author}{Varda, B.}, \bibinfo{author}{Briganti, A.},
\bibinfo{author}{Montorsi, F.}, \bibinfo{author}{Sun, M.},
\bibinfo{author}{Menon, M.}, \bibinfo{author}{Kibel, A.S.},
\bibinfo{author}{Trinh, Q.D.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{{Effect of Minimally Invasive Surgery on the Risk for
Surgical Site Infections: Results From the National Surgical Quality
Improvement Program (NSQIP) Database}}.
\newblock \bibinfo{journal}{JAMA Surgery} \bibinfo{volume}{149},
\bibinfo{pages}{1039--1044}.
%Type = Article
\bibitem[{Gu et~al.(2020)Gu, Wang, Song, Huang, Aertsen, Deprest, Ourselin,
Vercauteren and Zhang}]{Gu_2020}
\bibinfo{author}{Gu, R.}, \bibinfo{author}{Wang, G.}, \bibinfo{author}{Song,
T.}, \bibinfo{author}{Huang, R.}, \bibinfo{author}{Aertsen, M.},
\bibinfo{author}{Deprest, J.}, \bibinfo{author}{Ourselin, S.},
\bibinfo{author}{Vercauteren, T.}, \bibinfo{author}{Zhang, S.},
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Ca-net: Comprehensive attention convolutional neural
networks for explainable medical image segmentation}.
\newblock \bibinfo{journal}{IEEE Transactions on Medical Imaging} ,
\bibinfo{pages}{1–1}\DOIprefix\doi{10.1109/tmi.2020.3035253}.
%Type = Article
\bibitem[{He et~al.(2017)He, Gkioxari, Doll{\'{a}}r and Girshick}]{mask-rcnn}
\bibinfo{author}{He, K.}, \bibinfo{author}{Gkioxari, G.},
\bibinfo{author}{Doll{\'{a}}r, P.}, \bibinfo{author}{Girshick, R.B.},
\bibinfo{year}{2017}.
\newblock \bibinfo{title}{Mask {R-CNN}}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1703.06870}.
\newblock \href{http://arxiv.org/abs/1703.06870}{\tt arXiv:1703.06870}.
%Type = Misc
\bibitem[{He et~al.(2018)He, Gkioxari, Dollár and Girshick}]{He_2018}
\bibinfo{author}{He, K.}, \bibinfo{author}{Gkioxari, G.},
\bibinfo{author}{Dollár, P.}, \bibinfo{author}{Girshick, R.},
\bibinfo{year}{2018}.
\newblock \bibinfo{title}{Mask r-cnn}.
\newblock \href{http://arxiv.org/abs/1703.06870}{\tt arXiv:1703.06870}.
%Type = Misc
\bibitem[{He et~al.(2015)He, Zhang, Ren and Sun}]{He_2015}
\bibinfo{author}{He, K.}, \bibinfo{author}{Zhang, X.}, \bibinfo{author}{Ren,
S.}, \bibinfo{author}{Sun, J.}, \bibinfo{year}{2015}.
\newblock \bibinfo{title}{Deep residual learning for image recognition}.
\newblock \href{http://arxiv.org/abs/1512.03385}{\tt arXiv:1512.03385}.
%Type = Misc
\bibitem[{Huang et~al.(2020)Huang, Wang, Wei, Huang, Shi, Liu and
Huang}]{Huang_2020}
\bibinfo{author}{Huang, Z.}, \bibinfo{author}{Wang, X.}, \bibinfo{author}{Wei,
Y.}, \bibinfo{author}{Huang, L.}, \bibinfo{author}{Shi, H.},
\bibinfo{author}{Liu, W.}, \bibinfo{author}{Huang, T.S.},
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Ccnet: Criss-cross attention for semantic
segmentation}.
\newblock \href{http://arxiv.org/abs/1811.11721}{\tt arXiv:1811.11721}.
%Type = Misc
\bibitem[{Hurtik et~al.(2020)Hurtik, Molek, Hula, Vajgl, Vlasanek and
Nejezchleba}]{polyyolo}
\bibinfo{author}{Hurtik, P.}, \bibinfo{author}{Molek, V.},
\bibinfo{author}{Hula, J.}, \bibinfo{author}{Vajgl, M.},
\bibinfo{author}{Vlasanek, P.}, \bibinfo{author}{Nejezchleba, T.},
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Poly-yolo: higher speed, more precise detection and
instance segmentation for yolov3}.
\newblock \href{http://arxiv.org/abs/2005.13243}{\tt arXiv:2005.13243}.
%Type = Misc
\bibitem[{Isensee and Maier-Hein(2020)}]{orunet}
\bibinfo{author}{Isensee, F.}, \bibinfo{author}{Maier-Hein, K.H.},
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Or-unet: an optimized robust residual u-net for
instrument segmentation in endoscopic images}.
\newblock \href{http://arxiv.org/abs/2004.12668}{\tt arXiv:2004.12668}.
%Type = Inproceedings
\bibitem[{Jha et~al.(2021)Jha, Ali, Tomar, Riegler, Johansen, Johansen and
Halvorsen}]{Jha:BHI2021}
\bibinfo{author}{Jha, D.}, \bibinfo{author}{Ali, S.}, \bibinfo{author}{Tomar,
N.K.}, \bibinfo{author}{Riegler, M.A.}, \bibinfo{author}{Johansen, D.},
\bibinfo{author}{Johansen, H.D.}, \bibinfo{author}{Halvorsen, P.},
\bibinfo{year}{2021}.
\newblock \bibinfo{title}{Exploring deep learning methods for real-time
surgical instrument segmentation in laparoscopy}, in:
\bibinfo{booktitle}{2021 IEEE EMBS International Conference on Biomedical and
Health Informatics (BHI)}, pp. \bibinfo{pages}{1--4}.
\newblock \DOIprefix\doi{10.1109/BHI50953.2021.9508610}.
%Type = Inproceedings
\bibitem[{{Kaul} et~al.(2019){Kaul}, {Manandhar} and {Pears}}]{Kaul_2019}
\bibinfo{author}{{Kaul}, C.}, \bibinfo{author}{{Manandhar}, S.},
\bibinfo{author}{{Pears}, N.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{Focusnet: An attention-based fully convolutional
network for medical image segmentation}, in: \bibinfo{booktitle}{2019 IEEE
16th International Symposium on Biomedical Imaging (ISBI 2019)}, pp.
\bibinfo{pages}{455--458}.
\newblock \DOIprefix\doi{10.1109/ISBI.2019.8759477}.
%Type = Article
\bibitem[{Lee and Park(2019)}]{center-mask}
\bibinfo{author}{Lee, Y.}, \bibinfo{author}{Park, J.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{Centermask : Real-time anchor-free instance
segmentation}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1911.06667}.
%Type = Inproceedings
\bibitem[{Lin et~al.(2017a)Lin, Milan, Shen and Reid}]{Milan_2017}
\bibinfo{author}{Lin, G.}, \bibinfo{author}{Milan, A.}, \bibinfo{author}{Shen,
C.}, \bibinfo{author}{Reid, I.}, \bibinfo{year}{2017}a.
\newblock \bibinfo{title}{Refinenet: Multi-path refinement networks for
high-resolution semantic segmentation}, in: \bibinfo{booktitle}{Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
pp. \bibinfo{pages}{1925--1934}.
%Type = Article
\bibitem[{Lin et~al.(2017b)Lin, Goyal, Girshick, He and
Doll{\'{a}}r}]{retina-net}
\bibinfo{author}{Lin, T.}, \bibinfo{author}{Goyal, P.},
\bibinfo{author}{Girshick, R.B.}, \bibinfo{author}{He, K.},
\bibinfo{author}{Doll{\'{a}}r, P.}, \bibinfo{year}{2017}b.
\newblock \bibinfo{title}{Focal loss for dense object detection}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1708.02002}.
%Type = Misc
\bibitem[{Lin et~al.(2017c)Lin, Dollár, Girshick, He, Hariharan and
Belongie}]{Lin_2017}
\bibinfo{author}{Lin, T.Y.}, \bibinfo{author}{Dollár, P.},
\bibinfo{author}{Girshick, R.}, \bibinfo{author}{He, K.},
\bibinfo{author}{Hariharan, B.}, \bibinfo{author}{Belongie, S.},
\bibinfo{year}{2017}c.
\newblock \bibinfo{title}{Feature pyramid networks for object detection}.
\newblock \href{http://arxiv.org/abs/1612.03144}{\tt arXiv:1612.03144}.
%Type = Article
\bibitem[{Liu et~al.(2016)Liu, Anguelov, Erhan, Szegedy, Reed, Fu and
Berg}]{Liu_2016}
\bibinfo{author}{Liu, W.}, \bibinfo{author}{Anguelov, D.},
\bibinfo{author}{Erhan, D.}, \bibinfo{author}{Szegedy, C.},
\bibinfo{author}{Reed, S.}, \bibinfo{author}{Fu, C.Y.},
\bibinfo{author}{Berg, A.C.}, \bibinfo{year}{2016}.
\newblock \bibinfo{title}{Ssd: Single shot multibox detector}.
\newblock \bibinfo{journal}{Lecture Notes in Computer Science} ,
\bibinfo{pages}{21–37}\DOIprefix\doi{10.1007/978-3-319-46448-0_2}.
%Type = Inproceedings
\bibitem[{Long et~al.(2015a)Long, Shelhamer and Darrell}]{Long:CVPR15}
\bibinfo{author}{Long, J.}, \bibinfo{author}{Shelhamer, E.},
\bibinfo{author}{Darrell, T.}, \bibinfo{year}{2015}a.
\newblock \bibinfo{title}{Fully convolutional networks for semantic
segmentation}, in: \bibinfo{booktitle}{2015 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR)}, pp. \bibinfo{pages}{3431--3440}.
\newblock \DOIprefix\doi{10.1109/CVPR.2015.7298965}.
%Type = Misc
\bibitem[{Long et~al.(2015b)Long, Shelhamer and Darrell}]{Long_2015}
\bibinfo{author}{Long, J.}, \bibinfo{author}{Shelhamer, E.},
\bibinfo{author}{Darrell, T.}, \bibinfo{year}{2015}b.
\newblock \bibinfo{title}{Fully convolutional networks for semantic
segmentation}.
\newblock \href{http://arxiv.org/abs/1411.4038}{\tt arXiv:1411.4038}.
%Type = Misc
\bibitem[{Maier-Hein et~al.(2021a)Maier-Hein, Eisenmann, Sarikaya, März,
Collins, Malpani, Fallert, Feussner, Giannarou, Mascagni, Nakawala, Park,
Pugh, Stoyanov, Vedula, Cleary, Fichtinger, Forestier, Gibaud, Grantcharov,
Hashizume, Heckmann-Nötzel, Kenngott, Kikinis, Mündermann, Navab, Onogur,
Sznitman, Taylor, Tizabi, Wagner, Hager, Neumuth, Padoy, Collins, Gockel,
Goedeke, Hashimoto, Joyeux, Lam, Leff, Madani, Marcus, Meireles, Seitel,
Teber, Ückert, Müller-Stich, Jannin and Speidel}]{maierhein2021surgical}
\bibinfo{author}{Maier-Hein, L.}, \bibinfo{author}{Eisenmann, M.},
\bibinfo{author}{Sarikaya, D.}, \bibinfo{author}{März, K.},
\bibinfo{author}{Collins, T.}, \bibinfo{author}{Malpani, A.},
\bibinfo{author}{Fallert, J.}, \bibinfo{author}{Feussner, H.},
\bibinfo{author}{Giannarou, S.}, \bibinfo{author}{Mascagni, P.},
\bibinfo{author}{Nakawala, H.}, \bibinfo{author}{Park, A.},
\bibinfo{author}{Pugh, C.}, \bibinfo{author}{Stoyanov, D.},
\bibinfo{author}{Vedula, S.S.}, \bibinfo{author}{Cleary, K.},
\bibinfo{author}{Fichtinger, G.}, \bibinfo{author}{Forestier, G.},
\bibinfo{author}{Gibaud, B.}, \bibinfo{author}{Grantcharov, T.},
\bibinfo{author}{Hashizume, M.}, \bibinfo{author}{Heckmann-Nötzel, D.},
\bibinfo{author}{Kenngott, H.G.}, \bibinfo{author}{Kikinis, R.},
\bibinfo{author}{Mündermann, L.}, \bibinfo{author}{Navab, N.},
\bibinfo{author}{Onogur, S.}, \bibinfo{author}{Sznitman, R.},
\bibinfo{author}{Taylor, R.H.}, \bibinfo{author}{Tizabi, M.D.},
\bibinfo{author}{Wagner, M.}, \bibinfo{author}{Hager, G.D.},
\bibinfo{author}{Neumuth, T.}, \bibinfo{author}{Padoy, N.},
\bibinfo{author}{Collins, J.}, \bibinfo{author}{Gockel, I.},
\bibinfo{author}{Goedeke, J.}, \bibinfo{author}{Hashimoto, D.A.},
\bibinfo{author}{Joyeux, L.}, \bibinfo{author}{Lam, K.},
\bibinfo{author}{Leff, D.R.}, \bibinfo{author}{Madani, A.},
\bibinfo{author}{Marcus, H.J.}, \bibinfo{author}{Meireles, O.},
\bibinfo{author}{Seitel, A.}, \bibinfo{author}{Teber, D.},
\bibinfo{author}{Ückert, F.}, \bibinfo{author}{Müller-Stich, B.P.},
\bibinfo{author}{Jannin, P.}, \bibinfo{author}{Speidel, S.},
\bibinfo{year}{2021}a.
\newblock \bibinfo{title}{Surgical data science -- from concepts toward
clinical translation}.
\newblock \href{http://arxiv.org/abs/2011.02284}{\tt arXiv:2011.02284}.
%Type = Article
\bibitem[{Maier-Hein et~al.(2021b)Maier-Hein, Wagner, Ross, Reinke, Bodenstedt,
Full, Hempe, Mindroc-Filimon, Scholz, Tran, Bruno, Kisilenko, Müller,
Davitashvili, Capek, Tizabi, Eisenmann, Adler, Gröhl, Schellenberg,
Seidlitz, Lai, Pekdemir, Roethlingshoefer, Both, Bittel, Mengler,
Mündermann, Apitz, Kopp-Schneider, Speidel, Nickel, Probst, Kenngott and
Müller-Stich}]{Maierhein_2021}
\bibinfo{author}{Maier-Hein, L.}, \bibinfo{author}{Wagner, M.},
\bibinfo{author}{Ross, T.}, \bibinfo{author}{Reinke, A.},
\bibinfo{author}{Bodenstedt, S.}, \bibinfo{author}{Full, P.M.},
\bibinfo{author}{Hempe, H.}, \bibinfo{author}{Mindroc-Filimon, D.},
\bibinfo{author}{Scholz, P.}, \bibinfo{author}{Tran, T.N.},
\bibinfo{author}{Bruno, P.}, \bibinfo{author}{Kisilenko, A.},
\bibinfo{author}{Müller, B.}, \bibinfo{author}{Davitashvili, T.},
\bibinfo{author}{Capek, M.}, \bibinfo{author}{Tizabi, M.D.},
\bibinfo{author}{Eisenmann, M.}, \bibinfo{author}{Adler, T.J.},
\bibinfo{author}{Gröhl, J.}, \bibinfo{author}{Schellenberg, M.},
\bibinfo{author}{Seidlitz, S.}, \bibinfo{author}{Lai, T.Y.E.},
\bibinfo{author}{Pekdemir, B.}, \bibinfo{author}{Roethlingshoefer, V.},
\bibinfo{author}{Both, F.}, \bibinfo{author}{Bittel, S.},
\bibinfo{author}{Mengler, M.}, \bibinfo{author}{Mündermann, L.},
\bibinfo{author}{Apitz, M.}, \bibinfo{author}{Kopp-Schneider, A.},
\bibinfo{author}{Speidel, S.}, \bibinfo{author}{Nickel, F.},
\bibinfo{author}{Probst, P.}, \bibinfo{author}{Kenngott, H.G.},
\bibinfo{author}{Müller-Stich, B.P.}, \bibinfo{year}{2021}b.
\newblock \bibinfo{title}{Heidelberg colorectal data set for surgical data
science in the sensor operating room}.
\newblock \bibinfo{journal}{Scientific Data} \bibinfo{volume}{8},
\bibinfo{pages}{101}.
\newblock \DOIprefix\doi{10.1038/s41597-021-00882-2}.
%Type = Inproceedings
\bibitem[{Milletari et~al.(2018)Milletari, Rieke, Baust, Esposito and
Navab}]{milletari2018}
\bibinfo{author}{Milletari, F.}, \bibinfo{author}{Rieke, N.},
\bibinfo{author}{Baust, M.}, \bibinfo{author}{Esposito, M.},
\bibinfo{author}{Navab, N.}, \bibinfo{year}{2018}.
\newblock \bibinfo{title}{Cfcm: Segmentation via coarse to fine context
memory}, in: \bibinfo{editor}{Frangi, A.F.}, \bibinfo{editor}{Schnabel,
J.A.}, \bibinfo{editor}{Davatzikos, C.}, \bibinfo{editor}{Alberola-L{\'o}pez,
C.}, \bibinfo{editor}{Fichtinger, G.} (Eds.), \bibinfo{booktitle}{Medical
Image Computing and Computer Assisted Intervention -- MICCAI 2018},
\bibinfo{publisher}{Springer International Publishing},
\bibinfo{address}{Cham}. pp. \bibinfo{pages}{667--674}.
%Type = Article
\bibitem[{Ni et~al.(2019)Ni, Bian, Xie, Hou, Zhou and Zhou}]{rasnet}
\bibinfo{author}{Ni, Z.}, \bibinfo{author}{Bian, G.}, \bibinfo{author}{Xie,
X.}, \bibinfo{author}{Hou, Z.}, \bibinfo{author}{Zhou, X.},
\bibinfo{author}{Zhou, Y.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{Rasnet: Segmentation for tracking surgical
instruments in surgical videos using refined attention segmentation network}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1905.08663}.
%Type = Article
\bibitem[{Ren et~al.(2015)Ren, He, Girshick and Sun}]{faster-rcnn}
\bibinfo{author}{Ren, S.}, \bibinfo{author}{He, K.}, \bibinfo{author}{Girshick,
R.B.}, \bibinfo{author}{Sun, J.}, \bibinfo{year}{2015}.
\newblock \bibinfo{title}{Faster {R-CNN:} towards real-time object detection
with region proposal networks}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1506.01497}.
%Type = Inproceedings
\bibitem[{Ronneberger et~al.(2015a)Ronneberger, Fischer and
Brox}]{Ronneberger_2015}
\bibinfo{author}{Ronneberger, O.}, \bibinfo{author}{Fischer, P.},
\bibinfo{author}{Brox, T.}, \bibinfo{year}{2015}a.
\newblock \bibinfo{title}{U-net: Convolutional networks for biomedical image
segmentation}, in: \bibinfo{editor}{Navab, N.}, \bibinfo{editor}{Hornegger,
J.}, \bibinfo{editor}{Wells, W.M.}, \bibinfo{editor}{Frangi, A.F.} (Eds.),
\bibinfo{booktitle}{Medical Image Computing and Computer-Assisted
Intervention -- MICCAI 2015}, \bibinfo{publisher}{Springer International
Publishing}, \bibinfo{address}{Cham}. pp. \bibinfo{pages}{234--241}.
%Type = Article
\bibitem[{Ronneberger et~al.(2015b)Ronneberger, Fischer and Brox}]{unet}
\bibinfo{author}{Ronneberger, O.}, \bibinfo{author}{Fischer, P.},
\bibinfo{author}{Brox, T.}, \bibinfo{year}{2015}b.
\newblock \bibinfo{title}{U-net: Convolutional networks for biomedical image
segmentation}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{volume}{abs/1505.04597}.
\newblock \href{http://arxiv.org/abs/1505.04597}{\tt arXiv:1505.04597}.
%Type = Misc
\bibitem[{Roß and Reinke()}]{Ross_2019}
\bibinfo{author}{Roß, T.}, \bibinfo{author}{Reinke, A.}, .
\newblock \bibinfo{title}{Robustmis2019}.
\newblock \URLprefix \url{https://phabricator.mitk.org/source/rmis2019/}.
%Type = Article
\bibitem[{Roß et~al.(2021)Roß, Reinke, Full, Wagner, Kenngott, Apitz, Hempe,
Mindroc-Filimon, Scholz, Tran, Bruno, Arbeláez, Bian, Bodenstedt, Bolmgren,
Bravo-Sánchez, Chen, González, Guo, Halvorsen, Heng, Hosgor, Hou, Isensee,
Jha, Jiang, Jin, Kirtac, Kletz, Leger, Li, Maier-Hein, Ni, Riegler,
Schoeffmann, Shi, Speidel, Stenzel, Twick, Wang, Wang, Wang, Wang, Zhang,
Zhou, Zhu, Wiesenfarth, Kopp-Schneider, Müller-Stich and
Maier-Hein}]{Ross_2021}
\bibinfo{author}{Roß, T.}, \bibinfo{author}{Reinke, A.},
\bibinfo{author}{Full, P.M.}, \bibinfo{author}{Wagner, M.},
\bibinfo{author}{Kenngott, H.}, \bibinfo{author}{Apitz, M.},
\bibinfo{author}{Hempe, H.}, \bibinfo{author}{Mindroc-Filimon, D.},
\bibinfo{author}{Scholz, P.}, \bibinfo{author}{Tran, T.N.},
\bibinfo{author}{Bruno, P.}, \bibinfo{author}{Arbeláez, P.},
\bibinfo{author}{Bian, G.B.}, \bibinfo{author}{Bodenstedt, S.},
\bibinfo{author}{Bolmgren, J.L.}, \bibinfo{author}{Bravo-Sánchez, L.},
\bibinfo{author}{Chen, H.B.}, \bibinfo{author}{González, C.},
\bibinfo{author}{Guo, D.}, \bibinfo{author}{Halvorsen, P.},
\bibinfo{author}{Heng, P.A.}, \bibinfo{author}{Hosgor, E.},
\bibinfo{author}{Hou, Z.G.}, \bibinfo{author}{Isensee, F.},
\bibinfo{author}{Jha, D.}, \bibinfo{author}{Jiang, T.}, \bibinfo{author}{Jin,
Y.}, \bibinfo{author}{Kirtac, K.}, \bibinfo{author}{Kletz, S.},
\bibinfo{author}{Leger, S.}, \bibinfo{author}{Li, Z.},
\bibinfo{author}{Maier-Hein, K.H.}, \bibinfo{author}{Ni, Z.L.},
\bibinfo{author}{Riegler, M.A.}, \bibinfo{author}{Schoeffmann, K.},
\bibinfo{author}{Shi, R.}, \bibinfo{author}{Speidel, S.},
\bibinfo{author}{Stenzel, M.}, \bibinfo{author}{Twick, I.},
\bibinfo{author}{Wang, G.}, \bibinfo{author}{Wang, J.},
\bibinfo{author}{Wang, L.}, \bibinfo{author}{Wang, L.},
\bibinfo{author}{Zhang, Y.}, \bibinfo{author}{Zhou, Y.J.},
\bibinfo{author}{Zhu, L.}, \bibinfo{author}{Wiesenfarth, M.},
\bibinfo{author}{Kopp-Schneider, A.}, \bibinfo{author}{Müller-Stich, B.P.},
\bibinfo{author}{Maier-Hein, L.}, \bibinfo{year}{2021}.
\newblock \bibinfo{title}{Comparative validation of multi-instance instrument
segmentation in endoscopy: Results of the robust-mis 2019 challenge}.
\newblock \bibinfo{journal}{Medical Image Analysis} \bibinfo{volume}{70},
\bibinfo{pages}{101920}.
\newblock \DOIprefix\doi{https://doi.org/10.1016/j.media.2020.101920}.
%Type = Misc
\bibitem[{Seo et~al.(2018)Seo, Lin, Cohen, Shen and Han}]{Seo_2018}
\bibinfo{author}{Seo, P.H.}, \bibinfo{author}{Lin, Z.}, \bibinfo{author}{Cohen,
S.}, \bibinfo{author}{Shen, X.}, \bibinfo{author}{Han, B.},
\bibinfo{year}{2018}.
\newblock \bibinfo{title}{Progressive attention networks for visual attribute
prediction}.
\newblock \href{http://arxiv.org/abs/1606.02393}{\tt arXiv:1606.02393}.
%Type = Article
\bibitem[{Sheetz et~al.(2020)Sheetz, Claflin and Dimick}]{Sheetz2019:JAMANet}
\bibinfo{author}{Sheetz, K.H.}, \bibinfo{author}{Claflin, J.},
\bibinfo{author}{Dimick, J.B.}, \bibinfo{year}{2020}.
\newblock \bibinfo{title}{{Trends in the Adoption of Robotic Surgery for Common
Surgical Procedures}}.
\newblock \bibinfo{journal}{JAMA Network Open} \bibinfo{volume}{3},
\bibinfo{pages}{e1918911--e1918911}.
%Type = Inproceedings
\bibitem[{Shvets et~al.(2018)Shvets, Rakhlin, Kalinin and
Iglovikov}]{shvets2018}
\bibinfo{author}{Shvets, A.A.}, \bibinfo{author}{Rakhlin, A.},
\bibinfo{author}{Kalinin, A.A.}, \bibinfo{author}{Iglovikov, V.I.},
\bibinfo{year}{2018}.
\newblock \bibinfo{title}{Automatic instrument segmentation in robot-assisted
surgery using deep learning}, in: \bibinfo{booktitle}{2018 17th IEEE
International Conference on Machine Learning and Applications (ICMLA)}, pp.
\bibinfo{pages}{624--628}.
\newblock \DOIprefix\doi{10.1109/ICMLA.2018.00100}.
%Type = Article
\bibitem[{{Sinha} and {Dolz}(2021)}]{Sinha_2021}
\bibinfo{author}{{Sinha}, A.}, \bibinfo{author}{{Dolz}, J.},
\bibinfo{year}{2021}.
\newblock \bibinfo{title}{Multi-scale self-guided attention for medical image
segmentation}.
\newblock \bibinfo{journal}{IEEE Journal of Biomedical and Health Informatics}
\bibinfo{volume}{25}, \bibinfo{pages}{121--130}.
\newblock \DOIprefix\doi{10.1109/JBHI.2020.2986926}.
%Type = Article
\bibitem[{{Wang} et~al.(2020){Wang}, {Sun}, {Cheng}, {Jiang}, {Deng}, {Zhao},
{Liu}, {Mu}, {Tan}, {Wang}, {Liu} and {Xiao}}]{Wang_2020}
\bibinfo{author}{{Wang}, J.}, \bibinfo{author}{{Sun}, K.},
\bibinfo{author}{{Cheng}, T.}, \bibinfo{author}{{Jiang}, B.},
\bibinfo{author}{{Deng}, C.}, \bibinfo{author}{{Zhao}, Y.},
\bibinfo{author}{{Liu}, D.}, \bibinfo{author}{{Mu}, Y.},
\bibinfo{author}{{Tan}, M.}, \bibinfo{author}{{Wang}, X.},
\bibinfo{author}{{Liu}, W.}, \bibinfo{author}{{Xiao}, B.},
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Deep high-resolution representation learning for
visual recognition}.
\newblock \bibinfo{journal}{IEEE Transactions on Pattern Analysis and Machine
Intelligence} ,
\bibinfo{pages}{1--1}\DOIprefix\doi{10.1109/TPAMI.2020.2983686}.
%Type = Misc
\bibitem[{Wang et~al.(2020)Wang, Zhang, Kong, Li and Shen}]{solov2}
\bibinfo{author}{Wang, X.}, \bibinfo{author}{Zhang, R.}, \bibinfo{author}{Kong,
T.}, \bibinfo{author}{Li, L.}, \bibinfo{author}{Shen, C.},
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Solov2: Dynamic and fast instance segmentation}.
\newblock \href{http://arxiv.org/abs/2003.10152}{\tt arXiv:2003.10152}.
%Type = Inproceedings
\bibitem[{Wang et~al.(2018)Wang, Deng, Hu, Zhu, Yang, Xu, Heng and
Ni}]{Deng_2018}
\bibinfo{author}{Wang, Y.}, \bibinfo{author}{Deng, Z.}, \bibinfo{author}{Hu,
X.}, \bibinfo{author}{Zhu, L.}, \bibinfo{author}{Yang, X.},
\bibinfo{author}{Xu, X.}, \bibinfo{author}{Heng, P.A.}, \bibinfo{author}{Ni,
D.}, \bibinfo{year}{2018}.
\newblock \bibinfo{title}{Deep attentional features for prostate segmentation
in ultrasound}, in: \bibinfo{editor}{Frangi, A.F.},
\bibinfo{editor}{Schnabel, J.A.}, \bibinfo{editor}{Davatzikos, C.},
\bibinfo{editor}{Alberola-L{\'o}pez, C.}, \bibinfo{editor}{Fichtinger, G.}
(Eds.), \bibinfo{booktitle}{Medical Image Computing and Computer Assisted
Intervention -- MICCAI 2018}, \bibinfo{publisher}{Springer International
Publishing}, \bibinfo{address}{Cham}. pp. \bibinfo{pages}{523--530}.
%Type = Article
\bibitem[{Ward et~al.(2021)Ward, Mascagni, Ban, Rosman, Padoy, Meireles and
Hashimoto}]{Ward2021}
\bibinfo{author}{Ward, T.M.}, \bibinfo{author}{Mascagni, P.},
\bibinfo{author}{Ban, Y.}, \bibinfo{author}{Rosman, G.},
\bibinfo{author}{Padoy, N.}, \bibinfo{author}{Meireles, O.},
\bibinfo{author}{Hashimoto, D.A.}, \bibinfo{year}{2021}.
\newblock \bibinfo{title}{Computer vision in surgery}.
\newblock \bibinfo{journal}{Surgery} \bibinfo{volume}{169},
\bibinfo{pages}{1253--1256}.
\newblock \DOIprefix\doi{10.1016/j.surg.2020.10.039}.
%Type = Article
\bibitem[{Wiesenfarth et~al.(2021)Wiesenfarth, Reinke, Landman, Eisenmann,
Saiz, Cardoso, Maier-Hein and Kopp-Schneider}]{Wiesenfarth_2021}
\bibinfo{author}{Wiesenfarth, M.}, \bibinfo{author}{Reinke, A.},
\bibinfo{author}{Landman, B.A.}, \bibinfo{author}{Eisenmann, M.},
\bibinfo{author}{Saiz, L.A.}, \bibinfo{author}{Cardoso, M.J.},
\bibinfo{author}{Maier-Hein, L.}, \bibinfo{author}{Kopp-Schneider, A.},
\bibinfo{year}{2021}.
\newblock \bibinfo{title}{Methods and open-source toolkit for analyzing and
visualizing challenge results}.
\newblock \bibinfo{journal}{Scientific Reports} \bibinfo{volume}{11},
\bibinfo{pages}{1--15}.
%Type = Misc
\bibitem[{Woo et~al.(2018)Woo, Park, Lee and Kweon}]{Woo_2018}
\bibinfo{author}{Woo, S.}, \bibinfo{author}{Park, J.}, \bibinfo{author}{Lee,
J.Y.}, \bibinfo{author}{Kweon, I.S.}, \bibinfo{year}{2018}.
\newblock \bibinfo{title}{Cbam: Convolutional block attention module}.
\newblock \href{http://arxiv.org/abs/1807.06521}{\tt arXiv:1807.06521}.
%Type = Article
\bibitem[{Xu et~al.(2020)Xu, Wang, Wang and Guo}]{xu2020}
\bibinfo{author}{Xu, J.}, \bibinfo{author}{Wang, W.}, \bibinfo{author}{Wang,
H.}, \bibinfo{author}{Guo, J.}, \bibinfo{year}{2020}.
\newblock \bibinfo{title}{Multi-model ensemble with rich spatial information
for object detection}.
\newblock \bibinfo{journal}{Pattern Recognition} \bibinfo{volume}{99},
\bibinfo{pages}{107098}.
\newblock \DOIprefix\doi{https://doi.org/10.1016/j.patcog.2019.107098}.
%Type = Misc
\bibitem[{Xu et~al.(2016)Xu, Li, Liu, Wang, Lai and Chang}]{Xu_2016}
\bibinfo{author}{Xu, Y.}, \bibinfo{author}{Li, Y.}, \bibinfo{author}{Liu, M.},
\bibinfo{author}{Wang, Y.}, \bibinfo{author}{Lai, M.},
\bibinfo{author}{Chang, E.I.C.}, \bibinfo{year}{2016}.
\newblock \bibinfo{title}{Gland instance segmentation by deep multichannel side
supervision}.
\newblock \href{http://arxiv.org/abs/1607.03222}{\tt arXiv:1607.03222}.
%Type = Inproceedings
\bibitem[{You et~al.(2016)You, Jin, Wang, Fang and Luo}]{You_2016}
\bibinfo{author}{You, Q.}, \bibinfo{author}{Jin, H.}, \bibinfo{author}{Wang,
Z.}, \bibinfo{author}{Fang, C.}, \bibinfo{author}{Luo, J.},
\bibinfo{year}{2016}.
\newblock \bibinfo{title}{Image captioning with semantic attention}, in:
\bibinfo{booktitle}{2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, \bibinfo{publisher}{IEEE Computer Society},
\bibinfo{address}{Los Alamitos, CA, USA}. pp. \bibinfo{pages}{4651--4659}.
\newblock \DOIprefix\doi{10.1109/CVPR.2016.503}.
%Type = Article
\bibitem[{Zhao et~al.(2021)Zhao, Jin, Chen, Lu, Ng, Liu, Dou and
Heng}]{ZHAO2021102240}
\bibinfo{author}{Zhao, Z.}, \bibinfo{author}{Jin, Y.}, \bibinfo{author}{Chen,
J.}, \bibinfo{author}{Lu, B.}, \bibinfo{author}{Ng, C.F.},
\bibinfo{author}{Liu, Y.H.}, \bibinfo{author}{Dou, Q.},
\bibinfo{author}{Heng, P.A.}, \bibinfo{year}{2021}.
\newblock \bibinfo{title}{Anchor-guided online meta adaptation for fast
one-shot instrument segmentation from robotic surgical videos}.
\newblock \bibinfo{journal}{Medical Image Analysis} \bibinfo{volume}{74},
\bibinfo{pages}{102240}.
\newblock \DOIprefix\doi{https://doi.org/10.1016/j.media.2021.102240}.
%Type = Misc
\bibitem[{Zlocha et~al.(2019)Zlocha, Dou and Glocker}]{Zlocha_2019}
\bibinfo{author}{Zlocha, M.}, \bibinfo{author}{Dou, Q.},
\bibinfo{author}{Glocker, B.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{Improving retinanet for ct lesion detection with
dense masks from weak recist labels}.
\newblock \href{http://arxiv.org/abs/1906.02283}{\tt arXiv:1906.02283}.
\end{thebibliography}