Convolutional Nets Versus Vision Transformers for Diabetic Foot Ulcer Classification
Extracted source text · arXiv 2111.06894
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage[caption=false,font=footnotesize]{subfig}
\usepackage{caption}
\captionsetup[table]{skip=1pt}
\usepackage{booktabs}
\usepackage{amsmath,amssymb}
\usepackage{hyperref}
\usepackage[table]{xcolor}
\usepackage{collcell}
\usepackage{hhline}
\usepackage{pgf}
\usepackage{hyperref}
\usepackage{blindtext}
\usepackage{enumitem}
\usepackage{wrapfig}
\usepackage{soul}
\usepackage[normalem]{ulem}
\usepackage{booktabs}
\usepackage{makecell}
\usepackage{arydshln}
\usepackage{amsmath}
\usepackage[caption=false,font=footnotesize]{subfig}
\usepackage[export]{adjustbox}
\usepackage{xcolor}
\def\colorModel{hsb} %
\newcommand\ColCell[1]{
\pgfmathparse{#1<50?1:0} %
\ifnum\pgfmathresult=0\relax\color{white}\fi
\pgfmathsetmacro\compA{0} %
\pgfmathsetmacro\compB{#1/100} %
\pgfmathsetmacro\compC{1} %
\edef\x{\noexpand\centering\noexpand\cellcolor[\colorModel]{\compA,\compB,\compC}}\x #1
}
\newcolumntype{E}{>{\collectcell\ColCell}m{0.45cm}<{\endcollectcell}} %
\newcommand*\rot{\rotatebox{90}}
\usepackage[misc]{ifsym}
\def\Plus{\texttt{+}}
\definecolor{darkgreen}{rgb}{0.0,0.40,0.0}
\newcommand{\g}[1]{\textcolor{black}{#1}}
\begin{document}
\title{Convolutional Nets Versus Vision Transformers for Diabetic Foot Ulcer Classification}
\author{Adrian Galdran\inst{1,2} \and
Gustavo Carneiro\inst{2} \and
Miguel A. Gonz\'alez Ballester\inst{1,3}}
\authorrunning{A. Galdran et al.}
\institute{BCN Medtech, Dept. of Information and Communication Technologies, \\Universitat Pompeu Fabra, Barcelona, Spain, \email{adrian.galdran@upf.edu} \and
Australian Institute for Machine Learning, University of Adelaide, Australia \and ICREA, Barcelona, Spain}
\maketitle %
\begin{abstract}
This paper compares well-established Convolutional Neural Networks (CNNs) to recently introduced Vision Transformers for the task of Diabetic Foot Ulcer Classification, in the context of the DFUC 2021 Grand-Challenge, in which this work attained the first position.
Comprehensive experiments demonstrate that modern CNNs are still capable of outperforming Transformers in a low-data regime, likely owing to their ability for better exploiting spatial correlations.
In addition, we empirically demonstrate that the recent Sharpness-Aware Minimization (SAM) optimization algorithm improves considerably the generalization capability of both kinds of models.
Our results demonstrate that for this task, the combination of CNNs and the SAM optimization process results in superior performance than any other of the considered approaches.
\keywords{Diabetic Foot Ulcer Classification, Vision Transformers, \\Convolutional Neural Networks, Sharpness-Aware Optimization}
\end{abstract}
\section{Introduction and Problem Statement}
Diabetes affects 425 million people with a projection to reach 629 million affected people \g{by 2045 \cite{cho_idf_2018}}.
A particularly concerning development of diabetes are foot ulcers: \g{the risk of a diabetic patient developing a foot ulcer along their lifetime is estimated to be 19-34\% \cite{armstrong_diabetic_2017}}.
Diabetic Foot Ulcers (DFU) presented with ischaemia and/or infections can lead to limb gangrene, amputation, or even patient death. \g{Iscahemia is characterized by a deficient blood circulation, and it can be recognized by poor vascular reperfusion or dark gangrenous patterns. On the other hand, infection is associated to the presence of tissue inflammation or purulence \cite{goyal_recognition_2020}}. Although continuous monitoring and timely diagnosis is crucial to avoid these serious complications, \g{inappropriate care and lack of screening on diabetic population, as well as} the lack of medical specialists in developing countries poses a barrier towards early discovery and appropriate management of DFU.
In this setting, \g{computer-supported} diagnosis by means of computer vision techniques and conventional \g{smartphone} cameras represents a promising and economically viable alternative, with some recent research focusing already on developing this kind of technology \cite{yap_computer_2015,yap_new_2018,goyal_recognition_2020}.
\g{Therefore, previous research has addressed the need of automatic image understanding towards DFU analysis. Ranging from conventional feature engineering-based image classification \cite{veredas_binary_2010} based on colour and texture to more advanced two-stage approaches involving lesion detection or segmentation followed by classification \cite{brungel_detr_2021,amin_integrated_2020}, and more recently deep learning approaches \cite{das_recognition_nodate,rostami_multiclass_2021}. A detailed recent review on machine learning applied to DFU can be found in \cite{tulloch_machine_2020}.}
Within this context, the \g{Diabetic Foot Ulcer Challenge (DFUC) 2021} was held in conjunction with the MICCAI 2021 conference.
The task at hand is multi-class classification of foot images into four categories: no pathology, infection, ischaemia, or presence of both pathologies, as illustrated in Fig. \ref{fig_intro}.
This paper describes our approach to solving the challenge, in which we obtained the first position.
In addition, we make to contributions: 1) we compare the performance of \g{two popular Convolutional Neural Networks (CNNs) \cite{tan_efficientnet_2019,kolesnikov_big_2020} with that of two more recent Vision Transformers \cite{dosovitskiy_image_2021,touvron_training_2021}} in this problem, \g{revealing that CNNs may} still be the preferred choice for this kind of problems, and 2) we experiment with the Sharpness-Aware (SAM, \cite{foret_sharpness-aware_2021}) extension of the popular Stochastic Gradient Descent optimization process, demonstrating that SAM yields superior performance in all the analyzed cases.
More details on the DFUC 2021 organization, as well as live test leaderboard, can be found online at \url{https://dfu-2021.grand-challenge.org/} .
\g{In the next section, we first describe the two CNN and the two Transformer architectures we train for solving the task at hand. We then briefly recall the SAM optimization algorithm, and provide a description of the training process. Next, we present the data used for training and validating our analysis, and the performance obtained by each of the considered approaches, as well as the final test performance of our best model. We conclude the paper with some discussion and further research directions.}
\begin{figure*}[t]
\centering
\subfloat[]{\includegraphics[width = 0.24\textwidth]{images/none2.jpg}
\label{fig_deg_1}}
\hfil
\subfloat[]{\includegraphics[width = 0.24\textwidth]{images/infection2.jpg}
\label{fig_deg_2}}
\hfil
\subfloat[]{\includegraphics[width = 0.24\textwidth]{images/ischaemia1.jpg}
\label{fig_deg_3}}
\hfil
\subfloat[]{\includegraphics[width = 0.24\textwidth]{images/both2.jpg}
\label{fig_deg_4}}
\hfil
\caption{Randomly sampled diabetic foot images from the DFUC 2021 dataset. (a) No infection and no ischaemia, (b) presence of infection, (c) presence of ischaemia, (d) presence of both conditions. Images provided by the organization.}
\label{fig_intro}
\vspace{-3mm}
\end{figure*}
\section{Methodology}
We now describe the main methodological aspects of DFU classification in this paper: CNN and Vision Transformer architectures, model optimization process and training data.
\subsection{Convolutional Neural Networks and Vision Transformers}\label{sec_archs}
In this work we experiment with two different kind of neural networks for computer vision, namely conventional Convolutional Neural Networks (CNNs) and more recent Vision Transformers.
CNNs have become the de-facto approach to \g{most} computer vision tasks in the last decade, since their initial dominance of the 2012 ImageNet challenge \cite{krizhevsky_imagenet_2012}.
Essential to the learning ability of CNNs from image data are several distinctive features, namely their built-in inductive bias (spatial translation invariance imposed through convolutions with shared learnable filters across the image), or a representation hierarchy arising from their layer-wise architecture design, by which larger patterns are learned as sequential combinations of smaller patterns.
On the other hand, Transformers have emerged in the last few years as the most powerful architecture for Natural Language Processing applications, dominating virtually all public benchmarks\footnote{Transformers are based on the notion of attention, the detailed description of which falls beyond the scope of this paper \cite{vaswani_attention_2017}.}
Recently, the Transformer architecture has been extended to other domains and data types, like graphs, speech, or vision \cite{dong_speech-transformer_2018,yun_graph_2019,dosovitskiy_image_2021}.
After the initial exploration of Transformers for computer vision in \cite{dosovitskiy_image_2021}, where a variety of image recognition tasks were successfully approached with this architecture, the amount of research dedicated to proposing Vision Transformers as an alternative to CNNs has increased exponentially.
Generally speaking, Vision Transformers consider images as sequences of small patches akin to words or tokens, that can be supplied to standard Transformers.
However, and in contrast with CNNs, Transformers have no notion of distance within an image, since patches are processed sequentially.
Spatial relationships need to be modeled by means of positional encodings/embeddings and learned on extremely large datasets, in which case the increased flexibility of Transformers could favor them over CNNs.
In this paper, we test Transformers' ability to generalize for a relatively small dataset of lesion images like those shown on Fig. \ref{fig_intro}. Specifically, the four architectures we will analyze for our problem are enumerated below.
\begin{itemize}
\item \textbf{Big Image Transfer (BiT)} \cite{kolesnikov_big_2020}.
We use the ResNeXt50 architecture from this work, which contains 25,55M parameters, uses Group Normalization instead of BatchNorm, and implements convolutional layers with Weight Standardization.
\item \textbf{EfficientNet} \cite{tan_efficientnet_2019}, which is a family of architectures designed by neural architecture search, and carefully scaled so that they achieve state-of-the-art performance with a fraction of parameters.
In this paper we use the EfficientNet B3 architecture, which contains 12,23M weights.
\item \textbf{Vision Transformers (ViT)} \cite{dosovitskiy_image_2021}, the first architecture to replace convolutions by attention, this is a pure transformer applied on sequences of image patches.
We use the ViT-base configuration, containing 22,20M weights.
\item \textbf{Data-efficient Image Transformers (DeIT)} \cite{touvron_training_2021}, a refinement of ViT with improved pre-training strategies.
We use the DeiT-small variant, which contains 22M learnable parameters, and again a patch size of $16\times16$.
\end{itemize}
\subsection{Sharpness-Aware Optimization and Training Details}
In this paper we are also interested in comparing standard Stochastic Gradient Descent (SGD, \cite{bottou_tradeoffs_2008}) to a recently introduced approach for neural network optimization that tries to improve their generalization ability, namely \g{Sharpness-Aware} Optimization (\g{ SAM, \cite{foret_sharpness-aware_2021,korpelevich_extragradient_1976}}) .
SAM simultaneously minimizes the value of a loss function and the loss sharpness, thereby seeking to find parameters lying in neighborhoods with a uniformly low loss.
This way, the goal of SAM is to improve model generalization, and it has been shown to deliver State-of-\g{the}-Art performance for different applications, while also enforcing robustness against label noise.
Further technical details about this optimization algorithm can be found in \cite{foret_sharpness-aware_2021}.
Other than the optimization algorithm choice, all the models trained in this work follow the same process, similarly to \cite{galdran_little_2020}: network weights are optimized so as to minimize the cross-entropy loss with each of the optimizers adopting an initial learning rate of $l=0.01$ and a batch-size of $32$.
Note that these two values were selected by grid search on an initial train-validation split.
The learning rate is decayed following a cosine law from its initial value to $l=1e$-$8$ during $3$ epochs, which defines a training cycle.
This cycle is repeated $25$ times, restarting the learning back at the beginning.
During training images are augmented with standard techniques (random rotations, vertical/horizontal flipping, contrast/saturation/brightness changes).
The F1-score is monitored on an independent validation set and the best performing model is kept for testing purposes.
For testing, we generate four different versions of each image by horizontal/vertical flipping, predict on each of them, and average the results.
Note that in all cases, optimization starts from weights pre-trained on the ImageNet classification task\footnote{\g{All architectures and pretrained weights are taken from \cite{wightman_pytorch_2019}}} at a $224\times224$ resolution, which happens to coincide with the size of the image patches provided in the DFUC 2021 challenge, as explained in the next section.
\g{
\subsection{Training Data}
In DFUC 2021, participants were provided with a dataset composed of 15,683 DFU RGB image patches of a fixed 224$\times 224$ resolution, with 5,955 training and 5,734 testing unlabelled DFU patches for final ranking purposes. An additional set of 3,994 unlabeled DFU patches was released for training, although it was unused in this work.
These patches were extracted from close-ups of diabetic feet at a distance of approximately 30-40 cm perpendicularly to the plane of the ulcer. Natural data augmentation was included (repeated takes of the same lesion), and controlled for with automatic image similarity techniques while splitting the data into training and test subsets. More details on this dataset can be found in \cite{yap_analysis_2021}.
}
\section{Experimental Results and Analysis}
Below we analyze the performance of the four architectures described in section \ref{sec_archs} in the context of the \g{DFUC 2021}.
The organizers provided participants with a labeled training set and offered an unlabeled validation set \g{as well as} a public leaderboard to which preliminary submissions could be submitted.
After the validation period was over, the participants had to submit their predictions on a second hidden test set, and the final ranking was based on the latter.
In all cases, the models described below were trained on a 4-fold split of the labeled training data, with predictions generated by average ensembling.
\g{Models were implemented in PyTorch \cite{paszke_pytorch_2019} and trained on a workstation equipped with a NVIDIA GeForce RTX 3080 with 10 Gb memory size.}
\subsection{Impact of SAM optimization}
In this section we compare the performance of the aforementioned four architectures when trained as specified above, employing both standard SGD and Sharpness-Aware loss minimization with SGD as base optimizer.
Performance measures include F1-score (the metric that drives the DFU competition), as well as AUC, recall and precision on its macro versions.
For this experiment, these quantities are measured on the intermediate validation set released by the organizers. Table \ref{tab_results1} shows such results for all cases.
\begin{table*}[!h] %
\renewcommand{\arraystretch}{1.4}
\centering
\setlength\tabcolsep{8pt}
\begin{tabular}{l cccc}
\textbf{Model/Performance} & \textbf{F1-score} & \textbf{AUC} & \textbf{Recall} & \textbf{Precision} \\
\midrule
\textbf{BiT-ResNeXt50 SGD} & 51.34 & 84.79 & 54.99 & 51.05\\
\textbf{BiT-ResNeXt50 SAM} & 57.71 & 87.68 & 61.88 & 57.74 \\
\hdashline[2pt/5pt]
\textbf{Perf. Diff.} & \textbf{\Plus 6.37} & \textbf{\Plus 2.89} & \textbf{\Plus 6.89} & \textbf{\Plus 6.69}\\
\midrule
\textbf{EfficientNet B3 SGD} & 47.55 & 82.92 & 49.49 & 48.47 \\
\textbf{EfficientNet B3 SAM} & 57.65 & 84.81 & 60.74 & 57.13 \\
\hdashline[2pt/5pt]
\textbf{Perf. Diff.} & \textbf{\Plus 10.05} & \textbf{\Plus 1.89} & \textbf{\Plus11.25} & \textbf{\Plus 8.66}\\
\midrule
\textbf{ViT SGD} & 48.84 & 85.35 & 53.13 & 51.07 \\
\textbf{ViT SAM} & 51.94 & 87.85 & 56.52 & 54.38 \\
\hdashline[2pt/5pt]
\textbf{Perf. Diff.} & \textbf{\Plus 3.10} & \textbf{\Plus 2.50} & \textbf{\Plus3.39} & \textbf{\Plus 3.31} \\
\midrule
\textbf{DeiT SGD} & 53.87 & 84.92 & 58.01 & 53.01 \\
\textbf{DeiT SAM} & 53.98 & 85.97 & 58.12 & 54.03 \\
\hdashline[2pt/5pt]
\textbf{Perf. Diff.} & \textbf{\Plus 0.11} & \textbf{\Plus 1.05} & \textbf{\Plus0.11} & \textbf{\Plus 1.02} \\
\bottomrule\\[0.05cm]
\end{tabular}
\caption{Performance analysis of different combinations of architectures with SGD and SAM optimization procedures. \g{Note that all metrics are macro-averaged across categories.}}
\label{tab_results1}
\end{table*}%
\subsection{CNNs vs Transformers}
We also study the performance of each of our four architectures in the final hidden test set.
For this, Table \ref{tab_results2} shows the metrics obtained by each model, whereas Table \ref{tab_results3} describes the result of our final submission for the challenge classification, compared to the results obtained by the top-ranked contestants.
\begin{table*}[!h] %
\renewcommand{\arraystretch}{1.6}
\centering
\setlength\tabcolsep{8pt}
\begin{tabular}{l cccc}
\textbf{Model/Performance} & \textbf{F1-score} & \textbf{AUC} & \textbf{Recall} & \textbf{Precision} \\
\midrule
\textbf{BiT-ResNeXt50} & \textbf{61.53} & \textbf{88.49}& \textbf{65.59} & \textbf{60.53} \\
\textbf{EfficientNet B3} & 59.71 & 87.01 & 61.79 & 59.34 \\
\textbf{ViT} & 58.48 & 87.64 & 62.27 & 58.91 \\
\textbf{DeiT} & 57.29 & 87.98 & 61.35 & 57.01 \\
\bottomrule\\[0.05cm]
\end{tabular}
\caption{Performance analysis in the final test set for the four different architectures considered in our comparison. \g{Note that all metrics are macro-averaged across categories.}}
\label{tab_results2}
\end{table*}%
\begin{table*}[t] %
\renewcommand{\arraystretch}{1.6}
\centering
\setlength\tabcolsep{8pt}
\begin{tabular}{l cccc}
\textbf{Participant/Performance} & \textbf{F1-score} & \textbf{AUC} & \textbf{Recall} & \textbf{Precision} \\
\midrule
\textbf{agaldran (this work)} & \textbf{62.16} & \textbf{88.55}& \textbf{65.22} & 61.40\\
\textbf{Louise.Bloch} & 60.77 & 86.16 & 62.46 & \textbf{62.07} \\
\textbf{S. Ahmed} & 59.59 & 86.44 & 59.79 & 59.84 \\
\textbf{Abdul} & 56.91 & 84.88 & 61.04 & 58.14 \\
\textbf{orhunguley} & 56.10 & 87.02 & 57.59 & 59.17 \\
\bottomrule\\[0.05cm]
\end{tabular}
\caption{Performance analysis in the final test set for the five top competitors \g{sorted by ranking in terms of F1-score}. Note that our submission in this table corresponds to an ensemble of the predictions from the BiT-ResNeXt50 and EfficientNet B3 models in Table \ref{tab_results2}. \g{Note that all metrics are macro-averaged across categories.}}
\label{tab_results3}
\end{table*}%
\subsection{Discussion of the Results}
The first conclusion we can draw from the above results is that the SAM optimization generally benefits the end performance of both CNNs and Transformers, with CNNs always appearing to be superior to Transformers in all cases.
It should be noted that SAM requires two forward passes instead of one before weight update, which turns it into a relatively expensive alternative to SGD.
However, results in Table \ref{tab_results1} appear to indicate that the extra computational load is widely compensated, particularly when working with small datasets for which the cost of one training run is not \g{substantial}.
Next, we can see in Table \ref{tab_results2} that, as expected, the relative ranking of our four models in the validation set is preserved when evaluating them in the final test set.
However, there is a noticeable increase in the performance of Vision Transformers, which attain results close to those of the EfficientNet B3 architecture.
It is also worth stressing that, just as in the validation experiments on Table \ref{tab_results1}, the BiT-ResneXt50 model still achieves the highest performance, surpassing by a good margin the second best model, which is EfficientNet B3.
Finally, we see from the final ranking of the DFUC 2021 challenge\footnote{Ongoing (live) leaderboard can be accessed at \url{https://dfu-2021.grand-challenge.org/evaluation/live-testing-leaderboard/leaderboard/}.} in Table \ref{tab_results3} that our approach is validated when compared to other contestants' results.
Our final highest score was achieved by a linear combination of the predictions extracted from BiT-ResneXt50 and EfficientNet B3, but it is worth noting that BiT-ResneXt50 alone would still be the ranked the best method, and EfficientNet B3 would have been ranked second.
\section{Conclusion}
This paper describes the winning solution to the DFUC 2021 grand-challenge on Diabetic Foot Ulcer Classification, held in conjunction with MICCAI 2021, which is based on an ensemble of CNNs (BiT-Resnext50 and EfficientNet B3) trained on different data folds.
In addition, for weight optimization we employ Sharpness-Aware Minimization, which provides a noticeable improvement in performance for all considered models.
\g{Note that, in the literature, SAM has been benchmarked mostly with CNNs, which could introduce a bias of this optimization algorithm towards that option in this paper.
However, it has been recently shown that SAM has equally good impact in performance for Vision transformers \cite{chen_when_2021}.}
Finally, we also include a comparison of the performance of CNNs and Vision Transformers for this task, which reveals that despite the recent popularity of Transformers for Computer Vision tasks, CNNs may still be the preferred choice in few-data scenarios.
\section*{Acknowledgments}
Adrian Galdran was funded by a Marie Skłodowska-Curie Global Fellowship (No 892297).
Gustavo Carneiro was partially supported by Australian Research Council grants (DP180103232 and FT190100525)
\bibliographystyle{splncs04}
\bibliography{dfu_challenge.bib}
\end{document}
\section{Introduction}
Backed by the emergence of increasingly powerful convolutional neural networks, medical image classification has made remarkable advances over the last years, reaching unprecedented levels of accuracy \cite{litjens_survey_2017}.
However, due to the inherent difficulty in collecting labeled examples of rare diseases or other unusual instances in a medical context, these models are often trained and tested on large datasets containing a more balanced distribution with respect to image classes than the one found in real-life clinical scenarios, which typically has a long-tailed distribution.
Under such severe data imbalance, \sout{frequent} \gustavo{over-represented (i.e., majority)} classes tend to dominate the training process, resulting in a decrease of performance in \gustavo{under-represented (i.e., minority)} classes \cite{zhuang_care_2019,quellec_automatic_2020}.
Therefore, developing training methods that are adapted to strong data imbalance is essential for the advance of medical image classification.
Common solutions to address data imbalance involve re-sampling the data to achieve a balanced class distribution \cite{buda_systematic_2018} or the extension of loss functions to compensate for it, \textit{e.g.} cost-sensitive classification \cite{zhou_training_2006,galdran_cost-sensitive_2020}, or weighting the contribution of the different samples \cite{cui_class-balanced_2019,lin_focal_2020}.
Another approach consists of the synthetic manipulation of data and/or labels to drive the learning process towards a more suitable solution, like label smoothing \cite{galdran_non-uniform_2020} or SMOTE \cite{chawla_smote_2002}.
Our proposed technique is a direct combination of modified training data sampling strategies with synthetic data manipulation via the well-known MixUp regularization method \cite{zhang_mixup_2018}.
Therefore it is deeply connected with MixUp, with the fundamental difference that, while MixUp randomly combines training samples without taking into account their classes, we propose to carefully mix examples from minority categories with examples from the other classes in order to create a more suitable training data distribution. %
\begin{figure*}[t]
\centering
\subfloat[]{\includegraphics[width = 0.26\textwidth]{images/im1bbb.png}
\label{fig_deg_1}}
\hfil
\subfloat[]{\includegraphics[width = 0.72\textwidth]{images/im1a.png}
\label{fig_deg_2}}
\hfil
\caption{The two different data imbalance scenarios considered in this paper. Left: Highly imbalanced problem (DR grading from retinal images, $K=5$). Right: Long-tailed data (Gastro-intestinal image classification, $K=23$). %
}
\label{fig_intro}
\vspace{-4.85mm}
\end{figure*}
Some approaches have been recently explored that apply MixUp to combat data imbalance.
\sout{For example, ReMix \cite{chou_remix_2020}, where input data is mixed up differently to our proposed approach: instead of mixing up the associated labels, only the label corresponding to the minority class is kept.}
\gustavo{For example, ReMix \cite{chou_remix_2020}
mixes up by keeping the label corresponding to the minority class, instead of mixing up the labels, like we propose.}
Similarly, MixBoost \cite{kabra_mixboost_2020} attempts to combine active learning with MixUp to select which training samples to mix from each category, adding an extra complexity layer to the sampling process.
Another popular technique that is related to our approach is SMOTE \cite{chawla_smote_2002}.
However, SMOTE generates convex combinations of input samples only between nearest neighbors of the same class.
While this creates extra examples in regions of the space where there is less data, it does not provide the same regularizing benefits as MixUp.
\gustavo{In this paper, we propose Balanced-MixUp, which is a new imbalanced-robust training method that mixes up imbalanced (instance-based) and balanced (class-based) sampling of the data.}
Our experiments on two different medical image classification tasks with highly imbalanced and long-tailed data (as shown in Fig. \ref{fig_intro}), using neural networks of different complexities, demonstrate that Balanced-MixUp is highly effective in both creating a more evenly distributed training data and also regularizing overparametrized models, resulting in better performance than other conventional and recent approaches to handle data imbalance in all the considered cases.
\section{Methodology}
In this paper we intend to combine the MixUp regularization technique with a modified sampling strategy for imbalanced datasets. This section introduces each of these concepts and then our proposed combined technique.
\subsection{MixUp Regularization}
The MixUp technique was initially introduced in \cite{zhang_mixup_2018} as a simple regularization technique to decrease overfitting in deep neural networks.
If we denote $(x_i, y_i)$ a training example composed of an image $x_i$ and its associated one-hot encoded label $y_i$, with $f_\theta$ being a neural network to be trained to approximate the mapping $f(x_i)=y_i \ \forall i$, then MixUp creates synthetic examples and labels as follows:
\begin{equation}
\hat{x} = \lambda x_i + (1-\lambda) x_j, \ \ \hat{y} = \lambda y_i + (1-\lambda) y_j
\end{equation}
where $\lambda\sim\mathrm{Beta}(\alpha, \alpha)$, with $\alpha>0$.
Since $\lambda \in [0,1]$, $\hat{x}$ and $\hat{y}$ are random convex combinations of data and label inputs. Typical values of $\alpha$ vary in $[0.1, 0.4]$, which means that in practice $\hat{x}$ will likely be relatively close to either $x_i$ or $x_j$.
Despite its simplicity, it has been shown that optimizing $f_\theta$ on mixed-up data leads to better generalization and improves model calibration \cite{thulasidasan_mixup_2019}.
\subsection{Training Data Sampling Strategies}
When dealing with extremely imbalanced data, the training process is typically impacted by different per-class learning dynamics that result in underfitting of minority classes, which are rarely presented to the model and may end up being entirely ignored \cite{kang_decoupling_2020}.
In the context of image segmentation, recent works have found that it could also be the case that well-represented classes are adequately learned while minority classes may suffer from overfitting \cite{li_analyzing_2020} \gustavo{I'd probably remove this sentence because it weakens the underfitting of minority classes message. Besides this paper is not about segmentation}.
Modified sampling strategies can be utilized to mitigate this effect, like oversampling under-represented categories, although merely doing so often leads to counter-productive outcomes, \textit{e.g.} repeatedly showing to the model the same training examples may \sout{as well} lead to the ovefitting of minority classes \cite{chawla_smote_2002}.
Before detailing how to combine oversampling of \sout{infrequent} \gustavo{minority} classes with MixUp regularization, we introduce some further notation to describe sampling strategies.
Given a training set $\mathcal{D}=\{(x_i, y_i), \ i=1,...,N\}$ for a \gustavo{multi-class} problem with $K$ classes, if each class $k$ contains $n_k$ examples, we have that $\sum_{k=1}^K n_k=N$. We can then describe common data sampling strategies using the following equation:
\begin{equation}\label{sampling}
p_j = \frac{\displaystyle n_j^q}{\sum_{k=1}^K n_k^q\displaystyle},
\end{equation}
being $p_j$ the probability of sampling from class $j$ during training.
With this, selecting $q=1$ amounts to picking examples with a probability equal to the frequency of their class in the training set (\textit{instance-based sampling}), whereas choosing $q=0$ leads to a uniform probability $p_j = 1/K$ of sampling from each class, this is, \textit{class-based sampling} or oversampling of minority classes.
Another popular choice is \textit{square-root sampling}, which stems from selecting $q=1/2$.
\begin{figure*}[t]
\centering
\includegraphics[width = 0.4\textwidth, height=3cm]{images/place_holder.png}
\caption{Schematic illustration of how Balanced-MixUp fills less populated areas of the data space - according to less frequent categories - with synthetic examples.}
\label{fig_scheme}
\end{figure*}
\subsection{Balanced-MixUp}
We propose to combine the two training enhancements described above into a single technique that is suitable for learning under strong data imbalance.
We introduce an adaptation to this scenario of MixUp, referred to as Balanced-MixUp, in which two data points are not randomly sampled without considering their category, but rather sampling one following an instance-based strategy and the other following a class-based sampling.
Let us refer to a training dataset together with a sampling strategy as $(\mathcal{D}, \mathcal{S})$, with $\mathcal{S}_I$ and $\mathcal{S}_C$ referring to instance-based and class-based sampling respectively. With this notation, Balanced-MixUp can be described as follows:
\begin{equation}
\hat{x} = \lambda x_I + (1-\lambda) x_C, \ \ \hat{y} = \lambda y_I + (1-\lambda) y_C,
\end{equation}
for $(x_I, y_I) \in (\mathcal{D}, \mathcal{S}_I), (x_C, y_C) \in (\mathcal{D}, \mathcal{S}_C)$.
This induces a more balanced distribution of training examples by creating synthetic data points around regions of the space where minority classes provide less data density, as illustrated in Fig. \ref{fig_scheme}.
At the same time, \sout{the addition of noisy, non-integer labels helps regularizing the learning process.} \gustavo{adding noise to the labels helps regularizing the learning process.}
In our case, departing from the original MixUp formulation, we obtain the mixing coefficient as $\lambda\sim\mathrm{Beta}(\alpha,1)$. This results in an exponential-like distri-
\begin{wrapfigure}{r}{0.55\textwidth}
\vspace{-20pt}
\includegraphics[width=0.49\textwidth]{images/beta_dist.png}
\caption{$\mathrm{Beta}(\alpha,1)$ distribution for varying $\alpha$}\label{beta}
\end{wrapfigure}
bution that leads to convex combinations in which examples from $(\mathcal{D}, \mathcal{S}_I)$ receive more weight, preventing \sout{the appearance of} overfitting in minority classes.
In addition, it allows us to formulate our technique as depending of a single hyperparameter with an intuitive behavior: as $\alpha$ grows, examples from minority classes are mixed up with greater weights, increasing class balance at the expense of risking some overfitting.
\subsection{Training Details}
All the models in this paper are CNNs implemented in Pytorch \cite{paszke_pytorch_2019} and optimized to reduce the Cross-Entropy loss using Stochastic Gradient Descent with a batch size of $8$ and a learning rate of $0.01$.
During training, standard data augmentation operations are applied, and the learning rate is cyclically annealed towards 0.
The performance metric of interest in each task (see next section) is used to monitor model's error and the best performing one is kept \gustavo{can we say on the validation set?}.
After training, we generate predictions by averaging the result of the input image and its horizontally flipped version, which increases performance in almost all cases.
\section{Experimental Analysis}
In this section we detail our experimental setup: considered datasets with relevant statistics, baseline techniques for comparison, evaluation metrics, and a discussion on the numerical differences in performance between methods.
\subsection{Experimental Details}
We consider two different image classification tasks: Diabetic Retinopathy (DR) grading from retinal fundus images and gastrointestinal image (GI) classification.
\begin{itemize}[leftmargin=*]
\item For DR grading, we use the Eyepacs database\footnote{\url{https://www.kaggle.com/c/diabetic-retinopathy-detection}} the largest publicly available dataset for this task.
This database has images labeled as belonging to one out of five possible DR grades, ranging from DR0 (no DR) to DR4 (proliferative DR).
Due to the nature of this disease, and the very subtle signs associated to the DR1 class (just a few microaneurysms present), the class distribution is highly imbalanced, as shown in Fig. \ref{fig_intro}.
The database contains $\sim$35,000 training examples and $\sim$55,000 images assigned to a separate test set.
We use $10\%$ of the training set for validation and early-stopping purposes, and derive performance measures from the official test set.
As an external database to examine model generalization we also use the 1748 retinal images in the Messidor-2 dataset\footnote{\url{https://www.adcis.net/en/third-party/messidor2/}}, with the grades provided\footnote{\url{https://www.kaggle.com/google-brain/messidor2-dr-grades}} in \cite{krause_grader_2018}.
\item For GI image classification, we use the Hyper-Kvasir dataset\footnote{\url{https://endotect.com/}}, recently made available \cite{borgli_hyperkvasir_2020}. This database contains 10,662 images labeled with anatomical landmarks as well as pathological and normal findings, and it represents a challenging classification problem due to the larger amount of classes (23) and the varying class frequencies, with minority classes very rarely represented, see Fig. \ref{fig_intro}.
Please note that there is also an official test set \cite{hicks_endotect_2020}, but we found this to contain only 11 different classes.
Since our focus is on assessing model performance under a long-tail distribution, we prefer to discard this set, running all our experiments and computing performance for all models in a stratified 5-fold cross-validation manner.
\end{itemize}
For performance assessment in the DR case, we adopt as the main evaluation a measure based on the quadratic-weighted kappa score (quad-$\kappa$), which is the standard performance measure in grading problems, as it considers the correct ordering of predictions.
We also report Matthews Correlation Coefficient (MCC), which has been shown to be meaningful under strong class imbalance \cite{boughorbel_optimal_2017}, and the Kendall-$\tau$ coefficient.
For the GI image classification task, we adopt as the main assessment measure MCC, as it was used for performance ranking in a recent challenge associated to this database \cite{hicks_endotect_2020}.
Since we are dealing with a long-tail data distribution, we also report Balanced Accuracy (the mean of the diagonal in a normalized confusion matrix) and the macro-F1 score, which capture well performance in minority classes.
For comparison purposes, in both tasks we contrast Balanced-MixUp with other conventional sampling approaches: 1) Class-based sampling (oversampling of minority classes, as recommended in \cite{buda_systematic_2018}), 2) Instance-based sampling, and 3) sampling based on the square root of class-frequencies \cite{mahajan_exploring_2018}.
We also include training with loss functions that are meant to perform better under severe class imbalance, namely 4) the popular Focal Loss \cite{lin_focal_2017}, which modulates the contribution of each example according to the amount of loss it generates, and 5) a Class-Balanced loss recently introduced \sout{in which} \gustavo{that} weighs the loss of each class during training based on the \textit{effective} number of samples, see \cite{cui_class-balanced_2019}.
In all cases, in order to observe the behavior of the different techniques with models of varying sizes, we train two different CNN baseline architectures: a smaller MobileNet V2 \cite{sandler_mobilenetv2_2018} and more powerful ResNext50, modified as in \cite{kolesnikov_big_2020}.
For sensitivity analysis purposes, we measure the impact of varying the only hyperparameter $\alpha$ in Balanced-MixUP, with $\alpha\in\{0.1, 0.2, 0.3\}$ and report the corresponding results.
\subsection{Discussion on the Numerical Results on DR grading}
Assessment measures for the task of DR grading from retinal images are displayed in Table \ref{tab_results_dr}.
The leftmost three columns show performance of a MobileNet V2 in terms of quad-$\kappa$, MCC and Kendall-$\tau$.
In this case, it can be seen that Balanced-MixUP with $\alpha=0.1$ outperforms all other approaches in terms of the three considered figures of merit.
Note that setting $\alpha=0.2$ still outperforms all other techniques.
It is also interesting to observe the differences when we switch to a larger model, ResNext50.
In particular, we can see that: 1) the performance of almost all methods is increased, indicating that ResNext50 is a more suitable choice for this problem; and 2) Balanced-MixUP with $\alpha=0.2$ is now the dominant approach, reaching the highest scores in all considered measures. This is consistent with the intuition that Balanced-MixUP can regularize the learning process, and that increasing its hyperparameter $\alpha$ leads to a stronger regularization
Indeed, for this architecture the second best method is Balanced-MixUP with $\alpha=0.3$.
It is also worth stressing that, again, the three values of $\alpha$ translate in better results than any of the other considered alternatives.
\input{table_1}
It is important to stress that the results shown in Table \ref{tab_results_dr} do not represent the state-of-the-art in this dataset.
Recent methods are capable of reaching greater quad-$\kappa$ values by means of specialized architectures with attention mechanisms \cite{he_cabnet_2021}, or semi-supervised learning with segmentation branches \cite{zhou_collaborative_2019}.
Our results are however aligned with other recently published techniques, as shown in Table \ref{tab_results3}, which also contains test results in the Messidor-2 dataset (without any re-training), demonstrating their competitiveness and generalization ability. \gustavo{COMMENT: Awesome result!}
Moreover we consider Balanced-MixUP to be a supplementary regularizing technique of the learning process from which other methods could also benefit.
\begin{table}[!b] %
\renewcommand{\arraystretch}{1.3}
\centering
\setlength\tabcolsep{2pt}
\begin{tabular}{l c c c c c}
&
\begin{tabular}{@{}c@{}}\textbf{QWKL} \\ \small{PRL'18} \cite{de_la_torre_weighted_2018}\end{tabular}
&
\begin{tabular}{@{}c@{}} $\mathbf{DR\vert graduate}$ \\ \small{MedIA'20} \cite{araujo_drgraduate_2020} \end{tabular}
&
\begin{tabular}{@{}c@{}} \textbf{Cost-Sensitive} \\ \small{MICCAI'20} \cite{galdran_cost-sensitive_2020} \end{tabular}
&
\begin{tabular}{@{}c@{}} \textbf{Iter. Aug.} \\ \small{TMI'20} \cite{gonzalez-gonzalo_iterative_2020} \end{tabular}
&
\begin{tabular}{@{}c@{}} \textbf{Bal-Mxp} \\ Rx50, $\alpha=0.2$\end{tabular}\\
\cmidrule(lr{.0005em}){1-6}
\textbf{Eyepacs} & 74.00 & 74.00 & 78.71 & 80.00 & \textbf{80.78} \\
\textbf{Messidor-2} & ----- & 71.00 & 79.79 & ----- & \textbf{85.14} \\[0.25cm]
\end{tabular}
\caption{Performance comparison in terms of quad-$\kappa$ for methods trained on the Eyepacs dataset and tested (no retraining) on Eyepacs and Messidor-2.}
\label{tab_results3}
\vspace{-10mm}
\end{table}
\subsection{Discussion on the Numerical Results on GI image classification}
The second task we consider is Gastro-Intestinal image classification.
Here the class imbalance is greater than in the DR problem, in addition to the presence of 28 categories, which turn this into a more challenging dataset.
Table \ref{tab_results_gi} shows the median performance in a stratified 5-fold cross-validation process, when considering exactly the same techniques and architectures as in the previous section.
Although in this task the performance measures appear to be more mixed than in the DR problem, we can still see how Balanced-MixUp appears to bring improvements over the other competing approaches.
Again, the ResNext50 architecture delivers greater performance, although differences are not as high as previously, and some methods show a degraded performance when applied with this larger architecture (compared to using MobileNet).
Focusing first on the MCC score, we can appreciate again the regularizing effect of Balanced-MixUp: for both architectures, applying it leads to improved performance.
Moreover, we can again see the same trend as above: the MobileNet model performs better with less regularization ($\alpha=0.1$), and increasing $\alpha$ slowly deteriorates the MCC.
As expected, when using the ResNeXt architecture, a greater $\alpha=0.2$ delivers better performance.
It is worth noting that the MCC reached by the MobileNet model with $\alpha=0.1$ is greater than in any of the considered methods and architectures, excluding models regularized with Balanced-MixUp and varying $\alpha$.
Finally, it is relevant to observe that when we consider balanced accuracy and macro-F1, which can capture better performance in minority classes, for almost all methods it appears to be beneficial to use a small architecture, which is probably less able of overfitting when there are few examples available \gustavo{COMMENT: I found the over/under-fitting of minority classes message seems a bit inconsistent in the paper. In general, imbalanced problems under-fit minority classes, unless you apply some re-sampling that repeats the same minority sample many times, as you mentioned. However, this last comment may be a bit contradictory, because it mentions that more regularisation (from smaller model) avoids overfitting of the minority class. I'm not quite sure how to fix that, but it needs the message needs to be more consistent.}.
Interestingly, for these measures the best performing method is the MobileNet architecture when regularized with Balanced MixUp and a relatively higher $\alpha=0.3$.
\input{table_2}
\section{Conclusion}
Imbalanced data is present in almost any medical image analysis task.
Therefore, designing techniques for learning in such regimes is a challenge of great clinical significance.
This paper introduces Balanced-MixUp, a straightforward approach to deal with heavy imbalanced data distributions by means of the combination of the popular MixUp regularization technique and modified training data sampling strategies.
Balanced-MixUp is easy to implement and delivers consistent performance improvements when compared with other popular techniques. Its extension to other tasks beyond image classification, like multi-label problems or image segmentation, represents a promising future research direction.
\begin{table*}[t] %
\renewcommand{\arraystretch}{1.3}
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l ccc ccc }
& \multicolumn{3}{c}{\textbf{MobileNet V2}} & \multicolumn{3}{c}{\textbf{ResNeXt-50}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
& \textbf{quad-$\kappa$} & \textbf{MCC} & \textbf{Kendall-$\tau$} & \textbf{quad-$\kappa$} & \textbf{MCC} & \textbf{Kendall-$\tau$} \\
\midrule
Class-sampling & 74.58 & 49.34 & 63.30 & 74.45 & 51.12 & 64.14 \\
Instance-sampling & 78.75 & 61.79 & 73.39 & 80.17 & 62.78 & 74.32 \\
Sqrt-sampling \cite{mahajan_exploring_2018} & 79.32 & 59.66 & 72.22 & 79.38 & 58.77 & 72.22 \\
Focal Loss \cite{lin_focal_2017} & 78.59 & 60.74 & 72.70 & 79.73 & 62.66 & 73.37 \\
CB Loss \cite{cui_class-balanced_2019} & 77.84 & 61.08 & 72.85 & 79.19 & 61.83 & 74.32 \\
\bottomrule
\textbf{Bal-Mxp: $\alpha=0.1$} & \textbf{79.61} & \textbf{62.41} & \textbf{74.05} & 80.35 & 63.31 & 74.74 \\
\textbf{Bal-Mxp: $\alpha=0.2$} & 79.43 & 62.22 & 73.69 & \textbf{80.78} & \textbf{63.78} & \textbf{75.05} \\
\textbf{Bal-Mxp: $\alpha=0.3$} & 78.34 & 61.65 & 73.25 & 80.62 & 63.51 & 74.89 \\
\bottomrule\\[-1.5mm]
\end{tabular}
\caption{Performance comparison with two different CNN architectures for DR grading on the Eyepacs test set. Best results are marked bold.}
\label{tab_results_dr}
\vspace{-10mm}%
\end{table*}%
\begin{table*}[t] %
\renewcommand{\arraystretch}{1.3}
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l ccc ccc }
& \multicolumn{3}{c}{\textbf{MobileNet V2}} & \multicolumn{3}{c}{\textbf{ResNeXt-50}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
& \textbf{MCC} & \textbf{B-ACC} & \textbf{Macro-F1} & \textbf{MCC} & \textbf{B-ACC} & \textbf{Macro-F1} \\
\midrule
Class-sampling & 89.74 & 61.84 & 61.84 & 88.97 & 59.57 & 58.40 \\
Instance-sampling & 90.49 & 62.13 & 62.16 & 90.74 & 61.41 & 61.69 \\
Sqrt-sampling \cite{mahajan_exploring_2018} & 90.11 & 63.01 & 62.66 & 90.29 & 62.84 & 62.85 \\
Focal Loss \cite{lin_focal_2017} & 90.30 & 62.09 & 61.78 & 90.23 & 62.12 & 62.36 \\
CB Loss \cite{cui_class-balanced_2019} & 85.84 & 54.93 & 54.87 & 89.67 & \textbf{63.84} & 63.71 \\
\bottomrule
\textbf{Bal-Mxp: $\alpha=0.1$} & \textbf{90.90} & 63.49 & 62.77 & 91.05 & 62.55 & 62.92 \\
\textbf{Bal-Mxp: $\alpha=0.2$} & 90.54 & 63.44 & 63.85 & \textbf{91.15} & 62.80 & \textbf{64.00} \\
\textbf{Bal-Mxp: $\alpha=0.3$} & 90.39 & \textbf{64.76} & \textbf{64.07} & 90.84 & 62.34 & 62.35 \\
\bottomrule\\[-1.5mm]
\end{tabular}
\caption{Stratified 5-Fold cross-validation (median) results for GI image classification on the Hyper-Kvasir dataset. Best results are marked bold.}
\label{tab_results_gi}
\vspace{-9mm}
\end{table*}%
\begin{thebibliography}{10}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\providecommand{\doi}[1]{https://doi.org/#1}
\bibitem{amin_integrated_2020}
Amin, J., Sharif, M., Anjum, M.A., Khan, H.U., Malik, M.S.A., Kadry, S.: An
{Integrated} {Design} for {Classification} and {Localization} of {Diabetic}
{Foot} {Ulcer} {Based} on {CNN} and {YOLOv2}-{DFU} {Models}. IEEE Access
\textbf{8}, 228586--228597 (2020). \doi{10.1109/ACCESS.2020.3045732},
conference Name: IEEE Access
\bibitem{armstrong_diabetic_2017}
Armstrong, D.G., Boulton, A.J.M., Bus, S.A.: Diabetic {Foot} {Ulcers} and
{Their} {Recurrence}. The New England Journal of Medicine \textbf{376}(24),
2367--2375 (Jun 2017). \doi{10.1056/NEJMra1615439}
\bibitem{bottou_tradeoffs_2008}
Bottou, L., Bousquet, O.: The {Tradeoffs} of {Large} {Scale} {Learning}. In:
Platt, J., Koller, D., Singer, Y., Roweis, S. (eds.) Advances in {Neural}
{Information} {Processing} {Systems}. vol.~20. Curran Associates, Inc.
(2008),
\url{https://proceedings.neurips.cc/paper/2007/file/0d3180d672e08b4c5312dcdafdf6ef36-Paper.pdf}
\bibitem{brungel_detr_2021}
Brüngel, R., Friedrich, C.M.: {DETR} and {YOLOv5}: {Exploring} {Performance}
and {Self}-{Training} for {Diabetic} {Foot} {Ulcer} {Detection}. In: 2021
{IEEE} 34th {International} {Symposium} on {Computer}-{Based} {Medical}
{Systems} ({CBMS}). pp. 148--153 (Jun 2021).
\doi{10.1109/CBMS52027.2021.00063}, iSSN: 2372-9198
\bibitem{chen_when_2021}
Chen, X., Hsieh, C.J., Gong, B.: When {Vision} {Transformers} {Outperform}
{ResNets} without {Pre}-training or {Strong} {Data} {Augmentations}.
arXiv:2106.01548 [cs] (Oct 2021), \url{http://arxiv.org/abs/2106.01548},
arXiv: 2106.01548
\bibitem{cho_idf_2018}
Cho, N.H., Shaw, J.E., Karuranga, S., Huang, Y., Fernandes, J.D.d.R., Ohlrogge,
A.W., Malanda, B.: {IDF} {Diabetes} {Atlas}: {Global} estimates of diabetes
prevalence for 2017 and projections for 2045. Diabetes Research and Clinical
Practice \textbf{138}, 271--281 (Apr 2018).
\doi{10.1016/j.diabres.2018.02.023}, publisher: Elsevier
\bibitem{das_recognition_nodate}
Das, S.K., Roy, P., Mishra, A.K.: Recognition of ischaemia and infection in
diabetic foot ulcer: {A} deep convolutional neural network based approach.
International Journal of Imaging Systems and Technology \textbf{n/a}(n/a).
\doi{10.1002/ima.22598}, \_eprint:
https://onlinelibrary.wiley.com/doi/pdf/10.1002/ima.22598
\bibitem{dong_speech-transformer_2018}
Dong, L., Xu, S., Xu, B.: Speech-{Transformer}: {A} {No}-{Recurrence}
{Sequence}-to-{Sequence} {Model} for {Speech} {Recognition}. In: 2018 {IEEE}
{International} {Conference} on {Acoustics}, {Speech} and {Signal}
{Processing} ({ICASSP}). pp. 5884--5888 (Apr 2018).
\doi{10.1109/ICASSP.2018.8462506}, iSSN: 2379-190X
\bibitem{dosovitskiy_image_2021}
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X.,
Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S.,
Uszkoreit, J., Houlsby, N.: An {Image} is {Worth} 16x16 {Words}:
{Transformers} for {Image} {Recognition} at {Scale}. In: International
{Conference} on {Learning} {Representations} (2021)
\bibitem{foret_sharpness-aware_2021}
Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware
{Minimization} for {Efficiently} {Improving} {Generalization}. In:
International {Conference} on {Learning} {Representations} (2021),
\url{https://openreview.net/forum?id=6Tm1mposlrM}
\bibitem{galdran_little_2020}
Galdran, A., Anjos, A., Dolz, J., Chakor, H., Lombaert, H., Ayed, I.B.: The
{Little} {W}-{Net} {That} {Could}: {State}-of-the-{Art} {Retinal} {Vessel}
{Segmentation} with {Minimalistic} {Models} (Sep 2020),
\url{http://arxiv.org/abs/2009.01907}
\bibitem{goyal_recognition_2020}
Goyal, M., Reeves, N.D., Rajbhandari, S., Ahmad, N., Wang, C., Yap, M.H.:
Recognition of ischaemia and infection in diabetic foot ulcers: {Dataset} and
techniques. Computers in Biology and Medicine \textbf{117}, 103616 (Feb
2020)
\bibitem{kolesnikov_big_2020}
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S.,
Houlsby, N.: Big {Transfer} ({BiT}): {General} {Visual} {Representation}
{Learning}. In: {ECCV} (2020)
\bibitem{korpelevich_extragradient_1976}
Korpelevich, G.M.: The extragradient method for finding saddle points and other
problems. Ekonomika i Matematicheskie Metody \textbf{12}, 747--756 (1976)
\bibitem{krizhevsky_imagenet_2012}
Krizhevsky, A., Sutskever, I., Hinton, G.E.: {ImageNet} {Classification} with
{Deep} {Convolutional} {Neural} {Networks}. In: Pereira, F., Burges, C.J.C.,
Bottou, L., Weinberger, K.Q. (eds.) Advances in {Neural} {Information}
{Processing} {Systems}. vol.~25. Curran Associates, Inc. (2012),
\url{https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf}
\bibitem{paszke_pytorch_2019}
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen,
T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E.,
DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.,
Bai, J., Chintala, S.: {PyTorch}: {An} {Imperative} {Style},
{High}-{Performance} {Deep} {Learning} {Library}. In: Wallach, H.,
Larochelle, H., Beygelzimer, A., Alché-Buc, F.d., Fox, E., Garnett, R.
(eds.) Advances in {Neural} {Information} {Processing} {Systems} 32, pp.
8024--8035. Curran Associates, Inc. (2019),
\url{http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf}
\bibitem{rostami_multiclass_2021}
Rostami, B., Anisuzzaman, D.M., Wang, C., Gopalakrishnan, S., Niezgoda, J., Yu,
Z.: Multiclass wound image classification using an ensemble deep {CNN}-based
classifier. Computers in Biology and Medicine \textbf{134}, 104536 (Jul
2021). \doi{10.1016/j.compbiomed.2021.104536}
\bibitem{tan_efficientnet_2019}
Tan, M., Le, Q.: {EfficientNet}: {Rethinking} {Model} {Scaling} for
{Convolutional} {Neural} {Networks}. In: International {Conference} on
{Machine} {Learning}. pp. 6105--6114. PMLR (May 2019)
\bibitem{touvron_training_2021}
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H.:
Training data-efficient image transformers \& distillation through attention.
In: Proceedings of the 38th {International} {Conference} on {Machine}
{Learning} (Jul 2021)
\bibitem{tulloch_machine_2020}
Tulloch, J., Zamani, R., Akrami, M.: Machine {Learning} in the {Prevention},
{Diagnosis} and {Management} of {Diabetic} {Foot} {Ulcers}: {A} {Systematic}
{Review}. IEEE Access \textbf{8}, 198977--199000 (2020).
\doi{10.1109/ACCESS.2020.3035327}, conference Name: IEEE Access
\bibitem{vaswani_attention_2017}
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.,
Kaiser, L., Polosukhin, I.: Attention is {All} you {Need}. In: Guyon, I.,
Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S.,
Garnett, R. (eds.) Advances in {Neural} {Information} {Processing} {Systems}.
vol.~30. Curran Associates, Inc. (2017),
\url{https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf}
\bibitem{veredas_binary_2010}
Veredas, F., Mesa, H., Morente, L.: Binary {Tissue} {Classification} on {Wound}
{Images} {With} {Neural} {Networks} and {Bayesian} {Classifiers}. IEEE
Transactions on Medical Imaging \textbf{29}(2), 410--427 (Feb 2010).
\doi{10.1109/TMI.2009.2033595}, conference Name: IEEE Transactions on Medical
Imaging
\bibitem{wightman_pytorch_2019}
Wightman, R.: {PyTorch} {Image} {Models} (2019). \doi{10.5281/zenodo.4414861},
\url{https://github.com/rwightman/pytorch-image-models}, publication Title:
GitHub repository
\bibitem{yap_analysis_2021}
Yap, M.H., Cassidy, B., Pappachan, J.M., O'Shea, C., Gillespie, D., Reeves, N.:
Analysis {Towards} {Classification} of {Infection} and {Ischaemia} of
{Diabetic} {Foot} {Ulcers} (Jun 2021), \url{http://arxiv.org/abs/2104.03068}
\bibitem{yap_new_2018}
Yap, M.H., Chatwin, K.E., Ng, C.C., Abbott, C.A., Bowling, F.L., Rajbhandari,
S., Boulton, A.J.M., Reeves, N.D.: A {New} {Mobile} {Application} for
{Standardizing} {Diabetic} {Foot} {Images}. Journal of Diabetes Science and
Technology \textbf{12}(1), 169--173 (Jan 2018).
\doi{10.1177/1932296817713761}
\bibitem{yap_computer_2015}
Yap, M.H., Ng, C.C., Chatwin, K., Abbott, C.A., Bowling, F.L., Boulton, A.J.M.,
Reeves, N.D.: Computer {Vision} {Algorithms} in the {Detection} of {Diabetic}
{Foot} {Ulceration}. Journal of Diabetes Science and Technology
\textbf{10}(2), 612--613 (Oct 2015)
\bibitem{yun_graph_2019}
Yun, S., Jeong, M., Kim, R., Kang, J., Kim, H.J.: Graph {Transformer}
{Networks}. In: Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc,
F.d., Fox, E., Garnett, R. (eds.) Advances in {Neural} {Information}
{Processing} {Systems}. vol.~32. Curran Associates, Inc. (2019),
\url{https://proceedings.neurips.cc/paper/2019/file/9d63484abb477c97640154d40595a3bb-Paper.pdf}
\end{thebibliography}