DDI-CoCo: A Dataset For Understanding The Effect Of Color Contrast In Machine-Assisted Skin Disease Detection
Extracted source text ยท arXiv 2401.13280
% Template for ICASSP-2024 paper; to be used with:
% spconf.sty - ICASSP/ICIP LaTeX style file, and
% IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}
% Mystuff
\usepackage{dblfloatfix}
% \usepackage{graphicx}
% \usepackage{amsmath}
% \usepackage{subfig}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{adjustbox}
\usepackage{subcaption}
\usepackage[ruled]{algorithm2e}
\usepackage[noend]{algpseudocode}
% \usepackage{wrapfig}
\usepackage{breqn}
% math symbols
\DeclareMathOperator*{\argmin}{\arg\!\min}
\DeclareMathOperator*{\argmax}{\arg\!\max}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\E}{\mathbb{E}}
% Algorithm
\SetKwInput{KwInput}{Input} % Set the Input
\SetKwInput{KwOutput}{Output} % set the Output
\newcommand{\PY}[1]{{\color{blue}PY: #1}}
\newcommand{\PYB}[1]{{\color{blue}[PY: #1]}}
\newcommand{\SideNote}[2]{\todo[color=#1,size=\footnotesize]{#2}}
\newcommand{\maxside}[1]{\SideNote{red!40}{#1 --Max}}
% comments
\newcount\Comments % 0 suppresses notes to selves in text, 1 to show comments
\Comments=1
\usepackage{color}
\newcommand{\kibitz}[2]{\ifnum\Comments=1\textcolor{#1}{#2}\fi}
\newcommand{\ming}[1]{\kibitz{red} {[Ming:~#1]}}
\newcommand{\maxma}[1]{\kibitz{blue} {[Max:~#1]}}
% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}
% Title.
% ------
\title{DDI-COCO: A DATASET FOR UNDERSTANDING THE EFFECT OF COLOR CONTRAST IN MACHINE-ASSISTED SKIN DISEASE DETECTION}
%
% Single address.
% ---------------
% \name{Ming-Chang Chiu, Yingfei Wang, Jen-Ju Kuo, Pin-Yu Chen}
% \address{}
%
% For example:
% ------------
%\address{School\\
% Department\\
% Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
\name{Ming-Chang Chiu$^{\star}$~~Yingfei Wang$^{\star}$ \qquad Yen-Ju Kuo$^{\square}$ \qquad Pin-Yu Chen$^{\dagger}$\vspace{-3mm}}
\address{$^{\star}$ University of Southern California $^{\square}$ National Taiwan University Hospital
$^{\dagger}$ IBM Research \\
$^{\star}$Equal Contributions}
% \twoauthors
% {Ming-Chang Chiu*, Yingfei Wang*\footnote{Equal Contributions}}
% {University of Southern California\\
% Los Angeles, CA}
% {Jen-Ju Kuo}
% {Chung-Shan Medical School\\
% Medicine\\
% Address C-D}
% {Pin-Yu Chen}
% {IBM Research\\
% Yorktown Heights, NY\\
% Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
% \vspace{-5mm}
\begin{abstract}
Skin tone as a demographic bias and inconsistent human labeling poses challenges in dermatology AI. We take another angle to investigate color contrast's impact, beyond skin tones, on malignancy detection in skin disease datasets: We hypothesize that in addition to skin tones, the color difference between the lesion area and skin also plays a role in malignancy detection performance of dermatology AI models. To study this, we first propose a robust labeling method to quantify color contrast scores of each image and validate our method by showing small labeling variations. More importantly, applying our method to \textit{the only} diverse-skin tone and pathologically-confirmed skin disease dataset DDI, yields \textbf{DDI-CoCo Dataset}\footnote{https://github.com/charismaticchiu/DDI-CoCo}, and we observe a performance gap between the high and low color difference groups. This disparity remains consistent across various state-of-the-art (SoTA) image classification models, which supports our hypothesis. Furthermore, we study the interaction between skin tone and color difference effects and suggest that color difference can be an additional reason behind model performance bias between skin tones. %Lastly, to bridge the gap of lacking comprehensive evaluations with cutting-edge models, we assess the performance of SoTA convolutional neural networks (CNN) and vision transformer (ViT) models on our DDI-CoCo \& ISIC 2019.
Our work provides a complementary angle to dermatology AI for improving skin disease detection.
\end{abstract}
%
\begin{keywords}
Skin cancer, Skin lesion classification, Deep neural networks, Melanoma
\end{keywords}
\vspace{-2mm}
\section{Introduction}
\vspace{-2mm}
Recent advances in computer vision have made significant progress in aiding automated diagnostics and decision-making in dermatology AI \cite{tschandl2020human,esteva2017dermatologist}. This field aims to reduce barriers to dermatology care, which could benefit billions of people \cite{coustasse2019use}. However, recent studies have revealed biased performances among different skin tones and lesion types \cite{groh2021evaluating,Daneshjou2022DisparitiesID,daneshjou2021lack,Kinyanjui2020Fairness}, calling for more development in this research area. In addition, it is crucial to recognize that even though individuals with darker skin tones or people of color (PoC) have a lower risk of developing skin cancer, they face a higher mortality rate from the disease, primarily because of delays in detection or seeking medical attention \cite{gupta2016skin}. Therefore, we aim to provide more insights for evaluating darkly pigmented lesions in PoC by our technique.
Our work takes a different angle and asks whether, in addition to skin tone, the color contrast between the lesion and the background could be a factor that also affects the prediction of dermatology AI. %Furthermore, there is a lack of algorithm development using more cutting-edge computer vision models, and our work aims to fill this gap.
\input{fig_label_flow}
We introduce a novel \textit{dermatologist-reviewed and variability-controlled labeling technique} for the quantitative assessment of contrast scores in clinical images. Our aim is to establish a bias-mitigating labeling procedure that offers a cost-effective approach by involving advanced tests only when necessary. This approach is driven by the inadequacies in achieving a consensus among dermatologists, as highlighted in recent studies \cite{Daneshjou2022DisparitiesID}. We will validate the robustness of our labeling process to human factor in Sec.~\ref{sec:methods}.
Next, we apply our dermatologist-reviewed labeling method to the recently released Diverse Dermatology Images (DDI) dataset \cite{Daneshjou2022DisparitiesID}, yielding our proposed \textbf{DDI-Co}lor \textbf{Co}ntrast (\textbf{DDI-CoCo}) dataset that has contrast scoring on each image. We choose the DDI dataset as our test bed since it serves as \textit{the only} dermatologist-annotated and pathologically proven clinical image dataset with diverse skin tones and diverse diseases, and is shown to reduce skin tone bias.
We systematically assess color contrast's impact on malignancy detection in skin diseases using three deep neural network (DNN) models on DDI-CoCo dataset. Our approach serves a twofold purpose: first, we evaluate the performance of image classification models trained on the commonly-used ISIC 2019 dataset, and second, we investigate their performance after fine-tuning on our DDI-CoCo dataset. %Furthermore, we aim to explore the cutting-edge performance limits and select the SoTA image classification models, including the EfficientNetV2-S (EfficientNet) \cite{tan2021efficientnetv2} and Swin Transformer V2-B (SwinV2) \cite{liu2021swin} in addition to the widely used InceptionNet V3 (DeepDerm) \cite{esteva2017dermatologist,Daneshjou2022DisparitiesID}. Finally, we briefly explore the possibilities of using common data augmentations. Fig.~\ref{fig:label_flow} summarizes our experimental workflow.
We emphasize the development of a dermatologist-supervised labeling framework and its importance in enabling the study of color contrast effects in skin disease prediction. We also highlight the robustness of our labeling technique, which helps to address biases and offers valuable insights for improved understanding and diagnosis of skin diseases.
% Our contributions can be summarized in three folds,
% \begin{itemize}
% \item We curate our DDI-CoCo dataset by proposing a novel labeling framework supervised by dermatologists, which extends the existing Diverse Dermatology Images (DDI) dataset by incorporating contrast score annotations for skin disease images. This dataset enables the study of color contrast effects in skin disease prediction, offering complementary insights to the skin disease image analysis community.
% \item We propose and validate our robust and dermatologist-reviewed labeling technique which can serve as a cheaper pre-test alternative in dermatology AI.
% \item We thoroughly evaluate cutting-edge architectures and demonstrate the complementariness of our approach by showing the efficacy of DDI-CoCo in reducing performance gaps between color contrast groups.
%\item We conduct rigorous evaluations in out-of-distribution and fine-tuning setups on both ISIC 2019 and DDI-CoCo to call for adopting cutting-edge architectures and feature-targeting data augmentation in the skin disease detection domain.
% \end{itemize}
\vspace{-5mm}
\section{Related Work}\label{sec:related}
\vspace{-2mm}
%% melanoma detection
Research has shown superior results from DNNs \cite{esteva2017dermatologist, Cassidy2021AnalysisOT} than experts in clinical skin disease datasets, and the International Skin Imaging Collaboration (ISIC) has also organized valuable annotated datasets for this domain.
Prior works in dermatology AI have tested with setups like using InceptionNetV3 (DeepDerm\footnote{One of our goals is to fairly compare SoTA architectures with InceptionNet V3, so we avoid using pre-computed DeepDerm which trains on proprietary datasets.})\cite{Daneshjou2022DisparitiesID,esteva2017dermatologist} to train on \textit{proprietary} large-scale datasets, implementing small and computationally efficient models such as MobileNet \cite{chaturvedi2021skin} or modifying VGG-16 \cite{groh2021evaluating} for skin-lesion classification, or studying fair classification using automatic skin tone scoring using DenseNet \cite{Kinyanjui2020Fairness}. % However, the computer vision community has evolved with more advanced models such as ViT \cite{dosovitskiy2020vit} and others. Therefore, we think it is vital to consider those advanced architectures in a modern study.
Many prior works have relied on visual consensus labels from a small group of dermatologists \cite{esteva2017dermatologist,daneshjou2021lack,liu2020deep}. However, this process can be noisy as they may lack information useful for diagnoses such as clinical history, in-person visit review, and other diagnostic tests. \cite{hekler2020effects} showed that a melanoma classifier trained on dermatologists' consensus dataset performed significantly worse than models trained on pathologically confirmed ground truth dataset. This motivates us to design a labeling method that not only shows clinical value but also has a low human factor effect.
On the other hand, color contrast has been an aspect used by Ophthalmologist to test the vision capability of human beings \cite{biochem_texbook,neurosci_book,Chiu2022OnHV} and \cite{lester2021clinical} indicates biases in photography such as color balancing could contribute to the difficulty of capturing critical features. Classical computer-aided diagnosis methods often rely on image pre-processing or curating color-based features to assist classification or segmentation \cite{chen2003colour, cheng2008skin,madooei2016incorporating}. Recent work on deep learning based skin lesion detection also begin to incorporate color and illumination factors such as color bands, shading attenuation, and greyscale transformation \cite{abhishek2020illumination}, demonstrating the applicability of color theory to modern deep learning based frameworks. However, incorporating these factors as high dimensional features limits model interpretability, and fails to allow for direct interrogation of intrinsic characteristics presented in lesion-skin color contrast, and also, \cite{Chiu2022OnHV, Chiu_2023_ICCV} suggests DNNs has built-in color and contrast bias. Thus, we aim to study how lesion-skin color contrast plays a role in dermatology AI and specifically in malignancy detection. We highlight that the skin-lesion contrast in our work is distinct from the overall image contrast as measured by illumination or color theory, as adjusting the overall image contrast will not necessarily lead to a mitigation of skin-lesion contrast bias. Note that we use color contrast and color difference interchangeably in this work.
\input{fig_justifications}
\vspace{-5mm}
\section{Datasets}\label{sec:datasets}
\vspace{-2mm}
To enable fair comparisons of different models on DDI-CoCo, we follow common practice and adopt the publicly available ISIC 2019 as the base training dataset. ISIC 2019 is the most commonly used dataset for AI algorithm development in skin disease classification, and it also includes various patient metadata for potential deeper analysis. We do not possess the right to share the DDI dataset, but it is publicly available and we will share our DDI-CoCo.
\vspace{-5mm}
\subsection{ISIC 2019 Dataset}
\vspace{-2mm}
ISIC 2019 \cite{combalia2019bcn20000,Tschandl2018_HAM10000,codella2018skin} contains 25,331 images and includes the whole popular HAM10000 (ISIC2018 training set) in addition to the lesions in hard-to-diagnose locations such as nails and mucosa. % Therefore, instead of the widely used HAM10000 (ISIC2018) we use ISIC2019 for its comprehensivity. %Same as HAM10000, the dermoscopic images fall into the seven different skin diseases --- melanoma, melanocytic nevus, basal cell carcinoma, actinic keratosis, benign keratosis, dermatofibroma and vascular lesion.
% Also, they have metadata on age, gender, and location, but they do not contain annotated skin tone labels.
\vspace{-5mm}
\subsection{Diverse Dermatology Images (DDI) Dataset}
\vspace{-2mm}
The DDI dataset contains 656 images from 78 skin diseases and is \textit{the first and only} pathologically confirmed dataset with diverse Fitzpatrick skin types (FST), a clinical classification scheme for skin tone. It has relatively balanced sizes across three different skin tone groups, 208 images of FST I-II (light skin tones), 241 images of FST III-IV, and 207 images of FST V-VI (dark skin tones), and all skin tones are determined using in-person visits reviews by two board-certified dermatologists. %It is shown that DDI helps reduce the performance gap between light and dark skin tones \cite{Daneshjou2022DisparitiesID}.
Note that DDI was \textit{originally proposed to compare FST V-VI group and FST I-II group}, as a result, we will also \textit{only} consider these two groups in the analysis. %\faye{FST III-IV group result will be included in the supplementary?}. %One other dataset that has skin tone labels is Fitz17k but due to \cite{} instead of using Fitzpatrick17k, we use DDI since we do not want to .
\vspace{-5mm}
\section{Methods}\label{sec:methods}
\vspace{-2mm}
In this section, we present our dermatologist-reviewed labeling and contrast scoring process, which involves annotating the contrast score for each image. Also, we provide an overview of the experimental setups employed in our study.
\vspace{-5mm}
\subsection{Labeling Process}
\vspace{-2mm}
We apply a mathematical derivation based on the average pixel color intensities to annotate each image with a contrast score reflecting the contrast between the lesion and skin tone. This approach ensures robustness against human labeling bias and provides objective measurements for quantifying the contrast.
We justify our method in Sec.~\ref{sec:label} and detail our labeling process as follows:
\vspace{-2mm}
\begin{enumerate}
\item We manually retrieve the RGB values of three random points on the lesion area (within the circular diagnostic markings when presented). We avoid choosing points on active bleeding site of ulcerative lesions, dark-colored scar tissue regions or any consequences interfering the color of primary lesions. We name these points foreground points (red dots in Fig \ref{fig:label_flow}).
\vspace{-2mm}
\item We retrieve RGB values of three random points around the lesion area. We retrieve points on normal perilesional skin, which is the same color as the patient's skin tone, without any underlying erythema, nor fibrosis associated with possible inflammatory change. To decrease the risk of surrounding inflammation-induced color changes confounding the primary lesion color, we avoid choosing points immediately adjacent to the lesion. We name these points background points (green dots in Fig.~\ref{fig:label_flow}).
\vspace{-2mm}
\item When the lesion lies in an area with a special lighting condition, such as in the shadow, the labellers are instructed to choose foreground and background points in similar lightning conditions. % \ming{other corner cases: what about hair or nails?}
\vspace{-2mm}
\item We average the three RGB points for background and foreground, respectively, and compute the contrast score between the average foreground and background colors with the official WCAG formula \cite{w3c2008WCAG} for accessibility, emulating the idea from \cite{Chiu2022OnHV}.
\end{enumerate}
% \input{fig_violin_rgb}
\vspace{-4mm}
\subsection{Experimental Setups}
\vspace{-2mm}
We use Inception V3 for its commonality and explore the SoTA models such as the EfficientNetV2-S (more efficient) and Swin Transformer V2-B (bigger). All models are pre-trained on ImageNet. The objective of our evaluation is twofold: (1) to evaluate if SoTA models are also subject to contrast bias, similar to the skin tone bias observed in the DDI paper, and (2) to assess the efficacy of DDI-CoCo in reducing the performance gap between high and low contrast groups. For the first goal, we train on the comprehensive and publicly available ISIC 2019 dataset and evaluate on the entire DDI-CoCo dataset. This out-of-distribution (OOD) setup is practical as we do not always have in-distribution data in clinical evaluations. To achieve the second goal, instead of evaluating the entire DDI-CoCo, we fine-tune on 80\% of images and evaluate on the rest and we follow fine-tuning process in \cite{dosovitskiy2020vit}. % , and for reproducibility, unlike some previous works, we release details of our parameters in the Appendix.%Table~\ref{tab:appendix_setups}.
All experiments and data splits are performed with \textbf{five different random seeds}. The results of the experimental replicates are shown as error bars in all the model performance plots. We use weighted cross-entropy loss to account for any class imbalance, and the weights are obtained from the inverse frequency of each class.
We filter out 14 images due to their borderline pixel intensities causing abnormal contrast scores per WCAG formula, and then divide the rest of the images in DDI-CoCo into two contrast groups, ``high" and ``low" (see example images in Fig.~\ref{fig:label_flow}), by a cutoff at the median of contrast scores. Tab.~\ref{tab:contrast} summarizes the statistics of DDI-CoCo, a balanced high/low contrast presence among skin tones was achieved.
\input{fig_replicate_n_contrast}
\input{fig_contrast_n_skintone}
% \input{tab_setups}
\vspace{-5mm}
\section{Results and Analysis}
\vspace{-4mm}
In this section, we first demonstrate the robustness of our method by analyzing point selections and showcasing the high scoring consistency achieved between different labellers, then validate our twofold evaluations by comparing InceptionNet V3 (DeepDerm) performance on different skin tones. The skin tone effect result indicates that our experimental setup is able to replicate the result in the original DDI paper (Fig.~\ref{fig:replicate_n_contrast}(a)), which corroborates the validity of our experimental setups. Next, we discuss the effect of lesion-skin tone contrast on malignancy detection. Furthermore, we dive deeper into each skin tone type to discuss and explore the interaction of contrast and skin tone. Lastly, we discuss how SoTA models' behaviors. %Finally, we conduct an ablation study to encourage the use of other modern architectures.
\input{tab_contrast_n_size}
\vspace{-5mm}
\subsection{Justification of Labeling Process}\label{sec:label}
\vspace{-2mm}
We justify our labeling approach and demonstrate contrast score validity by showing high between-labeller consistency. Fig.~\ref{fig:justifications}(a) shows that the contrast score distributions are similar across two labellers trained by a dermatologist (F and M), and a paired t-test with a p-value of 0.95 further validates the identical contrast score distribution. %(Fig.~\ref{fig:justifications}(b)). We also observe a high correlation between the contrast scores derived from foreground and background points selected by each labellers (Fig.~\ref{fig:justifications}(c)).
These results show our designed labeling process is robust to human factors, and can reliably generate contrast scores, despite the randomness in the initial foreground and background point selection processes.
In addition, for background points that are chosen to capture skin tone colors, our labeling method shows a strong correlation between RGB values and skin tone, where darker skin tone (FST V-VI) has smaller RGB values (Fig.~\ref{fig:justifications}(b)), indicating the validity of background point selection.
This consistency is encouraging as it showcases we can apply our process as pretest in areas with scarce medical resources.
\vspace{-6mm}
\subsection{Color Contrast in Effect}
\vspace{-2mm}
In the initial phase of our dual evaluation, where we train models on ISIC 2019 and then assess their performance on the complete DDI-CoCo dataset, we observe a performance bias favoring images with high lesion-skintone color contrast. Fig.~\ref{fig:replicate_n_contrast}(b) shows the tested models' AUC of the high and low contrast groups. This finding underscores the vulnerability of dermatology AI models to bias associated with lesion-skin tone contrast when trained on clinical images. %, aligning with the observations made by \cite{Chiu2022OnHV}.
\vspace{-4mm}
\subsection{Fine-tuning closes the performance gap between low and high contrast groups}\vspace{-2mm}
In pursuit of our second evaluation objective - fine-tuning, as depicted in Fig.~\ref{fig:replicate_n_contrast}(c), we observe a more equitable performance distribution. This underscores two key points: (1) Color contrast, similar to skin tone, can introduce OOD issues when using dermatology AI algorithms without diverse representation in the training dataset, and (2) Fine-tuning on a diverse dataset effectively narrows the performance gap between high and low contrast groups.
% \input{fig_barplot}
% \input{fig_roc_architectures}
% \input{fig_roc_skin}
% \vspace{-2mm}
\subsection{Diving Deeper into Skin Tones}
% \vspace{-2mm}
The DDI paper shows that skin tone can be a major attribute behind dermatology AI's performance bias. However, we believe the unbalanced skin tone representation in the training set only partially explains why such a bias arises. Therefore, we propose that lesion-skin tone contrast is another complementary underlying reason, especially since the majority of lesions appears in dark color. Fig.~\ref{fig:contrast_n_skintone}(a) pinpoints the significant skin tone effect in low-contrast images, but nearly no difference can be observed in the high contrast group. Similarly, Fig.~\ref{fig:contrast_n_skintone}(b) shows that the contrast effect is more significant in the dark skin tones (FST V-VI). After fine-tuning on DDI-CoCo, both effects are mitigated (Fig.~\ref{fig:contrast_n_skintone}(c) \& (d)). %Per our observations, we suppose that a potential way to improve dermatology AI is a targeted data augmentation that can diversify clinical image color contrast. We di
\vspace{-2mm}
\subsection{Similar Behavior in SoTA Model}
% \vspace{-2mm}
A surprising discovery in our study is that SoTA models exhibit performance patterns similar to Inception V3 (Fig.~\ref{fig:replicate_n_contrast}(b) \& (c)), despite that ViT \cite{dosovitskiy2020vit} claims to possess less inductive bias. Across various model architectures, we consistently observe superior performance in the high-contrast group, echoing the findings in \cite{Chiu2022OnHV}. Initially, there is a substantial performance gap between the high and low contrast groups, which diminishes after fine-tuning. This unexpected consistency in trend challenges our expectations regarding the behavior of SoTA models, suggesting that fine-tuning plays a crucial role in closing performance gaps across different model architectures.
\vspace{-1mm}
\input{tab_performances}
% \input{tab_DA_overall_auc}
% \vspace{-2mm}
% \subsection{A Call for Adopting New Architectures}\label{sec:new_archi}
% \vspace{-2mm}
% The InceptionNet V3 (DeepDerm) architecture has been widely utilized in dermatology AI, as evidenced by previous studies \cite{esteva2017dermatologist, Daneshjou2022DisparitiesID}. However, our observations in Tab.~\ref{tab:performances_contrast} consistently demonstrates superior performance from advanced models across all contrast groups. The number of parameters for each architecture employed in this study is summarized in Tab.~\ref{tab:model_size}.
% Based on these findings, it becomes evident that there is a strong case for adopting more advanced architectures in evaluations within this domain. The increased average AUC after fine-tuning in FST V-VI (Tab.~\ref{tab:performances_skintone}), where more PoC are represented, as well as the consistently improved performances across contrast groups from advanced models, highlight the potential benefits of leveraging more sophisticated architectures in dermatology AI evaluations.
% \input{fig_auc_by_skintone}
% \subsubsection{ISIC 2019}
% As an additional experiment to reinforce the need for adopting new architectures in the field, we conduct an evaluation of the selected models solely on the ISIC 2019 dataset. The results, depicted in Fig.~\ref{fig:appendix_isic}, consistently demonstrate superior performance of both EfficientNet and SwinV2 compared to InceptionNet V3 across five random seeds. When combined with the findings from our DDI dataset in previous sections, these results further emphasize the necessity of embracing new classification architectures for effective clinical image evaluation.
% \input{fig_appendix_isic}
% \input{tab_DA_overall_auc}
% \subsection{Data Augmentation}\label{sec:DA}
% It is natural to explore, on advanced architectures, if existing data augmentations (DA) can help mitigate the issues around contrast after the discussions in previous sections. Prior works have shown that DAs work well on in-distribution large-scale datasets \cite{zhang2017mixup, yun2019cutmix, cubuk2020randaugment} or on OOD setups \cite{yang2022openood, wenzel2022assaying}. Nonetheless, \cite{Daneshjou2022DisparitiesID} has shown that Mixup \cite{zhang2017mixup} does not benefit the InceptionNet V3 on DDI dataset. We further explore two widely adopted DAs such as RandAugment (RA) \cite{cubuk2020randaugment} and Cutmix \cite{yun2019cutmix} in our setup.
% Tab.~\ref{tab:da_overall_auc} demonstrates that DAs do not necessarily lead to an increase in the overall average AUC performances, and the ineffectiveness of DA remains true for both of our evaluation objectives (only in 3 cases do we see a benefit).
% Then we ask if DA can mitigate the gap between contrast groups. Intuitively, RA possess various transformations such as \textit{AutoContrast}, which can manipulate image contrast and should help close the performance gaps between high and low contrast groups. Fig.~\ref{fig:DA_contrast} top row indeed show evidence that training on ISIC 2019 with RA helps mitigate the gaps between contrast groups in an OOD setup (with DDI-CoCo). However, using RA in fine-tuning (Fig.~\ref{fig:DA_contrast} bottom row) enhances the discrepancy. Moreover, we also do not observe obvious benefit in reducing contrast bias from Cutmix and Mixup for both objectives.
% Our explorations in existing DAs highlight the need for dedicated future works to investigate domain-specific and targeted DA method or having diverse datasets like DDI-CoCo for reducing contrast bias in skin disease datasets.
% \input{fig_DA_contrast}
\vspace{-3mm}
\section{Conclusion}
\vspace{-3mm}
Our systematic investigation focused on the influence of color contrast on skin disease malignancy detection. We introduced the DDI-CoCo dataset, incorporating a dermatologist-reviewed contrast labeling method to mitigate human labeling variations. This labeling method holds promise for application in resource-constrained settings. Our results highlight the significant impact of contrast on skin disease detection, with the high-contrast group consistently outperforming the low-contrast group. Additionally, we found that fine-tuning models on diverse datasets effectively mitigated both contrast and skin-tone performance biases, underscoring the complimentariness of our approach.
%As additional contributions, we empirically showed evidences to call for adopting more cutting-edge architectures in dermatology AI research and encourage future works to develop domain-specific data augmentations for skin disease datasets as existing methods do not show consistent benefits.
%In today's multicultural society, physicians encounter patients from diverse backgrounds due to globalization. The potential of dermatology AI to benefit billions of people worldwide is unprecedented. By utilizing these models, we can enhance our clinical assessments for individuals with varying skin tones, instilling greater confidence. Additionally, they can serve as valuable training tools for young residents to improve their ability to detect subtle color differences with the naked eye.
%However, the scarcity of high-quality, large-scale, and diverse clinical datasets can hinder the fairness of AI models. Based on our study, we propose that reducing the impact of contrast should be a crucial area of focus in future research endeavors to address performance discrepancies. By minimizing the effect of contrast, we can work towards improving the fairness and accuracy of AI models in dermatology.
% \vfill\pagebreak
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
{\small
\bibliography{strings,refs}
}
\end{document}
\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{figures/DDI_label_flow.pdf}\vspace{-3mm}
\caption{\textbf{Labeling process \& experimental setups.} %We construct our DDI-CoCo dataset atop the pathologically confirmed DDI dataset \cite{Daneshjou2022DisparitiesID} by retrieving three sets of RGB values on the lesion (red dots) and the same for the background (green dots) and then apply formula in \cite{w3contrast} to obtain the contrast score for each image. On the other hand, we conduct our evaluation in two ways --- (1) from ImageNet-pretrained models, we adopt transfer learning on ISIC 2019 and then directly evaluate on our DDI-CoCo, and (2) after transfer learning on ISIC 2019, we also fine-tune on 80\% of DDI-CoCo and finally evaluate on the rest of the dataset. We use training to mean transfer learning to avoid confusion and detail the labeling method design and experimental setups in Sec.~\ref{sec:methods}.
}
\label{fig:label_flow}
\vspace{-5mm}
\end{figure}
\begin{figure}[htb!]
\begin{minipage}[b]{0.48\linewidth}
\centering
\includegraphics[width=\textwidth]{figures/labeller_violinplot.pdf}\vspace{-2mm}
\centerline{(a)}
\end{minipage}
\begin{minipage}[b]{0.49\linewidth}
\centering
\includegraphics[width=\textwidth]{figures/RGB_violinplot.pdf}
\centerline{(b)}
\end{minipage}\vspace{-3mm}
% \begin{minipage}[b]{0.24\textwidth}
% \centering
% \includegraphics[width=\textwidth]{figures/KS_test.png}
% \centerline{(b)}
% \end{minipage}
% \begin{minipage}[b]{0.24\textwidth}
% \centering
% \includegraphics[width=\textwidth]{figures/correlation.png}
% \centerline{(c)}
% \end{minipage}
\caption{\textbf{(a) Distributions of contrast scores by labeled. (b) Background RGB by FST skin tones. Contrast scores show high consistency between the two labellers, and background point RGB values correlate with the dermatologists-confirmed FST skin tone groups.}% (a)(b)(c) violin plots, CDFs with the K-S test, and linear regression with the Pearson's correlation test showing high consistency between the two sets of contrast scores, which are derived respectively from the foreground and background points selected by the two labellers. (d) Labelled background points of the dark skin tone group (FST V-VI) have smaller RGB values than the light skin tone group (FST I-II), showing correspondence between our human-bias-robust point selection framework and the dermatologists-confirmed FST skin tone classification.
}
\label{fig:justifications}
\vspace{-6mm}
\end{figure}
\begin{figure*}[htb!]
\begin{minipage}[b]{0.33\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/poc_replicate_DDI.pdf}\vspace{-2mm}
\centerline{(a) Replicating DDI result on InceptionV3}
\end{minipage}
% \begin{minipage}[b]{0.33\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/auc_by_training_efficient.png}
% \caption{Replicating DDI}
% \end{minipage}\hfill
% \begin{minipage}[b]{0.33\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/auc_by_training_swin.png}
% \caption{Replicating DDI}
% \end{minipage}
\begin{minipage}[b]{0.33\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/three_models_auc.pdf}\vspace{-2mm}
\centerline{(b) color contrast in effect}
\end{minipage}
\begin{minipage}[b]{0.33\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/three_models_auc_finetune.pdf}\vspace{-2mm}
\centerline{(c) fine-tuning in effect}
\end{minipage}\vspace{-2mm}
\caption{\textbf{(a) We replicate the results of DDI paper by showing performance bias in different skin tone groups, and by showing fine-tuning reduces this performance gap. (b) DNNs perform consistently better in high color contrast group, showing the bias caused by lesion-skintone color contrast. (c) fine-tuning on DDI-CoCo reduces the performance gaps.}}
\label{fig:replicate_n_contrast}
\vspace{-4mm}
\end{figure*}
\begin{figure*}[htb!]
\begin{minipage}[b]{0.24\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/inceptionv3_TRANS_contrast.pdf}\vspace{-2mm}
\centerline{(a)}
\end{minipage}
\begin{minipage}[b]{0.24\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/inceptionv3_TRANS_skintone.pdf}\vspace{-2mm}
\centerline{(b)}
\end{minipage}
\begin{minipage}[b]{0.24\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/inceptionv3_FINETUNE_contrast.pdf}\vspace{-2mm}
\centerline{(c)}
\end{minipage}
\begin{minipage}[b]{0.24\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/inceptionv3_FINETUNE_skintone.pdf}\vspace{-2mm}
\centerline{(d)}
\end{minipage}\vspace{-2mm}
\caption{\textbf{(a) AUC by contrast group before fine-tuning. (b) AUC by skin tones before fine-tuning. (c) AUC by contrast group after fine-tuning. (d) AUC by skin tone after fine-tuning} %(a) The effect of skin tone bias is different between the low \& high contrast groups. Skin tone bias is more obvious for the low contrast group (b) Contrast effect is more obvious in dark skin tones. (c)~\&~(d) Fine-tuning on diverse datasets such as DDI can mitigate both contrast and skin tone effects.
}
\label{fig:contrast_n_skintone}
\vspace{-5mm}
\end{figure*}
\begin{table}[t]
\centering
\begin{tabular}{lcccc}\toprule
DDI-CoCo & I-II & III-IV & V-VI & Total\\\midrule
High & 88 & 134 & 92 & 314 \\
Low & 114 & 103 & 111 & 328 \\\bottomrule
\end{tabular}\vspace{-3mm}
\caption{\textbf{Number of images by contrast \& skin tones.}}
\label{tab:contrast}
\vspace{-4mm}
\end{table}
% \begin{table}[h]
% % \begin{subtable}[h]{0.5\textwidth}
% \centering
% \begin{tabular}{lc}\toprule
% Architecture & \#parameters \\\midrule
% InceptionNet V3 & 27.2M \\
% EfficientNetV2-S & 21.5M \\
% SwinTransformer V2-B & 87.9M \\\bottomrule
% \end{tabular}
% \caption{\textbf{Tested architectures model sizes.} We choose cutting-edge models like the more parameter efficient CNN and a bigger ViT to compare with the widely used InceptionNetV3.}
% \label{tab:model_size}
% \end{table}
% \begin{table}[h]
% \begin{subtable}[h]{0.5\textwidth}
% \centering
% \begin{tabular}{lcccc}\toprule
% & I-II & III-IV & V-VI & Total\\\midrule
% High & 88 & 134 & 92 & 314 \\
% Low & 114 & 103 & 111 & 328 \\\bottomrule
% \end{tabular}
% \caption{\textbf{\#images by contrast \& skin tones.}}
% \label{tab:contrast}
% \end{subtable}
% \begin{subtable}[h]{0.5\textwidth}
% \centering
% \begin{tabular}{lc}\toprule
% Architecture & \#parameters \\\midrule
% InceptionNet V3 & 27.2M \\
% EfficientNet & 21.5M \\
% SwinTransformer V2 & 87.9M \\\bottomrule
% \end{tabular}
% \caption{\textbf{Tested architecture sizes.}}
% \label{tab:model_size}
% \end{subtable}
% \caption{}
% % \vspace{-8mm}
% \end{table}
\begin{table}[t]
\centering
\begin{tabular}{cccc}\toprule
AUC & Inception & Efficient & SwinV2\\
\midrule
High & 0.810 & 0.822 & 0.852 \\
Low & 0.809 & 0.819 & 0.832 \\
% Overall & & & \\
\bottomrule
\end{tabular}\vspace{-2mm}
\caption{\textbf{Average AUC by contrast after fine-tuning.} % Advanced models such as EfficientNet and SwinV2 consistently perform better regardless of contrast groups.
}
\label{tab:performances_contrast}
\vspace{-3mm}
\end{table}
\begin{table}[t]
\centering
\begin{tabular}{cccc}\toprule
AUC & InceptionV3 & EfficientNet & SwinV2\\
\midrule
I-II & 0.792 & 0.776 & 0.814 \\
V-VI & 0.828 & 0.862 & 0.895 \\
Gap & +0.036 & +0.086 & +0.081\\
\bottomrule
\end{tabular}\vspace{-2mm}
\caption{\textbf{Average AUC by skin tones after fine-tuning.} %EfficientNet and SwinV2 show higher FST V-VI AUCs where more PoC represented and show bigger AUC gains.
}
\label{tab:performances_skintone}
\vspace{-5mm}
\end{table}
% \begin{table}[h]
% \begin{subtable}[h]{0.48\textwidth}
% \centering
% \begin{tabular}{cccc}\toprule
% & Inception & Efficient & SwinT\\
% \midrule
% I-II & 0.792 & 0.776 & 0.814 \\
% V-VI & 0.828 & 0.862 & 0.895 \\
% % Overall & & & \\
% \bottomrule
% \label{tab:performances_skintone}
% \end{tabular}
% \caption{\textbf{AUC by skin tones post-finetuning.}}
% \end{subtable}
% % \smallskip
% \begin{subtable}[h]{0.48\textwidth}
% \centering
% \begin{tabular}{cccc}\toprule
% & Inception & Efficient & SwinT\\
% \midrule
% High & 0.810 & 0.822 & 0.852 \\
% Low & 0.809 & 0.819 & 0.832 \\
% % Overall & & & \\
% \bottomrule
% \label{tab:performances_contrast}
% \end{tabular}
% \caption{\textbf{AUC by contrast post-finetuning.}}
% \end{subtable}
% \caption{}
% % \vspace{-7mm}
% \end{table}
\begin{thebibliography}{10}
\bibitem{tschandl2020human}
Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel
Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep
Malvehy, et~al.,
\newblock ``Human--computer collaboration for skin cancer recognition,''
\newblock {\em Nature Medicine}, 2020.
\bibitem{esteva2017dermatologist}
Andre Esteva, Brett Kuprel, Roberto~A Novoa, Justin Ko, Susan~M Swetter,
Helen~M Blau, and Sebastian Thrun,
\newblock ``Dermatologist-level classification of skin cancer with deep neural
networks,''
\newblock {\em nature}, 2017.
\bibitem{coustasse2019use}
Alberto Coustasse, Raghav Sarkar, Bukola Abodunde, Brandon~J Metzger, and
Chelsea~M Slater,
\newblock ``Use of teledermatology to improve dermatological access in rural
areas,''
\newblock {\em Telemedicine and e-Health}, vol. 25, no. 11, pp. 1022--1032,
2019.
\bibitem{groh2021evaluating}
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim,
Arash Koochek, and Omar Badri,
\newblock ``Evaluating deep neural networks trained on clinical images in
dermatology with the fitzpatrick 17k dataset,''
\newblock in {\em CVPR}, 2021.
\bibitem{Daneshjou2022DisparitiesID}
Roxana Daneshjou, Kailas Vodrahalli, Roberto~A. Novoa, Melissa Jenkins, Weixin
Liang, Veronica~M Rotemberg, Justin~M. Ko, Susan~M. Swetter, Elizabeth~E
Bailey, Olivier Gevaert, Pritam Mukherjee, Michelle Phung, Kiana Yekrang,
Bradley Fong, Rachna Sahasrabudhe, Johan Allerup, Utako Okata-Karigane, James
Zou, and Albert~S. Chiou,
\newblock ``Disparities in dermatology ai performance on a diverse, curated
clinical image set,''
\newblock {\em Science Advances}, 2022.
\bibitem{daneshjou2021lack}
Roxana Daneshjou, Mary~P Smith, Mary~D Sun, Veronica Rotemberg, and James Zou,
\newblock ``Lack of transparency and potential bias in artificial intelligence
data sets and algorithms: a scoping review,''
\newblock {\em JAMA dermatology}, 2021.
\bibitem{Kinyanjui2020Fairness}
Newton~M Kinyanjui, Timothy Odonga, Celia Cintas, Noel~CF Codella, Rameswar
Panda, Prasanna Sattigeri, and Kush~R Varshney,
\newblock ``Fairness of classifiers across skin tones in dermatology,''
\newblock in {\em MICCAI}, 2020.
\bibitem{gupta2016skin}
Alpana~K Gupta, Mausumi Bharadwaj, and Ravi Mehrotra,
\newblock ``Skin cancer concerns in people of color: risk factors and
prevention,''
\newblock {\em APJCP}, 2016.
\bibitem{Cassidy2021AnalysisOT}
Bill Cassidy, Connah Kendrick, Andrzej Brodzicki, Joanna Jaworek-Korjakowska,
and Moi~Hoon Yap,
\newblock ``Analysis of the isic image datasets: Usage, benchmarks and
recommendations,''
\newblock {\em Medical image analysis}, vol. 75, pp. 102305, 2021.
\bibitem{chaturvedi2021skin}
Saket~S Chaturvedi, Kajol Gupta, and Prakash~S Prasad,
\newblock ``Skin lesion analyser: an efficient seven-way multi-class skin
cancer classification using mobilenet,''
\newblock in {\em Proc. of AMLTA 2020}.
\bibitem{liu2020deep}
Yuan Liu, Ayush Jain, Clara Eng, David~H Way, Kang Lee, Peggy Bui, Kimberly
Kanada, Guilherme de~Oliveira~Marinho, Jessica Gallegos, Sara Gabriele,
et~al.,
\newblock ``A deep learning system for differential diagnosis of skin
diseases,''
\newblock {\em Nature medicine}, 2020.
\bibitem{hekler2020effects}
Achim Hekler, Jakob~N Kather, Eva Krieghoff-Henning, Jochen~S Utikal,
Friedegund Meier, Frank~F Gellrich, Julius Upmeier~zu Belzen, Lars French,
Justin~G Schlager, Kamran Ghoreschi, et~al.,
\newblock ``Effects of label noise on deep learning-based skin cancer
classification,''
\newblock {\em Frontiers in medicine}, 2020.
\bibitem{biochem_texbook}
G.H. Jacobs,
\newblock {\em Encyclopedia of Biological Chemistry (Second Edition)},
\newblock Elsevier, 2013.
\bibitem{neurosci_book}
Dale Purves, George~J Augustine, David Fitzpatrick, Lawrence~C Katz,
Anthony-Samuel LaMantia, James~O McNamara, and S~Mark Williams,
\newblock {\em Neuroscience. 2nd edition.},
\newblock 2001.
\bibitem{Chiu2022OnHV}
Ming-Chang Chiu, Yingfei Wang, Derrick Eui~Gyu Kim, Pin-Yu Chen, and Xuezhe Ma,
\newblock ``On human visual contrast sensitivity and machine vision robustness:
A comparative study,''
\newblock {\em ArXiv}, vol. abs/2212.08650, 2022.
\bibitem{lester2021clinical}
JC~Lester, L~Clark~Jr, E~Linos, and R~Daneshjou,
\newblock ``Clinical photography in skin of colour: tips and best practices,''
\newblock {\em British Journal of Dermatology}, 2021.
\bibitem{chen2003colour}
Jixiang Chen, R~Joe Stanley, Randy~H Moss, and William Van~Stoecker,
\newblock ``Colour analysis of skin lesion regions for melanoma discrimination
in clinical images,''
\newblock {\em Skin Research and Technology}, 2003.
\bibitem{cheng2008skin}
Yue Cheng, Ragavendar Swamisai, Scott~E Umbaugh, Randy~H Moss, William~V
Stoecker, Saritha Teegala, and Subhashini~K Srinivasan,
\newblock ``Skin lesion classification using relative color features,''
\newblock {\em Skin Research and Technology}, 2008.
\bibitem{madooei2016incorporating}
Ali Madooei and Mark~S Drew,
\newblock ``Incorporating colour information for computer-aided diagnosis of
melanoma from dermoscopy images: A retrospective survey and critical
analysis,''
\newblock {\em ISBI}, 2016.
\bibitem{abhishek2020illumination}
Kumar Abhishek, Ghassan Hamarneh, and Mark~S Drew,
\newblock ``Illumination-based transformations improve skin lesion segmentation
in dermoscopic images,''
\newblock in {\em CVPRW}, 2020.
\bibitem{Chiu_2023_ICCV}
Ming-Chang Chiu, Pin-Yu Chen, and Xuezhe Ma,
\newblock ``Better may not be fairer: A study on subgroup discrepancy in image
classification,''
\newblock in {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV)}, October 2023, pp. 4956--4966.
\bibitem{combalia2019bcn20000}
Marc Combalia, Noel~CF Codella, Veronica Rotemberg, Brian Helba, Veronica
Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan~C Halpern,
Susana Puig, et~al.,
\newblock ``Bcn20000: Dermoscopic lesions in the wild,''
\newblock {\em arXiv preprint arXiv:1908.02288}, 2019.
\bibitem{Tschandl2018_HAM10000}
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler,
\newblock ``The {HAM10000} dataset, a large collection of multi-source
dermatoscopic images of common pigmented skin lesions,''
\newblock {\em Sci. Data}, 2018.
\bibitem{codella2018skin}
Noel~CF Codella, David Gutman, M~Emre Celebi, Brian Helba, Michael~A Marchetti,
Stephen~W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald
Kittler, et~al.,
\newblock ``Skin lesion analysis toward melanoma detection: A challenge at the
2017 isbi,''
\newblock in {\em ISBI}, 2018.
\bibitem{w3c2008WCAG}
``Web content accessibility guidelines ({WCAG}) 2.0,'' 2008.
\bibitem{dosovitskiy2020vit}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,
\newblock ``An image is worth 16x16 words: Transformers for image recognition
at scale,''
\newblock {\em ICLR}, 2021.
\end{thebibliography}