Stain Based Contrastive Co-training for Histopathological Image Analysis
Extracted source text · arXiv 2206.12505
% This is samplepaper.tex, a sample chapter demonstrating the
% LLNCS macro package for Springer Computer Science proceedings;
% Version 2.21 of 2022/01/12
%
\documentclass[runningheads]{llncs}
%
\usepackage[T1]{fontenc}
% T1 fonts will be used to generate the final print and online PDFs,
% so please use T1 fonts in your manuscript whenever possible.
% Other font encondings may result in incorrect characters.
%
\usepackage{graphicx}
\usepackage{url}
% Used for displaying a sample figure. If possible, figure files should
% be included in EPS format.
%
% If you use the hyperref package, please uncomment the following two lines
% to display URLs in blue roman font according to Springer's eBook style:
%\usepackage{color}
%\renewcommand\UrlFont{\color{blue}\rmfamily}
%
\begin{document}
%
\title{Stain Based Contrastive Co-training for Histopathological Image Analysis}
%
%\titlerunning{Abbreviated paper title}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
% anonymous
%\author{Anonymous}
\author{Bodong Zhang\inst{1,2}\orcidID{0000-0001-9815-0303} \and
Beatrice Knudsen\inst{3}\orcidID{0000-0002-7589-7591} \and
Deepika Sirohi\inst{3}\orcidID{0000-0002-0848-4172} \and
Alessandro Ferrero\inst{2}\orcidID{0000-0002-8671-1211} \and
Tolga Tasdizen\inst{1,2}\orcidID{0000-0001-6574-0366}}%1{Zhang, Bodong}, 2{Knudsen, Beatrice},3{Sirohi, Deepika}, 4{Ferrero, Alessandro},5{Tasdizen, Tolga}
%Corresponding author/Registered author: Bodong Zhang
%\author{Bodong Zhang\inst{1,2} \and
%Beatrice Knudsen\inst{3} \and
%Deepika Sirohi\inst{3} \and
%Alessandro Ferrero\inst{2} \and
%Tolga Tasdizen\inst{1,2}}
% %
%\authorrunning{Anonymous}
\authorrunning{B. Zhang et al.}
%Corresponding author/Registered author: Bodong Zhang
% % First names are abbreviated in the running head.
% % If there are more than two authors, 'et al.' is used.
%\institute{Anonymous Organization\\
\institute{Electrical and Computer Engineering, University of Utah, SLC, UT, USA\and
Scientific Computing and Imaging Institute, University of Utah, SLC, UT, USA\and
Huntsman Cancer Institute, University of Utah, SLC, UT, USA
%\email{**@******.***}\\
\email{bodong.zhang@utah.edu,beatrice.knudsen@path.utah.edu,deepika.sirohi\\
@hsc.utah.edu,alessandro.ferrero@utah.edu,tolga@sci.utah.edu}
%\url{http://www.springer.com/gp/computer-science/lncs} \and ABC Institute, Rupert-Karls-University Heidelberg, Heidelberg, Germany\\
%\email{\{abc,lncs\}@uni-heidelberg.de}
}
% \institute{Electrical and Computer Engineering, University of Utah\and
% Scientific Computing and Imaging Institute, University of Utah\and
% Huntsman Cancer Institute, University of Utah
% %\email{lncs@springer.com}\\
% %\url{http://www.springer.com/gp/computer-science/lncs} \and ABC Institute, Rupert-Karls-University Heidelberg, Heidelberg, Germany\\
% %\email{\{abc,lncs\}@uni-heidelberg.de}
% }
% %
\maketitle % typeset the header of the contribution
%
\begin{abstract}
We propose a novel semi-supervised learning approach for classification of histopathology images. We employ strong supervision with patch-level annotations combined with a novel co-training loss to create a semi-supervised learning framework. Co-training relies on multiple conditionally independent and sufficient views of the data.
%The conditional independence of the views given the class labels ensures classification errors from the multiple views are not correlated. The sufficiency requirement states that each view should be able to support an accurate classification model on its own.
We separate the hematoxylin and eosin channels in pathology images using color deconvolution to create two views of each slide that can partially fulfill these requirements. Two separate CNNs are used to embed the two views into a joint feature space. We use a contrastive loss between the views in this feature space to implement co-training.
%learning strategy to ensure that the H and E views from the same patch are mapped close together in the feature space while H and E views from different tiles are mapped further apart.
We evaluate our approach in clear cell renal cell and prostate carcinomas, and demonstrate improvement over state-of-the-art semi-supervised learning methods.
\keywords{Histopathology \and Semi-supervised learning \and Co-training}
\end{abstract}
%
%
%
\section{Introduction}
% As a leading cause of death, cancer is accounted for millions of death worldwide every year. The analysis and diagnosis of cancer are crucial for improving human's lifes and reducing health risks. The dominant way to disgnose cancer is to check cell samples slides under microscope. However, it requires professional knowledge and training as well as long time spending and efforts for pathologists. In recent years, computer aided diagnosis alleviated some burdens of pathologists, especially after the prosperity of deep learning models. In this paper, we would mainly focus on Clear cell renal cell carcinoma (ccRCC) and prostate cancer.
% Clear cell renal cell carcinoma (ccRCC) is the most common form of kidney cancer. Genomic and morphologic heterogeneity in ccRCC presents a barrier to biomarker development for prognostication and treatment decisions. The contemporary clinical grading system for ccRCC is based on the size of nucleoli and does not incorporate other histopathologic features. The majority of ccRCCs (64-85\%) are assigned to intermediate grades which poorly predict disease progression~\cite{Dagher2017}. One histologic growth pattern (HGP) named "sarcomatoid" is traditionally associated with a high risk of disease progression and poor prognosis and used for clinical grading (Grade 4). However, only $\sim 10\%$ of ccRCCs that progress to develop metastatic disease possess areas of sarcomatoid morphology, suggesting that there must be other lethal HGPs. Comprehensive and systematic analysis of HGPs in ccRCC provides evidence for the prognostic significance of HGPs using the disease free survival endpoint after nephrectomy~\cite{Cai2020}. Recently, HGPs in ccRCC were separated into low risk and high risk groups ~\cite{Sirohi2022}. However, comprehensive clinical adoption of HGPs is difficult due to the morphologic complexity, and admixture of HGPs within a tumor. We propose a machine learning based approach to HGP classification.
% Prostate cancer is a cancer that happens in prostate and is very common among men. Many prostate cancer types grow slowly and usually don't have metastasis. However, some other prostate cancer types spread very fast and produce very harmful effect. The classification of benign vs cancer in prostate gland is crucial since early treatment greatly elevates the chance of survival rate.
Convolutional neural networks (CNNs) are commonly used in histopathology. Because digital whole slide images (WSIs) in pathology are much larger than typical input sizes for CNNs, workflows typically first tile the WSI into many smaller patches. There are two main approaches for training classification models with WSIs: strong and weak supervision. Strong supervision uses labels for the individual tiles, which requires expert annotation at a high cost~\cite{Dimitriou2019}. Weak supervision applies multiple instance learning with slide level labels~\cite{lerousseau:hal-03133239,Huang2019,Chikontwe2020,Campanella2019}. Weakly supervised methods have become popular due to the ease of obtaining labels for learning directly from pathology reports~\cite{Bulten:2022vv}. However, successful model training with weak learning requires thousands of WSIs, and strong supervision is still essential when a smaller number of WSIs are available for learning.
% Furthermore, compared to Gleason grading, classification of HGPs in ccRCC is more challenging because of the larger variability and greater heterogeneity in a single slide. Each HGP has a different characteristic spatial scale, and HGPs demonstrate larger variability visually compared to Gleason patterns. Co-occurence of various groups of HGP is common and can complicate weakly supervised learning. Also, in prostate glands, the different sizes and patterns of glands introduced vast variations. The number of glands that gathered together is quite different among cases. Therefore, we adopt a CNN model with strong, tile-level supervision for HGP classification.
Expert annotation at the tile level is infeasible to obtain beyond a small number of WSIs. Semi-supervised learning (SSL) seeks to leverage unlabeled data to improve the accuracy of models when only a limited amount of labeled data is available. One of the recent trends in SSL, consistency regularization~\cite{NIPS2016_30ef30b6,DBLP:conf/iclr/LaineA17}, has also found application in the classification of histopathology images. Teacher-student consistency~\cite{NIPS2017_68053af2} has been used to supplement tile-level labels for quantifying prognostic features in colorectal cancer~\cite{Shaw2020TeacherStudentCF} and in combination with weak supervision for Gleason grade classification in prostate cancer~\cite{Otalora2020}. The MixMatch model~\cite{NEURIPS2019_1cd138d0}
%, that guesses low-entropy labels for data-augmented unlabeled examples and mixes labeled and unlabeled data using MixUp,
has been tested on histology datasets with open-set noise~\cite{9287980}. Weak/strong data transformation consistency (FixMatch)~\cite{NEURIPS2020_06964dce} has been applied to detection of dysplasia of the esophagus~\cite{Pulido2020}. State-of-the-art SSL methods rely on enforcing prediction/representation consistency between various transformations of the data. Whereas consistency under model perturbations has been proposed~\cite{NIPS2016_30ef30b6}, it is a less explored area. On the other hand, the co-training~\cite{Blum1998} approach to SSL can provide excellent results when multiple views of each sample are available that meet the criteria of sufficiency (each view should be able to support accurate classification on its own) and conditional independence given the label of a sample.
Hematoxylin (H) and Eosin (E) are chemical stains that are used to highlight features of tissue architecture in formalin-fixed and paraffin-embedded tissue sections.
%that are displayed on a glass slide.
H and E provide complementary information for pathologists. H is a basic chemical compound that binds negatively charged nucleotides in DNA and RNA to provide a blue color. In contrast, E is acidic and reacts with basic side chains of amino acids resulting in pink coloration.
%When applied together, H and E stain tissues with shades of pink and purple.
Whereas proteins bind to DNA and RNA lead to overlapping H and E staining in the cell nucleus and cytoplasm, the extracellular matrix and vascular structures supporting cancer cells interact primarily with E since they are devoid of DNA and RNA. In contrast to RGB channels that cannot easily be linked to a biological interpretation, H and E allow separation of nuclei versus cytoplasm and extracellular matrix.
%The interpretation of coloration in H and E stained tissues is embedded in the learning of a pathologist and connected to the functional state of the tissue.
Therefore, we hypothesize that H and E stains, when separated into their own channels, can provide two views that can, to a large extent, satisfy the co-training assumptions. We also formulate a novel contrastive co-training with H and E views. We validate our approach on a dataset of 53 WSIs from clear cell renal cell carcinoma (ccRCC) patients for histologic growth pattern (HGP) classification and of 45 WSIs from prostate cancer patients for cancer vs. benign gland classification. We demonstrate that our approach outperforms state-of-the-art SSL methods. We perform further experiments to explain the suitability of H and E channels for co-training as opposed to RGB channels.
\section{Stain Based Contrastive Co-training}
\subsection{Stain Separation}
%Based on the higher concentration of DNA / RNA compared to protein in the nucleus, Hematoxylin (H) primarily stains cell nuclei with a purple color. On the other hand, Eosin (E) primarily stains the protein-rich cytoplasm and extracellular regions with a pink color.
We adopt an approach that separates an H\&E image in the RGB space into individual H and E stain channels using non-linear pixel-wise functions derived from dominant color profiles of each stain~\cite{Ruifrok2001QuantificationOH}. Concretely, we use the following approximate transformation between the two spaces:
\begin{equation}
\left[ \begin{array}{c}
H\\ E
\end{array} \right] =
\left[ \begin{array}{ccc}
1.838 & 0.034 & -0.760\\ -1.373 & 0.772 & 1.215
\end{array} \right]
\left[ \begin{array}{c}
\log _{10} 255/R\\
\log _{10} 255/G\\
\log _{10} 255/B\\
\end{array} \right].
\end{equation}
The H and E channels are normalized to the range $[0,1]$ after the transformation.
%Figure~\ref{fig:model} shows an example H\&E color (RGB) image separated into its H and E channels.
% \begin{figure}
% \begin{center}
% \begin{tabular}{ccc}
% \includegraphics[width=0.25\textwidth]{RGB.png}&
% \includegraphics[width=0.25\textwidth]{H.png}&
% \includegraphics[width=0.25\textwidth]{E.png}
% \end{tabular}
% \end{center}
% \caption{From left to right: H\&E color (RGB) image, hematoxylin, and eosin channels.}
% \label{fig:stain}
% \end{figure}
\begin{figure}
\centering
\includegraphics[width=0.67\textwidth]{PaperModel.png}
\caption{H\&E RGB image is separated into H and E channels and processed separately to generate two feature sets $f_H$ and $f_E$ trained with the proposed contrastive loss.}
\label{fig:model}
\end{figure}
\subsection{Contrastive Co-training}
% Since, the H channel is primarily associated with nuclear regions in contrast to the E channel which is primarily associated with cytoplasm and extracellular regions, they can be treated as providing two distinct views of the same specimen. Therefore,
We propose two ResNet models~\cite{He2016DeepRL} (same architecture, separate parameters) for H and E channels, respectively (Figure~\ref{fig:model}). Existing co-training methods enforce consistency of prediction between classifier outputs operating on different views of the data. The disadvantage of this approach is that the individual classifiers only make use of their respective views and are sub-optimal. Instead, we propose a contrastive loss in the feature space to implement co-training and define a single classifier which uses a combined view by averaging the features from the two channels (Figure~\ref{fig:model}). Our approach is inspired by recent works that use contrastive learning to create a shared feature space between multimodal data~\cite{Yuan2021MultimodalCT,Yuhao2020}. We use a contrastive loss to create a shared feature space between features extracted by the H and E networks. Let $f_{H}(x)$ and $f_{E}(x)$ denote the H and E features for input tile $x$, respectively. We use a triplet loss
%to implement the contrastive co-training loss
\begin{equation}
{\cal L}_{c.t.}(x_i)=\max\left(
\parallel f_H(x_i) - f_E(x_i)\parallel_2 - \parallel f_H(x_i) - f_E(x_k)\parallel_2 + m
,0\right),
\label{eqn:triplet}
\end{equation}
where random $k\neq i$, $\parallel a\parallel_2$ denotes the L2 norm of vector $a$, and $m$ is the margin hyperparameter. The triplet loss encourages $(f_H,f_E)$ pairs from the same H\&E tile $x_i$ to be mapped closer together than $(f_H,f_E)$ pairs from mismatched input tiles $x_i$ and $x_k$. Note that the output of the model is a linear+softmax layer applied to $0.5(f_H+f_E)$. Therefore, pushing the features $f_H$ and $f_E$ closer for the same tile also implicitly minimizes the difference between individual predictions, similar to co-training, if the final layer were applied to $f_H$ and $f_E$ alone.
Let $L=\{x_j,y_j\}$ denote the labeled training set where $y_i$ is the label corresponding to input tile $x_i$. Let $U=\{x_i\}$ denote the unlabeled training set.
The overall learning strategy combines supervised learning with cross-entropy on the labeled dataset with the triplet loss (\ref{eqn:triplet}) on the entire dataset:
\begin{equation}
{\cal L}=\sum_{x_j \in L} y_j\log \hat{y}_j
+ \lambda \sum_{x_i \in L\cup U} {\cal L}_{c.t.}(x_i),
\label{eqn:loss}
\end{equation}
where $\hat{y}_j$ denotes the output of the model for input $x_j$ and $\lambda$ is a hyperparameter controlling the relative contributions from the labeled and unlabeled losses.
\section{Experiments}
\subsection{Datasets}
% To better assess our model along with other state-of-the-art semi-supervised learning models, we prepared two datasets-clear cell renal cell carcinoma dataset and prostate cancer dataset and evaluate seperately.
\noindent {\bf ccRCC.}
%Comprehensive and systematic analysis of HGPs in ccRCC provides evidence for the prognostic significance of HGPs using the disease free survival endpoint after nephrectomy~\cite{Cai2020}.
H\&E slides from ccRCC patients at our institution were retrieved from the pathology archive and scanned
%on a 3DHistech Panoramic 600 slide scanner
at $40\times$ magnification. %Files were automatically converted from .mrsx to .svs formats for analysis. Digital slide images underwent quality control and deidentification.
%Digital slides were uploaded into QuIP, a slide annotation software~\cite{Saltz2017}.
HGPs in 53 WSIs were annotated by drawing polygons around them in QuIP~\cite{Saltz2017} by a GU-subspecialty trained pathologist. HGPs were divided into nested vs. diffuse (non-nested) histologic classes~\cite{Sirohi2022}. Diffuse HGPs are associated with a higher risk of cancer recurrence and metastatic progression. Each WSI contains multiple polygons. Images were downsampled by a factor of $2\times$ and a tile size of $400\times 400$ pixels was chosen to capture the visual characteristics of the HGPs after discussion with pathologists (Figure~\ref{fig:tiles_examples}). We sampled overlapping tiles by choosing a stride of $200$ pixels. We extracted 3014 tiles with nested HGPs and 2566 tiles with diffuse HGPs from the annotated polygons. We separated the WSIs into training, validation and testing sets to ensure a realistic experimental setting. This separation resulted in 2116/1990 nested/diffuse tiles for training, 386/246 nested/diffuse tiles for validation and 512/330 nested/diffuse tiles for testing. The validation set was used for choosing hyperparameters as discussed below. The test set was for final model evaluation. Tiles from same patient were in the same set. For the SSL experiments, we divided the annotated polygons from the training set into 10 groups and randomly picked one group to draw labeled tiles from in each run of our experiments. This is a more realistic and challenging scenario than randomly choosing $10\%$ of the training tiles as labeled data because tiles from the same polygon usually represent a smaller range of variations for learning.
% In addition, to prepare for semi-supervised learning, we further divided training set into 10 groups. First, for each HGP(nested/non-nested), we sorted tiles in training set by the priority of WSI index, polygon index in WSI and then tile index in a polygon. Second, we divided them into 10 chunks and each chunk has same number of tiles. So tiles from same polygon tend to be in the same group. In later experiments of semi-supervised learning, we picked one group as labeled training data and the rest as unlabeled training data. To be more specific, we always ran experiments 5 times for same settings by choosing group 1, 3, 5, 7, and 9 as labeled data to get mean and standard deviation of our results. In fully supervised learning experiments, we also ran 5 times to get results. If we only use 10\% of training data, then we picked group 1, 3, 5, 7 or 9 each time. If we need to use 100\% of training data, we used all the groups as training data and train 5 times. The reason why we sorted tiles and divided training tiles this way is to mimic the real situation that only a small portion of polygons/WSIs were annotated. This is more challenging than randomly choosing 10\% training tiles as labeled data because tiles in same polygon usually share more common features and our method has less to learn from the labeled data. Randomly choosing 10\% tiles could let 10\% labeled tiles reach all the polygons/WSIs, which is helpful in training but not a practical situation.
\begin{figure}
\centering
\begin{tabular}{cc||cc}
\includegraphics[height=0.21\textwidth]{SmallNest.png}&
\includegraphics[height=0.21\textwidth]{Rhabdoid.png} &
\includegraphics[height=0.21\textwidth]{prostate_train_benign_31.png}&
\includegraphics[height=0.21\textwidth]{prostate_train_cancer_270.png}\\
ccRCC nested & ccRCC diffuse &
Prostate benign & Prostate cancer
\end{tabular}
\caption{Examples of ccRCC and prostate gland tiles.
%Top left: example of ccRCC nested HGP. Top right: example of ccRCC HGPs without nested morphology illustrating the morphologic heterogeneity of ccRCC. These HGPs are associated with a higher risk of cancer recurrence and metastatic progression. Bottom left: benign prostate. Bottom right: prostate cancer
}
\label{fig:tiles_examples}
\end{figure}
\noindent {\bf Prostate Cancer.}
We collected 6992 benign gland images and 6992 prostate cancer images from our institution as training set using the same process as with the ccRCC dataset.
The tile size was chosen as $256\times 256$, which is sufficient to characterize gland features. The validation and test sets are from the The Cancer Genome Atlas Program (TCGA). We collected 477 benign and 472 cancer tiles from 18 cases. In each experiment, we randomly selected 8 cases for validation and 10 cases for testing. Examples of prostate gland images are shown in Figure~\ref{fig:tiles_examples}.
%For large glands, we cropped them to make sure the tile size is always $256\times 256$.
For SSL, we randomly divided training images into 20 groups and used one group as labeled data. The image sets from our institution are available through a material transfer agreement and the TCGA set is publicly available.
\subsection{Model Selection, Training and Hyperparameters}
%To make a fair comparison among models, we adopted same basic CNN structure for all experiments.
Considering the small number of training samples, we chose ImageNet pretrained ResNet18~\cite{He2016DeepRL} for all experiments. ResNet is a state-of-the-art model which has better performance with less parameters. For models that use single channel inputs, i.e., the H and E CNN pathways in Figure~\ref{fig:model}, we summed the convolutional weights of the R, G and B channels in the first layer of ResNet18. The final layer of the ResNet18 was also changed for binary classification.
%return a probability of belonging to the nested vs. diffuse (non-nested) ccRCC HGP class or benign vs. cancer in prostate experiments.
We used color jittering, random rotation, crop to $256\times 256$(ccRCC) or $224\times 224$(prostate) pixels, random horizontal/vertical flip and color normalization as data augmentation. For validation and test tiles, we performed center crop and color normalization to follow the same data format as in training. For the co-training model, H and E channels have independent color jitters but the rest of the augmentations are common, e.g., the same random rotation angle is applied to the H and E channels from the same tile. The rationale for independent color jitters is that color variations due to the amount of H or E chemical tissue stains used are common in practice, which leads to independent brightness variations in these channels. %Incorporating color jittering would make model more robust to color change.
We used the Adam optimizer with an initial learning rate of $10^{-3}$(100\% label only) or $10^{-4}$ and used a decaying learning rate.
% Bodong: clarify which learning rate is for which experiment
%We adopted a decaying learning rate to make smaller adjustment of model at larger epochs.
A batch size of $64$ was used in ccRCC dataset and 128 in prostate cancer dataset. Hyperparameters in (\ref{eqn:loss}) were chosen as $\lambda=0.2\times p$ and $m=40$, where $p$ is the percentage of training data used as labeled data. All hyperparameters were chosen to optimize accuracy over the validation set, including experiments on other state-of-the-art models. Batch normalization was applied to the features before computing the contrastive loss.
For comparison with other state-of-the-art SSL methods, we used consistency regularization~\cite{NIPS2016_30ef30b6,DBLP:conf/iclr/LaineA17}, MixMatch~\cite{NEURIPS2019_1cd138d0} and FixMatch~\cite{NEURIPS2020_06964dce}.
%We penalized the L2 norm of the difference in the output of the network, i.e. softmax probabilities, for pairs of random augmentations of the same RGB tile.
The same augmentations discussed above were used for the SSL experiments.
% Bodong: The following is redundant. You already said you used ResNet18 for all experiments
%The basic CNN models used in MixMatch and FixMatch are designed for processing $32\times 32$ images and have good performance on general datasets like CIFAR10. We changed it to ResNet18 for a fair comparison and make CNN model more suitable for our case.
We ran all experiments for 250 epochs in ccRCC experiments and 100 epochs in prostate cancer dataset and chose the epoch with the best validation accuracy.
%All experiments converged in the end.
Each experimental setting was run 5 times to calculate mean accuracy and standard deviation.
% {\bf Co-training: } For our proposed co-training model, we did two types of experiments. The first is semi-supervised learning, in which we used 10\% of training data as labeled data and the rest as unlabeled data. The second is fully supervised learning, in which we used 100\% training data as labeled data. In this model, we used image pairs(H image and E image) as input. We augmented H image and E image by color jittering separately, but then augmented them by random rotation, center crop, random horizontal/vertical flip and color normalization together, which means H image and E image have same random rotation angle and horizontal/vertical flip status. By this way, the two images tend to converge to more similar image features. In semi-supervised learning, we fine-tuned hyperparameters and chose initial learning rate as $10^{-4}$, weight of contrastive co-training loss as 0.02 and margin of contrastive co-training loss as 40. Before calculating contrastive co-training loss, batch normalization was implemented to the features. The weight of contrastive co-training loss is much smaller than 1 probably because the feature vector has dimension of 512, so the L2 norm distance has more variations than cross entropy loss. Also, only 10\% of training tiles have cross-entropy loss item while 100\% of training tiles have contrastive co-training loss item. Other hyperparameters were tested, like initial learning rate of $10^{-3}$, weight of contrastive co-training loss as 0.004, 0.01 and 0.04. With those different hyperparameters, the validation accuracy only dropped by around 1-2\%, which shows robustness to hyperparameter changes. The margin value selection was based on the fact that average gap between L2 norm distance of features from same image and different images after training is a little bit over 30. In fully supervised learning where 100\% training tiles are labeled, we also carefully tuned weight of contrastive co-training loss and learning rate because now we have 100\% training tiles that have cross-entropy loss item instead of only 10\%. We chose $10^{-3}$ for initial learning rate and 0.2 for contrastive co-training loss weight. We didn’t see much accuracy change in validation by slightly tuning contrastive co-training loss weight and only found 2\% accuracy drop by setting initial learning rate to $10^{-4}$.
% {\bf Fully supervised learning: } We also directly used RGB images as input to do fully supervised learning using ResNet18 for comparison. We directly used pre-trained ResNet18 model and tested on 10\% of training data and all training data. Each experiments were still run 5 times to get mean accuracy and standard deviation. We chose $10^{-3}$ for initial learning rate when we use all training data and $10^{-4}$ when only 10\% training data was used.
% {\bf Semi-supervised consistency regularization learning: } To better access our proposed co-training method, we did further experiments on consistency regularization which is one of state-of-the-art semi-supervised learning methods. In consistency regularization, we first pass augmented image into CNN model and get softmax score for each class, then we augment same image again and get another softmax score. The two softmax scores are different because augmentation is random and there could be random dropout in CNN network. So for all training images, we further calculate the L2 norm distance of the two softmax scores as another consistency loss, which forms semi-supervised learning since we are able to get consistency loss for unlabeled data.
% \begin{equation}
% {\cal L}_{cons}(x_i)=
% \parallel softmax(g(x_i')) - softmax(g(x_i''))\parallel_2^{2},
% \label{eqn:triplet}
% \end{equation}
% where $x_i’$ and $x_i’’$ denotes augmented images of $x_i$, $ x_i’\neq x_i’’$. $\parallel a\parallel_2^{2}$ denotes the square of L2 norm of vector $a$. g is CNN network that takes an image as input and output raw score for each class.
% The final loss function would be:
% \begin{equation}
% {\cal L}=\sum_{x_j \in L} y_j\log \hat{y}_j
% + \lambda \sum_{x_i \in L\cup U} {\cal L}_{cons}(x_i),
% \end{equation}
% where $\hat{y}_j$ denotes the output of the model for input $x_j$ and $\lambda$ is a hyperparameter controlling the relative contributions from the labeled and unlabeled losses.
% For comparison, we still used 10\% of training tiles as labeled data and the rest as unlabeled. We also fine-tuned the initial learning rate and weight of consistency loss. We chose $10^{-4}$ for initial learning rate and 3 as weight of consistency loss. We saw around 1\%-10\% accuracy drop in validation by changing weight of consistency loss to 0.3, 1 or 10 and around 3\% accuracy drop in validation by setting initial learning rate to $10^{-3}$.
We used Python 3.7.11 $+$ Pytorch 1.9.0 $+$ torchvision 0.10.0 $+$ CUDA 10.2 on virtual environment and ran on NVIDIA TITAN X and NVIDIA TITAN RTX. We also used Python 3.9.0 $+$ Pytorch 1.7.1 $+$ torchvision 0.8.2 $+$ CUDA 11.0 and ran on NVIDIA RTX A6000. With batchsize fixed to 64, co-training experiment on ccRCC occupied around 5300MB memory on GPU and needed 1.5-2.0 minutes for each epoch. The code is available at \url{https://github.com/BzhangURU/Paper_2022_Co-training}\par
\subsection{Results}
We compared proposed co-training with H and E views to a baseline ResNet18 model that uses RGB H\&E images as input, as well as other state-of-the-art SSL methods, such as MixMatch and FixMatch, considering they are already widely used in histopathology image analysis~\cite{9287980}.
%For all cases ResNet-18 models were employed.
The approaches were compared under two settings: using 100\% of the available labeled tiles in training set for supervised learning and using only a subset (10\% in ccRCC, 5\% in prostate) of the available tiles for supervised learning. The proposed model also employed the unsupervised co-training loss with 100\% of the training data (unlabeled) to set up an SSL method.
%The other SSL methods, consistency regularization, MixMatch and FixMatch, also read unlabeled data during training.
%While the validation set was used for selecting hyper-parameters for each model including early stopping as discussed above, the accuracy on test set was reported for final evaluation.
%the testing set remained unseen except for the final evaluation phase.
%The testing sets results for the baseline and proposed models are shown in rows 1 \& 2 and rows 6 \& 7 of Table~\ref{ref:tab}, respectively.
Mean accuracy and standard deviation over 5 runs reported for all methods are shown in Table~\ref{ref:tab} for both datasets.
%In addition to the inherent sources of randomness in training, for the 10\% labeled data setting, the samples used in the labeled set were also randomized for each run.
\begin{table}
\begin{center}
\caption{Mean accuracy and standard deviations of different models for the test sets in ccRCC and prostate experiments. Best performing model results for the 100\% and 10\%/5\% labeled data setting are shown in bold.}
\label{ref:tab}
\begin{tabular}{|l|c||l|c|}
\hline
ccRCC Model & Test Accuracy & Prostate Model & Test Accuracy\\
\hline
100\% label RGB ResNet & $84.8\pm 2.4\%$ & 100\% label RGB ResNet & $77.5\pm 2.5\%$ \\
100\% label H/E co-train & $\mathbf{92.0\pm 2.6\%}$ & 100\% label H/E co-train & $\mathbf{79.1\pm 2.0\%}$\\
\hline
10\% label RGB ResNet & $76.9\pm 5.9\%$ & 5\% label RGB ResNet & $73.4\pm 1.0\%$ \\
10\% label RGB consis & $86.8\pm 3.3\%$ & 5\% label RGB consis & $74.7\pm 1.3\%$\\
%\hline
10\% label RGB MixMatch & $85.9\pm 5.7\%$ & 5\% label RGB MixMatch & $73.7\pm 5.0\%$\\
%\hline
10\% label RGB FixMatch & $88.3\pm 3.8\%$ & 5\% label RGB FixMatch & $78.2\pm 3.8\%$\\
%\hline
10\% label H/E co-train & $\mathbf{92.3 \pm 2.1\%}$ & 5\% label H/E co-train & $\mathbf{78.7 \pm 1.9\%}$\\
\hline
%100\% labeled H/E & $84.7\pm 5.2\%$ \\
%10\% labeled H/E & $78.7\pm 8.0\%$ \\
%\hline
\end{tabular}
\end{center}
\end{table}
We note that the contrastive co-training strategy improved test accuracy, by a large margin in the case of ccRCC, when 100\% of the labeled data were used for supervised training (row 2 vs. 1, Table~\ref{ref:tab}), which suggests it provides a strong regularization effect against overfitting. Note that training accuracy for the fully supervised RGB ResNet and H/E co-train models were $99.97\pm 0.02\%$ and $99.78\pm 0.15\%$, respectively, in the ccRCC dataset. The same models achieve $98.34\pm 0.64\%$ and $98.32\pm 0.55\%$ training accuracy in prostate cancer. The fact that test accuracies on prostate cancer are lower than on ccRCC for all models is likely due to domain shift. In ccRCC dataset, all training, validation and test sets come from our institution. While in prostate cancer dataset, only training set comes our institution, the validation and test set come from TCGA dataset. Another possible reason is the fact that sometimes the gland size in prostate cancer is much smaller than the size of tiles, which could carry much less distinguishable features.
As expected, the proposed co-training strategy significantly outperforms the baseline approach (row 7 vs. 3, Table~\ref{ref:tab}) under the limited labeled data setting. Consistency regularization based SSL methods significantly improve the accuracy of RGB ResNet baseline when a limited amount of training data is available (rows 4-6 vs. 3, Table~\ref{ref:tab}). In line with results from computer vision, FixMatch even surpasses the baseline model trained with the entire labeled dataset. However, our proposed method outperformed all other SSL methods we compared against including FixMatch for both datasets. We note that hyperparameters for all SSL methods were independently fine-tuned to obtain the best validation accuracy. Finally, contrastive co-training was able to reach the same accuracy levels independent of the amount of labeled data that was used for supervised training (rows 2 and 7, Table~\ref{ref:tab}).
% setting both for the validation and testing sets. An additional observation is that for the 100\% labeled data setting, the large gap between testing and validation accuracy of the baseline model suggests an overfitting problem while the contrastive co-training model does not appear to suffer from overfitting. As a result, the testing accuracy for the proposed model is significantly higher than the baseline model even when using 100\% labeled data.
% We compared the proposed contrastive co-training to other state-of-the-art SSL models, including consistency regularization, MixMatch and FixMatch. Consistency regularization based semi-supervised learning used transformations of the RGB tiles for the $10\%/5\%$ labeled data setting (Table~\ref{ref:tab}, Row 3). While consistency regularization improved the accuracy of fully supervised learning with same labeled data, its testing accuracy was significantly below the proposed contrastive co-training. We also fine-tuned hyperparameters like learning rate, image augmentation methods for MixMatch and FixMatch to get best validation accuracy. The results are shown in (Table~\ref{ref:tab}, Row 4 \& 5). Our proposed model always has higher test accuracy and lower variation over MixMatch and FixMatch.
\subsection{Co-training View Analysis}
\label{sec:views}
In this section, we further study the suitability of the H and E channels for co-training in the context of the ccRCC dataset. First, we explore whether the H and E channels are sufficient on their own to provide a basis for accurate classification in a supervised setting. We train models that only use the H or only use the E channel as input. The 100\% labeled results in Table~\ref{ref:table_of_H_only_E_only} show that both channels carry sufficient information for the classification problem at hand. This is especially true for the E-channel, which is particularly informative for the nested vs diffuse classification task. However, as expected, the accuracy for both channels drops significantly when the labeled data is limited.
\begin{table}
\begin{center}
\caption{H-only and E-only models test accuracy for the ccRCC dataset.}
\setlength{\tabcolsep}{3mm}
\label{ref:table_of_H_only_E_only}
\begin{tabular}{|l|c|l|c|}
\hline
Model & Accuracy &
Model & Accuracy\\
\hline
100\% label H ResNet & $79.4\pm 3.7\%$ &
10\% label H ResNet & $73.5\pm 4.0\%$\\
\hline
100\% label E ResNet & $94.0\pm 1.4\%$&
10\% label E ResNet & $82.3\pm 7.0\%$\\
\hline
\end{tabular}
\end{center}
\end{table}
We next explore if the H and E channels are better suited for co-training than R, G and B channels due to a higher degree of independence. We trained an image-to-image regression model using the U-Net architecture~\cite{Ronneberger2015} between various channels, e.g., predicting the E-channel from the H-channel of the same tile. The final layer of the U-Net architecture was chosen to be linear, and we used the mean square error function for training. Table~\ref{ref:table_of_choosing_views} reports the coefficient of determination ($R^2$) achieved for various input/output channel combinations. We observe that the H and E channels are harder to predict from each other (lower $R^2$) compared to the R, G and B channels, hence demonstrating a higher degree of independence and suitability for co-training.
%We further verified different HGP has similar coefficient of determination($R^2$) score.
\begin{table}
\begin{center}
\caption{Coefficient of determination($R^2$) of image mapping between various channels on ccRCC validation set at epoch with the lowest MSE.
%If $R^2$ value is close to 0, then predicted output is more random and target is more independent with input. If $R^2$ value is close to 1, then predicted output is accurate, so input and target are more correlated.
}
\setlength{\tabcolsep}{3mm}
\label{ref:table_of_choosing_views}
\begin{tabular}{|l|c|l|c|}
\hline
Experiments & $R^2$ value & Experiments & $R^2$ value\\
\hline
H $\Rightarrow$ E & $0.5223$ &
E $\Rightarrow$ H & $0.4613$ \\
\hline
R $\Rightarrow$ G & $0.8464$ &
G $\Rightarrow$ R & $0.7833$\\
\hline
R $\Rightarrow$ B & $0.8207$ &
B $\Rightarrow$ R & $0.7713$ \\
\hline
G $\Rightarrow$ B & $0.8522$ &
B $\Rightarrow$ G & $0.8824$ \\
\hline
\end{tabular}
\end{center}
\end{table}
% Another requirement for co-training is, each view should be able to perform classification by itself. So we took H channel images alone and evaluate its test accuracies with 10\%/100\% labeled training data, and also repeat experiments for E channel. The results are shown in Comparing with RGB ResNet, the classification accuracy based on H channel only is slightly lower than RGB ResNet, while on E channel the performance is even better.
\subsection{Ablation Studies}
We also conducted ablation studies to separately analyze the role of the contrastive loss and the H and E channel selection in terms of classification accuracy on ccRCC.
Omitting the contrastive loss from training while using the H and E channel inputs lowered the accuracy from $92.0\pm 2.6\%$ to $84.7\pm 5.2\%$ for 100\% labeled data and from $92.3 \pm 2.1\%$ to $78.7\pm 8.0\%$ for 10\% labeled data. In the next ablation experiment, we used various pairs from the RGB channels as the basis for our co-training method, and compared with ResNet using the same pair as input, e.g., using only the R and B channels to form 2-channel images as input for ResNet. Results are reported in Table~\ref{ref:tab_ablation}. Unlike the H and E models, we observe that the results are approximately the same, which is expected considering the higher level of dependence among RGB channels shown in Section~\ref{sec:views}.
These observations suggest that the benefit of the proposed model is due to the contrastive co-training loss applied to the H and E view inputs rather than simply due to the change in the input space or the contrastive loss individually.
\begin{table}
\begin{center}
\caption{Ablation study on ccRCC. Test set accuracy of ResNet and co-training models taking only 2 channels from RGB as input with 10\% labeled data in training.}
\label{ref:tab_ablation}
\begin{tabular}{|l|c||l|c||l|c|}
\hline
Model & Accuracy &
Model & Accuracy &
Model & Accuracy \\
\hline
RB ResNet & $77.5\pm 6.6\%$ & RG ResNet & $80.2\pm 6.4\%$ &
GB ResNet & $78.4\pm 9.7\%$\\
\hline
R/B co-train & $78.2\pm 4.5\%$&
R/G co-train & $79.8\pm 5.6\%$&
G/B co-train & $76.6\pm 7.3\%$\\
\hline
\end{tabular}
\end{center}
\end{table}
\section{Conclusion}
We proposed a novel co-training approach for pathology image classification that leverages deconvolution of an H\&E image into individual H and E stains. We demonstrated the advantages of the proposed approach over fully supervised learning and other state-of-the-art SSL methods in the context of ccRCC and prostate cancer.
%The experiments also proved that our model has stable outstanding performance over various histopathology cases(ccRCC and prostate cancer).
%The proposed approach was validated for the classification on two types of cancers: ccRCC and prostate cancer.
%Successful classification holds prognostic value.
The proposed method could be used after segmentation of cancer regions from a WSI to drive prognostic markers. In future work, we will investigate finer-grained classification for further improved prognostic value.
%For instance, in ccRCC, alveolar HGPs represent an intermediate risk group between the small nests HGP and sarcomatoid and rhabdoid HGPs.
%Classification of growth patterns can also hold value for other cancer types like lung cancer and head\&neck cancers.
% Our contrastive loss is formulated in the feature space unlike MixMatch and FixMatch which formulate the SSL loss in output space. This makes it possible for our method to use tissue areas from the WSI that fall outside the label-space for the classification problem at hand, e.g. stroma areas in prostate gland classification, which greatly increases the amount of unlabeled data. We plan to investigate this possibility in future work.
Since our proposed approach uses a complementary learning strategy to consistency regularization approaches that use data transformations, a potential avenue for future research is to combine them for further improvements.
Another potential direction for further research is to investigate whether a more sophisticated separation into H and E stain channels can provide improved results with co-training. Methods based on Cycle-GAN have been used for stain-to-stain translation such as H\&E to immunohistochemistry and they could also be used for separation of H and E stains. However, this would require the acquisition of additional datasets with H only and E only stains for the discriminators.
%in a Cycle-GAN framework.
\noindent {\bf Acknowledgements.}
We are grateful for the support of the Computational Oncology Research Initiative (CORI) at the Huntsman Cancer Institute, University of Utah. We also acknowledge support of ARUP Laboratories and the Department of Pathology at University of Utah.
%
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
% \bibliographystyle{splncs04}
% \bibliography{mybibliography}
%
%\newpage
\bibliographystyle{splncs04}
\bibliography{pathology,ML}
\end{document}
\begin{thebibliography}{10}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\providecommand{\doi}[1]{https://doi.org/#1}
\bibitem{Sirohi2022}
et~al, D.S.: Histologic growth patterns in clear cell renal cell carcinoma
stratify patients into survival risk groups. Clin Genitourin Cancer
\textbf{22}, 17--9 (2022). \doi{10.1016/j.clgc.2022.01.005}, epub ahead of
print. PMID: 35125301.
\bibitem{Saltz2017}
et~al., J.S.: A containerized software system for generation, management, and
exploration of features from whole slide tissue images. In: Cancer Res.
vol.~77, pp. e79--e82 (2017)
\bibitem{NEURIPS2019_1cd138d0}
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel,
C.A.: Mixmatch: A holistic approach to semi-supervised learning. In: Advances
in Neural Information Processing Systems. vol.~32. Curran Associates, Inc.
(2019),
\url{https://proceedings.neurips.cc/paper/2019/file/1cd138d0499a68f4bb72bee04bbec2d7-Paper.pdf}
\bibitem{Blum1998}
Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training.
In: COLT: Proceedings of the Workshop on Computational Learning Theory. pp.
92--100 (1998)
\bibitem{Bulten:2022vv}
Bulten, W.e.a., the PANDA~challenge consortium: Artificial intelligence for
diagnosis and gleason grading of prostate cancer: the panda challenge. Nature
Medicine \textbf{28}(1), 154--163 (2022). \doi{10.1038/s41591-021-01620-2},
\url{https://doi.org/10.1038/s41591-021-01620-2}
\bibitem{Campanella2019}
Campanella, G., Hanna, M.G., Geneslaw, L., Miraflor, A., Werneck Krauss~Silva,
V., Busam, K.J., Brogi, E., Reuter, V.E., Klimstra, D.S., Fuchs, T.J.:
Clinical-grade computational pathology using weakly supervised deep learning
on whole slide images. Nature Medicine \textbf{25}(8), 1301--1309 (2019).
\doi{10.1038/s41591-019-0508-1},
\url{https://doi.org/10.1038/s41591-019-0508-1}
\bibitem{Dimitriou2019}
Dimitriou, N., Arandjelovi{\'c}, O., Caie, P.D.: Deep learning for whole slide
image analysis: An overview. Frontiers in Medicine \textbf{6} (2019).
\doi{10.3389/fmed.2019.00264},
\url{https://www.frontiersin.org/article/10.3389/fmed.2019.00264}
\bibitem{He2016DeepRL}
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image
recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) pp. 770--778 (2016)
\bibitem{DBLP:conf/iclr/LaineA17}
Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: 5th
International Conference on Learning Representations, {ICLR} 2017, Toulon,
France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net
(2017), \url{https://openreview.net/forum?id=BJ6oOfqge}
\bibitem{lerousseau:hal-03133239}
Lerousseau, M., Vakalopoulou, M., Classe, M., Adam, J., Battistella, E.,
Carr{\'e}, A., Estienne, T., Henry, T., Deutsch, E., Paragios, N.: {Weakly
Supervised Multiple Instance Learning Histopathological Tumor Segmentation}.
In: {MICCAI 2020 - Medical Image Computing and Computer Assisted
Intervention}. pp. 470--479. Lima, Peru (Oct 2020).
\doi{10.1007/978-3-030-59722-1\_45},
\url{https://hal.archives-ouvertes.fr/hal-03133239}
\bibitem{Otalora2020}
Ot\'{a}lora, S., Marini, N., M\"{u}ller, H., Atzori, M.: Semi-weakly supervised
learning for prostate cancer image classification with teacher-student deep
convolutional networks. In: Interpretable and Annotation-Efficient Learning
for Medical Image Computing: Third International Workshop, IMIMIC 2020,
Second International Workshop, MIL3ID 2020, and 5th International Workshop,
LABELS 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4--8,
2020, Proceedings. pp. 193--203. Springer-Verlag, Berlin, Heidelberg (2020)
\bibitem{Chikontwe2020}
P., C., M., K., S.J., N., H., G., S.H., P.: Multiple instance learning with
center embeddings for histopathology classification. In: Medical Image
Computing and Computer Assisted Intervention (2020)
\bibitem{9287980}
Pulido, J.V., Guleria, S., Ehsan, L., Fasullo, M., Lippman, R., Mutha, P.,
Shah, T., Syed, S., Brown, D.E.: Semi-supervised classification of noisy,
gigapixel histology images. In: 2020 IEEE 20th International Conference on
Bioinformatics and Bioengineering (BIBE). pp. 563--568 (2020).
\doi{10.1109/BIBE50027.2020.00097}
\bibitem{Pulido2020}
Pulido, J.V., Guleria, S., Ehsan, L., Fasullo, M., Lippman, R., Mutha, P.,
Shah, T., Syed, S., Brown, D.E.: Semi-supervised classification of noisy,
gigapixel histology images. In: 2020 IEEE 20th International Conference on
Bioinformatics and Bioengineering (BIBE). pp. 563--568 (2020).
\doi{10.1109/BIBE50027.2020.00097}
\bibitem{Ronneberger2015}
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for
biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M.,
Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted
Intervention -- MICCAI 2015. pp. 234--241. Springer International Publishing,
Cham (2015)
\bibitem{Ruifrok2001QuantificationOH}
Ruifrok, A.C., Johnston, D.A.: Quantification of histochemical staining by
color deconvolution. Analytical and quantitative cytology and histology
\textbf{23 4}, 291--9 (2001)
\bibitem{NIPS2016_30ef30b6}
Sajjadi, M., Javanmardi, M., Tasdizen, T.: Regularization with stochastic
transformations and perturbations for deep semi-supervised learning. In: Lee,
D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in
Neural Information Processing Systems. vol.~29. Curran Associates, Inc.
(2016),
\url{https://proceedings.neurips.cc/paper/2016/file/30ef30b64204a3088a26bc2e6ecf7602-Paper.pdf}
\bibitem{Shaw2020TeacherStudentCF}
Shaw, S., Pajak, M., Lisowska, A., Tsaftaris, S.A., O'Neil, A.Q.:
Teacher-student chain for efficient semi-supervised histology image
classification. ArXiv \textbf{abs/2003.08797} (2020)
\bibitem{NEURIPS2020_06964dce}
Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C.A.,
Cubuk, E.D., Kurakin, A., Li, C.L.: Fixmatch: Simplifying semi-supervised
learning with consistency and confidence. In: Larochelle, H., Ranzato, M.,
Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information
Processing Systems. vol.~33, pp. 596--608. Curran Associates, Inc. (2020)
\bibitem{NIPS2017_68053af2}
Tarvainen, A., Valpola, H.: Mean teachers are better role models:
Weight-averaged consistency targets improve semi-supervised deep learning
results. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R.,
Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information
Processing Systems. vol.~30. Curran Associates, Inc. (2017),
\url{https://proceedings.neurips.cc/paper/2017/file/68053af2923e00204c3ca7c6a3150cf7-Paper.pdf}
\bibitem{Huang2019}
Y, H., A.C.S., C.: Evidence localization for pathology images using weakly
supervised learning. In: Medical Image Computing and Computer Assisted
Intervention (2019)
\bibitem{Yuan2021MultimodalCT}
Yuan, X., Lin, Z.L., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A.,
Faieta, B.: Multimodal contrastive training for visual representation
learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR) pp. 6991--7000 (2021)
\bibitem{Yuhao2020}
Zhang, Y., Jiang, H., Miura, Y., Manning, C., Langlotz, C.: Contrastive
learning of medical visual representations from paired images and text (2020)
\end{thebibliography}