Cross-modal representation alignment of molecular structure and perturbation-induced transcriptional profiles
Extracted source text · arXiv 1911.10241
%\documentclass[wsdraft]{ws-procs11x85}
\documentclass{ws-procs11x85}
\usepackage{ws-procs-thm} % comment this line when `amsthm / theorem / ntheorem` package is used
\usepackage{booktabs}
% The package below in this class is throwing an error: https://tex.stackexchange.com/questions/96496/how-to-get-the-ws-procs9x6-document-class-to-work-with-subcaption-package
% \usepackage{caption} % fix vertical spacing of table captions
\usepackage{siunitx} % align numbers at the decimal point
\newcommand{\R}{\mathbb{R}}
\DeclareMathOperator*{\argmax}{arg\,max}
\begin{document}
\title{Cross-modal representation alignment of molecular structure and perturbation-induced transcriptional profiles}
\author{Samuel G. Finlayson$^{1,2,*}$, Matthew B.A. McDermott$^{2,*}$, Alex V. Pickering$^3$, Scott L. Lipnick$^3$, Isaac S. Kohane$^{3,\dag}$ }
% Annamarie Bair $^2$
\address{$^1$Department of Systems, Synthetic, and Quantitative Biology, Harvard Medical School, Boston, MA \\
$^2$Department of EECS, Massachusetts Institute of Technology, Cambridge, MA
\\
$^3$Department of Biomedical Informatics, Harvard Medical School, Boston, MA
\\
$^*$Co-first author
\\
$^\dag$E-mail: isaac\_kohane@harvard.edu}
\begin{abstract}
Modeling the relationship between chemical structure and molecular activity is a key goal in drug development. Many benchmark tasks have been proposed for molecular property prediction, but these tasks are generally aimed at specific, isolated biomedical properties. In this work, we propose a new cross-modal small molecule retrieval task, designed to force a model to learn to associate the structure of a small molecule with the transcriptional change it induces. We develop this task formally as multi-view alignment problem, and present a coordinated deep learning approach that jointly optimizes representations of both chemical structure and perturbational gene expression profiles. We benchmark our results against oracle models and principled baselines, and find that cell line variability markedly influences performance in this domain. Our work establishes the feasibility of this new task, elucidates the limitations of current data and systems, and may serve to catalyze future research in small molecule representation learning.
\end{abstract}
\keywords{Representation Learning, Therapeutics, Gene Expression, Deep Learning, Information Retrieval}
\copyrightinfo{\copyright\ 2020 The Authors. Open Access chapter published by World Scientific Publishing Company and distributed under the terms of the Creative Commons Attribution Non-Commercial (CC BY-NC) 4.0 License.}
\section{Introduction}
Identifying molecules that are likely to have a specific biological effect is a cornerstone of drug discovery and a key component of efforts to achieve precision medicine. Classically, computational profiling of small molecules has centered on predicting affinities for specific biological targets, using tools ranging from biophysics-driven techniques such as molecular docking\cite{hansson2002molecular} to literature-mined annotations.\cite{krallinger2017information} Small molecule modeling has also recently become a major area of interest in deep learning, a trend catalyzed by graph neural networks \cite{battaglia2018relational} and benchmarking datasets.\cite{wu2018moleculenet} Graph neural networks allow for end-to-end modeling of molecular graphs,\cite{kearnes2016molecular, yang2019analyzing, lo2018machine, ramsundar2015massively} and have yielded state-of-the-art performance on certain tasks.\cite{faber2017prediction, gilmer2017neural} In addition, deep learning approaches have been used at a more global scale modeling cross-molecule relationships.\cite{zitnik2018modeling}
% , even falling below the margin-of-error of gold-standard assays for certain biophysical properties
% As is frequently the case, the machine learning community has also been attracted to chemistry by the availability of benchmark datasets, the most prominent of which is the MoleculeNet dataset \cite{wu2018moleculenet}.
To date, deep learning efforts in this space have generally focused on two extremes: highly local, biochemical prediction problems, which test the model's ability to predict specific chemical properties, and more global, clinical modeling tasks, such as indication or side effect prediction.
% MoleculeNet and other similar datasets are predominantly focused on specific chemical properties, but do include several clinical tasks such as indication or side effect prediction.
Missing from the field, however, are benchmark tasks between these two extremes, that test the ability of deep models to encode rich, general representations of a molecule's broad-spectrum effect on cellular biology.
In parallel to these developments, connectivity mapping has emerged as a alternative approach for drug development\cite{musa2017review}. In connectivity mapping, compounds are foremost characterized not by individual chemical properties or downstream targets, but by the broad transcriptional effects they induce in cells. Connectivity mapping begins by first developing a large dataset of \emph{perturbational signatures} of molecules by physically treating cell lines with these molecules, then measuring the resultant changes in gene expression. These datasets are then compared to one or more \emph{query signatures}, which are typically differential gene expression (GE) signatures representing disease states that investigators hope to reverse. Various public datasets have been curated to enable these efforts,\cite{lamb2006connectivity,subramanian2017next} and researchers have sought to use these for drug repurposing, precision medicine, and analysis of gene expression data in general.\cite{musa2018harnessing,liu2018systematic,clark2014characteristic,donner2018drug,dincer2018deepprofile,rampavsek2019dr,cheng2013evaluation,lopez2018deep}
% Organizations such as the Broad Institute and the Library of Integrated Network-based Cellular Signatures (LINCS) Consortium have enabled the production of public datasets\cite{lamb2006connectivity, subramanian2017next} of more than a million perturbational signatures. Fueled by these data, researchers have sought to match existing drugs in an unbiased manner to new indications and even to specific patients\cite{musa2018harnessing,liu2018systematic,clark2014characteristic,donner2018drug,dincer2018deepprofile,rampavsek2019dr,cheng2013evaluation}.
Connectivity mapping is promising because it can be used to search for new indications of drugs without making any specific \emph{a priori} assumptions about their mechanism of action. However, the typical framework for connectivity mapping is limited by the fact that it can only query against drugs that have already been profiled using the transcriptional assay. In other words, connectivity mapping is -- in principle -- very flexible with respect to the disease signatures they accept as a query, but is transductive rather than inductive with respect to the target small molecule signatures.
% (In practice, connectivity mapping relies upon the assumption that drug responses \emph{in vitro} (typically in cancer cell lines) are representative of drug responses \emph{in vivo}.)
This is the perfect complement to structure-based computational chemistry, which is typically inductive to new drug structures but can only make predictions for diseases with known targets.
In this work, we combine these two fields, by using deep chemical embedders to learn the transcriptional space encoded by CMAP profiling. More specifically, we train coordinated networks to jointly embed chemical structures and perturbational gene expression profiles such that learned chemical representations are most similar to the encodings of the transcriptional patterns they induce.\footnote{Code available here: \url{https://github.com/sgfin/molecule_ge_coordinated_embeddings}}. Note that this task naturally fills the gap in inductive molecular modelling identified previously; by tasking the model to produce highly similar embeddings for chemical structures and the perturbational profiles they induce, we force the model to learn a \emph{transcriptome-wide} reflection of the drug's action on the cell. We then evaluate these chemical representations by using gene expression signatures as queries into the embedding space and recovering their corresponding compounds. (See Figure~\ref{aba:fig1}). Crucially, the evaluation is set up such that the validation and test set compounds and cell lines are not used in training, which allows us to test the ability of the model to generalize to new drugs and cell lines.
% In this work, we present this new benchmark task to assess the ability of learned chemical representations to capture broad functional properties of drugs. Second, to provide a potential proof-of-concept for a connectivity mapping strategy that can be applied to unassayed drugs.
\begin{figure*}
\begin{center}
\includegraphics[width=0.85\linewidth]{figures/figure1}
\end{center}
\caption[Aligned Chemical-Transcriptomic Representation Learning Method]{Our representation learning method. Neural networks are trained to embed gene expression profiles close to the small molecule structures that induce them. Given a cross-modal alignment, gene expression signatures can be used as queries to rank chemical structures by their likelihood to induce such a signature.}
\label{aba:fig1}
\end{figure*}
In the rest of this work, we first offer some background on joint embedding alignment, then detail the methods used in this work.
% Our experiments include the implementation of baseline models of increasing complexity, oracle models designed to estimate reasonable upper-bound performance, and introspection studies chosen to elucidate the changes in molecular representations induced by the training task.
Finally, we walk through the results and discussion of these experiments, then close with concluding thoughts. A version of this work, with supplementary material present, can be found here: \url{https://github.com/sgfin/molecule_ge_coordinated_embeddings/blob/master/paper_08_2020.pdf}.
\section{Background}
In \emph{multi-view representation alignment}, embeddings of two associated data modalities are learned separately but in a coordinated manner, such that the resulting embeddings are similar.
These methods have been used in comparing images to text but also in other domains.\cite{hsu2018unsupervised,hassani2020contrastive}
In this work, we learn aligned representations such that small molecules are embedded in close proximity to the differential gene expressions they induce. Multi-view representation alignment can be achieved through a variety of methods, including classical methods, such as \emph{canonical correlation analysis} (CCA)\cite{hotelling1992relations} and methods using distance, similarity, correlation, or ranking based penalties during training.\cite{li2018survey}
% , which produces a linear mapping between two datasets that maximizes the element-wise correlation between the embeddings of their rows.
% A wide range of additional methods for multi-modal alignment have also been proposed, including methods to align embeddings using distance-based, similarity-based, correlation-based, and ranking-based penalties during training .
Ranking-based methods for multi-view representation alignment, such as that described by Deng at al\cite{deng2018triplet}, allow the incorporation of ranking information into the training procedure, which may be important in tasks such as gene expression where perturbation signals may be small relative to baseline state. In addition, the field of rank-based embedding learning is intertwined with a broader literature of uni-modal embedding learning, which pioneered such architectures as Twin \cite{koch2015siamese, filzen2017representing} and Triplet networks \cite{hoffer2015deep}, which optimize embeddings to bring similar data together while driving dissimilar data apart. An analysis of best practices of these architecture can be found in Wu et al\cite{wu2017sampling}.
% \section{Formal Task Definition}
% As stated above, the . Put more formally, our dataset provides gene expression signatures $ \mathcal{G}_{\text{train}} $ and $\mathcal{G}_{\text{test}}$, small molecule encodings $\mathcal{M}_{\text{train}} $ and $\mathcal{M}_{\text{test}}$, and a mapping $ G2M : \mathcal{G}_{\text{train}} \rightarrow \mathcal{M}_{\text{train}}$ from training gene expression profiles to the small molecule that induced them.
\section{Methods}
\subsection{Dataset \& Tasks}
\noindent \textbf{Data Acquisition and Subsetting}
All data in this study comes from the LINCS Consortium/NIH Next-Generation Connectivity Map Level 3 L1000 data.\cite{subramanian2017next}
% The raw data comes in the form of gene expression profiles generated using the L1000 technology, a low-cost assay which measures 978 ``landmark" genes that have been deemed representative of the full transcriptome.
This dataset features 978-dimensional gene expression profiles from a variety of human cell lines treated with chemical and genetic perturbations. To ensure support over possible drugs, our data cut uses the most frequent 8 cell lines split into train, validation, and test sets such that no cell line or drug in the train set appears in the validation or test sets. To mitigate non-random missingness, we included only drugs assayed in all cell lines, and limited experiments to those incubated with small molecules for 24 hours at a dose of 10$\mu m$. Final statistics of these data are shown in Table~\ref{tab:dataset_stats}. For drug structures, we used the SMILES \cite{weininger1989smiles} structures provided by LINCS, canonicalized using RDKIT \cite{landrum2006rdkit}.
% Note that during one round of ablation experiments, outlined in Appendix Section~\ref{sec:ablations}, we also used a cut of data to the top 9 most frequent cell lines in order to be able to better assess the generalizability over cell lines alone.
% In order to eliminate as many sources of variance as possible beyond drug structure and cellular response, we limited our analysis to experiments conducted on the PC3 cell line and
\noindent \textbf{Preprocessing and Feature Engineering}
Gene expression intensity values from the training, validation, and test sets were centered and scaled at the gene-level based on the mean and standard deviation of each gene intensity across the training set. We augmented each gene expression profile with three additional sets of features: the corresponding gene expression intensities from a control signature on the same plate, the log$_2$ fold-change between the perturbation and control signatures, and the difference between these gene expression signatures.
%We also explored using a \emph{rank} transform, in which raw expression levels are replaced by their relative rank of over-expression, which is commonly used in bioinformatics to make gene expression data more robust, but found it to be uniformly unhelpful in our models.
%
For use in our baseline and oracle models, we also computed numerical representations of each small molecule: Morgan extended-connectivity fingerprints\cite{rogers2010extended} and the output of the ChemProp network.\cite{yang2019analyzing}%, wang2011pubchem}.
\noindent \textbf{Detailed Task Description}
Our goal is to learn embedders which map molecular structures and gene expression profiles into a vector space such molecular structure embeddings are close to the gene expression profile they induce while being far from other gene expression profiles (Figure 1). Formally, given a collection of gene expression signatures $\mathcal{G}$, chemical structures $\mathcal{M}$, and similarity function $\mathrm{Sim}:\R^d \times \R^d \rightarrow \R$, we seek to learn a gene expression embedder $E_g:\mathcal{G} \rightarrow \R^d$ and chemical embedder $E_m:\mathcal{M} \rightarrow \R^d$ to maximize $\mathrm{Sim}(E_g(g_i), E_m(m_j))$ while simultaneously minimizing $\mathrm{Sim}(E_g(g_i), E_m(m_{\neg j}))$, where gene expression $g_i$ was induced by molecule $m_j$. Unless otherwise specified, the similarity function can be assumed to be Pearson Correlation in our experiments. Across our baseline and oracle methods, we realize many variants of $E_g$ and $E_m$.
\subsection{Baseline and Oracle Methods}
\label{section:baselineMethod}
\noindent \textbf{Nearest Neighbor Baseline}
Nearest-neighbor (NN) methods have been previously shown to establish strong baselines for machine learning tasks on the L1000 data \cite{hodos2017cell, mcdermott2019deep}. In our cross-modal, information retrieval (IR) context, traditional NN methods are not applicable, so we employ the following ``double NN'' baseline: given a gene expression profile as a query, we first identify the nearest gene expression profile in the train set and look up its corresponding small molecule. We then take this small molecule (from the train set) as a query, and return the most structurally similar drug from the \textit{test} set as our final prediction.
In particular, given a mapping $G2M : \mathcal{G} \rightarrow \mathcal{M}$ from gene expression profiles to the small molecule that induced them, and a molecular embedding $E_m$ (which may include molecular fingerprints, Chemprop embeddings, or embeddings learned from other models), we define embedder $E_g: g_\text{query} \mapsto E_m(G2M(\argmax_{g_\text{tr} \in \mathcal G_\text{train}} \mathrm{Sim}(g_\text{query}, g_\text{tr})))$. Then, we perform information retrieval (IR) analyses with such embedders as usual.
% transcriptional and chemical similarity functions $\mathrm{Sim}_{\text{g}}: \mathcal{G} \times \mathcal{G} \rightarrow \R $ and $\mathrm{Sim}_{\text{m}}: \mathcal{M} \times \mathcal{M} \rightarrow \R $, each monotonically increasing with similarity. Given some gene expression query $g_{\text{query}}$, we then rank all molecular structures $m_{\text{test}} \in \mathcal{M}_{\text{test}}$ according to
% $\mathrm{Score}_{NN} = \mathrm{Sim}_{\text{m}} \left( G2M\left( \argmax_{g_{\text{tr}} \in \mathcal{G}_{\text{train}}} \mathrm{Sim}_{\text{g}} \left(g_{\text{\text{query}}}, g_{\text{tr}}\right)\right), m_{\text{test}}\right).$
% We use Pearson Correlation as our similarity functions $\mathrm{Sim}_{g,m}$, over various representations of genes/molecules, including extended connectivity fingerprints, Chemprop embeddings, or the aligned chemical embeddings generated by our joint modeling procedure\footnote{described in section~\ref{section:alignedMethodDescription}} for molecules and transcriptional signatures or our aligned gene expression embeddings for gene expression.
\noindent\textbf{Canonical Correlation Analysis Baseline}
% As described in Background, the canonical correlation procedure takes in a pair of data matrices (of potentially different dimensions) and produces a corresponding pair of linear mappings that maximizes the element-wise correlation between the mapped data matrices. This lends itself to a supervised baseline for this task, as follows:
Given training matrices of transcriptional $\mathcal{G}_{\text{train}}$ and molecular $\mathcal{M}_{\text{train}}$ encodings, we can learn a set of linear mappings $E_g : \mathcal G_\text{train} \to \mathbb{R}^d$ and $E_m: \mathcal M_\text{train} \to \mathbb{R}^d$ via $d$-dimensional CCA such that these mappings optimize the correlation between elements of $\mathcal G_\text{train}$ and $\mathcal M_\text{train}$.
% and some gene expression query $g_{\text{query}}$, we can rank all molecular structures $m_{\text{test}} \in \mathcal{M}_{\text{test}}$ according to a CCA baseline via
% $\mathrm{Score}_{CCA} = \text{Corr}( A \cdot g_{\text{query}}, B \cdot m_{\text{test}})$
% where $(A, B) = \mathrm{CCA}( G_{\text{train}}, M_{\text{train}} ).$
% %
% % This procedure defines an optimal \emph{linear} alignment between drug and transcriptions encodings in the training set.
% We computed CCA baseline results using all the same combinations of input representations as were employed by the double nearest neighbor approach.
% %
Note that this procedure requires a default numerical representation for molecules, which, as with other methods, can be either fingerprints, ChemProp embeddings, or learned embeddings by our learning model (described below). CCA can also be performed atop other embedding systems to further optimize embedding results. CCA was performed using SciKit Learn \cite{pedregosa2011scikit}, using 50 components, chosen to optimize validation set performance via a grid-search over a range of 5-125 components, run for 1000 iterations to ensure convergence.
% Default settings from SciKit Learn's CCA function were used, with two exceptions: first, the number of components was set to 50. The number 50 was chosen because a preliminary benchmark ranging from 5-125 components showed that 50 components had mildly superior performance in retrieval tasks on the validation set. Second, the maximum number of iterations was increased to 1000, since some initial experiments failed to converge at the default setting of 500.
% Note that other works have explored more involved, unsupervised methods for this problem, such as the use of generative adversarial networks or the Procrustes algorithm~\cite{grave2019unsupervised, hsu2018unsupervised}. In this work, given that we have fully paired data across our modalities, we did not feel the need to explore these other baselines as it was deemed unlikely they would offer performance improvements over the fully supervised methods.
\noindent \textbf{Oracle Models}
The central objective of our task is to learn small molecule embeddings that can stand in as surrogates for their corresponding gene expression signatures. To provide a rough upper-bound for expected performance on this task, we also implemented two ``oracle" models, each of which queries test set GE signatures against pseudo-``chemical embeddings" that are in reality the average GE signatures from each test set drug when it was measured on either (1) the \emph{train set cell lines},\footnote{Note that we can do this as we limited our choice of drugs to those that \emph{were} measured in all 8 cell lines, even though our actual data split prohibits training on any drug that appears in the validation or test sets.} to simulate an embedder that perfectly associates all structures to perturbational profiles, but cannot generalize beyond the train set cell lines, or (2) the \emph{test set cell lines}, which simulates a model of the same capabilities but able to generalize perfectly to the test set as well. These oracle models are still dependent on the underlying gene expression signature representation, so further innovation could offer improved upper bounds for this task. Formally, given $G2M$ mapping gene expression profiles to their corresponding perturbing molecule, we define oracle embeddings $E_g^\text{train}: g_\text{query} \mapsto \mathrm{Avg}(\{g_i \in \mathcal G_\text{train} | G2M(g_i) = G2M(g_\text{query})\})$, and $E_g^\text{test}: g_\text{query} \mapsto \mathrm{Avg}(\{g_i \in \mathcal G_\text{test} | G2M(g_i) = G2M(g_\text{query})\})$.
% Importantly, we note that while these oracle models simulate perfect alignment between the chemical embedder and the gene expression signatures, they do not necessarily represent a true upper bound on this task. This is because these oracle models still require some encoding of the gene expression signatures, which may not be the most generalizable representation of the GE data. Accordingly, we compute the performance for each of the above two oracle models using both raw GE signatures and the deep GE signatures learned through our joint models. This was done to identify the extent to which further innovation in GE embedding be required for improved performance on this task.
\subsection{Deep Coordinated Metric Learning Approach}
\label{section:alignedMethodDescription}
% We designed a cross-modal ranking algorithm to learn representations of chemical structure and gene expression such that each are jointly optimized to facilitate small molecule retrieval from gene expression signatures. An overview of this algorithm is depicted in Figure 1.
% %In brief, gene expression profiles and drug structures are each embedded through separate neural networks into a shared embedding space. These networks are optimized jointly to produce representations that minimize the distance in the embedding space between gene expression signatures and the drugs that induced them, while maximizing the distance between GE signatures and the embeddings of other structures.
% %
% % More formally,
% Formally, given a collection of gene expression signatures $\mathcal{G}_{\text{train}}$, chemical structures $\mathcal{M}_{\text{train}}$, and similarity function $\mathrm{Sim}:\R^d \times \R^d \rightarrow \R$, we seek to learn a gene expression embedder $E_g:\mathcal{G} \rightarrow \R^d$ and chemical embedder $E_m:\mathcal{M} \rightarrow \R^d $ to maximize $\mathrm{Sim}(E_g(g_i), E_m(m_j)$ while simultaneously minimizing $\mathrm{Sim}(E_g(g_i), E_m(m_{\neg j}))$, where gene expression $g_i$ was induced by molecule $m_j$.
For our learned model, we realize $E_g$ as a self-normalizing neural network (of size dictated by hyperparameter search), and $E_m$ as a directed message-passing neural network (D-MPNN), initialized by the Chemprop system, followed by a feed-forward output layer whose shape was dictated via hyperparameter search.\cite{yang2019analyzing} To train these architectures, we use a margin-based quadruplet loss, building on Wu et al's adaptive margin loss.\cite{wu2017sampling} The base of the adapted margin loss is defined over two data points $i$ and $j$ as $\mathrm{mar}_{\alpha, \beta} := \left( \alpha+y_{i,j} (D_{i j}-\beta) \right)_{+}$,
where $D$ is distance function (here euclidean distance), $\alpha$ defines a permissible margin of separation, $\beta$ controls the boundary between positive and negative pairs, and $y_{i,j}$ is an indicator variable equal to 1 if $i$ and $j$ are of the same class and 0 otherwise ($\alpha$ and $\beta$ were tuned as hyperparameters).
% Given this margin loss, we first implemented cross-modal triplet networks \cite{hoffer2015deep, deng2018triplet} to align gene expression embeddings to molecular structure embeddings. In these networks, we provide the loss function with
Given two pairs of matching gene expression and molecular structure embeddings, $(g_A, m_A), (g_B, m_B)$, our quadruplet loss is defined as the sum of the margin losses between all cross-modality pairs of embeddings: $\ell_\text{quad} = \mathrm{mar}_{\alpha, \beta}(g_A, m_A) + \mathrm{mar}_{\alpha, \beta}(g_A, m_B) + \mathrm{mar}_{\alpha, \beta}(g_B, m_A) + \mathrm{mar}_{\alpha, \beta}(g_B, m_B)$.
%
% \emph{anchor} gene expression embedding, a \emph{positive} (matching) molecular structure embedding, and a \emph{negative} (non-matching) molecular structure embedding, we define the triplet loss as the sum of the margin losses between (a) the anchor and the positive embedding, and (b) and the anchor and the negative embedding.
The network is thus optimized to bring the positive embedding within the margin of the anchor and negative embedding outside the margin.
% To give an example: if the anchor is a drug structure, the positive example is a gene expression signature that was induced by that drug structure, and the negative example is a gene expression signature induced by some other molecular structure. The loss function then tries to force the gene expression and its corresponding small molecule structure toward each other in embedding space and away from the other drug embedding.
%
% Throughout this paper, we refer to our triplet networks as ``Gene Expression-Anchored'' if their anchor embeddings are gene expressions and their positive/negative embeddings are drug structures, and as ``Chemical Structure-Anchored'' otherwise.
% We also implemented a quadruplet design that combined the Gene expression-Anchored and Chemical Structure-Anchored losses. In the quadruplet architecture, we sample a gene expression signature as anchor, positive and negative small molecule structures, and an additional gene expression signature induced by the negative small molecule structure. We then compute both the ``Gene expression-anchored'' triplet margin loss using the anchor gene expression signature and two molecular structures, as well as the ``Chemical structure-anchored'' margin loss using the first structure embedding as an anchor and the two gene expression embeddings as positive and negative examples. These two margin-based triplet losses are then summed together to form the quadruplet margin loss.
%
For sampling these two pairs (an analog of negative sampling for a more traditional triplet network), we first sample one matching pair, choose the molecular structure for the other pair based on the distance-weighted negative sampling scheme described in Wu et al, which was successful with their margin-based approach \cite{wu2017sampling}, then fill in the other gene expression profile to match the sampled molecular structure. To make this process computationally efficient we pre-computed the average distance in \emph{average post-perturbational gene expression space} between every pair of small molecule structures in the dataset.
%
We additionally tried other losses, including two varieties of traditional triplet losses, and a quintuplet loss, but ultimately found the quadruplet loss to be most performant via our hyperparameter search.
\noindent \textbf{Training and Hyperparameter Selection}
\label{sec:hyptun}
Each model was trained on a Nvidia GeForce GTX 1080 GPU. Early stopping was used to select the model with the best mean reciprocal rank on the validation set. Hyperparameter tuning via the Bayesian Hyperopt library~\cite{bergstra2013making} was performed over a wide range of possible hyperparameters, including network depth and width (parametrized by the size of the first hidden layer and a growth rate), learning rate, number of epochs, batch size, margin and $\beta$ parameters, triplet model orientation (i.e., gene first or compound first), activation function/network type (e.g., SNNN vs. unconstrained fully connected network), and dropout, with no early stopping for hyperparameter search runs. The optimal hyperparameters from this search are shown in Appendix Section~\ref{sec:final_params}.
\subsection{Experiments}
We designed a range of experiments with two purposes: First, we sought to evaluate if and how our deep coordinated representation learning method offers improvements over principled baselines. This entails a quantitative performance comparison against baseline methods and ablated versions of our model. Second, we introspect into the representations learned by training on this new task, to better understand the challenges and utility of the general framework. This entails a quantitative performance comparison against oracle models, a statistical analysis probing the ability of our models to generalize to new structures, and a qualitative exploration of the changes in chemical representations that are induced by our training scheme.
Quantitative performance analyses began by computing the embeddings of gene expression signatures and chemical structures in the test set, using the baseline, oracle, and deep coordinated methods defined in Sections~\ref{section:baselineMethod} and ~\ref{section:alignedMethodDescription}. Using each embedded gene expression signature as a query, we ranked all chemical structures in the test set based on their proximity to that gene expression signature in the embedding space. These rankings were then used to compute standard information-retrieval metrics (precision-recall curves, MR, MRR, and Hits at/H@ 10 or 100). For our ablation analyses, we repeated the above experiments using various combinations of raw and learned GE and chemical representations.
In addition to the information retrieval analyses, we probed the generalizability of our representations by analysing the statistical relationship between the average retrieval performance for each chemical structure and the structural similarity to the most similar chemical in the training set. Similarity was measured via the Tanimoto distance between all pairs of molecular fingerprints in the train and test set. We further examined performance vs. chemical specificity, using the number of genes that a molecule, on average, affected as a measure of specificity. Finally, we visualized the latent space of our chemical embedder (versus their pre-trained representations learned by ChemProp), and noted the relative position of drugs with the same mechanism of action (MOA) in each latent space.
% We designed a range of experiments with two primary purposes: First, we evaluate our proposed chemical alignment method on this task, benchmarked below by baseline methods and above by our oracle models. Second, we introspect into the resultant models trained over this task, in order to better understand what aspects of the task are most challenging and where additional improvements can be gleamed. In particular, we first profile basic results of our system, baseline models, and oracle models in this cross-modal retrieval task, examining results in terms of precision-recall curves, MR, MRR, and Hits at 10 or 100 (H@10 or H@100, respectively).
% Next, we analyze several baseline model ablations, using various alternative retrieval mechanisms based on our model's learned representations, rather than raw representations, to assess how big of an effect embedding quality has on baseline and oracle methods, as well as assessing the importance of post-embedding CCA on these metrics.
% Lastly, we additionally run several targeted experiments to assess which aspects of \emph{generalization} are most critical to this task. These include not only analyses of performance differences between oracle models (either that which requires only generalizability within train-set cell lines vs. test-set cell lines), but additionally examinations of the impact similarity between the query chemical structure and the train-set chemical structures has on performance (similarity as measured in Tanimoto distance over molecular fingerprints) or for the analogous relationship with gene expression signatures.
% \textbf{Qua ntitative IR Results}
% We evaluated all scoring strategies using the standard information retrieval metrics \emph{median rank} (MR), \emph{mean reciprocal rank} (MRR), and \emph{Hits} or \emph{Recall @ K} \cite{manning2010introduction}.
% % In addition, we plot the empirical cumulative distribution function (E-CDF) of the ranks of the true perturbagen among the results using all methods. This E-CDF plot is equivalent to a continuous-valued Hits @ K for all possible K in this dataset, and allows for a visual comparison between experiments.
% \textbf{Performance Analysis}
% In addition to the baseline and oracle analyses, we further examined the role of generalization in retrieval performance by comparing retrieval ranks for a given query to the distance from that query to the nearest neighbor in the training set. We examined these relationships both with respect to GE query signatures (computed both using raw GE profiles as well as embedded GE signatures) and for chemical structures to which the query corresponded (distance to the training set computed using Tanimoto distance using Molecular Fingerprints, and Pearson correlation using ChemProp and aligned embeddings).
% We also hypothesized that a compound's functional specificity may influence the difficulty of learning to encode its effect. To test this hypothesis, we first conducted a standard differential expression analysis for each compound, using the Limma package in R, and tabulating the total genes that are differentially expressed following treatment with each compound. Because the goal of this analysis was to get a simple measure of overall drug specificity and not to identify or report the effect on individual genes, we counted a gene as differentially expressed if Limma assigned it an unadjusted P-value less than 0.05.
% \textbf{Embedding Introspection Experiments}
% We were also interested in assessing the properties of the embeddings learned through our training task. To measure this, we computed UMAP representations of both our embeddings, and raw ChemProp embeddings of the small molecules in our training set,\cite{mcinnes2018umap} then assessed the concordance of any clusters in the UMAP space with clusters of drugs induced by drug MOA (if known).
% These analyses allowed us to assess qualitatively whether or not our jointly aligned representations appear to learn more or less rich notions of chemical embeddings than does the raw chemprop model.
% Using metadata from the LINCS consortium, we then labeled compounds with their known mechanism of action, if any. This enabled us to plot the location in the UMAP latent space of the drugs with the three most common mechanisms of actions, using which we compared the latent spaces of the pretrained Chemprop and aligned embeddings. Our hypothesis was that even though both models had comparable expressive capacity (both have identical message passing architectures), fine-tuning the embeddings using our GE-drug alignment task might force the representations to encode more clinically-relevant features than when trained only on the protein-binding task.
% Finally, we computed UMAP representations of the each compound using the averaged GE signatures from train and test, to examine where test set queries sit in relation to the training data in the latent transcriptional space. While all training set drugs were measured in 6 cell lines, there were a variable number of samples coming from each cell line-drug pair. To correct for cell line imbalances, we thus weighted the GE averages such that each cell line contributed equally to the overall average for each drug. Finally, in order to elucidate clusters in the latent space, additional UMAP projections were generated from the same data, but with raw GE measurements restricted to only those coming from control or perturbed cells.
\section{Results \& Discussion}
\subsection{Quantitative IR Experiments}
\textbf{Baseline and Proposed Method} Information retrieval results from the baseline and proposed methods are reported in Figure~\ref{fig:PRCurves}. This figure shows that our model variants offer significant performance improvements over either of the CCA or NN baselines, and even approach the performance of the train-set only oracle model. All models fall dramatically short of the test-set generalizability oracle, which indicates that while our tasks offer significant improvement over baseline models here, there are still major gains possible, primarily by focusing on improving the generalizability to the novel cell-types of the test set.
In addition, we show various ablation studies over the baseline models in Table~\ref{tab:main_results} to probe what gene or molecular representations would make them better or worse. Strikingly, we can note that uniformly using aligned representations (meaning representations based on our multi-view alignment neural network architecture) offers significant improvements over other representations, indicating that even with a baseline approach such as the double nearest neighbor (D-NN) model, improvements to the embedding quality translate to notable improvements to IR performance. Notably, this is true both for GE and Chemical embedders, with Aligned-Aligned representations yielding optimal performance for both D-NN and CCA query mechanisms. Additionally, it is also clear that CCA is the preferred query metric, over either raw correlation (Corr) based lookups or D-NN based lookups.
% As can be seen, the numerical representation of the chemical structures has a larger impact on performance than any other factor. Strikingly, the largest increase in performance comes when introducing our alignment training -- this is despite the fact that the Chemprop model is the core of the Aligned embedding architecture. In addition, while CCA seems to slightly outperform our nearest neighbors baseline given identical representations, both methods perform their best by far when using our aligned representations as their input. Of note, application of the CCA process to our embeddings yields a slight improvements in the results among early hits (as evidenced by MRR, H@10, the low recall region of Figure~\ref{fig:PRCurves}B). However, vanilla correlation-based retrieval yields slightly better results according to metrics that that do not actively emphasize earlier results (MR, H@100, etc.). The near-neighbors method provides no benefit over vanilla correlation-based ranking when using our aligned embeddings.
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{figures/precision_recall_all.png}
% \caption{Precision Recall curves for drug identification given gene expression signatures. The ``alignedGE\_alignedChem\_CCA" and ``alignedGE\_alignedChem\_noCCA" curves, plotted in blue and orange, respectively, in all plots, represents our approach as described in Section~\ref{section:alignedMethodDescription} with and without CCA post-processing. All other methods represent baseline or oracle methods as described in Section~\ref{section:baselineMethod}.}
% \label{fig:PRCurves}
% \end{figure}
\begin{figure}
\centering
\includegraphics[width=0.55\linewidth]{figures_nature_methods/main_results_figure.png}
\caption{Precision Recall curves for drug identification given gene expression signatures, across various baselines (dashed lines), oracles (dotted lines) and our model (solid lines).}
\label{fig:PRCurves}
\end{figure}
\begin{table}
\tbl{
IR metrics across various configurations of the model/baselines. `Chemprop' refers to pretrained model from Yang et al \cite{yang2019analyzing}. `Aligned' indicates representations learned from our method (see Section~\ref{section:alignedMethodDescription}). MR=median rank, MRR=mean reciprocal rank, H@K=Hit/Recall at K.
}{
\begin{tabular}{lllrrrr}
\toprule
\textbf{GE} & \textbf{Chemical} & \textbf{Method} & \textbf{MR} & \textbf{MRR} & \textbf{H@10} & \textbf{H@100} \\
\midrule
Raw & Morgan FP & D-NN & 206 & 0.025 & 0.037 & 0.240 \\
Raw & Chemprop & D-NN & 211 & 0.025 & 0.035 & 0.254 \\
Raw & Aligned & D-NN & 189 & 0.033 & 0.047 & 0.290 \\
Aligned & Morgan FP & D-NN & 214 & 0.025 & 0.041 & 0.256 \\
Aligned & Chemprop & D-NN & 196 & 0.022 & 0.037 & 0.278 \\
Aligned & Aligned & D-NN & 137 & 0.039 & 0.072 & 0.402 \\
\midrule
Raw & Morgan FP & CCA & 180 & 0.027 & 0.045 & 0.303 \\
Raw & Chemprop & CCA & 184 & 0.024 & 0.040 & 0.294 \\
Raw & Aligned & CCA & 134 & 0.039 & 0.076 & 0.412 \\
Aligned & Morgan FP & CCA & 177 & 0.027 & 0.050 & 0.319 \\
Aligned & Chemprop & CCA & 163 & 0.028 & 0.051 & 0.334 \\
Aligned & Aligned & CCA & 130 & \textbf{0.048} & \textbf{0.093} & 0.425 \\
\midrule
Aligned & Aligned & Corr & \textbf{126} & 0.042 & 0.085 &\textbf{ 0.432} \\
\bottomrule
\end{tabular}}
\label{tab:main_results}
\end{table}
\textbf{Oracle Model Analysis} Results from the oracle models are reported in Figure~\ref{fig:eCDF} and Table~\ref{tab:oracle_results}. As expected, oracle models using GE signatures from the test cell line greatly outperformed those using signatures from the train cell lines. This stark difference suggests that one of the largest barriers to performance here is the generalization gap between different cell lines. This further motivates for the curation of larger, cell-line heterogeneous datasets in the future.
In addition, aligned embeddings modestly improved the performance of test set oracles, and greatly improved the performance of train set oracles, consistent with the GE embedders learning slightly more generalizable representations. Of note, our proposed model achieved comparable results as the oracle model that leveraged raw GE signatures from the train cell lines. More specifically, our approach yielded slightly worse than the oracle on metrics (MRR, H@10) that emphasize early rankings, and slightly better on metrics (MR, H@100) that focus on more aggregate results. This is also apparent in the precision-recall plot, which shows the aligned embeddings curve starting out slightly below that of the raw/train oracle curve but then moving rapidly above it as further results are considered.
\begin{table}
\tbl{
IR metrics for the various oracle methods.
}{
\begin{tabular}{lrrrr}
\toprule
\textbf{Oracle Model} & \textbf{MR} & \textbf{MRR} & \textbf{H@10} & \textbf{H@100} \\
\midrule
Oracle - Train Cell Lines (Raw GE) & 138 & 0.057 & 0.101 & 0.400 \\
Oracle - Train Cell Lines (Embed) & 110 & 0.064 & 0.128 & 0.466 \\
Oracle - Test Cell Line (Raw GE) & 82 & 0.076 & 0.147 & 0.565 \\
Oracle - Test Cell Line (Embed) & 79 & 0.093 & 0.160 & 0.568 \\
\midrule
Our Approach & 126 & 0.042 & 0.085 & 0.432 \\
\bottomrule
\end{tabular}}
\label{tab:oracle_results}
\end{table}
\subsection{Introspection Analyses}
%
% \textbf{Performance as a function of distance from the training set.}
Figure~\ref{fig:perf_predictors}A
% and~\ref{fig:performance_predictors_append}
contains the results of our experiments comparing performance to distance from the training set. Regardless of the measure of chemical similarity, compound retrieval performance was inversely correlated with distance from the training set. As can be seen in Supplementary Figure~\ref{fig:performance_predictors_append}, the same trend held with learned gene expression embeddings, and was present but much weaker using raw gene expression profiles.
% Finally, as can be seen in the figure in the appendix, we note that some molecules were found to have a distance of zero from the training set according to the molecular fingerprints, but not according to the chemical embedding methods. This is due to the fact that chemical fingerprints fail to encode specific details of molecular structure; examples of structures from the dataset that have identical molecular fingerprints but distinct ChemProp and Aligned Embeddings are shown in Figure~\ref{fig:mols_w_small_fingerprints}.
\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{figures/performance_predictors}
\caption{Left: Performance (lower is better) vs. structural distance to the nearest compound in the training set. This plot demonstrates that compounds more structurally dissimilar to the train set show mildly worse performance than those that are more similar. See Appendix Figure~\ref{fig:performance_predictors_append} for analogous plots for four additional measures of distance from the training set. Right: Average Performance vs. \# of Genes deferentially expression following treatment with the molecule of interest, showing that compounds that have broader transcriptomic effects are better retrieved by this method.}
\label{fig:perf_predictors}
\end{figure}
% \textbf{Performance as a function of drug specificity.}
Figure~\ref{fig:perf_predictors}B depicts the relationship between the transcriptional specificity of a compound and its ability to be retrieved using our analysis. As can be seen, there is a mild negative correlation, implying that molecules that affect the expression of many genes are easier to retrieve using this approach. Note that this observation is concordant with our findings on the difficulty of generalizing to new cell lines -- drugs that affect a small, targeted set of genes are more likely to be cell line specific, and as our model is forced to surmount a significant generalization gap in evaluation, such cell-line specific signals are largely wiped out.
In addition, our analysis of the changes induced in the embedding space, shown in Supplementary Figure~\ref{fig:clustering_moas}, reveal that our model's embeddings of molecules appear to better cluster shared MOAs than do the raw ChemProp embeddings, from which our model is initialized. This suggests that, as hypothesized regarding the nature of this task, our model is learning rich representations of the underlying molecules, though additional work remains to investigate this effect more thoroughly.
% \section{Discussion}
% \subsection{Overall Summary}
% In this paper, we have sought to present both a new biomedical machine learning task -- cross-modal retrieval of molecular structure from perturbational signatures -- as well as a set of approaches to benchmark future work on this task.
% Our results indicate that the task is feasible, and that the richer models afforded by recent advances in deep learning show marked improvements over baseline methods. More specifically, learned representations of GE and small molecules appear to exhibit stronger retrieval performance on this task requiring generalization to both unseen cell line and chemical structures. In addition, our latent space introspection analyses indicate the the aligned training task may result in more robust and biomedically meaningful representations.
% However, even the strongest models studied provide only modest performance on this task overall, indicating we still have significant room to improve. In addition we observe that the gap between our chemical embedding and the train-set oracle was far narrower than the gap between the train-set oracle and the test-set oracle. This indicates that the problem of generalizing drug activity across \emph{cell types} may be at least as daunting as the generalization problem across new \emph{drug structures}. This raises questions not simply for our task, but also for classical connectivity mapping.
\subsection{Future Work}
We see several opportunities for further work on this task. First, expanding our data coverage, across molecules, cell lines, dosages, and treatment durations will allow us to measure and improve generalizability here. Second, exploring additional strategies to use the over-sampled nature of these data (e.g., ensembling together control and perturbational signatures to reduce variance) could be beneficial. Third, a more robust exploration of model architectures, losses, and deep metric learning/negative sampling methods, could offer improvements here.\cite{li2018survey} Additionally, other styles of multimodal embedding could be explored, such as the use of cycle generative adversarial network, which in particular would enable us to adopt a semi-supervised approach.\cite{felix2018multi,mcdermott_semi-supervised_2018}
The use of interpretability methods, particularly those used for graph analyses,\cite{ying2019gnnexplainer} as well as additional studies interrogating how our model's performance changes with the amount of available training data could also be insightful here. Fourth, we recommend exploring methods to improve cell-line generalizability, e.g. incorporating information across many cell lines when forming predictions. Finally, we also note that while our analyses only examine small-molecule therapeutics, similar methods could also be applied to other modalities, such as RNA-based therapies.
% From the perspective of task formulation, one area of future interest is the development of custom metrics to score model ranks. As it stands, the evaluation strategy counts the exact perturbagen as correct, and all other small molecules as equally incorrect. This will potentially under-estimate model performance, as it ignores the possibility of other plausible candidates. It would therefore be useful to consider metrics that incorporates functional or structural similarity to provide non-zero scores to reasonable alternatives. An additional concern is that the evaluation on unseen molecular structures is limited to only ~300 molecules, which means that those scores will likely have higher bias and variance. Finally, expanding the task to include -- and ideally translate between -- perturbational signatures from multiple cell types and dosages will be essential to assessing and optimizing generalizability to new biological settings. Given sufficiently robust results, we could then assess this method as an in silico nominating scheme, either as a drop-in replacement for connectivity mapping techniques or as a method to choose which drugs (from an arbitrarily large library) to profile using transcriptomics readouts.
% From the perspective of model formulation, there are several follow-up experiments that we are excited to pursue that would require only additional computational and human resources. These include a more dedicated hyperparameter optimization search and the implementation of test-time augmentation by ensembling predictions over repeat control sampling. In addition, we would like to compute confidence intervals for all experimental values computed by bootstrapping. Such exercises will be indispensable in identifying which if any of the comparisons in performance may be due to statistical aberration rather than substantive difference.
% Slightly more sophisticated follow-up work includes the investigation of more intelligent pretraining strategies and negative sampling techniques, which may prove essential for improved generalization. In addition, as described in Li et al, there are a wide variety of additional architectures that could be tried for this task, such as Cross-Modal Factor Analysis, Kernel CCA, and a range of novel deep learning architectures \cite{li2018survey}. Anecdotally, one model which we did begin to explore was Deep Canonical Correlation Analysis \cite{andrew2013deep}, which on first implementation has not produced useful embeddings for this task. These results were not included formally in this work, however, due to insufficient time to ensure the results weren't due to poor hyperparameters.
% Of greatest interest to us as authors is an even deeper examination of the failure modes that take place when the model tries to generalize to unseen molecules and cell types. In particular, we would like to characterize the properties of \emph{chemical libraries} that do or do not enable mutual generalizability. It is our hope that elucidating these pitfalls could lead both to better algorithms and to better chemical libraries. By the same token, we hope that in-depth investigation of generalizability of drug signatures across \emph{cell-lines} will improve not only improve our ability to train better drug embedders in this task, but also yield insights that are beneficial for classic connectivity mapping.
\section{Conclusion}
We present a new task: cross-modal multi-view alignment between drug structures and perturbational gene expression profiles, which links molecular structure to an objective, functional readout of drugs with very broad biomedical relevance. We profile state-of-the-art representation learning methods on this new task, and inspect the learned chemical embeddings. We find that this modeling task induces an embedding space reflective of drug mechanism of action -- which is not explicitly included in the training regime -- and see modest generalization to both new structures and a new biological environment. Our oracle experiments demonstrate major performance gaps when trying to generalize to new tissues. We hope that this new benchmark task will catalyze future research and ultimately help enable a rapid, in silico compound prioritization methods.
\section{Acknowledgements}
The authors thank Connor Coley and Kyle Swanson for providing the pretrained chemical embedder from Yang et al \cite{yang2019analyzing}. S.G.F. was supported by training grant T32GM007753 from the National Institute of General Medical Science. M.B.A.M. was funded in part by National Institutes of Health: National Institutes of Mental Health grant P50-MH106933 as well as a Mitacs Globalink Research Award. The content is solely the responsibility of the authors.
%%
%% The next two lines define the bibliography style to be used, and
%% the bibliography file.
\bibliographystyle{ws-procs11x85}
\bibliography{small_mol_align}
\clearpage
\appendix
\section{Appendix}
\subsection{Data Access}
Raw LINCS data are available for download at \url{https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE70138} and \url{https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE92742}. Our preprocessed versions of these files will be distributed with our Github repository.
\subsection{Final Optimal Parameters}
\label{sec:final_params}
\begin{table}
\centering
\tbl{
Dataset cell lines, number of samples, and number of drugs per split.
}{
\begin{tabular}{lp{0.4\linewidth}ll} \toprule
& Train & Validation & Test \\ \midrule
Cell Lines & HEPG2, HA1E, HCC515, VCAP, A375, PC3, MCF7& A549 & HT29 \\
\# Samples & 97210 & 1255 & 2317\\
\# Drugs & 3490 & 436 & 437 \\
% Most Sampled Drugs & & & \\
% Least Sampled Drugs & & & \\
\bottomrule
\end{tabular}
}
\label{tab:dataset_stats}
\end{table}
Final, optimal hyperparameters were selected according to the median rank of the parameter setting. Final values are shown in Table~\ref{tab:hyperparams}. We see lots of consistency between the two settings, despite the fact that both searches were independent. For example, both prefer very similar gene expression architectures, both run for 12 epochs, and both have relatively low margins.
\begin{table}[h!]
\centering
\tbl{Final optimal hyperparameters for the deep neural network architecture.}{
\begin{tabular}{lrr} \toprule
Parameter & Drug Split \\ \midrule
\# Epochs & 12 \\
LR & 5.6e-4 \\
Batch Size & 417 \\
Rank Transform & No \\
Neg. Samp. & N/A \\
Gene Exp. Emb. Arch. & 677, 1159 \\
Embed Size & 51 \\
Dropout Prob & 0.02 \\
Activation & SELU \\
Linear Bias & False \\
Beta & 1.3 \\
Margin & 0.31 \\
RdKit Ft. & True \\
\bottomrule \end{tabular}}
\label{tab:hyperparams}
\end{table}
% \begin{table}[h!]
% \centering
% \caption{Final optimal hyperparameters for the cell-line split and drug-level split experiments.}
% \begin{tabular}{lrr} \toprule
% Parameter & Cell Split & Drug Split \\ \midrule
% \# Epochs & 12 & 12 \\
% LR & 0.0003 & 5.6e-4 \\
% Batch Size & 104 & 417 \\
% Rank Transform & No & No \\
% Neg. Samp. & Expr & N/A \\
% Gene Exp. Emb. Arch. & 598, 1104 & 677, 1159 \\
% Embed Size & 65 & 51 \\c
% Dropout Prob & 0.3 & 0.02 \\
% Activation & SELU & SELU \\
% Linear Bias & True & False \\
% Beta & 1.74 & 1.3 \\
% Margin & 0.11 & 0.31 \\
% RdKit Ft. & True & True \\
% \bottomrule \end{tabular}
% \label{tab:hyperparams}
% \end{table}
%\subsection{Additional Validation Experiments}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/precision_recall_all.png}
\caption{Precision Recall curves for drug identification given gene expression signatures. The ``alignedGE\_alignedChem\_CCA" and ``alignedGE\_alignedChem\_noCCA" curves, plotted in blue and orange, respectively, in all plots, represents our approach as described in Section~\ref{section:alignedMethodDescription} with and without CCA post-processing. All other methods represent baseline or oracle methods as described in Section~\ref{section:baselineMethod}.}
\label{fig:eCDF}
\end{figure}
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{figures/ecdf_all.png}
% \caption{Empirical CDF representing the Recall @ K for drug identification given gene expression signatures. The "alignedGE\_alignedChem" curves, plotted in blue in all plots, represents our approach as described in Section~\ref{section:alignedMethodDescription}. All other methods represent baseline or oracle methods as described in Section~\ref{section:baselineMethod}.}
% \label{fig:eCDF}
% \end{figure}
\begin{figure}[!h]
\centering
\includegraphics[width=\linewidth]{figures/performance_predictors_append.png}
\caption{Alternative measures of performance as a function of distance from the training set. Accompanies Figure~\ref{fig:perf_predictors}. Note that while some test-set molecules have tanimoto distance of 0 to the train set, this does not imply they were shared between the two sets; rather, there are cases of two distinct molecules with identical molecular fingerprints. See Figure~\ref{fig:mols_w_small_fingerprints} for several examples.}
\label{fig:performance_predictors_append}
\end{figure}
\begin{figure}[!h]
\centering
\includegraphics[width=0.5\linewidth]{figures/mols_w_fp_in_train_4.png}
\caption{Examples of four molecules that have slightly distinct structures, but have identical Morgan Circular Fingerprints.}
\label{fig:mols_w_small_fingerprints}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/umap_drugs_w_moa.png}
\caption[Visualization of Latent Chemical Embedding Space]{This figure shows our induced embeddings (right) compared to established chemical embeddings (left), projected to 2-dimensions using UMAP, colored according to MOA. (Blue dots represent small molecules with different or unknown MOAs.)}
\label{fig:clustering_moas}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/ge_big_latent_viz_strat_controlsAll.png}
\caption{UMAP latent space of gene expression signatures, colored by train vs test set. A: Full raw gene expression signatures (post-perturbational signature, plate control signature, log-fold change). B: Raw gene expression from control signatures. C: Raw gene expression values, only the post-perturbational measurments. D: Aligned embedding representations.}
\label{fig:clustering_genes_controls}
\end{figure}
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{figures/ge_latent_cellLine_strat.png}
% \caption{UMAP latent space of gene expression signatures, colored by the cell line from which the most samples were originally measured. A: Raw gene expression signatures. B: Aligned embedding representation.}
% \label{fig:clustering_genes_cellLine}
% \end{figure}
% \begin{table}
% \caption{
% IR metrics across different configurations of the model and baselines. MR=median rank, MRR=mean reciprocal rank, H@K=Hits (Recall) at K. Chemprop means only pre-trained on \cite{yang2019analyzing}. Quadruplet indicates our twin embedder architecture.
% }
% \begin{tabular}{lll|rrrrr}
% \toprule
% % \multicolumn{3}{c}{Input} & \multicolumn{5}{c}{Test Set, All (3370 total)} & \multicolumn{4}{|c}{Test Set, Unseen Mols (312 total)} \\[3pt] \hline
% \textbf{GE} & \textbf{Chemical} & \textbf{Method} & \textbf{MR} & \textbf{MRR} & \textbf{H@10} & \textbf{H@100} & \textbf{H@500} \\ \midrule % & MR & MRR & H@10 & H@100 \\[3.5pt] \midrule
% Raw & Morgan FP & CCA & 1097 & 0.024 & 0.043 & 0.113 & 0.305 \\ % & & & & \\
% Raw & ChemProp & CCA & 1207 & 0.011 & 0.021 & 0.083 & 0.261 \\ % & & & & \\ \hline
% Raw & Aligned & CCA & 793 & 0.028 & 0.049 & 0.153 & 0.385 \\ % & & & & \\
% Aligned & Morgan FP & CCA & 1242 & 0.014 & 0.024 & 0.083 & 0.255 \\ % & & & & \\
% Aligned & ChemProp & CCA & 1275 & 0.008 & 0.016 & 0.075 & 0.247 \\ % & & & & \\
% Aligned & Aligned & CCA & 863 & 0.023 & 0.040 & 0.146 & 0.370 \\ % & & & & \\
% Aligned & Aligned & Corr & 1032 & 0.016 & 0.029 & 0.119 & 0.308 \\ % & & & & \\
% \bottomrule
% \end{tabular}
% \label{aba:tbl3}
% \end{table}
% \def\p{\phantom{$-$}}
% \def\pc{\phantom{,}}
% \def\p0{\phantom{0}}
% \begin{sidewaystable*}
% \caption{Information retrieval drug retrieval tasks. MR=median rank, MRR=mean reciprocal rank, H@K=Hits (Recall) at K.}
% \begin{tabular}{cc|c|ccccc|ccccc}
% \toprule\\
% \multicolumn{3}{c}{Input} & \multicolumn{5}{c}{Test Set, All (3370 total)} & \multicolumn{4}{|c}{Test Set, Unseen Mols (312 total)} \\[3pt]
% \midrule\\
% GE Encoding & Chemical Encoding & Alignment & MR & MRR & H@10 & H@100 & H@500 & MR & MRR & H@10 & H@100 \\[3.5pt]
% \hline\\
% Raw & Morgan FP & CCA & 1102 & 0.037 & 0.063 & 0.154 & 0.319 & 134 & 0.039 & 0.065 & 0.389 \\
% Raw & Chemprop & CCA & 874 & 0.045 & 0.078 & 0.192 & 0.380 & 116.5 & 0.055 & 0.096 & 0.447 \\
% Raw & Quadruplet & CCA & 604 & 0.057 & 0.099 & 0.242 & 0.460 & 100 & 0.059 & 0.123 & 0.503 \\
% \hline\\
% Quadruplet & Morgan FP & CCA & 644 & 0.035 & 0.063 & 0.206 & 0.444 & 131 & 0.041 & 0.068 & 0.400 \\
% Quadruplet & Chemprop & CCA & 403 & 0.049 & 0.091 & 0.267 & 0.538 & 101 & 0.048 & 0.079 & 0.499 \\
% \hline\\
% Triplet-GE-First & Triplet-GE-First & CCA & 236 & 0.070 & 0.123 & 0.341 & 0.666 & 97.5 & \textbf{0.062} & 0.126 & 0.511 \\
% Triplet-Chem-First & Triplet-Chem-First & CCA & 240 & 0.069 & 0.122 & 0.346 & 0.673 & 103 & 0.055 & 0.095 & 0.487 \\
% Quadruplet & Quadruplet & CCA & \textbf{196} & \textbf{0.073} & \textbf{0.135} & \textbf{0.384} & 0.706 & \textbf{93} & 0.056 & \textbf{0.136} & 0.521 \\
% \hline\\
% Triplet-GE-First & Triplet-GE-First & None & 306 & 0.055 & 0.099 & 0.289 & 0.626 & 101 & 0.051 & 0.098 & 0.499 \\
% Triplet-Chem-First & Triplet-Chem-First & None & 260 & 0.058 & 0.102 & 0.322 & 0.663 & 106.5 & 0.049 & 0.104 & 0.488 \\
% Quadruplet & Quadruplet & None & 199 & 0.066 & 0.122 & 0.372 & \textbf{0.711} & 96.5 & 0.060 & 0.124 & \textbf{0.523} \\
% \bottomrule
% \end{tabular}
% \label{aba:tbl4}
% \end{sidewaystable*}
% \subsection{Embedding Introspection Analysis}
% \vspace{1em}
% \textbf{Inspecting the Chemical Latent Space.} Figure~\ref{fig:clustering_moas} depicts the chemical latent space from test set compounds, annotated with the top three mechanisms of action. As hypothesized, fine-tuning the Chemprop model via our alignment task appears to have forced drugs with similar mechanisms of action to cluster closer together. (Note: Future work with more richly clinically annotated datasets will be required to truly validate this hypothesis.)
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{figures/umap_drugs_w_moa.png}
% \caption{This figure shows our induced embeddings (right) compared to established chemical embeddings (left), projected to 2-dimensions using UMAP, colored according to MOA.}
% \label{fig:clustering_moas}
% \end{figure}
% \textbf{Inspecting the Transcriptional Latent Space.} Figures~\ref{fig:clustering_genes},~\ref{fig:clustering_genes_controls}, and~\ref{fig:clustering_genes_cellLine} depict the latent space of the raw gene expression signatures and learned gene expression embeddings, as visualized using UMAP. Data points each represent averages across all samples from one perturbation, weighted to balance across cell lines. As can be seen, well-separated clusters are present in the latent spaces of both raw and embedded gene expression signatures. However, the latent space from the embeddings exhibits clusters that are less spread out and in which the test set signatures appear to be more integrated into the train set signatures. There are three broad categories of potential causes: true clusters of drug function, clusters due to cell-line specific effects, or technical artifact. To try to disentangle these categories of sources, we first tried to tease out drug-specific effects by producing plots of the raw GE signatures from \emph{only the control plates} matched to each perturbation experiment, as well as only the raw values of the perturbed cells. As can be seen in Figure~\ref{fig:clustering_genes_controls}, the clusters in these plots are essentially identical to those of the full raw GE signatures, implying that the clustering in the raw gene expression plots were likely not due to true drug-related signal. Finally, in Figure~\ref{fig:clustering_genes_cellLine}, we color points based on the cell line in which most of the samples had been originally measured, which aligns partially but not fully with the clusters, and more so in the raw GE space than in the embedded GE space. Given that data points had been weighted to correct for cell line imbalances, it is likely that the remaining correlation with clusters is due to technical confounding (for example, if specific drugs were primarily assayed at specific sites or operators, which also primarily utilized specific cell lines). Taken together, the results indicate that clusters are likely due primarily to non-drug effects, and it is thus a positive sign that the embedding process served to weaken and otherwise shift these clusters.
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{figures/ge_latent_trainTest_strat.png}
% \caption{UMAP latent space of gene expression signatures, colored by train vs test set. Panel A: Raw gene expression signatures. Panel B: Aligned embedding representations. See also Figures~\ref{fig:clustering_genes_controls} and~\ref{fig:clustering_genes_cellLine}, which demonstrate that visible clusters are likely due to non-drug effects, implying that the learned embeddings slightly improve robustness by weakening these effects.}
% \label{fig:clustering_genes}
% \end{figure}
\end{document}
\begin{thebibliography}{10}
\bibitem{hansson2002molecular}
T.~Hansson, C.~Oostenbrink and W.~van Gunsteren, Molecular dynamics
simulations, {\em Current opinion in structural biology} {\bf 12}, 190
(2002).
\bibitem{krallinger2017information}
M.~Krallinger, O.~Rabal, A.~Lourenco, J.~Oyarzabal and A.~Valencia, Information
retrieval and text mining technologies for chemistry, {\em Chemical reviews}
{\bf 117}, 7673 (2017).
\bibitem{battaglia2018relational}
P.~W. Battaglia, J.~B. Hamrick, V.~Bapst, A.~Sanchez-Gonzalez, V.~Zambaldi,
M.~Malinowski, A.~Tacchetti, D.~Raposo, A.~Santoro, R.~Faulkner {\em et~al.},
Relational inductive biases, deep learning, and graph networks, {\em arXiv
preprint arXiv:1806.01261} (2018).
\bibitem{wu2018moleculenet}
Z.~Wu, B.~Ramsundar, E.~N. Feinberg, J.~Gomes, C.~Geniesse, A.~S. Pappu,
K.~Leswing and V.~Pande, Moleculenet: a benchmark for molecular machine
learning, {\em Chemical science} {\bf 9}, 513 (2018).
\bibitem{kearnes2016molecular}
S.~Kearnes, K.~McCloskey, M.~Berndl, V.~Pande and P.~Riley, Molecular graph
convolutions: moving beyond fingerprints, {\em Journal of computer-aided
molecular design} {\bf 30}, 595 (2016).
\bibitem{yang2019analyzing}
K.~Yang, K.~Swanson, W.~Jin, C.~W. Coley, P.~Eiden, H.~Gao, A.~Guzman-Perez,
T.~Hopper, B.~Kelley, M.~Mathea {\em et~al.}, Analyzing learned molecular
representations for property prediction, {\em Journal of chemical information
and modeling} (2019).
\bibitem{lo2018machine}
Y.-C. Lo, S.~E. Rensi, W.~Torng and R.~B. Altman, Machine learning in
chemoinformatics and drug discovery, {\em Drug discovery today} {\bf 23},
1538 (2018).
\bibitem{ramsundar2015massively}
B.~Ramsundar, S.~Kearnes, P.~Riley, D.~Webster, D.~Konerding and V.~Pande,
Massively multitask networks for drug discovery, {\em arXiv preprint
arXiv:1502.02072} (2015).
\bibitem{faber2017prediction}
F.~A. Faber, L.~Hutchison, B.~Huang, J.~Gilmer, S.~S. Schoenholz, G.~E. Dahl,
O.~Vinyals, S.~Kearnes, P.~F. Riley and O.~A. Von~Lilienfeld, Prediction
errors of molecular machine learning models lower than hybrid dft error, {\em
Journal of chemical theory and computation} {\bf 13}, 5255 (2017).
\bibitem{gilmer2017neural}
J.~Gilmer, S.~S. Schoenholz, P.~F. Riley, O.~Vinyals and G.~E. Dahl, Neural
message passing for quantum chemistry, {\em Proceedings of the 34th
International Conference on Machine Learning-Volume 70} , 1263 (2017).
\bibitem{zitnik2018modeling}
M.~Zitnik, M.~Agrawal and J.~Leskovec, Modeling polypharmacy side effects with
graph convolutional networks, {\em Bioinformatics} {\bf 34}, i457 (2018).
\bibitem{musa2017review}
A.~Musa, L.~S. Ghoraie, S.-D. Zhang, G.~Glazko, O.~Yli-Harja, M.~Dehmer,
B.~Haibe-Kains and F.~Emmert-Streib, A review of connectivity map and
computational approaches in pharmacogenomics, {\em Briefings in
bioinformatics} {\bf 19}, 506 (2017).
\bibitem{lamb2006connectivity}
J.~Lamb, E.~D. Crawford, D.~Peck, J.~W. Modell, I.~C. Blat, M.~J. Wrobel,
J.~Lerner, J.-P. Brunet, A.~Subramanian, K.~N. Ross {\em et~al.}, The
connectivity map: using gene-expression signatures to connect small
molecules, genes, and disease, {\em science} {\bf 313}, 1929 (2006).
\bibitem{subramanian2017next}
A.~Subramanian, R.~Narayan, S.~M. Corsello, D.~D. Peck, T.~E. Natoli, X.~Lu,
J.~Gould, J.~F. Davis, A.~A. Tubelli, J.~K. Asiedu {\em et~al.}, A next
generation connectivity map: L1000 platform and the first 1,000,000 profiles,
{\em Cell} {\bf 171}, 1437 (2017).
\bibitem{musa2018harnessing}
A.~Musa, S.~Tripathi, M.~Kandhavelu, M.~Dehmer and F.~Emmert-Streib, Harnessing
the biological complexity of big data from lincs gene expression signatures,
{\em PloS one} {\bf 13}, p. e0201937 (2018).
\bibitem{liu2018systematic}
T.-P. Liu, Y.-Y. Hsieh, C.-J. Chou and P.-M. Yang, Systematic polypharmacology
and drug repurposing via an integrated l1000-based connectivity map database
mining, {\em Royal Society open science} {\bf 5}, p. 181321 (2018).
\bibitem{clark2014characteristic}
N.~R. Clark, K.~S. Hu, A.~S. Feldmann, Y.~Kou, E.~Y. Chen, Q.~Duan and
A.~Ma’ayan, The characteristic direction: a geometrical approach to
identify differentially expressed genes, {\em BMC bioinformatics} {\bf 15},
p.~79 (2014).
\bibitem{donner2018drug}
Y.~Donner, S.~Kazmierczak and K.~Fortney, Drug repurposing using deep
embeddings of gene expression profiles, {\em Molecular pharmaceutics} {\bf
15}, 4314 (2018).
\bibitem{dincer2018deepprofile}
A.~B. Dincer, S.~Celik, N.~Hiranuma and S.-I. Lee, Deepprofile: Deep learning
of cancer molecular profiles for precision medicine, {\em bioRxiv} , p.
278739 (2018).
\bibitem{rampavsek2019dr}
L.~Ramp{\'a}{\v{s}}ek, D.~Hidru, P.~Smirnov, B.~Haibe-Kains and A.~Goldenberg,
Dr. vae: improving drug response prediction via modeling of drug perturbation
effects, {\em Bioinformatics} {\bf 35}, 3743 (2019).
\bibitem{cheng2013evaluation}
J.~Cheng, Q.~Xie, V.~Kumar, M.~Hurle, J.~M. Freudenberg, L.~Yang and
P.~Agarwal, Evaluation of analytical methods for connectivity map data (World
Scientific, 2013) pp. 5--16.
\bibitem{lopez2018deep}
R.~Lopez, J.~Regier, M.~B. Cole, M.~I. Jordan and N.~Yosef, Deep generative
modeling for single-cell transcriptomics, {\em Nature methods} {\bf 15}, 1053
(2018).
\bibitem{hsu2018unsupervised}
T.-M.~H. Hsu, W.-H. Weng, W.~Boag, M.~McDermott and P.~Szolovits, Unsupervised
multimodal representation learning across medical images and reports, {\em
arXiv preprint arXiv:1811.08615} (2018).
\bibitem{hassani2020contrastive}
K.~Hassani and A.~H. Khasahmadi, Contrastive multi-view representation learning
on graphs, {\em arXiv preprint arXiv:2006.05582} (2020).
\bibitem{hotelling1992relations}
H.~Hotelling, Relations between two sets of variates (Springer, 1992) pp.
162--190.
\bibitem{li2018survey}
Y.~Li, M.~Yang and Z.~M. Zhang, A survey of multi-view representation learning,
{\em IEEE Transactions on Knowledge and Data Engineering} (2018).
\bibitem{deng2018triplet}
C.~Deng, Z.~Chen, X.~Liu, X.~Gao and D.~Tao, Triplet-based deep hashing network
for cross-modal retrieval, {\em IEEE Transactions on Image Processing} {\bf
27}, 3893 (2018).
\bibitem{koch2015siamese}
G.~Koch, R.~Zemel and R.~Salakhutdinov, Siamese neural networks for one-shot
image recognition, {\em ICML deep learning workshop} {\bf 2} (2015).
\bibitem{filzen2017representing}
T.~M. Filzen, P.~S. Kutchukian, J.~D. Hermes, J.~Li and M.~Tudor, Representing
high throughput expression profiles via perturbation barcodes reveals
compound targets, {\em PLoS computational biology} {\bf 13}, p. e1005335
(2017).
\bibitem{hoffer2015deep}
E.~Hoffer and N.~Ailon, Deep metric learning using triplet network, {\em
International Workshop on Similarity-Based Pattern Recognition} , 84 (2015).
\bibitem{wu2017sampling}
C.-Y. Wu, R.~Manmatha, A.~J. Smola and P.~Krahenbuhl, Sampling matters in deep
embedding learning, {\em Proceedings of the IEEE International Conference on
Computer Vision} , 2840 (2017).
\bibitem{weininger1989smiles}
D.~Weininger, A.~Weininger and J.~L. Weininger, Smiles. 2. algorithm for
generation of unique smiles notation, {\em Journal of chemical information
and computer sciences} {\bf 29}, 97 (1989).
\bibitem{landrum2006rdkit}
G.~Landrum {\em et~al.}, Rdkit: Open-source cheminformatics (2006).
\bibitem{rogers2010extended}
D.~Rogers and M.~Hahn, Extended-connectivity fingerprints, {\em Journal of
chemical information and modeling} {\bf 50}, 742 (2010).
\bibitem{hodos2017cell}
R.~Hodos, P.~Zhang, H.-C. Lee, Q.~Duan, Z.~Wang, N.~R. Clark, A.~Ma’ayan,
F.~Wang, B.~Kidd, J.~Hu {\em et~al.}, Cell-specific prediction and
application of drug-induced gene expression profiles, {\em Pacific Symposium
on Biocomputing} {\bf 23} (2017).
\bibitem{mcdermott2019deep}
M.~McDermott, J.~Wang, W.~N. Zhao, S.~D. Sheridan, P.~Szolovits, I.~Kohane,
S.~J. Haggarty and R.~H. Perlis, Deep learning benchmarks on l1000 gene
expression data, {\em IEEE/ACM transactions on computational biology and
bioinformatics} (2019).
\bibitem{pedregosa2011scikit}
F.~Pedregosa, G.~Varoquaux, A.~Gramfort, V.~Michel, B.~Thirion, O.~Grisel,
M.~Blondel, P.~Prettenhofer, R.~Weiss, V.~Dubourg {\em et~al.}, Scikit-learn:
Machine learning in python, {\em Journal of machine learning research} {\bf
12}, 2825 (2011).
\bibitem{bergstra2013making}
J.~Bergstra, D.~Yamins and D.~D. Cox, Making a science of model search:
Hyperparameter optimization in hundreds of dimensions for vision
architectures, {\em Proceedings of the 30th International Conference on
International Conference on Machine Learning - Volume 28} , p.
I–115–I–123 (2013).
\bibitem{felix2018multi}
R.~Felix, V.~B. Kumar, I.~Reid and G.~Carneiro, Multi-modal cycle-consistent
generalized zero-shot learning, {\em Proceedings of the European Conference
on Computer Vision (ECCV)} , 21 (2018).
\bibitem{mcdermott_semi-supervised_2018}
M.~B.~A. McDermott, T.~Yan, T.~Naumann, N.~Hunt, H.~Suresh, P.~Szolovits and
M.~Ghassemi, Semi-{Supervised} {Biomedical} {Translation} with {Cycle}
{Wasserstein} {Regression} {GANs}, p.~8 (2018).
\bibitem{ying2019gnnexplainer}
Z.~Ying, D.~Bourgeois, J.~You, M.~Zitnik and J.~Leskovec, Gnnexplainer:
Generating explanations for graph neural networks, {\em Advances in neural
information processing systems} , 9244 (2019).
\end{thebibliography}