Molecular De Novo Design through Transformer-based Reinforcement Learning
Extracted source text ยท arXiv 2310.05365
\documentclass[10pt]{article} % For LaTeX2e
\usepackage[preprint]{tmlr}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{tmlr}
% To de-anonymize and remove mentions to TMLR (for example for posting to preprint servers), instead use the following:
%\usepackage[preprint]{tmlr}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{times} % DO NOT CHANGE THIS
\usepackage{helvet} % DO NOT CHANGE THIS
\usepackage{courier} % DO NOT CHANGE THIS
\usepackage[hyphens]{url} % DO NOT CHANGE THIS
\usepackage{graphicx} % DO NOT CHANGE THIS
\usepackage{natbib} % DO NOT CHANGE THIS AND DO NOT ADD ANY OPTIONS TO IT
\usepackage{caption} % DO NOT CHANGE THIS AND DO NOT ADD ANY OPTIONS TO IT
\usepackage{float}
\restylefloat{table}
\usepackage{tabularx}
%
% These are recommended to typeset algorithms but not required. See the subsubsection on algorithms. Remove them if you don't have algorithms in your paper.
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{hyperref}
\usepackage{url}
\usepackage{enumitem}
\usepackage{xspace}
\usepackage{xcolor}
\usepackage{tabularx}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{makecell}
\setcounter{secnumdepth}{0} %May be changed to 1 or 2 if section numbers are desired.
% \newcommand{\method}{KGPrompt} % can be changed later
\newcommand{\method}{\texttt{CurricuLuM}\xspace} % can be changed later
\newcommand{\para}[1]{{\vspace{3pt} \bf \noindent #1 \hspace{0pt}}}
\title{Molecular De Novo Design through Transformer-based Reinforcement Learning}
% Authors must not appear in the submitted version. They should be hidden
% as long as the tmlr package is used without the [accepted] or [preprint] options.
% Non-anonymous submissions will be rejected without review.
% \author{\name Tao Feng \email ft19@tsinghua.org.cn \\
% \addr Department of Electrical and Electronic Engineering\\
% Tsinghua University
% \AND
% \name Pengcheng Xu \email px6@illinois \\
% \addr Department of Electrical and Computer Engineering
% University of Illinois Urbana-Champaign
% \AND
% \name Tianfan Fu \email futianfan@gmail.com \\
% \addr University of Illinois Urbana-Champaign\\
% \\
% \AND
% \name Siddhartha Laghuvarapu \email sl160@illinois.edu\\
% \addr University of Illinois Urbana-Champaign\\
% \\
% \AND
% \name Jimeng Sun \email jimeng@illinois.edu\\
% \addr University of Illinois Urbana-Champaign\\
% \\
% }
\author{\name Tao Feng \footnotemark[1] \email ft19@tsinghua.org.cn\\
\addr Department of Electrical and Electronic Engineering\\
Tsinghua University
\AND
\name Pengcheng Xu \footnotemark[1] \email px6@illinois.edu \\
\addr Department of Electrical and Computer Engineering\\
University of Illinois Urbana-Champaign
\AND
\name Tianfan Fu \email futianfan@gmail.com\\
\addr University of Illinois Urbana-Champaign
\\
\AND
\name Siddhartha Laghuvarapu \email sl160@illinois.edu\\
\addr University of Illinois Urbana-Champaign \\
\AND
\name Jimeng Sun \email jimeng@illinois.edu \\
\addr University of Illinois Urbana-Champaign
\footnotetext[1]{These authors contributed equally to this work.}
}
% The \author macro works with any number of authors. Use \AND
% to separate the names and addresses of multiple authors.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\def\month{MM} % Insert correct month for camera-ready version
\def\year{YYYY} % Insert correct year for camera-ready version
\def\openreview{\url{https://openreview.net/forum?id=XXXX}} % Insert correct link to OpenReview for camera-ready version
% REMOVE THIS: bibentry
% This is only needed to show inline citations in the guidelines document. You should not need it and can safely delete it.
\usepackage{bibentry}
% END REMOVE bibentry
\begin{document}
\maketitle
\begin{abstract}
In this work, we introduce a method to fine-tune a Transformer-based generative model for molecular de novo design. Leveraging the superior sequence learning capacity of Transformers over Recurrent Neural Networks (RNNs), our model can generate molecular structures with desired properties effectively. In contrast to the traditional RNN-based models, our proposed method exhibits superior performance in generating compounds predicted to be active against various biological targets, capturing long-term dependencies in the molecular structure sequence. The model's efficacy is demonstrated across numerous tasks, including generating analogues to a query structure and producing compounds with particular attributes, outperforming the baseline RNN-based methods. Our approach can be used for scaffold hopping, library expansion starting from a single molecule, and generating compounds with high predicted activity against biological targets.
\end{abstract}
\input{Introduction}
\input{Related_Work}
\input{Methodology}
\input{Experiment}
\input{Conclusion}
\newpage
% \subsubsection{Acknowledgments.}
% \appendix
\newpage
\appendix
\newpage
\bibliography{main}
\bibliographystyle{tmlr}
% \section{Submission of papers to TMLR}
% TMLR requires electronic submissions, processed by
% \url{https://openreview.net/}. See TMLR's website for more instructions.
% If your paper is ultimately accepted, use option {\tt accepted} with the {\tt tmlr}
% package to adjust the format to the camera ready requirements, as follows:
% \begin{center}
% {\tt {\textbackslash}usepackage[accepted]\{tmlr\}}.
% \end{center}
% You also need to specify the month and year
% by defining variables {\tt month} and {\tt year}, which respectively
% should be a 2-digit and 4-digit number. To de-anonymize and remove mentions
% to TMLR (for example for posting to preprint servers), use the preprint option,
% as in {\tt {\textbackslash}usepackage[preprint]\{tmlr\}}.
% Please read carefully the instructions below, and follow them
% faithfully.
% \subsection{Style}
% Papers to be submitted to TMLR must be prepared according to the
% instructions presented here.
% Authors are required to use the TMLR \LaTeX{} style files obtainable at the
% TMLR website. Please make sure you use the current files and
% not previous versions. Tweaking the style files may be grounds for rejection.
% \subsection{Retrieval of style files}
% The style files for TMLR and other journal information are available online on
% the TMLR website.
% The file \verb+tmlr.pdf+ contains these
% instructions and illustrates the
% various formatting requirements your TMLR paper must satisfy.
% Submissions must be made using \LaTeX{} and the style files
% \verb+tmlr.sty+ and \verb+tmlr.bst+ (to be used with \LaTeX{}2e). The file
% \verb+tmlr.tex+ may be used as a ``shell'' for writing your paper. All you
% have to do is replace the author, title, abstract, and text of the paper with
% your own.
% The formatting instructions contained in these style files are summarized in
% sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.
% \section{General formatting instructions}
% \label{gen_inst}
% The text must be confined within a rectangle 6.5~inches wide and
% 9~inches long. The left margin is 1~inch.
% Use 10~point type with a vertical spacing of 11~points. Computer Modern Bright is the
% preferred typeface throughout. Paragraphs are separated by 1/2~line space,
% with no indentation.
% Paper title is 17~point, in bold and left-aligned.
% All pages should start at 1~inch from the top of the page.
% Authors' names are
% set in boldface. Each name is placed above its corresponding
% address and has its corresponding email contact on the same line, in italic
% and right aligned. The lead author's name is to be listed first, and
% the co-authors' names are set to follow vertically.
% Please pay special attention to the instructions in section \ref{others}
% regarding figures, tables, acknowledgments, and references.
% \section{Headings: first level}
% \label{headings}
% First level headings are in bold,
% flush left and in point size 12. One line space before the first level
% heading and 1/2~line space after the first level heading.
% \subsection{Headings: second level}
% Second level headings are in bold,
% flush left and in point size 10. One line space before the second level
% heading and 1/2~line space after the second level heading.
% \subsubsection{Headings: third level}
% Third level headings are in bold,
% flush left and in point size 10. One line space before the third level
% heading and 1/2~line space after the third level heading.
% \section{Citations, figures, tables, references}
% \label{others}
% These instructions apply to everyone, regardless of the formatter being used.
% \subsection{Citations within the text}
% Citations within the text should be based on the \texttt{natbib} package
% and include the authors' last names and year (with the ``et~al.'' construct
% for more than two authors). When the authors or the publication are
% included in the sentence, the citation should not be in parenthesis, using \verb|\citet{}| (as
% in ``See \citet{Hinton06} for more information.''). Otherwise, the citation
% should be in parenthesis using \verb|\citep{}| (as in ``Deep learning shows promise to make progress
% towards AI~\citep{Bengio+chapter2007}.'').
% The corresponding references are to be listed in alphabetical order of
% authors, in the \textbf{References} section. As to the format of the
% references themselves, any style is acceptable as long as it is used
% consistently.
% \subsection{Footnotes}
% Indicate footnotes with a number\footnote{Sample of the first footnote} in the
% text. Place the footnotes at the bottom of the page on which they appear.
% Precede the footnote with a horizontal rule of 2~inches.\footnote{Sample of the second footnote}
% \subsection{Figures}
% All artwork must be neat, clean, and legible. Lines should be dark
% enough for purposes of reproduction; art work should not be
% hand-drawn. The figure number and caption always appear after the
% figure. Place one line space before the figure caption, and one line
% space after the figure. The figure caption is lower case (except for
% first word and proper nouns); figures are numbered consecutively.
% Make sure the figure caption does not get separated from the figure.
% Leave sufficient space to avoid splitting the figure and figure caption.
% You may use color figures.
% However, it is best for the
% figure captions and the paper body to make sense if the paper is printed
% either in black/white or in color.
% \begin{figure}[h]
% \begin{center}
% %\framebox[4.0in]{$\;$}
% \fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
% \end{center}
% \caption{Sample figure caption.}
% \end{figure}
% \subsection{Tables}
% All tables must be centered, neat, clean and legible. Do not use hand-drawn
% tables. The table number and title always appear before the table. See
% Table~\ref{sample-table}. Place one line space before the table title, one line space after the table
% title, and one line space after the table. The table title must be lower case
% (except for first word and proper nouns); tables are numbered consecutively.
% \begin{table}[t]
% \caption{Sample table title}
% \label{sample-table}
% \begin{center}
% \begin{tabular}{ll}
% \multicolumn{1}{c}{\bf PART} &\multicolumn{1}{c}{\bf DESCRIPTION}
% \\ \hline \\
% Dendrite &Input terminal \\
% Axon &Output terminal \\
% Soma &Cell body (contains cell nucleus) \\
% \end{tabular}
% \end{center}
% \end{table}
% \section{Default Notation}
% In an attempt to encourage standardized notation, we have included the
% notation file from the textbook, \textit{Deep Learning}
% \citep{goodfellow2016deep} available at
% \url{https://github.com/goodfeli/dlbook_notation/}. Use of this style
% is not required and can be disabled by commenting out
% \texttt{math\_commands.tex}.
% \centerline{\bf Numbers and Arrays}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1in}p{3.25in}}
% $\displaystyle a$ & A scalar (integer or real)\\
% $\displaystyle \va$ & A vector\\
% $\displaystyle \mA$ & A matrix\\
% $\displaystyle \tA$ & A tensor\\
% $\displaystyle \mI_n$ & Identity matrix with $n$ rows and $n$ columns\\
% $\displaystyle \mI$ & Identity matrix with dimensionality implied by context\\
% $\displaystyle \ve^{(i)}$ & Standard basis vector $[0,\dots,0,1,0,\dots,0]$ with a 1 at position $i$\\
% $\displaystyle \text{diag}(\va)$ & A square, diagonal matrix with diagonal entries given by $\va$\\
% $\displaystyle \ra$ & A scalar random variable\\
% $\displaystyle \rva$ & A vector-valued random variable\\
% $\displaystyle \rmA$ & A matrix-valued random variable\\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Sets and Graphs}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle \sA$ & A set\\
% $\displaystyle \R$ & The set of real numbers \\
% $\displaystyle \{0, 1\}$ & The set containing 0 and 1 \\
% $\displaystyle \{0, 1, \dots, n \}$ & The set of all integers between $0$ and $n$\\
% $\displaystyle [a, b]$ & The real interval including $a$ and $b$\\
% $\displaystyle (a, b]$ & The real interval excluding $a$ but including $b$\\
% $\displaystyle \sA \backslash \sB$ & Set subtraction, i.e., the set containing the elements of $\sA$ that are not in $\sB$\\
% $\displaystyle \gG$ & A graph\\
% $\displaystyle \parents_\gG(\ervx_i)$ & The parents of $\ervx_i$ in $\gG$
% \end{tabular}
% \vspace{0.25cm}
% \centerline{\bf Indexing}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle \eva_i$ & Element $i$ of vector $\va$, with indexing starting at 1 \\
% $\displaystyle \eva_{-i}$ & All elements of vector $\va$ except for element $i$ \\
% $\displaystyle \emA_{i,j}$ & Element $i, j$ of matrix $\mA$ \\
% $\displaystyle \mA_{i, :}$ & Row $i$ of matrix $\mA$ \\
% $\displaystyle \mA_{:, i}$ & Column $i$ of matrix $\mA$ \\
% $\displaystyle \etA_{i, j, k}$ & Element $(i, j, k)$ of a 3-D tensor $\tA$\\
% $\displaystyle \tA_{:, :, i}$ & 2-D slice of a 3-D tensor\\
% $\displaystyle \erva_i$ & Element $i$ of the random vector $\rva$ \\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Calculus}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% % NOTE: the [2ex] on the next line adds extra height to that row of the table.
% % Without that command, the fraction on the first line is too tall and collides
% % with the fraction on the second line.
% $\displaystyle\frac{d y} {d x}$ & Derivative of $y$ with respect to $x$\\ [2ex]
% $\displaystyle \frac{\partial y} {\partial x} $ & Partial derivative of $y$ with respect to $x$ \\
% $\displaystyle \nabla_\vx y $ & Gradient of $y$ with respect to $\vx$ \\
% $\displaystyle \nabla_\mX y $ & Matrix derivatives of $y$ with respect to $\mX$ \\
% $\displaystyle \nabla_\tX y $ & Tensor containing derivatives of $y$ with respect to $\tX$ \\
% $\displaystyle \frac{\partial f}{\partial \vx} $ & Jacobian matrix $\mJ \in \R^{m\times n}$ of $f: \R^n \rightarrow \R^m$\\
% $\displaystyle \nabla_\vx^2 f(\vx)\text{ or }\mH( f)(\vx)$ & The Hessian matrix of $f$ at input point $\vx$\\
% $\displaystyle \int f(\vx) d\vx $ & Definite integral over the entire domain of $\vx$ \\
% $\displaystyle \int_\sS f(\vx) d\vx$ & Definite integral with respect to $\vx$ over the set $\sS$ \\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Probability and Information Theory}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle P(\ra)$ & A probability distribution over a discrete variable\\
% $\displaystyle p(\ra)$ & A probability distribution over a continuous variable, or over
% a variable whose type has not been specified\\
% $\displaystyle \ra \sim P$ & Random variable $\ra$ has distribution $P$\\% so thing on left of \sim should always be a random variable, with name beginning with \r
% $\displaystyle \E_{\rx\sim P} [ f(x) ]\text{ or } \E f(x)$ & Expectation of $f(x)$ with respect to $P(\rx)$ \\
% $\displaystyle \Var(f(x)) $ & Variance of $f(x)$ under $P(\rx)$ \\
% $\displaystyle \Cov(f(x),g(x)) $ & Covariance of $f(x)$ and $g(x)$ under $P(\rx)$\\
% $\displaystyle H(\rx) $ & Shannon entropy of the random variable $\rx$\\
% $\displaystyle \KL ( P \Vert Q ) $ & Kullback-Leibler divergence of P and Q \\
% $\displaystyle \mathcal{N} ( \vx ; \vmu , \mSigma)$ & Gaussian distribution %
% over $\vx$ with mean $\vmu$ and covariance $\mSigma$ \\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Functions}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle f: \sA \rightarrow \sB$ & The function $f$ with domain $\sA$ and range $\sB$\\
% $\displaystyle f \circ g $ & Composition of the functions $f$ and $g$ \\
% $\displaystyle f(\vx ; \vtheta) $ & A function of $\vx$ parametrized by $\vtheta$.
% (Sometimes we write $f(\vx)$ and omit the argument $\vtheta$ to lighten notation) \\
% $\displaystyle \log x$ & Natural logarithm of $x$ \\
% $\displaystyle \sigma(x)$ & Logistic sigmoid, $\displaystyle \frac{1} {1 + \exp(-x)}$ \\
% $\displaystyle \zeta(x)$ & Softplus, $\log(1 + \exp(x))$ \\
% $\displaystyle || \vx ||_p $ & $\normlp$ norm of $\vx$ \\
% $\displaystyle || \vx || $ & $\normltwo$ norm of $\vx$ \\
% $\displaystyle x^+$ & Positive part of $x$, i.e., $\max(0,x)$\\
% $\displaystyle \1_\mathrm{condition}$ & is 1 if the condition is true, 0 otherwise\\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \section{Final instructions}
% Do not change any aspects of the formatting parameters in the style files.
% In particular, do not modify the width or length of the rectangle the text
% should fit into, and do not change font sizes (except perhaps in the
% \textbf{References} section; see below). Please note that pages should be
% numbered.
% \section{Preparing PostScript or PDF files}
% Please prepare PostScript or PDF files with paper size ``US Letter'', and
% not, for example, ``A4''. The -t
% letter option on dvips will produce US Letter files.
% Consider directly generating PDF files using \verb+pdflatex+
% (especially if you are a MiKTeX user).
% PDF figures must be substituted for EPS figures, however.
% Otherwise, please generate your PostScript and PDF files with the following commands:
% \begin{verbatim}
% dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps
% ps2pdf mypaper.ps mypaper.pdf
% \end{verbatim}
% \subsection{Margins in LaTeX}
% Most of the margin problems come from figures positioned by hand using
% \verb+\special+ or other commands. We suggest using the command
% \verb+\includegraphics+
% from the graphicx package. Always specify the figure width as a multiple of
% the line width as in the example below using .eps graphics
% \begin{verbatim}
% \usepackage[dvips]{graphicx} ...
% \includegraphics[width=0.8\linewidth]{myfile.eps}
% \end{verbatim}
% or % Apr 2009 addition
% \begin{verbatim}
% \usepackage[pdftex]{graphicx} ...
% \includegraphics[width=0.8\linewidth]{myfile.pdf}
% \end{verbatim}
% for .pdf graphics.
% See section~4.4 in the graphics bundle documentation (\url{http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps})
% A number of width problems arise when LaTeX cannot properly hyphenate a
% line. Please give LaTeX hyphenation hints using the \verb+\-+ command.
% \subsubsection*{Broader Impact Statement}
% In this optional section, TMLR encourages authors to discuss possible repercussions of their work,
% notably any potential negative impact that a user of this research should be aware of.
% Authors should consult the TMLR Ethics Guidelines available on the TMLR website
% for guidance on how to approach this subject.
% \subsubsection*{Author Contributions}
% If you'd like to, you may include a section for author contributions as is done
% in many journals. This is optional and at the discretion of the authors. Only add
% this information once your submission is accepted and deanonymized.
% \subsubsection*{Acknowledgments}
% Use unnumbered third level headings for the acknowledgments. All
% acknowledgments, including those to funding agencies, go at the end of the paper.
% Only add this information once your submission is accepted and deanonymized.
% \bibliography{main}
% \bibliographystyle{tmlr}
% \appendix
% \section{Appendix}
% You may include other additional sections here.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\section{Introduction}
% The enormous complexity and scale of chemical space, on the order of $10^{60}-10^{100}$ synthetically feasible molecules \citep{em:86}, pose significant challenges for drug discovery. Among these molecules, finding an ideal candidate that satisfies criteria such as bioactivity, drug metabolism and pharmacokinetic (DMPK) profile, and synthetic accessibility, is akin to finding a needle in a haystack. De novo design algorithms \citep{c:83} attempt to mitigate these challenges by virtually designing and assessing molecules to minimize the chemical space for efficient exploration.
% RNN-based generative models have been employed successfully in the de novo design of molecules \citep{c:84,r:80}, but they come with limitations. For instance, they may struggle to capture long-term dependencies in the sequence data. This limitation can be detrimental when dealing with complex molecular structures. To overcome this, we propose the use of Transformer-based architectures, known for their superior capability to capture long sequence features, for molecular de novo design.
The vast expanse of chemical space, encompassing an order of magnitude from \(10^{60}-10^{100}\) possible synthetically feasible molecules \citep{em:86}, presents formidable obstacles to drug discovery endeavors. In this colossal landscape, the task of pinpointing a molecule that simultaneously meets the prerequisites for bioactivity, drug metabolism and pharmacokinetic (DMPK) profile, and synthetic accessibility becomes an undertaking similar to the proverbial search for a needle in a haystack. Pioneering de novo design algorithms \citep{c:83} have attempted to address this by employing virtual strategies to design and evaluate molecules, thereby condensing the vast chemical space into a more navigable realm for exploration.
Traditional de novo design models, based on Recurrent Neural Networks (RNNs), have proven effective in molecule generation tasks \citep{c:84,r:80}. However, RNNs possess inherent architectural limitations, notably in their capability to capture long-term dependencies in sequential data, which can be particularly detrimental when modeling complex molecular structures. Recently, the Transformer architecture has emerged as a powerful alternative to RNNs in sequence modeling tasks across various domains. Some of the key advantages of Transformers over RNNs include:
\begin{enumerate}
\item \textbf{Parallelization:} Unlike RNNs which process sequences step-by-step, Transformers process all tokens in the sequence simultaneously, allowing for better computational efficiency.
\item \textbf{Long-term Dependency Handling:} Transformers utilize multi-head self-attention mechanisms which can capture long-range interactions in the data, making them particularly well-suited for modeling intricate molecular structures.
\item \textbf{Scalability:} Transformers are inherently more scalable, allowing for the processing of longer sequences which is a considerable advantage in molecular design.
\end{enumerate}
In light of these advantages, our work introduces a novel approach by integrating the Transformer architecture, specifically the Decision Transformer, for molecular de novo design. By leveraging the inherent strengths of Transformers, our model exhibits enhanced performance in generating molecular structures with desired attributes.
Furthermore, we emphasize the incorporation of the "oracle feedback reinforcement learning" method. Pretraining models on large datasets is beneficial, but downstream tasks often require fine-tuning on specific objectives. By integrating feedback from an oracle during the reinforcement learning phase, our approach can efficiently navigate the solution space, optimizing towards molecules with high predicted activity. Such oracle-guided optimization provides an added layer of precision, facilitating the generation of molecules that not only conform to structural constraints but also exhibit high bioactivity, thereby increasing the potential success rate in drug discovery endeavors.
Drawing inspiration from previous work that employed RNNs and reinforcement learning for molecular optimization \citep{c:83}, our approach distinguishes itself by the adoption and fine-tuning of the Transformer architecture, ensuring superior handling of long-sequence data and paving the way for innovative breakthroughs in the realm of molecular design.
In summary, this work presents a fresh perspective on molecular de novo design, underscoring the potential of Transformer-based architectures, complemented by oracle feedback reinforcement learning, to revolutionize drug discovery methodologies. We envision that our approach will not only set a new benchmark in molecular generation tasks but will also inspire future research in leveraging advanced machine learning architectures for complex scientific challenges.
\section{Related Works}
% Early de novo design algorithms were structure-based, aiming to grow ligands to fit the binding pocket of the target \citep{r:80x,hcr:83}. However, these methods often generated molecules with poor DMPK properties and could be synthetically intractable. Ligand-based approaches were introduced to create a vast virtual library of chemical structures and then searched with a scoring function \citep{hcrt:83,r:86}.
% Recently, generative models such as RNN-based methods have been used for de novo design of molecules \citep{c:79,c:21}. They have shown success in tasks like learning the underlying probability distribution over a large set of chemical structures, reducing the search over chemical space to only molecules seen as reasonable. Further fine-tuning of the models was done using reinforcement learning (RL) \citep{c:22}, which showed considerable improvement over the initial model.
% Despite these advancements, challenges such as capturing long-term dependencies in the sequence data persist. The Transformer architecture \citep{c:23}, known for its self-attention mechanism and ability to handle long sequences, has been highly successful in several sequence prediction tasks across domains. Motivated by these successes, we propose the use of Transformer-based architectures in place of RNNs for molecular de novo design.
Early de novo design algorithms were structure-based, aiming to grow ligands to fit the binding pocket of the target \citep{r:80x,hcr:83}. However, these methods often generated molecules with poor DMPK properties and could be synthetically intractable. Ligand-based approaches were introduced to create a vast virtual library of chemical structures and then searched with a scoring function \citep{hcrt:83,r:86}.
Recently, generative models such as RNN-based methods have been used for de novo design of molecules \citep{c:79,c:21}. They have shown success in tasks like learning the underlying probability distribution over a large set of chemical structures, reducing the search over chemical space to only molecules seen as reasonable. Further fine-tuning of the models was done using reinforcement learning (RL) \citep{c:22}, which showed considerable improvement over the initial model.
Despite these advancements, challenges such as capturing long-term dependencies in the sequence data persist. The Transformer architecture \citep{c:23}, known for its self-attention mechanism and ability to handle long sequences, has been highly successful in several sequence prediction tasks across domains. Motivated by these successes, we propose the use of Transformer-based architectures in place of RNNs for molecular de novo design.
Molecular assembly strategies, such as string-based approaches like SMILES and SELFIES \citep{weininger1988smiles,krenn2020self}, provide an efficient representation of molecules. Graph-based methods offer an intuitive two-dimensional representation of molecular structures, with nodes and edges representing atoms and bonds, respectively \citep{zhou2019optimization,jin2018junction}. Synthesis-based strategies, on the other hand, aim to generate only synthesizable molecules, ensuring that the design aligns with real-world applications \citep{bradshaw2020barking,gao2021amortized}.
Various optimization algorithms have been utilized for molecular design. Genetic Algorithms (GAs) mimic natural evolutionary processes and have been applied in the context of molecule generation using both SMILES and SELFIES representations \citep{brown2019guacamol,nigam2021beyond}. Bayesian optimization (BO) is another class of method that builds a surrogate for the objective function, with applications such as BOSS and ChemBO in the molecular domain \citep{moss2020boss,korovina2020chembo}. Variational autoencoders (VAEs) offer a generative approach, mapping molecules to and from a latent space, with notable methods including SMILES-VAE and JT-VAE \citep{gomez2018automatic,jin2018junction}. Reinforcement Learning (RL) techniques, like REINVENT, have also been applied to tune models for molecule generation \cite{olivecrona2017molecular}.
Furthermore, recent advancements in gradient ascent methods, such as Pasithea and Differentiable scaffolding tree (DST), have leveraged gradient-based optimization for molecular design \citep{shen2021deep,fu2021differentiable}.
In light of these developments, our approach integrates the benefits of the Transformer architecture with advanced reinforcement learning techniques, aiming to address the challenges present in current molecular de novo design methodologies.
\section{Methodology}
\begin{figure*}[t]
\centering
\includegraphics*[width=0.9\textwidth]{fig/reinvent.pdf}
\caption{The framework of our method.} \label{fig:framework}
\end{figure*}
Our method first pre-trains the real 2D molecule dataset based on the transformer. Then, based on the RL paradigm, fine-tuning is performed on the molecular attributes to be optimized.
\subsection{Transformer-based Molecular Pre-training}
% In this part, we train a graph neural network to fit a certain feature of the molecule. Specifically, we design our graph neural network based on EGNN. The core idea behind EGNN is to apply a series of learnable transformations to the node features in a graph, while ensuring that these transformations respect the symmetries of the data. By doing so, EGNN can capture and exploit the inherent rotational symmetries in the graph structure, leading to improved performance in tasks such as molecular property prediction, 3D shape analysis, and quantum chemistry.
The transformer is used for pre-training on real 2D molecules. Specifically, it treats the prediction of a 2D molecule as a sequence prediction and lets the transformer predict the next atom based on the molecular sequence history. The pre-training of the transformer is based on maximum likelihood.
\para{Transformers Overview}
Transformers are a neural network architecture designed to process sequential data, while also accounting for the importance of each input in relation to the others, despite their position in the sequence \citep{goodfellow2016deep}. They manage to do this by the introduction of an attention mechanism that assesses the significance of each input in the sequence (Figure 1). At any given step $t$, the transformer state at $t$ is influenced by all previous inputs $x^{1}, \ldots, x^{t-1}$ and the current input $x^{t}$. The transformer's ability to selectively focus on the parts of the input sequence that are most relevant for each step makes them especially well suited for tasks in the field of natural language processing. Sequences of words can be encoded into one-hot vectors with a length equivalent to our vocabulary size $X$. We may add two extra tokens, GO and EOS, to signify the beginning and end of a sequence, respectively.
\para{Learning the data}
Training a Transformer for sequence modeling typically involves using maximum likelihood estimation to predict the next token $x^{t}$ in the target sequence, given tokens from the previous steps (Figure 1). The model generates a probability distribution at every step, representing the likely next character, and the objective is to maximize the likelihood assigned to the correct token:
\begin{equation}
J(\Theta)=-\sum_{t=1}^T \log P\left(x^t \mid x^{t-1}, \ldots, x^1\right)
\end{equation}
The cost function $J(\Theta)$, often applied to a subset of all training examples known as a batch, is minimized with respect to the network parameters $\Theta$. Given a predicted $\log$ likelihood $\log P$ of the target at step $t$, the gradient
% \begin{center}
% \includegraphics[max width=\textwidth]{2023_07_16_14ca0e312c018dfb2653g-03(1)}
% \end{center}
of the prediction with respect to $\Theta$ is used to update $\Theta$. This method of fitting a neural network is called back-propagation. Changing the network parameters affects not only the immediate output at time $t$, but also influences the information flow into subsequent transformer states. This effect does not lead to problems of exploding or vanishing gradients due to the lack of recurrent connections in Transformers.
\para{Generating new samples}
Once a Transformer has been trained on target sequences, it can be used to generate new sequences that adhere to the conditional probability distributions learned from the training set. The first input is the GO token, and at every timestep following, we sample an output token $x^{t}$ from the predicted probability distribution $P\left(X^{t}\right)$ over our vocabulary $X$. The sampled $x^{t}$ is then used as our next input. The sequence is considered finished once the EOS token is sampled .%(Figure 2).
\para{Segmentation and Binary Coding of SMILES}
A Simplified Molecular Input Line Entry System (SMILES) \citep{SMILES} defines a molecule as a character sequence reflecting atoms as well as special symbols that illustrate ring opening and closure along with branching. In the majority of scenarios, SMILES are tokenized
% \begin{center}
% \includegraphics[max width=\textwidth]{2023_07_16_14ca0e312c018dfb2653g-03}
% \end{center}
% Figure 2. Sequence creation. Trained Transformer facilitates sequence generation. At each time-step $t$, the subsequent sequence token $x^{t}$ is derived from the Transformer-provided probability distribution, which is then utilized as the subsequent input.
on a single-character basis, with the exception of two-character atom types such as " $\mathrm{Cl}$ " and "Br", and unique environments indicated by square brackets (e.g., $[\mathrm{nH}])$, where they are processed as a single token. This approach to tokenization led to the identification of 86 tokens in the training data. Figure 3 illustrates how a chemical structure is converted to SMILES and binary-coded representations.
A single molecule can be represented in multiple ways using SMILES. Algorithms that consistently represent a particular molecule with the same SMILES are termed canonicalization algorithms~\citep{weininger1988smiles}. Nevertheless, different algorithm implementations may still yield diverse SMILES.
% \begin{center}
% \includegraphics[max width=\textwidth]{2023_07_16_14ca0e312c018dfb2653g-03(2)}
% \end{center}
% Figure 3. Three representations of 4-(chloromethyl)-1H-imidazole. A one-hot representation derived from a molecule's SMILES. In this case, a reduced vocabulary is presented, but in actuality, a much larger vocabulary encompassing all tokens in the training data is employed.
\subsection{Molecular Attribute Fine-tuning through Reinforcement Learning}
In this part, we load the pre-trained transformer network and fine-tune it based on RL. Here, our task is to generate some specific molecules with good attributes. Therefore, we use the generated molecules to measure the properties of the corresponding molecules through Oracle, and use them as rewards to finetune the neural network.
Assume an Agent that must decide on an action $a \in \mathbb{A}(s)$ to take given a particular state $s \in \mathbb{S}$, where $\mathbb{S}$ denotes the set of possible states and $\mathbb{A}(s)$ represents the set of potential actions for that state. The policy $\pi(a \mid s)$ of an Agent associates a state to the likelihood of each action executed within. Reinforcement learning challenges are often depicted as Markov decision processes, indicating that the current state provides all essential information to inform our action choice, and no additional benefit is gained from knowing past states' history. While this is more of an approximation than a fact for most real-life challenges, we can extend this concept to a partially observable Markov decision process where the Agent interacts with a partial environment representation. Let $r(a \mid s)$ be the reward serving as an indicator of the effectiveness of an action taken at a certain state, and the long-term return $G\left(a_{t}, S_{t}\right)=\sum_{t}^{T} r_{t}.$ represents the cumulative rewards collected from time $t$ to time $T$. As molecular desirability is only meaningful for a completed SMILES, we will only consider a complete sequence's return.
The main objective of reinforcement learning is to enhance the Agent's policy to increase the expected return $\mathbb{E}[G]$ based on a set of actions taken from some states and the obtained rewards. A task with a definitive endpoint at step $T$ is known as an episodic task \citep{sutton1999reinforcement}, where $T$ corresponds to the episode's length. SMILES generation is an example of an episodic task, which concludes once the EOS token is sampled.
The states and actions used for Agent training can be produced by the agent itself or through other means. If the agent generates them, the learning is called on-policy, and if generated by other means, it is off-policy learning \citep{sutton1999reinforcement}.
Reinforcement learning commonly employs two different strategies to determine a policy: value-based $\mathrm{RL}$ and policy-based RL \citep{sutton1999reinforcement}. In value-based RL, the aim is to learn a value function that describes a given state's expected return. Once this function is learned, a policy can be established to maximize a certain action's expected state value. In contrast, policy-based RL aims to learn a policy directly. For the problem we are addressing, we believe policy-based methods are the most suitable for the following reasons:
\begin{itemize}
\item Policy-based methods can explicitly learn an optimal stochastic policy \citep{sutton1999reinforcement}, which aligns with our objective.
\item The used method starts with a prior sequence model. The goal is to fine-tune this model based on a specific scoring function. Since the prior model already embodies a policy, fine-tuning might require only minimal changes to the prior model. The short and fast-sampling episodes in this case decrease the gradient estimate's variance impact.
\end{itemize}
\section{Experiment}
\subsection{Dataset}
% The Alpaca dataset \footnote{https://huggingface.co/datasets/yahma/alpaca-cleaned.} captures the rich and nuanced nature of human language, providing a wide range of linguistic patterns, including formal writing, informal conversations, technical jargon, and colloquial expressions. It covers different domains such as news articles, social media posts, forum discussions, and more, reflecting the diverse ways people communicate and express themselves online.
For any method that necessitates a database, we exclusively use the ZINC 250K dataset~\footnote{https://www.kaggle.com/datasets/basu369victor/zinc250k}. This dataset comprises approximately 250K molecules, selected from the ZINC database due to their pharmaceutical significance, manageable size, and widespread recognition. Both Screening and MolPAL conduct searches within this database. Additionally, generative models like VAEs and LSTMs are pretrained on it. Any fragments essential for JT-VAE~\citep{jin2018junction}, MIMOSA~\citep{fu2021mimosa}, and DST~\citep{fu2021differentiable} are also derived from this very database.
\subsection{Baseline}
To make a comprehensive comparison,
eight baseline methods are adopted in performance evaluation.
First, we compare two rule-based baselines, including:
\begin{itemize}
\item \textbf{REINVENT~\citep{olivecrona2017molecular}.}
\begin{itemize}
\item \textit{Overview:} A method that employs a policy-based reinforcement learning approach to instruct RNNs to produce SMILES strings.
\item \textit{Technical Details:} It formulates the molecular design as a Markov decision process (MDP), where states represent partially generated molecules, and actions correspond to string-based manipulations. The rewards arise from properties of interest in the generated molecules.
\item \textit{Advantage:} Adaptable and can be modified to generate other string representations, such as SELFIES.
\item \textit{Disadvantage:} Relies heavily on the definition and design of rewards.
\end{itemize}
\item \textbf{Graph-GA~\citep{jensen2019graph}.}
\begin{itemize}
\item \textit{Overview:} A genetic algorithm that manipulates molecular representations using graphs. It integrates crossover operations derived from graph matching and comprises both atom- and fragment-level mutations.
\item \textit{Technical Details:} Unlike string-based genetic algorithms, which primarily involve mutation steps, Graph-GA introduces crossover operations based on graph representations.
\item \textit{Advantage:} Offers a richer set of operations due to its graph-based nature, potentially exploring more diverse chemical spaces.
\item \textit{Disadvantage:} Increased complexity due to the need for graph-based operations.
\end{itemize}
\item \textbf{SELFIES-REINVENT.}
\begin{itemize}
\item \textit{Overview:} An extension of REINVENT to generate SELF-referencing Embedded Strings (SELFIES).
\item \textit{Technical Details:} Like REINVENT, it utilizes a policy-based RL approach but specifically for the SELFIES representation, ensuring syntactical validity.
\item \textit{Advantage:} Can produce molecules with fewer syntactical errors due to the nature of SELFIES.
\item \textit{Disadvantage:} Still dependent on the definition of the reward system.
\end{itemize}
\item \textbf{GP BO~\citep{tripp2021fresh}.}
\begin{itemize}
\item \textit{Overview:} Incorporates Gaussian process Bayesian optimization, combining surrogate GP models with Graph-GA methods in inner loops.
\item \textit{Technical Details:} While BO traditionally employs non-parametric models, GP BO leverages the GP acquisition function and integrates Graph-GA techniques for sampling.
\item \textit{Advantage:} Balances exploration and exploitation by combining Bayesian optimization with genetic algorithms.
\item \textit{Disadvantage:} The interplay between GP and GA might lead to higher computational costs.
\end{itemize}
\item \textbf{STONED~\citep{nigam2021beyond}.}
\begin{itemize}
\item \textit{Overview:} A modified genetic algorithm that manipulates tokens within the SELFIES strings representation.
\item \textit{Technical Details:} Unlike traditional string-based GAs, STONED directly interacts with the tokens in the SELFIES strings.
\item \textit{Advantage:} A more direct approach that can potentially reduce invalid chemical representations.
\item \textit{Disadvantage:} Limited to SELFIES, may not generalize to other representations.
\end{itemize}
\item \textbf{SMILES-LSTM HC~\citep{brown2019guacamol}.}
\begin{itemize}
\item \textit{Overview:} An iterative learning method leveraging LSTM to comprehend the molecular distribution represented in SMILES strings.
\item \textit{Technical Details:} Uses a variant of the cross-entropy method and integrates generated high-scoring molecules into training data, subsequently fine-tuning the model.
\item \textit{Advantage:} The iterative approach refines the generative process at each step.
\item \textit{Disadvantage:} Convergence might be slow if the initial model is far from optimal.
\end{itemize}
\item \textbf{SMILES-GA~\citep{brown2019guacamol}.}
\begin{itemize}
\item \textit{Overview:} Genetic algorithm that defines actions based on SMILES context-free grammar.
\item \textit{Technical Details:} Works on SMILES strings and implements genetic mutations and crossovers based on their grammar.
\item \textit{Advantage:} Exploits the inherent structure of SMILES for effective exploration.
\item \textit{Disadvantage:} Limited to the nuances of SMILES grammar, possibly missing out on novel structures.
\end{itemize}
\item \textbf{SynNet~\citep{gao2021amortized}.}
\begin{itemize}
\item \textit{Overview:} A synthesis-based genetic algorithm that operates on binary fingerprints and decodes to synthetic pathways.
\item \textit{Technical Details:} Focuses on the synthesis pathway, ensuring the synthesizability of generated molecules.
\item \textit{Advantage:} Prioritizes synthesizability, ensuring generated molecules can be created in the lab.
\item \textit{Disadvantage:} The emphasis on synthesis may limit the diversity of the molecular space explored.
\end{itemize}
\item \textbf{DoG-Gen~\citep{bradshaw2020barking}.}
\begin{itemize}
\item \textit{Overview:} This method is tailored to learn the distribution of synthetic pathways.
\item \textit{Technical Details:} DoG-Gen represents synthetic pathways as Directed Acyclic Graphs (DAGs) and employs an RNN generator to model the distribution. By focusing on synthetic pathways, the method inherently emphasizes the synthesizability of molecules.
\item \textit{Advantages:} Provides a structured approach to learning synthetic pathways.
\item \textit{Disadvantages:} Reliance on RNNs might lead to issues in capturing very long sequences if not designed effectively.
\end{itemize}
\item \textbf{DST~\citep{fu2021differentiable}.}
\begin{itemize}
\item \textit{Overview:} DST stands for Differentiable Scaffolding Tree, a gradient ascent method designed for molecular optimization.
\item \textit{Technical Details:} DST abstracts molecular graphs into scaffolding trees and makes use of a graph neural network for gradient estimation. This gradient-focused approach enables fine-tuned molecular modifications based on property landscapes in the chemical space.
\item \textit{Advantages:} Offers a more direct way to optimize molecular structures by computing gradients.
\item \textit{Disadvantages:} The abstraction to scaffolding trees may result in loss of information.
\end{itemize}
\end{itemize}
% \begin{itemize}
% \item \textbf{REINVENT~\citep{olivecrona2017molecular}.} Uses SMILES representation and RNN with GRUs. Employs a combination of prior likelihood and reward function. $\sigma$ plays a crucial role in optimization, set to 500.
% \item \textbf{Graph-GA~\citep{jensen2019graph}.} Manipulates molecular graphs using crossover and mutation. Sets a mutation rate of 6.7\% with various mutation operations.
% \item \textbf{SELFIES-REINVENT.} Uses SELFIES for molecular representation. Shares the setup with REINVENT, but differs in vocabulary.
% \item \textbf{GP BO~\citep{tripp2021fresh}.} Uses Gaussian process for the surrogate model. Operates on 2-radius 2048 bit molecular fingerprints. Has various parameters for optimization, including a maximum of 10,000 BO iterations.
% \item \textbf{STONED~\citep{nigam2021beyond}.} Genetic algorithm on SELFIES string without crossover. Optimized with a generation size of 500.
% \item \textbf{SMILES-LSTM HC~\citep{brown2019guacamol}.} Employs a three-layer LSTM as the neural model. Uses hill climbing with specific parameters like a batch size of 256 and maximum SMILES length of 100.
% \item \textbf{SMILES-GA~\citep{brown2019guacamol}.} Operates on SMILES strings with mutations only. Has set parameters such as a population size of 100 and a mutation involving flipping of SMILES bits.
% \item \textbf{SynNet~\citep{gao2021amortized}.} Utilizes GA on binary molecular fingerprints with an MLP neural architecture. Consists of four major modules, each containing specific tasks and dimensions.
% \item \textbf{DoG-Gen~\citep{bradshaw2020barking}.} Hill climbing variant of DoG. Samples 3,000 molecules in each iteration, retaining the top 1,000. Uses GGNN for molecular embedding and GRU for molecule generation.
% \item \textbf{DST~\citep{fu2021differentiable}.} Employs GCN as property predictor with set layer numbers and dimensions. Uses GA-like processes and determinantal point process to improve population diversity.
% \end{itemize}
\subsection{Metric}
In order to evaluate both optimization capability and sample efficiency, we utilize the area under the curve (AUC) of the top-K average property value in relation to the number of oracle calls. This metric, which we refer to as AUC top-K, serves as our primary measure of performance. Distinct from the straightforward top-K average property, the AUC provides greater reward to methods that achieve high values with a reduced number of oracle calls. In this paper, we set K at 1, 10, and 100. This choice is motivated by the importance of pinpointing a limited set of unique molecular candidates for subsequent phases of development. We cap the number of oracle calls at 10,000, though we anticipate that effective methods should ideally optimize with just hundreds of calls during experimental evaluations. All AUC values reported have been min-max scaled to fit within the range [0, 1].
% \begin{itemize}
% \item \textbf{Pandalm}: This metric is based on the Pandalm large model to compare the quality of the language model for answering some questions. This metric will give scores to rank the strength of the language model.
% \item \textbf{Bleu}: This metric measures the accuracy and fluency of the model-generated answers.
% \item \textbf{Perplexity}: This metric indicates how difficult it is for the model to correctly predict the probability of each word in the test set, given a language model and a test set. .
% \end{itemize}
\subsection{Evaluation Results}
Our result is shown in Table. \ref{tab:main result}. From the table, we can observe that our method is better than the baseline method on multiple Oracles, which proves the effectiveness of the transformer in our problem.
\para{Overall Molecular Generation Result}
\begin{table*}[t]
\centering
\caption{Performance comparison between Reinvent-Trans, REINVENT, and other methods over all oracles for AUC Top-10}
\resizebox{1\textwidth}{!}{
\begin{tabular}{c||cccccccc}
\Xhline{1pt} \textbf{Method}
& \textbf{\textbf{REINVENT-Trans}} & \textbf{REINVENT}
& \textbf{Graph GA}
& \textbf{REINVENT}
& \textbf{GP BO}
& \textbf{STONED}\\
\textbf{Assembly} & \textbf{SMILES} & \textbf{SMILES} &\textbf{Fragments}& \textbf{SELFIES} & \textbf{Fragments} & \textbf{SELFIES}
\\\Xhline{1pt}
Albuterol\_Similarity & \textbf{0.910$\pm$ 0.008} & 0.882$\pm$ 0.006 & 0.838$\pm$ 0.016 & 0.826$\pm$ 0.030 & 0.898$\pm$ 0.014 & 0.745$\pm$ 0.076\\
Amlodipine\_MPO & 0.653$\pm$ 0.029 & 0.635$\pm$ 0.035 & \textbf{0.661$\pm$ 0.020} & 0.607$\pm$ 0.014 & 0.583$\pm$ 0.044 & 0.608$\pm$ 0.046\\
Celecoxib\_Rediscovery & 0.457$\pm$ 0.071 & 0.713$\pm$ 0.067 & 0.630$\pm$ 0.097 & 0.573$\pm$ 0.043 & \textbf{0.723$\pm$ 0.053} & 0.382$\pm$ 0.041\\
DRD2 & 0.931$\pm$ 0.006 & 0.945$\pm$ 0.007 & 0.964$\pm$ 0.012 & 0.943$\pm$ 0.005 & 0.923$\pm$ 0.017 & 0.913$\pm$ 0.020\\
Deco\_Hop & 0.645$\pm$ 0.038 & 0.666$\pm$ 0.044 & 0.619$\pm$ 0.004 & 0.631$\pm$ 0.012 & 0.629$\pm$ 0.018 & 0.611$\pm$ 0.008\\
Fexofenadine\_MPO & 0.796$\pm$ 0.007 & 0.784$\pm$ 0.006 & 0.760$\pm$ 0.011 & 0.741$\pm$ 0.002 & 0.722$\pm$ 0.005 & \textbf{0.797$\pm$ 0.016}\\
Isomers\_C9H10N2O2PF2Cl & 0.809$\pm$ 0.040 & 0.642$\pm$ 0.054 & 0.719$\pm$ 0.047 & 0.733$\pm$ 0.029 & 0.469$\pm$ 0.180 & 0.805$\pm$ 0.031\\
Median 1 & 0.354$\pm$ 0.008 & \textbf{0.356$\pm$ 0.009} & 0.294$\pm$ 0.021 & 0.355$\pm$ 0.011 & 0.301$\pm$ 0.014 & 0.266$\pm$ 0.016\\
Median 2 & 0.263$\pm$ 0.006 & 0.276$\pm$ 0.008 & 0.273$\pm$ 0.009 & 0.255$\pm$ 0.005 & \textbf{0.297$\pm$ 0.009} & 0.245$\pm$ 0.032\\
Mestranol\_Similarity & \textbf{0.685$\pm$ 0.032} & 0.618$\pm$ 0.048 & 0.579$\pm$ 0.022 & 0.620$\pm$ 0.029 & 0.627$\pm$ 0.089 & 0.609$\pm$ 0.101\\
Osimertinib\_MPO & 0.813$\pm$ 0.010 & \textbf{0.837$\pm$ 0.009} & 0.831$\pm$ 0.005 & 0.820$\pm$ 0.003 & 0.787$\pm$ 0.006 & 0.822$\pm$ 0.012\\
Perindopril\_MPO & 0.525$\pm$ 0.011 & 0.537$\pm$ 0.016 & 0.538$\pm$ 0.009 & 0.517$\pm$ 0.021 & 0.493$\pm$ 0.011 & 0.488$\pm$ 0.011\\
QED & \textbf{0.942$\pm$ 0.000} & 0.941$\pm$ 0.000 & 0.940$\pm$ 0.000 & 0.940$\pm$ 0.000 & 0.937$\pm$ 0.000 & 0.941$\pm$ 0.000\\
Ranolazine\_MPO & 0.761$\pm$ 0.012 & 0.742$\pm$ 0.009 & 0.728$\pm$ 0.012 & 0.748$\pm$ 0.018 & 0.735$\pm$ 0.013 & \textbf{0.765$\pm$ 0.029}\\
Scaffold\_Hop & \textbf{0.560$\pm$ 0.013} & 0.536$\pm$ 0.019 & 0.517$\pm$ 0.007 & 0.525$\pm$ 0.013 & 0.548$\pm$ 0.019 & 0.521$\pm$ 0.034\\
Sitagliptin\_MPO & \textbf{0.563$\pm$ 0.025} & 0.451$\pm$ 0.003 & 0.433$\pm$ 0.075 & 0.194$\pm$ 0.121 & 0.186$\pm$ 0.055 & 0.393$\pm$ 0.083\\
Thiothixene\_Rediscovery & 0.556$\pm$ 0.016 & 0.534$\pm$ 0.013 & 0.479$\pm$ 0.025 & 0.495$\pm$ 0.040 & \textbf{0.559$\pm$ 0.027} & 0.367$\pm$ 0.027\\
Troglitazone\_Rediscovery & \textbf{0.451$\pm$ 0.015} & 0.441$\pm$ 0.032 & 0.390$\pm$ 0.016 & 0.348$\pm$ 0.012 & 0.410$\pm$ 0.015 & 0.320$\pm$ 0.018\\
Valsartan\_Smarts & \textbf{0.165$\pm$ 0.278} & 0.165$\pm$ 0.358 & 0.000$\pm$ 0.000 & 0.000$\pm$ 0.000 & 0.000$\pm$ 0.000 & 0.000$\pm$ 0.000\\
Zaleplon\_MPO & \textbf{0.544 $\pm$ 0.041} & 0.358$\pm$ 0.062 & 0.346$\pm$ 0.032 & 0.333$\pm$ 0.026 & 0.221$\pm$ 0.072 & 0.325$\pm$ 0.027 \\
sum &12.197& 12.047& 11.526& 11.092& 11.152& 10.598\\
rank & 1&2&3&5&4&6\\
\hline
\hline
\Xhline{1pt} \textbf{Method}
& \textbf{\textbf{LSTM HC}} & \textbf{SMILES GA}
& \textbf{SynNet}
& \textbf{DoG-Gen}
& \textbf{DST}\\
\textbf{Assembly} & \textbf{SMILES} & \textbf{SMILES} & \textbf{Synthesis} & \textbf{Synthesis} & \textbf{Fragments}
\\\Xhline{1pt}
Albuterol\_similarity & 0.719$\pm$ 0.018 & 0.661$\pm$ 0.066 & 0.584$\pm$ 0.039 & 0.676$\pm$ 0.013 & 0.619$\pm$ 0.020 \\
Amlodipine\_MPO & 0.593$\pm$ 0.016 & 0.549$\pm$ 0.009 & 0.565$\pm$ 0.007 & 0.536$\pm$ 0.003 & 0.516$\pm$ 0.007 \\
Celecoxib\_Rediscovery & 0.539$\pm$ 0.018 & 0.344$\pm$ 0.027 & 0.441$\pm$ 0.027 & 0.464$\pm$ 0.009 & 0.380$\pm$ 0.006 \\
DRD2 & 0.919$\pm$ 0.015 & 0.908$\pm$ 0.019 & \textbf{0.969$\pm$ 0.004} & 0.948$\pm$ 0.001 & 0.820$\pm$ 0.014 \\
Deco\_Hop & \textbf{0.826$\pm$ 0.017} & 0.611$\pm$ 0.006 & 0.613$\pm$ 0.009 & 0.800$\pm$ 0.007 & 0.608$\pm$ 0.008 \\
Fexofenadine\_MPO & 0.725$\pm$ 0.003 & 0.721$\pm$ 0.015 & 0.761$\pm$ 0.015 & 0.695$\pm$ 0.003 & 0.725$\pm$ 0.005 \\
%gsk3b & \textbf{0.839$\pm$ 0.015} & 0.629$\pm$ 0.044 & 0.789$\pm$ 0.032 & 0.831$\pm$ 0.021 & 0.671$\pm$ 0.032 \\
%isomers\_c7h8n2o2 & 0.485$\pm$ 0.045 & \textbf{0.913$\pm$ 0.021} & 0.455$\pm$ 0.031 & 0.465$\pm$ 0.018 & 0.548$\pm$ 0.069 \\
Isomers\_C9H10N2O2PF2Cl & 0.342$\pm$ 0.027 & \textbf{0.860$\pm$ 0.065} & 0.241$\pm$ 0.064 & 0.199$\pm$ 0.016 & 0.458$\pm$ 0.063 \\
%jnk3 & \textbf{0.661$\pm$ 0.039} & 0.316$\pm$ 0.022 & 0.630$\pm$ 0.034 & 0.595$\pm$ 0.023 & 0.556$\pm$ 0.057 \\
Median 1 & 0.255$\pm$ 0.010 & 0.192$\pm$ 0.012 & 0.218$\pm$ 0.008 & 0.217$\pm$ 0.001 & 0.232$\pm$ 0.009 \\
Median 2 & 0.248$\pm$ 0.008 & 0.198$\pm$ 0.005 & 0.235$\pm$ 0.006 & 0.212$\pm$ 0.000 & 0.185$\pm$ 0.020 \\
Mestranol\_Similarity & 0.526$\pm$ 0.032 & 0.469$\pm$ 0.029 & 0.399$\pm$ 0.021 & 0.437$\pm$ 0.007 & 0.450$\pm$ 0.027 \\
Osimertinib\_MPO & 0.796$\pm$ 0.002 & 0.817$\pm$ 0.011 & 0.796$\pm$ 0.003 & 0.774$\pm$ 0.002 & 0.785$\pm$ 0.004 \\
Perindopril\_MPO & 0.489$\pm$ 0.007 & 0.447$\pm$ 0.013 & \textbf{0.557$\pm$ 0.011} & 0.474$\pm$ 0.002 & 0.462$\pm$ 0.008 \\
QED & 0.939$\pm$ 0.000 & 0.940$\pm$ 0.000 & 0.941$\pm$ 0.000 & 0.934$\pm$ 0.000 & 0.938$\pm$ 0.000 \\
Ranolazine\_MPO & 0.714$\pm$ 0.008 & 0.699$\pm$ 0.026 & 0.741$\pm$ 0.010 & 0.711$\pm$ 0.006 & 0.632$\pm$ 0.054 \\
Scaffold\_Hop & 0.533$\pm$ 0.012 & 0.494$\pm$ 0.011 & 0.502$\pm$ 0.012 & 0.515$\pm$ 0.005 & 0.497$\pm$ 0.004 \\
Sitagliptin\_MPO & 0.066$\pm$ 0.019 & 0.363$\pm$ 0.057 & 0.025$\pm$ 0.014 & 0.048$\pm$ 0.008 & 0.075$\pm$ 0.032 \\
Thiothixene\_Rediscovery & 0.438$\pm$ 0.008 & 0.315$\pm$ 0.017 & 0.401$\pm$ 0.019 & 0.375$\pm$ 0.004 & 0.366$\pm$ 0.006 \\
Troglitazone\_Rediscovery & 0.354$\pm$ 0.016 & 0.263$\pm$ 0.024 & 0.283$\pm$ 0.008 & 0.416$\pm$ 0.019 & 0.279$\pm$ 0.019 \\
Valsartan\_Smarts & 0.000$\pm$ 0.000 & 0.000$\pm$ 0.000 & 0.000$\pm$ 0.000 & 0.000$\pm$ 0.000 & 0.000$\pm$ 0.000 \\
Zaleplon\_MPO & 0.206$\pm$ 0.006 & 0.334$\pm$ 0.041 & 0.341$\pm$ 0.011 & 0.123$\pm$ 0.016 & 0.176$\pm$ 0.045 \\
sum &10.227 & 10.185 & 9.613 & 9.554 & 9.203\\
rank & 7&8&9&10&11\\
\Xhline{1pt}
\end{tabular}
}
\label{tab:main result}
\end{table*}
% \begin{table*}[htbp]
% \centering
% \caption{Performance comparison between REINVENT-Trans and REINVENT over all oracles for AUC Top-10}
% \resizebox{1\textwidth}{!}{
% \begin{tabular}{c||cc}
% \Xhline{1pt} \textbf{Task}
% & \textbf{REINVENT-Trans}
% & \textbf{REINVENT}\\ \Xhline{1pt}
% Albuterol\_Similarity & 0.910 & 0.898 \\
% Amlodipine\_MPO & 0.653 & 0.623 \\
% Celecoxib\_Rediscovery & 0.457 & 0.657 \\
% DRD2 & 0.931 & 0.577 \\
% Deco\_Hop & 0.645 & 0.645 \\
% Fexofenadine\_MPO & 0.796 & 0.798 \\
% Isomers\_C9H10N2O2PF2Cl & 0.809 & 0.820 \\
% Median 1 & 0.354 & 0.364 \\
% Median 2 & 0.263 & 0.268 \\
% Mestranol\_Similarity & 0.685 & 0.625 \\
% Osimertinib\_MPO & 0.813 & 0.822 \\
% Perindopril\_MPO & 0.525 & 0.526 \\
% QED & 0.934 & 0.933 \\
% Ranolazine\_MPO & 0.761 & 0.742 \\
% Scaffold\_Hop & 0.560 & 0.536 \\
% Sitagliptin\_MPO & 0.563 & 0.451 \\
% Thiothixene\_Rediscovery & 0.556 & 0.503 \\
% Troglitazone\_Rediscovery & 0.451 & 0.414 \\
% Valsartan\_Smarts & 0.165 & 0.165 \\
% Zaleplon\_MPO & 0.544 & 0.521 \\\Xhline{1pt}
% \end{tabular}
% }
% \label{tab:main result}
% \end{table*}
% \begin{table*}[htbp]
% \centering
% \caption{Performance comparison between REINVENT-Trans and REINVENT over all baselines}
% \resizebox{1\textwidth}{!}{
% \begin{tabular}{c||ccc|ccc}
% \Xhline{1pt} \textbf{Task}
% & \multicolumn{3}{c|}{\textbf{REINVENT-Trans}}
% & \multicolumn{3}{c}{\textbf{REINVENT}}\\
% & AUC Top-1&AUC Top-10&AUC Top-100&AUC Top-1&AUC Top-10&AUC Top-100\\ \Xhline{1pt}
% Albuterol\_Similarity & \textbf{0.923} & \textbf{0.910} & \textbf{0.877} & 0.920 & 0.898 & 0.838 \\
% Amlodipine\_MPO & \textbf{0.665} & \textbf{0.653} & \textbf{0.632} & 0.636 & 0.623 & 0.596 \\
% Celecoxib\_Rediscovery & 0.479 & 0.457 & 0.427 & \textbf{0.665} & \textbf{0.657} & \textbf{0.641} \\
% DRD2 & \textbf{0.956} & \textbf{0.931} & \textbf{0.904} & 0.689 & 0.577 & 0.531 \\
% Deco\_Hop & \textbf{0.652} & \textbf{0.645} & \textbf{0.634} & \textbf{0.652} & \textbf{0.645} & \textbf{0.634} \\
% Fexofenadine\_MPO & 0.811 & 0.796 & 0.771 & \textbf{0.821} & \textbf{0.798} & \textbf{0.765} \\
% %Isomers\_C7H8N2O2 & 0.583 & 0.555 & 0.508 & \textbf{0.966} & \textbf{0.944} & \textbf{0.854} \\
% Isomers\_C9H10N2O2PF2Cl & \textbf{0.844} & \textbf{0.809} & \textbf{0.762} & 0.839 & 0.820 & 0.777 \\
% Median 1 & 0.367 & 0.354 & 0.320 & \textbf{0.372} & \textbf{0.364} & \textbf{0.340} \\
% Median 2 & 0.277 & 0.263 & 0.245 & \textbf{0.280} & \textbf{0.268} & \textbf{0.249} \\
% Mestranol\_Similarity & \textbf{0.710} & \textbf{0.685} & \textbf{0.648} & 0.643 & 0.625 & 0.583 \\
% Osimertinib\_MPO & 0.822 & 0.813 & 0.789 & \textbf{0.835} & \textbf{0.822} & \textbf{0.794} \\
% Perindopril\_MPO & 0.533 & 0.525 & 0.506 & \textbf{0.540} & \textbf{0.526} & \textbf{0.499} \\
% QED & \textbf{0.936} & \textbf{0.934} & \textbf{0.921} & 0.935 & 0.933 & 0.919 \\
% Ranolazine\_MPO & \textbf{0.781} & \textbf{0.761} & \textbf{0.723} & 0.766 & 0.742 & 0.703 \\
% Scaffold\_Hop & \textbf{0.572} & \textbf{0.560} & \textbf{0.538} & 0.549 & 0.536 & 0.516 \\
% Sitagliptin\_MPO & \textbf{0.584} & \textbf{0.563} & \textbf{0.511} & 0.482 & 0.451 & 0.383 \\
% Thiothixene\_Rediscovery & \textbf{0.575} & \textbf{0.556} & \textbf{0.522} & 0.528 & 0.503 & 0.467 \\
% Troglitazone\_Rediscovery & \textbf{0.465} & \textbf{0.451} & \textbf{0.426} & 0.426 & 0.414 & 0.389 \\
% Valsartan\_Smarts & \textbf{0.210} & \textbf{0.165} & \textbf{0.094} & \textbf{0.210} & \textbf{0.165} & \textbf{0.094} \\
% Zaleplon\_MPO & \textbf{0.562} & \textbf{0.544} & \textbf{0.499} & 0.544 & 0.521 & 0.481 \\\Xhline{1pt}
% \end{tabular}
% }
% \label{tab:main result}
% \end{table*}
\begin{table}[t]
\centering
\begin{tabular}{c|c|c|c}
\Xhline{1pt}
\textbf{Oracle} & \textbf{Model} & \textbf{Avg SA}$\downarrow$ & \textbf{Diversity Top100} $\uparrow$\\
\Xhline{1pt}
Albuterol Similarity & reinvent & 3.177 & 0.394 \\
& reinvent trans & \textbf{3.173} & \textbf{0.408} \\
Amlodipine MPO & reinvent & \textbf{3.478} & \textbf{0.391} \\
& reinvent trans & 3.888 & 0.311 \\
Celecoxib Rediscovery & reinvent & 3.458 & \textbf{0.551} \\
& reinvent trans & \textbf{3.245} & 0.357 \\
DRD2 & reinvent & \textbf{2.788} & \textbf{0.868} \\
& reinvent trans & 2.914 & 0.464 \\
Deco Hop & reinvent & 3.458 & \textbf{0.551} \\
& reinvent trans & \textbf{3.240} & 0.457 \\
Fexofenadine MPO & reinvent & 4.163 & 0.325 \\
& reinvent trans & \textbf{4.113} & \textbf{0.411} \\
GSK3B & reinvent & \textbf{3.146} & \textbf{0.884} \\
& reinvent trans & \textbf{3.146} & \textbf{0.884} \\
Isomers C7H8N2O2 & reinvent & 4.273 & 0.712 \\
& reinvent trans & \textbf{2.589} & \textbf{0.796} \\
Isomers C9H10N2O2PF2Cl & reinvent & 3.261 & 0.585 \\
& reinvent trans & \textbf{3.245} & \textbf{0.686} \\
Median 1 & reinvent & 4.571 & \textbf{0.408} \\
& reinvent trans & \textbf{3.532} & 0.371 \\
Median 2 & reinvent & \textbf{2.772} & \textbf{0.411} \\
& reinvent trans & 2.877 & 0.389 \\
Mestranol Similarity & reinvent & \textbf{3.799} & 0.267 \\
& reinvent trans& 4.394 & \textbf{0.434} \\
Osimertinib MPO
& reinvent & \textbf{3.174} & \textbf{0.504} \\
& reinvent trans& 3.799 & 0.447 \\
Perindopril MPO
& reinvent & 3.819 & \textbf{0.479} \\
& reinvent trans& \textbf{3.766} & 0.357\\
QED
& reinvent & \textbf{1.883} & \textbf{0.573} \\
& reinvent trans& 3.422 & 0.540\\
Ranolazine MPO
& reinvent & 3.468 & 0.421 \\
& reinvent trans& \textbf{2.727} & \textbf{0.434}\\
Scaffold Hop
& reinvent & \textbf{2.857} & \textbf{0.555} \\
& reinvent trans& 4.355 & 0.382 \\
Sitagliptin MPO
& reinvent & \textbf{2.639} & \textbf{0.692} \\
& reinvent trans& 5.279 & 0.391\\
Thiothixene Rediscovery
& reinvent & \textbf{2.899} & 0.373 \\
& reinvent trans& 3.275 & \textbf{0.441}\\
Troglitazone Rediscovery
& reinvent & \textbf{3.275} & \textbf{0.441} \\
& reinvent trans& 4.435 & 0.204\\
Valsartan Smarts
& reinvent & 3.421 & 0.874 \\
& reinvent trans& 3.421 & 0.874\\
Zaleplon MPO
& reinvent & \textbf{1.991}& \textbf{0.614} \\
& reinvent trans& 2.465 & 0.486\\
\hline
\end{tabular}
\caption{Avg SA and Diversity Top100}
\end{table}
The evaluation results depict a thorough comparison between the REINVENT-Transformer (referred to as REINVENT-Trans) and other prominent models across multiple oracles.
\para{Overall Molecular Generation Result Performance Overview}
\textbf{REINVENT-Trans} demonstrates its strength in molecular generation, consistently achieving top results in several oracles. For instance, the model achieved the highest performance for `Albuterol\_Similarity', `Mestranol\_Similarity', `QED', `Scaffold\_Hop', and `Sitagliptin\_MPO'. This suggests that the transformer's architecture potentially excels in capturing intricate molecular patterns and relations, and effectively optimizing towards desired properties.
\para{Comparative Insight}
1. \textbf{Versus REINVENT (SMILES and SELFIES):} REINVENT-Trans has outperformed the REINVENT model (using SMILES) in multiple instances. However, it's worth noting that in some oracles like `Osimertinib\_MPO', REINVENT achieves a marginally better score. It's also evident that SELFIES representation in REINVENT doesn't always improve the performance as compared to its SMILES counterpart. This underscores the importance of the underlying model's architecture and how different representations can influence its performance.
2. \textbf{Graph-based Models:} Both `Graph GA' and `GP BO' exhibit competitive performance in certain oracles like `Amlodipine\_MPO' and `Celecoxib\_Rediscovery' respectively. However, their performance isn't consistently at the top across all oracles. This implies that while graph-based models can be effective in certain scenarios, they may not always generalize well across diverse tasks.
3. \textbf{Genetic Algorithms:} STONED (using SELFIES representation) achieves the highest score in the `Fexofenadine\_MPO' oracle. Genetic algorithms, despite their inherent stochasticity, have potential in some specific optimization tasks.
\para{Sample Efficiency}
The primary metric, AUC top-K, emphasizes not just optimization capability but also sample efficiency. High values in this metric would imply fewer oracle calls, thereby maximizing performance with limited data evaluations. REINVENT-Trans, achieving high scores in several oracles, suggests its effectiveness in rapidly optimizing toward desirable molecular properties without exhaustive database searches.
\para{Future Implications}
These results showcase the potential of transformers in the domain of molecular generation. Given the success of transformer models in natural language processing tasks, it's no surprise that their capabilities translate effectively into molecular representations as well.
However, it's essential to consider the versatility of the tasks presented. Some models might specialize in particular tasks but may not be universally applicable. Hence, it's beneficial to have an ensemble or a selection mechanism based on the specific task at hand.
\subsection{Ablation Study: Long Sequence Molecule Generation Comparison with REINVENT SMILES}
\begin{figure}[t]
\centering
\includegraphics*[width=0.7\textwidth]{fig/evaluation_score_vs_molecule_length.png}
\caption{Evaluation score vs molecular length for comparison of Reinvent\_Transformer and Reinvent on oracle Mestranol\_Similarity} \label{fig:Oracle_mestranol_similarity_evaluation_score_vs_molecular_length}
\end{figure}
The box plot visualizes the distribution of evaluation scores across different molecular lengths for both the reinvent\_transformer method and the baseline reinvent method.
Based on the visual representation, we can derive the following observations:
1. For shorter molecular lengths, both methods exhibit similar distributions of scores.\\
2. As the molecular length increases, the reinvent\_transformer method consistently achieves higher averge scores.\\
3. For longer molecular sequences, the difference in scores between the two methods becomes more pronounced. This suggests that the reinvent\_transformer method is better suited for longer sequences, maintaining high evaluation scores.\\
4. The spread (interquartile range) of scores for the reinvent\_transformer
method remains relatively consistent across molecular lengths, indicating stable performance.\\
For shorter molecular lengths, both methods exhibit similar distributions of scores.
As the molecular length increases, the reinvent\_transformer method consistently achieves higher median scores.
For longer molecular sequences, the difference in scores between the two methods becomes more pronounced. This suggests that the reinvent\_transformer method is better suited for longer sequences, maintaining high evaluation scores.
The spread (interquartile range) of scores for the reinvent\_transformer method remains relatively consistent across molecular lengths, indicating stable performance.
In conclusion, the reinvent\_transformer method outperforms the baseline reinvent method, particularly in the context of longer molecular sequences.
\begin{figure}[t]
\centering
\includegraphics*[width=0.6\textwidth]{fig/new_Black_and.png}
\caption{Evaluation score vs short and long sequence for comparison of Reinvent\_Transformer and Reinvent on oracle Mestranol\_Similarity} \label{fig:Oracle_mestranol_similarity_evaluation_score_vs_short_long_sequence}
\end{figure}
We set a threshold=50 for the length of generated molecular string. If the generated string is longer than the threshold, it will be considered as "long", other it's considered as "short"
. From the Figure~\ref{fig:Oracle_mestranol_similarity_evaluation_score_vs_short_long_sequence}, we can see the our method Reinvent-Transformer has better average score when generating long sequences.
\subsection{Case Study: Convergence rate Comparison between Reinvent-Transformer and Reinvent}
We plotted the auc\_topk curve and number of epoches is the x-axis. From the figure as follows, we can see that our method Reinvent-Transformer converges faster than Reinvent method.
From Fig. \ref{fig:avg_top100_mestranol_Similarity}, the evolution of the average accuracy for the top 100 predictions is evident. Upon examination, across equivalent epochs, the mean accuracy of Reinvent-Transformer consistently surpasses that of Reinvent. This indicates a more expedient convergence rate for the Reinvent-Transformer compared to Reinvent.
\begin{figure}[t]
\centering
\includegraphics*[width=0.7\textwidth]{fig/Mean_and_Standard_avg_top100.png}
\caption{Mean and Standard Deviation of avg\_top100 over Epochs for Reinvent and Reinvent-Transformer on oracle Mestranol\_Similarity} \label{fig:avg_top100_mestranol_Similarity}
\end{figure}
The avg\_top100 curve initially displays a steep incline, eventually plateauing post approximately 6000 epochs. Notably, beginning from the 2500th epoch, the performance differential between Reinvent-Transformer and Reinvent significantly widens.
It is also observed that the Reinvent-Transformer possesses a higher standard deviation relative to Reinvent, suggesting potential variability in its performance. Despite this, the difference between the average top100 accuracy and the standard deviation for Reinvent-Transformer remains superior to the mean accuracy of Reinvent, reaffirming the enhanced efficacy of the Reinvent-Transformer method.
\begin{figure}[t]
\centering
\includegraphics*[width=0.7\textwidth]{fig/Mean_and_Standard_Deviation_of_auc_top100.png}
\caption{Mean and Standard Deviation of auc\_top100 over Epochs for REINVENT and REINVENT-Transformer on oracle Albuterol\_Similarity} \label{fig:avg_top100_Albuterol_Similarity}
\end{figure}
Furthermore, the AUC top100 curve for Albuterol Similarity is illustrated in Fig. \ref{fig:avg_top100_Albuterol_Similarity}. In this context, the differential in performance between REINVENT-Transformer and REINVENT is more nuanced. It isn't until the 8000th epoch that a discernible gap emerges. Ultimately, the REINVENT-Transformer exhibits marginally superior performance relative to Reinvent in this scenario.
\begin{figure}[t]
\centering
\includegraphics*[width=0.7\textwidth]{fig/Mean_and_Standard_Deviation_of_auc_top10.png}
\caption{Mean and Standard Deviation of auc\_top10 over Epochs for Reinvent and Reinvent-Transformer on oracle Mestranol\_Similarity} \label{fig:auc_top10_mestranol_similarity}
\end{figure}
In Fig. \ref{fig:auc_top10_mestranol_similarity}, the AUC top10 curve for Mestranol Similarity is presented. Contrasted with the average accuracy curve, this AUC curve demonstrates a milder inclination initially, followed by a pronounced rise. Specifically, for the Reinvent-Transformer, the mean AUC top10 consistently surpasses that of Reinvent. Although the disparity is subtle during the initial epochs, it becomes more pronounced post the 5000th epoch and remains so thereafter.
\section{Conclusion}
Navigating the vast chemical space in molecular design remains a challenge. The introduction of the REINVENT-Transformer marks a significant advancement, harnessing the Transformer architecture's strengths such as parallelization and long-term dependency handling. Our experimental findings reinforce the REINVENT-Transformer's superior performance across multiple oracles, especially in tasks requiring longer sequence data. By integrating oracle feedback reinforcement learning, our approach achieves heightened precision, favorably impacting drug discovery efforts. In essence, the REINVENT-Transformer not only sets a benchmark in molecular de novo design but also illuminates the path for future research, highlighting the promise of Transformer-based architectures in drug discovery.
% \section{Conclusion}
% In the vast and intricate realm of molecular design, efficient navigation of the expansive chemical space remains a formidable challenge. Traditional Recurrent Neural Networks (RNNs), while effective, have shown their limitations, particularly in modeling complex molecular structures due to their incapacity to handle long-term dependencies in sequential data. Our introduction of the REINVENT-Transformer presents a pivotal shift in molecular de novo design, leveraging the unique strengths of the Transformer architecture, such as parallelization, long-term dependency handling, and scalability.
% Our experimental evaluations elucidate the REINVENT-Transformer's prowess in molecular generation. Across multiple oracles, the REINVENT-Transformer consistently outperforms established models, underlining its capability in capturing intricate molecular patterns and optimizing towards desired properties. Especially in scenarios requiring longer sequence data, the advantages of the Transformer architecture become pronounced, substantiating our hypothesis and demonstrating its innovative application in the domain of molecular design.
% Furthermore, the integration of oracle feedback reinforcement learning augments the precision of our approach, facilitating the generation of molecules that not only satisfy structural constraints but also demonstrate high bioactivity. Such enhanced precision becomes pivotal for increasing success rates in drug discovery efforts.
% However, it's imperative to acknowledge the multifaceted nature of the tasks in molecular generation. While REINVENT-Transformer establishes itself as a leader in many oracles, the versatility of tasks suggests the need for adaptability and, potentially, model ensembles tailored to specific challenges.
% In conclusion, the REINVENT-Transformer sets a new precedent in molecular de novo design, emphasizing the potential of Transformer-based architectures in revolutionizing drug discovery methodologies. As we continue to push the boundaries of molecular design, the innovations presented in this work offer a beacon for future research, highlighting the transformative impact of advanced machine learning architectures on scientific challenges.
\begin{thebibliography}{32}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bouville(2008)]{c:22}
Mathieu Bouville.
\newblock Crime and punishment in scientific research, 2008.
\bibitem[Bradshaw et~al.(2020)Bradshaw, Paige, Kusner, Segler, and Hern{\'a}ndez-Lobato]{bradshaw2020barking}
John Bradshaw, Brooks Paige, Matt~J Kusner, Marwin Segler, and Jos{\'e}~Miguel Hern{\'a}ndez-Lobato.
\newblock Barking up the right tree: an approach to search over molecule synthesis dags.
\newblock \emph{Advances in Neural Information Processing Systems}, 33:\penalty0 6852--6866, 2020.
\bibitem[Brown et~al.(2019)Brown, Fiscato, Segler, and Vaucher]{brown2019guacamol}
Nathan Brown, Marco Fiscato, Marwin~HS Segler, and Alain~C Vaucher.
\newblock {GuacaMol}: benchmarking models for de novo molecular design.
\newblock \emph{Journal of chemical information and modeling}, 59\penalty0 (3):\penalty0 1096--1108, 2019.
\bibitem[Clancey(1979)]{c:79}
William~J. Clancey.
\newblock \emph{{Transfer of Rule-Based Expertise through a Tutorial Dialogue}}.
\newblock {Ph.D.} diss., Dept.\ of Computer Science, Stanford Univ., Stanford, Calif., 1979.
\bibitem[Clancey(1983)]{c:83}
William~J. Clancey.
\newblock {Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education}.
\newblock In \emph{Proceedings of the Eighth International Joint Conference on Artificial Intelligence {(IJCAI-83)}}, pp.\ 556--560, Menlo Park, Calif, 1983. {IJCAI Organization}.
\bibitem[Clancey(1984)]{c:84}
William~J. Clancey.
\newblock {Classification Problem Solving}.
\newblock In \emph{Proceedings of the Fourth National Conference on Artificial Intelligence}, pp.\ 45--54, Menlo Park, Calif., 1984. AAAI Press.
\bibitem[Clancey(2021)]{c:21}
William~J. Clancey.
\newblock {The Engineering of Qualitative Models}.
\newblock Forthcoming, 2021.
\bibitem[Engelmore \& Morgan(1986)Engelmore and Morgan]{em:86}
Robert Engelmore and Anthony Morgan (eds.).
\newblock \emph{Blackboard Systems}.
\newblock Addison-Wesley, Reading, Mass., 1986.
\bibitem[Fu et~al.(2021)Fu, Xiao, Li, Glass, and Sun]{fu2021mimosa}
Tianfan Fu, Cao Xiao, Xinhao Li, Lucas~M Glass, and Jimeng Sun.
\newblock {MIMOSA}: Multi-constraint molecule sampling for molecule optimization.
\newblock \emph{AAAI}, 2021.
\bibitem[Fu et~al.(2022)Fu, Gao, Xiao, Yasonik, Coley, and Sun]{fu2021differentiable}
Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor~W Coley, and Jimeng Sun.
\newblock Differentiable scaffolding tree for molecular optimization.
\newblock \emph{International Conference on Learning Representations}, 2022.
\bibitem[Gao et~al.(2022)Gao, Mercado, and Coley]{gao2021amortized}
Wenhao Gao, Roc{\'\i}o Mercado, and Connor~W Coley.
\newblock Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design.
\newblock \emph{International Conference on Learning Representations}, 2022.
\bibitem[G{\'o}mez-Bombarelli et~al.(2018)G{\'o}mez-Bombarelli, Wei, Duvenaud, Hern{\'a}ndez-Lobato, S{\'a}nchez-Lengeling, Sheberla, Aguilera-Iparraguirre, Hirzel, Adams, and Aspuru-Guzik]{gomez2018automatic}
Rafael G{\'o}mez-Bombarelli, Jennifer~N Wei, David Duvenaud, Jos{\'e}~Miguel Hern{\'a}ndez-Lobato, Benjam{\'\i}n S{\'a}nchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy~D Hirzel, Ryan~P Adams, and Al{\'a}n Aspuru-Guzik.
\newblock Automatic chemical design using a data-driven continuous representation of molecules.
\newblock \emph{ACS central science}, 2018.
\bibitem[Goodfellow et~al.(2016)Goodfellow, Bengio, and Courville]{goodfellow2016deep}
Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
\newblock \emph{Deep learning}.
\newblock MIT press, 2016.
\bibitem[Hasling et~al.(1983)Hasling, Clancey, Rennels, and Test]{hcrt:83}
Diane~Warner Hasling, William~J. Clancey, Glenn~R. Rennels, and Thomas Test.
\newblock {Strategic Explanations in Consultation---Duplicate}.
\newblock \emph{The International Journal of Man-Machine Studies}, 20\penalty0 (1):\penalty0 3--19, 1983.
\bibitem[Hasling et~al.(1984)Hasling, Clancey, and Rennels]{hcr:83}
Diane~Warner Hasling, William~J. Clancey, and Glenn Rennels.
\newblock Strategic explanations for a diagnostic consultation system.
\newblock \emph{International Journal of Man-Machine Studies}, 20\penalty0 (1):\penalty0 3--19, 1984.
\newblock ISSN 0020-7373.
\newblock \doi{https://doi.org/10.1016/S0020-7373(84)80003-6}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0020737384800036}.
\bibitem[Jensen(2019)]{jensen2019graph}
Jan~H Jensen.
\newblock A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space.
\newblock \emph{Chemical science}, 10\penalty0 (12):\penalty0 3567--3572, 2019.
\bibitem[Jin et~al.(2018)Jin, Barzilay, and Jaakkola]{jin2018junction}
Wengong Jin, Regina Barzilay, and Tommi Jaakkola.
\newblock Junction tree variational autoencoder for molecular graph generation.
\newblock \emph{ICML}, 2018.
\bibitem[Korovina et~al.(2020)Korovina, Xu, Kandasamy, Neiswanger, Poczos, Schneider, and Xing]{korovina2020chembo}
Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing.
\newblock {ChemBO}: Bayesian optimization of small organic molecules with synthesizable recommendations.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, pp.\ 3393--3403. PMLR, 2020.
\bibitem[Krenn et~al.(2020)Krenn, H{\"a}se, Nigam, Friederich, and Aspuru-Guzik]{krenn2020self}
Mario Krenn, Florian H{\"a}se, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik.
\newblock Self-referencing embedded strings ({SELFIES}): A 100\% robust molecular string representation.
\newblock \emph{Machine Learning: Science and Technology}, 1\penalty0 (4):\penalty0 045024, 2020.
\bibitem[Moss et~al.(2020)Moss, Leslie, Beck, Gonzalez, and Rayson]{moss2020boss}
Henry Moss, David Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson.
\newblock {BOSS}: Bayesian optimization over string spaces.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 15476--15486, 2020.
\bibitem[{NASA}(2015)]{c:23}
{NASA}.
\newblock Pluto: The 'other' red planet.
\newblock \url{https://www.nasa.gov/nh/pluto-the-other-red-planet}, 2015.
\newblock Accessed: 2018-12-06.
\bibitem[Nigam et~al.(2021)Nigam, Pollice, Krenn, dos Passos~Gomes, and Aspuru-Guzik]{nigam2021beyond}
AkshatKumar Nigam, Robert Pollice, Mario Krenn, Gabriel dos Passos~Gomes, and Alan Aspuru-Guzik.
\newblock Beyond generative models: superfast traversal, optimization, novelty, exploration and discovery ({STONED}) algorithm for molecules using {SELFIES}.
\newblock \emph{Chemical science}, 12\penalty0 (20):\penalty0 7079--7090, 2021.
\bibitem[Olivecrona et~al.(2017)Olivecrona, Blaschke, Engkvist, and Chen]{olivecrona2017molecular}
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen.
\newblock Molecular de-novo design through deep reinforcement learning.
\newblock \emph{Journal of cheminformatics}, 9\penalty0 (1):\penalty0 1--14, 2017.
\bibitem[Rice(1986)]{r:86}
James Rice.
\newblock {Poligon: A System for Parallel Problem Solving}.
\newblock Technical Report KSL-86-19, Dept.\ of Computer Science, Stanford Univ., 1986.
\bibitem[Robinson(1980{\natexlab{a}})]{r:80}
Arthur~L. Robinson.
\newblock New ways to make microcircuits smaller.
\newblock \emph{Science}, 208\penalty0 (4447):\penalty0 1019--1022, 1980{\natexlab{a}}.
\newblock ISSN 0036-8075.
\newblock \doi{10.1126/science.208.4447.1019}.
\newblock URL \url{https://science.sciencemag.org/content/208/4447/1019}.
\bibitem[Robinson(1980{\natexlab{b}})]{r:80x}
Arthur~L. Robinson.
\newblock {New Ways to Make Microcircuits Smaller---Duplicate Entry}.
\newblock \emph{Science}, 208:\penalty0 1019--1026, 1980{\natexlab{b}}.
\bibitem[Shen et~al.(2021)Shen, Krenn, Eppel, and Aspuru-Guzik]{shen2021deep}
Cynthia Shen, Mario Krenn, Sagi Eppel, and Alan Aspuru-Guzik.
\newblock Deep molecular dreaming: {I}nverse machine learning for de-novo molecular design and interpretability with surjective representations.
\newblock \emph{Machine Learning: Science and Technology}, 2021.
\bibitem[Sutton \& Barto(1999)Sutton and Barto]{sutton1999reinforcement}
Richard~S Sutton and Andrew~G Barto.
\newblock Reinforcement learning: An introduction.
\newblock \emph{Robotica}, 17\penalty0 (2):\penalty0 229--235, 1999.
\bibitem[Tripp et~al.(2021)Tripp, Simm, and Hern{\'a}ndez-Lobato]{tripp2021fresh}
Austin Tripp, Gregor~NC Simm, and Jos{\'e}~Miguel Hern{\'a}ndez-Lobato.
\newblock A fresh look at de novo molecular design benchmarks.
\newblock In \emph{NeurIPS 2021 AI for Science Workshop}, 2021.
\bibitem[Weininger(1988)]{weininger1988smiles}
David Weininger.
\newblock {SMILES}, a chemical language and information system. 1. {I}ntroduction to methodology and encoding rules.
\newblock \emph{Journal of chemical information and computer sciences}, 28\penalty0 (1):\penalty0 31--36, 1988.
\bibitem[Weininger(2017)]{SMILES}
David Weininger.
\newblock Smiles.
\newblock Accessed 7 April 2017, 2017.
\newblock \url{http://www.daylight.com/dayhtml/doc/theory/theory.smiles.html}.
\bibitem[Zhou et~al.(2019)Zhou, Kearnes, Li, Zare, and Riley]{zhou2019optimization}
Zhenpeng Zhou, Steven Kearnes, Li~Li, Richard~N Zare, and Patrick Riley.
\newblock Optimization of molecules via deep reinforcement learning.
\newblock \emph{Scientific reports}, 9\penalty0 (1):\penalty0 1--10, 2019.
\end{thebibliography}