BioNeMo Framework: a modular, high-performance library for AI model development in drug discovery
Extracted source text · arXiv 2411.10548
\documentclass{article}
\usepackage{PRIMEarxiv}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\hypersetup{
colorlinks=true,
linkcolor=blue,
filecolor=magenta,
urlcolor=cyan,
pdftitle={BioNeMo Framework},
pdfpagemode=FullScreen,
}
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\microtypesetup{protrusion=false}
%%\usepackage[protrusion=false,expansion=true]{microtype}
\usepackage{lipsum}
\usepackage{xcolor}
\usepackage{fancyhdr} % header
\usepackage{graphicx} % graphics
\usepackage{setspace} % spacing
\usepackage[backend=biber]{biblatex}
\addbibresource{references.bib} % Replace "references.bib" with your file name
\graphicspath{{media/}} % organize your images and other figures under media/ folder
%Header
\pagestyle{fancy}
\thispagestyle{empty}
\rhead{ \textit{ }}
% Update your Headers here
\fancyhead[LO]{BioNeMo Framework}
% \fancyhead[RE]{Firstauthor and Secondauthor} % Firstauthor et al. if more than 2 - must use \documentclass[twoside]{article}
%% Title
\title{BioNeMo Framework: A Modular, High-performance Library for AI Model Development in Drug Discovery}
\author{
\begin{tabular}{c}
\textbf{Peter St. John}$^{a}$, \textbf{Dejun Lin}$^{a}$, \textbf{Polina Binder}$^{a}$, \textbf{Malcolm Greaves}$^{a}$, \textbf{Vega Shah}$^{a}$, \textbf{John St. John}$^{a}$, \\
\textbf{Adrian Lange}$^{b}$, \textbf{Patrick Hsu}$^{c}$, \textbf{Rajesh Illango}$^{c}$, \textbf{Arvind Ramanathan}$^{d}$, \textbf{Anima Anandkumar}$^{e}$, \\
\textbf{David H Brookes}$^{f}$, \textbf{Akosua Busia}$^{f}$, \textbf{Abhishaike Mahajan}$^{f}$, \textbf{Stephen Malina}$^{f}$, \textbf{Neha Prasad}$^{f}$, \\
\textbf{Sam Sinai}$^{f}$, \textbf{Lindsay Edwards}$^{g}$, \textbf{Thomas Gaudelet}$^{g}$, \textbf{Cristian Regep}$^{g}$, \textbf{Martin Steinegger}$^{h}$, \\
\textbf{Burkhard Rost}$^{i}$, \textbf{Alexander Brace}$^{jk}$, \textbf{Kyle Hippe}$^{jk}$, \textbf{Luca Naef}$^{l}$, \textbf{Keisuke Kamata}$^{m}$, \\
\textbf{George Armstrong}$^{a}$, \textbf{Kevin Boyd}$^{a}$, \textbf{Zhonglin Cao}$^{a}$, \textbf{Han-Yi Chou}$^{a}$, \textbf{Simon Chu}$^{a}$, \\
\textbf{Allan dos Santos Costa}$^{a}$, \textbf{Sajad Darabi}$^{a}$, \textbf{Eric Dawson}$^{a}$, \textbf{Kieran Didi}$^{a}$, \textbf{Cong Fu}$^{a}$, \\
\textbf{Mario Geiger}$^{a}$, \textbf{Michelle Gill}$^{a}$, \textbf{Darren J Hsu}$^{a}$, \textbf{Gagan Kaushik}$^{a}$, \textbf{Maria Korshunova}$^{a}$, \\
\textbf{Steven Kothen-Hill}$^{a}$, \textbf{Youhan Lee}$^{a}$, \textbf{Meng Liu}$^{a}$, \textbf{Micha Livne}$^{a}$, \textbf{Zachary McClure}$^{a}$, \\
\textbf{Jonathan Mitchell}$^{a}$, \textbf{Alireza Moradzadeh}$^{a}$, \textbf{Ohad Mosafi}$^{a}$, \textbf{Youssef Nashed}$^{a}$, \textbf{Saee Paliwal}$^{a}$, \\
\textbf{Yuxing Peng}$^{a}$, \textbf{Sara Rabhi}$^{a}$, \textbf{Farhad Ramezanghorbani}$^{a}$, \textbf{Danny Reidenbach}$^{a}$, \textbf{Camir Ricketts}$^{a}$, \\
\textbf{Brian C Roland}$^{a}$, \textbf{Kushal Shah}$^{a}$, \textbf{Tyler Shimko}$^{a}$, \textbf{Hassan Sirelkhatim}$^{a}$, \textbf{Savitha Srinivasan}$^{an}$, \\
\textbf{Abraham C Stern}$^{a}$, \textbf{Dorota Toczydlowska}$^{a}$, \textbf{Srimukh Prasad Veccham}$^{a}$, \textbf{Niccolò Alberto Elia Venanzi}$^{a}$, \\
\textbf{Anton Vorontsov}$^{a}$, \textbf{Jared Wilber}$^{a}$, \textbf{Isabel Wilkinson}$^{a}$, \textbf{Wei Jing Wong}$^{a}$, \textbf{Eva Xue}$^{a}$, \\
\textbf{Cory Ye}$^{a}$, \textbf{Xin Yu}$^{a}$, \textbf{Yang Zhang}$^{a}$, \textbf{Guoqing Zhou}$^{a}$, \textbf{Becca Zandstein}$^{a}$, \\
\textbf{Alejandro Chacon}$^{a}$, \textbf{Prashant Sohani}$^{a}$, \textbf{Maximilian Stadler}$^{a}$, \textbf{Christian Hundt}$^{a}$, \textbf{Feiwen Zhu}$^{a}$, \\
\textbf{Christian Dallago}$^{ao}$, \textbf{Bruno Trentini}$^{ap}$, \textbf{Emine Kucukbenli}$^{a}$, \textbf{Saee Paliwal}$^{a}$, \textbf{Timur Rvachov}$^{a}$, \textbf{Eddie Calleja}$^{a}$, \\
\textbf{Johnny Israeli}$^{a}$, \textbf{Harry Clifford}$^{a}$, \textbf{Risto Haukioja}$^{a}$, \textbf{Nicholas Haemel}$^{a}$, \\ \textbf{Kyle Tretina}$^{a*}$,
\textbf{Neha Tadimeti}$^{a*}$, \textbf{Anthony B Costa}$^{a*}$ \\
\end{tabular}
}
\begin{document}
\maketitle
\vspace{-2em} % Adjust this value as needed to reduce space between authors and abstract
\begin{abstract}
Artificial Intelligence models encoding biology and chemistry are opening new routes to high-throughput and high-quality in-silico drug development. However, their training increasingly relies on computational scale, with recent protein language models (pLM) training on hundreds of graphical processing units (GPUs). We introduce the BioNeMo Framework to facilitate the training of computational biology and chemistry AI models across hundreds of GPUs. Its modular design allows the integration of individual components, such as data loaders, into existing workflows and is open to community contributions. We detail technical features of the BioNeMo Framework through use cases such as pLM pre-training and fine-tuning. On 256 NVIDIA A100s, BioNeMo Framework trains a three billion parameter BERT-based pLM on over one trillion tokens in 4.2 days. The BioNeMo Framework is open-source and free for everyone to use.
\end{abstract}
\begingroup
\renewcommand{\thefootnote}{}
\footnotetext{\textbf{Affiliations:}
$^a$NVIDIA,
$^b$A-Alpha Bio Inc.,
$^c$Arc Institute,
$^d$Argonne National Laboratory,
$^e$Caltech,
$^f$Dyno Therapeutics,
$^g$Relation Therapeutics,
$^h$Seoul National University,
$^i$Technical University of Munich,
$^j$University of Chicago,
$^k$Argonne National Laboratory,
$^l$VantAI,
$^m$Weights \& Biases,
$^n$Stanford University,
$^o$Duke University,
$^p$University of Oxford.
$^*$\textbf{Correspondence to}: {Anthony B Costa: acosta@nvidia.com; Neha Tadimeti: ntadimeti@nvidia.com; Kyle Tretina: ktretina@nvidia.com.}}
\addtocounter{footnote}{-1}
\endgroup
% Apply left indentation from Introduction onward
\setlength{\leftskip}{2em} % Set left indentation for all subsequent text
% keywords can be removed
\keywords{artificial intelligence \and drug discovery \and deep learning \and framework \and biomolecules}
\section{Introduction}
The field of Artificial Intelligence (AI) has revolutionized computational biology and chemistry, culminating in the 2024 Nobel Prize in Chemistry for advancements in computational protein design and protein structure prediction. These breakthroughs have been driven by the rapid growth of high-throughput experimental biology \cite{behjati2013next}, extensive sequence databases \cite{uniprot2023uniprot,lee2024petabase}, and significant progress in large-scale neural network architectures \cite{vaswani2017attention} leading to Large Language Models (LLMs). As such, protein Language Models (pLMs) \cite{elnaggar2021prottrans, Lin2023-sb}, trained on vast protein sequence databases \cite{uniprot2023uniprot}, are now achieving unprecedented accuracy in tasks such as structure prediction \cite{Lin2023-sb}, function prediction \cite{bernhofer2021predictprotein}, and de novo protein design \cite{ferruz2023sequence}.
With recent understanding of LLM scaling laws, pLMs have been growing in size to improve performance, with ESM3 featuring 98 billion parameters and requiring 10\textsuperscript{24} floating-point operations (FLOPS) to train \cite{Lin2023-sb, Kaplan2020-en}. While improvements in training data quality may also help improve performance \cite{Fournier2024-fp}, we believe that frontier pLMs will continue to increase in size and training requirements. Beyond protein design, LLMs are also driving new techniques in chemical and material discovery \cite{Boiko2023-mj, Kononova2021-cq}.
AI-driven molecular research is driving a rapid increase in the need for specialized machine-learning frameworks tailored to drug discovery. While traditional tools like PyTorch\cite{Paszke2019-ch}, TensorFlow\cite{tensorflow2015-whitepaper}, and JAX\cite{jax2018github} have been successful in advancing deep learning, these frameworks are not optimized for the complex biomolecular data and large-scale models required in drug discovery. Many of these frameworks are excellent for general-purpose deep learning but often require intricate adaptation for tasks such as protein structure prediction or chemical reaction modeling, resulting in trade-offs between flexibility and performance. Efficiently training very large models requires careful tuning of data and model-parallelism strategies and requires careful use of fused CUDA kernels to minimize memory usage and maximize training throughput \cite{Kuchaiev2019-am}\cite{Narayanan2021-qc}. High-performance data loaders are also a critical component of a pre-training pipeline, as ensuring that GPUs are fully utilized requires that pre-processing and memory access is done as efficiently as possible. Minimizing energy consumption and maximizing model performance are especially important as model training increases in scale. Further progress in these fields also requires that these techniques remain accessible to smaller research groups and domain scientists in chemical and biological research areas.
In this paper, we present the BioNeMo Framework, a suite of open-source python packages designed to simplify the process of training high-performance AI models in the biomolecular and chemical space. These libraries build on top of NVIDIA NeMo\cite{Kuchaiev2019-am} and Megatron-LM \cite{Narayanan2021-qc} to optimize training and inference throughput, and include high-performance data loaders for both protein sequence sampling and single-cell data. In this release, BioNeMo Framework includes the ESM-2 \cite{Lin2023-sb} and Geneformer \cite{Theodoris2023-np} models, where we demonstrate an over 2x improvement in training throughput over PyTorch implementations and near-linear scaling in multi-device training up to 256 GPUs. We also discuss future directions for the BioNeMo Framework and how external contributors can extend the framework to accelerate additional families of models.
\section{Overview of BioNeMo Framework}
\label{sec:headings}
NVIDIA BioNeMo Framework is a collection of programming tools, libraries, and models for computational drug discovery. It accelerates the most time-consuming and costly stages of building and adapting biomolecular AI by providing domain-specific, optimized models and tooling that can be easily integrated into any GPU-based compute environment.
The BioNeMo Framework codebase is designed with a sub-project structure that facilitates ease of contribution and focus among its users. While the entire codebase can be installed as a cohesive unit, it is also segmented into independently installable components, referred to as sub-projects, organized by their specific purposes and scopes. This modular architecture enhances the capability for extensibility, allowing new contributions to take two forms: the creation of new sub-projects (wide contributions) or the extension of existing sub-projects (deep contributions).
At the center of BioNeMo lies bionemo-core, which provides essential interfaces and common data processing and model building blocks built around PyTorch and the Lightning framework. This core dependency supports all other sub-projects within the framework. For instance, the bionemo-llm sub-package is built upon NVIDIA’s NeMo and Megatron libraries, providing bio-specific customizations and foundational model architectures. The bionemo-geometric sub-package is developing support for Graph Neural Networks(GNNs) leveraging PyTorch Geometric, highlighting the breadth of innovations possible within the BioNeMo ecosystem.
Specific models and data loaders are encapsulated within their sub-packages, enabling users to selectively incorporate only the components relevant to their particular use cases. This modular design not only streamlines customization but also enhances usability.
To foster easy integration into private projects, BioNeMo sub-projects are designed to be hosted on the Python Package Index\footnote{More details available at https://pypi.org/} (PyPI). Docker images enable rapid development, allowing users to set up a complete environment quickly. Internal tools within the repository further assist users in creating and integrating their own Python projects into workflows, streamlining development.
BioNeMo Framework currently supports building large-scale BERT-based models through NVIDIA NeMo Megatron, a scalable framework for developing custom LLMs, multimodal, and speech AI. BioNeMo Framework uses NeMo Megatron to build biomolecular BERT model support.
\section{Functionalities enabled through BioNeMo Framework}
\subsection{BERT Model Architecture Support}
Two example BERT implementations -- ESM-2 for protein sequence modeling and Geneformer for single-cell expression modeling -- are currently available in the BioNeMo Framework codebase. The ESM-2 model is packaged into the submodule \textit{bionemo-esm2}, and the Geneformer model is packaged into an independent submodule \textit{bionemo-geneformer}.
Users can install and use either of these packages off-the-shelf and perform training, fine-tuning, and inference with custom data. Each sub-package has its own independent \textit{src/} directory, which hosts data modules, data tokenizers, and model implementations. Customizing the models can be as simple as making a copy of the sub-package and swapping out the modules, such as data loaders, with custom implementations.
Users can also easily customize loss functions, change different layers in the model architecture, etc., using the reference packages as examples.
One of BioNeMo Framework's most significant advantages, beyond its ease of use and customizability, is its seamless, ready-to-use scalability. With built-in support for data, model, tensor, and pipeline parallelism, scaling models across devices or increasing the number of parameters is simplified. Users only need to adjust the appropriate configuration parameters, requiring minimal or no changes to their code.
\subsection{Benchmarks and Results}
\begin{figure}
\centering
\includegraphics[width=0.5\linewidth]{Figure1.eps}
\caption{The BioNeMo Framework improves single-node memory usage and throughput compared to a standard PyTorch implementation. The ESM-2 650M parameter model trained with Hugging Face Accelerate using \textit{torch.compile} (gray, ``Baseline'') reached a maximum batch size of 16 on an NVIDIA 80GB A100 before Out of Memory (OOM) errors were encountered. The equivalent model trained with the BioNeMo Framework (green) reached a maximum batch size of 46 before OOM errors.}
\label{esm2-batch-scale}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=1\linewidth]{Figure2.eps}
\caption{The BioNeMo Framework enables higher-throughput training across a range of device types and model sizes. The ESM-2 650M or 3B parameter model trained with Hugging Face Accelerate using \textit{torch.compile} (gray, "Baseline") is compared to the equivalent model trained with the BioNeMo Framework (green). Results are shown using the largest possible batch size for each model and framework and training on 16 NVIDIA A100s or H100s split over two compute nodes.}
\label{esm2-device-scale}
\end{figure}
To assess the performance of the BioNeMo Framework, we compared training speeds for the ESM-2 model at various sizes and in different parallel configurations. First, we looked at single-device throughput for the 33-layer, 650M parameter variant. Benchmark speeds for the model as implemented in the Hugging Face Transformers library (4.44.0) were obtained using the Accelerate training library (0.33.0), using \textit{torch.compile’s} default dynamo backend, running inside the NVIDIA Deep Graph Library (DGL) PyTorch container\footnote{Access via \href{http://nvcr.io/nvidia/pytorch:24.07-py3}{nvcr.io/nvidia/pytorch:24.07-py3}}, PyTorch version 2.4.0). Training was performed on a single NVIDIA A100 GPU. Batches were increased until a CUDA out-of-memory (OOM) error occurred after a batch size of 16, with the resulting training throughput at each batch size shown in Figure \ref{esm2-batch-scale}. The BioNeMo Framework training throughputs were obtained on the same hardware with version 2.0 of the framework container. A maximum batch size of 46 was possible before an OOM error occurred, with an overall single-device training throughput 1.47x faster than that obtained with the Transformers library. BioNeMo Framework achieved a 59.2\% model flops utilization (MFU) \cite{Chowdhery2022-hg}, while the model trained with the Accelerate library reached an MFU of 40.1\%.
Next, we quantified how BioNeMo Framework performed in distributed training, at larger model sizes, and on H100 GPUs in addition to A100 GPUs. When training on 16 devices split across two compute nodes, the acceleration observed over the baseline training examples remained consistent. Training the 3B parameter variant of ESM-2 yields an MFU of 62.3\% for BioNeMo Framework on A100 GPUs, which drops to 49.1\% on the H100 system (a speedup of only 2.43x compared to the theoretical maximum of 3x). Hugging Face models were trained using DeepSpeed (0.15.1) with ZeRO level 1 using distributed data parallel \cite{Rajbhandari2019-wp}.
To determine how efficiently model training can be accelerated through parallelization, we looked at training speed for the 3B parameter model as we increased the number of GPUs. Each library's single-node (8 A100 GPUs) training throughput was used as a reference for perfect ‘linear’ scalability. Next, we measured training throughput at 2 nodes (16 GPUs), 4 nodes (32 GPUs), 8 nodes (64 GPUs), 16 nodes (128 GPUs), and finally 32 nodes (256 GPUs), as shown in Figure \ref{esm2-device-scale}. The BioNeMo Framework achieved a training speed at 256 GPUs that was 96.9\% of the extrapolated single-node throughput corresponding to a 60\% MFU and a total time to train over 1 trillion tokens of just 4.2 days. The models trained using DeepSpeed reached 80\% of their extrapolated single-node performance, with a total time to train over 1 trillion tokens of 8.8 days.
\subsection{BioNeMo Single-Cell Data Loader}
BioNeMo-SCDL provides an independent PyTorch-compatible dataset class for single-cell data with a consistent API. This package can be run independently from BioNeMo. It is designed to accelerate the training of transformer models with single-cell data. A data loader utilizing the Single-Cell Data Loader (SCDL) interface provides noticeable acceleration over comparable AnnData Loaders. SCDL does not require loading data into memory, however, it runs 1.1 to 2.75 times faster than a comparable AnnData loader. SCDL is also easy to customize and currently includes rank-order transformations for seamless integration into training Geneformer-like single-cell foundational models. It also supports pulling in user-defined metadata from the AnnData source files, which makes it highly adaptable for fine-tuning workflows. SCDL does all this by efficiently storing and accessing data from compressed sparse matrices, which are an integral part of single-cell datasets, using NumPy's memory-mapped format. This allows it to handle arbitrarily large datasets rapidly, unlike other single-cell data loading approaches. Currently, SCDL supports conversion from the AnnData format, but the conversion code is modular to facilitate the development of other custom converters in the future. BioNeMo SCDL's API resembles that of AnnData, making it is easy to use for practitioners familiar with the AnnData API.
\begin{figure}
\centering
\includegraphics[width=0.5\linewidth]{Figure3.eps}
\caption{Dataset iteration time for a data loader using SCDL and an AnnData data loader. SCDL shows a consistent speed-up. }
\label{fig:scdl}
\end{figure}
\subsection{WebDataModule}
The WebDataset is a popular Python library for high-performance data loading in training neural network models. It implements stream processing of input data with parallel data access via data sharding. It allows streaming data from local storage as well as cloud storage endpoints\footnote{https://github.com/webdataset/webdataset}. On the other hand, the \textit{LightningDatamodule} is the standard way of preparing, loading, and presenting data to the training workflow in the Lightning framework. A customized \textit{LightningDatamodule} requires user implementation of a few Lightning APIs, which could be cumbersome in the case of \textit{WebDataset}. In particular, WebDataset requires a specific file name scheme in the input data shards. It also requires a specific type of Python generator chained together to form the data transformation used in the dataset and data loader. To make it easy for the user to leverage WebDataset, we implemented the \textit{webdatamodule} in the BioNeMo Framework to create LightningDatamodule specifically from WebDataset. There are two components in our WebDataModule: 1) a general ``WebDataModule'' class. This is a subclass of the \textit{LightningDatamodule} and maps various WebDataset inputs, e.g., the data shard directory and the data file naming scheme, to the corresponding LightningDatamodule setup. 2) a ``PickledDataWDS'' that inherits from the WebDataModuleclass mentioned above with the additional built-in utilities to create the WebDataset data shards from Python pickle data objects. Here we refer to the BioNeMo Framework documentation for the webdatamodule usage and examples.
\subsection{Size-aware Batching}
Many applications of neural network models require processing data of varying dimensions. Especially in the case of GNNs, the input data exhibit variations in their graph topologies, which are defined by the structure of their node sets and edge sets. In drug discovery and molecular modeling, graphs can represent the molecular bond structure of proteins or small molecules and the general spatial interactions among the atoms or higher-level chemical structures \cite{David2020-gz}. Instead of processing the data samples individually, processing them in batches is often more efficient for better hardware utilization via parallel data processing and for more stable gradient estimation during training, which can reduce training times. Perhaps the most obvious way of batching graphs is via padding, where a multi-dimensional array represents certain graph properties, with the leading dimension being the batch dimension, i.e., indexing the samples in the batch, while the remaining dimensions represent the inherent feature dimensions of the graph property in question. Padding is typically required in this scheme so that slicing along the batch dimension results in feature arrays of the same size. At the cost of extra memory space -- and perhaps extra processing time due to the padding values -- the padded array representation is convenient in indexing or slicing the samples, and it is usually machine learning framework friendly where a large number of arithmetic operations have been developed to work on dense arrays. A more memory-efficient approach to batching graph data leverages a graph adjacency matrix’s sparse representation to collate multiple graphs into a disjoint meta-graph, where sparse arrays represent the meta-graph itself. This approach typically concatenates samples’ array of a graph property, e.g., node features, into a single array and can discern the operations applied on different graphs in the batch via auxiliary data like an array that masks the individual graphs in the concatenation. No padding is required, and efficient algorithms have been developed to parallelize the sparse batch of samples subgraph sampling and message-passing techniques, among others. This approach is widely used in GNN frameworks such as PyG \cite{Fey2019-hu} and DGL \cite{Wang2019-ch}. While this approach is memory efficient and often saves unnecessary computation compared to the padding approach, it can become unintuitive as to what utility functions can be used for certain operations and what the time and space complexities are for those operations as a function of the input data size.
In the BioNeMo Framework, we have developed tools to help address memory utilization issues often arising from batching data of varying dimensions. Specifically for the application of GNN model training, we have developed the \textit{size-aware batching} Python module to make GPU memory utilization more efficient. There are two main components in the size-aware batching module: a size-aware batcher and a bucket batch sampler.
\subsubsection{Size-aware batcher}
As mentioned previously, mini-batching is essential in training neural networks. In a static batch size scheme, where the same number of samples is batched on every training step, the mini-batch size is often deemed a critical hyper-parameter to be tuned, considering the tradeoff between memory consumption and training efficiency. In the case of GNN training using the sparse mini-batch approach, it is often challenging to compute the relationship between batch size and the resulting memory consumption due to the inherent heterogeneous computational requirements among the samples. Too small a batch size can lead to volatile gradient estimation that can lower the training efficiency, and too big a batch size can often lead to aborting the entire batch due to out-of-memory error. To address this problem, we have developed the size-aware batcher module to adjust the batch size in real-time during training to maintain a relatively constant level of GPU memory consumption. There are three critical steps in the size-aware batcher: (1) a utility function called \textit{``collect\_cuda\_peak\_alloc''}that collects CUDA memory allocation for a given workflow and can optionally output some features the user may want to use in fitting or profiling the CUDA memory usage; (2) a user implementation of memory allocation model that can be trained to predict memory consumption of the workflow based on the features as described in the previous steps; and (3) A Python Iterable that loops over an input sequence of data samples and outputs mini-batches of samples, ensuring none of the batches exceed CUDA memory capacity by accumulating the memory consumption requirements from each sample based on the memory predictor from step (2). While the Python Iterable in step (3) is designed to work directly in PyTorch iterable-style dataset, a size-aware batch sampler is also implemented to make it easy to adapt the size-aware batching method in PyTorch’s indexable or map-style dataset.
\subsubsection{Bucket batch sampler}
In the padding-based mini-batching approach, minimizing the amount of padding in creating the data property arrays is often desirable. Under the assumption that sufficient data samples exist in each subset of data size, the padding minimization problem can be solved by mini-batching data of similar sizes. We have implemented the bucket batch sampler, a specialization of the PyTorch \textit{BatchSampler}, that bins the data samples into buckets of desirable sizes and returns the resulting indices to the data sample in each bucket. The size measurement of each data sample and the lower and upper boundaries of each bucket are taken from user input. We have also provided a utility function \textit{``create\_buckets''} to help build buckets given input constraints, such as their maximal width and the minimal number of samples each can contain. Smaller bucket width translates into more homogeneous data sizes in the resulting mini-batches, while the minimal sample count for each bucket is a fuzzy way to control the bucket size.
\subsubsection{Benchmarks and results}
Here we demonstrate the efficiency of size-aware batching in the GEOM-Drug data \cite{Axelrod2022-lo} by comparing three different data loading methods: the baseline data loading with static batch size, the MiDi data loader used by the Megalodon model, which implements an adaptive batch size \cite{megaloden2024}, and our data loader using the size-aware batcher that takes the input samples from the buckets of the bucket batch sampler. We generate as many batches as 10 times the dataset size to simulate 10 training epochs using each method. We then examine these batches of data in terms of their data size or molecular size as well as the amount of padding involved. \ref{fig:mol-size} shows the probability of samples of different sizes in the GEOM-Drug dataset. The MiDi and the baseline static size batching often undersample the moderate to large data for different reasons. The large samples are skipped in the baseline method due to OOM error. In the MiDi method, sampling is biased towards the most prominent sizes by the design of the method. The bucket size-aware batching gives a uniform distribution across a broad spectrum of data sizes, i.e., it does not introduce sampling bias.
\begin{figure}
\centering
\includegraphics[width=0.5\linewidth]{Figure4.eps}
\caption{Shown here is the probability of samples of different sizes in the GEOM-Drug dataset. The MiDi and the baseline static size batching often undersample the moderate to large data for different reasons. The large samples are skipped in the baseline method due to OOM error. In the MiDi method, sampling is biased towards the most prominent sizes by the design of the method. The bucket size-aware batching gives a uniform distribution across a wide spectrum of data sizes, i.e., it does not introduce sampling bias.}
\label{fig:mol-size}
\end{figure}
We then compare the amount of memory padding required to generate data batches from the three methods in question. In \ref{fig:batch-pad}, we create a histogram with the numerical elements required for padding the batches. The bucket size-aware batching requires very little padding in most batches, while the MiDi and baseline methods require moderate and large amounts of padding, respectively.
\begin{figure}
\centering
\includegraphics[width=0.5\linewidth]{Figure5.eps}
\caption{A histogram of the numerical elements required for padding the batches. The bucket size-aware batching requires very little padding in most batches while the MiDi and baseline methods require moderate and large amounts of padding respectively.}
\label{fig:batch-pad}
\end{figure}
\section{Discussion}
\subsection{Contributions from the Research Community}
As open-source software, the BioNeMo Framework is a community-driven platform that enables users to contribute across various biomolecular domains and optimizes performance on GPU hardware. There have already been many contributions, from protein design workflows to DNA data loading, derived from real-world applications in drug discovery.
Dyno Therapeutics, a biotechnology company applying AI to discover novel gene therapy vectors with transformative delivery properties, integrated parameter-efficient fine-tuning to the BioNeMo Framework's ESM-2 implementation by enabling freezing of specific layers and only fine-tuning the most relevant parts of the network. Traditional fine-tuning requires updating the entire model, which can be costly and inefficient. In contrast, this approach reduces the risk of overfitting and significantly speeds up training, making it more resource-efficient.
Flagship Pioneering is harnessing AI to drive greater innovation in the life sciences through their initiative Flagship Intelligence, which added protein inverse folding model RL-DIF\cite{ektefaie2024reinforcementlearningstructureconditionedcategorical} to the BioNeMo Framework. Unlike traditional protein folding models, which predict a structure from a given sequence, RL-DIF does the reverse: it generates diverse sequences that can fold into a specific target structure, allowing researchers to design new proteins that fit precise structural requirements and opening vast possibilities in drug discovery and biotechnology.
Relation Therapeutics leverages AI with a Lab-in-the-Loop approach to deepen our understanding of disease biology and therapeutic discovery and has added a new DNADL data loading tool to the BioNeMo Framework for simplifying data loading of common genomics data formats. This allows users to easily develop AI models for DNA sequences, including enabling the training of models on specific genome regions or sequences with known variants for more precise insights.
Finally, Weights \& Biases, the leading AI developer platform for experiment management when training and fine-tuning models, has also integrated their tooling into BioNeMo Framework to automatically enable experiment management for users, tracking training runs across restarts and help users monitor their experiments more effectively.
\subsection{Scaling BioNeMo Framework Training with Cloud Computing Resources}
The scalability of cloud services allows BioNeMo Framework to support a wide range of project requirements and large datasets without the need for extensive on-site infrastructure. This environment enables teams to access and customize pre-trained models, making it feasible to conduct proof-of-concept studies before deployment to a production environment. For example, BioNeMo Framework is fully compatible with Amazon Web Services (AWS) infrastructure, providing scalable compute resources, including EC2 instances with NVIDIA GPUs, to meet the computational demands of training. AWS storage solutions, such as S3, enable data access and management throughout the training process. This cloud-based setup supports iterative model refinement, eliminating the need for on-premise hardware and accelerating the development of models tailored to specific research objectives in biopharma.
A-Alpha Bio, a biotechnology company specializing in protein-protein interaction (PPI) prediction, has used the BioNeMo Framework on AWS to enhance its computational pipeline for drug discovery. By deploying BioNeMo on Amazon EC2 P5 instances equipped with NVIDIA H100 Tensor Core GPUs, A-Alpha Bio achieved a 12-fold increase in inference speed and processed over 108 million inference calls in two months — significantly higher than their initial target. The increase in inference capacity enabled the evaluation of 10 times more protein binding predictions, thereby expanding the scope of potential drug candidates under consideration. Integrating BioNeMo with AWS Batch allowed A-Alpha Bio to scale their computational workflows efficiently. By using AWS's capacity reservation options, such as EC2 Capacity Blocks, they secured predictable access to high-performance compute resources. This setup enabled iterative model fine-tuning and optimization, reducing the need for multiple experimental cycles, which lowered overall costs and accelerated the process of therapeutic design.
Using the BioNeMo Framework also facilitated A-Alpha Bio's exploration of a wider mutational landscape, improving the likelihood of identifying high-affinity binders for therapeutic applications. This approach reduced the number of wet-lab experiments required in the design-build-test cycle, allowing them to shift more of their exploration to computational testing. Consequently, they were able to explore more complex protein modifications, statistically increasing their chances of discovering potent therapeutic candidates.
As part of this effort, A-Alpha Bio also contributed their notebook for zero-shot protein design with ESM-2 back to the BioNeMo Framework codebase, now available for other users to deploy. This leverages masked language modeling to predict likely amino acid substitutions at specific positions, empowering researchers to efficiently design and optimize proteins for anyone aiming to perform fine protein perturbations for downstream applications.
\section{Conclusion}
The BioNeMo Framework is a significant step in enabling drug discovery workflows through a robust, open-source ecosystem of GPU-accelerated tools. With its demonstrated capability in training throughput and scalability, coupled with its modular and extensible design, BioNeMo Framework has the potential to enable further advancements across diverse biomolecular domains. In the future, the BioNeMo Framework team plans to continue refining the framework APIs to facilitate modularity, component-level reusability, customization, and experimentation. We aim to support our users in being able to contribute framework-level improvements back to the project while independently sharing and maintaining packages that make use of the framework. We see this as an evolving collection of sub-packages that may grow or shrink as improved abstractions become available. We aim to prioritize workflows that demonstrate our framework’s strengths, namely the ability to train, infer, and fine-tune with very large models that do not fit in single devices. We will focus future efforts on pushing the envelope with state-of-the-art performance in these settings. As BioNeMo uses other actively used frameworks within NVIDIA, including NeMo, Megatron, and Transformer Engine, it will continue to benefit from performance and usability improvements across the NVIDIA software ecosystem.
%Bibliography
%\bibliographystyle{unsrt}
%\bibliography{references}
\printbibliography
\end{document}
% $ biblatex auxiliary file $
% $ biblatex bbl format version 3.3 $
% Do not modify the above lines!
%
% This is an auxiliary file used by the 'biblatex' package.
% This file may safely be deleted. It will be recreated by
% biber as required.
%
\begingroup
\makeatletter
\@ifundefined{ver@biblatex.sty}
{\@latex@error
{Missing 'biblatex' package}
{The bibliography requires the 'biblatex' package.}
\aftergroup\endinput}
{}
\endgroup
\refsection{0}
\datalist[entry]{nty/global//global/global/global}
\entry{megaloden2024}{inproceedings}{}{}
\name{author}{1}{}{%
{{hash=7079c72c21415131774625ba1d64f4b0}{%
family={Anonymous},
familyi={A\bibinitperiod}}}%
}
\strng{namehash}{7079c72c21415131774625ba1d64f4b0}
\strng{fullhash}{7079c72c21415131774625ba1d64f4b0}
\strng{fullhashraw}{7079c72c21415131774625ba1d64f4b0}
\strng{bibnamehash}{7079c72c21415131774625ba1d64f4b0}
\strng{authorbibnamehash}{7079c72c21415131774625ba1d64f4b0}
\strng{authornamehash}{7079c72c21415131774625ba1d64f4b0}
\strng{authorfullhash}{7079c72c21415131774625ba1d64f4b0}
\strng{authorfullhashraw}{7079c72c21415131774625ba1d64f4b0}
\field{sortinit}{A}
\field{sortinithash}{2f401846e2029bad6b3ecc16d50031e2}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{Submitted to The Thirteenth International Conference on Learning Representations}
\field{note}{under review}
\field{title}{Applications of Modular Co-Design for De Novo 3D Molecule Generation}
\field{year}{2024}
\verb{urlraw}
\verb https://openreview.net/forum?id=9UoBuhVNh6
\endverb
\verb{url}
\verb https://openreview.net/forum?id=9UoBuhVNh6
\endverb
\endentry
\entry{Axelrod2022-lo}{article}{}{}
\name{author}{2}{}{%
{{hash=3887997daa394693492e845aa8eecde5}{%
family={Axelrod},
familyi={A\bibinitperiod},
given={Simon},
giveni={S\bibinitperiod}}}%
{{hash=a37688f976e048ffbc85a45699635c91}{%
family={Gómez-Bombarelli},
familyi={G\bibinithyphendelim B\bibinitperiod},
given={Rafael},
giveni={R\bibinitperiod}}}%
}
\list{language}{1}{%
{en}%
}
\list{publisher}{2}{%
{Springer Science}%
{Business Media LLC}%
}
\strng{namehash}{bf047b1d43790498e21978bc676fe405}
\strng{fullhash}{bf047b1d43790498e21978bc676fe405}
\strng{fullhashraw}{bf047b1d43790498e21978bc676fe405}
\strng{bibnamehash}{bf047b1d43790498e21978bc676fe405}
\strng{authorbibnamehash}{bf047b1d43790498e21978bc676fe405}
\strng{authornamehash}{bf047b1d43790498e21978bc676fe405}
\strng{authorfullhash}{bf047b1d43790498e21978bc676fe405}
\strng{authorfullhashraw}{bf047b1d43790498e21978bc676fe405}
\field{sortinit}{A}
\field{sortinithash}{2f401846e2029bad6b3ecc16d50031e2}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Machine learning (ML) outperforms traditional approaches in many molecular design tasks. ML models usually predict molecular properties from a 2D chemical graph or a single 3D structure, but neither of these representations accounts for the ensemble of 3D conformers that are accessible to a molecule. Property prediction could be improved by using conformer ensembles as input, but there is no large-scale dataset that contains graphs annotated with accurate conformers and experimental data. Here we use advanced sampling and semi-empirical density functional theory (DFT) to generate 37 million molecular conformations for over 450,000 molecules. The Geometric Ensemble Of Molecules (GEOM) dataset contains conformers for 133,000 species from QM9, and 317,000 species with experimental data related to biophysics, physiology, and physical chemistry. Ensembles of 1,511 species with BACE-1 inhibition data are also labeled with high-quality DFT free energies in an implicit water solvent, and 534 ensembles are further optimized with DFT. GEOM will assist in the development of models that predict properties from conformer ensembles, and generative models that sample 3D conformations.}
\field{journaltitle}{Sci. Data}
\field{month}{4}
\field{number}{1}
\field{title}{{GEOM}, energy-annotated molecular conformations for property prediction and molecular generation}
\field{volume}{9}
\field{year}{2022}
\field{pages}{185}
\range{pages}{1}
\endentry
\entry{behjati2013next}{article}{}{}
\name{author}{2}{}{%
{{hash=8876f8b8fad9468386c44f48b8452db0}{%
family={Behjati},
familyi={B\bibinitperiod},
given={Sam},
giveni={S\bibinitperiod}}}%
{{hash=4fca9b902d4734317c5c891516a9b462}{%
family={Tarpey},
familyi={T\bibinitperiod},
given={Patrick\bibnamedelima S},
giveni={P\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
}
\list{publisher}{2}{%
{Royal College of Paediatrics}%
{Child Health}%
}
\strng{namehash}{fe11bf35490294f81e967e4a4bb664a1}
\strng{fullhash}{fe11bf35490294f81e967e4a4bb664a1}
\strng{fullhashraw}{fe11bf35490294f81e967e4a4bb664a1}
\strng{bibnamehash}{fe11bf35490294f81e967e4a4bb664a1}
\strng{authorbibnamehash}{fe11bf35490294f81e967e4a4bb664a1}
\strng{authornamehash}{fe11bf35490294f81e967e4a4bb664a1}
\strng{authorfullhash}{fe11bf35490294f81e967e4a4bb664a1}
\strng{authorfullhashraw}{fe11bf35490294f81e967e4a4bb664a1}
\field{sortinit}{B}
\field{sortinithash}{d7095fff47cda75ca2589920aae98399}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Archives of Disease in Childhood-Education and Practice}
\field{number}{6}
\field{title}{What is next generation sequencing?}
\field{volume}{98}
\field{year}{2013}
\field{pages}{236\bibrangedash 238}
\range{pages}{3}
\endentry
\entry{bernhofer2021predictprotein}{article}{}{}
\true{moreauthor}
\true{morelabelname}
\name{author}{10}{}{%
{{hash=2343292bcf82a0a3b3ccefe7940daa9b}{%
family={Bernhofer},
familyi={B\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=381edeaec659df830008a0b2d8646834}{%
family={Dallago},
familyi={D\bibinitperiod},
given={Christian},
giveni={C\bibinitperiod}}}%
{{hash=264297d94afe70d3c26d466124fff157}{%
family={Karl},
familyi={K\bibinitperiod},
given={Tim},
giveni={T\bibinitperiod}}}%
{{hash=ea12676a0d370ce8b1214694e97b61b5}{%
family={Satagopam},
familyi={S\bibinitperiod},
given={Venkata},
giveni={V\bibinitperiod}}}%
{{hash=c04fca9e06da18cb002f115133913d24}{%
family={Heinzinger},
familyi={H\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=5c357491f0feef183b4870354e7c1093}{%
family={Littmann},
familyi={L\bibinitperiod},
given={Maria},
giveni={M\bibinitperiod}}}%
{{hash=f7db09e1ac73053f0897a18a5032094c}{%
family={Olenyi},
familyi={O\bibinitperiod},
given={Tobias},
giveni={T\bibinitperiod}}}%
{{hash=09028ca70d6a9b25a0f9b42a36c60b86}{%
family={Qiu},
familyi={Q\bibinitperiod},
given={Jiajun},
giveni={J\bibinitperiod}}}%
{{hash=e51a34288860233b0844f4617a90b3c1}{%
family={Schütze},
familyi={S\bibinitperiod},
given={Konstantin},
giveni={K\bibinitperiod}}}%
{{hash=7b63b4d7f6c5d9f81101fd69ee465574}{%
family={Yachdav},
familyi={Y\bibinitperiod},
given={Guy},
giveni={G\bibinitperiod}}}%
}
\list{publisher}{1}{%
{Oxford University Press}%
}
\strng{namehash}{835f92fe25077285df4cbfd71d6b42cf}
\strng{fullhash}{3c384880d11b825fb3ff31e15dcafa78}
\strng{fullhashraw}{3c384880d11b825fb3ff31e15dcafa78}
\strng{bibnamehash}{835f92fe25077285df4cbfd71d6b42cf}
\strng{authorbibnamehash}{835f92fe25077285df4cbfd71d6b42cf}
\strng{authornamehash}{835f92fe25077285df4cbfd71d6b42cf}
\strng{authorfullhash}{3c384880d11b825fb3ff31e15dcafa78}
\strng{authorfullhashraw}{3c384880d11b825fb3ff31e15dcafa78}
\field{sortinit}{B}
\field{sortinithash}{d7095fff47cda75ca2589920aae98399}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Nucleic acids research}
\field{number}{W1}
\field{title}{PredictProtein-predicting protein structure and function for 29 years}
\field{volume}{49}
\field{year}{2021}
\field{pages}{W535\bibrangedash W540}
\range{pages}{-1}
\endentry
\entry{Boiko2023-mj}{article}{}{}
\name{author}{4}{}{%
{{hash=f2a914d7070929581c5e0bbdb665c2c1}{%
family={Boiko},
familyi={B\bibinitperiod},
given={Daniil\bibnamedelima A},
giveni={D\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=bfa021f99c97bfde54db2e3b478565a7}{%
family={MacKnight},
familyi={M\bibinitperiod},
given={Robert},
giveni={R\bibinitperiod}}}%
{{hash=60d2ca932460740bc469696c0b4e1518}{%
family={Kline},
familyi={K\bibinitperiod},
given={Ben},
giveni={B\bibinitperiod}}}%
{{hash=b9f7e54a748919899c2ae410a72c5c04}{%
family={Gomes},
familyi={G\bibinitperiod},
given={Gabe},
giveni={G\bibinitperiod}}}%
}
\list{language}{1}{%
{en}%
}
\strng{namehash}{61b9ad707de951530d5d861b519f9f39}
\strng{fullhash}{a9df9a7c5bc62c4b315991945c7643f0}
\strng{fullhashraw}{a9df9a7c5bc62c4b315991945c7643f0}
\strng{bibnamehash}{61b9ad707de951530d5d861b519f9f39}
\strng{authorbibnamehash}{61b9ad707de951530d5d861b519f9f39}
\strng{authornamehash}{61b9ad707de951530d5d861b519f9f39}
\strng{authorfullhash}{a9df9a7c5bc62c4b315991945c7643f0}
\strng{authorfullhashraw}{a9df9a7c5bc62c4b315991945c7643f0}
\field{sortinit}{B}
\field{sortinithash}{d7095fff47cda75ca2589920aae98399}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Transformer-based large language models are making significant strides in various fields, such as natural language processing1-5, biology6,7, chemistry8-10 and computer programming11,12. Here, we show the development and capabilities of Coscientist, an artificial intelligence system driven by GPT-4 that autonomously designs, plans and performs complex experiments by incorporating large language models empowered by tools such as internet and documentation search, code execution and experimental automation. Coscientist showcases its potential for accelerating research across six diverse tasks, including the successful reaction optimization of palladium-catalysed cross-couplings, while exhibiting advanced capabilities for (semi-)autonomous experimental design and execution. Our findings demonstrate the versatility, efficacy and explainability of artificial intelligence systems like Coscientist in advancing research.}
\field{journaltitle}{Nature}
\field{month}{12}
\field{number}{7992}
\field{title}{Autonomous chemical research with large language models}
\field{volume}{624}
\field{year}{2023}
\field{pages}{570\bibrangedash 578}
\range{pages}{9}
\endentry
\entry{jax2018github}{software}{}{}
\name{author}{11}{}{%
{{hash=b75383e6b48c8360c7a60031424c85cf}{%
family={Bradbury},
familyi={B\bibinitperiod},
given={James},
giveni={J\bibinitperiod}}}%
{{hash=29563c986154ca2b45d286b4dd5ef92a}{%
family={Frostig},
familyi={F\bibinitperiod},
given={Roy},
giveni={R\bibinitperiod}}}%
{{hash=7ab2d7e7f2bfceec36e42ad1962fde11}{%
family={Hawkins},
familyi={H\bibinitperiod},
given={Peter},
giveni={P\bibinitperiod}}}%
{{hash=e4d515e534ba184f553d5a2a926e0ea7}{%
family={Johnson},
familyi={J\bibinitperiod},
given={Matthew\bibnamedelima James},
giveni={M\bibinitperiod\bibinitdelim J\bibinitperiod}}}%
{{hash=0101ae5b219809cf901e7645ca40b9ae}{%
family={Leary},
familyi={L\bibinitperiod},
given={Chris},
giveni={C\bibinitperiod}}}%
{{hash=07e607fb522f06610feb023ecfa712e2}{%
family={Maclaurin},
familyi={M\bibinitperiod},
given={Dougal},
giveni={D\bibinitperiod}}}%
{{hash=bca7d753be3217d63404d860acb6e5f5}{%
family={Necula},
familyi={N\bibinitperiod},
given={George},
giveni={G\bibinitperiod}}}%
{{hash=56bf0b340039cf8594436a624ff548a9}{%
family={Paszke},
familyi={P\bibinitperiod},
given={Adam},
giveni={A\bibinitperiod}}}%
{{hash=0fd9a0e34f1b2adda41357c948d14986}{%
family={Vander{P}las},
familyi={V\bibinitperiod},
given={Jake},
giveni={J\bibinitperiod}}}%
{{hash=02f809ed2547f721c304772c06378c2d}{%
family={Wanderman-{M}ilne},
familyi={W\bibinithyphendelim M\bibinitperiod},
given={Skye},
giveni={S\bibinitperiod}}}%
{{hash=fe167f78433aabd363ac3c06710a1945}{%
family={Zhang},
familyi={Z\bibinitperiod},
given={Qiao},
giveni={Q\bibinitperiod}}}%
}
\strng{namehash}{be9c2f36b976223518e841214253df92}
\strng{fullhash}{12f83c04441cfb40a3c9b086929c12bf}
\strng{fullhashraw}{12f83c04441cfb40a3c9b086929c12bf}
\strng{bibnamehash}{be9c2f36b976223518e841214253df92}
\strng{authorbibnamehash}{be9c2f36b976223518e841214253df92}
\strng{authornamehash}{be9c2f36b976223518e841214253df92}
\strng{authorfullhash}{12f83c04441cfb40a3c9b086929c12bf}
\strng{authorfullhashraw}{12f83c04441cfb40a3c9b086929c12bf}
\field{sortinit}{B}
\field{sortinithash}{d7095fff47cda75ca2589920aae98399}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{title}{{JAX}: composable transformations of {P}ython+{N}um{P}y programs}
\field{version}{0.3.13}
\field{year}{2018}
\verb{urlraw}
\verb http://github.com/jax-ml/jax
\endverb
\verb{url}
\verb http://github.com/jax-ml/jax
\endverb
\endentry
\entry{Chowdhery2022-hg}{article}{}{}
\name{author}{67}{}{%
{{hash=fd55b702c004c13506e70bb76a4e987c}{%
family={Chowdhery},
familyi={C\bibinitperiod},
given={Aakanksha},
giveni={A\bibinitperiod}}}%
{{hash=a2da348cbbad750c6fe76dda9c35ffcd}{%
family={Narang},
familyi={N\bibinitperiod},
given={Sharan},
giveni={S\bibinitperiod}}}%
{{hash=13202969e372bc82318f9629cbdd199b}{%
family={Devlin},
familyi={D\bibinitperiod},
given={Jacob},
giveni={J\bibinitperiod}}}%
{{hash=3fd5b56bfa48546f2caae7b4aa56a29f}{%
family={Bosma},
familyi={B\bibinitperiod},
given={Maarten},
giveni={M\bibinitperiod}}}%
{{hash=94c647b98d5a26af384caedf7f3c933b}{%
family={Mishra},
familyi={M\bibinitperiod},
given={Gaurav},
giveni={G\bibinitperiod}}}%
{{hash=243c03e5ae8031582d6af9fcbc70f629}{%
family={Roberts},
familyi={R\bibinitperiod},
given={Adam},
giveni={A\bibinitperiod}}}%
{{hash=08cf514c08137f94d73b590060797f5b}{%
family={Barham},
familyi={B\bibinitperiod},
given={Paul},
giveni={P\bibinitperiod}}}%
{{hash=10118dd89bd210596b82c1b6feb26faf}{%
family={Chung},
familyi={C\bibinitperiod},
given={Hyung\bibnamedelima Won},
giveni={H\bibinitperiod\bibinitdelim W\bibinitperiod}}}%
{{hash=fd59f6a0549332a54ac82d8840ea6ff1}{%
family={Sutton},
familyi={S\bibinitperiod},
given={Charles},
giveni={C\bibinitperiod}}}%
{{hash=80c831c617902a22d4fa4074948f3aff}{%
family={Gehrmann},
familyi={G\bibinitperiod},
given={Sebastian},
giveni={S\bibinitperiod}}}%
{{hash=e59fde4458205333c1a2baba12329b29}{%
family={Schuh},
familyi={S\bibinitperiod},
given={Parker},
giveni={P\bibinitperiod}}}%
{{hash=30ccfd41b610ee68f088bcfcf3a28368}{%
family={Shi},
familyi={S\bibinitperiod},
given={Kensen},
giveni={K\bibinitperiod}}}%
{{hash=e6aac14eea90afa28e8f35f25b0909ee}{%
family={Tsvyashchenko},
familyi={T\bibinitperiod},
given={Sasha},
giveni={S\bibinitperiod}}}%
{{hash=2eda226c5ef11a030c1a898ef763b643}{%
family={Maynez},
familyi={M\bibinitperiod},
given={Joshua},
giveni={J\bibinitperiod}}}%
{{hash=36a0ca08ebb0acddfbd61b67274f81da}{%
family={Rao},
familyi={R\bibinitperiod},
given={Abhishek},
giveni={A\bibinitperiod}}}%
{{hash=be9805a1e793fbd638411a4d76ecd398}{%
family={Barnes},
familyi={B\bibinitperiod},
given={Parker},
giveni={P\bibinitperiod}}}%
{{hash=7579658a0aca8cb0e17a89deb28b8ccc}{%
family={Tay},
familyi={T\bibinitperiod},
given={Yi},
giveni={Y\bibinitperiod}}}%
{{hash=62efade83d70f0323fe248755e6c90c5}{%
family={Shazeer},
familyi={S\bibinitperiod},
given={Noam},
giveni={N\bibinitperiod}}}%
{{hash=c18c48f3c746611d61b3517651ec5c07}{%
family={Prabhakaran},
familyi={P\bibinitperiod},
given={Vinodkumar},
giveni={V\bibinitperiod}}}%
{{hash=bf36a95184bcf06abe636f74e9bcdc70}{%
family={Reif},
familyi={R\bibinitperiod},
given={Emily},
giveni={E\bibinitperiod}}}%
{{hash=a926c34a0eebdaaa30da36b46bd2192c}{%
family={Du},
familyi={D\bibinitperiod},
given={Nan},
giveni={N\bibinitperiod}}}%
{{hash=f8663b0ce8cef0efd0317d72945209e8}{%
family={Hutchinson},
familyi={H\bibinitperiod},
given={Ben},
giveni={B\bibinitperiod}}}%
{{hash=d8be19e523fa307d44a1fddbff485f5c}{%
family={Pope},
familyi={P\bibinitperiod},
given={Reiner},
giveni={R\bibinitperiod}}}%
{{hash=b75383e6b48c8360c7a60031424c85cf}{%
family={Bradbury},
familyi={B\bibinitperiod},
given={James},
giveni={J\bibinitperiod}}}%
{{hash=9b0426325e91e9da0e91dbf27a7379f6}{%
family={Austin},
familyi={A\bibinitperiod},
given={Jacob},
giveni={J\bibinitperiod}}}%
{{hash=3b62e7eedeec9f7a7f7f3c3ec88637f1}{%
family={Isard},
familyi={I\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=4150a424a61199cfaf36bb11ca192b7b}{%
family={Gur-Ari},
familyi={G\bibinithyphendelim A\bibinitperiod},
given={Guy},
giveni={G\bibinitperiod}}}%
{{hash=2996beb75be77b26db18b01e40ad65fc}{%
family={Yin},
familyi={Y\bibinitperiod},
given={Pengcheng},
giveni={P\bibinitperiod}}}%
{{hash=05efb48d668f0b7eeb075a73f1d28570}{%
family={Duke},
familyi={D\bibinitperiod},
given={Toju},
giveni={T\bibinitperiod}}}%
{{hash=738351d7ac35edab0cd231cb64407ab4}{%
family={Levskaya},
familyi={L\bibinitperiod},
given={Anselm},
giveni={A\bibinitperiod}}}%
{{hash=193bcec5240237591ad8fb697869f013}{%
family={Ghemawat},
familyi={G\bibinitperiod},
given={Sanjay},
giveni={S\bibinitperiod}}}%
{{hash=c30a044c9daf83a028f234d3fc1fa150}{%
family={Dev},
familyi={D\bibinitperiod},
given={Sunipa},
giveni={S\bibinitperiod}}}%
{{hash=88b44aba31eead485bb3a63154798388}{%
family={Michalewski},
familyi={M\bibinitperiod},
given={Henryk},
giveni={H\bibinitperiod}}}%
{{hash=006d0e4e90ea3a3afbe8207b594b3737}{%
family={Garcia},
familyi={G\bibinitperiod},
given={Xavier},
giveni={X\bibinitperiod}}}%
{{hash=7e5089b1b1b0dd763ad19b53771227ac}{%
family={Misra},
familyi={M\bibinitperiod},
given={Vedant},
giveni={V\bibinitperiod}}}%
{{hash=4ad67c4643b792d3bb7d1a7e37af0f22}{%
family={Robinson},
familyi={R\bibinitperiod},
given={Kevin},
giveni={K\bibinitperiod}}}%
{{hash=806ceeb371e478d9f180febef772fb08}{%
family={Fedus},
familyi={F\bibinitperiod},
given={Liam},
giveni={L\bibinitperiod}}}%
{{hash=797f9a226fd7356db5edc6feba1a0e87}{%
family={Zhou},
familyi={Z\bibinitperiod},
given={Denny},
giveni={D\bibinitperiod}}}%
{{hash=f0b22a1a9ec5b2901f5e0cb457612e93}{%
family={Ippolito},
familyi={I\bibinitperiod},
given={Daphne},
giveni={D\bibinitperiod}}}%
{{hash=63b32614460a4ab72dbbbbfe223220ea}{%
family={Luan},
familyi={L\bibinitperiod},
given={David},
giveni={D\bibinitperiod}}}%
{{hash=4498aad690f2c6e0698b89a1d2dde41b}{%
family={Lim},
familyi={L\bibinitperiod},
given={Hyeontaek},
giveni={H\bibinitperiod}}}%
{{hash=91aafc93e68e8c6a55e8cd804c2acaf2}{%
family={Zoph},
familyi={Z\bibinitperiod},
given={Barret},
giveni={B\bibinitperiod}}}%
{{hash=6452adc1249c2abc94741a4113987a47}{%
family={Spiridonov},
familyi={S\bibinitperiod},
given={Alexander},
giveni={A\bibinitperiod}}}%
{{hash=7f45cbea23420eb9e29c75c69725248f}{%
family={Sepassi},
familyi={S\bibinitperiod},
given={Ryan},
giveni={R\bibinitperiod}}}%
{{hash=7369065ea0b748a65621d6b6197996ac}{%
family={Dohan},
familyi={D\bibinitperiod},
given={David},
giveni={D\bibinitperiod}}}%
{{hash=aea4a56bc73650d2a59fd044b9b80fde}{%
family={Agrawal},
familyi={A\bibinitperiod},
given={Shivani},
giveni={S\bibinitperiod}}}%
{{hash=90870f8d2c2dd318f093ef32f3b7b56a}{%
family={Omernick},
familyi={O\bibinitperiod},
given={Mark},
giveni={M\bibinitperiod}}}%
{{hash=978bf3b58698c55cce487279eb72f59f}{%
family={Dai},
familyi={D\bibinitperiod},
given={Andrew\bibnamedelima M},
giveni={A\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=2f95b48f49e1c9df6179a525a5c498a3}{%
family={Pillai},
familyi={P\bibinitperiod},
given={Thanumalayan\bibnamedelima Sankaranarayana},
giveni={T\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=077c25462b324153bc432beee5b95779}{%
family={Pellat},
familyi={P\bibinitperiod},
given={Marie},
giveni={M\bibinitperiod}}}%
{{hash=725df010acbaaf6e169430d58421a5ed}{%
family={Lewkowycz},
familyi={L\bibinitperiod},
given={Aitor},
giveni={A\bibinitperiod}}}%
{{hash=d5d5571648a28ab0a6b3f3742925b34c}{%
family={Moreira},
familyi={M\bibinitperiod},
given={Erica},
giveni={E\bibinitperiod}}}%
{{hash=5c2f5c2e6d4a9ec8681377f8a8e5e6af}{%
family={Child},
familyi={C\bibinitperiod},
given={Rewon},
giveni={R\bibinitperiod}}}%
{{hash=21029155fee7b53a534e92b5dca700ba}{%
family={Polozov},
familyi={P\bibinitperiod},
given={Oleksandr},
giveni={O\bibinitperiod}}}%
{{hash=bfca12eba160cc79988e61b7fb6ded42}{%
family={Lee},
familyi={L\bibinitperiod},
given={Katherine},
giveni={K\bibinitperiod}}}%
{{hash=335463264b30d83d055bbbda6e0ac19b}{%
family={Zhou},
familyi={Z\bibinitperiod},
given={Zongwei},
giveni={Z\bibinitperiod}}}%
{{hash=27d22d8f05f44b47f7dcfe9b7fb85584}{%
family={Wang},
familyi={W\bibinitperiod},
given={Xuezhi},
giveni={X\bibinitperiod}}}%
{{hash=7e8263bdfc0415de5b52a96d0cc15a5b}{%
family={Saeta},
familyi={S\bibinitperiod},
given={Brennan},
giveni={B\bibinitperiod}}}%
{{hash=0964763c1bedba75b847574ad1504a2f}{%
family={Diaz},
familyi={D\bibinitperiod},
given={Mark},
giveni={M\bibinitperiod}}}%
{{hash=199f475d073184300c78ea1ea7d001e1}{%
family={Firat},
familyi={F\bibinitperiod},
given={Orhan},
giveni={O\bibinitperiod}}}%
{{hash=4d8195891e857816c8f42626f464a3a8}{%
family={Catasta},
familyi={C\bibinitperiod},
given={Michele},
giveni={M\bibinitperiod}}}%
{{hash=61331a11b490f3d927555985facd25b3}{%
family={Wei},
familyi={W\bibinitperiod},
given={Jason},
giveni={J\bibinitperiod}}}%
{{hash=5d010242b58bad75a9d0574fdd339b63}{%
family={Meier-Hellstern},
familyi={M\bibinithyphendelim H\bibinitperiod},
given={Kathy},
giveni={K\bibinitperiod}}}%
{{hash=b2c4631ad93e93115d9b3ceea6587550}{%
family={Eck},
familyi={E\bibinitperiod},
given={Douglas},
giveni={D\bibinitperiod}}}%
{{hash=62f954323339abeba6c6a240e9d2855b}{%
family={Dean},
familyi={D\bibinitperiod},
given={Jeff},
giveni={J\bibinitperiod}}}%
{{hash=6d4804cc2b12eeb7ab41eb5ee03fc7ab}{%
family={Petrov},
familyi={P\bibinitperiod},
given={Slav},
giveni={S\bibinitperiod}}}%
{{hash=65440831849e813585a469e2dd81dca9}{%
family={Fiedel},
familyi={F\bibinitperiod},
given={Noah},
giveni={N\bibinitperiod}}}%
}
\strng{namehash}{b3fd9cbf2512d200b431ebc50e088d71}
\strng{fullhash}{ec52ff366b46bf821ecffbbf98f825aa}
\strng{fullhashraw}{ec52ff366b46bf821ecffbbf98f825aa}
\strng{bibnamehash}{b3fd9cbf2512d200b431ebc50e088d71}
\strng{authorbibnamehash}{b3fd9cbf2512d200b431ebc50e088d71}
\strng{authornamehash}{b3fd9cbf2512d200b431ebc50e088d71}
\strng{authorfullhash}{ec52ff366b46bf821ecffbbf98f825aa}
\strng{authorfullhashraw}{ec52ff366b46bf821ecffbbf98f825aa}
\field{sortinit}{C}
\field{sortinithash}{4d103a86280481745c9c897c925753c0}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{month}{4}
\field{title}{{PaLM}: Scaling language modeling with Pathways}
\field{year}{2022}
\verb{eprint}
\verb 2204.02311
\endverb
\endentry
\entry{David2020-gz}{article}{}{}
\name{author}{4}{}{%
{{hash=347dcd5f1a23a55a565e4a4c12d5220b}{%
family={David},
familyi={D\bibinitperiod},
given={Laurianne},
giveni={L\bibinitperiod}}}%
{{hash=84fb578e9f4d1b5d144725ac2f398690}{%
family={Thakkar},
familyi={T\bibinitperiod},
given={Amol},
giveni={A\bibinitperiod}}}%
{{hash=7d2332835bf59fb8333300e1d43024fe}{%
family={Mercado},
familyi={M\bibinitperiod},
given={Rocío},
giveni={R\bibinitperiod}}}%
{{hash=07fc5197917d83cd3df96202400b36bc}{%
family={Engkvist},
familyi={E\bibinitperiod},
given={Ola},
giveni={O\bibinitperiod}}}%
}
\strng{namehash}{a5b3719de9156f387c233ef326156dfe}
\strng{fullhash}{054e95cb9f52ff3387b7b9a0924d43ed}
\strng{fullhashraw}{054e95cb9f52ff3387b7b9a0924d43ed}
\strng{bibnamehash}{a5b3719de9156f387c233ef326156dfe}
\strng{authorbibnamehash}{a5b3719de9156f387c233ef326156dfe}
\strng{authornamehash}{a5b3719de9156f387c233ef326156dfe}
\strng{authorfullhash}{054e95cb9f52ff3387b7b9a0924d43ed}
\strng{authorfullhashraw}{054e95cb9f52ff3387b7b9a0924d43ed}
\field{sortinit}{D}
\field{sortinithash}{6f385f66841fb5e82009dc833c761848}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{The technological advances of the past century, marked by the computer revolution and the advent of high-throughput screening technologies in drug discovery, opened the path to the computational analysis and visualization of bioactive molecules. For this purpose, it became necessary to represent molecules in a syntax that would be readable by computers and understandable by scientists of various fields. A large number of chemical representations have been developed over the years, their numerosity being due to the fast development of computers and the complexity of producing a representation that encompasses all structural and chemical characteristics. We present here some of the most popular electronic molecular and macromolecular representations used in drug discovery, many of which are based on graph representations. Furthermore, we describe applications of these representations in AI-driven drug discovery. Our aim is to provide a brief guide on structural representations that are essential to the practice of AI in drug discovery. This review serves as a guide for researchers who have little experience with the handling of chemical representations and plan to work on applications at the interface of these fields.}
\field{journaltitle}{Journal of Cheminformatics}
\field{month}{9}
\field{number}{1}
\field{title}{Molecular representations in {AI-driven} drug discovery: a review and practical guide}
\field{volume}{12}
\field{year}{2020}
\field{pages}{56}
\range{pages}{1}
\endentry
\entry{ektefaie2024reinforcementlearningstructureconditionedcategorical}{misc}{}{}
\name{author}{6}{}{%
{{hash=9706737cdd9e6111fbc0ee5fdaa848c7}{%
family={Ektefaie},
familyi={E\bibinitperiod},
given={Yasha},
giveni={Y\bibinitperiod}}}%
{{hash=bd07079d17a38095649966fd73e12d9b}{%
family={Viessmann},
familyi={V\bibinitperiod},
given={Olivia},
giveni={O\bibinitperiod}}}%
{{hash=264ef12be9e6128fa4555de211600463}{%
family={Narayanan},
familyi={N\bibinitperiod},
given={Siddharth},
giveni={S\bibinitperiod}}}%
{{hash=55734e5d1f458f5ee13f98aae3745387}{%
family={Dresser},
familyi={D\bibinitperiod},
given={Drew},
giveni={D\bibinitperiod}}}%
{{hash=55e32a997f91b5550d5d929b07d4fa74}{%
family={Kim},
familyi={K\bibinitperiod},
given={J.\bibnamedelimi Mark},
giveni={J\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=05b94720eb1ee52c902a106df20af1da}{%
family={Mkrtchyan},
familyi={M\bibinitperiod},
given={Armen},
giveni={A\bibinitperiod}}}%
}
\strng{namehash}{82993c530f85986f9b34b0dd6f280369}
\strng{fullhash}{c9016f06f1bd6c9197ddb427437a9583}
\strng{fullhashraw}{c9016f06f1bd6c9197ddb427437a9583}
\strng{bibnamehash}{82993c530f85986f9b34b0dd6f280369}
\strng{authorbibnamehash}{82993c530f85986f9b34b0dd6f280369}
\strng{authornamehash}{82993c530f85986f9b34b0dd6f280369}
\strng{authorfullhash}{c9016f06f1bd6c9197ddb427437a9583}
\strng{authorfullhashraw}{c9016f06f1bd6c9197ddb427437a9583}
\field{sortinit}{E}
\field{sortinithash}{8da8a182d344d5b9047633dfc0cc9131}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.AI}
\field{eprinttype}{arXiv}
\field{title}{Reinforcement learning on structure-conditioned categorical diffusion for protein inverse folding}
\field{year}{2024}
\verb{eprint}
\verb 2410.17173
\endverb
\verb{urlraw}
\verb https://arxiv.org/abs/2410.17173
\endverb
\verb{url}
\verb https://arxiv.org/abs/2410.17173
\endverb
\endentry
\entry{elnaggar2021prottrans}{article}{}{}
\true{moreauthor}
\true{morelabelname}
\name{author}{10}{}{%
{{hash=ffe73c593e2a87fe360a38cbda274f31}{%
family={Elnaggar},
familyi={E\bibinitperiod},
given={Ahmed},
giveni={A\bibinitperiod}}}%
{{hash=c04fca9e06da18cb002f115133913d24}{%
family={Heinzinger},
familyi={H\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=381edeaec659df830008a0b2d8646834}{%
family={Dallago},
familyi={D\bibinitperiod},
given={Christian},
giveni={C\bibinitperiod}}}%
{{hash=f4f7b453c6235e072a10a092abf0ae74}{%
family={Rehawi},
familyi={R\bibinitperiod},
given={Ghalia},
giveni={G\bibinitperiod}}}%
{{hash=b2eccb185095fccfeb40fdffd8b2105f}{%
family={Wang},
familyi={W\bibinitperiod},
given={Yu},
giveni={Y\bibinitperiod}}}%
{{hash=2fd2982e30ebcec93ec1cf76e0d797fd}{%
family={Jones},
familyi={J\bibinitperiod},
given={Llion},
giveni={L\bibinitperiod}}}%
{{hash=ef99e5f5de2e799b18169a1945451fdd}{%
family={Gibbs},
familyi={G\bibinitperiod},
given={Tom},
giveni={T\bibinitperiod}}}%
{{hash=fe3319a4eb730f5484bfff0eb08493f4}{%
family={Feher},
familyi={F\bibinitperiod},
given={Tamas},
giveni={T\bibinitperiod}}}%
{{hash=6f148a1f82c1e04b4a3ef4350c3f0ad5}{%
family={Angerer},
familyi={A\bibinitperiod},
given={Christoph},
giveni={C\bibinitperiod}}}%
{{hash=c06ac6a6fdcf2ee2092d61fc58c1f21b}{%
family={Steinegger},
familyi={S\bibinitperiod},
given={Martin},
giveni={M\bibinitperiod}}}%
}
\list{publisher}{1}{%
{IEEE}%
}
\strng{namehash}{65ca48ebfe01fff7e9b3d9e91ff50edf}
\strng{fullhash}{9a4cdadc49e27a6b60a6426c4f007548}
\strng{fullhashraw}{9a4cdadc49e27a6b60a6426c4f007548}
\strng{bibnamehash}{65ca48ebfe01fff7e9b3d9e91ff50edf}
\strng{authorbibnamehash}{65ca48ebfe01fff7e9b3d9e91ff50edf}
\strng{authornamehash}{65ca48ebfe01fff7e9b3d9e91ff50edf}
\strng{authorfullhash}{9a4cdadc49e27a6b60a6426c4f007548}
\strng{authorfullhashraw}{9a4cdadc49e27a6b60a6426c4f007548}
\field{sortinit}{E}
\field{sortinithash}{8da8a182d344d5b9047633dfc0cc9131}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE transactions on pattern analysis and machine intelligence}
\field{title}{ProtTrans: towards cracking the language of lifes code through self-supervised deep learning and high performance computing}
\field{year}{2021}
\endentry
\entry{ferruz2023sequence}{article}{}{}
\name{author}{6}{}{%
{{hash=96cae7e0762a2f32bcb3565218b3ac9b}{%
family={Ferruz},
familyi={F\bibinitperiod},
given={Noelia},
giveni={N\bibinitperiod}}}%
{{hash=c04fca9e06da18cb002f115133913d24}{%
family={Heinzinger},
familyi={H\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=d910722253b97163b0c026eb26e8fdfc}{%
family={Akdel},
familyi={A\bibinitperiod},
given={Mehmet},
giveni={M\bibinitperiod}}}%
{{hash=84096a90774ea137e2ae529a6d4d9b03}{%
family={Goncearenco},
familyi={G\bibinitperiod},
given={Alexander},
giveni={A\bibinitperiod}}}%
{{hash=e8172fab565d9a1621b7af78ecce8399}{%
family={Naef},
familyi={N\bibinitperiod},
given={Luca},
giveni={L\bibinitperiod}}}%
{{hash=381edeaec659df830008a0b2d8646834}{%
family={Dallago},
familyi={D\bibinitperiod},
given={Christian},
giveni={C\bibinitperiod}}}%
}
\list{publisher}{1}{%
{Elsevier}%
}
\strng{namehash}{0c7d9b7f8fb97cf0fadf368a53c8fc49}
\strng{fullhash}{47f87ec8b0b0082ad5dcdc0d81d10723}
\strng{fullhashraw}{47f87ec8b0b0082ad5dcdc0d81d10723}
\strng{bibnamehash}{0c7d9b7f8fb97cf0fadf368a53c8fc49}
\strng{authorbibnamehash}{0c7d9b7f8fb97cf0fadf368a53c8fc49}
\strng{authornamehash}{0c7d9b7f8fb97cf0fadf368a53c8fc49}
\strng{authorfullhash}{47f87ec8b0b0082ad5dcdc0d81d10723}
\strng{authorfullhashraw}{47f87ec8b0b0082ad5dcdc0d81d10723}
\field{sortinit}{F}
\field{sortinithash}{2638baaa20439f1b5a8f80c6c08a13b4}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Computational and Structural Biotechnology Journal}
\field{title}{From sequence to function through structure: Deep learning for protein design}
\field{volume}{21}
\field{year}{2023}
\field{pages}{238\bibrangedash 250}
\range{pages}{13}
\endentry
\entry{Fey2019-hu}{article}{}{}
\name{author}{2}{}{%
{{hash=2667edba6b12b863d4fc7ec3afba0ede}{%
family={Fey},
familyi={F\bibinitperiod},
given={Matthias},
giveni={M\bibinitperiod}}}%
{{hash=ec18b27cbe6047529e35940b52997e45}{%
family={Lenssen},
familyi={L\bibinitperiod},
given={Jan\bibnamedelima Eric},
giveni={J\bibinitperiod\bibinitdelim E\bibinitperiod}}}%
}
\strng{namehash}{590626506334262dd0c764fafa024ab0}
\strng{fullhash}{590626506334262dd0c764fafa024ab0}
\strng{fullhashraw}{590626506334262dd0c764fafa024ab0}
\strng{bibnamehash}{590626506334262dd0c764fafa024ab0}
\strng{authorbibnamehash}{590626506334262dd0c764fafa024ab0}
\strng{authornamehash}{590626506334262dd0c764fafa024ab0}
\strng{authorfullhash}{590626506334262dd0c764fafa024ab0}
\strng{authorfullhashraw}{590626506334262dd0c764fafa024ab0}
\field{sortinit}{F}
\field{sortinithash}{2638baaa20439f1b5a8f80c6c08a13b4}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{We introduce PyTorch Geometric, a library for deep learning on irregularly structured input data such as graphs, point clouds and manifolds, built upon PyTorch. In addition to general graph data structures and processing methods, it contains a variety of recently published methods from the domains of relational learning and 3D data processing. PyTorch Geometric achieves high data throughput by leveraging sparse GPU acceleration, by providing dedicated CUDA kernels and by introducing efficient mini-batch handling for input examples of different size. In this work, we present the library in detail and perform a comprehensive comparative study of the implemented methods in homogeneous evaluation scenarios.}
\field{eprintclass}{cs.LG}
\field{title}{Fast graph representation learning with {PyTorch} Geometric}
\field{year}{2019}
\verb{eprint}
\verb 1903.02428
\endverb
\endentry
\entry{Fournier2024-fp}{unpublished}{}{}
\name{author}{6}{}{%
{{hash=de40290534f565e873d5eab079d20013}{%
family={Fournier},
familyi={F\bibinitperiod},
given={Quentin},
giveni={Q\bibinitperiod}}}%
{{hash=f63e7f53247f20b36aa96fcaaa3d36db}{%
family={Vernon},
familyi={V\bibinitperiod},
given={Robert\bibnamedelima M},
giveni={R\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=c9ca4d4d85e15285bef45eb66f9d20d3}{%
family={Sloot},
familyi={S\bibinitperiod},
given={Almer},
giveni={A\bibinitperiod},
prefix={van\bibnamedelima der},
prefixi={v\bibinitperiod\bibinitdelim d\bibinitperiod}}}%
{{hash=994d19ca7aee77feba3ecb93ffcc7aed}{%
family={Schulz},
familyi={S\bibinitperiod},
given={Benjamin},
giveni={B\bibinitperiod}}}%
{{hash=5fc5e175d6caeca6cf596c994cc02cd8}{%
family={Chandar},
familyi={C\bibinitperiod},
given={Sarath},
giveni={S\bibinitperiod}}}%
{{hash=7c5ce1be896fa06911efac1abd3237e0}{%
family={Langmead},
familyi={L\bibinitperiod},
given={Christopher\bibnamedelima James},
giveni={C\bibinitperiod\bibinitdelim J\bibinitperiod}}}%
}
\strng{namehash}{fca9793ec3f8edd564c6002c9f5a8370}
\strng{fullhash}{0890dcbe8802c72a36c8369e89e55f60}
\strng{fullhashraw}{0890dcbe8802c72a36c8369e89e55f60}
\strng{bibnamehash}{fca9793ec3f8edd564c6002c9f5a8370}
\strng{authorbibnamehash}{fca9793ec3f8edd564c6002c9f5a8370}
\strng{authornamehash}{fca9793ec3f8edd564c6002c9f5a8370}
\strng{authorfullhash}{0890dcbe8802c72a36c8369e89e55f60}
\strng{authorfullhashraw}{0890dcbe8802c72a36c8369e89e55f60}
\field{sortinit}{F}
\field{sortinithash}{2638baaa20439f1b5a8f80c6c08a13b4}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{AbstractPublic protein sequence databases contain samples from the fitness landscape explored by nature. Protein language models (pLMs) pre-trained on these sequences aim to capture this landscape for tasks like property prediction and protein design. Following the same trend as in natural language processing, pLMs have continuously been scaled up. However, the premise that scale leads to better performance assumes that source databases provide accurate representation of the underlying fitness landscape, which is likely false. By developing an efficient codebase, designing a modern architecture, and addressing data quality concerns such as sample bias, we introduce AMPLIFY, a best-in-class pLM that is orders of magnitude less expensive to train and deploy than previous models. Furthermore, to support the scientific community and democratize the training of pLMs, we have open-sourced AMPLIFY's pre-training codebase, data, and model checkpoints.}
\field{journaltitle}{bioRxiv}
\field{month}{9}
\field{title}{Protein language models: Is scaling necessary?}
\field{year}{2024}
\endentry
\entry{Kaplan2020-en}{article}{}{}
\name{author}{10}{}{%
{{hash=9e85370e215d552f72c95a2b63a37802}{%
family={Kaplan},
familyi={K\bibinitperiod},
given={Jared},
giveni={J\bibinitperiod}}}%
{{hash=b51e7c5fe92844f39ce52b8a5fa5675f}{%
family={McCandlish},
familyi={M\bibinitperiod},
given={Sam},
giveni={S\bibinitperiod}}}%
{{hash=eb1d3044b466619459c76843f0e98bb9}{%
family={Henighan},
familyi={H\bibinitperiod},
given={Tom},
giveni={T\bibinitperiod}}}%
{{hash=7c16e8d36475e58d4f2d161e6ecb7704}{%
family={Brown},
familyi={B\bibinitperiod},
given={Tom\bibnamedelima B},
giveni={T\bibinitperiod\bibinitdelim B\bibinitperiod}}}%
{{hash=01f70651539bbd7dccc01e86ed9c78c3}{%
family={Chess},
familyi={C\bibinitperiod},
given={Benjamin},
giveni={B\bibinitperiod}}}%
{{hash=5c2f5c2e6d4a9ec8681377f8a8e5e6af}{%
family={Child},
familyi={C\bibinitperiod},
given={Rewon},
giveni={R\bibinitperiod}}}%
{{hash=7006ca8c1ce969019b89de50fece60dd}{%
family={Gray},
familyi={G\bibinitperiod},
given={Scott},
giveni={S\bibinitperiod}}}%
{{hash=a812c46caad94fc8701be37871f303ba}{%
family={Radford},
familyi={R\bibinitperiod},
given={Alec},
giveni={A\bibinitperiod}}}%
{{hash=495187f3a2c93ddb8083bd18a5702527}{%
family={Wu},
familyi={W\bibinitperiod},
given={Jeffrey},
giveni={J\bibinitperiod}}}%
{{hash=1e6adbf36ab730cd5fdadb838b4d2667}{%
family={Amodei},
familyi={A\bibinitperiod},
given={Dario},
giveni={D\bibinitperiod}}}%
}
\strng{namehash}{b2dab6d559c573eed2d134686678fe84}
\strng{fullhash}{e6d1bc69376f936360a0a70ddcc99051}
\strng{fullhashraw}{e6d1bc69376f936360a0a70ddcc99051}
\strng{bibnamehash}{b2dab6d559c573eed2d134686678fe84}
\strng{authorbibnamehash}{b2dab6d559c573eed2d134686678fe84}
\strng{authornamehash}{b2dab6d559c573eed2d134686678fe84}
\strng{authorfullhash}{e6d1bc69376f936360a0a70ddcc99051}
\strng{authorfullhashraw}{e6d1bc69376f936360a0a70ddcc99051}
\field{sortinit}{K}
\field{sortinithash}{c02bf6bff1c488450c352b40f5d853ab}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.}
\field{eprintclass}{cs.LG}
\field{eprinttype}{arXiv}
\field{month}{1}
\field{title}{Scaling laws for neural language models}
\field{year}{2020}
\verb{eprint}
\verb 2001.08361
\endverb
\endentry
\entry{Kononova2021-cq}{article}{}{}
\name{author}{6}{}{%
{{hash=ace37c98c98bc2a596b5381d6c559097}{%
family={Kononova},
familyi={K\bibinitperiod},
given={Olga},
giveni={O\bibinitperiod}}}%
{{hash=50fcdf0b35a310b3cbc483405b3f75a8}{%
family={He},
familyi={H\bibinitperiod},
given={Tanjin},
giveni={T\bibinitperiod}}}%
{{hash=15262b287126ec73016108e9523c3a2f}{%
family={Huo},
familyi={H\bibinitperiod},
given={Haoyan},
giveni={H\bibinitperiod}}}%
{{hash=3226c692e171965fbec7956776dba506}{%
family={Trewartha},
familyi={T\bibinitperiod},
given={Amalie},
giveni={A\bibinitperiod}}}%
{{hash=122ff1d7f7d0bace64fc22217e270eed}{%
family={Olivetti},
familyi={O\bibinitperiod},
given={Elsa\bibnamedelima A},
giveni={E\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=1119594242bcd7037a1f252b0626e411}{%
family={Ceder},
familyi={C\bibinitperiod},
given={Gerbrand},
giveni={G\bibinitperiod}}}%
}
\list{language}{1}{%
{en}%
}
\list{publisher}{1}{%
{Elsevier BV}%
}
\strng{namehash}{638098adb68658ab2169cd2717f4e191}
\strng{fullhash}{f465d2e5c195a56fd87ac8361fcbff9b}
\strng{fullhashraw}{f465d2e5c195a56fd87ac8361fcbff9b}
\strng{bibnamehash}{638098adb68658ab2169cd2717f4e191}
\strng{authorbibnamehash}{638098adb68658ab2169cd2717f4e191}
\strng{authornamehash}{638098adb68658ab2169cd2717f4e191}
\strng{authorfullhash}{f465d2e5c195a56fd87ac8361fcbff9b}
\strng{authorfullhashraw}{f465d2e5c195a56fd87ac8361fcbff9b}
\field{sortinit}{K}
\field{sortinithash}{c02bf6bff1c488450c352b40f5d853ab}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Research publications are the major repository of scientific knowledge. However, their unstructured and highly heterogenous format creates a significant obstacle to large-scale analysis of the information contained within. Recent progress in natural language processing (NLP) has provided a variety of tools for high-quality information extraction from unstructured text. These tools are primarily trained on non-technical text and struggle to produce accurate results when applied to scientific text, involving specific technical terminology. During the last years, significant efforts in information retrieval have been made for biomedical and biochemical publications. For materials science, text mining (TM) methodology is still at the dawn of its development. In this review, we survey the recent progress in creating and applying TM and NLP approaches to materials science field. This review is directed at the broad class of researchers aiming to learn the fundamentals of TM as applied to the materials science publications.}
\field{journaltitle}{iScience}
\field{month}{3}
\field{number}{3}
\field{title}{Opportunities and challenges of text mining in aterials research}
\field{volume}{24}
\field{year}{2021}
\field{pages}{102155}
\range{pages}{1}
\keyw{Computational Materials Science; Computing Methodology; Data Analysis; Materials Design}
\endentry
\entry{Kuchaiev2019-am}{article}{}{}
\name{author}{14}{}{%
{{hash=bf07d85e23f5d109ed19745b901d2ce8}{%
family={Kuchaiev},
familyi={K\bibinitperiod},
given={Oleksii},
giveni={O\bibinitperiod}}}%
{{hash=757ee43203ad9bbb1f0624d13926a6a9}{%
family={Li},
familyi={L\bibinitperiod},
given={Jason},
giveni={J\bibinitperiod}}}%
{{hash=e05e57ebdec3de2a583909f4ec5786b4}{%
family={Nguyen},
familyi={N\bibinitperiod},
given={Huyen},
giveni={H\bibinitperiod}}}%
{{hash=3807e05e275b68005774cbce2aa2ae04}{%
family={Hrinchuk},
familyi={H\bibinitperiod},
given={Oleksii},
giveni={O\bibinitperiod}}}%
{{hash=82e56064304001a087dc9a965b3683a4}{%
family={Leary},
familyi={L\bibinitperiod},
given={Ryan},
giveni={R\bibinitperiod}}}%
{{hash=57b6ee8703cf89267a2d51429a4e61a7}{%
family={Ginsburg},
familyi={G\bibinitperiod},
given={Boris},
giveni={B\bibinitperiod}}}%
{{hash=12456e17452a91539d567fa71a517742}{%
family={Kriman},
familyi={K\bibinitperiod},
given={Samuel},
giveni={S\bibinitperiod}}}%
{{hash=b57e500c01c13ea269e1fab5aedc14b3}{%
family={Beliaev},
familyi={B\bibinitperiod},
given={Stanislav},
giveni={S\bibinitperiod}}}%
{{hash=ea50ad27e3aeecaaeee4cf5837a6b13a}{%
family={Lavrukhin},
familyi={L\bibinitperiod},
given={Vitaly},
giveni={V\bibinitperiod}}}%
{{hash=2b6820af875a90f16678763c99828a0b}{%
family={Cook},
familyi={C\bibinitperiod},
given={Jack},
giveni={J\bibinitperiod}}}%
{{hash=1b0203f2627b122423f6565fb0555737}{%
family={Castonguay},
familyi={C\bibinitperiod},
given={Patrice},
giveni={P\bibinitperiod}}}%
{{hash=cce7fe8d1e7c2268cda64e37d63d57db}{%
family={Popova},
familyi={P\bibinitperiod},
given={Mariya},
giveni={M\bibinitperiod}}}%
{{hash=d903c2b0b1d5db5145c140da071d1e2e}{%
family={Huang},
familyi={H\bibinitperiod},
given={Jocelyn},
giveni={J\bibinitperiod}}}%
{{hash=0ad5c4dabcde69d033e571909d94886e}{%
family={Cohen},
familyi={C\bibinitperiod},
given={Jonathan\bibnamedelima M},
giveni={J\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
}
\strng{namehash}{0167b484a1abf66770e8ba2cacd02c71}
\strng{fullhash}{e8614f407fe15896d5f85941d1dc3d0d}
\strng{fullhashraw}{e8614f407fe15896d5f85941d1dc3d0d}
\strng{bibnamehash}{0167b484a1abf66770e8ba2cacd02c71}
\strng{authorbibnamehash}{0167b484a1abf66770e8ba2cacd02c71}
\strng{authornamehash}{0167b484a1abf66770e8ba2cacd02c71}
\strng{authorfullhash}{e8614f407fe15896d5f85941d1dc3d0d}
\strng{authorfullhashraw}{e8614f407fe15896d5f85941d1dc3d0d}
\field{sortinit}{K}
\field{sortinithash}{c02bf6bff1c488450c352b40f5d853ab}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{NeMo (Neural Modules) is a Python framework-agnostic toolkit for creating AI applications through re-usability, abstraction, and composition. NeMo is built around neural modules, conceptual blocks of neural networks that take typed inputs and produce typed outputs. Such modules typically represent data layers, encoders, decoders, language models, loss functions, or methods of combining activations. NeMo makes it easy to combine and re-use these building blocks while providing a level of semantic correctness checking via its neural type system. The toolkit comes with extendable collections of pre-built modules for automatic speech recognition and natural language processing. Furthermore, NeMo provides built-in support for distributed training and mixed precision on latest NVIDIA GPUs. NeMo is open-source https://github.com/NVIDIA/NeMo}
\field{eprintclass}{cs.LG}
\field{title}{{NeMo}: a toolkit for building {AI} applications using Neural Modules}
\field{year}{2019}
\verb{eprint}
\verb 1909.09577
\endverb
\endentry
\entry{lee2024petabase}{article}{}{}
\name{author}{9}{}{%
{{hash=34e025c7b9ede25114e0436f27b7c697}{%
family={Lee},
familyi={L\bibinitperiod},
given={Sewon},
giveni={S\bibinitperiod}}}%
{{hash=a406ca5f70e60215746c2383b3d5c62c}{%
family={Kim},
familyi={K\bibinitperiod},
given={Gyuri},
giveni={G\bibinitperiod}}}%
{{hash=9403613893f81faf4c45881049692264}{%
family={Karin},
familyi={K\bibinitperiod},
given={Eli\bibnamedelima Levy},
giveni={E\bibinitperiod\bibinitdelim L\bibinitperiod}}}%
{{hash=2dac539cd49fcc48ee89ea09ce5e40c3}{%
family={Mirdita},
familyi={M\bibinitperiod},
given={Milot},
giveni={M\bibinitperiod}}}%
{{hash=38bbe07268d00b1fc685f72a7cae371e}{%
family={Park},
familyi={P\bibinitperiod},
given={Sukhwan},
giveni={S\bibinitperiod}}}%
{{hash=2c4c17b9e143709d5453c268900358f8}{%
family={Chikhi},
familyi={C\bibinitperiod},
given={Rayan},
giveni={R\bibinitperiod}}}%
{{hash=deeeb710d17b4d9afb5c6e3886ad1422}{%
family={Babaian},
familyi={B\bibinitperiod},
given={Artem},
giveni={A\bibinitperiod}}}%
{{hash=ba6a5be81953dca645cba617720be57e}{%
family={Kryshtafovych},
familyi={K\bibinitperiod},
given={Andriy},
giveni={A\bibinitperiod}}}%
{{hash=c06ac6a6fdcf2ee2092d61fc58c1f21b}{%
family={Steinegger},
familyi={S\bibinitperiod},
given={Martin},
giveni={M\bibinitperiod}}}%
}
\list{publisher}{1}{%
{Cold Spring Harbor Lab}%
}
\strng{namehash}{f50b96ff5d6eca20ec4f752aca887e38}
\strng{fullhash}{d5bc97d1fb2d68bb5e8a594e11b0308e}
\strng{fullhashraw}{d5bc97d1fb2d68bb5e8a594e11b0308e}
\strng{bibnamehash}{f50b96ff5d6eca20ec4f752aca887e38}
\strng{authorbibnamehash}{f50b96ff5d6eca20ec4f752aca887e38}
\strng{authornamehash}{f50b96ff5d6eca20ec4f752aca887e38}
\strng{authorfullhash}{d5bc97d1fb2d68bb5e8a594e11b0308e}
\strng{authorfullhashraw}{d5bc97d1fb2d68bb5e8a594e11b0308e}
\field{sortinit}{L}
\field{sortinithash}{7c47d417cecb1f4bd38d1825c427a61a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Cold Spring Harbor perspectives in biology}
\field{number}{5}
\field{title}{Petabase-Scale Homology Search for Structure Prediction}
\field{volume}{16}
\field{year}{2024}
\field{pages}{a041465}
\range{pages}{-1}
\endentry
\entry{Lin2023-sb}{article}{}{}
\name{author}{15}{}{%
{{hash=c40352c194e60a3ef458ee7e8685afb5}{%
family={Lin},
familyi={L\bibinitperiod},
given={Zeming},
giveni={Z\bibinitperiod}}}%
{{hash=15ffaf478ac80f96892efadcc2a135a6}{%
family={Akin},
familyi={A\bibinitperiod},
given={Halil},
giveni={H\bibinitperiod}}}%
{{hash=0f3a8501ead10f233eccda0b53bf0bd1}{%
family={Rao},
familyi={R\bibinitperiod},
given={Roshan},
giveni={R\bibinitperiod}}}%
{{hash=2218b6e69c35a7ab5d3c0ebfbe634d92}{%
family={Hie},
familyi={H\bibinitperiod},
given={Brian},
giveni={B\bibinitperiod}}}%
{{hash=47065c85b39cd5a2d076a7acf7f6280d}{%
family={Zhu},
familyi={Z\bibinitperiod},
given={Zhongkai},
giveni={Z\bibinitperiod}}}%
{{hash=883441545fe75217474c1c94d7936838}{%
family={Lu},
familyi={L\bibinitperiod},
given={Wenting},
giveni={W\bibinitperiod}}}%
{{hash=51a9fc39f16c7c9c6d331de0bbdaf57d}{%
family={Smetanin},
familyi={S\bibinitperiod},
given={Nikita},
giveni={N\bibinitperiod}}}%
{{hash=46be631e87cb978c860e4b0fa1d4887b}{%
family={Verkuil},
familyi={V\bibinitperiod},
given={Robert},
giveni={R\bibinitperiod}}}%
{{hash=f457c482acb006e884966be55f6e745a}{%
family={Kabeli},
familyi={K\bibinitperiod},
given={Ori},
giveni={O\bibinitperiod}}}%
{{hash=1db73894e13374e91b04144d7376349e}{%
family={Shmueli},
familyi={S\bibinitperiod},
given={Yaniv},
giveni={Y\bibinitperiod}}}%
{{hash=43a4aa96aaaf0415ae3065e1e46630fb}{%
family={Dos\bibnamedelimb Santos\bibnamedelima Costa},
familyi={D\bibinitperiod\bibinitdelim S\bibinitperiod\bibinitdelim C\bibinitperiod},
given={Allan},
giveni={A\bibinitperiod}}}%
{{hash=3574995111995617f1b94c9b2646d290}{%
family={Fazel-Zarandi},
familyi={F\bibinithyphendelim Z\bibinitperiod},
given={Maryam},
giveni={M\bibinitperiod}}}%
{{hash=21e17f70ffb714b5a29005c2a54cec93}{%
family={Sercu},
familyi={S\bibinitperiod},
given={Tom},
giveni={T\bibinitperiod}}}%
{{hash=2499e63435eed12a242cf416ef474e2a}{%
family={Candido},
familyi={C\bibinitperiod},
given={Salvatore},
giveni={S\bibinitperiod}}}%
{{hash=77c951217a087dd3f5981a0a79a87222}{%
family={Rives},
familyi={R\bibinitperiod},
given={Alexander},
giveni={A\bibinitperiod}}}%
}
\list{language}{1}{%
{en}%
}
\strng{namehash}{e5658d7879c7784982486a4a7cba055b}
\strng{fullhash}{f058bbbf059fa4ae9aa5e4940e8d3957}
\strng{fullhashraw}{f058bbbf059fa4ae9aa5e4940e8d3957}
\strng{bibnamehash}{e5658d7879c7784982486a4a7cba055b}
\strng{authorbibnamehash}{e5658d7879c7784982486a4a7cba055b}
\strng{authornamehash}{e5658d7879c7784982486a4a7cba055b}
\strng{authorfullhash}{f058bbbf059fa4ae9aa5e4940e8d3957}
\strng{authorfullhashraw}{f058bbbf059fa4ae9aa5e4940e8d3957}
\field{sortinit}{L}
\field{sortinithash}{7c47d417cecb1f4bd38d1825c427a61a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Recent advances in machine learning have leveraged evolutionary information in multiple sequence alignments to predict protein structure. We demonstrate direct inference of full atomic-level protein structure from primary sequence using a large language model. As language models of protein sequences are scaled up to 15 billion parameters, an atomic-resolution picture of protein structure emerges in the learned representations. This results in an order-of-magnitude acceleration of high-resolution structure prediction, which enables large-scale structural characterization of metagenomic proteins. We apply this capability to construct the ESM Metagenomic Atlas by predicting structures for >617 million metagenomic protein sequences, including >225 million that are predicted with high confidence, which gives a view into the vast breadth and diversity of natural proteins.}
\field{journaltitle}{Science}
\field{month}{3}
\field{number}{6637}
\field{title}{Evolutionary-scale prediction of atomic-level protein structure with a language model}
\field{volume}{379}
\field{year}{2023}
\field{pages}{1123\bibrangedash 1130}
\range{pages}{8}
\endentry
\entry{tensorflow2015-whitepaper}{misc}{}{}
\name{author}{40}{}{%
{{hash=396d6419316ec52f4c63b2f85912b61b}{%
family={Martín\bibnamedelima Abadi},
familyi={M\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=f337a7c116835c22bb206d2f0d7c70e0}{%
family={Ashish\bibnamedelima Agarwal},
familyi={A\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=84ac9fcb6c15dcd79c092bc8e20586ba}{%
family={Paul\bibnamedelima Barham},
familyi={P\bibinitperiod\bibinitdelim B\bibinitperiod}}}%
{{hash=d8574748e3086e0b279a58cdba71763d}{%
family={Eugene\bibnamedelima Brevdo},
familyi={E\bibinitperiod\bibinitdelim B\bibinitperiod}}}%
{{hash=c0b56f741b5a5bddfe77f1881c3cc67a}{%
family={Zhifeng\bibnamedelima Chen},
familyi={Z\bibinitperiod\bibinitdelim C\bibinitperiod}}}%
{{hash=8b8dd2e01366c855f42e47027cf23e98}{%
family={Craig\bibnamedelima Citro},
familyi={C\bibinitperiod\bibinitdelim C\bibinitperiod}}}%
{{hash=978a7d9601bf09e03d1bb3f6cce7a0ce}{%
family={Greg\bibnamedelima S.\bibnamedelimi Corrado},
familyi={G\bibinitperiod\bibinitdelim S\bibinitperiod\bibinitdelim C\bibinitperiod}}}%
{{hash=3b500b0dfd88e6e151d29108fdcb82f0}{%
family={Andy\bibnamedelima Davis},
familyi={A\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=2fd376ea3b3a3da11704c0ee86753dcf}{%
family={Jeffrey\bibnamedelima Dean},
familyi={J\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=5b34e641dd8a00f97c6242ae0353eb90}{%
family={Matthieu\bibnamedelima Devin},
familyi={M\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=5b4490947d4e91359646ce3c93cbd2f7}{%
family={Sanjay\bibnamedelima Ghemawat},
familyi={S\bibinitperiod\bibinitdelim G\bibinitperiod}}}%
{{hash=1fdef10b94ee122ef6136197f99e3df3}{%
family={Ian\bibnamedelima Goodfellow},
familyi={I\bibinitperiod\bibinitdelim G\bibinitperiod}}}%
{{hash=166ae8a0b435eded68e39e9e2d2a1ee8}{%
family={Andrew\bibnamedelima Harp},
familyi={A\bibinitperiod\bibinitdelim H\bibinitperiod}}}%
{{hash=7e9f7006151cf312bc133568336c68c6}{%
family={Geoffrey\bibnamedelima Irving},
familyi={G\bibinitperiod\bibinitdelim I\bibinitperiod}}}%
{{hash=08c1890e1c33279b8c63c71fa8f19263}{%
family={Michael\bibnamedelima Isard},
familyi={M\bibinitperiod\bibinitdelim I\bibinitperiod}}}%
{{hash=9fce03efe6b3331a1b93ed2e7c0da9d5}{%
family={Jia},
familyi={J\bibinitperiod},
given={Yangqing},
giveni={Y\bibinitperiod}}}%
{{hash=c0c0eea5379268c0c5b68732c90984b6}{%
family={Rafal\bibnamedelima Jozefowicz},
familyi={R\bibinitperiod\bibinitdelim J\bibinitperiod}}}%
{{hash=cff46cb4603a73d83b11ea7a9ded9d79}{%
family={Lukasz\bibnamedelima Kaiser},
familyi={L\bibinitperiod\bibinitdelim K\bibinitperiod}}}%
{{hash=d088e0f635523b8b5b18662331e4f44a}{%
family={Manjunath\bibnamedelima Kudlur},
familyi={M\bibinitperiod\bibinitdelim K\bibinitperiod}}}%
{{hash=1c24291ae15b979c82aa09a33790cb62}{%
family={Josh\bibnamedelima Levenberg},
familyi={J\bibinitperiod\bibinitdelim L\bibinitperiod}}}%
{{hash=796a3a98ff7545fe10f6a4c17ba016fa}{%
family={Dandelion\bibnamedelima Mané},
familyi={D\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=1ee98d232eb1fc1208a8f8ca649e970b}{%
family={Rajat\bibnamedelima Monga},
familyi={R\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=b2a15ec3d90955ece50ea26d31100b9a}{%
family={Sherry\bibnamedelima Moore},
familyi={S\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=1494c573fadad736c58cf1119ac59239}{%
family={Derek\bibnamedelima Murray},
familyi={D\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=ecf58eb1684af6cba2c1f126405eedab}{%
family={Chris\bibnamedelima Olah},
familyi={C\bibinitperiod\bibinitdelim O\bibinitperiod}}}%
{{hash=9f43befd94cd09a9aaa7ea8489405a83}{%
family={Mike\bibnamedelima Schuster},
familyi={M\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=4712800a228b1179529b9f6e0d1b1838}{%
family={Jonathon\bibnamedelima Shlens},
familyi={J\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=41ad6ff6c026d5a3730269072b31caf1}{%
family={Benoit\bibnamedelima Steiner},
familyi={B\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=b02f7871db6fc5524cec4ce38e104410}{%
family={Ilya\bibnamedelima Sutskever},
familyi={I\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=63288446e47b1d383f522ede84aa6fcc}{%
family={Kunal\bibnamedelima Talwar},
familyi={K\bibinitperiod\bibinitdelim T\bibinitperiod}}}%
{{hash=1dec75595b55bf77971f6a932d146b81}{%
family={Paul\bibnamedelima Tucker},
familyi={P\bibinitperiod\bibinitdelim T\bibinitperiod}}}%
{{hash=b6680dbb0176cb9bd87a3b26fa6f5cfb}{%
family={Vincent\bibnamedelima Vanhoucke},
familyi={V\bibinitperiod\bibinitdelim V\bibinitperiod}}}%
{{hash=e030c9d199c66657e26138be29814d81}{%
family={Vijay\bibnamedelima Vasudevan},
familyi={V\bibinitperiod\bibinitdelim V\bibinitperiod}}}%
{{hash=04426b798803cfaf3e8aa9280a5d0a58}{%
family={Fernanda\bibnamedelima Viégas},
familyi={F\bibinitperiod\bibinitdelim V\bibinitperiod}}}%
{{hash=fa7242e11c7d955de2ac1be94ca29073}{%
family={Oriol\bibnamedelima Vinyals},
familyi={O\bibinitperiod\bibinitdelim V\bibinitperiod}}}%
{{hash=8c9ee8f70a3c3d97f85efd01c4e9cbe6}{%
family={Pete\bibnamedelima Warden},
familyi={P\bibinitperiod\bibinitdelim W\bibinitperiod}}}%
{{hash=8e4243c228c72a5e5279e31252887b32}{%
family={Martin\bibnamedelima Wattenberg},
familyi={M\bibinitperiod\bibinitdelim W\bibinitperiod}}}%
{{hash=c6a6eb2597f23589fc9141bdda275996}{%
family={Martin\bibnamedelima Wicke},
familyi={M\bibinitperiod\bibinitdelim W\bibinitperiod}}}%
{{hash=3ea39e6dc6ef47029ae996c7e63f1a48}{%
family={Yuan\bibnamedelima Yu},
familyi={Y\bibinitperiod\bibinitdelim Y\bibinitperiod}}}%
{{hash=b69feb3a3d59a312b20dbef0b1d2d6de}{%
family={Xiaoqiang\bibnamedelima Zheng},
familyi={X\bibinitperiod\bibinitdelim Z\bibinitperiod}}}%
}
\strng{namehash}{7fdd865be502254047a3b2638dc0cfeb}
\strng{fullhash}{9b332dc9b33a2f6316d71d525269bd0f}
\strng{fullhashraw}{9b332dc9b33a2f6316d71d525269bd0f}
\strng{bibnamehash}{7fdd865be502254047a3b2638dc0cfeb}
\strng{authorbibnamehash}{7fdd865be502254047a3b2638dc0cfeb}
\strng{authornamehash}{7fdd865be502254047a3b2638dc0cfeb}
\strng{authorfullhash}{9b332dc9b33a2f6316d71d525269bd0f}
\strng{authorfullhashraw}{9b332dc9b33a2f6316d71d525269bd0f}
\field{sortinit}{M}
\field{sortinithash}{4625c616857f13d17ce56f7d4f97d451}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{note}{Software available from tensorflow.org}
\field{title}{{TensorFlow}: Large-Scale Machine Learning on Heterogeneous Systems}
\field{year}{2015}
\verb{urlraw}
\verb https://www.tensorflow.org/
\endverb
\verb{url}
\verb https://www.tensorflow.org/
\endverb
\endentry
\entry{Narayanan2021-qc}{inproceedings}{}{}
\name{author}{12}{}{%
{{hash=b34c3675fa6935732be9302df09d61e7}{%
family={Narayanan},
familyi={N\bibinitperiod},
given={Deepak},
giveni={D\bibinitperiod}}}%
{{hash=4af7939b0e0aac6c1e9dec565975b46c}{%
family={Shoeybi},
familyi={S\bibinitperiod},
given={Mohammad},
giveni={M\bibinitperiod}}}%
{{hash=0068bfe8f24d2afb2959214f0b5928a4}{%
family={Casper},
familyi={C\bibinitperiod},
given={Jared},
giveni={J\bibinitperiod}}}%
{{hash=83e4e2d744c8cf00198e36bf6b82cb64}{%
family={LeGresley},
familyi={L\bibinitperiod},
given={Patrick},
giveni={P\bibinitperiod}}}%
{{hash=ed280cff66c1c79d7a05b33c82dd9038}{%
family={Patwary},
familyi={P\bibinitperiod},
given={Mostofa},
giveni={M\bibinitperiod}}}%
{{hash=0813fa5cfb76cce7c22bbe49a41c6d63}{%
family={Korthikanti},
familyi={K\bibinitperiod},
given={Vijay},
giveni={V\bibinitperiod}}}%
{{hash=148ec2fd697cb7bce427c9b7b9c88614}{%
family={Vainbrand},
familyi={V\bibinitperiod},
given={Dmitri},
giveni={D\bibinitperiod}}}%
{{hash=c7ad727e766ff04b536a3e01139f9577}{%
family={Kashinkunti},
familyi={K\bibinitperiod},
given={Prethvi},
giveni={P\bibinitperiod}}}%
{{hash=106c481eed1ff0e50c5484328eb1c68b}{%
family={Bernauer},
familyi={B\bibinitperiod},
given={Julie},
giveni={J\bibinitperiod}}}%
{{hash=3c8bbc381484f2cfbbd08866be36590e}{%
family={Catanzaro},
familyi={C\bibinitperiod},
given={Bryan},
giveni={B\bibinitperiod}}}%
{{hash=472262704806870baaf3b41b5305c994}{%
family={Phanishayee},
familyi={P\bibinitperiod},
given={Amar},
giveni={A\bibinitperiod}}}%
{{hash=00dc518b94dfa6349c86e70661a1a5ef}{%
family={Zaharia},
familyi={Z\bibinitperiod},
given={Matei},
giveni={M\bibinitperiod}}}%
}
\list{location}{1}{%
{St. Louis Missouri}%
}
\list{publisher}{1}{%
{ACM}%
}
\strng{namehash}{48e4a5b8e1fae46817befd7f8ab7985f}
\strng{fullhash}{1cfb8d3da4e957e774a768d37dfab108}
\strng{fullhashraw}{1cfb8d3da4e957e774a768d37dfab108}
\strng{bibnamehash}{48e4a5b8e1fae46817befd7f8ab7985f}
\strng{authorbibnamehash}{48e4a5b8e1fae46817befd7f8ab7985f}
\strng{authornamehash}{48e4a5b8e1fae46817befd7f8ab7985f}
\strng{authorfullhash}{1cfb8d3da4e957e774a768d37dfab108}
\strng{authorfullhashraw}{1cfb8d3da4e957e774a768d37dfab108}
\field{sortinit}{N}
\field{sortinithash}{22369a73d5f88983a108b63f07f37084}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis}
\field{month}{11}
\field{title}{Efficient large-scale language model training on {GPU} clusters using {megatron-LM}}
\field{year}{2021}
\endentry
\entry{Paszke2019-ch}{article}{}{}
\name{author}{21}{}{%
{{hash=56bf0b340039cf8594436a624ff548a9}{%
family={Paszke},
familyi={P\bibinitperiod},
given={Adam},
giveni={A\bibinitperiod}}}%
{{hash=4ba5062e5919c814aceec188d54c01f2}{%
family={Gross},
familyi={G\bibinitperiod},
given={Sam},
giveni={S\bibinitperiod}}}%
{{hash=e5dfae4582081d649e3a0d5342050016}{%
family={Massa},
familyi={M\bibinitperiod},
given={Francisco},
giveni={F\bibinitperiod}}}%
{{hash=b5815e1692fa2d0c1f44eecf509bd7c4}{%
family={Lerer},
familyi={L\bibinitperiod},
given={Adam},
giveni={A\bibinitperiod}}}%
{{hash=b75383e6b48c8360c7a60031424c85cf}{%
family={Bradbury},
familyi={B\bibinitperiod},
given={James},
giveni={J\bibinitperiod}}}%
{{hash=f897ed422c34d95af2e22778dfc2607e}{%
family={Chanan},
familyi={C\bibinitperiod},
given={Gregory},
giveni={G\bibinitperiod}}}%
{{hash=046269e070246feb6f394141db80ed87}{%
family={Killeen},
familyi={K\bibinitperiod},
given={Trevor},
giveni={T\bibinitperiod}}}%
{{hash=c40352c194e60a3ef458ee7e8685afb5}{%
family={Lin},
familyi={L\bibinitperiod},
given={Zeming},
giveni={Z\bibinitperiod}}}%
{{hash=6e45f49ec618e619efad90c8e8a61f0c}{%
family={Gimelshein},
familyi={G\bibinitperiod},
given={Natalia},
giveni={N\bibinitperiod}}}%
{{hash=f65a80959d520337ae99a0798515036c}{%
family={Antiga},
familyi={A\bibinitperiod},
given={Luca},
giveni={L\bibinitperiod}}}%
{{hash=954cf7680b6ce14813973eccdca3c4bc}{%
family={Desmaison},
familyi={D\bibinitperiod},
given={Alban},
giveni={A\bibinitperiod}}}%
{{hash=048232cf7c525fbc0bc93052fe8cee03}{%
family={Köpf},
familyi={K\bibinitperiod},
given={Andreas},
giveni={A\bibinitperiod}}}%
{{hash=b9e701339e56fd0b171145b08288a1b7}{%
family={Yang},
familyi={Y\bibinitperiod},
given={Edward},
giveni={E\bibinitperiod}}}%
{{hash=42ac264897098b400e1367e5922c9b0d}{%
family={DeVito},
familyi={D\bibinitperiod},
given={Zach},
giveni={Z\bibinitperiod}}}%
{{hash=d814afaa50b9e22ab92cc9f8f9a9e43a}{%
family={Raison},
familyi={R\bibinitperiod},
given={Martin},
giveni={M\bibinitperiod}}}%
{{hash=3feeeebee8583ecc208f7fb3e0a55068}{%
family={Tejani},
familyi={T\bibinitperiod},
given={Alykhan},
giveni={A\bibinitperiod}}}%
{{hash=e18536d5cb7543731fbf2ca1a4908732}{%
family={Chilamkurthy},
familyi={C\bibinitperiod},
given={Sasank},
giveni={S\bibinitperiod}}}%
{{hash=0a0b028c6b85c46f368317d0c5bfe3a0}{%
family={Steiner},
familyi={S\bibinitperiod},
given={Benoit},
giveni={B\bibinitperiod}}}%
{{hash=998a001f16bb57c079c1d5afb1cb02c8}{%
family={Fang},
familyi={F\bibinitperiod},
given={Lu},
giveni={L\bibinitperiod}}}%
{{hash=3f19c633bbfb847db6a0e71d3659eacd}{%
family={Bai},
familyi={B\bibinitperiod},
given={Junjie},
giveni={J\bibinitperiod}}}%
{{hash=8ef51a0906e47d2b4472c4e714ed598f}{%
family={Chintala},
familyi={C\bibinitperiod},
given={Soumith},
giveni={S\bibinitperiod}}}%
}
\strng{namehash}{724e74fc18651eb78eb82fbcd1d9dfb1}
\strng{fullhash}{4842db6c92a33147f588935fdde44a69}
\strng{fullhashraw}{4842db6c92a33147f588935fdde44a69}
\strng{bibnamehash}{724e74fc18651eb78eb82fbcd1d9dfb1}
\strng{authorbibnamehash}{724e74fc18651eb78eb82fbcd1d9dfb1}
\strng{authornamehash}{724e74fc18651eb78eb82fbcd1d9dfb1}
\strng{authorfullhash}{4842db6c92a33147f588935fdde44a69}
\strng{authorfullhashraw}{4842db6c92a33147f588935fdde44a69}
\field{sortinit}{P}
\field{sortinithash}{ff3bcf24f47321b42cb156c2cc8a8422}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.}
\field{eprintclass}{cs.LG}
\field{title}{{PyTorch}: An imperative style, high-performance deep learning library}
\field{year}{2019}
\verb{eprint}
\verb 1912.01703
\endverb
\endentry
\entry{Rajbhandari2019-wp}{article}{}{}
\name{author}{4}{}{%
{{hash=d780451abfe83fa36b8188128dd49256}{%
family={Rajbhandari},
familyi={R\bibinitperiod},
given={Samyam},
giveni={S\bibinitperiod}}}%
{{hash=bf851cabe7082a61a8b46a398a4ab4e7}{%
family={Rasley},
familyi={R\bibinitperiod},
given={Jeff},
giveni={J\bibinitperiod}}}%
{{hash=2e3b810171475601e7a29638c20597a3}{%
family={Ruwase},
familyi={R\bibinitperiod},
given={Olatunji},
giveni={O\bibinitperiod}}}%
{{hash=2a1be1c510b34082b3cb846cf8147969}{%
family={He},
familyi={H\bibinitperiod},
given={Yuxiong},
giveni={Y\bibinitperiod}}}%
}
\strng{namehash}{1997458e7d38752f4374499bf4bbb94f}
\strng{fullhash}{638fa57a9116b097f06e3bae828dd47e}
\strng{fullhashraw}{638fa57a9116b097f06e3bae828dd47e}
\strng{bibnamehash}{1997458e7d38752f4374499bf4bbb94f}
\strng{authorbibnamehash}{1997458e7d38752f4374499bf4bbb94f}
\strng{authornamehash}{1997458e7d38752f4374499bf4bbb94f}
\strng{authorfullhash}{638fa57a9116b097f06e3bae828dd47e}
\strng{authorfullhashraw}{638fa57a9116b097f06e3bae828dd47e}
\field{sortinit}{R}
\field{sortinithash}{5e1c39a9d46ffb6bebd8f801023a9486}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Large deep learning models offer significant accuracy gains, but training billions to trillions of parameters is challenging. Existing solutions such as data and model parallelisms exhibit fundamental limitations to fit these models into limited device memory, while obtaining computation, communication and development efficiency. We develop a novel solution, Zero Redundancy Optimizer (ZeRO), to optimize memory, vastly improving training speed while increasing the model size that can be efficiently trained. ZeRO eliminates memory redundancies in data- and model-parallel training while retaining low communication volume and high computational granularity, allowing us to scale the model size proportional to the number of devices with sustained high efficiency. Our analysis on memory requirements and communication volume demonstrates: ZeRO has the potential to scale beyond 1 Trillion parameters using today's hardware. We implement and evaluate ZeRO: it trains large models of over 100B parameter with super-linear speedup on 400 GPUs, achieving throughput of 15 Petaflops. This represents an 8x increase in model size and 10x increase in achievable performance over state-of-the-art. In terms of usability, ZeRO can train large models of up to 13B parameters (e.g., larger than Megatron GPT 8.3B and T5 11B) without requiring model parallelism which is harder for scientists to apply. Last but not the least, researchers have used the system breakthroughs of ZeRO to create the world's largest language model (Turing-NLG, 17B parameters) with record breaking accuracy.}
\field{eprintclass}{cs.LG}
\field{eprinttype}{arXiv}
\field{month}{10}
\field{title}{{ZeRO}: Memory optimizations toward training Trillion parameter models}
\field{year}{2019}
\verb{eprint}
\verb 1910.02054
\endverb
\endentry
\entry{Theodoris2023-np}{article}{}{}
\name{author}{11}{}{%
{{hash=db9bb3bc0604199de17ce7fd678a551d}{%
family={Theodoris},
familyi={T\bibinitperiod},
given={Christina\bibnamedelima V},
giveni={C\bibinitperiod\bibinitdelim V\bibinitperiod}}}%
{{hash=91c798495eb64a5e7ee8f04fe8a36c00}{%
family={Xiao},
familyi={X\bibinitperiod},
given={Ling},
giveni={L\bibinitperiod}}}%
{{hash=31374d66ede2f6080a15fadcef088436}{%
family={Chopra},
familyi={C\bibinitperiod},
given={Anant},
giveni={A\bibinitperiod}}}%
{{hash=61558ceeb38ed81350c7787b14753062}{%
family={Chaffin},
familyi={C\bibinitperiod},
given={Mark\bibnamedelima D},
giveni={M\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=7000dfde3ec299b959c6b08cfa003c88}{%
family={Al\bibnamedelima Sayed},
familyi={A\bibinitperiod\bibinitdelim S\bibinitperiod},
given={Zeina\bibnamedelima R},
giveni={Z\bibinitperiod\bibinitdelim R\bibinitperiod}}}%
{{hash=44b105d010aee14a2cb3ae7d14ba4caa}{%
family={Hill},
familyi={H\bibinitperiod},
given={Matthew\bibnamedelima C},
giveni={M\bibinitperiod\bibinitdelim C\bibinitperiod}}}%
{{hash=ec5f1921e4fa00025af2b0aefa9be7b6}{%
family={Mantineo},
familyi={M\bibinitperiod},
given={Helene},
giveni={H\bibinitperiod}}}%
{{hash=d414ad5f541163967a35db57965529dd}{%
family={Brydon},
familyi={B\bibinitperiod},
given={Elizabeth\bibnamedelima M},
giveni={E\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=7f09613e1486d45804d75d555079625d}{%
family={Zeng},
familyi={Z\bibinitperiod},
given={Zexian},
giveni={Z\bibinitperiod}}}%
{{hash=ce053aa23e50f63feec80cc5cd4dad0d}{%
family={Liu},
familyi={L\bibinitperiod},
given={X\bibnamedelima Shirley},
giveni={X\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=f66439929a95109d7912306caa769bcd}{%
family={Ellinor},
familyi={E\bibinitperiod},
given={Patrick\bibnamedelima T},
giveni={P\bibinitperiod\bibinitdelim T\bibinitperiod}}}%
}
\list{language}{1}{%
{en}%
}
\strng{namehash}{34b91c78d6352e0e0c737491fd55667f}
\strng{fullhash}{213647b357b295bb9d7078d33195a752}
\strng{fullhashraw}{213647b357b295bb9d7078d33195a752}
\strng{bibnamehash}{34b91c78d6352e0e0c737491fd55667f}
\strng{authorbibnamehash}{34b91c78d6352e0e0c737491fd55667f}
\strng{authornamehash}{34b91c78d6352e0e0c737491fd55667f}
\strng{authorfullhash}{213647b357b295bb9d7078d33195a752}
\strng{authorfullhashraw}{213647b357b295bb9d7078d33195a752}
\field{sortinit}{T}
\field{sortinithash}{9af77f0292593c26bde9a56e688eaee9}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Mapping gene networks requires large amounts of transcriptomic data to learn the connections between genes, which impedes discoveries in settings with limited data, including rare diseases and diseases affecting clinically inaccessible tissues. Recently, transfer learning has revolutionized fields such as natural language understanding1,2 and computer vision3 by leveraging deep learning models pretrained on large-scale general datasets that can then be fine-tuned towards a vast array of downstream tasks with limited task-specific data. Here, we developed a context-aware, attention-based deep learning model, Geneformer, pretrained on a large-scale corpus of about 30 million single-cell transcriptomes to enable context-specific predictions in settings with limited data in network biology. During pretraining, Geneformer gained a fundamental understanding of network dynamics, encoding network hierarchy in the attention weights of the model in a completely self-supervised manner. Fine-tuning towards a diverse panel of downstream tasks relevant to chromatin and network dynamics using limited task-specific data demonstrated that Geneformer consistently boosted predictive accuracy. Applied to disease modelling with limited patient data, Geneformer identified candidate therapeutic targets for cardiomyopathy. Overall, Geneformer represents a pretrained deep learning model from which fine-tuning towards a broad range of downstream applications can be pursued to accelerate discovery of key network regulators and candidate therapeutic targets.}
\field{journaltitle}{Nature}
\field{month}{6}
\field{number}{7965}
\field{title}{Transfer learning enables predictions in network biology}
\field{volume}{618}
\field{year}{2023}
\field{pages}{616\bibrangedash 624}
\range{pages}{9}
\endentry
\entry{uniprot2023uniprot}{article}{}{}
\list{publisher}{1}{%
{Oxford University Press}%
}
\field{sortinit}{U}
\field{sortinithash}{6901a00e45705986ee5e7ca9fd39adca}
\field{labeltitlesource}{title}
\field{journaltitle}{Nucleic acids research}
\field{number}{D1}
\field{title}{UniProt: the universal protein knowledgebase in 2023}
\field{volume}{51}
\field{year}{2023}
\field{pages}{D523\bibrangedash D531}
\range{pages}{-1}
\endentry
\entry{vaswani2017attention}{article}{}{}
\name{author}{1}{}{%
{{hash=dd8654367c923279fc35fe561acfbf40}{%
family={Vaswani},
familyi={V\bibinitperiod},
given={A},
giveni={A\bibinitperiod}}}%
}
\strng{namehash}{dd8654367c923279fc35fe561acfbf40}
\strng{fullhash}{dd8654367c923279fc35fe561acfbf40}
\strng{fullhashraw}{dd8654367c923279fc35fe561acfbf40}
\strng{bibnamehash}{dd8654367c923279fc35fe561acfbf40}
\strng{authorbibnamehash}{dd8654367c923279fc35fe561acfbf40}
\strng{authornamehash}{dd8654367c923279fc35fe561acfbf40}
\strng{authorfullhash}{dd8654367c923279fc35fe561acfbf40}
\strng{authorfullhashraw}{dd8654367c923279fc35fe561acfbf40}
\field{sortinit}{V}
\field{sortinithash}{afb52128e5b4dc4b843768c0113d673b}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Advances in Neural Information Processing Systems}
\field{title}{Attention is all you need}
\field{year}{2017}
\endentry
\entry{Wang2019-ch}{article}{}{}
\name{author}{15}{}{%
{{hash=3f4efd00ddc6a60912e47de240273894}{%
family={Wang},
familyi={W\bibinitperiod},
given={Minjie},
giveni={M\bibinitperiod}}}%
{{hash=26769a305fa30533cb369be20dd8bb69}{%
family={Zheng},
familyi={Z\bibinitperiod},
given={Da},
giveni={D\bibinitperiod}}}%
{{hash=700c5e0bb72126fd71161b560ed1d445}{%
family={Ye},
familyi={Y\bibinitperiod},
given={Zihao},
giveni={Z\bibinitperiod}}}%
{{hash=78043a6d80bed30053efee8fc010f5cd}{%
family={Gan},
familyi={G\bibinitperiod},
given={Quan},
giveni={Q\bibinitperiod}}}%
{{hash=a5f00e0019689914807d4a4ef6623ce6}{%
family={Li},
familyi={L\bibinitperiod},
given={Mufei},
giveni={M\bibinitperiod}}}%
{{hash=7471051e2eae9f757440aafaf7adb3f8}{%
family={Song},
familyi={S\bibinitperiod},
given={Xiang},
giveni={X\bibinitperiod}}}%
{{hash=8b1bba6efae7fc9dcfc6d5407df2b33c}{%
family={Zhou},
familyi={Z\bibinitperiod},
given={Jinjing},
giveni={J\bibinitperiod}}}%
{{hash=26f3b8f065f337f327a99920fa1e63cc}{%
family={Ma},
familyi={M\bibinitperiod},
given={Chao},
giveni={C\bibinitperiod}}}%
{{hash=889dfe42c6869cf12a941fdb9c148c58}{%
family={Yu},
familyi={Y\bibinitperiod},
given={Lingfan},
giveni={L\bibinitperiod}}}%
{{hash=36049221dee33effccf50457c289ec6d}{%
family={Gai},
familyi={G\bibinitperiod},
given={Yu},
giveni={Y\bibinitperiod}}}%
{{hash=4484930d127005d0277016f70f2a4b07}{%
family={Xiao},
familyi={X\bibinitperiod},
given={Tianjun},
giveni={T\bibinitperiod}}}%
{{hash=399cf1aabecf0867957b50a0628a019a}{%
family={He},
familyi={H\bibinitperiod},
given={Tong},
giveni={T\bibinitperiod}}}%
{{hash=3db7df6401b206bb040d6735adbe81e2}{%
family={Karypis},
familyi={K\bibinitperiod},
given={George},
giveni={G\bibinitperiod}}}%
{{hash=20e2f54cf4cbc45d48bb0ed78215a321}{%
family={Li},
familyi={L\bibinitperiod},
given={Jinyang},
giveni={J\bibinitperiod}}}%
{{hash=fcdb4c7704baa79893b29d652aecc77f}{%
family={Zhang},
familyi={Z\bibinitperiod},
given={Zheng},
giveni={Z\bibinitperiod}}}%
}
\strng{namehash}{063f02df51131b80c25ac063b2ac78c9}
\strng{fullhash}{36ecc07a6eebd3b4565fab267dbe46ab}
\strng{fullhashraw}{36ecc07a6eebd3b4565fab267dbe46ab}
\strng{bibnamehash}{063f02df51131b80c25ac063b2ac78c9}
\strng{authorbibnamehash}{063f02df51131b80c25ac063b2ac78c9}
\strng{authornamehash}{063f02df51131b80c25ac063b2ac78c9}
\strng{authorfullhash}{36ecc07a6eebd3b4565fab267dbe46ab}
\strng{authorfullhashraw}{36ecc07a6eebd3b4565fab267dbe46ab}
\field{sortinit}{W}
\field{sortinithash}{4315d78024d0cea9b57a0c6f0e35ed0d}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Advancing research in the emerging field of deep graph learning requires new tools to support tensor computation over graphs. In this paper, we present the design principles and implementation of Deep Graph Library (DGL). DGL distills the computational patterns of GNNs into a few generalized sparse tensor operations suitable for extensive parallelization. By advocating graph as the central programming abstraction, DGL can perform optimizations transparently. By cautiously adopting a framework-neutral design, DGL allows users to easily port and leverage the existing components across multiple deep learning frameworks. Our evaluation shows that DGL significantly outperforms other popular GNN-oriented frameworks in both speed and memory consumption over a variety of benchmarks and has little overhead for small scale workloads.}
\field{eprintclass}{cs.LG}
\field{title}{Deep Graph Library: A graph-centric, highly-performant package for graph neural networks}
\field{year}{2019}
\verb{eprint}
\verb 1909.01315
\endverb
\endentry
\enddatalist
\endrefsection
\endinput