NeutronOrch: Rethinking Sample-based GNN Training under CPU-GPU Heterogeneous Environments

Extracted source text · arXiv 2311.13225

% VLDB template version of 2020-08-03 enhances the ACM template, version 1.7.0:
% https://www.acm.org/publications/proceedings-template
% The ACM Latex guide provides further information about the ACM template

\documentclass[sigconf, nonacm]{acmart}
% \usepackage{cite}

\usepackage{amsmath,amsfonts}
\let\Bbbk\relax
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\usepackage{xspace}
\usepackage{graphics}
\usepackage{epsfig}
\usepackage{enumitem}
\newcommand{\Paragraph} [1] {\smallskip\noindent{\bf #1 }}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{bbding}
\usepackage{url}
\usepackage{tikz}
\usepackage{filecontents}

%\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{balance}  % for  \balance command ON LAST PAGE  (only there!)
\usepackage{subfig}
\usepackage{verbatim}
\usepackage{color,xcolor}
\usepackage{array}
\usepackage{bbding}
% Include information below and uncomment for camera ready
\settopmatter{printacmref=false,printfolios=true}
% \usepackage{algorithm}
% %\usepackage{algorithmic}
% \usepackage{algpseudocode}
% \renewcommand{\algorithmicrequire}{\textbf{Input:}}  % Use Input in the format of Algorithm  
% \renewcommand{\algorithmicensure}{\textbf{Output:}} % Use 

\usepackage{algorithm}
%\usepackage{algorithmic}
\usepackage[noend]{algpseudocode}
\usepackage{algpseudocode}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}  % Use Input in the format of Algorithm  
\renewcommand{\algorithmicensure}{\textbf{Output:}} % Use Output in the
%% The following content must be adapted for the final version
% paper-specific
\newcommand\vldbdoi{XX.XX/XXX.XX}
\newcommand\vldbpages{XXX-XXX}
% issue-specific
\newcommand\vldbvolume{14}
\newcommand\vldbissue{1}
\newcommand\vldbyear{2022}
% should be fine as it is
\newcommand\vldbauthors{\authors}
\newcommand\vldbtitle{\shorttitle} 
% leave empty if no availability url should be set
\newcommand\vldbavailabilityurl{https://github.com/AiX-im/Sample-based-GNN}
% whether page numbers should be shown or not, use 'plain' for review versions, 'empty' for camera ready
\newcommand\vldbpagestyle{plain} 
\newcommand{\zyf}[1]{\textcolor{blue}{#1}}
\newcommand{\wqg}[1]{\textcolor{red}{#1}}
\newcommand{\ccy}[1]{\textcolor{orange}{#1}}
\newcommand{\yh}[1]{\textcolor{brown}{#1}}
\newcommand{\sanzo}[1]{\textcolor{brown}{#1}}

\newcommand{\notecolor}[1]{\textcolor{red}{#1}}

\newcommand{\system}{NeutronOrch\xspace}
\begin{document}
\title{NeutronOrch: Rethinking Sample-based GNN Training under CPU-GPU Heterogeneous Environments}

%\thanks{Qiange and Xin contributed equally. Yanfeng is the corresponding author.}
\vspace{-0.25in}
\author{Xin Ai, Qiange Wang, Chunyu Cao, Yanfeng Zhang, Chaoyi Chen, Hao Yuan, Yu Gu, Ge Yu}
\affiliation{%
  \institution{Northeastern University\country{China}}}
\email{{aixin0,wangqiange,chunyucao,chenchaoy,yuanhao}@stumail.neu.edu.cn, {zhangyf, guyu,yuge}@mail.neu.edu.cn}



%%
%% The "author" command and its associated commands are used to define the authors and their affiliations.
% \author{Xin Ai}
% \affiliation{%
%   \institution{Northeastern University}
%   \city{Shenyang}
%   \state{China}
% }
% \email{aixin0@stumail.neu.edu.cn}

%plan total
%ddl 1.1 2023 8 weeks
%paper 1.introduction 2.background 3.analysis: a motivating study 4.FtmGNN overview
%5.
%
%%
%% The abstract is a short summary of the work to be presented in the
%% article.
%plan  deadline 2023.4.1
%12.5 - 12.20 
%ax: 完成前三章
%第一章捋顺故事脉络：现有什么问题，本文解决什么问题，怎么解决，论文的核心贡献是什么。
%第二章介绍背景，铺垫引出问题，介绍现有优化及特点，为阅读本文做前置知识。
%第三章正式引出要解决问题，并且逐个分析，证明问题真的存在，最后引出并验证解决方法的合理性，即使用CPU做部分training的合理性。
%cy:部署调试熟悉DGL GNNlab的代码，为后续实验做准备，熟悉前三章论文，熟悉全文故事。
%12-20 - 12-30
%ax: 迭代一次前三章
% 与老师和千阁和春榆sync一下现有内容，整理修改，迭代前三章一次。
% 需要明确几点
% 1.故事是否可行
% 2.现有实验是否合理，如需重做或补充还需要安排
% 3.英语写作问题
% cy：撰写第七章论文，检查前三章论文问题。 related work
%1.1 - 1.10
%ax: 完成论文第四章 系统的high level设计，系统的架构设计，给出系统的overview。
%cy: 完成第七章论文，学习多GPU代码。
%1.10 - 1.20  
%ax,cy:完成多GPU部分代码，测试系统在多GPU中的性能。主要是与GNNlab,DGL对比。主要做overall性能对比，确保性能优越性。
%1.20 - 1.30 holiday 
%2.1  - 2.10  % 基于工作时间的任务分配，预跑一次。
%ax: 编排论文第五章内容，设计动态push down CPU training模块，设计动态策略，不能固定一层pushdown
%cy：在多GPU与单GPU中，除原本的GCN算法外，测试GAT  GraphSage算法性能，确保性能优越性
%2.10 - 2.20  
%ax:完成论文第五章，与cy一起完成上述代码任务。
%cy: 同上。
%2.20 - 2.30  
%ax: 迭代论文 四五章一次
%与老师和千阁和春榆sync一下现有内容，整理修改，迭代四五章一次。
%cy ：添加动态调度策略代码，这个可选，可以先手动搞
%3.1  - 3.5  
%ax，cy: 设计论文第六章，设计实验，设计后讨论可行性，难点。准备实验数据，实验环境。
%3.5 - 3.20  
%ax，cy：完成全部实验，完成第六章论文撰写。
%3.20 - 3.30 
%ax，cy: 与老师和千阁和春榆sync一下现有内容，整理修改，迭代全部论文一次。
%4.1 deadline


%随着图形神经网络的兴起，各种基于GPU加速的系统已经被开发出来，以提高GNN的效率。为了解决这两个瓶颈，基于GPU的采样方法与GPU内存中的特征缓存相继被提出，同时CPU只用于存储图数据。为了同时优化GGNN训练的两个瓶颈，这些优化通常考虑被同时使用。但在只有一个GPU的异构场景下，GPU的计算资源与存储资源都会受到争用。
%我们重新思考在异构环境中，CPU与GPU的任务分配，相比于采样，CPU更适合处理一部分GNN训练任务。新的任务分配中，为了解决GPU计算资源争用问题，GPU负责采样，CPU根据采样结果将一阶顶点特征聚合后再传输给GPU进行计算。进一步的，为了解决GPU的内存争用问题，缓存场所从GPU转移至CPU，对于可重用的聚合结果进行缓存，不再重新进行采样与聚合。NtaGNN在减轻GPU计算资源与存储资源压力的同时，减少采样与特征传输开销。
%FtmGNN重新考虑在GPU-GPU异构场景下，

% Graph neural networks(GNNs) are increased in large-scale graph-based recommender systems. Due to the GPU's limited memory capacity, which might become inadequate for big datasets, training GNN requires the mini-batch generator to sample the sparsely distributed nearby vertices and transfer their features to GPU memory. Meanwhile, feature transfer and sampling also start to slow down GNN training's performance. Feature caching in GPU memory and GPU-based sampling techniques have both been suggested as solutions to these two bottlenecks, with the CPU merely being required to store the graph data. These optimizations are usually thought of as being applied concurrently to eliminate the two bottlenecks of GNN training. However, in a heterogeneous scenario with only one or less GPU, both computational and storage resources of the GPU are contested. 


% In this work, we propose \system, a GPU-accelerated system that support general and efficient sampling-based GNN training with new task assignment. In a heterogeneous system, where the CPU is better equipped to execute some GNN training tasks than sampling, we rethink the task assignment between CPU and GPU. The CPU collects the first layer vertex features based on the sample results before sending them to the GPU for computation. Furthermore, the cache is moved from the GPU to the CPU, and the reusable aggregation results are cached without resampling and aggregation. \system reduces the contention on GPU computational and storage resources while reducing the sampling and feature transfer overhead.
%However, training GNNs efficiently on large graphs poses significant challenges due to its high computational complexity.
%Sample-based GNN training has gained widespread acceptance due to its reduced memory footprint and comparable accuracy to full-graph training. Existing frameworks utilize GPUs to accelerate the training process and combine mini-batch and sampling techniques to overcome the memory limitation of GPUs, enabling efficient model training on large graphs. 
\begin{abstract}

Graph Neural Networks (GNNs) have demonstrated outstanding performance in various applications. Existing frameworks utilize CPU-GPU heterogeneous environments to train GNN models and integrate mini-batch and sampling techniques to overcome the GPU memory limitation. 
 % This enables efficient model training on large graphs. 
In CPU-GPU heterogeneous environments, we can divide sample-based GNN training into three steps: sample, gather, and train. Existing GNN systems use different task orchestrating methods to employ each step on the CPU or GPU. After extensive experiments and analysis, we find that existing task orchestrating methods fail to fully utilize the heterogeneous resources, limited by inefficient CPU processing or GPU resource contention. 
%As a result, they limit the overall performance by either inefficient CPU processing or contention for GPU resources.

%In the early works, sampling and part of the data loading work was performed on the CPU, causing CPU contention. To solve the bottleneck caused by CPU contention, various GPU-led optimizations have been proposed one after another, bringing performance improvements while introducing GPU contention and CPU idle. 

%\zyf{The writing of the whole article needs significant improvement. too many small errors or typos would infuriate the reviewers. } 
% \zyf{is it too narrow if only push down the only one bottom layer? see Sec 4.1}
In this paper, we propose \system, a system for sample-based GNN training that incorporates a hotness-aware layer-based task orchestrating method and ensures balanced utilization of the CPU and GPU. \system decouples the training process by layer and pushes down the training task of the bottom layer to the CPU. This significantly reduces the computational load and memory footprint of GPU training. To avoid inefficient CPU processing, \system only offloads the training of frequently accessed vertices to the CPU and lets GPU reuse their embeddings with bounded staleness. Furthermore, \system provides a fine-grained pipeline design for the layer-based task orchestrating method, fully overlapping different tasks on heterogeneous resources while strictly guaranteeing bounded staleness. The experimental results show that compared with the state-of-the-art GNN systems, \system can achieve up to 11.51$\times$ performance speedup.

\end{abstract}

\maketitle
%we rethink sample-based GNN training under CPU-GPU heterogeneous environment by analyzing the task orchestrating on both the CPU and GPU. Firstly, we propose a hotness-aware layer-based task orchestrating method to maximize the utilization of computational and memory resources on both the CPU and GPU. Secondly, we propose a super-batch pipelined training approach that interleaves CPU computation, GPU computation, and gathering steps to improve overall performance. Based on these designs, we propose \system, a system for sample-based GNN training that incorporates a new task orchestrating method and ensures balanced utilization of the CPU and GPU. The experimental results show that, compared with the current works \cite{PAGRAPH_SOCC_2020, GNNlab_EUROSYS_2022, DGL_CORR_2019}, \system can achieve up to 8.6X performance speedup in the end-to-end performance.


%we designed a CPU-centric, GPU-assisted caching strategy based on this computing framework. The cache object will be changed from a vertex feature to a vertex embedding in the bottom layer. Finally,
%Meanwhile, data loading and sampling also start to slow down GNN training's performance. Both GPU-based sampling methods and data caching in GPU memory have been proposed as solutions for these two bottlenecks. These two approaches will compete for GPU resources. More focus should be placed on the CPU's function in a heterogeneous environment than on continually adding GPUs to obtain more resources.


% \zyf{better to motivate compute-cache relationships in CPU-GPU environments. provide insights. In my opinion, the approach that the first time you present by decoupling and analyzing the key steps (sampling, transferring, caching, ....,) and their compute-cache-transfer requirements is more interesting.}




%%% do not modify the following VLDB block %%
%%% VLDB block start %%%
\pagestyle{\vldbpagestyle}

% \begingroup\small\noindent\raggedright\textbf{PVLDB Reference Format:}\\
% \vldbauthors. \vldbtitle. PVLDB, \vldbvolume(\vldbissue): \vldbpages, \vldbyear.\\
% \href{https://doi.org/\vldbdoi}{doi:\vldbdoi}
% \endgroup
% \begingroup
% \renewcommand\thefootnote{}\footnote{\noindent
% This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit \url{https://creativecommons.org/licenses/by-nc-nd/4.0/} to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing \href{mailto:info@vldb.org}{info@vldb.org}. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. \\
% \raggedright Proceedings of the VLDB Endowment, Vol. \vldbvolume, No. \vldbissue\ %
% ISSN 2150-8097. \\
% \href{https://doi.org/\vldbdoi}{doi:\vldbdoi} \\
% }\addtocounter{footnote}{-1}\endgroup
% VLDB block end %%%

%% do not modify the following VLDB block %%
%% VLDB block start %%%
\ifdefempty{\vldbavailabilityurl}{}{
\vspace{-0.0cm}
\begingroup\small\noindent\raggedright\textbf{PVLDB Artifact Availability:}\\
The source code, data, and/or other artifacts have been made available at \url{\vldbavailabilityurl}.
\endgroup
}
%%% VLDB block end %%%
\input{sec1}
\input{sec2}
\input{sec3}
\input{sec4}
\input{sec5}
\input{sec6}

% \begin{acks}
%  This work was supported by the [...] Research Fund of [...] (Number [...]). Additional funding was provided by [...] and [...]. We also thank [...] for contributing [...].
% \end{acks}

%\clearpage
\balance
\bibliographystyle{ACM-Reference-Format}
\bibliography{sample}

\end{document}
\endinput

\vspace{-0.2cm}
\section{Introduction}

%GNN背景
%GNN由传统DNN发展衍生而出，但传统DNN不能coverGNN任务，GNN有独特的图操作，要考虑都结构

Graph Neural Networks (GNNs) \cite{SURVEY_TNNLS_2021} are a novel class of Deep Neural Networks (DNNs) designed to process graph-structured data. They have demonstrated remarkable effectiveness across various graph applications\cite{GCN_iclr_2017,link_www_2021,link_neurips_2018,NMT_EMNLP_2017,Recom_kdd_22,REC_WWW_2019}.
GNNs aim to learn low-dimensional feature representations (i.e., embeddings) \cite{rep_ieee_2017} for each vertex in a graph and using the representations to graph-related tasks.
Recently, GPUs have been extensively used to accelerate GNN training owing to their high memory bandwidth and massive parallelism \cite{GNNGPU_PPOPP_2021,ROC_mlsys_2020}.
Graph learning systems like PyG \cite{PYG_corr_2019} and DGL \cite{DGL_CORR_2019} offer optimized GPU implementations and convenient programming interfaces to enhance both performance and usability of GNN training.

%As many past works \cite{Nextdoor_eurosys_2021, PAGRAPH_SOCC_2020, DATATIRING_VLDB_2022,lazygcn_2020_neuips,GNNlab_EUROSYS_2022} have pointed out, CPU-based sampling and gathering steps are slow, leading to significant waiting time for the training step on GPUs to receive the input data. In contrast, Figure \ref{fig:system_classification} illustrates that many systems \cite{DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,wholegraph_SC_2022,TurboGNN_TOC_2023,DGL_CORR_2019,PAGRAPH_SOCC_2020} rely on GPUs to handle two or three steps. Although GPU-based optimizations can enhance the performance of each step, there is a risk of GPU resource contention if many steps are deployed on the GPU. 


%Specifically, each layer of GNN can compute representations of vertices in parallel. In addition, for larger graphs, GNN can also accelerate the computation process by dividing batches among vertices and then computing in parallel on different batches.
% \wqg{ From here to the end of this paragraph, nothing usefull, can remove.}
% GNN models must take into account vertex programming abstractions and graph operation primitives in comparison to conventional deep learning models, which presents a challenge to the current frameworks supporting DNNs like TensorFlow \cite{Tensflow_osdi_2016}, PyTorch \cite{pytorch_NeurIPS_2019}, and MXNet \cite{Mxnet_corr_2915}. Hence, multiple efforts exist to develop open-source machine learning frameworks adapted for GNNs, such as PyTorch Geometric (PyG) \cite{PYG_corr_2019} and Deep Graph Library (DGL) \cite{DGL_CORR_2019}, in response to the rapidly expanding demand to apply GNNs in a variety of fields. These GNN frameworks include several message-passing operators that are highly efficient and graph-specific computation layers that were missing from the earlier DNN frameworks. 

%\wqg{Comments 1:}%我觉得你可以在这里说一下：因为 GNN具有庞大的计算量，和大量的随机内存访问需求，使用GPU成为了广泛使用的方法。 
%aix：删减+修改第一段

%仅仅有支持普通GNN模型的框架是不够的，GNN与DNN相同，都有支持大规模数据的需求。
% However, the computational layer or message passing operator implementation is just one aspect in the complexity of GNN training. Real-world graphs usually have large scales and high-dimensional features, frequently 300 to 600 dimensions or more, at the vertices and edges. Early GNN implementations tended to concentrate on small-scale networks and made the assumption that the entire graph could fit in a single GPU memory. However, for large graphs whose node/edge feature data cannot fit into the GPU memory, at least part of the graph needs to be placed into the CPU memory. One common practice to train GNNs in such scenario is to create a smaller set of problem by performing a mini-batched training. With the mini-batch training, only a subset of nodes are randomly picked along with their neighboring nodes and sent to GPU. This method is very effective when training GNNs on large graphs as it practically reduces the memory footprint of the application. Not only that, several recently introduced GNN models showed that the mini-batched based approaches are superior in achieving high training accuracy as well.

%aix:补充说明 sampling-based GNN的训练过程和在CPU-GPU异构环境中的三个阶段
Considering the growing sizes of graphs in real-world applications, Full-graph GNN training that loads and trains the entire graph on the GPU is impractical due to the limited the GPU memory capacity \cite{hongtu_SIGMOD_2024,SAGA_SIGMOD_2022, MOSAIC_EUROSYS_2017, Marius++_arxiv_2022,SANCUS_VLDB_2022}. 
As a result, the sample-based approach for training GNNs has emerged as a promising solution. This approach performs computations on sampled subgraphs, allowing the training of large input graphs with limited GPU resources \cite{Graphsage_2017,GCNWS_SIGKDD_2018,DLG_TKDE_2022}.
%这里你说一下训练构成，简单但是准确，1）sampling-based training 把训练顶点划分batch。然后训练中，以每个batch的训练点为一个处理单元，首先通过采样操作，获取邻居子图（基于大概什么样的策略）。然后根据采样子图，收集涉及到的feature信息。 然后将采样子图和收集到的feature一起发送到GPU端执行训练。 并且将执行过程分为三个阶段
In this approach, the input data, including vertex features and graph structure, are stored in the host memory while the training process is offloaded to GPUs \cite{DGL_CORR_2019,distdgl_ieee_2020}. 
The sample-based GNN training divides the training vertices into multiple batches, treating each batch as the basic unit for training.
To train on a batch, the training process first samples the $K$-hop subgraphs for the training vertices, following a given neighbor sampling rule, then gathers the required vertex features based on the sampled subgraph and loads the subgraph and the involved vertex features onto the GPU. Finally, the GPU performs a forward and backward pass and updates the corresponding model parameters. 

% \wqg{I think there is no need becuase we have detailed introduction in the background}
\begin{comment}
For each training vertex, it uses same graph sampling algorithms to sample a fixed number of neighbors within $L$ hops. A sample is made up of the training vertex and its corresponding sampled neighbors. A batch contains a certain number of samples and the features of sampled vertices. Meanwhile, the graph data and the high-dimensional features are often stored in the host memory. This needs to frequently transfer batch data from CPU to GPU through PCIe interconnect. Therefore, we can summarize the sampling-based GNN training in the CPU-GPU heterogeneous environment into three steps, \textbf{Sampling}, \textbf{Data loading}, and \textbf{Training}. The entire training process repeats the three steps until the GNN model converges to the expected accuracy.
\end{comment}

\begin{comment}
Early GNN implementations concentrated on small-scale networks and assumed that the entire graph could fit in a single GPU memory. However, the size of real-world graph data and the feature dimension of vertices or edges are increasing, and GPU memory usually cannot accommodate. The method of \textbf{Sampling-based GNN training} is proposed \cite{Graphsage_2017} to solve the scalability problem. For each training vertex, it uses same graph sampling algorithms to sample a fixed number of neighbors within $L$ hops. A sample is made up of the training vertex and its corresponding sampled neighbors, and all samples are processed in a mini-batch. The entire training process repeats this iteration (i.e.,epoch) until the GNN model converges to the expected accuracy.
%minibatch可以分为三个阶段，sampling ,data loading，training,早期工作中，sampling以及data loading中的feature collection都是在CPU执行的，且feature collection占用大部分data loading时间，甚至超过CPU-GPU data movement的时间。我们将sampling与data loadig瓶颈产生的原因归为 High CPU contention。
\end{comment}


%For sampling-based GNN training, the CPU-GPU heterogeneous environment is the typical scenario. First, GPUs can accelerate the GNN training process very well. Meanwhile, the limited GPU memory cannot fully accommodate the training data, especially high-dimensional features. The graph data and feature data are often stored in the host memory. This needs to frequently transfer training data from CPU to GPU through PCIe interconnect. Therefore, we can summarize the sampling-based GNN training in the CPU-GPU heterogeneous environment into three steps, \textbf{Sampling}, \textbf{Data loading}, and \textbf{Training}.

% \wqg{Then you need to discuss the resource they used and briefly summarize the existing frameworks. It is not necessary to "define" the processing method, it is not that clear compared to the GPU-accelerated graph processing, just talk about How they process each stage, briefly!:}


%\multirow{3}*
%& {\textbf{Pros}} &\textbf{Cons}


%S D T可以在CPU或GPU，
This training mechanism, known as the \textbf{sample-gather-train} paradigm has gained widespread adoption in various GNN systems \cite{DGL_CORR_2019,PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,PYTDIRECT_VLDB_2021,Marius_OSDI_2021,Marius++_arxiv_2022,lazygcn_2020_neuips,TurboGNN_TOC_2023,wholegraph_SC_2022,quiver_2023_arxiv,FASTGcn_iclr_2018,DSP_PPOPP_2023,ducati_sigmod_2023,GNNautoscale_icml_2021,refresh_arxiv_2023,legion_atc_2023}. These systems decouple the three steps and deploy them on different computation devices to achieve high performance. 
%Depending on the resources they mainly utilized, each step can work in a CPU-centric or GPU-centric mode.  
Early systems \cite{PAGRAPH_SOCC_2020,DGL_CORR_2019,lazygcn_2020_neuips,FASTGcn_iclr_2018,Marius++_arxiv_2022} employ graph sampling and feature gathering on CPUs. These systems store the graph data in the CPU memory, perform CPU-based graph sampling, and collect the required features on the CPU side before transferring them to GPUs. This approach allows the GPUs to be dedicated solely to training tasks. On the other hand, some other systems \cite{Nextdoor_eurosys_2021,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,DGL_CORR_2019,TurboGNN_TOC_2023,wholegraph_SC_2022,ducati_sigmod_2023,DSP_PPOPP_2023,legion_atc_2023} leverage GPU-accelerated graph sampling. They store the graph topology in either GPU or pinned CPU memory and perform the sampling on GPUs, relieving CPUs from the heavy random memory access. Moreover, recent systems \cite{PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,TurboGNN_TOC_2023,wholegraph_SC_2022,legion_atc_2023,DSP_PPOPP_2023,ducati_sigmod_2023} also employ GPU-based feature gathering to optimize overall performance. They store the vertex feature completely or partially (with a caching mechanism) on GPUs, converting the heavy host-GPU data communication and CPU memory access to efficient GPU global memory access. The GPU-based graph sampling and GPU-based feature caching can be used jointly to optimize performance.

%The training step is usually accelerated using GPUs. 
%Therefore, various task orchestrating strategies exist in sample-based GNN systems for the sampling-gathering-training process. 

Figure \ref{fig:system_classification} provides an overview of existing GNN systems and their task orchestrating methods using a 3-layer binary tree. Each path from the root to the leaf represents a specific task orchestrating method, where the CPU is selected when traversing the left child, and the GPU is selected when traversing the right child. 
%Among the possible eight methods, since performing training on the CPU is not a wise option under CPU-GPU heterogeneous environments, the existing task orchestrating methods can be summarized into four categories.
Since performing training on the CPU is not a efficient option when GPU is equipped, the existing task orchestrating methods on CPU-GPU heterogeneous framework contains only four categories.

Having conducted intensive experiments and analysis, we find that existing task orchestrating methods that decouple the computation based on the step does not fully utilize CPU-GPU heterogeneous resources due to inefficient CPU processing or GPU resource contention. Assigning one or two
steps to the CPU may cause inefficient CPU processing to become
a bottleneck and leading to long waiting time for the GPU training process to receive the input training data.
In contrast, assigning two or more steps to the GPU may result in GPU resource contention although GPU parallel processing can significantly enhance the performance of each single step. This is because the training data, cached features, and graph topology need to be simultaneously stored within the constrained GPU memory while the computation kernels of sampling and training are completed for GPU cores. 
% The step-based task orchestrating methods fail to achieve adequate and balanced CPU and GPU utilization. 


% In contrast, Figure \ref{fig:system_classification} illustrates that many systems \cite{DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,wholegraph_SC_2022,TurboGNN_TOC_2023,DGL_CORR_2019,PAGRAPH_SOCC_2020} rely on GPUs to handle two or three steps. Although GPU-based optimizations can enhance the performance of each step, there is a risk of GPU resource contention if many steps are deployed on the GPU. 
%which also prevents us from overlapping the computation of different batches. 
%These two cases, \textbf{(1) CPU contention with GPU idle}, and \textbf{(2) CPU idle with CPU contention}. 
%GNN training usually overlaps different batches for concurrent execution, and the sampling kernel and training kernel may compete for GPU cores. Therefore, 
%We summarize the major problems of existing computation orchestrating strategies into two types: \textbf{(1) GPU contention but CPU idle}, \textbf{(2) GPU wait caused by CPU contention}. 
%Most applications developed for heterogeneous systems are more suitable for the GPU than for the CPU. Only considering the device suitability when scheduling the applications overloads the GPU and leaves the CPU idle. On the other hand, overusing the CPU and making the GPU wait only makes the situation worse. 
%Figure \ref{fig:utilization} displays four task orchestrating methods that we configure in DGL \cite{DGL_CORR_2019}. 
To illustrate this, we implement the four task orchestrating methods in DGL \cite{DGL_CORR_2019} and compare their performance on a 3-layer GCN model with the Reddit dataset \cite{GCN_iclr_2017}. We evaluate the GPU utilization, CPU utilization, and per-epoch runtime as shown in Figure \ref{fig:utilization}. 
We can observe that the existing task orchestrating methods do not effectively utilize CPU-GPU heterogeneous resources due to either unbalanced resource utilization or low overall resource utilization, resulting in sub-optimal performance.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\linewidth,page={1}]{figures/system-classification.pdf}
  \vspace{-0.8cm} %调整图片与上文的垂直距离
  \caption{Classification tree of existing GNN systems and their task orchestrating methods. }
     \vspace{-0.4cm}
  \label{fig:system_classification}
\end{figure}
% \ccy{(Modified: 1. GNNLab 2. Alignment: Sample Gather Train 3. Last row horizontally aligned)}

\begin{figure}[!t]
  \centering
  \includegraphics[width=\linewidth,page={1}]{figures/utilization.pdf}
  \vspace{-0.6cm} %调整图片与上文的垂直距离
  \caption{Comparison of resource utilization and per-epoch execution time for different task orchestrating methods. The S, G, and T represent the sample, gather, and train steps of sample-based GNN training.}
     \vspace{-0.5cm}
  %\zyf{font style is times new roman, but the font style in Fig. 1 is Arial or . You should make the fond style consistent in all figures.}
  %\zyf{the font style is not consistent with others}} %\zyf{S,D,T should be explained in fig or captain}}
  \label{fig:utilization}
\end{figure}

%\wqg{To balance the resource utilization and improve the overall performance, we reposition the roles of CPU and GPUs in the pipeline of sampling-based GNN training, and propose a new computation orchestrating method. In traditional GNN systems, the CPU is often excluded from computation tasks due to its low parallelism. However, we observe that using the CPU can help alleviate the contention of GPU, especially, reduce the transmission overhead between CPU and GPUs. Different from existing GNN training methods that place each step into a single device, we decouple the computation by layers and distribute the computation to both CPU and GPU.}
%In traditional GNN systems, the CPU is often excluded from computation tasks due to its low parallelism. However, we observe that using the CPU can help alleviate the resource contention of the GPU and decrease the total transfer amount.

In this work, we rethink the roles of CPU and GPU in sample-based GNN training and propose a hotness-aware layer-based task orchestrating method to optimize performance. Unlike existing methods that allocate each step to a single device, we decouple the training task by layers and employ the computation of each sub-task (sample-gather-train) to a single device. This is based on our observation that the multiple hops of neighbor sampling cause most of the training resources (especially the memory) consumption to be in computations from the bottom. Offloading computation of the bottom layer to the CPU can significantly reduce the computation and memory requirement of the GPU. Furthermore, the volume of CPU GPU communication can be significantly reduced by transferring computed feature embeddings instead of raw features. Considering offloading the computation of a complete GNN layer to the CPU may make the CPU processing a new bottleneck if not carefully optimized, we propose a hotness-aware embedding reusing method that reduces CPU computation by periodically computing the embeddings of frequently accessed vertices (i.e., hot vertices) and reusing them across batches.
To sustain the correctness of training with embedding reuse, we extend bounded staleness \cite{ssp_2013_nips,SANCUS_VLDB_2022,ST_ICML_2018,Marius_OSDI_2021} processing to guarantee that the reused embedding is within a given stale constraint. Moreover, to further improve performance, we propose a super-batch pipelined training that combines $k$ adjacent batches into a super-batch and controls the staleness of hot embeddings among super-batches, overlapping GPU and CPU computation tasks while strictly and efficiently implementing bounded staleness. 


%To accommodate different CPU-GPU environments while minimizing CPU-GPU communication, we propose a hot vertice hybrid process policy that adjusts the ratio of hot vertices assigned to CPU embedding computation and GPU feature caching based on available heterogeneous resources.

% Moreover, to further improve resource utilization on the CPU-GPU heterogeneous platform, we propose an embedding-feature-hybrid cache policy that adjusts the ratio of hot vertices assigned to caching and CPU computation.
% A super-batch pipelined training that combines $k$ adjacent batches into a super-batch and controls the update and reuse of hot embeddings among super-batches, overlapping GPU and CPU computation tasks while strictly guaranteeing bounded staleness.
%为了最小化CPU-GPU通信同时考虑到CPU资源与GPU资源的变化关系，我们提出了Embedding-Feature-Hybrid Cache Policy。对于给定热顶点比例，在GPU资源匮乏时，使用CPU为更多热点计算嵌入，GPU只缓存少量顶点特征。相反，在GPU资源充足时，减少CPU中的嵌入计算同时提升GPU中缓存顶点特征的比例。
% \notecolor{ 
% To minimize CPU-GPU communication while considering the changing relationship between CPU resources and GPU resources, we propose an embedding-feature-hybrid cache policy. For a given ratio of hot vertices, when GPU resources are scarce, the CPU is used to compute embeddings for more hot vertices, and the GPU caches only a small number of vertex features. In contrast, when GPU resources are sufficient, the embedding computation in the CPU is reduced while the proportion of cached vertex features in the GPU is increased.
% }

% 







%By employing this hotness-aware layer-based task orchestrating method, inefficient CPU processing and GPU resource contention can be effectively eliminated, thus maximizing the utilization of heterogeneous resources.
%Through this method, the computation volume of the CPU can be reduced while maintaining the model's accuracy. 

%The engines of \system require different resources, including CPU for bottom layer training of hot vertices, GPU for non-hot vertices and other layers training, and PCIe for the gathering step. A fine-grained pipeline design is necessary to ensure that the different resources in \system are fully utilized.

%Concurrently executing tasks deployed on the GPU and CPU is essential to achieve high performance. However, it is challenging in \system because the bounded staleness training creates cross-layer data dependencies between the CPU and GPU training process. To efficiently resolve the CPU-GPU staleness control, we propose a super-batch pipelined training to fully overlap various tasks while strictly guaranteeing bounded staleness. 

%Typically, a historical embedding becomes stale after a few batches \cite{SANCUS_VLDB_2022}. Selecting hot vertices from a limited number of upcoming batches is more reasonable and efficient than selecting hot vertices from the entire training set.

%Specifically, we combine $k$ batches into a super-batch and select the hot vertices for each super-batch. Within each super-batch, the CPU computes the bottom layer embeddings of hot vertices for the next super-batch. At the same time, the GPU pulls historical embeddings from the previous super-batch on demand. This pipeline design ensures that historical embeddings from the last super-batch are only utilized within $k$ batches in the current super-batch, enabling effective control of bounded staleness.

%总结概括一下这个过程
%To further reduce CPU computation and ensure bounded staleness, We shrink the selection granularity of the hot vertices queue to only the multiple batches. Before training starts, we combine $K$ batches into a super-batch and select the hot vertices queue for each super-batch. During the pipeline training, CPU computation and GPU computation are overlapped. In each super-batch, the CPU computes the bottom layer embeddings of hot vertices for the next super-batch, and the GPU uses the historical embeddings from the last super-batch on demand. Meanwhile, in each super-batch, the data loading step of each batch also overlaps with the computation. In this process, the version difference (staleness) between the historical embedding and latest model parameter is limited to $2k$. 

%The CPU and GPU resources can be utilized in a balanced manner. However, simply pushing the computation for the bottom layer down to CPU may still lead to inferior performance as the heavy computation in CPU may cause GPU wait.

%\wqg{comment}The bottom layer can naturally be executed on the CPU since vertex features are stored here. By pushing down the bottom layer computation to the CPU, firstly, the GPU no longer needs to cache the vertex features, and secondly, the computation task of the GPU is effectively reduced. In addition, this new computation orchestrating method may decrease the total transfer amount because the transfer object from CPU to GPU is changed from vertex features to vertex embedding.

%To solve the above issues, we need to design a new computation orchestrating strategy. Therefore, we rethink the role of the CPU in sample-based GNN training. The CPU is often excluded from computational tasks because it can become a performance bottleneck. However, in GNN training, the CPU may be the key to resolving GPU resource contention. Different from dividing GNN training into three steps, we decoupled the GNN training by layers. Finally we found that instead of being responsible for sampling or data loading, the CPU is better suited to perform the training task of the bottom layer. It can naturally be executed on the CPU since vertex features are stored here. The bottom layer is the most crucial in a GNN model. It dominates the time overhead of sampling, data loading, and training. By pushing down the bottom layer to the CPU, firstly, the GPU no longer needs to cache the vertex features, and secondly, the computation task of the GPU is effectively reduced. In addition, this new computing framework may decrease the total transfer amount by approximately ten times because the transfer object from CPU to GPU is changed from vertex features to vertex embedding. With this new GPU-centric, CPU-assisted computing framework, we eliminate the contention on the GPU and reduce the total transfer amount. 


%原本的都是静态缓存，缓存feature，内存开销大。 设计一个针对于embedding的动态缓存，兼顾节省内存和减少传输量。同时还能防止CPU成为瓶颈。

% 我觉得这里history embedding caching 是为了优化CPU计算，甚至比GPU的优化更重要。

%Based on this static task decoupling method, we further propose a set of optimizations to improve performance. 
%A fixed division of computation by layer is not enough to ensure a balanced workload between CPU and GPU. 


%Through this method, the CPU can reduce GPU resource contention with the less computation volume while maintaining the model accuracy. 

%we suggest that the CPU only computes the embeddings of the frequently accessed vertices for the bottom layer, and let GPU reuse these embeddings among batches with a bounded staleness \cite{Marius_OSDI_2021,SANCUS_VLDB_2022}. Therefore, we propose the idea of Hot Vertices Embeddding Reusing based on our computation orchestrating method.
%To further balance the workload between CPU and GPU, we design a computation reduction method based on historical embedding reusing method.
%The core ideas is reusing the frequently accessed vertex embeddings of the bottom layer among batches with a bounded staleness \cite{Marius_OSDI_2021,SANCUS_VLDB_2022}.

%Specifically, a set of frequently accessed vertices are selected as hot vertices. During computation, the bottom layer embeddings of hot vertices are periodically recomputed by the CPU, and in each batch, the GPU reuses the historical embeddings from the CPU on demand. In this process, we monitor the version difference (staleness) between the historical embedding and latest model parameter, and adjust hot vertices amount to control the historical embedding update period in the CPU.

%\wqg{Do we really need the following sentence?}Moreover Users can tune the computation volume by adjusting the ratio of cached vertices and staleness parameter.

%Unlike prior studies that \cite{PAGRAPH_SOCC_2020,DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022} cache vertex features in the GPU to reduce GPU-CPU transfer overhead, 

\begin{comment}
However, simply pushing the computation for the bottom layer down to CPU may still lead to inferior performance as the heavy computation in CPU may cause GPU wait. To further balance the workload between CPU and GPU, we design a dynamic cache policy based on historical embedding. Unlike past works that \cite{PAGRAPH_SOCC_2020,DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022} cache vertex features in the GPU to reduce GPU-CPU transfer overhead, Our approach caches the computed embeddings in the GPU with bounded staleness \cite{Marius_OSDI_2021,SANCUS_VLDB_2022} to reduce the computation overhead in the CPU. 
Specifically, after selecting the cached vertices, the CPU periodically computes the embedding of all cached vertices to ensure that the embedding version within the bound. In each batch, for the cached vertices, the GPU pulls the historical embeddings from the CPU on demand and caches them until their version are stale. For the other vertices in the bottom layer, the GPU pulls the features of sampled vertices. 
Through this dynamic caching policy, the computation amount in the CPU can be reduced. Users can tune the computation volume by adjusting the ratio of cached vertices and staleness parameter.
\end{comment}

%To prevent the embedding version from becoming too stale, a complete cached vertices embedding update will not take longer than the training time for a given number of batches.


%And the users also have the  by adjusting the ratio of cached vertices, which help to ensure a balanced CPU-GPU workload. 

%In this way, CPU can share GPU resource pressure with the least computation amount. 

%Past works \cite{PAGRAPH_SOCC_2020,DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022} have shown that a portion of the vertices is more likely to be accessd, and caching the features of them in the GPU can significantly reduce the transfer volume. Following this idea, we suggest that the limited CPU computing power should be provided for the vertices with higher sampling frequency instead of all the vertices in the bottom layer. In addition, past works \cite{Marius_OSDI_2021,SANCUS_VLDB_2022} have shown that using some historical embeddings with bounded staleness allowed does not affect the accuracy. Therefore, caching historical embedding in the GPU and reusing it as many times as bounded staleness allows can further save computation in the CPU. 


%the CPU no longer computes the complete bottom layer but only the cached vertices. 

%In addition, we design a super-batch pipeline based on our caching policy to maximize GPU and CPU utilization. 

%To further improve CPU-GPU resource utilization and reduce the transfer amount with less GPU memory, we propose a dynamic cache policy based on historical embedding.  Past works \cite{PAGRAPH_SOCC_2020,DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022} design static cache policies on features, which have a lot of additional memory overhead. It is not suitable for situations such as low GPU memory, a huge model. Moreover, the hit ratio of the static cache policy decreases as the batch size increases, because the proportion of low degree vertices increases and available memory for caching decreases. 


%Like the traditional caching strategy, we need to select a set of vertices that are accessed more frequently. This set may be 10\%-20\% or more of the total vertices. However, unlike traditional caching strategies, the cached objects are embedding in our caching strategy. Therefore, the CPU no longer performs the complete bottom layer but only the cached vertices. These historical embeddings are cached on the CPU or GPU and used as many times as bounded staleness allows. Secondly, we propose a super-batch pipeline with historical embedding based on our caching strategy to maximize GPU and CPU utilization under balanced task scheduling. Historical embedding will be used multiple times in a super-batch, which contains many batches. In addition, \system adds a CPU-GPU balance check in the pipeline design. \system will check the pipeline status and adjust the amount of cached vertex set to control the volume of CPU tasks. Through these designs, we ensure that the GPU does not wait for tasks in the CPU to maximize GPU utilization. As shown in Figure \ref{fig:utilization}, \system makes full use of the CPU and GPU resources. 

%A fixed division of tasks by layer is not enough to ensure a balanced workload between CPU and GPU. Therefore, to prevent CPU tasks from growing into a new bottleneck, we design a new caching strategy and a super-batch pipeline with historical embedding.

%\zyf{I think it is a well-motivated problem and a nice brief introduction}


% reduce the total amount of computation, we introduced asynchronous training in the bottom layer. As long as bounded staleness\cite{SANCUS_VLDB_2022} allows, all of the bottom layer vertices in the CPU are trained in full-neighborhood without sampling, and the vertex embedding results are cached for subsequent batches.


%另外，为了加速training过程，同时防止CPU上的计算成为新的瓶颈，我们设计了一种新的缓存策略。首先缓存对象不再是顶点特征，而是第一层的顶点嵌入，同时缓存场所从GPU转为CPU，GPU的有限内存不再是缓存策略扩展性的瓶颈。CPU中的所有第一层顶点进行不采样的全邻居训练，并保存顶点嵌入结果，在bounded staleness允许的情况下供后续batch使用。为了进一步节省传输开销，GPU也可以作为一部分高热度顶点嵌入的缓存场所。

%1.底层计算量/传输量大，PCIe瓶颈，CPU带宽比pcie大，feature size小了，顶点数也少了，
%针对这个实验，确定问题在GPU资源不够，第一转到CPU部分任务，改变更小的缓存结果。
%In addition, we develop a novel caching strategy to speed up the training step and avoid the CPU computation from growing into a new bottleneck. First, the cache object is no longer the vertex feature, but the vertex embedding of the first layer, and the cache site is switched from GPU to CPU so that the limited memory of GPU is no longer the bottleneck for the scalability of the caching strategy. As long as bounded staleness allows, all of the first layer vertices in the CPU are trained in full-neighborhood without sampling, and the vertex embedding results are cached for subsequent batches. A portion of the \textcolor{red}{high hotnees} vertex embeddings can also be cached on the GPU to further reduce data loading overhead.

% In addition, as observed in section \ref{sec3}, as sampling and data loading are optimized, training progressively becomes a significant overhead. To train GNNs in the limited memory of GPU, minibatch is proposed, and batch size as a very important hyperparameter determines the size of memory occupied by a batch in minibatch training. Minibatch is changed to full batch when the batch size reaches the maximum value, the number of training points. Past work\cite{} has shown that full batch training has higher accuracy and shorter training time compared to minibatch training. We further find that the above conclusion is also reflected in the variation of batch size, i.e., the larger the batch size, the higher the training accuracy and the shorter the time. In our computing framework, we have two advantages for providing larger batch sizes. Firstly, there is no longer a need to cache vertex features to reduce the amount of data transfer, and GPU memory is fully available for use by batch data. Second, as the most memory-consuming bottom layer in batch data, its computation occurs in the CPU, which facilitates the provision of a larger batch size.

%我们重新考虑CPU-GPU异构系统下的GNN训练，提出一种新的GPU为中心，CPU为辅助计算模式，并依赖此计算模式设计新的CPU为中心，GPU为辅助的缓存策略，使GNN训练不再受GPU的资源争用问题困扰。
% We rethink Mini-Batch GNN training in CPU-GPU heterogeneous systems, redefine the computational tasks that CPUs and GPUs are responsible for and propose a new GPU-centric and CPU-assisted computational framework. We eliminate resource contention within the GPU. In terms of computational resources, we lighten the GPU task load and reduce the total task load by training asynchronously; in terms of storage resources, limited GPU memory is no longer required to cache vertex features, thus having the ability to support training closer to full batch and improving overall performance. Based on the above analysis, we propose \system, a system for sample-based GNN training with a new computing framework and memory allocation strategy. 

%对比DGL, GNNlab。

By integrating the above techniques, we propose \system, a sample-based GNN system that can efficiently utilize the resources of heterogeneous CPU-GPU environments. 
% We evaluate the performance of \system using two popular GNN models, namely GCN \cite{GCN_iclr_2017} and GraphSAGE \cite{Graphsage_2017}, across five real-world graphs. Experimental results demonstrate that, compared with three state-of-the-art GNN systems, DGL \cite{DGL_CORR_2019}, Pagraph \cite{PAGRAPH_SOCC_2020}, and GNNlab \cite{GNNlab_EUROSYS_2022}, \system achieves up to 8.64X, 7.11X, and 2.02X speedups, respectively. 
% 
We make the following contributions.

\begin{itemize}[leftmargin=*]
    \item We provide a comprehensive analysis of resource utilization issues associated with the task orchestrating methods for sample-based GNN systems on GPU-CPU heterogeneous platforms.
    
    \item We propose a hotness-aware layer-based task orchestrating method that effectively leverages the computation and memory resources of the GPU-CPU heterogeneous system.
    
    \item We propose a super-batch pipelined task scheduling method that seamlessly overlaps different tasks on heterogeneous resources and efficiently achieves strict bounded staleness.
    %simultaneously reduce the amount of computation and transfer and increase the value of CPU computing power.
    %\item Based on the GPU-centric, CPU-assisted computing framework, we design and implement \system, a CPU-GPU balanced sample-based GNN system with a new caching strategy and well-designed pipeline \wqg{I think we need to "rethink" this, isn't new caching our main method ?}
\end{itemize}

We evaluate the performance of \system using three popular GNN models, GCN \cite{GCN_iclr_2017}, GraphSAGE \cite{Graphsage_2017}, and GAT \cite{GAT_ICLR_2018}. 
Experimental results demonstrate that, compared with five state-of-the-art sample-based GNN systems, DGL \cite{DGL_CORR_2019}, Pagraph \cite{PAGRAPH_SOCC_2020}, GNNlab \cite{GNNlab_EUROSYS_2022}, DSP \cite{DSP_PPOPP_2023}, and GNNAutoScale \cite{GNNautoscale_icml_2021}, \system achieves speedups of up to 11.51$\times$, 9.72$\times$, 2.43$\times$, 1.63$\times$, and 9.18$\times$, respectively. 


% The rest of this work is summarized as follows. Section \ref{sec2} presents the background information relevant to this work. Section \ref{sec3} presents the experimental analysis of limitations related to existing task orchestrating methods. Section \ref{sec4} provides a detailed description of the proposed optimization techniques, as well as the architecture and implementation of \system. Section \ref{sec5} analyzes the experimental results. Section \ref{sec6} presents a discussion on related work. Section \ref{sec7} concludes the paper.




%\zyf{it is not so obvious that the four configurations in the figure cover all cases. need more explanations. }
%For optimal performance, a good CPU-GPU heterogeneous framework should make full use of CPU and GPU simultaneously.
% \ccy{Figure \ref{fig:utilization} presents the results of NeutronOrch without explanation. Also, do we  need to give the results of NeutronOrch here?}

%The efficiency of a heterogeneous system requires considering both of the issues we mentioned above. 

% In addition, in order to eliminate GPU contention in GPU-centric framework, existing work \cite{GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022} opts to employ a single multi-GPU environment, adding extra GPU to get adequate resources and leaving the CPU idle, which results in more energy consumption. For optimal performance and cost friendly, a good CPU-GPU heterogeneous framework should make balanced use of CPU and GPU.

% Based on the above analysis, we can observe that minibatch-based GNN training shifts from high CPU contention to high GPU contention. We conduct experiments in DGL\cite{DGL_CORR_2019} to compare the performance of CPU sampling and GPU sampling. As observed in Table \ref{tab:breakdown-GPU_Sampling}, although GPU sampling can accelerate the sample process better (0.33 vs. 0.83), the simultaneous execution of training and sampling in the GPU causes GPU contention, which also wastes CPU computing power and ultimately leads to an overall performance worse than that of CPU sampling (See Section\ref{sec3} for details). In order to eliminate GPU contention, existing work\cite{GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022} opts to employ a single multi-GPU environment, adding extra GPU to get adequate resources and leaving the CPU idle, which results in more energy consumption. For optimal performance and cost friendly, a good CPU-GPU heterogeneous framework should make balanced use of CPU and GPU.

%Given a mini-batch size, CPUs repeatedly sample input graph and collect the scattered features into a contiguous memory before transferring them. Finally, GNN forward and backward computation for the given minibatch data is performed on the GPU. Although the GPU can accelerate the training process very well, the excessive tasks on the CPU causes the GPU to wait frequently. As observed in section \ref{sec3}, the feature collection takes much longer than the CPU-GPU data movement. Due to high CPU contention, early mini-Batched GNN training is often throttled by sampling and data loading. 

%Meanwhile, this needs to frequently transfer mini-batch data from CPU to GPU through a slow PCIe interconnect. 
%为了解决High CPU contention，三种GPU主导的优化被提出，分别是，zero copy based data loading，GPU sampling，GPU feature caching。
% Three GPU-led optimizations have been proposed to alleviate the CPU contention. First, a novel GPU-oriented data loading approach was proposed, where GPU threads directly access sparse features in host memory through zero-copy accesses without feature collection in CPU. Second, in many cases, a major reason for such a large size of GNN datasets is not the graph topological data itself, but the features of vertices. Based on this observation, some studies attempted to transfer graph topological data into the GPU memory and apply GPUs to accelerate sampling. Finally, prior works found that some vertices are more frequently sampled than others, and thus some static caching strategies are introduced to reduce data movement from the host memory to the GPU memory.

%但这三种优化的使用，将Mini-Batched GNN Training从原本的High CPU contention变为另一个极端，High GPU contention。首先，在GPU存储资源角度，三种完全不重合的数据都需要被保存在GPU内存，分别是 training batch data, cached feature, and graph topology data。而且根据第三节的分析，batchdata和cachedfeature均是与内存正相关的存在，即使用更多内存就更有利于训练。其次，在GPU计算资源角度，Mini-Batched GNN Training通常是流水线执行的，这就会导致sampling transfer与trainig阶段有重合执行的时候。sampling与training都是GPU中执行，同时执行会竞争GPU cores导致性能下降，而zero copy作为GPU主导的传输，也会消耗部分GPU cores。在DGL中，依次加入优化，观察优化效果，而当将几种优化同时使用时，由于GPU资源的争用，效果反而变差。另外当前工作主要关注优化sampling与data loading阶段，这是没错的，但随着sampling与data loading的不断加速，training会成为新的瓶颈，这是没人关注的。而且基于第三节的分析，traning的时间与采样子图的规模相关，且当规模不断增大时，training时间会主导整个end to end的过程，所以优化training是很有必要的。
% While these GPU-led optimizations continue to improve the performance of GNN training, they leave GPU resources contention and CPU resources idle. First, in terms of storage resources, three types of data must be stored simultaneously in the limited GPU memory: training batch data, cached feature, and graph topology data. Second, in terms of computational resources, minibatch-based GNN training usually overlaps different batches for concurrent execution, and the sampling kernel and training kernel may compete for GPU cores. Meanwhile, zero-copy, a GPU-led data transfer, eliminates the feature collection phase of the CPU, but also requires the use of some GPU cores in the transfer process. This is understandable, as these optimizations were proposed by separate works and were not considered compatible with each other. 

%\textcolor{red}{In order to observe the improvement of these optimizations, we start from the pure Mini-Batched GNN Training in DGL\cite{DGL_CORR_2019} and integrate zero-copy-based data loading, GPU-based caching and GPU-based Sampling one by one. As observed in Table \ref{tab:breakdown}, each optimization can deliver significant performance gains when used individually, but when all of them are used together, the performance gains are not as good as the individual optimizations due to GPU resource contention.} In addition, as observed in section \ref{sec3}, as sampling and transmission are optimized, training progressively becomes a significant overhead.

%根据前文，我们可以总结发现，GNN training从CPU contention 变为GPU contention，现有工作如通过扩展GPU数量获得更多的资源。但是，只有一个或较少的GPU的异构框架内，这些优化难以被同时使用。为了获得最佳的性能，一个优秀的CPU-GPU异构框架应该平衡的利用CPU与GPU。
% Based on the above analysis, we can observe that minibatch-based GNN training shifts from high CPU contention to high GPU contention. We conduct experiments in DGL\cite{DGL_CORR_2019} to compare the performance of CPU sampling and GPU sampling. As observed in Table \ref{tab:breakdown-GPU_Sampling}, although GPU sampling can accelerate the sample process better (0.33 vs. 0.83), the simultaneous execution of training and sampling in the GPU causes GPU contention, which also wastes CPU computing power and ultimately leads to an overall performance worse than that of CPU sampling (See Section\ref{sec3} for details). In order to eliminate GPU contention, existing work\cite{GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022} opts to employ a single multi-GPU environment, adding extra GPU to get adequate resources and leaving the CPU idle, which results in more energy consumption. For optimal performance and cost friendly, a good CPU-GPU heterogeneous framework should make balanced use of CPU and GPU.

%However, the benefits of these optimizations cannot be simultaneously achieved in a heterogeneous system with only one or less GPU.
%实验一拿到这个位置，CPU sample加GPU training和GPU sampling加GPUtraining对比，直接提出GPU contention的
%1. CPU 采样 GPU训练  无pipline
%2. CPU 采样 GPU训练  有pipeline
%3. GPU 采样 GPU训练  无pipeline
%4. GPU 采样 GPU训练  有pipline

% 1.在有了缓存以及zc之后，传输瓶颈得到了较好的解决，而对于GNN的另一瓶颈sample，GPU 加速的sampling虽然能较好的加速sample过程，但同时在GPU中执行 training与 sampling ，造成GPU的contention，CPU idle 浪费了CPU资源。同时对于整体性能，并不如传统的CPU sampling。（这里需要实验支撑）。首先没有piepline的情况 GPU采样肯定是整体更好，因为这时CPU采样的情况有等待。 但是加入pipeline后，GPU采样并没有明显提升，可是CPU采样的情况则因为解决GPU等待的同时因为CPU算力的加入，得到可观的性能。

%现有先进工作 gnnlab 使用单机多卡模式，使得单个GPU分别执行采样或者训练。CPU只负责聚集feature的任务，但在zero copy被应用后，CPU的feature聚集是不再被需要的。在使用GPU同时进行sampling与training的情况下，会造成CPU idle。 在单机多卡的情况下，浪费一定CPU资源不会对整体性能有多大影响，但是在仅有一个或少量GPU的情况下，CPU与GPU共同工作是一种更好的方式。

%为了在仅有一个或少量GPU的前提下，我们提出一种新的GNN训练模式，在同时利用现有GPU主导的优化的前提下，保证性能
% CPU采样通常被认为

%在CPU-GPU异构场景中，CPU常常被排除在计算任务之外，因为这可能会成为性能瓶颈。但在GNN训练中，CPU可能是解决GPU资源争用的关键。我们将GNN的训练过程解耦并重新审视每一个步骤，最后我们发现CPU与其执行采样或特征收集，其更适合执行部分训练任务,尤其是最下层训练。CPU作为顶点特征的储存位置，可以很自然的执行GNN最下层的计算。一个GNN模型通常由多层构成，其中最关键的是第一层的计算，它主导着GNN训练三个阶段的时间开销。通过下沉它的计算到CPU，首先GPU不再需要缓存顶点feature，其次GPU的计算任务也得到一定程度减少。另外，在这种新的计算模式中，因为CPU-GPU之前的传输对象从顶点特征转为顶点嵌入，在没有任何缓存的情况下，传输总量可能被减少近十倍
%high level 讲一下 balanced 的必要。


% \begin{table}[t]
% 	\caption{Summary of existing systems.}
% 	\hspace{-0.05in}
% 	\label{tab:system_overview}
% 	\centering
% 	{\renewcommand{\arraystretch}{1.2}
%      \resizebox{8cm}{!}{

% 	\begin{tabular}{l c c c c c}
% 		\hline
		
% 		\hline
% 		\textbf{Systems}& {\textbf{Sampling}} &
% 		{\textbf{Data Loading}} &{\textbf{Training}} 
%         %&{\textbf{CPU Utilization}} &{\textbf{GPU Utilization}}
%         \\
% 		\hline
%   	{LazyGCN \cite{lazygcn_2020_neuips}}&CPU & CPU & GPU %& high & low
%    \\
%   	{MariusGNN \cite{Marius++_arxiv_2022}}& CPU & CPU & GPU %&high & low
%    \\
% 		{PaGraph \cite{PAGRAPH_SOCC_2020}}&{CPU} &GPU & GPU %& medium & medium
%    \\
% 	    {PyTorch-Direct \cite{PYTDIRECT_VLDB_2021}} & CPU & GPU & GPU %& medium& medium
%    \\
% 		{Data Tiering \cite{DATATIRING_VLDB_2022}} & GPU & GPU & GPU %&low & high
%    \\
% 		{GNNlab \cite{GNNlab_EUROSYS_2022}}&GPU & GPU& GPU %&low & high
%    \\
%         {WholeGraph \cite{wholegraph_SC_2022}}&GPU&GPU&GPU
%    \\
% 		{DGL \cite{DGL_CORR_2019}}&{CPU/GPU} & CPU/GPU & CPU/GPU %& medium & medium
%    \\
%         \hline
		
% 		\hline
% 		\vspace{-0.15in}
% 	\end{tabular}
% 	} } 
% 	\vspace{-0.1in}
% \end{table}

%For sampling-data loading-training processing, the left subtree represents the utilization of CPU resources and the right subtree represents the utilization of GPU resources. 
%Illustration of task orchestrating method and related systems.

% \wqg{Find a space to insert a table, summarize the existing systems and the resources their steps use}



\section{Background}
%GNN; mini-batch training; existing framework and their challenges; summary;
\label{sec2}
%We first briefly introduce Graph Neural Networks (GNNs) and the process of sample-based GNN training. 


%Then we present an overview of existing frameworks and address the challenges associated with their step-based task orchestrating strategies.


% \begin{table*}[t]
% 	\vspace{-0.03in}
%         \centering
%     \caption{The runtime breakdown (in seconds) of a training epoch on DGL with key optimizations(GZC:zero-copy based data transfer; GC: GPU-based Caching; GS: GPU-based Sampling). \textbf{GNN}: A 2-layer GCN with random neighborhood sampling. \textbf{Dataset}:Orkut. \textbf{Environment}:Intel Silver 4210 10-core CPU,  128GB DRAM, 1 NVIDIA-GTX 2080Ti GPU( 34 SMXs,  4352 cores, 11GB GDDR6 RAM).}	
%     \label{tab:breakdown}
% 	\centering
%  \small
% 	{\renewcommand{\arraystretch}{1.2}
%  %\renewcommand{\arraystretch}{1}
% 	%\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c c c c c c c}
% 		\hline
		
% 		\hline
% 		\multirow{1}*{\textbf{GNN Systems}} \textcolor{red}{here}&%这里建议写全了，然后把表格字体调小
%             \multirow{1}*{\textbf{CPU Utilization}}  &
% 		\multirow{1}*{\textbf{GPU Utilization}} &
%         \multirow{1}*{\textbf{Sampling}} &
% 		\multirow{1}*{\textbf{Data Loading}}& {\textbf{Training}} & {\textbf{Total}}\\
% 		\hline
%     	{DGL}& {35.4\%}&{81\%}&{1.39}&{3.67} &{4.90} & {5.28} \\ %batchsize 1000
%     	{w/ zero-copy}& {34.6\%} & {78\%} & {1.39} &{3.75} &{4.79} & {5.32}  \\   %batchsize 1000
% 	    {w/ GPU-base Caching}& {30.6\%} & {82\%} & {1.53} &{4.09} &{5.87} & {6.01} \\     %batchsize 500
% 	    {w/ GPU-base Sampling}& {14.3\%}  & {72\%}& {0.46} &{3.64} &{4.73} & {8.94} \\      %batchsize 500
% 	    {ALL}& {12.5\%}  & {72\%} & {0.46} &{3.47} &{5.41} & {9.58} \\   %batchsize 200
% 		\hline
	
% 		\hline
% 	\end{tabular}
% 	}
% \end{table*}


%待解决问题
%1.第一章motivation实验，可以明确的是，我们是要说，平衡利用CPU及GPU的重要性
% 实验体现出CPU GPU利用率，突出CPU GPU 利用率都高的情况更好，体现为3会表现的比1和2更好。但是会暴露一个问题就是，3 本身就是一种较为平衡的方法，我们要解决的其实就优化3，介绍1和2会显得冗余。而3又不是一种的最先进方法，因为现有异构都在做单机多卡，2才是公认最先进的方法。而在单机多卡情况下，其实不存在GPU contention的情况，因为GPU足够多，资源够用。  
%总之，我们要解决的问题，单机单卡，或GPU很少的情况下才存在，或者说，GPU资源不够多的情况下，平衡利用CPU与GPU才有价值。 这样的话可能没法对比GNNlab，GNNlab的优化没法在单机单卡中用，他是不同GPU只做Sample或者training，然后加一个presample-based的caching。
%如果只针对单机单卡，会不会让人感觉这个问题是我们创造的，一是因为这些优化是被单独提出和设计系统的，二是同时用这些优化的如gnnlab，GNNtier都是单机多卡去用的。 怎么能让人觉得CPU和GPU同时使用是合理的很关键。
%1.提出问题
%三种模式下的CPU GPU利用率 一个epoch的时间 

%1. CPU Ccontention : CPU采样+EXP传输+GPU训练         
%2. GPU contention  : GPU采样+ZR传输+Cache+GPU训练      

%3. CPU GPU banlance ： CPU采样  + ZR传输 + CACHE + GPU训练      

%在只有1和2的情况下，感觉没有直接说出平衡利用CPU和GPU是更好的方法，所以我想补个3，一种相对来说CPU GPU balanced的模式。然后观察它们三者一个epoch的时间。我想用这个说明CPU GPU balance的是有机会超过两种contention的方法。
%如果上面的实验ok，那我会继续引出我们的方法。

%2.总结问题，解释现象
%对于3，我们可以观察到sampling的时间要小于data loading+ training，在一个良好的流水线设计下，不仅不会因为使用CPU算力而导致产生瓶颈，反而会得到更加的性能。
%原因如下。
%（1）引入了更多算力。但在任务均衡的情况下，不会导致GPU等待。（2）假设。我们抽象整个GNN训练任务的任务量为T，T包括Sampling，data loading，training。 如第三节分析，过度的使用GPU资源所导致的GPU contention。 由于可用的cache比例及batchsize大小降低，会使得总的任务量 T 增加。

%但是简单的使用3，并不能时刻保证最佳性能。我们知道，针对复杂的GNN模型以及不同的CPU-GPU架构，任务量T与算力C是不固定的，固定的GNN任务分配可能会打破CPU与GPU之间的平衡。

%3.解决问题：我们的方法  
%第一， 我们可以抽象CPU负责的任务量为T1, 算力为C1; GPU负责的任务量为T2,算力为C2。 传输方向若为CPU-GPU则归为T2, 反之归为T1。 T = T1 + T2。 

%倘若满足一个条件，即在当前工作负载与CPU-GPU架构下，T1/C1 约等于 T2/C2，系统将会得到最佳性能。

%为此，我们解耦并分析GNN的每个阶段的工作量及所需算力，并针对不同的GNN工作负载以及CPU-GPU异构算力，制定算力-任务量平衡 且 任务量最小化（CPU GPU balanced）的workflow。
%4. 我们方法

%老师的行文思路
%做实验分析的文章，

% \begin{table}[t]
% 	\vspace{-0.03in}
%         \centering
%     \caption{The runtime breakdown (in seconds) of a training epoch on DGL with key optimizations(GZC:zero-copy based data transfer; GC: GPU-based Caching; GS: GPU-based Sampling). \textbf{GNN}: A 2-layer GCN with random neighborhood sampling. \textbf{Dataset}:Orkut. \textbf{Environment}:Intel Silver 4210 10-core CPU,  128GB DRAM, 1 NVIDIA-GTX 2080Ti GPU( 34 SMXs,  4352 cores, 11GB GDDR6 RAM).}	
%     \label{tab:breakdown}
% 	\centering
% 	{\renewcommand{\arraystretch}{1}
% 	\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c c c c c}
% 		\hline
		
% 		\hline
% 		\multirow{1}*{\textbf{Opt}} &
% 		\multirow{1}*{\textbf{CPU Utilization}} &
%             \multirow{1}*{\textbf{CPU Utilization}} &
%             \multirow{1}*{\textbf{Sampling}} &
% 		\multirow{1}*{\textbf{Data Loading}}& {\textbf{Training}} & {\textbf{Total}}\\
% 		\hline
%     	{CPU contention}& {100\%}& {20\%} & {7}&{0} & {8}  & {19.8}\\
%     	{GPU contention}& {0\%}& {100\%} & {5}&{0} & {9}  & {19.8}\\
% 	    {CPU-GPU balanced}& {80\%} & {9} & {6}& {6} &{5}  & {19.8}\\
% 		\hline
	
% 		\hline
% 		\vspace{-0.35in}
% 	\end{tabular}}
% 	}
% \end{table}

\subsection{Graph Neural Networks}

% \zyf{lowercase $k$ is a specific layer, uppercase $K$ is the total number of layers}
%首先是传统GNN的训练过程，按层计算，聚合邻居。其次，通过图的规模不断增大的特点以及图的幂律分布特性，引出Mini-batched gnn training
Graph-structured data serves as the input for Graph Neural Networks (GNNs), where each vertex or edge is associated with a high-dimensional feature vector.
A typical GNN model comprises multiple layers that compute a low-dimensional embedding for each vertex. 
Each layer contains an aggregation phase and an update phase \cite{GNNlayer-2020}. 
For example, in a GNN with $L$ layers, during the aggregation phase of layer $l$, each vertex $v$ combines its neighbors' embedding vectors at the $l - 1$ layer with its own embedding vector to generate the aggregation result $a_{v}^l$ using an aggregation function:
\begin{align}
    a_{v}^{l} &=AGGREGATE(h_u^{l-1} | \forall {u} \in {N_{in}(v)} \cup  \{v\})
 \label{form:agg_forward}
\end{align} 
\noindent where ${N_{in}(v)}$ represents the incoming neighbors of vertex $v$, $h_v^{l}$ represents the node embedding vector of vertex $v$ at l-th layer, and $h_v^0$ is the input feature of vertex $v$. The aggregate functions can be \texttt{sum}, \texttt{average}, \texttt{max/min}, etc. Next, during the update phase, each vertex computes its output embedding vector $h_{v}^l$ by applying an update function to the aggregation result $a_{v}^l$:
\begin{align}
    h_v^{l}&=UPDATE(a_{v}^{l}, W^{l})
 \label{form:update_forward}
\end{align}
After $L$ layers, each vertex’s feature vector becomes a low-dimensional embedding of its neighbors up to $L$ hops away. These embeddings can be utilized for performing a wide range of tasks, such as node classification and link prediction.


% Forward Propagation
% 
% \begin{itemize}[leftmargin=*]
% \item Forward Propagation
% \begin{align}
%     a_{v}^{l-1} &=AGGREGATE(h_u^{l-1} | \forall {u} \in {N_{in}(v)} \cup  \{v\})
%  \label{form:agg_forward}
% \end{align}
% \begin{align}
%     h_v^{l}&=UPDATE(a_{v}^{l-1}, W^{l-1})
%  \label{form:update_forward}
% \end{align}
% % $\nabla_{\mathbf{W}^{l}} \mathcal{L}$ & Gradient matrix of the loss $\mathcal{L}$ with respect to $\mathbf{W}^{(l)}$ at the $l^{th}$ layer  \\
% \end{itemize}

% In backpropagation, we first derive the recurrence to backpropagate the gradient. Let  $\nabla_{\mathbf{h}^{l-1}_{v}} \mathcal{L} $ denote the gradient of the loss $ \mathcal{L}$ with respect to $\nabla_{\mathbf{h}^{l-1}_{v}} $. By the chain rule, the relation between  $\nabla_{\mathbf{h}^{l-1}_{v}} \mathcal{L} $ and $\nabla_{\mathbf{h}^{l}_{v}} \mathcal{L} $ is: 
% \begin{align}
%       \nabla_{\mathbf{h}^{l-1}_{v}} \mathcal{L} &= \frac{\partial \mathcal{L}}{\partial h^l_v} \frac{\partial h^l_v}{\partial a_{v}^{l}}  \frac{\partial a_{v}^{l}}{\partial h_v^{l-1}}
% \end{align}
% The gradient $\nabla_{\mathbf{h}^{l-1}_{v}} \mathcal{L} $ can be computed in two steps. First, compute the gradient of the aggregation result of vertex $v$. Then, compute the gradient  $\nabla_{\mathbf{h}^{l-1}_{v}} \mathcal{L} $ of vertex  $v$.
% \begin{align}
%      \frac{\partial h^l_v}{\partial a_{v}^{l}} &= \frac{\partial \mathcal{L}}{\partial h^l_v}  \frac{\partial h^l_v}{\partial a_{v}^{l}}  = \delta^{l}_{v}  UPDATE^{'}
%      \label{form:agg_backward}
% \end{align}
% \begin{align}
%      \nabla_{\mathbf{h}^{l-1}_{v}} \mathcal{L} &=  AGGREGATE^{'}(a_u^{l} | \forall {u} \in {N_{out}(v)} \cup  \{v\})
%      \label{form:emb_backward}
% \end{align} 
% where $\delta^{l}_{v}$ denotes $\nabla_{\mathbf{h}^{l}_{v}} \mathcal{L} $. $UPDATE^{'}$ is the derivative of the activation function of $UPDATE$, and  $AGGREGATE^{'}$ is the derivative of the activation function of $AGGREGATE$.  When all vertices compute the aggregated gradient, each vertex pulls the aggregated gradient of its outgoing neighbors and computes the gradient $\nabla_{\mathbf{h}^{l-1}_{v}} \mathcal{L} $ according to $AGGREGATE^{'}$.





\begin{table}[tbp]
	\centering 
	\caption{\label{tab:test}Notations}
 \vspace{-0.1in}
  \scalebox{0.75}{
	\begin{tabular}{lcr}
		\toprule
		Notation & Description  \\
		\midrule
        $\mathbf{a}_v^{l}$ & aggregation results vector of vertex $v$ at the $l^{th}$ layer \\
        $\mathbf{h}_v^{l}$ & embedding vector of vertex $v$ at the $l^{th}$ layer \\
		$\mathbf{A_i}$ & Adjacency matrix of the $i^{th}$ iteration \\
		$\mathbf{\hat{A}_i}$ & Adjacency matrix after symmetric normalization of the $i^{th}$ iteration \\
		$\mathbf{W}_i^{l}$ &   Weight matrix at the $l^{th}$  layer of the $i^{th}$ iteration\\
		$\mathbf{H}_i^{l}$ & Embedding matrix at the $l^{th}$ layer of the $i^{th}$ iteration  \\
		$\mathbf{Z}_i^{l}$ & Input matrix to activation function at the $l^{th}$ layer of the $i^{th}$ iteration \\
        $\nabla \mathcal{L}$ & Gradient of the loss function \\
		$\nabla_{\mathbf{Z}^{l}} \mathcal{L}=\delta^{l}$ &  Gradient matrix of the loss $\mathcal{L}$ with respect to $\mathbf{Z}^{l}$ at the $l^{th}$ layer \\
		$\nabla_{\mathbf{W}^{l}} \mathcal{L}$ & Gradient matrix of the loss $\mathcal{L}$ with respect to $\mathbf{W}^{l}$ at the $l^{th}$ layer  \\
        $g(W_i)$ & Gradient of the loss function with respect to $W$ at the $i^{th}$ iteration \\
		$\mathcal{L}$& Loss of the GNN \\
        $I$ & Number of iterations in an epoch or number of batches \\
        $L$ & Number of layers of GNN model \\
		$\eta$& Learning rate \\
		$\epsilon$& Staleness bound \\
		\bottomrule
	\end{tabular}
 }
 \vspace{-0.2cm}
\end{table}

% At a high level, the computation in a GNN can be broken down into two main steps: (1) \textbf{Aggregation} refers to the process of gathering information from a node's in-coming neighbors and summarizing it in some way. This summarized information can then be used to update the node's own feature representation. There are several ways to perform aggregation, such as summing, averaging, or taking the maximum of the neighbors' features. (2) \textbf{Combination} refers to the process of combining information from a node's own features with the summarized information obtained from aggregation. There are several ways to perform combination, such as concatenation, element-wise multiplication, or summation. Then using a neural network usually MLP to transform the feature vector of each vertex. The aforementioned computation on a graph can be formally expressed by

% \begin{align}
%     a_{v}^l &=AGGREGATE(\{h_u^{l-1} | {u} \in {N(v)}\})\\
%     h_v^{l}&=\sigma(W^{(l)} \cdot COMBINE(h_v^{l-1}, a_{v}^l))
% \end{align}

% \noindent Where, $h_v^{(l)}$ denotes the node embedding vector of vertex v at l-th layer, $N(v)$ represents the incoming neighbors of vertex v, $\sigma$ is a non-linear function (e.g., ReLU). $W^{(l)}$ is the weight matrix that transform the node embedding at (l-1)-th layer, Specially, $h_v^0$ is the input feature of vertex v. AGGREGATE and COMBINE are two key components of GNNs that allow nodes to share information with their neighbors and update their feature representations based on that information. By combining these two operations in different ways, GNNs can learn powerful representations of complex graphs that can be used for a wide range of tasks.



%\zyf{what do circle and square mean? 0,2,8 should be drawn under the embedding but not inside. It should be better to add 1..5..10 to the original features in the CPU graph. place sampling under the graph in CPU. place training under the subgraph and features in GPU.}


 % \zyf{rewrite caption, please check.} \ccy{Modified: 1. The topmost symbol description is displayed in the center   2.  "two-hop", uppercase t}

% \begin{figure*}[htbp]
%   \centering
%   \includegraphics[width=\textwidth,page={1}]{figures/CPU-GPU.pdf}
%   \vspace{-0.4cm} %调整图片与上文的垂直距离
%   \caption{Two frameworks for sample-based GNN training 
%   %\wqg{This figure delivers to less information, consider remove and it and summary the existing systems with a figure.By the way, discuss it in section 2}
%   }
%   \vspace{-0.4cm}
%   \label{fig:CPU-GPU}
% \end{figure*}


\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth,page={1}]{figures/Sample.pdf}
   \vspace{-0.4cm}
  \caption{An example of sample-based GNN training for a two-layer GNN, where vertex 4 is the vertex with a ground-truth label for training.}
   \vspace{-0.4cm}
  \label{fig:Sample}
\end{figure}



\begin{algorithm}[t]
\caption{Sample-based mini-batch GNN training (per epoch)} \footnotesize 
\label{alg:sampleflow}  
\begin{algorithmic}[1]
    \Require graph structure  $G(V,E)$, vertex features $\{\textbf{h}^{0}_v\mid v\in V\}$, training set of vertices with ground-truth labels $V_{\mathbb{L}}$, number of layers $L$, initial model parameters $W^{l}_0$ for each layer $l$, and batch size $b$
  %number of layers $L$,  vertex labels $\{\mathbb{L}_v\mid v\in V_{\mathbb{L}}\}$, initial model parameters $W^{(l)}$ for each layer $l$, fanout $\{k_1, k_2, \cdots, k_L\}$, batch size $b$, number of iterations in an epoch $I$
    \Ensure updated parameters
    \State $\{V_0,\ldots V_I\} \leftarrow \texttt{split}{V_{\mathbb{L}}}$ by batch size $b$; \textcolor{gray}{//split training vertices into $I$ batches}
    %\State randomize vertex order in $V_{\mathbb{L}}$;
    \For{$i$ = $1$ to $I$}
        \State \textcolor{gray}{/* Sample a subgraph for a batch of training vertices */}
        \State $G_i\leftarrow \emptyset$; $V^F$= $V_i$; 
        \For{$l$ = $L$ to $1$} 
            \State $G^{l-1}(V^{l-1}, E^{l-1})\leftarrow$ \texttt{sample}($V^F$) \textcolor{gray}{// Sample one hop subgraph }
            \State $G_i=G_i\cup G^{l-1}$ ; $V^F\leftarrow V^{l-1}$
        \EndFor
            % \For{each $u$ in $V^l$}
            %         \State $V_{smpl}\leftarrow$ randomly select at most $k_l$ in-neighbors of $u$;
            %         \State $E_{smpl} \leftarrow$ the edges between $u$ and $V_{smpl}$;
            %         \State $V^{l-1} \leftarrow V^{l-1} \cup V_{smpl}$; $E^{l-1} \leftarrow E^{l-1}\cup E_{smpl}$;
            %     \EndFor
            %     \State $G_{i}\leftarrow G_{i}\cup V^{l-1} \cup E^{l-1}$;
            % \EndFor
        \State \textcolor{gray}{/* Gather the vertex features of sampled subgraph $G_i$ */}
        \State $H_i\leftarrow \{\textbf{h}_v^0\mid v\in G_i\}$
        \State \textcolor{gray}{/* Train $V_i$ with sampled subgraph $G_i$ and update parameters */}
        \For{$l$ = $1$ to $L$}
            \State Forward compute the vertex embeddings with $G_i, H_i, W^l_{i-1}$ based on Equation (\ref{form:agg_forward}) and (\ref{form:update_forward}); 
        \EndFor
        \State Compute the loss $\mathcal{L}$ and generate backward gradients;
        \For{$l$ = $L$ to $1$}
                \State Backward compute the gradients with $G_i, H_i, W^l_{i-1}$; 
                % based on Equation (\ref{form:agg_backward}) and (\ref{form:emb_backward});
                \State Update $W^{l}_i$ based on $W^{l}_{i-1}$ and $\nabla W^{l}_{i-1}$;
        \EndFor
    \EndFor                              

	\end{algorithmic}  
\end{algorithm}


% \begin{comment}
% \begin{algorithm}[t]
% \caption{Sample-based mini-batch GNN training (per epoch)} \small 
% \label{alg:sampleflow}  
% \begin{algorithmic}[1]
%     \Require graph structure  $G(V,E)$, vertex features $\{\textbf{h}^{0}_v\mid v\in V\}$, training set of vertices with ground-truth labels $V_{\mathbb{L}}$, number of layers $L$, initial model parameters $W^{l}_0$ for each layer $l$, sampling fanout $k_l$ for each layer $l$, and batch size $b$
%   %number of layers $L$,  vertex labels $\{\mathbb{L}_v\mid v\in V_{\mathbb{L}}\}$, initial model parameters $W^{(l)}$ for each layer $l$, fanout $\{k_1, k_2, \cdots, k_L\}$, batch size $b$, number of iterations in an epoch $I$
%     \Ensure updated parameters
%     \State $num\_batch \leftarrow \lceil |V_{\mathbb{L}}|/b\rceil$; \textcolor{gray}{//get number of batches}
%     %\State randomize vertex order in $V_{\mathbb{L}}$;
%     \For{$i$ = $1$ to $num\_batch$}
%         \State \textcolor{gray}{/* Sample a subgraph for a batch of training vertices */}
%         \State $V_i\leftarrow \emptyset$; $G_i\leftarrow \emptyset$; \textcolor{gray}{//training set and sampled subgraph of batch $i$}
%         \For{each $v$ in $V_{\mathbb{L}}$}
%             \State $V_i\leftarrow V_i\cup v$; $G_{i}\leftarrow G_{i}\cup v$;
%             \State $V^L \leftarrow \{v\}$; \textcolor{gray}{//for obtaining sampled subgraph targeted at $v$}
%             \State \textcolor{gray}{/* construct the sampled subgraph of $v$ layer by layer */}
%             \For{$l$ = $L$ to $1$} 
%                 \For{each $u$ in $V^l$}
%                     \State $V_{smpl}\leftarrow$ randomly select at most $k_l$ in-neighbors of $u$;
%                     \State $E_{smpl} \leftarrow$ the edges between $u$ and $V_{smpl}$;
%                     \State $V^{l-1} \leftarrow V^{l-1} \cup V_{smpl}$; $E^{l-1} \leftarrow E^{l-1}\cup E_{smpl}$;
%                 \EndFor
%                 \State $G_{i}\leftarrow G_{i}\cup V^{l-1} \cup E^{l-1}$;
%             \EndFor
%             \If{$|V_i| \geq b$}
%                 \textbf{break;}
%             \EndIf
%         \EndFor
%         \State \textcolor{gray}{/* Gather the vertex features of sampled subgraph $G_i$ */}
%         \State $H_i\leftarrow \{\textbf{h}_v^0\mid v\in G_i\}$
%         \State \textcolor{gray}{/* Train $V_i$ with sampled subgraph $G_i$ and update parameters */}
%         \For{$l$ = $1$ to $L$}
%             \State Forward compute the vertex embeddings with $G_i, H_i, W^l_{i-1}$ based on Equation (\ref{form:agg_forward}) and (\ref{form:update_forward}); 
%         \EndFor
%         \State Compute the loss $\mathcal{L}$ and generate backward gradients;
%         \For{$l$ = $L$ to $1$}
%                 \State Backward compute the gradients with $G_i, H_i, W^l_{i-1}$ based on Equation (\ref{form:agg_backward}) and (\ref{form:emb_backward});
%                 \State Update $W^{l}_i$ based on $W^{l}_{i-1}$ and $\nabla W^{l}_{i-1}$;
%         \EndFor
%     \EndFor

%   $batch \leftarrow \{\emptyset,\ldots,\emptyset\}$
%             \State $batch$, $I$ = SplitMiniBatch( $\{v \mid v\in V_{\mathbb{L}}\}$,  $b$) \textcolor{gray}{//Split mini-batches}
%             \For{$i$ = $1$ to $I$}       \textcolor{gray}{//Sample}
%                 \State $nbrs = batch[i]$, $sampledgraph\leftarrow \emptyset$, $requiredfeature\leftarrow \emptyset$
%                 \State $sampledgraph.$append( $nbrs$);
%                 \For{$l$ = $L$ to $1$}
% 		            \State $nbrs$= OneHopSample($nbrs$, $k_l$);
%                     \State $sampledgraph.$append($nbrs$);
% 	            \EndFor  
%                 \For{each $v$ in $sampledgraph$} \textcolor{gray}{//Gather}
%                     \State $requiredfeature.$push(${h}^{(0)}_v$);
% 	            \EndFor  
%             \State Input $sampledgraph$, $requiredfeature$ to GNN model with $W_{i-1}^{(l)}$;
%             \newline \underline{ \textbf{Forward:}}   \textcolor{gray}{//Train}
%                 \For{$l$ = $1$ to $L$}
%                     \State execute Formulas  (\ref{form:agg_forward}), (\ref{form:update_forward}) 
%                 \EndFor
% 		  \newline \underline{ \textbf{Backward:}}
% 		      \For{$l$ = $L$ to $1$}
%                     \State execute Formulas  (\ref{form:agg_backward}), (\ref{form:emb_backward})
%                     \State update $W_{i}^{(l)}$ based on $\nabla W_{i-1}^{(l)}$ 
%                 \EndFor
%             \EndFor
% %                              

% % 	\end{algorithmic}  
% % \end{algorithm}
% \end{comment}

% \begin{algorithm}[t]
% 	\caption{Sample-based GNN training process} \small 
% 	\label{alg:sampleflow}  
% 	\begin{algorithmic}[1]
% 		\Require graph structure  $G(V,E)$, number of layers $L$, vertex features $\{\textbf{h}^{(0)}_v\mid v\in V\}$, vertex labels $\{\mathbb{L}_v\mid v\in V_{\mathbb{L}}\}$, initial model parameters $W^{(l)}$ for each layer $l$, fanout $\{k_1, k_2, \cdots, k_L\}$, batch size $b$, number of iterations in an epoch $I$
% 		\Ensure updated parameters $W^{(l)}$ for each layer $l$ 
% 		\State $batch_t\leftarrow \{\emptyset,\ldots,\emptyset\}$, $subgraph\leftarrow \{\emptyset,\ldots,\emptyset\}$
%             \State $batch_t$, $I$ = DetermineBatch( $\{v \mid v\in V_{\mathbb{L}}\}$,  $b$   )
%             \For{each $batch$ in $batch_t$}       \textcolor{gray}{//Sample}
%                 \State $dst = \{batch\}$, $sampledgraph\leftarrow \emptyset$
%                     \For{$l$ = $L$ to $1$}
%                     \State $sampled_{(ngbr)}\leftarrow \emptyset$
%                      \For{each $v$ in $dst$}
% 		              \State $nbrs= uniform\_select(v, k_l)$;
%                         \State  $sampled_{(ngbr)}.$append( $\{nbrs\}$)
% 	            \EndFor        
%                     \State $sampledgraph.$appendLayerGraph($dst, sampled_{(ngbr)}, l$)
%                     \State $dst = sampled_{(ngbr)}$
%                 \EndFor
%             \State $subgraph.$appendGraph($sampledgraph$)
%              \EndFor
%             \For{each $subg$ in $subgraph$}   \textcolor{gray}{//Train}
%             \newline \underline{ \textbf{Forward:}}   
%             \For{$l$ = $1$ to $L$}
%                     \State execute Formulas  (\ref{form:agg_forward}), (\ref{form:update_forward}) 
%              \EndFor
%              \For{each $v\in V_{\mathbb{L}}$}
% 		    \State $\hat{\mathbb{L}_v}=\overrightarrow{\mathcal{P}} \big(\textbf{h}^{(L)}_v\big)$ 
% 		\EndFor
% 		\State $loss=loss\_func \big(\{\hat{\mathbb{L}_v}\mid v\in V_\mathbb{L}\}, \{\mathbb{L}_v\mid v\in V_{\mathbb{L}}\}\big)$
% 		\newline \underline{ \textbf{Backward:}}
%             \For{each $v\in V_{\mathbb{L}}$}
% 		    \State $\nabla \textbf{h}^{(L)}_v=\overleftarrow{\mathcal{P}} \big(loss,\textbf{h}^{(L)}_v\big)$
% 		\EndFor
% 		\For{$l$ = $L$ to $1$}
%                  \State execute Formulas  (\ref{form:agg_backward}), (\ref{form:emb_backward})
%              \State update $W^{(l)}$ based on $\nabla W^{(l)}$ 
%            \EndFor
%          \EndFor
% 	\end{algorithmic}  
% \end{algorithm}




\vspace{-0.3cm}
\subsection{Sample-based Mini-batch GNN Training}

%将minibatch gnn分为三个阶段，sampling，data loading， training，配合训练图
% \wqg{WQG:comments}
%这里写的太虚了，根本看不懂为啥要用sampling-based GNN。你解释一下说法，GPU的memory limitation咋来的. 而且这里就讲sampling-based GNN training 不要去讲GPUCPU的问题。
%Specifically, each layer of a GNN can parallelly compute representations of vertices.
%The aggregation and update phase of GNNs involves complex matrix and vector operations on large-scale graph data. GPUs are generally more suitable for handling such computational tasks. Machine learning systems like PyTorch Geometric (PyG) \cite{PYG_corr_2019} and Deep Graph Library (DGL) \cite{DGL_CORR_2019} offer optimized GPU implementations and user-friendly programming interfaces, enhancing the training of GNNs on GPUs.A GNN model can be trained simply on GPUs by using whole-graph training \cite{GNNGPU_PPOPP_2021,ROC_mlsys_2020,neutron_sigmod_2022}, in which the whole graph is loaded into GPUs and each vertex aggregates its neighbor features. However, real-world graphs are often massive and include high-dimensional vertex/edge features, which may exceed the memory capacity of GPUs \cite{SAGA_SIGMOD_2022, MOSAIC_EUROSYS_2017, Marius++_arxiv_2022}. To address the GPU memory limitation in training GNN models, state-of-the-art systems adopt sample-based mini-batch GNN training \cite{PAGRAPH_SOCC_2020,lazygcn_2020_neuips,Marius++_arxiv_2022,DGL_CORR_2019,DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,wholegraph_SC_2022,TurboGNN_TOC_2023}.

% The key idea of sample-based GNN training is to construct sampled subgraphs. According to the granularity of sampling algorithm execution, sampling algorithms can be divided into vertex-wise, layer-wise, and subgraph-wise sampling. Taking vertex-wise sampling as an example, Subsequently, neighbor sampling is performed for each vertex to obtain a layer of sampling results. This process is iterated multiple times to obtain a L-layer sampled subgraph.


Sample-based mini-batch GNN training splits the training vertices into multiple mini-batches. Each time it loads a batch of training vertices (with ground-truth labels) along with the sampled subgraph targeted at these vertices into GPUs for training \cite{Graphsage_2017,GCNWS_SIGKDD_2018,DLG_TKDE_2022}. Algorithm \ref{alg:sampleflow} elaborates the sample-based mini-batch GNN training process for a single epoch. It first splits training vertices $V_{\mathbb{L}}$ into $I$ mini batches according to the batch size $b$ (Line 1), then samples the multi-hop subgraph $G_i$ of each batch (Line 3-9). The sampled subgraph is generated by a reverse traversal from the training vertices (Line 5-7). There is a \textit{fanout} parameter to specify the number of sampled neighbors in the \texttt{sample} function per layer or per vertex for controlling the size of the sampled subgraph. It is noticeable that there exist other sampling schemes that help improve the accuracy, such as importance-aware sampling \cite{FASTGcn_iclr_2018}. We show a simple and general case since the sampling scheme is not our focus in this paper. According to the sampled subgraph $G_i$, we launch the gathering step to collect the features $H_i$ of sampled vertices (Line 9). The batch of training vertices $V_i$, the gathered features $H_i$, and sampled subgraph $G_i$ are then fed into the GNN model for training (Line 11-16). Specifically, there is a forward computation process of vertex embeddings (Line 11-12), loss computation with predicted labels and ground-truth labels (Line 13), and a backward computation process of gradients (Line 14-15). The layer-wise parameters $W^l$ are updated after each batch processing, so the parameters are updated multiple times instead of once in full-graph training.

%The algorithm first randomly splits batches of the training label vertices (target vertices) according to the batch size $b$ (Line 2). The number of iterations $I$ is the number of batches in an epoch, which represents that the model weights will be updated $I$ times. Compared with the full-graph training method, the model weights of the sample-based GNN training are updated more frequently. Then, to obtain sampled subgraph, the same graph sampling algorithms are utilized for each training vertex in a batch to select the random neighbors within $L$ hops (Line 3-8). The number of neighbors sampled in each layer depends on the fanout $\{k_1, k_2, \cdots, k_L\}$. Subsequently, the features of each vertex in the sampled subgraph are gathered into $requiredfeature$ and fed into the GNN model with $sampledgraph$ for training (Line 9-11). Finally, GNN training is performed on the mini-batched and sampled subgraph and their corresponding features (Line 12-16).
% For large graphs, mini-batch training can achieve faster convergence and higher accuracy \cite{distdglv2_sigkdd22}.

% \zyf{how is this subgraph generated? it seems that you first time mention subgraph but I don't know what subgraph is. I think the description is not clear. mini-batch (parameters updated more frequently than whole-batch), sample-based that generates subgraph. what do you mean by target vertex in fig 3. Is it possible to give a formal definition of the sampled subgraph? or an algorithm to describe the sample-based mini-batch training?}.

Figure \ref{fig:Sample} shows an illustrative example of sample-based mini-batch GNN training model mapping to CPU-GPU heterogeneous execution environment. This is a 2-layer GNN model on a graph of 9 vertices with a training set containing only a single vertex 4. 
The input data, including graph data and high-dimensional features, are typically stored in host memory, while GPUs perform the GNN training on mini-batched and sampled subgraphs.
In each training batch, this approach follows a \textbf{sample-gather-train} processing flow.
Firstly, the graph sampling algorithm uniformly selects two neighboring vertices for each vertex.
Secondly, the required vertex features are gathered based on the sampled subgraph and the training vertices. Then, they are loaded from the CPU to the GPU through a PCIe interconnect.
Finally, the GPU performs forward and backward computations, computing and updating the model parameters correspondingly.
% ccy comments: 在第一段介绍GPU 训练时，是不是可以直接介绍sample-gather-train(in each iteration, the training process can be abstract into three steps: (1) sample..  (2)gather...  (3)train...), 然后下一段用图解释(As shown in Figure 3...)。  现在这么写可能表达不是特别直接

% %传统minbatched GNN 的workflow
% Prior work has shown that the sample-based approach can achieve almost the same training accuracy but with much less computational cost and scales well for large graphs \cite{ST_ICML_2018,sampling_caa_2022,GNNSampling_sigops_2021}. Hence, it has been widely adopted by existing GNN systems. Furthermore, since the sample-based approach forms the grid-structured data with fixed size (i.e., sampled vertices and their features), GPUs are becoming popular in GNN training. Figure \ref{fig:CPU-GPU} (a) shows a baseline for sample-based GNN training in CPU-GPU heterogeneous environment. All graph topological data and features are kept in the CPU. For each mini-batch, graph sampling is performed on CPUs sequentially; then the sampled vertices and their features are transferred to GPU memory for model training. 

% %传统Minibatched GNN中 采样和传输成为瓶颈
% However, with the increase of input data size—large-scale graphs and high-dimensional features, sampling the graph on CPUs and transferring features to GPU memory become two main performance bottlenecks. Past work has often focused on only one of two bottlenecks. Table \ref{tab:breakdown} reports the runtime breakdown of representative GNN systems for training a 3-layer GCN on Reddit. After enabling GPU-based training, the Sampling and Data loading stages dominate the end-to-end GNN training time, accounting for 24\% and 54\%, respectively, on DGL. Consequently, this motivates recent research efforts to further improve GNN training in these two bottlenecks.

% The running time of the Data loading stage is mainly dominated by loading features of sampled vertices from host memory to GPU memory due to the limited PCIe bandwidth (normally less than 16 GB/s). To solve this problem two optimizations have been proposed.

% \vspace{-0.05in}
% \subsection{Existing Frameworks and Their Challenges} 
% \label{sec2.3}
% %\wqg{这里你应该先说}

% A key challenge in achieving high-performance sample-based GNN training is orchestrating tasks on GPU-CPU heterogeneous platforms. Existing sample-based GNN systems \cite{PAGRAPH_SOCC_2020,lazygcn_2020_neuips,Marius++_arxiv_2022,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,PYTDIRECT_VLDB_2021,TurboGNN_TOC_2023,DGL_CORR_2019,wholegraph_SC_2022} typically separate the computation based on the operations, i.e., \textbf{sampling}, \textbf{data loading}, and \textbf{training}, and assign each operation exclusively to the GPU or the CPU. 
% %different computation orchestrating method based on their hardware configuration to achieve high performance.
% In Figure \ref{fig:system_classification}, we classify these strategies and list representative systems for each category. Among the eight possible methods, CPU-based training is not utilized if a GPU is available on the platform.
% Step-based computation orchestrating methods are commonly employed in CPU-GPU heterogeneous platforms. However, they often exhibit suboptimal performance due to inefficient CPU processing or GPU resource contention. Therefore, we conducted an experimental analysis to identify the issues associated with step-based computation orchestration methods.
% Since training is always executed on GPUs in existing frameworks, we consider only sampling and data loading in the analysis. Further information regarding the utilized datasets, test platforms, and system configurations can be found in Section \ref{evaluation}. 

% %All of computation orchestrating strategies requires dividing the three steps into two devices, so there must be one device responsible for two or three steps. This step-based computation orchestrating strategies inevitably leave one side of heterogeneous system in a state of resource contention. Therefore, we experimentally analyze the challenges that may arise in each of the four strategies. The details of the used dataset, test platform, and system configurations are given in Section \ref{evaluation}.

% %Therefore, we will analyze the challenges caused by resource contention in the existing step-based computation orchestrating strategies.
% %Therefore, we classify systems which the CPU is responsible for two steps as CPU-centric frameworks \cite{lazygcn_2020_neuips,mari}, and conversely, systems which the GPU is responsible for two or three steps as GPU-centric frameworks \cite{PAGRAPH_SOCC_2020,DGL_CORR_2019,DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,wholegraph_SC_2022,TurboGNN_TOC_2023}.
% %\wqg{Please rewrite this part according to Figure 1, according to our previous discussion.}


% %As shown in Figure \ref{fig:CPU-GPU} (a), in the CPU-centric framework \cite{lazygcn_2020_neuips, Marius++_arxiv_2022}, the CPU will be responsible for the sampling and data loading step. Given a mini-batch size, CPUs repeatedly sample the input graph and collect the scattered features into a contiguous memory for the data loading step. 
% %In addition, for those systems \cite{PAGRAPH_SOCC_2020, DGL_CORR_2019} which the CPU is only responsible for the sampling or the data loading, we also group them as CPU-centric frameworks because the steps in the CPU are usually the bottleneck of the overall performance \cite{DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,Nextdoor_eurosys_2021}.
% %On the other hand, some other systems adopt a GPU-centric framework \cite{DATATIRING_VLDB_2022, GNNlab_EUROSYS_2022, TurboGNN_TOC_2023, wholegraph_SC_2022}. Through a series of GPU-based optimizations, as shown in Figure \ref{fig:CPU-GPU} (b), the existing systems can not only complete the sampling and training step in GPU but also use GPU feature caching policy to accelerate the data loading step. When the CPU or GPU is responsible for two to three steps in the sample-based GNN training, existing computation orchestrating strategies inevitably leave one side of heterogeneous system in a state of resource contention. Therefore, we will analyze the challenges caused by resource contention in the CPU-centric and GPU-centric frameworks.

% %In these systems, the CPU only keeps a portion of the infrequently accessed vertex feature and doing less feature collection work for them when GPU cache miss. 

% %DGL-UVA 的 data loading 算不算CPU 负责
% \Paragraph{Case 1: Placing sampling and data loading operation on CPUs suffers from inefficient CPU processing.} In this case, inefficient sampling and data loading on the CPU block the training pipeline, making data preparation the main bottleneck for overall performance.
% %the GPU executes only the training stage. The CPU executes the sampling step and stores the training data.%leading to high CPU contention and GPU wait. 
% % However, with the increase of input data size—large-scale graphs and high-dimensional features, sampling the graph on CPUs and transferring features to GPU memory become two main performance bottlenecks. Table \ref{tab:breakdown-GPU_Sampling} reports the runtime breakdown of representative GNN systems for training a 2-layer GCN on Orkut. After enabling GPU-based training, the Sampling and Data loading stages dominate the end-to-end GNN training time, accounting for 24\% and 29\%, respectively, on DGL. Consequently, this motivates recent research efforts to further improve GNN training in these two bottlenecks.
% %\textbf{CPU-based sampling.}
% %In practice, sampling from a large graph requires many random accesses to the memory, which inherently could perform better in GPUs, which are designed for regular parallel accesses. 
% %A heterogeneous system comprising CPUs and GPUs allows some tradeoffs between these two – CPUs are more capable of performing random memory access compared to GPUs but do not have the high degree of parallelism offered by the latter.
% %\textbf{CPU-based data loading.}
% % There is a simple and practical way to transfer data between main memory and GPU, explicit copy. Early GNN systems \cite{DGL_CORR_2019,PAGRAPH_SOCC_2020,lazygcn_2020_neuips} use explicit copy due to its simplicity and full use of bandwidth. Explicit copy is specialized in transferring large blocks of data independently. By providing source and destination memory pointers along with the data size, explicit copy transfers data behind the scenes while keeping processing units available for other tasks. However, since explicit copy may only transfer contiguous space, it adds extra feature collection overhead. CPU will collect the scattered features into a contiguous memory for explicit copy.

% \begin{table}[t]
% 	\vspace{-0.03in}
%         \centering
%     \caption{The runtime breakdown (in seconds) of the data loading step on DGL with different datasets. GNN:A 2-layer GCN \cite{GCN_iclr_2017}.}	
%     \label{tab:feature collection}
% 	\centering
%  \small
% 	{\renewcommand{\arraystretch}{1.2}
%  %\renewcommand{\arraystretch}{1}
% 	%\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c c}
% 		\hline
% 		\multirow{1}*{\textbf{Dataset}}  & 
%         \multirow{1}*{\textbf{Feature Collection}}  &
%         \multirow{1}*{\textbf{Feature Transfer}}  &
%         \multirow{1}*{\textbf{Total}} \\
% 		\hline
%     	{Reddit} & {0.608} & {0.363} & {3.63}\\ 
%     	{Lj-large} & {13.46} & {9.34} & {31.57}\\  
% 	    {Orkut} & {9.94} & {8.11} & {22.93}\\      
% 	    {Wikipedia} & {34.03} & {25.13} & {80.93}\\     
%         {OGB-products} & {4.04} & {2.25} & {10.93} \\  
% 		\hline
	
% 		\hline
% 	\end{tabular}
% 	}
%  \vspace{-0.4cm}
% \end{table}

% %we often think of sampling as the main task in the CPU, while the data loading relies on PCIe. However, when we look further into the data loading step, we find that slow external interconnects are not the biggest bottleneck limiting the data loading performance. GNNlab \cite{GNNlab_EUROSYS_2022} and Pagraph \cite{PAGRAPH_SOCC_2020} as well as early DGL (DGL-UVA) \cite{DGL_CORR_2019} use explicit copy to transfer vertex features.

% Neighbor sampling traverses the graph to obtain a multi-hop sampled subgraph for each training vertex. It incurs massive random memory access, which the limited computational resources of the CPU struggle to accelerate. \cite{Nextdoor_eurosys_2021}. 
% On the other hand, employing CPUs to handle the data loading step, including storing vertex features in CPU memory and performing feature extraction and transfer, also proves to be inefficient \cite{PAGRAPH_SOCC_2020}. This inefficiency arises from two main factors.
% Firstly, the external PCIe interconnect introduces additional latency to the host-GPU feature transfer, resulting in slower performance compared to CPU and GPU memory access \cite{PYTDIRECT_VLDB_2021}. Secondly, when loading the features of outermost neighbors, collecting and organizing the fragmented vertex data into contiguous memory (to leverage the cuda memory copy engine, \texttt{cudaMemcpy}) places a significant burden on CPU resources due to the combination of random memory access and sequential feature writing.
% We execute a 2-layer GCN on all five real-world graphs and analyze the time distribution for the data loading step.
% As shown in Table \ref{tab:feature collection}, the data loading step occupies 70.8\% of the total runtime. Specifically, feature collection and feature transfer account for about 41.6\% and 29.2\%, respectively, both of which have a considerable impact on the overall data loading overhead.

% On the other hand, the long sampling and data loading time also cause low GPU utilization. The training step in the GPU needs to wait for the slow CPU computation. Figure \ref{fig:utilization} presents the average GPU utilization of the GCN algorithm trained on the Reddit dataset. We can observe that merely around 25\% of the GPU computational resources are utilized when the CPU executes both sampling and data loading steps. Even if only separate sampling or data loading is performed on the CPU, the overall GPU utilization is still less than 50\%.


% %To fully investigate the hardware parallelism in the GPU, the mini-batch data in GPU must be more sufficient. 

% %to leverage explicit memory copy engine(cudaMemcpy)

% % \wqg{%完全不知道你要说啥，感觉像喝完酒蹲在饭桌上写的。每一句和下一句都不挨着。删掉吧
% % Given a mini-batch size, CPUs repeatedly sample the input graph and  usually takes a significant portion of total runtime. The neighbor sampling of each vertex is independently on each other, but the limited computational resources of CPU cannot sufficiently accelerate the sampling step in parallel \cite{Nextdoor_eurosys_2021}.
% % On the other hand, when we look further into the data loading step, we find that slow external interconnects are not the biggest bottleneck limiting the data loading performance. With the CPU-based data loading method, fragmented vertex features in CPU memory must be collected into a contiguous buffer before transferring them to GPU through explicit memory copy engine (cudaMemcpy). In real-world graphs, each vertex can be connected to thousands of neighbors. To collect relational information from those neighboring nodes, we may need to access thousands of scattered locations in memory.}




% %Through a series of GPU-based optimizations, as shown in Figure \ref{fig:CPU-GPU} (b), the existing systems can not only complete the sampling and training step in GPU but also use GPU-based caching policy to accelerate the data loading step. 
% %GPU-centric framework contains three different computation orchestrating strategies. 

% %Graph sampling takes a substantial portion of the end-to-end GNN training time. Thus, leveraging GPUs to accelerate graph sampling has appeared in academic and open-sourced projects like NextDoor \cite{Nextdoor_eurosys_2021} and DGL \cite{DGL_CORR_2019}. Figure \ref{fig:CPU-GPU} (b) shows that graph topological data is first loaded into GPU memory and then sampled on the GPU for each mini-batch. For the data loading step, prior works proposed to selectively cache the features associated with frequently sampled vertices in GPU memory. Many feature caching strategies \cite{PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022} are adopted to avoid the overhead of dynamic data tracking and swapping. 

% %two types of GPU-based methods are proposed. To eliminate the feature collection process in the CPU, Seung Won Min et al \cite{PYTDIRECT_VLDB_2021} introduce a GPU-based feature transfer method with zero copy. This approach allows the application developers to direct the GPU cores to the locations holding the data needed for computation. On the other hand, 


% %As shown in Table \ref{tab:breakdown-SDT}, in the GPU-centric framework, the GPU-led optimizations bring considerable acceleration to each step. In addition, the pipeline is equally essential because that can make full use of different resources in heterogeneous systems. However, when all optimizations are simply used simultaneously, it causes GPU contention and CPU idle. As shown in Table \ref{tab:breakdown-SDT}, when all optimizations are applied together, each step of SDT does not get the desired acceleration. Optimal end-to-end performance comes from pipeline optimization, which makes better use of heterogeneous resources. 
% %Therefore, in order to analyze the impact of GPU contention on performance, we will analyze the problem in the GPU-centric framework in terms of computational and storage resources.

% % Sampling is an irregular computation that is typically performed using the CPU, whereas training is performed on the GPU. As pointed out by past works \cite{Nextdoor_eurosys_2021,lazygcn_2020_neuips}, accelerating graph sampling is important to improve the end-to-end training time. As shown in Table \ref{tab:breakdown-SDT}, we found that graph sampling can take up to 24\% of an epoch’s time in baseline of DGL \cite{DGL_CORR_2019}. After adding caching and zero copy optimization, the time occupation of graph sampling is increased to 45\%. However GPU sampling's capacity to decrease end-to-end time is not optimal.
% %\textbf{Contention 1: Computational Resources.} 
% %There may be contention for computational resources when the GPU accelerates sampling and training simultaneously. 

% %Figure \ref{fig:pipeline} shows three examples of multi-stream pipeline in different configurations. First, When there is only one stream, the current batch needs to wait for the previous batch to finish before it can start execution, which can cause a waste of computational resources. 
% \Paragraph{Case 2: placing sampling on the GPU suffers from GPU resource contention.} In this case, the slow sampling can be accelerated through the massively parallel processing capability of GPUs. However, in this case, the two operations will compete for the limited GPU computation resources, leading to suboptimal overall performance.

% Sampling-based GNN training typically employs a pipeline design that overlaps the sampling, data loading, and training steps to achieve high performance in heterogeneous systems \cite{DGL_CORR_2019,PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,Marius++_arxiv_2022}. Figure \ref{fig:pipeline} depicts an ideal pipeline implementation where the operations of three batches (numbered 1, 2, and 3) are fully overlapped. However, this optimization requires different steps to be executed on different resources. When placing sampling on the GPU, it competes for computation resources with the training process, leading to inefficient pipeline processing, as shown in Figure \ref{fig:pipeline}. To illustrate this, we evaluate the performance of DGL with pipelined optimization and non-pipelined under both CPU-based sampling and GPU-based sampling. We run a 2-layer GCN on the Reddit graph and report the results in Table \ref{tab:breakdown-GPU_Sampling}. GPU-based sampling with pipeline optimization reduces the runtime of its non-pipelined version by 34.0\%, which is a lower improvement than the effect of pipeline optimization on CPU-based sampling. Furthermore, with pipeline optimization, GPU-based sampling shows inferior overall performance compared to CPU-based sampling.

% % However, when GPU sampling adds pipeline optimization, the overall performance improvement is only 1.43x. This is because sampling and training in different GPU stream are competing for GPU resources, and makes the execution time of sampling and training step longer (1.18 vs 0.33 and 2.03 vs 1.74). 

% % \wqg{Therefo re, we study the performance of DGL \cite{DGL_CORR_2019} with CPU sampling or GPU sampling under normal vision and pipeline vision. As observed in Table \ref{tab:breakdown-GPU_Sampling}, with CPU sampling, the pipeline optimization improves the end-to-end time by 1.89x with no change in the time of each step.}
 

% %In the baseline, the processing engines of sample-based GNN training require different resources, including CPU for sampling kernel, GPU for the training kernel, and PCIe for the host-GPU data loading. To overlap the resource utilization and improve the parallelism, Existing works use multiple CUDA streams to process the tasks concurrently. During the iterative processing, the task scheduler monitors the available streams and assigns them to tasks that have not been scheduled. The operating system will automatically overlap data transfer and kernel computation of different streams. 

% \begin{figure}[t]
%   \centering
%   \includegraphics[width=8cm,page={1}]{figures/pipeline.pdf}
%   \vspace{-0.4cm} %调整图片与上文的垂直距离
%   \caption{Two examples of multi-stream pipeline in different situations. }
%   \label{fig:pipeline}
%    \vspace{-0.4cm}
% \end{figure}




% \begin{table}[t]
% 	\vspace{-0.03in}
%     \setlength{\tabcolsep}{1mm}{
%     \centering
%     \caption{The runtime breakdown (in seconds) of a training epoch on DGL with CPU sampling or GPU sampling. GNN: A 2-Layer GCN \cite{GCN_iclr_2017}. Dataset: Reddit.}	
%     \label{tab:breakdown-GPU_Sampling}
% 	\centering
%  \small
% 	{\renewcommand{\arraystretch}{1.2}
%  %\renewcommand{\arraystretch}{1}
% 	%\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c c c c c c c}
% 		\hline
  
%         \hline
% 		\multirow{1}*{\textbf{Configuration}}  &
%         %\multirow{1}*{\textbf{GPU Utilization}} &
%         \multirow{1}*{\textbf{S}} &
% 		\multirow{1}*{\textbf{D}}& {\textbf{T}} &{\textbf{Total}}&{\textbf{+pipeline}}\\
% 		\hline
%     	{CPU sampling}&{0.84}&{1.07} &{1.72} & {3.63}& {1.92} (-47.8\%)&\\ 
%     	%{CPU sampling + pipeline}  & {0.83} &{1.03} &{1.82} & {1.92} & 1.76 (47.8\%)\\  
% 	    {GPU sampling}& {0.33} &{0.93} &{1.74}& {3.08}& {2.03} (-34.0\%)\\      
% 	    %{GPU sampling + pipeline}  & {1.18} &{1.13} &{2.03} &{2.15} &\\     
% 		\hline
	
% 		\hline
% 	\end{tabular}}
% 	}
%  \vspace{-0.4cm}
% \end{table}

% % \begin{figure*}[t]
	
% % 	\begin{minipage}{0.32\linewidth}
% % 		\vspace{3pt}
% %         %这个图片路径替换成你的图片路径即可使用
% % 		\centerline{\includegraphics[width=\textwidth]{figures/batchacc.pdf}}
% %           % 加入对这列的图片说明

% % 	\end{minipage}
% % 	\begin{minipage}{0.32\linewidth}
% % 		\vspace{3pt}
% % 		\centerline{\includegraphics[width=\textwidth]{figures/batch-RM.pdf}}

% % 	\end{minipage}
% % 	\begin{minipage}{0.32\linewidth}
% % 		\vspace{3pt}
% % 		\centerline{\includegraphics[width=\textwidth]{figures/cache-RM.pdf}}

% % 	\end{minipage}
 
% % 	\caption{(a) Accuracy comparison between different batch sizes of GCN on the Reddit dataset. (b) The Runtime of one epoch and Memory usage comparison between different batch sizes of GCN on Reddit dataset. (c) The runtime of one epoch and Memory usage comparison between different cache rates of GCN on the Reddit dataset.}
% % 	\label{fig3}
% %  \vspace{-0.4cm}
% % \end{figure*}


% \begin{figure*}
%   \centering
%   \includegraphics[width=\textwidth,page={1}]{figures/batchsize-cache.pdf}
%   \caption{(a) GPU utilization comparison between different batch sizes of GraphSAGE on the Wikipedia dataset. (b) The Runtime of one epoch and Memory usage comparison between different batch sizes of GraphSAGE on Wikipedia dataset. (c) The transfer volume of one epoch and Memory usage comparison between different cache ratios of GraphSAGE on the Wikipedia dataset.}
%    \vspace{-0.4cm}
%   \label{fig:batchsize-cache}
% \end{figure*}

% % \begin{figure}[t]
% %   \centering
% %   \includegraphics[width=8cm,page={1}]{figures/computer-batch.pdf}
% %   \vspace{-0.4cm} %调整图片与上文的垂直距离
% %   \caption{Calculation under different batch size}
% %   \label{fig:compute-batch}
% % \end{figure}



% %As shown in Figure \ref{fig:pipeline}, in the version of the GPU Sampling pipeline, the training kernel in stream 1 shares GPU resources with the sampling kernel in stream 3. The limited GPU resources cannot accelerate both the sampling and training steps at the same time. With sufficient GPU computing resources, kernels in different streams can be executed in parallel to achieve optimal performance. However, a simple pipeline design can cause different kernels to compete for resources and fail to fully utilize GPU resources. 
% %Table \ref{tab:breakdown-GPU_Sampling} shows the average utilization of GPU computing resources in different configurations. We can observe that in the CPU Sampling pipeline version, the utilization can already be as high as 78\% after three steps are fully pipelined. In the GPU sampling version, GPU utilization reaches 72\% and sampling step accelerated by 2.5x (0.83 vs 0.33). After further adding pipeline optimization to hide the data loading step, GPU utilization improves by just 3\% (81\% vs 78\%). 

% %Based on the above analysis, although GPU sampling greatly accelerates the sampling step, end-to-end performance is still sub-optimal due to GPU resource contention. As shown in Table \ref{tab:breakdown-GPU_Sampling}, the end-to-end performance of the CPU sampling pipeline version is 1.11x faster than the GPU sampling pipeline version. That means that we need to ensure that the GPU can efficiently accelerate the sampling step while avoiding GPU computational resource contention.

% %On memory limited GPUs.
% \Paragraph{Case 3: placing data loading on the GPU suffers from GPU memory contention.}
% GPU-based data loading leverages GPU memory to cache the vertex features, converting the heavy host-GPU data communication to GPU memory access as much as possible \cite{PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,TurboGNN_TOC_2023}. However, the performance of GPU-based data loading is influenced by the size of the configured memory buffer. In actual training, a substantial amount of global memory must be allocated for storing the training data and intermediate result, leaving only a small portion of memory available for data caching. If using a large cache buffer, the memory allocated for training has to be reduced, resulting in the training with a small batch size. Unfortunately, this trade-off may lead to inferior performance as the computational power of GPUs cannot be fully utilized. \cite{gnngaps_ppopp_2021,graphite_isca_2022}. 
% To illustrate the impact of batch sizes and cache ratios on performance, we conduct experiments using the GraphSAGE model on the Wikipedia dataset.
% %去掉准确率实验，换为GPU利用率实验，准确率实验有争议，并不一定batchsize大就好，用wiki数据集
% Figure \ref{fig:batchsize-cache} (a) and (b) display the results, demonstrating that training with a larger batch size is advantageous for GNNs in terms of achieving better GPU utilization. Although this consumes more GPU memory, it results in faster execution time for running an epoch.

% %However, it is not viable to store all vertex features because to the limited GPU memory. In addition, as analyzed above, batch data also needs enough memory to accelerate the tracing step. The caching policy's ability to accelerate the data loading step depends on the cache rate
% %As shown in Figure \ref{fig3} (c), the memory consumption of cache data depends on the ratio of cached vertices to all vertices in graph data. 
% Figure \ref{fig:batchsize-cache} (c) illustrates that a larger cache ratio contributes to a reduction in feature transfer volume during data loading. However, when using the GraphSAGE model with a batch size of 10000, we can only achieve a maximum cache ratio of less than 0.05, leading to numerous cache misses. The maximum cache ratio is influenced by three factors: available GPU memory, total number of vertices, and feature dimensions. Therefore, the performance of cache policy will get worse when the graph topological data and feature dimensions increase, aligning with trends observed in GNN workloads \cite{p3_osdi_2021,Graphsage_2017}. Due to GPU memory contention, the GPU cannot guarantee efficient acceleration of the data loading and training steps simultaneously. 
% %Hence, we needs to design a method that can reduce the total transfer volume without using much GPU memory.
% %Moreover, the hit rate decreases as the batch size increases, because the available GPU memory for cache data will be reduced.
% %Hence, feature caching needs to be optimized by a method that can reduce the total amount of transfers without using much GPU memory.



% %\textbf{Analysis of case 3: CPU executes sampling step and GPU executes the data loading step.} In this case, the GPU executes the training step and caches features to speed up the data loading step. GPU feature caching policy is widely used in existing works \cite{PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,TurboGNN_TOC_2023}. While caching policy speed up the data loading step, it also competes with the batch data for the GPU memory. More importantly, both data loading and training steps require sufficient GPU memory for cache data and batch data to execute effectively. 

% %In the basic sample-based GNN training, just the batch data—which includes the sample graph, feature, and aggregation results—is maintained in the GPU memory. But now, GPU caching policy needs to cache the frequently accessed vertices features to reduce the transfer volume.
% %Unfortunately, the memory of a single GPU (e.g., 16 GB) cannot simultaneously accommodate all the training data, especially for large-scale graphs with high-dimensional features. 
% %Thus, although GPU memory can usually keep all graph topological data, it will greatly limit the available memory capacity for batch data and caching data. %When further making the GPU responsible for the sampling step, the GPU memory needs to additionally hold the graph topology data, which can make the situation worse.

% %Due to the connectivity of the graph, vertices have many common neighbors with each other. Therefore, more training vertices in a batch also mean more common neighbors, which may repeatedly sampled and computed when training vertices are scattered in different batches. Figure \ref{fig:compute-batch} shows two training examples under different batch sizes. When training vertices are scattered in different batches, vertex $3$ and vertex $7$ are sampled and computed repeatedly. In addition, expanding the batch size can reduce this redundant computation.

% %更大的batch size对计算的好处有两方面，第一可以充分的利用GPU的并行能力，第二对于共同邻居不会重复采样和计算。
% % The memory consumption of batch data mainly depends on the batch size. When the batch size increased, more memory is used since there is a larger scale sample graph with more training vertices. The tiny batch size is not suitable for fully utilizing the computational power of the GPU \cite{gnngaps_ppopp_2021,graphite_isca_2022}. After further experiments, we found that a larger batch size can greatly reduce the training time. As shown in Figure \ref{fig3} (b), training with a larger batch size has higher memory consumption and faster average execution time for an epoch. However, less training time does not mean poor model accuracy \cite{ROC_mlsys_2020,graphite_isca_2022}. Figure \ref{fig3} (a) shows the GCN accuracy of test data under different batch sizes (from 512 to 153451). We can observe that larger batch size is more advantageous for the GNN model to converge to better accuracy. Hence, in order to improve training accuracy and reduce training time, sample-based training needs to provide adequate memory space for batch data. 

% % %However, it is not viable to store all vertex features because to the limited GPU memory. In addition, as analyzed above, batch data also needs enough memory to accelerate the tracing step. The caching policy's ability to accelerate the data loading step depends on the cache rate
% % On the other hand, the memory consumption of cache data depends on the cache rate. The cache rate is the ratio of cached vertices to all vertices in graph data. The larger the cache rate, the better it is for reducing the feature transfer volume during the data loading, which is positively correlated as shown in Figure \ref{fig3}(c). The maximum cache rate that can be given is influenced by three factors: available GPU memory, total number of vertices, and feature dimensions. Therefore, the cache hit rate will get worse when the graph topological data and feature dimensions increase—this aligns well with the trends in GNN workloads \cite{p3_osdi_2021, Graphsage_2017}. Moreover, the hit rate decreases as the batch size increases, because the available GPU memory for cache data will be reduced.
% %Hence, feature caching needs to be optimized by a method that can reduce the total amount of transfers without using much GPU memory.

% %Based on the above analysis, when using GPUs to accelerate both the training and data loading steps, there will be inevitable contention for GPU storage resources. Using a tiny batch size affects model quality and training efficiency. Hence, we need a new method that can reduce the total transfer volume without using much GPU memory.

% % \begin{table}[t]
% % 	\vspace{-0.03in}
% %     \centering
    
% %     \caption{The runtime breakdown (in seconds) of a training epoch on DGL with different optimization and all of them. We run a 3-Layer GraphSAGE model \cite{GCN_iclr_2017} on the Wikipedia graph}	
% %     \label{tab:breakdown-SDT}
   
% % 	\centering
% %     \small
% % 	{\renewcommand{\arraystretch}{1.2}
% %  %\renewcommand{\arraystretch}{1}
% % 	%\resizebox{0.97\linewidth}{!}{
% % 	\begin{tabular}{l c c c c c c}
% % 		\hline
		
% % 		\hline
% % 		\multirow{1}*{\textbf{Configuration}}  &
% %         \multirow{1}*{\textbf{S}} &
% % 		\multirow{1}*{\textbf{D}}& {\textbf{T}} &{Total} &{\textbf{+pipeline}} &{\textbf{all}}\\
% % 		\hline
% %     	{Baseline}& {0.62} &{6.39} & {3.92} &{10.93} &{6.63} (-47.8\%)\\  
% %         {Feature Cache} & {0.63} &{2.32} & {3.91}&{4.25} &{4.25} (-47.8\%)\\ 
% %         {GPU Sampling} & {0.33} &{6.33} &{3.94}& {3.08}&{6.62}  (-47.8\%)\\  
% %         {Both} & {0.33} &{4.73} &{3.92}& {3.08} &{}  (-47.8\%)\\  
% % 		\hline
	
% % 		\hline
% % 	\end{tabular}}
% %  \vspace{-0.4cm}
% % \end{table}

% \Paragraph{Case 4: placing sampling and data loading on the GPU suffers from GPU memory and resource contention} When all steps of sample-based GNN training are executed on the GPU, it results in GPU contention and CPU idle. The reasons for this situation have been discussed in cases 2 and 3. Firstly, the computation kernels of sampling and training compete for the limited GPU cores, leading to a slowdown in both steps. Secondly, the batch data for training and cache data for data loading contend with the limited GPU memory. Furthermore, when further making the GPU responsible for the sampling step, the GPU needs to additionally hold the graph topology data, which can make the GPU memory contention worse.  
% %As shown in Table \ref{tab:breakdown-SDT}, the GPU-led optimizations bring considerable acceleration to each step, and the pipeline is equally essential optimization because that can make full use of different resources in heterogeneous systems. Due to the GPU contention, when all optimizations are applied together, each step does not get the desired acceleration. Through a series of GPU-based optimizations, the existing systems can not only complete the sampling and training step in GPU but also use GPU-based caching policy to accelerate the data loading step. 

% %Optimal end-to-end performance comes from pipeline optimization, which makes better use of heterogeneous resources. 


% \vspace{-0.05in}
% \subsection{Summary}
% We conducted experimental analysis on different computation orchestrating strategies in GPU-CPU heterogeneous systems that follow step-based task partitioning. 
% %这里应该概要地总结一下他们有什么问题
% Our observations reveal that step-based task partitioning leads to an imbalanced allocation of computational and memory resources. Assigning two or more steps to the GPU may result in memory or GPU resource contention. On the other hand, assigning one or two steps to the CPU may cause inefficient CPU processing becomes a bottleneck. 
% %inferior performance
% %which inevitably leads to resource contention at one end of the heterogeneous resource. After analysis and experiments, we summarize the major problems of the step-based computation orchestrating strategies into two types: (1) GPU resource contention. (2) Inefficient CPU processing. 
% %Most applications developed for heterogeneous systems are more suitable for the GPU than the CPU. Only considering the device's suitability when scheduling the applications overloads the GPU and leaves the CPU idle. This results in a longer execution time. On the other hand, overusing the CPU and making the GPU wait only makes the situation worse. 
% To achieve optimal performance, a well-designed CPU-GPU heterogeneous framework should ensure balanced utilization of both the CPU and GPU. However, the step-based computation orchestrating method fails to achieve this balance. This motivates us to design a resource-balanced computation orchestrating method.
% %The efficiency of a heterogeneous system requires considering both of the issues we mentioned above. 


% \vspace{-0.05in}
% \section{Analysis: a motivating study}
% \label{sec3}
\vspace{-0.05in}
\section{Existing systems and Their Limitations}
% \zyf{see comments}
% % frameworks是不是不太好？太大了？
\label{sec3}
%\wqg{这里你应该先说}

A key challenge in achieving high-performance sample-based GNN training is orchestrating tasks on the GPU-CPU heterogeneous environments.
% In Figure \ref{fig:system_classification}, we classify task orchestrating methods and list representative systems for each method. Among the eight possible methods, CPU-based training is not utilized if a GPU is available on the platform.
Existing sample-based GNN systems \cite{PAGRAPH_SOCC_2020,lazygcn_2020_neuips,Marius++_arxiv_2022,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,PYTDIRECT_VLDB_2021,TurboGNN_TOC_2023,DGL_CORR_2019,wholegraph_SC_2022} typically separate the training process based on the operations, i.e., \textbf{sample}, \textbf{gather}, and \textbf{train}, and assign each operation to the GPU or the CPU. 
However, they often exhibit suboptimal performance due to inefficient CPU processing or GPU resource contention. To illustrate this, we conduct a series of experiments to analyze the problems of existing task orchestration methods.
%different computation orchestrating method based on their hardware configuration to achieve high performance. 


%All of computation orchestrating strategies requires dividing the three steps into two devices, so there must be one device responsible for two or three steps. This step-based computation orchestrating strategies inevitably leave one side of heterogeneous system in a state of resource contention. Therefore, we experimentally analyze the challenges that may arise in each of the four strategies. The details of the used dataset, test platform, and system configurations are given in Section \ref{evaluation}.

%Therefore, we will analyze the challenges caused by resource contention in the existing step-based computation orchestrating strategies.
%Therefore, we classify systems which the CPU is responsible for two steps as CPU-centric frameworks \cite{lazygcn_2020_neuips,mari}, and conversely, systems which the GPU is responsible for two or three steps as GPU-centric frameworks \cite{PAGRAPH_SOCC_2020,DGL_CORR_2019,DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,wholegraph_SC_2022,TurboGNN_TOC_2023}.
%\wqg{Please rewrite this part according to Figure 1, according to our previous discussion.}


%As shown in Figure \ref{fig:CPU-GPU} (a), in the CPU-centric framework \cite{lazygcn_2020_neuips, Marius++_arxiv_2022}, the CPU will be responsible for the sampling and data loading step. Given a mini-batch size, CPUs repeatedly sample the input graph and collect the scattered features into a contiguous memory for the data loading step. 
%In addition, for those systems \cite{PAGRAPH_SOCC_2020, DGL_CORR_2019} which the CPU is only responsible for the sampling or the data loading, we also group them as CPU-centric frameworks because the steps in the CPU are usually the bottleneck of the overall performance \cite{DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,Nextdoor_eurosys_2021}.
%On the other hand, some other systems adopt a GPU-centric framework \cite{DATATIRING_VLDB_2022, GNNlab_EUROSYS_2022, TurboGNN_TOC_2023, wholegraph_SC_2022}. Through a series of GPU-based optimizations, as shown in Figure \ref{fig:CPU-GPU} (b), the existing systems can not only complete the sampling and training step in GPU but also use GPU feature caching policy to accelerate the data loading step. When the CPU or GPU is responsible for two to three steps in the sample-based GNN training, existing computation orchestrating strategies inevitably leave one side of heterogeneous system in a state of resource contention. Therefore, we will analyze the challenges caused by resource contention in the CPU-centric and GPU-centric frameworks.

%In these systems, the CPU only keeps a portion of the infrequently accessed vertex feature and doing less feature collection work for them when GPU cache miss. 

%DGL-UVA 的 data loading 算不算CPU 负责

\begin{figure*}
% \vspace{-0.3in}
  \centering
  \includegraphics[width=0.95\textwidth,page={1}]{figures/task_method.pdf}
  \vspace{-0.4cm}
  \caption{Illustration of the four task orchestrating methods and \system.
  (a) Case 1 \cite{lazygcn_2020_neuips,Marius++_arxiv_2022}: Placing sampling and gathering on CPUs. (b) Case 2 \cite{DGL_CORR_2019}: Placing sampling on the GPUs. (c) Case 3 \cite{PAGRAPH_SOCC_2020,refresh_arxiv_2023}: Placing gathering on the GPUs. (d) Case 4 \cite{DATATIRING_VLDB_2022,GNNlab_EUROSYS_2022,wholegraph_SC_2022,TurboGNN_TOC_2023,quiver_2023_arxiv,DSP_PPOPP_2023,ducati_sigmod_2023}: Placing sampling and gathering on the GPUs. (e) Hotness-aware layer-based task orchestrating method of \system. The width of the arrow is positively proportional to the transfer data volume. The volumes of the circles S, G, and T are proportional to the task amount.} 
  % \ccy{(Question: 1. Why is the feature represented by a vertical line instead of being represented by a rectangle consistent with Figure 3?  2. Blue arrows indicate transfer, written as "Feature". In addition, do you need to explain that the width indicates the transfer volume?)}
   \vspace{-0.2cm}
  \label{fig:task-method}
\end{figure*}

\vspace{-0.05in}
\subsection{Existing Task Orchestrating Methods}

In Figure \ref{fig:system_classification}, we classify task orchestrating methods and list representative systems. Among eight possible methods, CPU-based training is not utilized if a GPU is available on the platform. Therefore, we only 
consider placing sampling and gathering on different devices in the analysis. Further information regarding the used graphs, test platforms, and system configurations can be found in Section \ref{sec5}. 


\Paragraph{Case 1: Placing sampling and gathering on CPUs suffers from inefficient CPU processing} We show an example of this case in Figure \ref{fig:task-method} (a). In this case, inefficient sampling and feature gathering on the CPU block the training pipeline, making data preparation the main bottleneck for overall performance.
%the GPU executes only the training stage. The CPU executes the sampling step and stores the training data.%leading to high CPU contention and GPU wait. 
% However, with the increase of input data size—large-scale graphs and high-dimensional features, sampling the graph on CPUs and transferring features to GPU memory become two main performance bottlenecks. Table \ref{tab:breakdown-GPU_Sampling} reports the runtime breakdown of representative GNN systems for training a 2-layer GCN on Orkut. After enabling GPU-based training, the Sampling and Data loading stages dominate the end-to-end GNN training time, accounting for 24\% and 29\%, respectively, on DGL. Consequently, this motivates recent research efforts to further improve GNN training in these two bottlenecks.
%\textbf{CPU-based sampling.}
%In practice, sampling from a large graph requires many random accesses to the memory, which inherently could perform better in GPUs, which are designed for regular parallel accesses. 
%A heterogeneous system comprising CPUs and GPUs allows some tradeoffs between these two – CPUs are more capable of performing random memory access compared to GPUs but do not have the high degree of parallelism offered by the latter.
%\textbf{CPU-based data loading.}
% There is a simple and practical way to transfer data between main memory and GPU, explicit copy. Early GNN systems \cite{DGL_CORR_2019,PAGRAPH_SOCC_2020,lazygcn_2020_neuips} use explicit copy due to its simplicity and full use of bandwidth. Explicit copy is specialized in transferring large blocks of data independently. By providing source and destination memory pointers along with the data size, explicit copy transfers data behind the scenes while keeping processing units available for other tasks. However, since explicit copy may only transfer contiguous space, it adds extra feature collection overhead. CPU will collect the scattered features into a contiguous memory for explicit copy.

\begin{table}[t]
	\vspace{-0.03in}
        \centering
    \caption{The runtime breakdown (in seconds) of sample and gather steps on DGL with different datasets. The FC and FT represent the feature collection and feature transfer of the gather step. GNN: A 3-layer GCN \cite{GCN_iclr_2017}.}
    \vspace{-0.1in}
    \label{tab:feature collection}
    \footnotesize
	\centering
	{\renewcommand{\arraystretch}{1.2}
 %\renewcommand{\arraystretch}{1}
	%\resizebox{0.97\linewidth}{!}{
	\begin{tabular}{l c c c c}
		\hline
		\multirow{1}*{\textbf{Dataset}}  & 
        \multirow{1}*{\textbf{Sample}}  & 
        \multirow{1}*{\textbf{Gather (FC)}}  &
        \multirow{1}*{\textbf{Gather (FT)}}  &
        \multirow{1}*{\textbf{Total}} \\
		\hline
    	% {Reddit}  & {0.84/23\%} & {0.608/17\%} & {0.372/10\%} & {3.63}\\ 
    	% {Lj-large} & {4.95/15\%} & {13.46/42\%} & {9.34/29\%} & {31.57}\\  
	    % {Orkut} & {2.43/10\%} & {9.94/43\%} & {8.11/35\%} & {22.93}\\      
	    % {Wikipedia} & {10.14/12\%} & {34.03/42\%} & {25.13/31\%} & {80.93}\\     
     %    {OGB} & {3.11/28\%} & {4.04/36\%} & {2.25/20\%} & {10.93} \\  
        {Reddit}  & {2.7/11\%} & {9.1/38\%} & {6.0/25\%} & {23.7}\\ 
    	{Lj-large} & {128.8/14\%} & {384.4/41\%} & {252.5/27\%} & {935.3}\\  
	    {Orkut} & {78.8/10\%} & {384.3/48\%} & {249.1/31\%} & {813.3}\\      
	    {Wikipedia} & {209.4/12\%} & {651.8/40\%} & {570.9/33\%} & {1669.1}\\     
        {Products} & {9.9/37\%} & {7.2/27\%} & {4.1/15\%} & {26.8} \\
        {Papers100M} & {11.5/32\%} & {8.6/24\%} & {6.4/18\%} & {36.84} \\  
		\hline
	
		\hline
	\end{tabular}
	}
 \vspace{-0.4cm}
\end{table}


%we often think of sampling as the main task in the CPU, while the data loading relies on PCIe. However, when we look further into the data loading step, we find that slow external interconnects are not the biggest bottleneck limiting the data loading performance. GNNlab \cite{GNNlab_EUROSYS_2022} and Pagraph \cite{PAGRAPH_SOCC_2020} as well as early DGL (DGL-UVA) \cite{DGL_CORR_2019} use explicit copy to transfer vertex features.

Neighbor sampling traverses the graph to obtain a multi-hop sampled subgraph for each training vertex. It incurs massive computation and irregular memory access, which makes the limited computational resources of the CPU hard to accelerate \cite{Nextdoor_eurosys_2021}. 
On the other hand, deploying the gathering step on the CPU, including storing vertex features in the host memory and performing feature collection and transfer, also proves to be inefficient \cite{PAGRAPH_SOCC_2020}. This inefficiency arises from two main factors.
Firstly, the slow external PCIe interconnect makes the host-GPU feature transfer much slower than GPU memory access \cite{PYTDIRECT_VLDB_2021}. Secondly, when loading the features of neighbors, collecting and organizing the fragmented vertex feature into contiguous memory before communication (to leverage the CUDA memory copy engine, \texttt{cudaMemcpy}) also consumes substantial CPU resources due to massive random memory accesses.
%\zyf{is it about CPU memory operation? why do you mention CUDA memory copy?}
%and sequential feature writing \zyf{sequential write of ... why sequential write is slow?}
We run this task orchestrating method with a 3-layer GCN on all six real-world graphs and analyze the time distribution for sampling and gathering steps.
As shown in Table \ref{tab:feature collection}, sampling and gathering steps occupy 19.3\% and 61.2\% of the total runtime, respectively. 
Specifically, the most significant overhead is the feature collection in the gathering step, which accounts for 36.3\% of the total runtime.

On the other hand, the long sampling and gathering time also causes low GPU utilization. The training step in the GPU needs to wait for the slow CPU tasks. % shows the average GPU utilization of the GCN trained on the Reddit dataset.
We can observe from Figure \ref{fig:utilization} that merely around 25\% of the GPU computational resources are utilized when the CPU executes both sampling and gathering steps. Even if only separate sampling or gathering is performed on the CPU, the overall GPU utilization is still less than 65\%. 
%\zyf{CPU utilization is low, why? is it because memory access is slow?}


%To fully investigate the hardware parallelism in the GPU, the mini-batch data in GPU must be more sufficient. 

%to leverage explicit memory copy engine(cudaMemcpy)

% \wqg{%完全不知道你要说啥，感觉像喝完酒蹲在饭桌上写的。每一句和下一句都不挨着。删掉吧
% Given a mini-batch size, CPUs repeatedly sample the input graph and  usually takes a significant portion of total runtime. The neighbor sampling of each vertex is independently on each other, but the limited computational resources of CPU cannot sufficiently accelerate the sampling step in parallel \cite{Nextdoor_eurosys_2021}.
% On the other hand, when we look further into the data loading step, we find that slow external interconnects are not the biggest bottleneck limiting the data loading performance. With the CPU-based data loading method, fragmented vertex features in CPU memory must be collected into a contiguous buffer before transferring them to GPU through explicit memory copy engine (cudaMemcpy). In real-world graphs, each vertex can be connected to thousands of neighbors. To collect relational information from those neighboring nodes, we may need to access thousands of scattered locations in memory.}




%Through a series of GPU-based optimizations, as shown in Figure \ref{fig:CPU-GPU} (b), the existing systems can not only complete the sampling and training step in GPU but also use GPU-based caching policy to accelerate the data loading step. 
%GPU-centric framework contains three different computation orchestrating strategies. 

%Graph sampling takes a substantial portion of the end-to-end GNN training time. Thus, leveraging GPUs to accelerate graph sampling has appeared in academic and open-sourced projects like NextDoor \cite{Nextdoor_eurosys_2021} and DGL \cite{DGL_CORR_2019}. Figure \ref{fig:CPU-GPU} (b) shows that graph topological data is first loaded into GPU memory and then sampled on the GPU for each mini-batch. For the data loading step, prior works proposed to selectively cache the features associated with frequently sampled vertices in GPU memory. Many feature caching strategies \cite{PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022} are adopted to avoid the overhead of dynamic data tracking and swapping. 

%two types of GPU-based methods are proposed. To eliminate the feature collection process in the CPU, Seung Won Min et al \cite{PYTDIRECT_VLDB_2021} introduce a GPU-based feature transfer method with zero copy. This approach allows the application developers to direct the GPU cores to the locations holding the data needed for computation. On the other hand, 


%As shown in Table \ref{tab:breakdown-SDT}, in the GPU-centric framework, the GPU-led optimizations bring considerable acceleration to each step. In addition, the pipeline is equally essential because that can make full use of different resources in heterogeneous systems. However, when all optimizations are simply used simultaneously, it causes GPU contention and CPU idle. As shown in Table \ref{tab:breakdown-SDT}, when all optimizations are applied together, each step of SDT does not get the desired acceleration. Optimal end-to-end performance comes from pipeline optimization, which makes better use of heterogeneous resources. 
%Therefore, in order to analyze the impact of GPU contention on performance, we will analyze the problem in the GPU-centric framework in terms of computational and storage resources.

% Sampling is an irregular computation that is typically performed using the CPU, whereas training is performed on the GPU. As pointed out by past works \cite{Nextdoor_eurosys_2021,lazygcn_2020_neuips}, accelerating graph sampling is important to improve the end-to-end training time. As shown in Table \ref{tab:breakdown-SDT}, we found that graph sampling can take up to 24\% of an epoch’s time in baseline of DGL \cite{DGL_CORR_2019}. After adding caching and zero copy optimization, the time occupation of graph sampling is increased to 45\%. However GPU sampling's capacity to decrease end-to-end time is not optimal.
%\textbf{Contention 1: Computational Resources.} 
%There may be contention for computational resources when the GPU accelerates sampling and training simultaneously. 

%Figure \ref{fig:pipeline} shows three examples of multi-stream pipeline in different configurations. First, When there is only one stream, the current batch needs to wait for the previous batch to finish before it can start execution, which can cause a waste of computational resources. 
\Paragraph{Case 2: Placing sampling on the GPU suffers from GPU resource contention} We show an example of this case in Figure \ref{fig:task-method} (b). In this case, the slow sampling can be accelerated through the massively parallel processing capability of GPUs. However, the sampling and training operations will compete for the GPU resources, leading to suboptimal overall performance.

Sample-based GNN training typically employs a pipeline design that overlaps the different steps to achieve high performance in heterogeneous systems \cite{DGL_CORR_2019,PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,Marius++_arxiv_2022}. Figure \ref{fig:pipeline} (a) depicts an ideal pipeline implementation where the operations of three batches (numbered 1, 2, and 3) are fully overlapped. However, this optimization requires different steps to be executed on independent resources. When placing sampling on the GPU, it competes for computation resources with the training step, leading to inefficient pipelining, as shown in Figure \ref{fig:pipeline} (b). To illustrate this, we evaluate the performance of DGL with pipelined optimization and non-pipelined under both CPU-based and GPU-based sampling. We run a 3-layer GCN on the Reddit graph and report the results in Table \ref{tab:breakdown-GPU_Sampling}. The GPU-based sampling with pipeline optimization reduces the runtime of its non-pipelined version by 43.1\%, which is a lower improvement than the effect of pipeline optimization on CPU-based sampling. Furthermore, with pipeline optimization, GPU-based sampling shows inferior overall performance compared to CPU-based sampling.

\begin{figure}[t]  
  \centering
  \includegraphics[width=7cm,page={1}]{figures/pipeline.pdf}
  \vspace{-0.4cm} %调整图片与上文的垂直距离
  \caption{Two examples of multi-stream pipeline design in the ideal and actual situations.}
  \label{fig:pipeline}
   \vspace{-0.4cm}
\end{figure}




\begin{table}[t]
%\vspace{-0.03in}
    \setlength{\tabcolsep}{1mm}{
    \centering
    \caption{The runtime breakdown (in seconds) of a training epoch on DGL with CPU-based or GPU-based sampling. S, G, and T represent sample, gather, and train steps. GNN: A 3-Layer GCN \cite{GCN_iclr_2017}. Batch size 10000.}	
    \vspace{-0.1in}
    \label{tab:breakdown-GPU_Sampling}
	\centering
 \footnotesize
	{\renewcommand{\arraystretch}{1.2}
 %\renewcommand{\arraystretch}{1}
	%\resizebox{0.97\linewidth}{!}{
	\begin{tabular}{l c c c c c c c c}
		\hline
  
        \hline
		\multirow{1}*{\textbf{Configuration}}  &
        %\multirow{1}*{\textbf{GPU Utilization}} &
        \multirow{1}*{\textbf{S}} &
		\multirow{1}*{\textbf{G}}& {\textbf{T}} &{\textbf{Total}}&{\textbf{+pipeline}}\\
		\hline
    	% {CPU-based sampling}&{0.84}&{1.07} &{1.72} & {3.63}& {1.92} (-47.8\%)&\\ 
	    % {GPU-based sampling}& {0.33} &{0.93} &{1.74}& {3.08}& {2.03} (-34.0\%)\\
        {CPU-based sampling}&{2.28}&{2.84} &{2.76} & {7.88}& {3.42} (-56.6\%)&\\ 
	    {GPU-based sampling}& {0.78} &{2.69} &{2.75}& {6.22}& {3.54} (-43.1\%)\\  
	    %{GPU sampling + pipeline}  & {1.18} &{1.13} &{2.03} &{2.15} &\\     
		\hline
	
		\hline
	\end{tabular}}
	}
 \vspace{-0.3cm}
\end{table}

\begin{figure}[t]  
  \centering
  \includegraphics[width=0.45\textwidth,page={1}]{figures/batchsize-cache.pdf}
    \vspace{-0.4cm}
   \caption{(a) GPU utilization with different batch sizes. (b) Per-epoch runtime and memory usage with different batch sizes. (c) Transfer volume and memory usage with different ratios of cached vertices (cache ratio).}
   \vspace{-0.4cm}
  \label{fig:batchsize-cache}
\end{figure}


% However, when GPU sampling adds pipeline optimization, the overall performance improvement is only 1.43x. This is because sampling and training in different GPU stream are competing for GPU resources, and makes the execution time of sampling and training step longer (1.18 vs 0.33 and 2.03 vs 1.74). 

% \wqg{Therefo re, we study the performance of DGL \cite{DGL_CORR_2019} with CPU sampling or GPU sampling under normal vision and pipeline vision. As observed in Table \ref{tab:breakdown-GPU_Sampling}, with CPU sampling, the pipeline optimization improves the end-to-end time by 1.89x with no change in the time of each step.}
 

%In the baseline, the processing engines of sample-based GNN training require different resources, including CPU for sampling kernel, GPU for the training kernel, and PCIe for the host-GPU data loading. To overlap the resource utilization and improve the parallelism, Existing works use multiple CUDA streams to process the tasks concurrently. During the iterative processing, the task scheduler monitors the available streams and assigns them to tasks that have not been scheduled. The operating system will automatically overlap data transfer and kernel computation of different streams. 

% \begin{figure*}[t]
	
% 	\begin{minipage}{0.32\linewidth}
% 		\vspace{3pt}
%         %这个图片路径替换成你的图片路径即可使用
% 		\centerline{\includegraphics[width=\textwidth]{figures/batchacc.pdf}}
%           % 加入对这列的图片说明

% 	\end{minipage}
% 	\begin{minipage}{0.32\linewidth}
% 		\vspace{3pt}
% 		\centerline{\includegraphics[width=\textwidth]{figures/batch-RM.pdf}}

% 	\end{minipage}
% 	\begin{minipage}{0.32\linewidth}
% 		\vspace{3pt}
% 		\centerline{\includegraphics[width=\textwidth]{figures/cache-RM.pdf}}

% 	\end{minipage}
 
% 	\caption{(a) Accuracy comparison between different batch sizes of GCN on the Reddit dataset. (b) The Runtime of one epoch and Memory usage comparison between different batch sizes of GCN on Reddit dataset. (c) The runtime of one epoch and Memory usage comparison between different cache rates of GCN on the Reddit dataset.}
% 	\label{fig3}
%  \vspace{-0.4cm}
% \end{figure*}



% \begin{figure}[t]
%   \centering
%   \includegraphics[width=8cm,page={1}]{figures/computer-batch.pdf}
%   \vspace{-0.4cm} %调整图片与上文的垂直距离
%   \caption{Calculation under different batch size}
%   \label{fig:compute-batch}
% \end{figure}



%As shown in Figure \ref{fig:pipeline}, in the version of the GPU Sampling pipeline, the training kernel in stream 1 shares GPU resources with the sampling kernel in stream 3. The limited GPU resources cannot accelerate both the sampling and training steps at the same time. With sufficient GPU computing resources, kernels in different streams can be executed in parallel to achieve optimal performance. However, a simple pipeline design can cause different kernels to compete for resources and fail to fully utilize GPU resources. 
%Table \ref{tab:breakdown-GPU_Sampling} shows the average utilization of GPU computing resources in different configurations. We can observe that in the CPU Sampling pipeline version, the utilization can already be as high as 78\% after three steps are fully pipelined. In the GPU sampling version, GPU utilization reaches 72\% and sampling step accelerated by 2.5x (0.83 vs 0.33). After further adding pipeline optimization to hide the data loading step, GPU utilization improves by just 3\% (81\% vs 78\%). 

%Based on the above analysis, although GPU sampling greatly accelerates the sampling step, end-to-end performance is still sub-optimal due to GPU resource contention. As shown in Table \ref{tab:breakdown-GPU_Sampling}, the end-to-end performance of the CPU sampling pipeline version is 1.11x faster than the GPU sampling pipeline version. That means that we need to ensure that the GPU can efficiently accelerate the sampling step while avoiding GPU computational resource contention.

%On memory limited GPUs.
\Paragraph{Case 3: Placing gathering on the GPU suffers from GPU memory contention} 
GPU-based gathering method leverages GPU memory to cache the vertex features, converting the slow host-GPU data communication into fast GPU memory access as much as possible \cite{PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,TurboGNN_TOC_2023,DSP_PPOPP_2023,ducati_sigmod_2023,legion_atc_2023,wholegraph_SC_2022}. However, the performance of GPU data caching is affected by the size of the available GPU memory. We show an example of this case in Figure \ref{fig:task-method} (c). In actual training, a substantial amount of global memory must be allocated for storing the training data and intermediate results, leaving only a small portion of memory available for feature caching. If using a large cache buffer, the memory allocated for training has to be reduced, resulting in the training with a small batch size. Unfortunately, this trade-off may lead to inferior performance since the computational power of GPUs cannot be fully utilized \cite{gnngaps_ppopp_2021,graphite_isca_2022}. 
To illustrate the impact of batch sizes and the ratio of cached vertices (denoted by cache ratio) on performance, we conduct experiments on DGL using the 3-layer GCN model on the Wikipedia graph. 
% \zyf{You always change GNN model and dataset, I suspect that you intend to gain the results you would like to see but this might not be true on other models/datasets.}
%去掉准确率实验，换为GPU利用率实验，准确率实验有争议，并不一定batchsize大就好，用wiki数据集
The results in Figure \ref{fig:batchsize-cache} (a) and (b) demonstrate that training with a larger batch size is beneficial for GNNs in terms of achieving better GPU utilization. Although this consumes more GPU memory, it results in faster execution.
The results in Figure \ref{fig:batchsize-cache} (c) reveal that a larger cache ratio results in a linear transfer reduction of features in the gathering step. 
However, when increasing the batch size from 128 to 4096, the ratio of cached vertices is decreased from 0.37 to 0.05 due to insufficient GPU memory, which results in numerous cache misses. Moreover, as the graph topology data and feature dimensions increase, the benefits brought by caching vertices will further diminish. 

%This aligns with the observed trends in GNN workloads \cite{p3_osdi_2021, Graphsage_2017}.

% \zyf{The results varying batch size can't directly verify your analysis on GPU memory contention.}
%zyf:这个感觉扯的有点远，你给我show batch size和cache ratio的实验，跟你说的GPU memory contention没有直接联系。或者绕了几个弯，所以可能会让reviewer confuse不理解你给我看这些结果干啥。
%Hence, we needs to design a method that can reduce the total transfer volume without using much GPU memory.
%Moreover, the hit rate decreases as the batch size increases, because the available GPU memory for cache data will be reduced.
%Hence, feature caching needs to be optimized by a method that can reduce the total amount of transfers without using much GPU memory.



%\textbf{Analysis of case 3: CPU executes sampling step and GPU executes the data loading step.} In this case, the GPU executes the training step and caches features to speed up the data loading step. GPU feature caching policy is widely used in existing works \cite{PAGRAPH_SOCC_2020,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,TurboGNN_TOC_2023}. While caching policy speed up the data loading step, it also competes with the batch data for the GPU memory. More importantly, both data loading and training steps require sufficient GPU memory for cache data and batch data to execute effectively. 

%In the basic sample-based GNN training, just the batch data—which includes the sample graph, feature, and aggregation results—is maintained in the GPU memory. But now, GPU caching policy needs to cache the frequently accessed vertices features to reduce the transfer volume.
%Unfortunately, the memory of a single GPU (e.g., 16 GB) cannot simultaneously accommodate all the training data, especially for large-scale graphs with high-dimensional features. 
%Thus, although GPU memory can usually keep all graph topological data, it will greatly limit the available memory capacity for batch data and caching data. %When further making the GPU responsible for the sampling step, the GPU memory needs to additionally hold the graph topology data, which can make the situation worse.

%Due to the connectivity of the graph, vertices have many common neighbors with each other. Therefore, more training vertices in a batch also mean more common neighbors, which may repeatedly sampled and computed when training vertices are scattered in different batches. Figure \ref{fig:compute-batch} shows two training examples under different batch sizes. When training vertices are scattered in different batches, vertex $3$ and vertex $7$ are sampled and computed repeatedly. In addition, expanding the batch size can reduce this redundant computation.

%更大的batch size对计算的好处有两方面，第一可以充分的利用GPU的并行能力，第二对于共同邻居不会重复采样和计算。
% The memory consumption of batch data mainly depends on the batch size. When the batch size increased, more memory is used since there is a larger scale sample graph with more training vertices. The tiny batch size is not suitable for fully utilizing the computational power of the GPU \cite{gnngaps_ppopp_2021,graphite_isca_2022}. After further experiments, we found that a larger batch size can greatly reduce the training time. As shown in Figure \ref{fig3} (b), training with a larger batch size has higher memory consumption and faster average execution time for an epoch. However, less training time does not mean poor model accuracy \cite{ROC_mlsys_2020,graphite_isca_2022}. Figure \ref{fig3} (a) shows the GCN accuracy of test data under different batch sizes (from 512 to 153451). We can observe that larger batch size is more advantageous for the GNN model to converge to better accuracy. Hence, in order to improve training accuracy and reduce training time, sample-based training needs to provide adequate memory space for batch data. 

% %However, it is not viable to store all vertex features because to the limited GPU memory. In addition, as analyzed above, batch data also needs enough memory to accelerate the tracing step. The caching policy's ability to accelerate the data loading step depends on the cache rate
% On the other hand, the memory consumption of cache data depends on the cache rate. The cache rate is the ratio of cached vertices to all vertices in graph data. The larger the cache rate, the better it is for reducing the feature transfer volume during the data loading, which is positively correlated as shown in Figure \ref{fig3}(c). The maximum cache rate that can be given is influenced by three factors: available GPU memory, total number of vertices, and feature dimensions. Therefore, the cache hit rate will get worse when the graph topological data and feature dimensions increase—this aligns well with the trends in GNN workloads \cite{p3_osdi_2021, Graphsage_2017}. Moreover, the hit rate decreases as the batch size increases, because the available GPU memory for cache data will be reduced.
%Hence, feature caching needs to be optimized by a method that can reduce the total amount of transfers without using much GPU memory.

%Based on the above analysis, when using GPUs to accelerate both the training and data loading steps, there will be inevitable contention for GPU storage resources. Using a tiny batch size affects model quality and training efficiency. Hence, we need a new method that can reduce the total transfer volume without using much GPU memory.

% \begin{table}[t]
% 	\vspace{-0.03in}
%     \centering
    
%     \caption{The runtime breakdown (in seconds) of a training epoch on DGL with different optimization and all of them. We run a 3-Layer GraphSAGE model \cite{GCN_iclr_2017} on the Wikipedia graph}	
%     \label{tab:breakdown-SDT}
   
% 	\centering
%     \small
% 	{\renewcommand{\arraystretch}{1.2}
%  %\renewcommand{\arraystretch}{1}
% 	%\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c c c c c}
% 		\hline
		
% 		\hline
% 		\multirow{1}*{\textbf{Configuration}}  &
%         \multirow{1}*{\textbf{S}} &
% 		\multirow{1}*{\textbf{D}}& {\textbf{T}} &{Total} &{\textbf{+pipeline}} &{\textbf{all}}\\
% 		\hline
%     	{Baseline}& {0.62} &{6.39} & {3.92} &{10.93} &{6.63} (-47.8\%)\\  
%         {Feature Cache} & {0.63} &{2.32} & {3.91}&{4.25} &{4.25} (-47.8\%)\\ 
%         {GPU Sampling} & {0.33} &{6.33} &{3.94}& {3.08}&{6.62}  (-47.8\%)\\  
%         {Both} & {0.33} &{4.73} &{3.92}& {3.08} &{}  (-47.8\%)\\  
% 		\hline
	
% 		\hline
% 	\end{tabular}}
%  \vspace{-0.4cm}
% \end{table}

\Paragraph{Case 4: Placing sampling and gathering on the GPU suffers from GPU memory and resource contention} We show an example of this case in Figure \ref{fig:task-method} (d). When all steps of sample-based GNN training are executed on the GPU, it results in GPU contention and CPU idle. The reasons for this situation have been discussed in cases 2 and 3. Firstly, the computation kernels of sampling and training compete for the limited GPU cores, leading to a slowdown in both. Secondly, the batch data for training and cache data for gathering contend with the limited GPU memory. When further making the GPU responsible for the sampling step, the GPU needs to additionally hold the graph topology data, which can make the GPU memory contention worse.  
%As shown in Table \ref{tab:breakdown-SDT}, the GPU-led optimizations bring considerable acceleration to each step, and the pipeline is equally essential optimization because that can make full use of different resources in heterogeneous systems. Due to the GPU contention, when all optimizations are applied together, each step does not get the desired acceleration. Through a series of GPU-based optimizations, the existing systems can not only complete the sampling and training step in GPU but also use GPU-based caching policy to accelerate the data loading step. 

%Optimal end-to-end performance comes from pipeline optimization, which makes better use of heterogeneous resources. 


\vspace{-0.05in}
\subsection{Summary}
We conduct experimental analysis on different task orchestrating methods in GPU-CPU heterogeneous environments. 
%这里应该概要地总结一下他们有什么问题
Our observations reveal that step-based task orchestrating leads to an imbalanced allocation of computational and memory resources. Assigning two or more steps to the GPU may result in memory or GPU resource contention. On the other hand, assigning one or two steps to the CPU may cause inefficient CPU processing to become a bottleneck. 
%inferior performance
%which inevitably leads to resource contention at one end of the heterogeneous resource. After analysis and experiments, we summarize the major problems of the step-based computation orchestrating strategies into two types: (1) GPU resource contention. (2) Inefficient CPU processing. 
%Most applications developed for heterogeneous systems are more suitable for the GPU than the CPU. Only considering the device's suitability when scheduling the applications overloads the GPU and leaves the CPU idle. This results in a longer execution time. On the other hand, overusing the CPU and making the GPU wait only makes the situation worse. 
A well-designed CPU-GPU heterogeneous system should ensure adequate and balanced CPU and GPU utilization to achieve optimal performance. However, the step-based task orchestrating methods fail to achieve this. This motivates us to design a resource-balanced task orchestrating method.
%The efficiency of a heterogeneous system requires considering both of the issues we mentioned above. 

% In Section \ref{sec2}, we summarize sample-based GNN training into the SDT model and summarize existing systems in two frameworks, CPU-centric framework and GPU-centric framework. Observing the resource utilization of CPU and GPU in Figure \ref{fig:utilization}, we find that both frameworks can only fully utilize one resource in the heterogeneous environment. Therefore, we present an in-depth analysis of two frameworks and pinpoint the main challenges to fully utilizing CPU and GPU at the same time.

% %分析两种framework的问题，分析只利用异构系统中一个器件带来的问题，
% % Although the aforementioned three optimizations can individually improve the performance of GNN systems, they only fouce on one step of the SDT model. Zero copy and feature caching focus on solving data loading, and the sampling step chooses to use GPU to accelerate. Each step in the SDT model depends on and waits for the completion of the previous step, so any slow step results in a long end-to-end time. 

% \vspace{-0.05in}
% \subsection{Analysis of CPU-centric framework}
% % %早期的Mini-batched gnn training中，只有training阶段使用GPU加速，CPU执行Sampling阶段以及大部分data loading阶段，这造成了高的CPU contention和低的GPU resource utilization. 
% As Figure \ref{fig:CPU-GPU}(a) shows, the CPU-centric framework only utilized GPU acceleration during the training step; the CPU handled the sampling step and most of the data loading step, leading to high CPU contention and GPU wait. 

% \begin{table}[t]
% 	\vspace{-0.03in}
%         \centering
%     \caption{The runtime breakdown (in seconds) of the data loading step on DGL with different dataset.}	
%     \label{tab:feature collection}
% 	\centering
%  \small
% 	{\renewcommand{\arraystretch}{1.2}
%  %\renewcommand{\arraystretch}{1}
% 	%\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c }
% 		\hline
% 		\multirow{1}*{\textbf{Dataset}}  & 
%         \multirow{1}*{\textbf{Feature Collection}}  &
%         \multirow{1}*{\textbf{Feature Transfer}}  \\
% 		\hline
%     	{reddit} & {1.358} & {1.339} \\ 
%     	{livejournal} & {35.324} & {33.043} \\  
% 	    {lj-links} & {38.645} & {35.957} \\      
% 	    {lj-large} & {24.993} & {20.163} \\     
%         {lj-large} & {108.038} & {95.124}\\  
% 		\hline
	
% 		\hline
% 	\end{tabular}
% 	}
% \end{table}

% \textbf{High CPU contention.} Given a mini-batch size, CPUs repeatedly sample the input graph. We often think of sampling as the main task in the CPU, while data loading relies on PCIe. However, when we look further into the data loading step, we find that slow external interconnects are not the biggest bottleneck limiting the data loading step. GNNlab \cite{GNNlab_EUROSYS_2022} and Pagraph \cite{PAGRAPH_SOCC_2020} as well as early DGL \cite{DGL_CORR_2019} use explicit copy to transfer vertex features. In this CPU-based data loading method, fragmented vertex features in CPU memory must be collected into a contiguous buffer before transferring them to GPU through explicit memory copy engine (cudaMemcpy). In real-world graphs, each vertex can be connected to thousands of neighbors. To collect relational information from those neighboring nodes, we may need to access thousands of scattered locations in memory. Without a doubt, such data access patterns make the traditional block data transfer method ineffective. As shown in Table \ref{tab:feature collection}, we break down the time spent in different stages of the data loading and find that the feature collection step takes much longer than the data movement through the PCIe.

% \textbf{GPU wait.} Low GPU utilization is also a major problem in the CPU-centric framework. The training step in the GPU needs to wait for the CPU to provide it with sample graph and vertex features. We counted the average GPU utilization of the GCN algorithm trained on the Reddit dataset. Figure \ref{fig:utilization} shows that only around 25\% of the GPU computational resources are in use. To fully investigate the hardware parallelism in the GPU, the mini-batch data in GPU must be more sufficient. 



% %This is because the mini-batch data that the CPU sent to GPU needs to be more sufficient to explore the hardware parallelism in GPU fully. Meanwhile, the GPU is idle, waiting for training data to arrive at most time. 

% % \vspace{-0.05in}
% % \subsection{Mini-batched GNN Training With High GPU contention}
% % Three GPU-led optimizations have been proposed to alleviate the CPU contention of training GNNs in a heterogeneous system. These optimizations are often used in isolation when they are proposed and are not considered compatible with each other. When these GPU-led optimizations are simply used simultaneously, they can cause GPU resource contention.

% % \textbf{Contention 1: Storage Resources.} 
% % While GPU sampling and feature Caching on GPU speed up GNN end-to-end time, they also place more strain on the GPU memory and create contention for storage resources. In the basic Mini-batched GNN training, just the batch data—which includes the sample graph, feature, and aggregation results—is maintained in the GPU memory. But now, GPU sampling needs to save the graph topological data in the GPU, and data loading needs to cache the frequently accessed features, as shown in Figure \ref{fig:CPU-GPU}(b). Unfortunately, the memory of a single GPU (e.g., 16 GB) cannot accommodate all three of these at the same time, especially for large-scale graphs with high-dimensional features. Thus, although GPU memory is usually able to keep all graph topological data, it will greatly limit the available memory capacity for batch data and feature cache. 

% % The memory consumption of batch data depends on the batch size. When the batch size of mini-batch training is increased, more memory is used since there is a larger scale sample graph with more training vertices. More training vertices also means more common neighbors, which are repeatedly sampled and computed when training vertices are scattered in different batches. The training with a larger batch size has a smaller computational amount because the common neighbors are only computed once. As shown in Figure \ref{fig3} (b), a larger batch size has higher memory consumption and faster average runtime for an epoch. However, less computational amount does not mean poorer model accuracy. Figure \ref{fig3} (a) shows the GCN accuracy of test data under different batch size (from 512 to 153451). We can observe that a larger batch size is more advantageous for the GNN model to converge to a better accuracy. Hence, in order to improve training accuracy and reduce training time, mini-batched training need to provide adequate memory space for batch data. 

% % The memory consumption of batch data depends on the cache rate. The improvement of the data loading time by caching features becomes trivial as it is positively correlated with the cache ratio of features in GPU memory, as shown in Figure \ref{fig3} (c). More importantly, it will get worse when the topological data and feature dimensions increase—this aligns well with the trends in GNN workloads. Hence, in order to reduce data loading time, mini-batched training need to provide adequate memory space for batch data. 

% % \textbf{Contention 2: Computational Resources.} 
% % The processing engines of Mini-batched GNN training require different resources, including GPU for the sampling kernel and training kernel, and PCIe for the host-GPU data loading. To overlap the resource utilization and improve the parallelism, Existing works use multiple CUDA streams to process the tasks concurrently. During the iterative processing, the task scheduler monitors the available streams and assigns them to tasks that have not been scheduled. The operating system will automatically overlap data transfer and kernel computation of different streams. However, when the GPU executes the Sampling and training processes simultaneously and different batches are fully pipelined, there will be contention for GPU computational resources. We observe the execution time of Sampling and training for different number of streams. As observed in Table \ref{tab:pipiline}, the single stream has the longest total execution time, but has a shorter sampling and training execution time. Because, when only one CUDA stream is working, the Sampling and Training step each use all GPU cores.

% \begin{table}[t]
% 	\vspace{-0.03in}
%     \centering
    
%     \caption{The runtime breakdown (in seconds) of a training epoch on DGL with different optimization and all of them.}	
%     \label{tab:breakdown-SDT}
   
% 	\centering
%     \small
% 	{\renewcommand{\arraystretch}{1.2}
%  %\renewcommand{\arraystretch}{1}
% 	%\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c c c c c}
% 		\hline
		
% 		\hline
% 		\multirow{1}*{\textbf{Configuration}}  &
%         \multirow{1}*{\textbf{S}} &
% 		\multirow{1}*{\textbf{D}}& {\textbf{T}} &{Total}\\
% 		\hline
%     	{Baseline}& {0.324}&{0.849} &{0.711} & {1.884}  \\  
%         {Zerocopy}& {0.331} &{0.408} &{0.705} &{1.444}\\
%         {Cache} & {0.328} &{0.279} &{0.709} &{1.316}  \\ 
%         {GPU sampling} & {0.084} &{0.850} &{0.709} &{1.643}  \\ 
%         {Pipeline} & {0.325}&{0.846} &{0.714} & \textbf{0.924} \\
%         {All optimizations}& {0.375} &{0.332} &{0.907} &{1.164} \\  
% 		\hline
	
% 		\hline
% 	\end{tabular}}
% \end{table}

% \vspace{-0.05in}
% \subsection{Analysis of GPU-centric framework}
% As shown in Table \ref{tab:breakdown-SDT}, in the GPU-centric framework, the GPU-led optimizations bring considerable acceleration to each step in the SDT model. In addition, the pipeline is equally essential because that can make full use of different resources in heterogeneous systems. However, when all optimizations are simply used simultaneously, it causes GPU contention and CPU idle. As shown in Table \ref{tab:breakdown-SDT}, when all optimizations are applied together, each step of SDT does not get the desired acceleration. Optimal end-to-end performance comes from pipeline optimization, which makes better use of heterogeneous resources. Therefore, in order to analyze the impact of GPU contention on performance, we will analyze the problem in the GPU-centric framework in terms of computational and storage resources.

% % Sampling is an irregular computation that is typically performed using the CPU, whereas training is performed on the GPU. As pointed out by past works \cite{Nextdoor_eurosys_2021,lazygcn_2020_neuips}, accelerating graph sampling is important to improve the end-to-end training time. As shown in Table \ref{tab:breakdown-SDT}, we found that graph sampling can take up to 24\% of an epoch’s time in baseline of DGL \cite{DGL_CORR_2019}. After adding caching and zero copy optimization, the time occupation of graph sampling is increased to 45\%. However GPU sampling's capacity to decrease end-to-end time is not optimal.
% \textbf{Contention 1: Computational Resources.} There may be contention for computational resources when the GPU accelerates sampling and training simultaneously. Figure \ref{fig:pipeline} shows three examples of multi-stream pipeline in different configurations. First, When there is only one stream, the current batch needs to wait for the previous batch to finish before it can start execution, which can cause a waste of computational resources. Therefore, the three steps of the SDT model are usually overlapped in heterogeneous systems to achieve accelerated overall performance. However, when the GPU executes the sampling and training step simultaneously, there will be contention for GPU computational resources. 

% %In the baseline, the processing engines of sample-based GNN training require different resources, including CPU for sampling kernel, GPU for the training kernel, and PCIe for the host-GPU data loading. To overlap the resource utilization and improve the parallelism, Existing works use multiple CUDA streams to process the tasks concurrently. During the iterative processing, the task scheduler monitors the available streams and assigns them to tasks that have not been scheduled. The operating system will automatically overlap data transfer and kernel computation of different streams. 

% \begin{figure}[htbp]
%   \centering
%   \includegraphics[width=8cm,page={1}]{figures/pipeline.pdf}
%   \vspace{-0.4cm} %调整图片与上文的垂直距离
%   \caption{Three examples of multi-stream pipeline in different configurations}
%   \label{fig:pipeline}
% \end{figure}




% \begin{table}[t]
% 	\vspace{-0.03in}
%     \setlength{\tabcolsep}{1mm}{
%     \centering
%     \caption{The runtime breakdown (in seconds) of a training epoch on DGL with CPU sampling or GPU sampling.}	
%     \label{tab:breakdown-GPU_Sampling}
% 	\centering
%  \small
% 	{\renewcommand{\arraystretch}{1.2}
%  %\renewcommand{\arraystretch}{1}
% 	%\resizebox{0.97\linewidth}{!}{
% 	\begin{tabular}{l c c c c c c c}
% 		\hline
% 		\multirow{1}*{\textbf{Configuration}}  &
%         \multirow{1}*{\textbf{GPU Utilization}} &
%         \multirow{1}*{\textbf{S}} &
% 		\multirow{1}*{\textbf{D}}& {\textbf{T}} & {\textbf{Total}}\\
% 		\hline
%     	{CPU sampling}&{59\%}&{0.84}&{1.07} &{1.72} & {3.63} \\ 
%     	{CPU sampling + pipeline} & {78\%} & {0.83} &{1.03} &{1.82} & {1.92}  \\  
% 	    {GPU sampling}&{72\%}& {0.33} &{0.93} &{1.74} & {3.08} \\      
% 	    {GPU sampling + pipeline}  & {81\%}& {1.18} &{1.13} &{2.03} & {2.15} \\     
% 		\hline
	
% 		\hline
% 	\end{tabular}}
% 	}
% \end{table}

% \begin{figure*}[htbp]
	
% 	\begin{minipage}{0.32\linewidth}
% 		\vspace{3pt}
%         %这个图片路径替换成你的图片路径即可使用
% 		\centerline{\includegraphics[width=\textwidth]{figures/batchacc.pdf}}
%           % 加入对这列的图片说明

% 	\end{minipage}
% 	\begin{minipage}{0.32\linewidth}
% 		\vspace{3pt}
% 		\centerline{\includegraphics[width=\textwidth]{figures/batch-RM.pdf}}

% 	\end{minipage}
% 	\begin{minipage}{0.32\linewidth}
% 		\vspace{3pt}
% 		\centerline{\includegraphics[width=\textwidth]{figures/cache-RM.pdf}}

% 	\end{minipage}
 
% 	\caption{(a) Accuracy comparison between different batchsize of GCN on Reddit dataset. (b)Runtime of one epoch and Memory usage comparison between different batchsize of GCN on Reddit dataset. (c)Runtime of one epoch and Memory usage comparison between different cache rate of GCN on Reddit dataset.}
% 	\label{fig3}
% \end{figure*}


% \begin{figure}[htbp]
%   \centering
%   \includegraphics[width=8cm,page={1}]{figures/computer-batch.pdf}
%   \vspace{-0.4cm} %调整图片与上文的垂直距离
%   \caption{Calculation under different batch size}
%   \label{fig:compute-batch}
% \end{figure}


% Therefore, we observe the performance of CPU sampling and GPU sampling under normal vision and pipeline vision. As observed in Table \ref{tab:breakdown-GPU_Sampling}, the pipeline is an engineering optimization with significant performance improvement. With CPU sampling, the pipeline improves the end-to-end time by 1.89x with no change in the time of each step. However, when GPU sampling adds pipeline optimization, the overall performance improvement is only 1.43x. This is because sampling and training in the GPU are competing for GPU resources, we can observe that the sample and training steps in the GPU sampling pipeline version are much slower than GPU sampling version (1.18 vs 0.33 and 2.03 vs 1.74). As shown in Figure \ref{fig:pipeline}, in the version of the GPU Sampling pipeline, the training kernel in stream 1 shares GPU resources with the sampling kernel in stream 3. The limited GPU resources cannot accelerate both the sampling and training steps at the same time. With sufficient GPU computing resources, kernels in different streams can be executed in parallel to achieve optimal performance. However, a simple pipeline design can cause different kernels to compete for resources and fail to fully utilize GPU resources. Table \ref{tab:breakdown-GPU_Sampling} shows the average utilization of GPU computing resources in different configurations. We can observe that in the CPU Sampling pipeline version, the utilization can already be as high as 78\% after three steps are fully pipelined. In the GPU sampling version, GPU utilization reaches 72\% and sampling step accelerated by 2.5x (0.83 vs 0.33). After further adding pipeline optimization to hide the data loading step, GPU utilization improves by just 3\% (81\% vs 78\%). 

% Based on the above analysis, although GPU sampling greatly accelerates the sampling step, end-to-end performance is still sub-optimal due to GPU resource contention. As shown in Table \ref{tab:breakdown-GPU_Sampling}, the end-to-end performance of the CPU sampling pipeline version is 1.11x faster than the GPU sampling pipeline version. That means that we need to ensure that the GPU can efficiently accelerate the sampling step while avoiding GPU computational resource contention to maximize GPU utilization.

% %ensuring that the computational tasks do not exceed the GPU resource limit.


% %Meanwhile, in the GPU sampling pipeline version, the GPU utilization is only increased by 3\%, which means that the GPU does not provide enough resources for sampling, and the contention of resources will make both sampling and training steps slow.

% % Existing work\cite{GNNlab_EUROSYS_2022,PAGRAPH_SOCC_2020,DATATIRING_VLDB_2022} usually uses a caching strategy to reduce the amount of transfers by residing the more frequently used features in the GPU. As shown in \ref{tab:breakdown-SDT}, after caching 20\% of the vertex features, the data loading time drops by 52\%. 

% % Instead of using Explict copy, Seung Won Min et al program GPU cores to directly access host memory with zero copy memory access. With this optimization, zero-copy PCIe bandwidth can match up to 93\% of block transfer PCIe bandwidth and eliminate feature collection overhead. As shown in \ref{tab:breakdown-SDT}, after adding zero copy optimization, the data loading time drops by 64\%. 
% % Acceleration of modern machine learning models is often severely limited by insufficient memory bandwidth. To provide the best possible memory bandwidth, data is usually placed in memory closest to the processing units of the accelerators. However, with extremely large datasets, it is inevitable to put data farther from the processing units to take advantage of larger capacity (e.g., host memory). In this case directly accessing remote data from the processing units can be very inefficient due to slow external interconnects. The sample-based GNN training faces a data loading problem, i.e., the data movement from the host memory to the GPU memory, especially for loading vertex features.
% % It's fortunate that feature caching and zero copy can cooperate without conflicting with each other. As shown in \ref{tab:breakdown-SDT}, After adding both feature caching and zero copy optimization, the data loading time drops by 79\%. In conclusion, zero copy and feature caching speed up the data loading process from a different perspective. Zero copy eliminates the feature collection step of the data loading and improves the average bandwidth utilization of PCIe. 
% % Feature caching reduces the total amount of features transferred during data loading. Compared to zero copy, feature caching requires enough GPU memory to perform well.
% % \vspace{-0.05in}
% % \subsection{Analysis of Data Loading}


% \textbf{Contention 2: Storage Resources.} While GPU sampling and feature caching speed up the end-to-end time, they also place more strain on the GPU memory and bring contention for storage resources. In the basic sample-based GNN training, just the batch data—which includes the sample graph, feature, and aggregation results—is maintained in the GPU memory. But now, GPU sampling needs to save the graph topological data in the GPU, and data loading needs to cache the frequently accessed features, as shown in Figure \ref{fig:CPU-GPU}(b). Unfortunately, the memory of a single GPU (e.g., 16 GB) cannot simultaneously accommodate all three of these, especially for large-scale graphs with high-dimensional features. Thus, although GPU memory can usually keep all graph topological data, it will greatly limit the available memory capacity for batch data and caching data. However, both data loading and training steps require sufficient memory for batch data and caching data to accelerate effectively.

% % As shown in Figure \ref{fig3} (b), a larger batch size has higher memory consumption and faster average runtime for an epoch. However, less computational amount does not mean poorer model accuracy. Figure \ref{fig3} (a) shows the GCN accuracy of test data under different batch size (from 512 to 153451). We can observe that a larger batch size is more advantageous for the GNN model to converge to a better accuracy. Hence, in order to improve training accuracy and reduce training time, mini-batched training need to provide adequate memory space for batch data. 

% The memory consumption of batch data depends on the batch size. When the batch size of training is increased, more memory is used since there is a larger scale sample graph with more training vertices. Due to the connectivity of the graph data, vertices have many common neighbors with each other. More training vertices in a batch also mean more common neighbors, which are repeatedly sampled and computed when training vertices are scattered in different batches. Figure \ref{fig:compute-batch} shows two training examples under different batch sizes. When training vertices are scattered in different batches, vertex $3$ and vertex $7$ are sampled and computed repeatedly. Therefore, the training with a larger batch size has a smaller computational amount because the common neighbors $3$ and $7$ are only computed once. After further experiments, we found that batch size can greatly affect the training time. As shown in Figure \ref{fig3} (b), training with a larger batch size has higher memory consumption and faster average runtime for an epoch. However, the less computational amount does not mean poor model accuracy. Figure \ref{fig3} (a) shows the GCN accuracy of test data under different batch sizes (from 512 to 153451). We can observe that a larger batch size is more advantageous for the GNN model to converge to better accuracy. Hence, in order to improve training accuracy and reduce training time, sample-based training needs to provide adequate memory space for batch data. 

% %The reason is that as the batch size increases, the total computation amount of training decreases.


% The memory consumption of cache data depends on the cache rate. The cache rate is the ratio of cached vertices to all vertices in graph data. The larger the cache rate, the better it is for reducing the amount of feature transferred during the data loading. However, it is not viable to store all vertex features because to the limited GPU memory. In addition, as analyzed above, batch data also needs enough memory to accelerate the tracing step. The maximum cache rate that can be given is influenced by three factors: available GPU memory, total number of vertices, and feature dimensions. The caching strategy's ability to accelerate the data loading step depends on the size of the cache rate, which is positively correlated as shown in the Figure \ref{fig3}(c). Therefore, it will get worse when the graph topological data and feature dimensions increase—this aligns well with the trends in GNN workloads. Hence, feature caching needs to be replaced by a method that can reduce the total amount of transfers without using much GPU memory.



% % \vspace{-0.05in}
% % \subsection{Analysis of Training}
% % %分析一下，在改变fanout后，training也可能成为瓶颈，其次training 的batch对整体性能和模型质量都有影响，应该使用更接近fullbatch的训练，而不是把有限的GPU内存给cache
% % Past work has always focused on solving the sampling and data loading step. However the training step can also be a major time overhead for the overall process. The reason is twofold. First, the sampling phase samples the neighbors of the vertices at each layer, and the number of neighbors sampled in each layer is fixed, which is usually called fanout. The fanout is an important parameter that determines the amount of tasks in each step. Because the size of the sampling graph rises as the fanout increases, which determines the amount of computation in the sampling and training steps. Meanwhile, the amount of data transfer in the data loading step depends on the leaf vertices of the sampling graph. The data loading step will de-duplicate the leaf vertices and transfer all non-duplicate vertex features to the GPU. It is worth noting that as the fanout increases, the time ratio of the three steps will change, and the time of sampling and training will gradually exceed the data loading, as shown in the figure \ref{fig:Bottleneck Analysis}. Because data loading is a point-level extension, while the computation of Sampling and data loading is an edge-level extension. Secondly, after sampling and data loading are optimized, the time of training does not change, as shown in the table \ref{tab:breakdown-SDT}, after applying all the optimizations, training will take the longest time of the whole process.

% % \begin{figure}[htbp]
% %   \centering
% %   \includegraphics[width=8cm,page={1}]{figures/SDT-breakdonw.pdf}
% %   \vspace{-0.4cm} %调整图片与上文的垂直距离
% %   \caption{Training epoch time breakdown of Mini-Batched GNN training on a GPU with different data loading method and fanout.}
% %   \label{fig:Bottleneck Analysis}
% % \end{figure}




% % The memory consumption of batch data depends on the batch size. When the batch size of mini-batch training is increased, more memory is used since there is a larger scale sample graph with more training vertices.

% \vspace{-0.05in}
% \subsection{Summary}
% We discuss the two frameworks regarding resource utilization in heterogeneous systems. Therefore, We summarize the major problems of the two frameworks into two types: (1) GPU contention but CPU idle, (2) GPU wait caused by CPU contention. Most applications developed for heterogeneous systems are more suitable for the GPU than the CPU. Only considering the device's suitability when scheduling the applications overloads the GPU and leaves the CPU idle. This results in a longer execution time. On the other hand, overusing the CPU and making the GPU wait only makes the situation worse. The efficiency of a heterogeneous system requires considering both of the issues we mentioned above. For optimal performance, a good CPU-GPU heterogeneous framework should make balanced use of CPU and GPU.
\section{\system}
\label{sec4}


% \begin{figure}
%   \centering
%   \includegraphics[width=8cm,page={1}]{figures/Architecture.pdf}
%   \caption{Overall architecture of \system}
%    \vspace{-0.4cm}
%   \label{fig:architecture}
% \end{figure}

%改名 
%Motivated by the analysis and experimental results presented in Section \ref{sec2},
%Specifically, our architecture can achieve an average $\sim$ 90\% GPU utilization. 

We propose \system, a sample-based GNN training system that effectively improves CPU and GPU resource utilization through two critical techniques. 
% First, a hotness-aware layer-based task orchestrating method with bounded staleness that partitions and deploy the task among layers, improving resource utilization by eliminating inefficient CPU processing and GPU resource contention. Second, a super-batch pipelined training, which fully overlaps different tasks on heterogeneous devices while strictly guaranteeing bounded staleness.
% \wqg{WGQ:I think this statement is not strong\#
% Moreover, \system also provides a set of GPU-CPU optimizations to achieve high performance.}
% %\wqg{Finally, a super-batch pipeline with historical embedding is designed to fully Leverage heterogeneous resources and manage bounded staleness of historical embedding.}
% \zyf{comments}
%\zyf{If you only place the bottom layer on CPU, that would limit the contribution. After reading the motivation of improving CPU-GPU resources utilization, it looks like you are selling a big idea, but your approach is too specific and simple. It makes me feel I was cheated. Can you allow multiple layers? with a flexible configuration? Otherwise, is it OK for a super-powerful CPU with a poor GPU? or how about a super-poor CPU with a powerful GPU? I think your key contribution is allowing fine-grained task orchestrating (\textbf{layer level rather than task level}).comments}

%\subsection{System Overview}
%通过第三章的分析，我们可以发现GPU中sampling与training的对计算资源的争用
%After in-depth analysis and experiments, we found that GPU resource contention, which leads to poor end-to-end performance. In this computation orchestrating strategy, \system avoids two possible performance bottlenecks in the heterogeneous platform, GPU resource contention, and inefficient CPU processing.

\Paragraph{Hotness-aware layer-based task Orchestrating. }
Unlike step-based task orchestrating methods, \system decouples the training process by layers rather than steps and employs the sample-gather-train computation of each sub-task to a single device, eliminating the constraint of computing each step entirely on the CPU or GPU. To prevent CPU computation from becoming a bottleneck, \system allows the CPU to compute embeddings only for the hot vertices that are frequently accessed. Moreover, \system extends stale synchronous processing \cite{ssp_2013_nips} to guarantee bounded staleness of embedding reuse, thereby guaranteeing final accuracy. 

%To accommodate different CPU-GPU resource configurations, we also control the proportion of hot vertices assigned to CPU embedding computation and GPU feature caching. Our design can effectively balance CPU and GPU resource utilization and reduce GPU-CPU data communication.

% \notecolor{Finally, to further optimize CPU-GPU communication, we will adjust the cache ratio of embeddings and features based on the heterogeneous resources.}

%while enabling the GPU to reuse these embeddings across batches within bounded staleness
%In terms of computing resources, decoupling GNN tasks by layer allow the flexible introduction of CPU resources and share the computation tasks of the GPU. In terms of storage resources, we can make more GPU memory available for training data without caching any feature. 
%In addition, during the training, the training data is maintained by both CPU and GPU, which enables less GPU memory to be used for the same batch size. 
%Supporting larger batch size is good for overall training, as shown in section \ref{sec2}. 
%GPU training can use the historical embeddings provided by the CPU which are staleness in fixed version bound. Our pipeline design is based on two observations: (1)  (2) Hot vertices are duplicated between batches and the selection of hot vertices can be done within part of the batches locally.
%To ensure optimal resource utilization, a fine-grained pipeline design is crucial. 
%Additionally, in order to reduce CPU computation and maintain bounded staleness, we refine the selection granularity of the hot vertices queue to span multiple batches.
%Concurrently executing tasks deployed on the GPU and CPU is essential to achieve high performance. However, it is challenging in \system because the bounded staleness training creates cross-layer data dependencies between the CPU and GPU training process. To efficiently resolve the CPU-GPU staleness control, we propose a super-batch pipelined training to fully overlap various tasks while strictly guaranteeing bounded staleness. 

%Typically, a historical embedding becomes stale after a few batches \cite{SANCUS_VLDB_2022}. Selecting hot vertices from a limited number of upcoming batches is more reasonable and efficient than selecting hot vertices from the entire training set.

%Specifically, we combine $k$ batches into a super-batch and select the hot vertices for each super-batch. Within each super-batch, the CPU computes the bottom layer embeddings of hot vertices for the next super-batch. At the same time, the GPU pulls historical embeddings from the previous super-batch on demand. This pipeline design ensures that historical embeddings from the last super-batch are only utilized within $k$ batches in the current super-batch, enabling effective control of bounded staleness.

\Paragraph{Super-batch pipelined training. } 
Concurrently executing sub-tasks deployed on the GPU and CPU is essential to achieve high performance. However, it is challenging to design a task scheduling method for layer-based task orchestrating because of the cross-layer data dependencies between the CPU and GPU training process. To solve this, we propose a super-batch pipelined training. In each super-batch, the CPU and GPU execute sub-tasks concurrently and independently of each other so that the training process is fully pipelined. In addition, this pipeline design ensures that the historical embeddings from the previous super-batch are only accessible within the current super-batch, naturally and efficiently enabling strict bounded staleness. 

%to eliminate data dependencies in CPU and GPU at the same time and resolve the CPU-GPU staleness control efficiently.
%and combine $N$ adjacent batches into a super-batch. 
%control the update and reuse of historical embeddings among super-batches
% Maximizing resource utilization in heterogeneous systems requires not only effective task orchestration but also the complete overlap of tasks across diverse resources.
% The engines of \system rely on various resources, including CPU for training hot vertices in the bottom layer, GPU for training non-hot vertices and other layers, and PCIe for data loading. Designing overlapping execution between computation and data loading is straightforward.
% However, the overlapping execution of the training process in CPU and GPU requires further design to ensure bounded staleness.
% %Therefore, we reduce the selection granularity of the hot vertices to span multiple continuous batches.
% To achieve this, we create super-batches by combining $K$ individual batches before the training begins. Each super-batch has a selected hot vertices queue that contains the frequently accessed vertices within the super-batch.
% Within each super-batch, CPU and GPU computations are overlapped. The CPU calculates the bottom layer embeddings for the hot vertices of the next super-batch, while the GPU utilizes the historical embeddings from the previous super-batch as needed. This approach guarantees that the staleness between historical embeddings and the latest model parameter is limited to two super-batches.
%Furthermore, the data loading step within each super-batch overlaps with the computation.

%Before training starts, we combine $K$ batches into a super-batch and select the hot vertices queue for each super-batch. During pipelined training, CPU and GPU computations are overlapped. Within each super-batch, the CPU computes the bottom layer embeddings for the hot vertices of the next super-batch, while the GPU utilizes the historical embeddings from the previous super-batch on demand. Meanwhile, in each super-batch, the data loading step of each batch also overlaps with the computation. This process ensures that the difference (staleness) between the historical embedding and the latest model parameter is limited to $2k$.

%\system can adjust hot vertices rate and batch number $k$ in a super-batch to control the historical embeddings computation period in the CPU.

%Like the traditional caching strategy, we need to select a set of vertices that are accessed more frequently. These vertices may be 10\%-20\% or more of the total vertices. However, unlike traditional caching strategies, the cached objects are historical embedding in the bottom layer. Therefore, the CPU no longer performs the complete bottom layer in every batch but only the cached vertices. These historical embeddings are cached in the CPU or GPU and reused as many times as bounded staleness allows. Our strategy consumes less GPU memory than traditional caching strategies. First, the dimensions of vertex embedding is much smaller than that of vertex feature. Second, the GPU does not cache all historical embeddings. In each batch, the GPU pulls the current version of the historical embeddings from the CPU on demand and caches them until the version is stale.


% First, when the bottom layer is calculated in the CPU, the cache object is no longer the vertex feature, but the vertex embedding of the bottom layer. Caching embedding is more memory efficient compared to feature. Meanwhile, besides the GPU, the CPU can also be used as a cache site for embedding. Because cache embedding not only save transfers but also save computation amount and prevents the CPU from becoming a new bottleneck. As long as bounded staleness allows \cite{SANCUS_VLDB_2022}, for the frequently accessed vertices in the bottom layer, we compute and cache their historical embedding in the CPU and the GPU.

% \textbf{Super-batch pipeline with Historical embeddings.}
% With the above optimizations, we successfully solve the resource contention problem in step-based computation orchestrating method. A CPU-GPU balanced task division is crucial, but a fine-grained pipeline design is also required to fully leverage the resources of heterogeneous systems. Meanwhile, in the pipeline design, we can better identify and manage the version staleness caused by the use of historical embedding. Therefore, \system designs a super-batch pipeline for the new computation orchestrating method to ensure maximal CPU and GPU utilization. 

% Figure \ref{fig:architecture} shows the overall architecture of \system. The CPU side is responsible for computing part of the underlying training task and caching the computation results on the CPU side. When bounded staleness conditions allow, vertex embeddings are reused without the need for resampling and retraining. The CPU side is also responsible for saving the graph structure as well as the feature. On the GPU side, thanks to new computational model of \system, the GPU caches objects from vertex features to vertex embeddings, with a nearly tenfold difference in their data size, and the cache location is no longer limited to the GPU. Because the vertex embedding-based caching strategy not only reduces the transmission volume, but also saves the sampling and training overhead. In order to further save GPU memory resources and thus support larger batchsize training, \system supports out-of-memory sampling based on zero-copy, and the GPU side no longer needs to save graph topology data. 

% \subsection{\system Architecture}
% \vspace{-0.05in}

%As discussed in Section \ref{sec3}, existing task orchestrating methods requires dividing the three steps between two devices, resulting in one device being responsible for two or three steps. Step-based task orchestrating methods inevitably lead to resource contention on one side of the heterogeneous environments. Consequently, these methods often result in suboptimal performance due to inefficient CPU processing or GPU resource contention. The imbalance in distributing computation tasks among heterogeneous resources is a significant challenge. \zyf{looks like redundant, you repeat this several times.}
%In heterogeneous systems, the CPU is usually used as the data storage unit and all computational tasks are scheduled on the GPU.
%However, through the analysis in Section \ref{sec2}, we find that sampling and training in GPUs compete for computational resources, and batch data and cache data compete for storage resources. 

%Balancing the workload of such systems is a very complex task. Most applications developed for heterogeneous systems are more suitable for the GPU than for the CPU. Only considering the device's suitability when scheduling the applications overloads the GPU and leaves the CPU idle. This results in longer execution times and more energy consumption. 

%这还需要改
%Existing works typically introduce more GPU resources to perform two steps simultaneously and ignore the remaining computing power in the CPU. 
%Achieving workload balance in heterogeneous systems is a complex and challenging task. \ccy{(see comments)}
% ccy: 实现异构环境的负载均衡？负载均衡可能用词不准确，负载均衡指的是指在设备之间平均分配工作量。 maybe  实现异构环境的资源利用最大化？
% %The GPU acceleration of the three steps is significant, making it detrimental to shift them back to the CPU in terms of performance. However, simultaneous processing of these steps in the GPU can lead to unfavorable acceleration due to GPU resource contention. On the other hand, assigning one or two steps to the CPU may cause inefficient CPU processing becomes a bottleneck.  \ccy{(I suggest adjusting the order of the above two sentences, first talk about CPU, then talk about GPU.)}
% \wqg{WQG@:TOO MANY THINGS TALKED BEFORE, I remove them all, Just talk about what we discussed in this section
% }
%An example is a typical computation orchestrating method in the CPU-GPU heterogeneous system, where the CPU handles the sampling while the data loading and training steps are accelerated by the GPU. In this training method, previous researchers have frequently viewed sampling as the main bottleneck \cite{lazygcn_2020_neuips, Nextdoor_eurosys_2021,Marius++_arxiv_2022}. The main reason is that the sampling task can sometimes overload the CPU's computing power, leading to GPU waiting times.
\subsection{Hotness-Aware Layer-based Task Orchestrating}
In this section, we give a detailed discussion on the hotness-aware layer-based task orchestrating. It maximizes CPU-GPU resource utilization based on two principles. Firstly, partitioning the tasks in a fine-grained manner to balance the workloads on the CPUs and GPUs. Secondly, combining CPU and GPU resources to minimize CPU-GPU communication overhead. 

% Secondly, the computation in the CPU should be minimal yet valuable \zyf{minimal yet valuable, why not fully utilize CPU?}, enabling the CPU to assist the GPU in sharing computation tasks without making GPU wait.

%Firstly, the computation in the GPU should be adequate and reasonable, ensuring that GPU computing resources are fully utilized but do not cause memory or GPU resource contention. Second, the computation in the CPU should be minor and valuable, ensuring that the CPU can help the GPU share computation tasks and not cause the GPU to wait.


% \begin{table}%[!htbp]
% \vspace{-0.1in}
% \small
% \renewcommand{\arraystretch}{1.2}
% \centering
% 	\caption{Memory size of the feature and embedding for different hops in . Batch size :10000 Fanout: 4-4.}
% 	\label{tab:Memory size comparision}
% \begin{tabular}{|c|c|c|c|c|} 
% \hline
% Dataset & Hop & Vertex & Dimension & Memory (GB)\\ 
% \hline
% \multirow{2}*{Reddit}&source & 28706 &128 & 14.1\\ %600 * num * 4 / 1024 / 1024
% \cline{2-5}
% &destination & 66175 & 602 & 64.6\\

% \hline
% \multirow{2}*{Lj-large}& source & 24012 &128 &11.7\\
% \cline{2-5}
% &destination & 46475& 600 & 45.4\\

% \hline
% \multirow{2}*{Orkut}&source &24899 &128 &12.2\\
% \cline{2-5}
% &destination & 45813 &600 &44.7\\

% \hline
% \multirow{2}*{Wikipedia}&source &17216 &128 &8.4\\
% \cline{2-5}
% &destination & 26310 & 600 & 25.7\\

% \hline

% \end{tabular}
% \end{table}

\begin{figure}
  \centering
% \vspace{-0.05in}  
\includegraphics[width=8cm,page={1}]{figures/layer-task.pdf}
     \vspace{-0.4cm}
      \caption{Workload and transfer volume of all layers on GPU and the layer-based task orchestrating, where |V| represents the number of vertices and Dim represents the dimension of features or embeddings. GNN: A 2-Layer GCN \cite{GCN_iclr_2017}. Dataset: Reddit.}
   \vspace{-0.3cm}
  \label{fig:layer-task}
\end{figure}

% Existing distributed GNN systems employ the data parallel training approach \cite{neutron_sigmod_2022,ROC_mlsys_2020,distgnn_sc_2021}. 
% These systems enable each vertex to collect its neighbors' embeddings from remote workers through communication. The graph propagation (including both the forward and backward propagation) occurs across workers. This design can be implemented by treating the CPU and GPU as separate workers and dividing the training process by layer between them. \zyf{I am not sure whether you want to mention distributed training. it might be unnecessary.} 

\subsubsection{Layer-based Task Orchestrating}
%Following these two principles, we decouple and analyze the sample-based GNN training to determine the task orchestrating method.
%We find that the training process of sample-based GNN training can be divided not only by the three steps but also by GNN layers. Fortunately, the CPU has a natural execution advantage for the bottom layer.

We find that the training process of sample-based GNN training can be divided not only by step but also by the GNN layer. As shown in Figure \ref{fig:task-method} (e), the CPU is responsible for the bottom layer computation of GNN training, while the GPU is responsible for the computation of the other layers. 
The CPU exhibits an inherent advantage when executing the bottom layer. Firstly, as the storage of features, the CPU can directly perform the GNN computation of the bottom layer without executing the time-consuming feature collection stage.
Secondly, the CPU-GPU communication overhead will decrease as the transfer objects are changed from the neighbors' features to the computed embeddings.
%The CPU can directly aggregate fragmented vertex features and write embeddings to continuous space after aggregation and update.
% \zyf{why not need feature collection stage?}
%of sampled vertices are aggregated and updated before the transfer begins.
% \wqg{comments: Lack of complete introduction or picture on Layer-based TO}
% %我咋感觉，你还没说 layer-based orchestrating 是啥呢？ 
% comment sanzo: backward pass or backward computation?
In layer-based task orchestrating, an additional data transfer stage is required in the backward pass for transferring the gradient back to the bottom layer training in the CPU. To improve training efficiency, we move the backward pass of the bottom layer to the GPU. Specifically, the required data for the gradient computation is transferred to the GPU along with the computed embeddings in the forward computation. Take the GCN \cite{GCN_iclr_2017} algorithm as an example. The aggregated neighbor representation and newly computed vertex embedding are transferred to the GPU.  

%毫无用处，且不知道在干什么的说法
%{we combine the two data transfers into one host-to-GPU data transfer.} 

%The required data for the backward pass is transferred together with the embeddings. 
%\zyf{the writing needs polish}  
% Compared to the traditional method, our method still decreases the total transfer volume because the aggregated results reduce the amount of the vertices to be transferred. 
% \zyf{too detailed, not reflected in the figure, the figure is hard to understand. Maybe you can use a concrete graph with 6-8 nodes associated with dimensional features to illustrate.}
%and four random neighbors are sampled for a vertex in each layer

We execute a two-layer GCN on the Reddit dataset to evaluate the impact of the layer-based task orchestrating method on the workload and transfer volume in the training process.
The training batch size is 10000, and four random neighbors are sampled for a vertex in each layer. We count the number of vertices and the feature or embedding dimension of each model layer.
As shown in Figure \ref{fig:layer-task}, the number of vertices in the bottom layer is three times that in the middle layer (86175 vs. 28706), positively related to the number of sampled neighbors (i.e., fanout). 
Meanwhile, the embedding dimension of the middle layer is generally smaller than the features in the outermost layer. Thus, despite introducing additional transmission for backward computation on the GPU, the layer-based task orchestrating method still has a smaller transfer volume.
On the other hand, this method makes more GPU memory available for training data, as it maintains the training data in both CPU and GPUs, resulting in reduced GPU memory usage. 

By employing layer-based task orchestrating, \system significantly reduces the computational workload and memory footprint of GPU training. 

% Because we don't need to cache vertex features, the vertex features are aggregated and updated on the CPU.
% \zyf{-- because ---, logical gap} 

% \begin{figure}
%   \centering
%   \includegraphics[width=5cm,page={1}]{figures/computing_example.pdf}
%   \caption{An example of GPU-centric, CPU-assisted Computing Framework}
%    \vspace{-0.4cm}
%   \label{fig:computing-example}
% \end{figure}

% Figure \ref{fig:computing-example} shows an example of our computation orchestrating strategy for training a 2-layer GNN. The CPU aggregates and updates the bottom layer while the GPU performs the sampling and subsequent training. The GPU receives the embedding result from the CPU forward and returns the corresponding grad backward. Transferred objects changed from features of source vertices in blue to the embeddings and the grad of destination vertices in yellow. 

% Further refinement of the layer-based computation orchestrating strategy's workflow is necessary to ensure optimal utilization of both the CPU and GPU.
%We need to go further in designing the workflow of the layer-based computation orchestrating strategy to ensure that the CPU and GPU are fully utilized.

%Second, the new computational framework has a more complex execution process, which makes the simple pipeline no longer applicable. As shown in Figure \ref{fig: pipeline design} (a), in the synchronous model, the beginning of training in the current batch must wait until the conclusion of CPU backward in the previous batch. 
%Second, although we reduce the total number of single transfers, we introduce an additional data transfer because of the bottom layer backward in the CPU.
%Second, the new computational framework has a more complex execution process, which makes the simple pipeline no longer applicable. As shown in Figure \ref{fig: pipeline design} (a), in the synchronous model, the beginning of training in the current batch must wait until the conclusion of CPU backward in the previous batch. This can result in a huge waste of resources. Even if we introduce asynchronous to the full pipeline, we still face the situation where the GPU performs sampling and training simultaneously. 

%Finally, although we reduce the total number of single transfers, we introduce an additional data transfer because of the bottom layer backward in the CPU. We need to go further in designing the workflow of the GPU-centric, CPU-assisted computing framework to ensure that the CPU and GPU are fully utilized while minimizing the number of transfers.


%算法图
%workflow图
%In GNN training, the various steps are always performed in a specific order, and each step is based on the results of the previous one. Simple and fixed work assignments may not always result in the best performance. High CPU and GPU utilization must be maintained simultaneously in an efficient heterogeneous system to prevent CPU or GPU idle.



%todolist : CPU GPU算力抽象，CPU GPU 任务抽象
%（1）引入了更多算力。但在任务均衡的情况下，不会导致GPU等待。（2）假设。我们抽象整个GNN训练任务的任务量为T，T包括Sampling，data loading，training。 如第三节分析，过度的使用GPU资源所导致的GPU contention。 由于可用的cache比例及batchsize大小降低，会使得总的任务量 T 增加。

%但是简单的使用3，并不能时刻保证最佳性能。我们知道，针对复杂的GNN模型以及不同的CPU-GPU架构，任务量T与算力C是不固定的，固定的GNN任务分配可能会打破CPU与GPU之间的平衡。

%3.解决问题：我们的方法  
%第一， 我们可以抽象CPU负责的任务量为T1, 算力为C1; GPU负责的任务量为T2,算力为C2。 传输方向若为CPU-GPU则归为T2, 反之归为T1。 T = T1 + T2。 

%倘若满足一个条件，即在当前工作负载与CPU-GPU架构下，T1/C1 约等于 T2/C2，系统将会得到最佳性能。

%为此，我们解耦并分析GNN的每个阶段的工作量及所需算力，并针对不同的GNN工作负载以及CPU-GPU异构算力，制定算力-任务量平衡 且 任务量最小化（CPU GPU 

%用流水线引入新计算框架下的问题， 接着为了full pipeline 也就是充分利用CPU GPU，引入 1. 要full neibor bottom 训练 2. 要额外考虑 bottom在CPU的执行时间与

%CPU按访问频率执行全部顶点的聚合计算，聚合全部邻居。Sample先执行数轮，等待CPU中顶点完成部分计算，随着CPU聚合完成的顶点增多，sample data loading training 的任务量都会逐渐降低。 sampling 对于已经聚合完成的点，在bottom layer不再进行采样。 data loading 对于已经完成聚合顶点，传输embedding 对于未完成的顶点，按照采样结果传输feature。 GPU中的training阶段也只计算未被CPU聚合的顶点，根据sample的结果聚合embedding。 

% \vspace{-0.05in}
% \subsection{Hot Vertices Embedding Reusing}
% \zyf{do we need such an algorithm? If so, the current shape is not ready, should be heavily rewritten.}
% \begin{algorithm}[t]
% 	\caption{Prepare hot vertices queue} \small 
% 	\label{alg:prepare}  
% 	\begin{algorithmic}[1]
% 		\Require training vertices set $T$, vertex sampling frequency matrix $S_{V}$, vertex hotness vector $A_{V}$, hot vertices ratio $R$.
% 		\Ensure hot vertices queue $H_{V}$. 
%         \For{each $v$ in $T$}
%         \State PreSample($S_{V}$,$v$); \textcolor{gray}{//Record the vertex sampling frequency in $S_{V}$ as hotness.}
% 		\EndFor
%         % \State $A_{V}$ = $H_{V}.columnSum()$; \textcolor{gray}{//Accumulate vertex sampling frequency in $A_{V}$ as hotness.}
%         \State $HotnessRank$ = Sort($S_{V}$); \textcolor{gray}{//Sort vertices by hotness.}
%         \For{$i = 0$ to $HotnessRank.size() * R$}
%         \State $H_{V}[i]$ = $HotnessRank[i]$;
% 		\EndFor
% 	\end{algorithmic}  
% \end{algorithm}


% \zyf{too much information in the figure. hard to follow. If you cannot draw an illustrative figure, I prefer to remove it.}
\begin{figure}
  \centering
  \includegraphics[width=8cm,page={1}]{figures/computing_framework.pdf}
     \vspace{-0.4cm}
  \caption{An illustrative of CPU and GPU workload in naive layer-based task orchestrating and hotness-aware layer-based task orchestrating.}
   \vspace{-0.2cm}
  \label{fig: computing framework}
\end{figure}




% \begin{algorithm}[t]
% 	\caption{Workflow in GPU} \small 
% 	\label{alg:workflow}  
% 	\begin{algorithmic}[1]
% 		\Require $G(V,E)$, $L$, $\{\textbf{h}^{0}_v\mid v\in V\}$, $\{\mathbb{L}_v\mid v\in V_{\mathbb{L}}\}$, initial model parameters $W^{(l)}$ for each layer $l$
% 		\Ensure updated parameters $W^{(l)}\}$ for each layer $l$
% 		\State initialize vector $Emebedding\_{GPU}$, $Hotness$, $AGGMap$ on GPU.
        
%         \For{$l$ = $1$ to $L - 1$}
% 		\For{each $V_{i}$ in $l$} in parallel
% 		\State $nbrs = uniform\_select(V_{i}, fanout)$;
%         \State $Sample\_{GPU}.push(numbers)$;
% 		\EndFor
%         \EndFor
%         \State $L\_{unhot} = Remove\_hotness(L, Hotness)$; 
%         \For{each $V_{i}$ in $L\_{unhot}$} in parallel \textcolor{gray}{//bottom layer}
%         \State $nbrs = uniform\_select(V_{i}, fanout)$;
%         \State $Sample\_{GPU}.push(numbers)$;
% 		\EndFor
%         \For{each $V_{i}$ in $L$} in parallel
%         \If{$AGGFlag[V_{i}] == 1$}
%         \State copy $Embedding\_{CPU}[AGGMap[V_{i}]]$ to GPU
%         \ElsIf{$AGGFlag[V_{i}] == 0$}
%         \State copy feature of $Sample\_{CPU}[V_{i}]$
%         \EndIf
%         \EndFor
%         \For{each $V_{i}$ in $L$} in parallel
%         \If{$AGGFlag[V_{i}] == 1$}
%         \State $Embedding\_L[V_{i}] = Embedding\_{CPU}[AGGMap[V_{i}]]$;
%         \ElsIf{$AGGFlag[V_{i}] == 0$}
%         \State $Embedding\_L[V_{i}] = forward(\textbf{h}^{0}_v, W, Sample\_{CPU}[V_{i}])$;
%         \State $Emebedding\_{GPU}.push(Embedding\_L[V_{i}])$;
%         \State $AGGMap[V_{i}] = Emebedding\_{GPU}.size()$;
%         \State $AGGFlag[V_{i}] = 2$; \textcolor{gray}{//caching in the GPU}
%         \ElsIf{$AGGFlag[V_{i}] == 2$}
%         \State $Embedding\_L[V_{i}] = Embedding\_{GPU}[AGGMap[V_{i}]]$;
%         \EndIf
%         \EndFor
%         \State forward and backward of L - 1 layer to layer 1
%         \State copy Grad of Embedding\_{CPU} to CPU
%         \State backward of the bottom layer 
% 	\end{algorithmic}  
% \end{algorithm}
%Some works use machine learning \cite{mlbalance_2019_JPDC} to keep load balancing in CPU-GPU heterogeneous systems. \system takes a simpler but equally effective approach. 
%Based on the layer-based computation orchestrating strategy, a fixed computation tasks division by layer is insufficient to prevent the CPU from growing into a new bottleneck. 

\subsubsection{Hotness-aware Embedding Reusing}
As shown in Figure \ref{fig: computing framework} (a), executing a complete bottom layer in the CPU may cause the CPU processing a new bottleneck because the computation volume of the entire layer is high.
% In other words, relying on the GPU as the primary computing power, we cannot afford to let it frequently wait for the CPU's results. This situation can lead to significant resource wastage.
Previous studies on caching policies have demonstrated that a subset of vertices is frequently accessed during sample-based GNN training \cite{PAGRAPH_SOCC_2020, DATATIRING_VLDB_2022, GNNlab_EUROSYS_2022}, which is commonly referred to as the "hot vertices". By caching the features of these vertices in the GPU, existing works can avoid repeatedly loading them from the host memory, thereby reducing CPU-GPU communications. 
% \zyf{logical gap, why caching can reduce transfer volume?}
\system follows similar ideas to design a hotness-aware computation offloading method, which computes only the embeddings for hot vertices in the CPU and allows the GPU to reuse these embeddings between batches within bounded staleness \cite{SANCUS_VLDB_2022, ST_ICML_2018,ssp_2013_nips}, our task orchestrating method can substantially decrease the CPU workload and CPU-GPU communication.

The main idea of reusing embeddings within bounded staleness is to maintain the historical embedding $\tilde{h}_i^{(l)}$ for exact embedding ${h}_i^{(l)}$ as an affordable approximation with a given bound \cite{ssp_2013_nips, ST_ICML_2018, SANCUS_VLDB_2022}. Bounded staleness expects $\tilde{h}_i^{(l)}$ and ${h}_i^{(l)}$ to be similar if the model weights do not change too fast during the training. In \system, we use the number of model parameter updates, i.e., the number of batches, as the bound. Specifically, if the bound is $N$, a recently computed embedding can be reused within the subsequent $N$ batches.
Using historical embeddings avoids the need for aggregating neighbor features and the associated backward pass.
This not only reduces the amount of raw features to load but also minimizes the CPU workload.
We will provide the theoretical analysis of the convergence guarantee for \system in Section \ref{prove}. 
%Historical embeddings with parameter versions within the bounds allowed can be reused without recalculation. 
% Each time when ${h}_i^{(l)}$ is computed, we update $\tilde{h}_i^{(l)}$ with ${h}_i^{(l)}$.

% But honestly, I don't think non-expert readers can understand bounded staleness after reading your writing. I don't even know what you talk about historical embeddings with bounded staleness.}

%Similarly, if the hot vertices can be aggregated and updated in the CPU and let GPU reuse the historical embedding between batches within bounded staleness, our computation orchestrating strategy can significantly reduce the computation amount in the CPU. 

%The training task in the CPU shifts from computing the whole bottom layer for each batch to only adding the bottom layer training of the hot vertices. 
%Only the bottom layer training of hot vertices should be performed in the CPU to help the GPU share the training tasks with less computation. 
% Then, we sort the vertices based on their hotness and allocate the higher hotness vertices into $H_{V}$ according to the hot vertices rate $R$ (lines 4-6).
%determine the hotness of vertices in the bottom layer and places them into the hot vertices queue from high hotness to low hotness. The length of the hot vertices queue depends on the given hot vertices rate, i.e., the percentage of hot vertices to current batch bottom vertices.
%and the version of the embedding is updated according to the latest parameters. 

%we introduce an additional data transfer because of the bottom layer backward in the CPU. In the backward of the bottom layer, the GPU needs to transfer back the gradient of the embedding to the CPU and synchronize the gradient of the model parameters with the CPU. This process will cause the GPU to wait, which has a negative impact on the overall performance.
%We need to monitor the version difference (staleness) between the historical embedding and latest model parameter, and adjust hot vertices rate to control the historical embedding update period in the CPU.
%In addition, adjusting the hot vertices rate allows us to control the computational load on the CPU. 

%thereby helping the GPU share the training workload with minimal computational requirements.

%Following this idea, the CPU should handle the training of the bottom layer limited to the hot vertices. This allows for a higher reuse probability with minimal CPU load. 
Before the end-to-end training starts, \system requires a preprocessing stage to determine the hot vertices and set up a hot vertices queue for the processing. We employ the pre-sampling method of GNNLab \cite{GNNlab_EUROSYS_2022} to sample multi-hop neighbor multiple times for each training vertex and record the accessed frequencies (i.e., hotness) of the vertices. Then, we sort the vertices based on the hotness and determine the hot vertices used for CPU computation according to the hot vertex ratio.
 
Figure \ref{fig: computing framework} (b) illustrates an example of the hotness-aware layer-based task orchestrating method. During training, for hot vertices, the CPU samples them in one hop and computes their embeddings, which will be fetched and reused by the GPU training process to reduce computation and communication. For the cold vertices in the bottom layer, the GPU pulls the features of their vertices and computes them locally. 
To guarantee model accuracy, it is essential to control the version of reused embeddings within a specified range of batches during training, which is known as a concept of bounded staleness \cite{ssp_2013_nips}. To achieve this, we propose a super-batch pipelined training method that achieves bounded staleness among batches by packing each consecutive $N$ batches into a super-batch and controlling the embedding reusing among super-batches during computation. In this way, each embedding of the hot vertex is updated at least once within $N$ batches. A detailed discussion is provided in Section \ref{sec4.2}. Additionally, we offer theoretical analysis in Section 4.3 to demonstrate the correctness of our design. By adopting this approach, \system effectively offloads computations to the GPU and prevents CPU computation from becoming a performance bottleneck.
%achieve bounded staleness among batches,

%VR-GCN主要提供使用历史嵌入的理论证明，证明了能收敛，没有特殊的bounded staleness策略，只是以epoch为bound，纯CPU，算法论文。
%GAS为计算图中所有顶点使用历史嵌入，减少训练所需的GPU内存，并且把所有点的嵌入保存在CPU内存，在需要的时候再传到GPU，没有特殊的bounded staleness策略，只是以epoch为bound。CPU-GPU异构
%Refresh在GPU内存中缓存历史嵌入，减少计算量和传输量，根据设计的bound策略判断历史嵌入是重用还是重计算。在使用历史嵌入前会先判断历史嵌入的参数落后多少轮，再看梯度大小，两个bound条件都满足才会继续使用。CPU-GPU异构
%SANCUS在分布式环境中为边界点使用历史嵌入，减少通信，提供bound策略。
%然而这些系统与NeutronOrch有如下区别，1）NeutronOrch selectively computes HEs for frequently accessed vertices on the CPU and combines HE version control with CPU-GPU pipelining techniques to ensure bounded staleness

%CPU-side updates and GPU-side reuse are concurrently executed while limiting the version gap within two adjacent super-batches.

%The existing HE solutions \cite{ST_ICML_2018,SANCUS_VLDB_2022,GNNautoscale_icml_2021,refresh_arxiv_2023} either target entirely different training paradigms \cite{SANCUS_VLDB_2022}, reuse HE for all vertices \cite{ST_ICML_2018,refresh_arxiv_2023,GNNautoscale_icml_2021}, or do not strictly guarantee bounded staleness \cite{ST_ICML_2018, GNNautoscale_icml_2021}, thus cannot address our problems.
%感觉还差点 todo。。。
% \system uses the CPU to periodically recompute the historical embeddings without incurring the overhead and latency caused by staleness HE checking.

%The existing HE solutions \cite{ST_ICML_2018,SANCUS_VLDB_2022,GNNautoscale_icml_2021,refresh_arxiv_2023} cannot address our problems.

%现有的HE reuse solutions, 要么针对完全不同的训练模型，要么使用general的HE策略来重用所有顶点的结果, 要么，无法保证bounded staleness, 因此不能应用到\system 关注的GPU-CPU异构加速场景中。

%这些系统侧重于优化专门在 GPU 上部署训练的场景，当面对异构环境时，不能充分利用异构资源。相反，NeutronOrch侧重于在异构环境中使用 HE，并探索其在提高异构资源利用率方面的独特优势。1）NeutronOrch使用CPU周期性地重新计算历史嵌入，不在GPU中产生因为历史嵌入陈旧性检查带来的开销和延迟，保证GPU端可以高效训练。2）NeutronOrch 将 HE 版本控制与 CPU-GPU 流水线技术相结合，以确保有界的滞后性（第 4.3 节）。CPU 端更新和 GPU 端重用同时执行，并把版本差距限制在相邻两个super-batch内。3）NeutronOrch会根据CPU与GPU的资源情况，控制分配给CPU计算和GPU特征缓存的热顶点比例，以确保在不同的异构环境中都可以最小化CPU-GPU传输
%However, \system differs from these frameworks in the recompute and reuse of HE. 
% In a word, \system focuses on using HE in a CPU-GPU environment and explores its unique advantages for improving heterogeneous resource utilization.
% \system selectively computes HEs for frequently accessed vertices on the CPU and combines HE version control with CPU-GPU pipelining techniques to ensure bounded staleness (Section 4.3). 
% These frameworks focus on optimizing scenarios where computation is exclusively deployed on GPUs. In contrast, our work extends HE reuse to the CPU-GPU environment, achieving both high performance and scalability.
% NeutronOrch selectively computes HEs for frequently accessed vertices on the CPU. This allows for a higher reuse probability with minimal CPU load. Furthermore, by transferring and reusing HEs, we convert expensive feature-based transfer into cheaper embedding-based transfer, which minimizes the transfer and reduces the GPU memory consumption for caching. 
% In addition, we propose a super-batch pipelined training, which combines HE version control with CPU-GPU pipelining techniques to ensure bounded staleness (Section 4.3). CPU-side updates and GPU-side reuse are concurrently executed while limiting the version gap within two adjacent super-batches.

\subsubsection{Hybrid Hot Vertices Processing for high-performance GPU servers}
%基于层的任务编排方式在考虑充分利用CPU与GPU算力的同时也要兼顾最小化数据传输。在第三章我们讨论了训练数据与缓存数据对GPU资源的争用可能导致无法提供足够的feature cache ratio。为此，NeutronOrch使用闲置的CPU算力为热点计算嵌入并在有界陈旧性允许下缓存在GPU重复使用。因为热点嵌入通常比热点feature具有更小的维度，NeutronOrch减少了数据传输的同时也减少了热点缓存所需内存。
%在GPU资源有限的情况下，使用CPU算力带来的帮助是有价值的。然而，当GPU内的资源压力不大，或者有充足的 GPU 资源（如在多 GPU 环境中）时，CPU 提供的帮助就变得相对有限。当给定热点比例较大的情况下，CPU计算可能成为整体性能瓶颈。因此我们提出了一种Embedding-Feature-Hybrid Cache Policy 去适应不同的计算任务与硬件环境，以提供稳定的传输缩减。对于给定热点比例，一部分热点将由CPU负责计算嵌入，一部分热点的特征会被静态缓存在GPU。具体来说，当GPU资源有限，CPU会负责更多的热点嵌入计算，当GPU资源充足，CPU会减少热点嵌入的计算比例，同时增加热点特征的缓存比例。cpu和gpu的强弱关系，决定数据传输缩减是依赖cpu算力，还是gpu资源。CPU所能承担的热点比例会在super-batch pipeline的过程中通过监控CPU和GPU的计算时间自适应的调整（see more details in 4.3）。
%相比于传统的静态特征缓存策略，Embedding-Feature-Hybrid Cache Policy在减少CPU-GPU通信的同时对GPU内存的要求更低
%\notecolor{
% The hotness-aware layer-based task orchestrating method aims to maximize the utilization of both CPU and GPU resources while minimizing CPU-GPU communication. 
% In Section 3, we discuss the potential contention for GPU resources between training data and cache data, which may lead to an insufficient feature cache ratio (i.e., hot vertices ratio). To address this challenge, \system leverages idle CPU computing power to calculate hot vertex embeddings and caches them on the GPU for reuse within bounded staleness. As embeddings generally have smaller dimensions than features, \system reduces both CPU-GPU communication and the required memory for caching.
%}
% Varying heterogeneous resource relationships may limit the embedding that the CPU can provide.
%Orchestrating tasks across CPUs and GPUs requires adjusting workloads based on available heterogeneous resources.
%变化异构资源关系可能限制CPU可以提供的嵌入
%在CPU与GPU间调度任务需要适应不同的异构资源关系
% \notecolor{
% Orchestrating tasks across CPUs and GPUs necessitates consideration of different CPU-GPU resource configurations.
% When GPU resources are constrained, leveraging CPU resources provides benefits for alleviating GPU burden. However, when there are sufficient GPU resources available, such as in a single-CPU-multi-GPU environment, the contribution of the CPU resources becomes relatively restricted. 
% To address this problem, we propose a hybrid computing and GPU caching policy for hot vertices to adapt to diverse heterogeneous environments by controlling the proportion of hot vertices assigned to CPU computation and GPU feature caching.
% When the number of GPUs is limited, \system assigns hot vertices to CPU computation, aiming to alleviate the pressure on GPU memory and computation. 
% When GPU resources are sufficient, \system will assign some of the hot vertices to GPU utilizing feature caching to alleviate CPU computation burdens and fully leveraging GPU resources.%e3e3e3
% %allocates certain hot vertices to the GPU, utilizing feature caching to alleviate CPU computational burdens and make optimal use of GPU computing resources.
% %我们通过监测CPU可工作时间和GPU可用内存确定分配给CPU计算和GPU缓存的热点数量。CPU可工作时间会在Super-batch流水线调度的过程中确定，GPU可用内存会根据训练所需内存和GPU总内存来确定 CUDA Runtime API
% We determine the allocation of hot vertices for CPU computation and GPU caching by monitoring the available CPU working time and GPU memory. The CPU working time is determined during the scheduling process of the super-batch pipelining (Section 4.2), while GPU available memory is determined based on the training memory consumption and total GPU memory, which can be achieved through the CUDA runtime.
% Compared to only computing embedding on the CPU, the hot vertice hybrid processing cache policy maximizes the utilization of both CPU and GPU resources while further reducing CPU-GPU communication.}
Orchestrating tasks across CPUs and GPUs balances CPU and GPU resource utilization and reduces CPU-GPU data communication by offloading the computation of hot vertices to the CPU. 
% However, in cases where the computing server is equipped with multiple GPUs but has limited CPU computing power, such as in a single-CPU-multi-GPU environment, the contribution of using CPU computation decreases because the improvement may not surpass the benefit brought about by the increased GPU resources. 
% 这里不是GPU计算算力的提高吧？感觉像是GPU内存的增加？或者GPU计算资源的充裕？（下面这句话用的是 GPU resources）
To address this problem, we propose a hybrid processing policy for hot vertices that further balances CPU and GPU utilization by assigning hot vertices to both CPU computation and GPU feature caching. When GPU resources significantly outnumber CPU resources, \system adaptively assigns hot vertices to the GPU, utilizing feature caching to avoid computing them on the CPUs. 
Specifically, during execution, \system monitors the time elapsed on GPU idleness caused by CPU computation and the remaining available GPU memory. 
It adjusts the allocation of hot vertices to GPU and CPU through a worklist, ensuring that GPU memory does not overly subscribe while minimizing GPU idle time. 
Note that \system stops assigning hot vertices to GPU when GPU memory is exhausted or the GPU idle time reaches zero, as appropriately utilizing CPU computation can effectively reduce data transfers. 
Compared to computing embedding of all hot vertices on the CPU, our hybrid processing optimization can further improve heterogeneous resource utilization.


\subsubsection{Discussion of \system and other HE-based GNN training frameworks.}
Recent studies \cite{ST_ICML_2018,SANCUS_VLDB_2022,GNNautoscale_icml_2021,refresh_arxiv_2023} have shown the benefits of using historical embedding (HE) for GNN training acceleration. VR-GCN \cite{ST_ICML_2018} uses HE in GNN training to reduce the number of vertices to sample. GNNAutoScale \cite{GNNautoscale_icml_2021} reuses HE for all vertices to reduce training memory consumption on the GPU.
Refresh \cite{refresh_arxiv_2023} caches HE in GPU memory to reduce the CPU-GPU feature communication and GPU training time. SANCUS \cite{SANCUS_VLDB_2022} reuses HE for boundary vertices to reduce communication in distributed full-graph training. 
However, these frameworks focus on scenarios where training is exclusively deployed on GPUs \cite{ST_ICML_2018,SANCUS_VLDB_2022,GNNautoscale_icml_2021,refresh_arxiv_2023}, and either target different training paradigms \cite{SANCUS_VLDB_2022}, uniformly reuse HE for all vertices \cite{ST_ICML_2018,refresh_arxiv_2023,GNNautoscale_icml_2021} which does not consider the impact of vertex hotness to the resource utilization, or do not strictly guarantee bounded staleness \cite{ST_ICML_2018, GNNautoscale_icml_2021} among batches, which is not applicable to \system that focuses on balancing GPU-CPU heterogeneous resource utilization and guaranteeing bounded accuracy. 
To achieve this goal, \system adopts several critical designs.
1) \system selectively computes HEs for the frequently accessed vertices and maximizes CPU-GPU resource utilization by adjusting the proportion of hot vertices assigned to CPU computation and GPU feature cache (Section 4.1.3).
2) We conduct a theoretical analysis on the convergence of \system's HE to demonstrate its correctness (Section 4.3). Furthermore, \system provides a super-batch pipelining method, combining HE version control with pipelined task scheduling to achieve efficient and strict bounded staleness (Section 4.2). 


%我们通过监控CPU可用工作时长和可用GPU内存来确定热顶点的分配。其中CPU可用工作时长是指在训练pileline中,CPU可以进行嵌入计算而不会造成等待的时间槽，其在可以在我们提出的Super-batch流水线（见第4.2节）调度中被自然地计算。而GPU可用内存则是通过CUDA运来进行监控。
%The CPU computation and system idle time is determined through the scheduler of the super-batch pipelining (Section 4.2), while GPU available memory is determined based on the CUDA runtime.

% Assuming that the number of vertices that can be processed by our hybrid caching policy is $T$, we have $T$ = $T_{C} + T_{G}$, where $T_{C}$ denotes the number of vertices for which the embedding is computed within the CPU and $T_{G}$ denotes the number of vertices for which features are cached in the GPU. 
% %我们会优先为$T_{C}$分配高热度顶点，因为缓存嵌入节省更多传输同时需要更少内存
% We will prioritize the allocation of high-hotness vertices for $T_{C}$ because cache embedding saves more transfers while requiring less GPU memory.
% To determine $T_{C}$, we need to determine the maximum workload that the CPU can support while avoiding GPU waits. To achieve this goal, we make CPU computation fully concurrent with GPU computation by super-batch pipelining (Section 4.2), and adjust the CPU workload so that the CPU and GPU working time within each super-batch is close to each other.  
% Then, we determine $T_{G}$ according to available GPU memory. 
% %To achieve this, we leverage the fact that the memory consumption is similar across training iterations.


%The CPU workload (the number of vertices using caching embedding) is adaptively adjusted during the super-batch pipelining by monitoring CPU and GPU computation times (see more details in Section 4.3).
% Specifically, when GPU resources are scarce, the CPU handles a larger proportion of hot vertice embedding computations. 
% Conversely, when GPU resources are abundant, the CPU limits the ratio of hot vertice embedding computations while increasing the ratio of hot vertice feature caching.
% We will maximize the embedding cache ratio while avoiding GPU waits.
% Then, for the other hot vertices, their feature will be statically cached on the GPU according to available GPU memory.  
% The balance between CPU and GPU capabilities determines whether the CPU-GPU communication reduction relies on CPU resources or GPU resources.
% For a given hot vertice ratio, we determine the embedding caching ratio and feature caching ratio based on available heterogeneous resources.
% \system first determines the embedding caching ratio according to available CPU resources. 
%a portion of hot vertices is computed by the CPU for embedding, while features of other hot vertices are statically cached on the GPU.
%当给定热点比例，
%cpu和gpu的强弱关系，决定我数据传输缩减是依赖cpu算力，还是gpu内存
%gpu没内存做缓存这说明gpu可能也比较弱，那我cpu算力可以提供给你热点的历史嵌入，cpu算力在这时很关键。gpu变多或者gpu很强，我为了防止cpu瓶颈，我减少cpu的历史嵌入计算，但我同样受益与变强的gpu，我可以缓存更多feature
%Therefore, the CPU must recompute embeddings for the hot vertices after each epoch finishes.

%The CPU's workload can be managed with the help of this method. This means that we can change the hot vertices rate to adjust the computation amount that needs to be performed in the CPU. With this method, we make the computation volume in the CPU manageable, preventing CPU training from becoming an overall performance bottleneck. 


%The CPU will update the vertex embedding in order of hotness with the latest version parameters and loop through this process for every batch.
%The difference is that SANCUS is full graph training. The SANCUS's staleness check exists between epochs, while we need to adjust the check frequency to between batches. SANCUS has three bounded staleness methods, and we choose Epoch-Fixed Embedding Staleness, which in \system is Batch-Fixed Embedding Staleness. Using the simple and general bounded staleness check method is enough because we only involve the historical embedding of the bottom layer. 



%为了适应CPU-GPU异构环境，batch-fixed方法需要进一步改进。因为CPU中的热度顶点并不属于同一个batch，在一个batch中，往往只涉及其中一部分顶点。CPU作为热度顶点的前向传播地点是非常适合的，但是在反向传播过程中，CPU可能会成为性能瓶颈。在bottom layer的反向传播中，GPU需要传回梯度至CPU，并与CPU同步对模型参数的梯度。这个过程会造成GPU 等待，这是我们不希望的。为此，我们决定去除CPU中的反向传播过程，所有反向传播转至GPU执行。在data loading过程中，在传输embedding与feature的同时，传输反向传播所需数据，具体地，CPU需要传输与模型参数相乘之前的聚合结果至GPU。注意，与传统方法相比，这并不会造成总传输量增加。因为从传输顶点规模来看，传输聚合后的顶点特征减少了需要传输的顶点规模。

%讨论这种方式下的传输，节省反向传回与等待开销
%这块引起的额外传输和前面的节省传输有点冲突
%The batch-fixed approach requires a further design to adopt the CPU-GPU heterogeneous environment. The forward of hot vertices can be executed simply in order of hotness and cache their embedding in the CPU. However, the hot vertices in the CPU do not belong to the same batch, and usually, only part of them are pulled by the GPU in a set. Due to this, the backward of hot vertices is fragmented and irregular. 


%If the GPU finds that the current historical embedding is stale, the gradient is transferred back to the CPU backward to update the parameters and recompute the embedding.

%we show the algorithm of the caching strategy, where $Embedding\_CPU$ keeps the history embedding, $Hotness$ keeps the set of high hotness vertices, $AGGflag$ keeps the flag of whether the aggregation of high hotness vertices is finished, and $AGGMap$ keeps the mapping relationship between vertices and historical embedding. First, the CPU will sample all the vertices in $Hotness$ in one hop. Second, the results are aggregated and updated based on the sampling results to get the embedding and saved in the $Embedding\_CPU$. Second, the CPU will calculate the embedding of all vertices in $hotness$ and cache in the CPU or GPU. Finally, the GPU will use the embedding provided by the CPU in subsequent training and check the staleness bound of the historical embedding during epochs. If the GPU finds that the current historical embedding is stale, the gradient is transferred back to the CPU for backward to update the parameters and recompute the embedding.

% To solve this issue, \system defined the device load as the time required to execute all the applications assigned to it. We abstract the entire GNN training task as $T$. $T$ includes sampling, data loading, and training. In addition, we outline the task of CPU as $T_{C}$ and computing power as $P_{C}$; abstract the task volume of GPU as $T_{G}$ and computing power as $P_{G}$. To determine $T$, we assume that the GPU has a computing power of 1. Sampling and training are executed in the GPU at the beginning of the training phase.

%需要一个算法图
%与传统cache方法相同，对顶点排序，并选出前10%的热度点。
%开始训练，CPU中执行前 10%热度点 的采样，仅执行一跳。
%GPU正常执行采样算法，但是，对于bottom layer中的热度点，跳过采样过程。
%GPU多次重复采样过程，设次数为k， k = 4
%CPU完成 20%热度点采样后，进行热度点的bottom layer 聚合， 并在一块CPU GPU同时可见的锁页内存中维护一个 热度顶点 聚合状态完成数组，标志当前顶点是否完成 聚合 以及 聚合位置。
%在GPU第一次采样结束后，开始传输，检查聚合状态完成数组，若当前点未完成聚合，则根据CPU中的采样结果，传输对应feature，将聚合状态设为完成，聚合位置设为GPU，并将聚合结果 embedding缓存在GPU。
%对于已完成聚合热度顶点，判断其聚合位置，若聚合位置在CPU，直接传输其embedding；若聚合位置在GPU 直接获取其embedding
%在 k/2次 训练结束后，将聚合位置为CPU的顶点累积梯度传回，至CPU进行反向传播
%在 k次 训练结束后，将聚合位置为GPU的顶点做反向传播。
%平衡检查 
%上界 1：CPU训练不会超过 K-1次的data loading。 2. CPU sample 不会超过 传回CPU的梯度 data loading 
%下界 1：CPU训练不会少于K/2次的data loading。

% \zyf{why Gather on PCIe? Why sample on both CPU and GPU?How does (a) map to the 4.2 approach? More efforts to elaborate. The first character should be lowercase}

\begin{figure*}
  \centering
  \includegraphics[width=0.9\textwidth,page={1}]{figures/super-batch.pdf}
     \vspace{-0.3cm}
  \caption{An illustrative example of super-batch pipelined training.}
   \vspace{-0.4cm}
  \label{fig:pipeline design}
\end{figure*}

% \zyf{I think the problem is not well-motivated. I don't understand what the problem is. Would it be better to formalize the problem?}

% We present a comprehensive exploration of the super-batch pipeline design, starting with an overview, followed by a detailed examination of each stage.

% \vspace{-0.05in}
\subsection{Super-batch Pipelined Training}
\label{sec4.2}

%我们讨论了由CPU和GPU交错计算所导致的数据依赖性，并提出。。。

In this section, we discuss the data dependencies arising from the integration of CPU and GPU computations and propose a super-batch pipelined training method to manage them efficiently. 
%感觉这句没啥问题啊

\subsubsection{Data Dependencies.} Seamlessly overlapping tasks across diverse computing resources through pipelining is essential to achieve high performance on heterogeneous systems \cite{PYTDIRECT_VLDB_2021,TurboGNN_TOC_2023,DSP_PPOPP_2023}. However, implementing high-performance task pipelining in \system is a non-trivial task because the layer-based task orchestrating creates cross-layer data dependencies between the CPU and GPU training process. 
Figure \ref{fig:pipeline design} (a) illustrates data dependencies in \system when setting the stale bound to one epoch \cite{ST_ICML_2018,GNNautoscale_icml_2021}.
The CPU updates the embeddings for hot vertices when it reaches the stale bound, and the training on the GPU must wait for the CPU to complete the embedding computation for hot vertices. Consequently, the CPU and GPU need to fall back to sequential computation, resulting in inefficient resource utilization.

%In addition, the model weights are updated after one epoch finishes. Setting one epoch as stale bound makes the embeddings for hot vertices too stale for batches that start training later. 

%Past works \cite{SANCUS_VLDB_2022,Marius_OSDI_2021,ssp_2013_nips} have shown that introducing asynchronous pipeline training and using stale embeddings within certain bound can speed up training and maintain accuracy.

% Therefore, selecting hot vertices from a limited number of upcoming batches is more reasonable and efficient than selecting hot vertices from the entire training set.
% \zyf{I don't know what the intuition is to merge n batches.} 
%In sample-based GNN training, each epoch contains many batches. As shown in Figure \ref{fig:pipeline design} (a), computing embeddings for hot vertices for all batches in an epoch is time-consuming and results in GPU resources idle.


\subsubsection{Pipeline Design.} 
To address the above issues, we propose a super-batch pipelined training method.
Firstly, the embeddings update for hot vertices in the CPU is partitioned into multiple sub-computing tasks, each of which computes the embeddings for hot vertices for subsequent multiple batches, which provides more opportunities for pipelined designs. Secondly, we limit the stale bound to several batches instead of one epoch because more tightly controlled bounded staleness can help ensure model accuracy.


Before training starts, \system combines $n$ batches into a single super-batch. Then, we select a hot vertices queue for each super-batch. As shown in Figure \ref{fig:pipeline design} (b), this is an illustrative example of a super-batch pipelined training when $n = 4$. During the pipelined training, the CPU computes embeddings for hot vertices for the next super-batch. In each super-batch, the GPU computes $n$ batches of training for a given batch size, sharing the historical embeddings provided by the CPU from the last super-batch.
%与每一次为一个epoch中所有热点更新嵌入相比，每次只为下一个super-batch更新嵌入的时间开销更小，同时减小了历史嵌入与真实嵌入之间的版本差距。
Compared to updating the embedding for all hot vertices, updating the embedding for only the next super-batch has less computation overhead and reduces the version gap between historical embeddings and exact embeddings.
% The super-batch pipeline has a shorter CPU embedding update period than selecting hot vertices from all training vertices, effectively maintaining bounded staleness.
Our pipeline design consists of four stages, each serving a specific purpose. We now elaborate on these individual stages.

% \zyf{may need rewrite, hard to follow.}
\Paragraph{Stage 1: Sampling.} 
The sampling tasks are divided between the CPU and GPU. Firstly, the CPU samples the hot vertices and generates a one-hop subgraph for bottom-layer training. Next, the GPU initiates n-hop sampling, and upon reaching the bottom layer, it skips the vertices previously sampled by the CPU. Meanwhile, the GPU completes $n$ rounds of sampling before $n$ training rounds to mitigate potential resource contention between the sampling and training kernels. 
%Moreover, CPU sampling is advanced at the start of the pipeline to make it easier for the GPU to benefit from the results. %The GPU will perform $K$ rounds of sampling, which will help the CPU finish training more hot vertices. 
% and incorporates the CPU's sampling results


\Paragraph{Stage 2: Embedding computation for hot vertices.} The CPU is responsible for updating the embeddings of the hot vertices intended for the subsequent super-batch within each super-batch. The CPU conducts training based on the hotness order of the vertices and utilizes the most recent version of parameters. 
% Once the CPU completes the sampling of hot vertices for the next super-batch, it promptly performs aggregation and updates on these vertices to obtain embeddings. 

%The CPU also keeps an array in pinned memory visible to both the CPU and GPU that records whether the hot vertices have completed bottom layer training and its version.
%In the backward, the CPU updates the parameters of the bottom layer based on the gradients transferred back from the GPU. As shown in Figure\ref{fig:pipeline design}, in order not to cause the GPU to wait for the backward phase of the CPU, the gradient will be transferred back at $k - 1$ batches and the model parameters in the bottom layer are updated once. 

\Paragraph{Stage 3: Feature gathering.} 
During the gathering step, the embeddings of hot vertices and the features of cold vertices are transferred. After being transferred, the embeddings of the hot vertices will be cached in the GPU for use in other batches within a super-batch. 
% The transfer object is determined by the CPU's sampling result and the hot vertices queue.
% The features corresponding to the non-hot vertices in the bottom layer are transferred based on their sampled vertices. 
%Additionally, the data necessary for the backward computation of hot vertices is transferred to the GPU along with the embeddings, ensuring version consistency.

%For the non-hot vertices in the bottom layer, the corresponding features are transferred according to their sampled vertices. The embeddings are transferred directly to the hot vertices, and cache the embeddings in the GPU for other batches in the super-batch. In addition, the data required for hot vertices backward is transferred to the GPU along with embedding, and the version is kept consistent. 
%In order to fully utilize PCIe and GPU resources, the GPU doesn't wait for the CPU to finish bottom layer training for all hot vertices. When the GPU ends sampling the first batch, it immediately starts data loading for the first batch. 
%If the hot vertices are sampled but not yet processed in the CPU, transfer the required feature to the GPU for training. Therefore, for $K$ rounds of data loading within a super-batch, the time required is gradually reduced as the embedding of the hot vertices is gradually cached in the CPU or GPU. 
%With this adaptive data loading design, the GPU benefits from the CPU's computing power without causing GPU to wait.
% \zyf{is these stages description too wordy? To me, it seems like no interesting idea is presented but only engineering details.}

\Paragraph{Stage 4: Training on GPU.}
The GPU training starts after $n$ rounds of GPU sampling. During the forward pass, the GPU pulls the historical embeddings of hot vertices in the bottom layer from either the CPU or the GPU cache. For other layers, the GPU performs aggregation and updates alternately as normal. In the backward pass, the GPU updates the parameters of each layer and shares the parameters of the bottom layer with the CPU. 

%For other layers, the GPU performs aggregation and updates alternately as normal. During the forward, the data required for backward of the hot vertices pulled from the CPU is also filled in the corresponding position. In the backward, the GPU updates the parameters of each layer and shares the parameters of the bottom layer with the CPU. 
%As shown in Figure \ref{fig:pipeline design} (b), this is an illustrative example of a super-batch pipelined training when $n = 4$.
%Therefore, the versions of the model parameters vary within a hot vertices queue. 
\Paragraph{Bounded Staleness.} 
Model parameters are updated once within each batch, so we record the model parameter version by batch number.
The CPU trains hot vertices in order of hotness and utilizes the latest model parameters. 
As shown in Figure \ref{fig:pipeline design} (b), during the first super-batch, the embeddings for hot vertices in the CPU may have versions ranging from $0$ to $n-1$, where $n$ is the number of batches contained within a super-batch. 
In the second super-batch, when the GPU pulls embeddings for hot vertices from the CPU, it may receive historical embeddings with model parameters that are older than the current version. 
The CPU must complete the embeddings update for hot vertices for the next super-batch within the current super-batch. This constraint ensures that the version gap remains within two super-batches, effectively preventing excessive staleness. Figure \ref{fig:pipeline design} (b) illustrates that the most significant version gap may occur in the last batch of the second super-batch. During this batch, the model parameter version is $2n - 1$, and it may utilize historical embeddings from the previous super-batch with a model parameter version of $0$. For the other batches, the version gap between historical embeddings and exact embeddings is smaller than the upper bound of $2n - 1$. 
%During execution, we will monitor the CPU and GPU working time in a super-batch. To avoid GPU wait, if the CPU fails to complete the embeddings update before the GPU completes training the last batch in a super-batch, the CPU embedding computation will be decreased. Conversely, the CPU will raise the hot vertices ratio, maximizing resource utilization.


%  and the CPU must complete the embeddings for hot vertices update for the next super-batch within the current super-batch. 
% This limitation not only avoids GPU waits but also ensures that the version gap remains within two super-batches, effectively preventing excessive staleness. 

% If the CPU fails to complete the embeddings update before the GPU completes training the last batch in a super-batch, the hot vertices ratio will be decreased. Conversely, the CPU will raise the hot vertices ratio, maximizing the CPU utilization.

% with a version gap ranging from $1$ to $2n - 1$

%For the bottom layer, from the first round to $k - 1$ matches of batch, the model parameters are updated each batch using the gradients of non-hotness vertices. The GPU will reduce and cache the gradients of hotness vertices. As discussed in the data loading, the aggregation and update of hotness points is done in both the CPU and the GPU. For hotness vertices in CPU, at the end of the $k - 1$ rounds of batch, the gradients of them are transferred back to the CPU. In the backward of the $K$ rounds of the batch, the parameters update of the bottom layer will use the accumulated gradients of all hotness vertices. Specifically, the hotness vertices in the CPU will perform all reduce of the gradient with the hotness vertices in the GPU and other vertices in the bottom layer in the $K$ rounds of batch, and finally update the model parameters.

%修改陈旧性检测方法，super-batch，去掉CPU瓶颈

% The CPU trains hot vertices in order of hotness and utilizes the latest model parameters. Therefore, the versions (batch number) of the model parameters vary within a hot vertices queue. As shown in Figure \ref{fig:pipeline design} (b), during the first super-batch, the hot vertices in the CPU may have versions ranging from $0$ to $n-1$. In the second super-batch, when the GPU pulls embeddings for hot vertices from the CPU, it may receive historical embeddings with model parameters that are older than the current version. 
% \zyf{I really think there should be a formal description of sampled mini-batch trainning algorithm in sec 2. otherwise, what is batch number? did you mention this before? Without the background of mini-batch training and the update process of vertex embeddings, people are likely to confuse about the description of bounded staleness.}
%To illustrate this, consider a batch entering the pipeline with the embedding for hot vertex $A$. Once this batch reaches the GPU, the gradients for the embedding for $A$ will be computed. While the gradient is being computed, consider another batch containing the embedding for hot vertex $A$ entering the pipeline. Now. While the updates from the first batch are being transferred back to the CPU and applied to parameters, the second batch has already entered the pipeline, and thus it contains a stale version of the embedding for hot vertex $A$. 
%In the super-batch pipeline training, besides using heterogeneous resources, it also guarantees bounded staleness.
% \Paragraph{CPU Workload Adjustment.} We can control the CPU workload by adjusting the number of vertices that use the historical embedding. 
% During the execution, we will monitor the CPU and GPU working time in a super-batch, and the CPU must complete the embeddings for hot vertices update for the next super-batch within the current super-batch. 
% This limitation not only avoids GPU waits but also ensures that the version gap remains within two super-batches, effectively preventing excessive staleness. 
% Figure \ref{fig:pipeline design} (b) illustrates that the most significant version gap may occur in the last batch of the second super-batch. During this batch, the model parameter version is $2n - 1$, and it may utilize historical embeddings from the previous super-batch with a model parameter version of $0$.
% For the other batches, most embeddings experience staleness smaller than the upper bound of $2n - 1$, with a version gap ranging from $1$ to $2n - 1$. 
% If the CPU fails to complete the embeddings update before the GPU completes training the last batch in a super-batch, the CPU workload (the number of vertices using caching embeddings) will be decreased. Conversely, the CPU will raise the workload, maximizing the utilization.




% \begin{figure}[t]
%   \centering
%   \includegraphics[width=8cm,page={1}]{figures/data management.pdf}
%      \vspace{-0.4cm}
%   \caption{An example of the \system's buffer management policy. n represents the model parameter version, and HE represents the historical embedding. }
%    \vspace{-0.6cm}
%   \label{fig:data management}
% \end{figure}

% and DMA-based feature transfer
% \label{sec:databuffer}
% \subsubsection{Buffer management}
% To prevent resource contention with training processing, it is necessary to design the GPU memory allocation for caching the embeddings for hot vertices.
% As implemented in the super-batch pipelined training, the GPU retrieves the historical embeddings from the CPU as needed and caches them for reusing. 
% The cached data of hot vertices can be managed by maintaining a shared memory space in the GPU and a pinned memory space in the CPU. As depicted in Figure \ref{fig:data management}, within each super-batch, the GPU replaces the outdated embeddings with the embeddings provided by the previous super-batch.
% It is necessary to estimate the maximum GPU memory allocation for this space. Considering the batch size and the sample fanout, we can compute the maximum number of vertices in the bottom layer, denoted as $V_{\text{max}}$. Subsequently, the maximum number of bottom layer vertices for a super-batch can be determined as $n \times V_{\text{max}}$. Therefore, determining the appropriate GPU memory allocation for the shared memory space depends on the hot vertices rate ranging from 0 to 1 and $k \times V_{\text{max}}$.
%   \vspace{-0.2cm}

%As shown in Figure \ref{fig:pipeline} (b), the worst version gap may come from the last batch in the second super-batch. In this batch, the model parameter version is $2k - 1$, and the historical embedding with model parameter version $0$ from the last super-batch may be used.  
%This method is also a Batch-Fixed Embedding Staleness which bound the most stale vision in $2k$. Most of the embedding's staleness is smaller than the upper bound of $2k$, and the version gap range from $1$ to $2k$.
%The CPU and GPU computing power should be considered when determining the ratio of hot vertices to be computed in a super-batch training. In one super-batch, the CPU has to complete the training and sampling of all the hot vertices. 


\label{prove}
\subsection{Convergence Analysis}


%SANCUS performs full-batch GNN training without sampling, avoiding communication by caching historical embeddings with stale constraints. \system's hotness-aware design has a similar goal but is distinct from it in two aspects. Firstly, \system performs sampling-based GNN training in mini-batches. Secondly, \system only uses stale embeddings in the bottom layer. 
% To ensure high model accuracy, besides setting the bound for the version (batch number) bound of the stale embeddings, we also set the bound for the model parameters of the stale embeddings \cite{ST_ICML_2018}. In this process, we monitor the model parameter variation between adjacent super-batches, and the maximum model parameter variation $\epsilon{W}$ that can be tolerated be defined as $\begin{Vmatrix} \tilde{W} - W \end{Vmatrix}\le \epsilon{W}$, where $\tilde{W}$ is the model parameter of stale embeddings, $W$ is the model parameter of the current batch. If the model parameters changes between adjacent super-batches are within the bound, the GPU can use the stale historical embeddings from the last super-batch.
% We deduce the convergence guarantee of \system as follows.

In this section, we provide a theoretical analysis of convergence guarantee for \system.
Bounded staleness has been widely used by machine learning systems \cite{SANCUS_VLDB_2022,ssp_2013_nips,ST_ICML_2018,Marius_OSDI_2021,GNNlab_EUROSYS_2022}, and we present the theoretical analysis referring to the SANCUS \cite{SANCUS_VLDB_2022} and VR-GCN \cite{ST_ICML_2018}. 
To ensure bounded staleness, \system designs super-batch pipelined training to limit the version (batch number) bound to $2n$, where $n$ is the number of batches in a super-batch. In this process, we monitor the model weight variation between adjacent super-batches, and the maximum model weight variation $\epsilon{}$ that can be tolerated be defined as $\epsilon{} = max \bigtriangleup \begin{Vmatrix} W \end{Vmatrix} \times 2n$, where $max \bigtriangleup \begin{Vmatrix} W \end{Vmatrix}$ denotes the maximum value variation of $W$ in a model weight update. 
With this staleness bound, we deduce the convergence guarantee as follows.

\begin{itemize}[leftmargin=*]
    \item \textbf{Proposition 1} provides the necessary and fundamental inequality operations required for the theoretical analysis;
    
    \item \textbf{Lemma 1} states that by imposing bounded staleness on the weights, the approximations of the embeddings and intermediate matrix results are close to the exact results;
    
    \item \textbf{Lemma 2} further demonstrates that the approximations of gradients in the training process closely match the exact gradients;

    \item \textbf{Theorem 1} concludes that the weight changes during training occur at a sufficiently slow rate, ensuring that the gradients are asymptotically unbiased and guaranteeing convergence;
\end{itemize}


%\setlength\abovedisplayskip{3pt}%shrink space \setlength\belowdisplayskip{3pt}
\textbf{Proposition 1.} \textit{Let} 
$\begin{Vmatrix} A \end{Vmatrix}_{\infty} = max_{ij} \begin{vmatrix}
A_{ij} \end{vmatrix}$, \textit{then we have}:
\begin{gather*}
\begin{Vmatrix} AB \end{Vmatrix}_{\infty} \le col(A) \begin{Vmatrix} A \end{Vmatrix}_{\infty} \begin{Vmatrix} B \end{Vmatrix}_{\infty} \\
\begin{Vmatrix} A \odot B \end{Vmatrix}_{\infty} \le \begin{Vmatrix} A \end{Vmatrix}_{\infty} \begin{Vmatrix} B \end{Vmatrix}_{\infty} \\
\begin{Vmatrix} A + B \end{Vmatrix}_{\infty} \le \begin{Vmatrix} A \end{Vmatrix}_{\infty} + \begin{Vmatrix} B \end{Vmatrix}_{\infty}
\end{gather*}
\noindent \textit{where}, $col(A)$ \textit{denotes the number of columns of the matrix} $A$, $\odot$ \textit{denotes the element wise product}.

This proposition has been proved by VR-GCN \cite{ST_ICML_2018}, and we omit the proof. We further denote $C$ as the maximum number of columns that exist in our analysis.

Lemma 1, lemma 2, and Theorem 1 have also been proved by SANCUS \cite{SANCUS_VLDB_2022} and VR-GCN \cite{ST_ICML_2018}, and we omit the proof and give the necessary and sufficient conditions for their establishment.

\textbf{lemma 1.} 
\textit{Assume all the activations are $\rho$-Lipschitz},  \textit{the} $\begin{Vmatrix} W_i \end{Vmatrix}_{\infty}$ and $\begin{Vmatrix} \hat{A}_i \end{Vmatrix}_{\infty}$ \textit{are bounded by some constant} $B$, \textit{and the historical weights} $\tilde{W}_i$ \textit{are close to the exact weights} $W_i$ \textit{with the staleness bound} $\epsilon$ \textit{where} $\begin{Vmatrix} \tilde{W}_i - W_i \end{Vmatrix}$ $\le$ $\epsilon$, $\forall i$. \textit{Then the approximation error of the stale embedding} $\tilde{H}$ \textit{and stale activation} $\tilde{Z}$ \textit{is bounded by some constant} $K$ \textit{that depends on} $\rho, C, B$: 
$\begin{Vmatrix} H_{i}^{l} - \tilde{H}_{i}^{l} \end{Vmatrix}_{\infty}$ $<$ $\epsilon K$, $\forall i$ $>$ $I$, $l = 1,..., L-1$;
$\begin{Vmatrix} Z_{i}^{l} - \tilde{Z}_{i}^{l} \end{Vmatrix}_{\infty}$ $<$ $\epsilon K$, $\forall i > I$, $l = 1,..., L$.

\textbf{lemma 2.}
\textit{Assume that activation function} $\sigma \left ( \cdot  \right )$ \textit{and the gradient} $\nabla \mathcal{L}$ \textit{are} $\rho-Lipschitz$, the $\begin{Vmatrix} \nabla \mathcal{L} \end{Vmatrix}_{\infty} $, 
$\begin{Vmatrix} \hat{A}_{i} \end{Vmatrix}_{\infty} $,
$\begin{Vmatrix} W_{i} \end{Vmatrix}_{\infty} $, \textit{and}
$\begin{Vmatrix} \sigma' \left ( Z_{i} \right ) \end{Vmatrix}_{\infty} $ \textit{are bounded by some constant} $B$, \textit{and the historical weights} $\tilde{W}_i$ \textit{are close to the exact weights} $W_i$ \textit{with the staleness bound} $\epsilon$ \textit{where} $\begin{Vmatrix} \tilde{W}_i - W_i \end{Vmatrix}$ $\le$ $\epsilon$, $\forall i$.
\textit{Then the approximation error of the gradient} $\tilde g(W_i)$ \textit{is bounded by some contents}: $ \begin{Vmatrix} \mathbb{E} \tilde g(W_i) - \nabla \mathcal{L}(W_i) \end{Vmatrix}_{\infty} \le \epsilon K$ \textit{and} $\forall i > I$, \textit{where K depends on} $\rho, C, B$. 


%$\nabla_{\tilde{Z}^{(l)} }\tilde{\mathcal{L}}$ and
%, the initial weights $W_{(1)}$
\textbf{Theorem 1.} 
\textit{Given the local minimizer} $W^{\star}$.
\textit{Assume that 
(1) the activation} $\sigma(\cdot)$ \textit{is $\rho$-Lipschitz}, 
\textit{(2) the gradient of the loss function} $\nabla \mathcal{L}(W_i)$ \textit{is $\rho$-Lipschitz and bounded}, 
\textit{(3) The gradient matrices}
$\begin{Vmatrix} \tilde{g}(W) \end{Vmatrix}_{\infty}$, 
$\begin{Vmatrix} g(W) \end{Vmatrix}_{\infty}$ and 
$\begin{Vmatrix} \nabla \mathcal{L}(W) \end{Vmatrix}_{\infty}$ \textit{are bounded by some constant} $G > 0$. 
\textit{(4) The loss} $\mathcal{L}(W)$ \textit{is $\rho$-smooth}, \textit{i.e.},
$$\lvert \mathcal{L}(W_2) - \mathcal{L}(W_1)- 	\left \langle \nabla \mathcal{L}(W_1), W_2 - W_1  \right \rangle \rvert \leq \frac{\rho}{2} \begin{Vmatrix} W_2 - W_1 \end{Vmatrix}_{F}^{2}, \forall W_1, W_2$$
, \textit{where $\left \langle A, B \right \rangle = tr(A^{T}B)$ is the inner product of matrix $A$ and matrix $B$}. 
\textit{Then, there exists $K > 0$, s.t., $\forall N > I$, if we run SGD for $R \leq N$ iterations, where R is chosen uniformly from $[I+1, ..., N ]$ and the learning rate }
$\eta = min\{{\frac{1}{\rho}, \frac{1}{\sqrt{N}}\}}$, , we have: 
$$\mathbb{E}_{R} \Vert \nabla \mathcal{L} (W_{R}) \Vert_{F}^{2} \leq 2\frac{\mathcal{L} (W_{1})- \mathcal{L}(W^{\star}) + \frac{\rho K}{2}}{\sqrt{N}}$$

when $N \rightarrow \infty$ , $\mathbb{E}_{R} \Vert \nabla \mathcal{L} (W_{R}) \Vert_{F}^{2} \rightarrow 0$, the above concludes that the convergence is guaranteed.



%Therefore, if the CPU can't complete one embedding update for the hot vertices queue of the next super-batch when the GPU finishes training the last one batch in current super-batch, the hot vertices rate will be reduced. In contrast, the CPU will increase the hot vertices rate, further utilizing the CPU computing power.

%We need to control the hot vertices rate in the bottom layer and the batch amount in a super-batch. Specifically
% \vspace{-0.05in}
% \subsection{\zyf{Implementation}}
% \label{sec4.5}

% \zyf{If the implementation tech is not evaluated in experiment section. I suggest to remove it and briefly introduce in the beginning of Sec 5. }
% \system uses NeutronStar \cite{neutron_sigmod_2022} and LibTorch (v1.9) \cite{pytorch_NeurIPS_2019} as the underlying frameworks. 
% %NeutronStar decouples the forward and backward execution flows into multiple pairs of graph operations and neural network functions. Each operation pair corresponds to a stage in the forward and backward computation, enabling the design of a flexible computation orchestration strategy. 
% In addition to main functions, \system provides the following two implementations to further improve performance.

% \Paragraph{Out-of-memory sampling}
% %We also use the Graph Store Server implemented by NeutronStar to store graph structural data and features. 





% \textbf{Local shuffling.}
% Shuffling the training vertices is necessary to improve the empirical performance of sample-based GNN training. Shuffling can be performed globally across all training vertices or locally within each super-batch, which facilitates the sharing of hot vertices across different batches within the super-batch. 
% Local shuffling is a widely adopted practice in real-world scenarios, and recent studies have demonstrated its comparable performance to global shuffling \cite{PAGRAPH_SOCC_2020,shuffle_2019}. 
% Thus, \system shuffles the execution order of the super-batch first and then shuffles the training vertices within each super-batch. Furthermore, we demonstrate in \ref{} that local shuffling enhances the hit rate of hot vertices in the bottom layer without affecting the model accuracy.

%We need to estimate the maximum amount of GPU memory allocation for this space. Given the batch size and the sample fanout, we can calculate the maximum number of vertices at the bottom layer, denoted as $V_max$. Then the maximum number of bottom layer vertices for a super-batch can be used $k * V_max$. 
%To achieve this, we leverage the fact that its memory consumption is similar across training iterations. It is sufficient to decide the available GPU memory for the shared memory space with the hot vertices rate from 0 to 1.


%To limit this staleness, we use Batch-Fixed Embedding Staleness which bound the number of batches in the pipeline at any given time. For example, if the bound is 4, embeddings in the pipeline will be at worst 4 updates behind. However, different from full graph training in SANCUS\cite{SANCUS_VLDB_2022}, due to the sparsity of node embedding updates, it is unlikely a node embedding will even become stale. To give a realistic example, take the lj-large graph which has 10.68 million nodes. The typical batch size is 10240. As shown in \ref{tab: Memory size comparision}, Each batch will have an average of 24000 node embeddings. If we give the staleness bound 4 and there can be an average of 96000 node embeddings in the pipeline at any given time, which is just 0.9\%. of all node embeddings. We can observe that in most cases, only a very small fraction of node embeddings will be operated on at a given time. 




% \begin{algorithm}[t]
% 	\caption{Workflow in CPU} \small 
% 	\label{alg:workflow}  
% 	\begin{algorithmic}[1]
% 		\Require $G(V,E)$, $L$, $\{\textbf{h}^{0}_v\mid v\in V\}$, $\{\mathbb{L}_v\mid v\in V_{\mathbb{L}}\}$, initial model parameters $W$ for bottom layer
% 		\Ensure updated parameters $W$ for bottom layer
% 		\State initialize vector $Emebedding\_{CPU}$, $Hotness$, $AGGFlag$, $ AGGMap$ on CPU.

% 		\For{each $V_{i}$ in $Hotness$} in parallel
% 		\State $nbrs = uniform\_select(V_{i}, fanout)$;
%         \State $Sample\_{CPU}.push(numbers)$;
% 		\EndFor
%         \For{each $V_{i}$ in $Hotness$} in parallel
%         \If{$AGGFlag[V_{i}] == 0$}
% 	    \State $embedding = forward(\textbf{h}^{0}_v, W, Sample\_{CPU}[V_{i}])$;
%         \State $Emebedding\_{CPU}.push(embedding)$;
%         \State $AGGMap[V_{i}] = Emebedding\_{CPU}.size()$;
%         \State $AGGFlag[V_{i}] = 1$; \textcolor{gray}{//1: Embedding in CPU}
% 		\EndIf
% 		\EndFor
% 		\State Wait Grad from GPU;
%         \For{each $V_{i}$ in $Hotness$} in parallel
%         \If{$AGGFlag[V_{i}] == 1$}
%         \State $embedding = Emebedding\_{CPU}[AGGMap[V_{i}]]$;
% 	    \State $backward(embedding, W, Sample\_{CPU}[V_{i}])$;
%         \State $AGGFlag[V_{i}] = 0$;
%         \EndIf
%         \EndFor
% 	\end{algorithmic}  
% \end{algorithm}



% \begin{algorithm}[t]
% 	\caption{Balance check} \small 
% 	\label{alg:workflow}  
% 	\begin{algorithmic}[1]
% 		\Require 
% 		\Ensure 
% 		\State Balance check

% 	\end{algorithmic}  
% \end{algorithm}
\section{Evaluation}
\label{sec5}
% We implement \system based on NeutronStar \cite{neutron_sigmod_2022} and LibTorch (v1.9) \cite{pytorch_NeurIPS_2019}. 
This section evaluates the performance of \system using three representative GNN models and six real-world graph datasets. 
%\zyf{the following words can be removed.} Specifically, we investigate the overall performance and the efficiency achieved through hotness-aware layer-based task orchestrating and super-batch pipelined training. Additionally, we conduct a breakdown analysis under various conditions.

\subsection{Experimental Setup}

\Paragraph{Environments.}
%单机单卡
Single GPU experiments are conducted on an Aliyun server equipped with an Intel Xeon Platinum 8163 CPU (48 cores and 368 GB main memory) and NVIDIA V100 (16GB) GPU. The multi-GPU experiments are conducted on an Aliyun server equipped with an Intel Xeon Platinum 8163 CPU (96 cores and 736 GB main memory) and eight NVIDIA V100 (16GB) GPUs. The eight GPUs are connected to the CPU via four PCIe-3.0 switches and equipped with NVLink interconnects similar to NVIDIA DGX-1 \cite{DGX1}. The GPU is enabled with CUDA 11.4 runtime and 418.67 drivers, and the host side is running Ubuntu 18.04 with Linux kernel version 4.13.0. All the source codes are compiled with O3 optimization.

%We adopt the same model architectures and configurations as in the reference.
\Paragraph{GNNs and model configurations.}
% We evaluate \system using two representative GNN models: Graph Convolutional Network (GCN) \cite{GCN_iclr_2017} and GraphSAGE \cite{Graphsage_2017}.
We evaluate \system using three representative GNN models: GCN \cite{GCN_iclr_2017}, GraphSAGE \cite{Graphsage_2017}, and GAT \cite{GAT_ICLR_2018}. Compared with other systems, the training batch size is set to 1024, the model depth is set to 3, and the sampling fan-out is set to [25, 10, 5]. When using more layers for performance evaluation, sampling fan-out beyond 3 layers will be set to 5, i.e., [25, 10, 5, 5 $\cdots$].


% Graph Convolutional Network (GCN) utilizes 2-hop random neighborhood sampling and GraphSAGE utilizes 3-hop random neighborhood sampling. Graph Convolutional Network (GCN) \cite{GCN_iclr_2017} generalizes the graph convolution operation as follows. Each vertex in a GCN layer aggregates the features of its neighboring vertices using a summation operation. Subsequently, the aggregated feature is passed through a fully-connected layer followed by a ReLU activation to generate output representations. GraphSAGE \cite{Graphsage_2017} is an inductive learning model that employs different aggregation functions depending on the number of hops. GraphSAGE offers four aggregation types: GCN(sum), Mean, LSTM, and Pooling. We present the results of the GraphSAGE-Mean model because all four aggregators exhibit similar execution costs \cite{Graphsage_2017}. 
% %Past works \cite{sampleacc_icml_2018,ST_ICML_2018,GNNSampling_sigops_2021, sampling_caa_2022} have demonstrated that sampling-based training can achieve comparable model accuracy to full graph training, even when sampling only a few neighbors at each neighborhood layer.
% In our evaluation, we set the sampling fanout to 4 for every layer. In addition, we set the training batch size to 10000 as it typically allows for full utilization of GPU.

\begin{table}[!t]
\vspace{-0.05in}
	\caption{Dataset description.}
	\vspace{-0.1in}
	%\hspace{-0.5in}
	\label{tab:Dataset}
	\centering
	\footnotesize
	{\renewcommand{\arraystretch}{1.2}
	\begin{tabular}{l r r c c c c}
		\hline
		
		\hline
		{\textbf{Dataset}} &
		{\textbf{|V|}} &
		{\textbf{|E|}}  &
		{\textbf{ftr. dim}}&
		{\textbf{\#$\mathbb{L}$}}&
		{\textbf{hid. dim}}\\
		\hline
        {Reddit \cite{Graphsage_2017}} & 232.96K &114.61M & 602 &41 &256\\
		{Lj-large \cite{LIVE_KDD_2006}} & 10.69M & 224.61M & 400 & 60 &256\\	%没有feature
		{Orkut \cite{ORUKT_KIS_2015}} &3.1M& 117M &600&20&160\\	%没有feature
        {Wikipedia \cite{konect_www_2013}} & 13.6M &437.2M & 600 &16&128\\
        {Products (PR) \cite{ogb_2020_neurIPS}} & 2.4M &61.9M & 100 &47&64\\
        {Papers100M (PA) \cite{ogb_2020_neurIPS}} & 111M &1.6B & 128&172&64\\
		\hline
		
		\hline
		
	\end{tabular}
	}
	\vspace{-0.1in}
\end{table}

% {ogbn-products} & 2.4M &61.9M & 128 - 4096 &16 &256\\
% {amazon}& 14.7M  & 64.0M & 578 & 16&160\\
% {ogbn-papers100M} & 111.1M & 1.6B &128&41&256\\	
% \begin{table}[!t]
% \vspace{-0.05in}
% 	\caption{The competitor systems.} 
% 	\vspace{-0.1in}
% 	%\hspace{-0.5in}
% 	\label{tab:system}
% 	\centering
% 	\footnotesize
% 	{\renewcommand{\arraystretch}{1.2}
% 	\begin{tabular}{l c c c c}
% 		\hline
		
% 		\hline
% 		{\textbf{System}} &
% 		{\textbf{Sample}}  &
% 		{\textbf{Gather}}&
% 		{\textbf{Train}}\\
% 		\hline
% 		{DGL \cite{DGL_CORR_2019}} &CPU & CPU & GPU\\
% %		\hline
% 		{PaGraph \cite{PAGRAPH_SOCC_2020}} & CPU & GPU & GPU\\
% %		\hline
% 		{GNNLab \cite{GNNlab_EUROSYS_2022}} & GPU & GPU & GPU\\		
           
%         {DGL-UVA \cite{PYTDIRECT_VLDB_2021}} & GPU & CPU &GPU\\
        
%         {GAS \cite{GNNautoscale_icml_2021}} & GPU & CPU & GPU \\
        
%         % {DSP \cite{DSP_PPOPP_2023}} & GPU & GPU &GPU \\
        
%         {\system (ours)} & CPU-GPU & CPU-GPU &CPU-GPU\\	
% 		\hline
		
% 		\hline
		
% 	\end{tabular}
% 	}
% 	\vspace{-0.1in}
% \end{table}

\Paragraph{Datasets.} 
For evaluation, we utilize {six} real-world graph datasets, as listed in Table \ref{tab:Dataset}. These include Reddit \cite{Graphsage_2017} and Orkut \cite{ORUKT_KIS_2015}, which are social networks, the Wikipedia (Wiki) network \cite{konect_www_2013} comprising wikilinks from the English Wikipedia, the LiveJournal communication network (Lj-large) \cite{LIVE_KDD_2006}. The Products (PR) \cite{ogb_2020_neurIPS} dataset is based on Amazon's co-purchasing network. The Papers100M (PA) \cite{ogb_2020_neurIPS} is a citation graph, where each vertex represents a paper and the edge represents the citation relation.
The "ftr. dim" column represents the dimension of vertex features, the "\#L" column represents the number of vertex classes, and the "hid. dim" column represents the embedding dimension of the hidden layer output.
For graphs without ground-truth properties (Lj-large, Orkut, and Wikipedia), we use randomly generated features, labels, training (65\%), test (10\%), and validation (25\%) set division.

%that incorporates CPU-based graph sampling and CPU-based feature gathering
%that incorporates GPU-accelerated graph sampling and zero-copy optimizations

% In addition, we compare \system with GNNAutoScale (GAS) \cite{GNNautoscale_icml_2021}, a representative system based on historical embedding.

% PaGraph designs a degree-based feature cache policy to reduce CPU-GPU communication. GNNLab proposes a pre-sampling-based feature cache policy to accommodate different graph datasets and sampling algorithms. stores graph topology and popular vertex features among the GPUs, fully utilizing the memory of the GPUs and reducing the CPU-GPU communication prunes the computation graph and reduces the GPU memory requirement of GNN training by utilizing historical embeddings


\Paragraph{Baselines.}
We compare \system with DGL \cite{DGL_CORR_2019}, DGL-UVA \cite{PYTDIRECT_VLDB_2021} , PaGraph \cite{PAGRAPH_SOCC_2020}, GNNLab \cite{GNNlab_EUROSYS_2022}, GNNAutoScale (GAS) \cite{GNNautoscale_icml_2021}, and DSP \cite{DSP_PPOPP_2023}. 
All comparison systems use GPUs to conduct training.
DGL and DGL-UVA store both the graph structure and features in CPU memory. The difference is that DGL conducts sampling using the CPU, while DGL-UVA conducts sampling using the GPU by utilizing the UVA technique.  
PaGraph and GNNLab utilize GPU-based feature caching to reduce CPU-GPU communication. PaGraph conducts the sampling using the CPU, and GNNLab stores the graph structure in GPU memory and conducts the sampling using the GPU.
GAS conducts feature gathering in the CPU and utilizes historical embedding to accelerate training. 
% The utilization of historical embeddings in GAS is relatively coarse-grained. Initially, GAS calculates and saves historical embeddings for all vertices. However, GAS does not provide bounded staleness and historical embeddings are reused among an epoch. 
%GAS 对历史嵌入的利用相对粗粒度。最初，GAS 会计算并保存所有顶点的历史嵌入。但是，GAS 并不提供有界限的滞后性，历史嵌入会在一个时代内重复使用。
DSP is a multi-GPU GNN training system that uses multi-GPU cooperative sampling. It also caches the graph topology and popular vertex features in GPU memory to accelerate the gathering step.
All these systems use the four specific step-based task orchestrating methods as discussed in Section \ref{sec3}. 
In the single GPU experiments, we compare \system with five systems: DGL \cite{DGL_CORR_2019}, DGL-UVA \cite{PYTDIRECT_VLDB_2021} , PaGraph \cite{PAGRAPH_SOCC_2020}, GNNLab \cite{GNNlab_EUROSYS_2022}, and GAS \cite{GNNautoscale_icml_2021}.
As GAS only provides a single GPU implementation, in multi-GPU experiments, we compare \system with four systems: PaGraph \cite{PAGRAPH_SOCC_2020}, DGL-UVA \cite{PYTDIRECT_VLDB_2021}, GNNLab \cite{GNNlab_EUROSYS_2022}, and DSP \cite{DSP_PPOPP_2023}.
% We present the details of the compared systems in Table \ref{tab:system}.

%DGL, DGL-UVA, PabGraph, and GNNLab are typical GNN systems using the four specific step-based task orchestrating methods as discussed in Section \ref{sec3}. 

\begin{figure}
  \centering
\includegraphics[width=8cm,page={1}]{figures/overall.pdf}
    \vspace{-0.4cm}
  \caption{Overall training performance comparison (``OOM'' denotes out of memory).}
   \vspace{-0.6cm}
  \label{fig:overall-performance}
\end{figure}

\subsection{Single GPU Performance}
%We first compare \system with its competitors. 
Figure \ref{fig:overall-performance} shows the average training time of one epoch for all GNN systems. Note that GNNLab \cite{GNNlab_EUROSYS_2022} and PaGraph \cite{PAGRAPH_SOCC_2020} do not provide GAT support, and GAS \cite{GNNautoscale_icml_2021} does not provide GraphSAGE support.

(1) Comparison with DGL \cite{DGL_CORR_2019}: \system achieves a speed-up ranging from 2.91$\times$ (for GAT on Orkut) to 11.51$\times$ (for GCN on PR) compared to DGL.
DGL exhibits inferior performance compared to the other systems due to inefficient CPU-based sampling and gathering steps.
% As discussed in Section \ref{sec3}, DGL's task orchestrating method requires the CPU to perform both the sampling step and feature collection in the gathering step. 
The CPU resources are insufficient for effectively accelerating the sampling step in parallel and are also unable to handle the frequent random memory access required for feature collection. These two steps account for a substantial portion of the total runtime, resulting in considerable waiting time for the GPU.
\system effectively avoids inefficient CPU processing by selectively computing historical embeddings for hot vertices on the CPU. Hot vertices typically account for only a fraction of all training vertices, and historical embeddings can be reused multiple times across multiple batches.
% , effectively reducing computation on the CPU. 

%Second, the training data, mainly the features of the hot vertices' sampled neighbors, are stored in the CPU memory. 
(2) Comparison with PaGraph \cite{PAGRAPH_SOCC_2020}: \system achieves a speed-up ranging from 2.68$\times$ (for GraphSAGE on PA) to 9.72$\times$ (for GCN on PR) compared to PaGraph. PaGraph uses GPU-based gathering, reserving a portion of GPU memory for caching the features of high-degree vertices. The performance of PaGraph is limited by inefficient CPU sampling and GPU memory contention. 
\system provides more flexible task orchestration based on heterogeneous resources. When GPU resources are limited, increasing the CPU embedding computation reduces the memory requirement for training and caching in the GPU. When GPU resources are sufficient, reduce CPU embedding computation while increasing the feature cache ratio to avoid GPU waiting.
%As discussed in Section \ref{sec3}, the hit ratio of cache policy decreases as the feature dimension and batch size increase due to the limited GPU memory. 
% By employing the hotness-aware layer-based task orchestrating method, \system is memory-efficient. First, NeutronOrch saves the training memory requirement by offloading the computation to the CPU. Second, the cached objects of \system are only historical embeddings for hot vertices within a super-batch, consuming only a small amount of GPU memory. Therefore, \system can enable more GPU memory for accelerating sample and train steps while avoiding GPU memory contention. 


(3) Comparison with GNNLab \cite{GNNlab_EUROSYS_2022}: \system achieves a speed-up ranging from 1.52$\times$ (for GCN on Orkut) to 2.43$\times$ (for GraphSAGE on PR) compared to GNNLab.
NeutronOrch performs better when training with larger models or datasets. GNNLab needs to maintain the graph topology, cache data, and training data in the GPU memory, which constrains its scalability, especially when training billion-scale graphs like Papers100M for users with limited GPU resources. When the training memory requirement increases, GNNab's performance degrades due to a decreased cache hit rate. Moreover, when handling deeper GNN models, GNNLab encounters out-of-memory (OOM) issues due to GPU memory exhaustion, as shown in Table \ref{table:vary_layer}, Table \ref{table:vary-batchsize}, and Section 5.7.

%In contrast, NeutronOrch reduces computation, communication, and memory consumption by reusing historical embeddings and scheduling training tasks across CPUs and GPUs. As shown in Table \ref{table:vary_layer}, when training a 3-layer, 4-layer, and 5-layer GCN on Products, our speedups are 1.80$\times$, 1.89$\times$, and 2.24$\times$, respectively. 

%GNNLab proposes a factored system that places sampling and training steps across GPUs and introduces a pre-sampling-based caching policy to accelerate the data loading step. However, in a heterogeneous environment with only one GPU, as discussed in Section \ref{sec2}, the three steps executed on the GPU can lead to memory and resource contention issues. The GPU resource contention and CPU computing power idling can both adversely affect the performance of heterogeneous systems. Through the hotness-aware layer-based task orchestrating method, \system allocates reasonable and sufficient tasks for both CPU and GPU, avoiding resource contention. On the other hand, the super-batch pipeline design ensures that \system has high CPU and GPU utilization. 

%By assigning rational computational tasks to the CPU, \system fully utilizes memory and computation resources in heterogeneous environments.
% By capitalizing on increased CPU and GPU utilization simultaneously, \system demonstrates better overall performance.
% \wqg{What engine does DGL use?}
% \wqg{If it is deployed in \system, then what are the differences between our system and DGL-UVA}, which is also deployed on the \system

\begin{figure*}[th]
  \centering
  \vspace{-0.3in}
  \includegraphics[width=0.9\textwidth,page={1}]{figures/multi-GPU.pdf}
     \vspace{-0.4cm}
    \caption{Per-epoch runtime of NeutronOrch and other systems under multi-GPU environment. (bs denotes batch size, ``x'' denotes illegal memory access, and ``OOM'' denotes out of memory).}
   \vspace{-0.2cm}
  \label{fig:mulit-GPU}
\end{figure*}


(4) Comparison with DGL-UVA \cite{DGL_CORR_2019}: \system achieves a speed-up ranging from 1.81$\times$ (for GCN on Orkut) to 9.18$\times$ (for GCN on PR) compared to DGL-UVA. DGL-UVA supports accessing graph topology and features via the zero-copy transfer engine \cite{PYTDIRECT_VLDB_2021}. We categorize DGL-UVA's gather step on the CPU since it keeps all the features in the CPU memory without feature caching. 
% The zero-copy transfer engine helps the GPU accelerate the sampling step without saving the graph topology in GPU memory and eliminates the feature collection processing in the gathering step. 
Compared to saving the graph topology in the GPU for sampling, DGL-UVA saves GPU memory while introducing access latency between CPU and GPU. On the other hand, DGL-UVA has a larger communication compared to \system because it transfers all features needed for training in every iteration. 
% as discussed in Section \ref{sec3}, performing entire sample and train steps on the GPU can result in resource contention and prolonged execution time. 

(5) Comparison with GAS \cite{GNNautoscale_icml_2021}: \system achieves a speed-up ranging from 7.08$\times$ (for GAT on Reddit) to 11.05$\times$ (for GAT on PR) compared to GAS. This is because GAS computes historical embeddings for all vertices and transfers them back to the CPU memory. Although GPU memory is saved, additional overhead is incurred due to frequent CPU-GPU communication. GAS also faces CPU memory limitations on graphs with many vertices, as it needs to store embeddings for all vertices across every layer.
In contrast, NeutronOrch selectively computes historical embeddings for frequently accessed vertices on the CPU, achieving efficient historical embedding reuse while reducing memory overhead. 
%This allows for a higher reuse probability with minimal CPU load.
Furthermore, NeutronOrch achieves higher accuracy with strictly bounded staleness than GAS, which arbitrarily reuses historical embeddings within an epoch (see more details in Section \ref{train_convergence}).

% In general, \system achieves its performance over competitors through two primary aspects. First, by employing the hotness-aware layer-based task orchestrating method and a super-batch pipelined training, \system achieves balanced utilization of heterogeneous resources and higher CPU and GPU utilization compared to step-based task orchestrating methods. Second, by utilizing hotness-aware embeddings reusing with bounded staleness, \system effectively reduces the overall computation and transfer data volume while ensuring model accuracy.
% \wqg{ DO WE HAVE SUCH A METHOD? I dont think you clearly explain why DGL-UVA is worse than us, the reason seems quite general and not convenience\# and lacks a GPU-based caching method. }



\vspace{-0.2cm}
\subsection{Multi-GPU Performance}
% \Paragraph{Overall comparison.}
We conduct a comparative analysis of NeutronOrch, PaGraph \cite{PAGRAPH_SOCC_2020}, DGL-UVA \cite{PYTDIRECT_VLDB_2021}, GNNLab \cite{GNNlab_EUROSYS_2022}, and DSP \cite{DSP_PPOPP_2023} to evaluate the scalability by varying the number of GPUs used in training. Figure \ref{fig:mulit-GPU} shows the results of training GraphSAGE against two real-world datasets with different numbers of GPUs. 
\system consistently demonstrates superior performance over the baselines with different batch sizes and different numbers of GPUs.
Compared to DGL-UVA \cite{PYTDIRECT_VLDB_2021} and PaGraph \cite{PAGRAPH_SOCC_2020}, NeutronOrch achieves on average 6.33$\times$ and 5.20$\times$ speedups. The performance of DGL-UVA and PaGraph is limited by extensive CPU-GPU communication and inefficient CPU sampling. In contrast, NeutronOrch minimizes CPU-GPU communication by reusing the historical embeddings and adaptively adjusting the workload between CPU and GPUs.
Compared to GNNLab \cite{GNNlab_EUROSYS_2022} and DSP \cite{DSP_PPOPP_2023}, NeutronOrch achieves on average 2.28$\times$ and 1.36$\times$ speedups. 
GNNLab and DSP deploy all steps (sample-gather-train) on the GPU and leave the CPU idle. When the number of GPUs decreases, GPU memory contention makes the benefit of their caching method decrease. 
% On Products with 4 and 8 GPUs, DSP and GNNLab show similar performance because their cache design effectively reduces CPU-GPU communication. 
In addition, when handling large-scale graphs (Papers100M), both DSP and GNNLab report memory errors due to memory exhaustion. %Notably, DSP even fails to operate on small graphs when using a batch size of 1024. 
NeutronOrch effectively trains large-scale GNNs by offloading computations to the CPU, reducing both CPU-GPU communication and GPU memory overhead.

% \Paragraph{Scaling Performance.}
% The execution time for all systems decreases as more GPUs are added. Specifically, when the number of GPUs increases from 1 to 8, the runtimes of \system, PaGraph, DGL-UVA, GNNLab, and DSP decrease by 3.54$\times$, 3.01$\times$, 2.97$\times$, 4.34$\times$, and 6.07$\times$, respectively. 
% % The scaling performance of DGL-UVA is limited by extensive CPU-GPU communication. In our GPU server, the 8 GPUs are connected to the CPU through 4 PCIe switches, with each two GPUs sharing the same PCIe link to fetch data from the CPU, causing bandwidth contention during training.
% % The scaling performance of PaGraph is limited by inefficient CPU sampling. 
% NeutronOrch, PaGraph, and DGL-UVA fail to achieve a speedup greater than 4$\times$ due to their reliance on CPU-GPU communication for data transfer. In our GPU server, the 8 GPUs are connected to the CPU through 4 PCIe switches, with each two GPUs sharing the same PCIe link to fetch data from the CPU, causing bandwidth contention during training.
% Among these three systems, NeutronOrch achieves better scaling performance by reducing CPU-GPU communication through historical embedding reuse and feature caching. 
% GNNLab and DSP achieve better speedup in successful runs. Nevertheless, their design, which involves caching vertex features and the entire graph topology, limits their ability to handle large-scale graphs. NeutronOrch consistently exhibits high efficiency in all scenarios thanks to the memory-efficient task orchestrating method.

\begin{figure}
  \centering
   % \vspace{-0.1in}
  \includegraphics[width=8cm,page={1}]{figures/breakdown.pdf}
     \vspace{-0.2cm}
  \caption{Performance gain analysis. L for the layer-based task orchestrating, HE for the hotness-aware embedding reusing, HH for the hybrid hot vertices processing, and S for the super-batch pipelined training.}
   \vspace{-0.2cm}
  \label{fig:breakdown}
\end{figure}

%In the baseline+L configuration, the entire bottom layer is executed in the CPU. 
%1.477942079	2.844361375	 1.642410566
% \wqg{I don't know what is the basic optimization, also, I dont know what is the baseline}
% \wqg{In NeutronStar and HytGraph, we have mature words to introduce the perf. gain analysis, but you still invente a set of  new scentences}
% \vspace{-0.2cm}
\subsection{Performance Gain Analysis}
In this section, we analyze the performance gain of layer-based task orchestrating (L), hotness-aware embedding reusing (HE), hybrid hot vertices processing (HH), and super-batch pipelined training (S) on the GCN model with six datasets. 
We start from a baseline with \system’s codebase for a fair comparison and gradually integrate four optimizations.
The baseline is a step-based task orchestrating method that employs GPU-based graph sampling, GPU-based training, and CPU-based gather step.
Figure \ref{fig:breakdown} shows the normalized speedups. 
%\sanzo{replace with: layer-based task orchestrating}
The layer-based task orchestrating aggregates and updates all vertex features on the CPU and transfers the vertex embedding to the GPU for computation, significantly reducing CPU-GPU communication. 
It can achieve an average speedup of 1.52$\times$ compared to the baseline. 
The hotness-aware embedding reusing optimizes the layer-based task orchestrating of \system, significantly reducing CPU computation. It provides an additional average speedup of 1.96$\times$ over the baseline+L. 
The hybrid hot vertices processing further reduces CPU-GPU communication by effectively balancing CPU and GPU resource utilization. 
It provides an additional average speedup of 1.53$\times$ over the baseline+L+HE. 
Finally, the super-batch pipeline design provides an additional average speedup of 1.61$\times$ over the baseline+L+HE+HH. This design enables the overlap of computations between the CPU and GPU, reducing overall execution time substantially.


% \wqg{Why place the following paragraph here? ? SEE COMMENTS}
%1这个东西看起来应该出现在5.2 而不是这里。
%2）这个东西全是废话，没看出来要说明什么。
%3这种毫无营养的废话会消磨耐心

%三个算法 GCN GraphSage，GAT（pagraph gnnlab不支持GAT） 六个数据集（待定，可以减为4-5个，考虑性能挑选），对比四个计算模式的代表系统

% 固定T在GPU，S D在CPU或GPU四种情况 4 - 4 
%1.GPU： S  T     CPU: D        GNNlab 关掉cache （GNNlab使用显式拷贝，关闭cache即可）
%2.GPU： D  T     CPU: S       Pagraph GNNlab CPU sampling （基于cache策略的 Dataloading，同时采样在CPU）
%3.GPU： S  D  T  CPU: idle     GNNlab 单GPU （性能最佳，打败这个即可） 
%4.GPU： T        CPU: S  D     DGL 0.6版本 (比较朴素的实现，最垃圾)
%5.our system
%整体性能对比，100个eopch，平均一个epoch的时间. 三个算法
% \zyf{don't understand what degree, presample, HER are.}

\begin{figure}[t]
  \centering
  \includegraphics[width=8cm,page={1}]{figures/hot-vertices.pdf}
  \vspace{-0.2cm} %调整图片与上文的垂直距离
  \caption{The memory consumption and transfer volume comparison among different caching policies.}
  \label{fig:hot vertices}
   \vspace{-0.2cm}
\end{figure}

% \wqg{Do we have a outline? I dont see any intuition why evaluating the mentioned things needs implementing caching.}
% \wqg{You should: First, we.... ; Second We .... On the otherhand, the following discussion has very very very very LOW information entropy. Simply repeat our design, nothing interesting, nothing useful, And seems have no "analysis"; See COMMENTS}
% %能不能把最基本的图的顺序和文字顺序做到一致？在caption里面把信息都补足


%节省GPU内存，GPU内存保存两个superbatch内版本的embedding，当前版本和上一版本。
\vspace{-0.1cm}
\label{sec:hot-layer}
\subsection{Benefit of Hotness-Aware Layer-Based Task Orchestrating}

We next explore how the hotness-aware layer-based task orchestrating method impacts the CPU-GPU communication, GPU memory consumption, and GPU training time.

We implement the degree-based cache policy (Degree) \cite{PAGRAPH_SOCC_2020} and the pre-sample-based cache policy (PreSample) \cite{GNNlab_EUROSYS_2022} in \system and compare them with hybrid hot vertices processing (Hybrid). We conduct experiments using the GCN model on the Wikipedia dataset. Figure \ref{fig:hot vertices} (a) illustrates that the Hybrid in \system leads to an average reduction of 55.1\% in GPU memory consumption compared to static cache policies, as embeddings generally have smaller dimensions than features.
On the other hand, although the embedding cache needs to be updated dynamically, each cache hit saves more raw feature transfers.
Under different hot vertices ratios, the average transfer volume using the Hybrid method is 63.2\% and 75.8\% of the static caching strategies Degree and PreSample, respectively. 

% Unlike static caching policies that require substantial GPU memory, \system utilizes the available computational power of CPUs and minimal GPU memory to achieve comparable reductions in transfer volume. 
% GNNlab and PaGraph propose reserving a portion of GPU memory for static caching of frequently accessed vertex features. In contrast, \system offloads the bottom layer training of these vertices to the CPU and pulls vertex embeddings on-demand on the GPU side. 
% Thanks to the hotness-aware embedding reusing method (HER) with bounded staleness, \system achieves reduced transfer volume and saves GPU memory. 
% Each hot vertex embedding in the CPU pulled by the GPU will save the transfer volume of its neighboring features, and hot vertex embedding in the GPU cache hit will save more transfer volume. 

% Meanwhile, in the GPU, \system only needs to maintain the space needed for a hot vertices queue of one super-batch. This greatly saves GPU memory compared to traditional static caching policies and provides fine-grained historical embeddings for each super-batch.

% To illustrate it, we conduct experiments using the GCN model on the Wikipedia dataset. Figure \ref{fig:hot vertices} (a) illustrates that the dynamic caching of historical embeddings in \system leads to an average reduction of 98\% in GPU memory usage compared to static cache policies like Degree and PreSample for different cache ratios and hot vertex ratios. Unlike static caching policies that require substantial GPU memory, \system utilizes the available computational power of CPUs and minimal GPU memory to achieve comparable reductions in transfer volume. Under different cache ratios and hot vertices ratios, the average transfer volume using the HER method is 1.44$\times$ and 1.73$\times$ of the static caching strategies Degree and PreSample, respectively. 

%As discussed in Section \ref{sec2.3}, a single GPU with 11 GB global memory can support a cache ratio of up to 0.05 for a batch size of 10,000 on the Wikipedia dataset. 
%Although \system requires relatively more transfer volume for the same cache ratio and hot vertices ratio. As discussed in Section \ref{sec2.3}, a single GPU can support a cache ratio of up to 0.05 for a batch size of 10,000 on the Wikipedia dataset. It's easier to support larger hot vertices ratios than cache ratios. 

%However, limited by CPU power and model convergence problems, the proportion of hotspots supported by H is limited, usually no more than 0.25
% 对比传输量，传输量 = CPU to GPU的feature 或 embeddding 的维度 * 顶点个数。
% 对比对象 ：1.our solution 2.静态cache方法（GNNlab）presample cacha policy
%与cache对比， 在使用更少GPU内存的情况下，节省comparable的传输量，画GPU内存-节省传输量的折线图
% 两个数据集，GCN算法 orkut livejournal cache rate 0.05 - 0.3

\begin{figure}[!t]
  \centering
  \vspace{-0.2cm}
  \includegraphics[width=8cm,page={1}]{figures/computation_volume.pdf}
  \vspace{-0.4cm} %调整图片与上文的垂直距离
  \caption{Average GPU training time in one epoch of Baseline and \system.}
  \label{fig:computation volume}
   \vspace{-0.2cm}
\end{figure}


\begin{table}[!t]
% \vspace{-0.1in}
\caption{Per-epoch runtime of different systems with different model depths (GCN).}
\scriptsize
\vspace{-0.1in}
\label{table:vary_layer}
\centering
\begin{tabular}{c|c|c|c|c|c|c} 
    % \toprule
    \hline
    % \cline{4}
    \multirow{2}{*}{\textbf{Systems}} & \multicolumn{3}{c|}{\textbf{Products}} & \multicolumn{3}{c}{\textbf{Wiki}} \\ 
    \cline{2-7}
    & 3-layer & 4-layer & 5-layer & 3-layer & 4-layer & 5-layer\\ 
    \hline
    DGL & 28.1 & 55.4 &  114.8& 1669.1 & OOM  & OOM \\ 
    \hline 
    PaGraph & 20.1 & 45.0 & 78.0  & 693.2 & OOM  & OOM \\ 
    \hline 
    DGL-UVA & 21.8 & 41.6 & 71.3& 911.7 & 1782.3 & OOM \\ 
    \hline 
    GNNLab & 5.66 & 10.7 & 22.9 & 384.2 & OOM  & OOM \\ 
    \hline 
    GAS & 26.2 & 30.9 & 35.2 & OOM  & OOM  & OOM  \\ 
    \hline 
    % DSP & OOM & OOM & OOM & OOM  & OOM  &OOM \\ 
    % \hline 
    NeutronOrch & \textbf{2.33} & \textbf{4.58} &\textbf{10.2} & \textbf{221.1} & \textbf{483.4} & \textbf{1852.3}\\ 
    \hline
\end{tabular}
\vspace{-0.1in}
\end{table}


\begin{table}[!t]
\vspace{-0.1in}
\caption{Per-epoch runtime of different systems with different batch sizes. (3-layer GCN)}
\scriptsize
\vspace{-0.1in}
\label{table:vary-batchsize}
\centering
\begin{tabular}{c|c|c|c|c|c|c|c|c} 
    % \toprule
    \hline
    % \cline{4}
    \multirow{3}{*}{\textbf{Systems}} & \multicolumn{4}{c|}{\textbf{Products}} & \multicolumn{4}{c}{\textbf{Wiki}} \\ 
    \cline{2-9}
    &\multicolumn{4}{c|}{Batch size} & \multicolumn{4}{c}{Batch size} \\ 
    \cline{2-9}
    & 256 & 1024 & 4096 & 10000 & 256 & 1024 & 4096 & 10000\\ 
    \hline
    DGL & 35.1 & 28.1 & 11.9& 5.24 & 2104.1 & 1669.1 & 861.9  & OOM \\ 
    \hline 
    PaGraph & 31.6 & 20.1 & 15.5  & 11.9 & 1054.6  & 693.2 & OOM  & OOM\\ 
    \hline 
    DGL-UVA & 30.1 & 21.8 & 8.87 & 4.99 & 1624.5  & 911.7 & 592.4 & 301.6\\ 
    \hline 
    GNNLab & 12.3 & 5.65 &  2.78 & 1.65 & 774.8  & 384.2 & OOM  & OOM \\ 
    \hline 
    GAS & 56.4 & 26.2 & 18.1 & 15.1  & OOM  & OOM  & OOM  & OOM\\ 
    \hline 
    % DSP & OOM & OOM & OOM & OOM  & OOM  &OOM & OOM  & OOM\\ 
    % \hline 
    NeutronOrch & \textbf{4.19}  & \textbf{2.33} & \textbf{1.35} & \textbf{0.71} & \textbf{409.2} & \textbf{221.1} &\textbf{129.9} & \textbf{84.6}\\ 
    \hline
\end{tabular}
\vspace{-0.12in}
\end{table}


On the other hand, the GPU training time is significantly reduced since the CPU-provided hot vertex embeddings can be reused multiple times within the super-batch.
A larger hot vertices ratio facilitates more GPU computation savings. The proportion of hot vertices is proportional to the CPU's computing power. In order to guarantee the bound, the CPU must complete the hot vertices embeddings update of the next super-batch within one super-batch. The proportion of hot vertices that can be supported by different datasets is usually 10\%-30\%. We use the \system with a hot vertices ratio of 0 as a baseline and compare the average GPU training time of one epoch with \system.
Figure \ref{fig:computation volume} presents the observation of GPU training time reduction for GCN algorithms with six datasets. \system reduced GPU training time by an average of 36.5\% on six datasets. 
In addition, the reduction of GPU training time is most significant on Reddit and Orkut, with an average of 48.3\%. This is because they have a larger average degree, which is great for improving hots vertices hits. 

\begin{figure}
  \centering
  \includegraphics[width=8cm,page={1}]{figures/uti-exp.pdf}
   \vspace{-0.13in}
  \caption{GPU utilization and CPU utilization comparison.}
   \vspace{-0.4cm}
  \label{fig:uti-exp}
\end{figure}

\vspace{-0.1in}
\subsection{Analysis of Resource Utilization}
% 对比对象 ：1.在本系统中配置四种计算模式 2.our system
%2.统计四种模式和本系统的CPU与GPU计算资源利用率，证明在异构环境下的新任务分配对资源的充分利用
%3.存储资源利用率怎么对比不知道。
%六个数据集，GCN算法，Graphsage算法
We evaluate the utilization of GPU and CPU resources during the training of GCN on Lj-large and Orkut using five systems: DGL, PaGraph, GNNLab, DGL-UVA, and \system. Figure \ref{fig:uti-exp} presents a comparison of GPU and CPU utilization. GPU and CPU utilization were recorded every 100 milliseconds and averaged over ten epochs. \system exhibits superior utilization of both CPU and GPU resources by scheduling training tasks across CPUs and GPUs. DGL and PaGraph have good CPU utilization and poor GPU utilization. This is because performing a complete sampling or gathering step on the CPU improves CPU utilization but causes the GPU to wait. DGL-UVA and GNNLab have poor CPU utilization and good GPU utilization. They have good performance with sufficient GPU resources, but exploiting CPU resources can further improve performance. 
\system achieves both good CPU and GPU utilization, averaging 44.5\% and 92.9\%, respectively.

% DGL exhibits the second highest CPU utilization, averaging 33.8\%, and the lowest GPU utilization, averaging only 45.3\%. This is because DGL executes the sampling step and feature collection for the gathering step on the CPU.
% In contrast, PaGraph improves GPU utilization (averaging at 65.6\%) by caching features in the GPU to accelerate the gathering step. However, its performance is still hindered by CPU-based sampling and GPU memory contention. GNNlab achieves a substantial improvement in GPU utilization, averaging 76.6\%, through the integration of GPU sampling and cache policies. However, this approach introduces resource contention on the GPU, resulting in performance limitations and increased cache miss features 
% DGL-UVA exhibits high GPU utilization (91.2\%) thanks to GPU sampling and the zero-copy transfer engine. However, it lacks a reduction in transfer volume and introduces GPU resource contention. 
% Since GNNlab and DGL-UVA do not schedule computation on the CPU, they both have a low CPU utilization of averaging 11.4\% and 2.8\%, respectively. In contrast, \system achieves the highest CPU and GPU utilization, averaging 57.5\% and 96.9\%, respectively.


%Meanwhile, compared to PaGraph, GNNlab requires storage graph topology data in the GPU, cache available GPU memory decreases and generates more cache miss features in the CPU. 
% Figure \ref{fig:pipeline design} illustrates the CPU and GPU computation in \system, enabled by hotness-aware layer-based task orchestrating and super-batch pipelined training. This architecture ensures uninterrupted and intensive computations on \system, leading to enhanced utilization of CPU and GPU averaging 57.5\% and 96.9\%, respectively.

%We abstract the computation volume as the number of edges involved in each layer multiplied by the feature or embedding dimension of the corresponding source vertex. 
% 1. 以边为指标，统计整体的计算量 分析节省了多少计算量
% 对比 使用热点嵌入重用 与 不使用 的情况。 也就是热点嵌入重用的计算量和完整的bottom layer计算量相比节省了多少。
% 两个数据集， GCN算法

% \begin{figure}[th]
%   \centering
%   \includegraphics[width=8cm,page={1}]{figures/GAT.pdf}
%      \vspace{-0.4cm}
%   \caption{\notecolor{The performance comparison of different systems with GAT algorithm.}}
%    \vspace{-0.6cm}
%   \label{fig:gat}
% \end{figure}


% \subsection{\notecolor{Generalization of GAT}}
% \notecolor{We run GAT on NeutronOrch, DGL, DGL-UVA, and GNNautoscale (GAS) to demonstrate its effectiveness. Pagraph, GNNlab, and DSP are excluded from the evaluation because they do not support the GAT algorithm. 
% Firstly, we report the per-epoch runtime results in Figure \ref{fig:gat} (a). We can observe that NeutronOrch outperforms other systems, achieving on average 3.07$\times$, 1.75$\times$, and 3.63$\times$ speedup over DGL, DGL-UVA, and GNNautoscale, respectively. In terms of epoch-to-accuracy comparison as shown in Figure \ref{fig:gat} (b), NeutronOrch achieves convergence accuracy nearly on par with DGL ($\leq 0.01$) and outperforms GNNautoscale.}

% \notecolor{
% Compared to DGL and DGL-UVA, NeutronOrch can reduce computation and communication overhead by using historical embeddings (HE).
% Compared to GNNautoscale, which uses HE, NeutronOrch can save memory consumption and alleviate accuracy losses by strictly controlling 
% versions through the super-batch pipeline.
% }


\subsection{Varying Model Configurations}

\Paragraph{Performance with varying model depths.}
As the model depth increases, the effectiveness of NeutronOrch will not decrease, although NeutronOrch only offloads the lowest-level calculations to the CPU. In sample-based GNN training, the number of vertices exhibits exponential growth across layers \cite{Graphsage_2017,FASTGcn_iclr_2018}. As a result, more than half of the entire training workload comes from the bottom layer. For example, on Wiki dataset with a 3-layer model, the bottom layer has 1.94M vertices, occupying 65\% of the total workload, while the other layers only have 1.04M vertices in total. For the 4-layer model and 5-layer model, the bottom layer occupies 61\% and 59\% of the total workload, respectively. 
We run a GCN model on all systems and report the per-epoch runtime in Table \ref{table:vary_layer}, with the model depth ranging from 3 to 5. For the 3-layer model, NeutronOrch achieves on average 6.43$\times$ speedup over the different systems. For the 4-layer model and 5-layer model, the speedups are 5.84$\times$ and 6.31$\times$, respectively.


\Paragraph{Performance with varying batch sizes.}
To study the effectiveness of NeutronOrch under different batch sizes, we run a 3-layer GCN model on all systems and report the per-epoch runtime in Table \ref{table:vary-batchsize}, with the batch size ranging from 256 to 10000.
For a batch size of 256, NeutronOrch achieves on average 5.64$\times$ speedups over different systems. For batch sizes of 1024, 4096, and 10000, the speedups are 6.43$\times$, 9.85$\times$, and 7.25$\times$, respectively. The setting of batch size will not affect the effectiveness. Furthermore, on large-scale graphs, NeutronOrch's CPU offloading efficiently mitigates the increasing memory overhead when using large batch sizes.



%第一个时间，一整个的bottom layer 的CPU计算 算完给GPU
%第二个时间，只有hot vertice 的CPU计算，GPU按需的拿
%第三个时间，多流的
%依次开启，新的CPU-GPU任务摆放，热点嵌入重用，看性能提升
%六个数据集，GCN算法

%分析性能提升来源
%我们系统的性能优势应该来自于以下三点，重要性按序号排序： 1.更充分的异构资源利用率 2.更少的传输量 3.更小的计算量 


\begin{figure}
  \centering
  \includegraphics[width=8cm,page={1}]{figures/acc-new.pdf}
     \vspace{-0.4cm}
  \caption{Epoch-to-accuracy}
   \vspace{-0.6cm}
  \label{fig:acc}
\end{figure}

\vspace{-0.1cm}
\label{sec:acc}
\subsection{Training Convergence}
\label{train_convergence}
%验证方法的time to accrcuy 
%一个数据集，GCN算法，GAT算法，对比CPU中不同热点个数，对准确率收敛的影响
%证明调节热点个数，可以改善embedding版本陈旧情况
% Benefit from strict control of bounded staleness by super-batch pipelined training, \system optimally utilizes heterogeneous resources without compromising the accuracy of GNNs. Stale bound in the \system depends on the number of batches in the super-batch. In actual training, we find that using a larger bound benefits the overall performance, but the accuracy rise slows down \cite{SANCUS_VLDB_2022,ST_ICML_2018,ssp_2013_nips}. The tradeoff between system performance and model convergence is difficult to choose. Moreover, differences in applications, datasets, implementations, and configurations may affect the practical staleness in the GNN training as well. In our practical tests, we find that setting the number of batches between 4 and 8 provides an acceptable balance between runtime and errors.
We plot the epoch-to-accuracy curve on different systems for both GCN and GAT algorithms.
Benefiting from the strict version control based on super-batch pipelining, \system accelerates GNN training while maintaing high accuracy. As shown in Figure \ref{fig:acc}, compared to other GNN systems that use historical embeddings but do not strictly control versions (e.g., GAS\cite{GNNautoscale_icml_2021}), NeutronOrch achieves higher accuracy due to the ability to achieve smaller cumulative errors across batches. In comparison to systems that do not use historical embeddings\cite{DGL_CORR_2019}, NeutronOrch achieves comparable convergence with an accuracy loss of no more than 1\%.


% With the super-batch pipeline design, NeutronOrch can strictly control the version gap of historical embedding between two adjacent super-batches. As shown in Figure \ref{fig:acc}, NeutronOrch achieves a convergence to nearly the same accuracy ( $\leq 0.01$) as the other systems. GNNauotoscale is a representative system based on historical embedding. To improve the scalability of GNN training, GAS uses historical embedding to reduce memory consumption during training.
% However, it does not specify the range of vertices using historical embeddings and also does not guarantee that historical embeddings will always be updated within a given version gap (batch num). NeutronOrch achieves higher accuracy than GAS on every dataset. In the largest dataset, Ogbn-paper100M, GAS has the most significant decrease in accuracy. This is because in large datasets, each epoch contains more batches, i.e., more model parameter updates, and if the update of the historical embedding is not actively controlled, it may fall behind by more version gaps.}
% We evaluate the validation accuracy of training GCN on the Reddit dataset using DGL, PaGraph, GNNlab, DGL-UVA, and \system with four batches in a super-batch. As shown in Figure \ref{fig:acc}, \system achieves a convergence to nearly the same accuracy ($\leq 0.005$) as the other systems. Furthermore, the convergence time required to achieve satisfactory accuracy is significantly reduced. In summary, based on the bounded staleness technique, we conclude that the performance of the proposed training system is remarkably similar to the original GNN training, as demonstrated in previous studies such as SANCUS \cite{SANCUS_VLDB_2022}, VR-GCN \cite{ST_ICML_2018} and Marius \cite{Marius_OSDI_2021}. 
%We did not perform similar experiments on datasets other than Reddit due to the random initialization of vertex features in those datasets.

%多GPU的扩展性感觉不好，因为在单机多卡里，有很多GPU，CPU能提供的帮助有限。
%所有，做在单GPU内的训练扩展，因为我们的方法在相同图数据，相同batchsize，相同model（hidden layer size）下，消耗更少GPU内存。

% \subsection{Scale GNN Training With One GPU}
%逐步扩大feature维度，提高batchsize，使用更大的图结构数据。在单GPU下，我们的系统能训练，其他的系统会oom。
%两个图数据，其中一个图拓扑要大。 然后观察扩大feature维度，提高batchsize后，对比性能，以及是否能正常训练，其他系统是否会oom
%GCN算法


 \vspace{-0.2cm}
\section{Related Work}
\label{sec6}
% The topic of how to effectively enable GNN training has recently gotten more and more attention \cite{PYG_corr_2019,DGL_CORR_2019, neugraph_atc_2019,DBLP:journals/pvldb/ZhuZYLZALZ19, Seastar_EuroSys_2021,p3_osdi_2021,DGCL_eurosys_2021,distdgl_ieee_2020,ROC_mlsys_2020,PAGRAPH_SOCC_2020,PYTDIRECT_VLDB_2021,neutron_sigmod_2022,Ginex_vldb_2022, GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,ByteGNN_VLDB_2022,Marius++_arxiv_2022,lazygcn_2020_neuips,wholegraph_SC_2022,SANCUS_VLDB_2022,PCG_VLDB_2022,Sgs_2022,TurboGNN_TOC_2023,refresh_arxiv_2023,ducati_sigmod_2023}. The two major categories of existing solutions are full-batch training and sample-based mini-batch training.
% 原来的引用：\cite{neugraph_atc_2019,neutron_sigmod_2022,DGCL_eurosys_2021,distdgl_ieee_2020,ROC_mlsys_2020,PAGRAPH_SOCC_2020,PYTDIRECT_VLDB_2021,DGL_CORR_2019,GNNlab_EUROSYS_2022,DATATIRING_VLDB_2022,Marius++_arxiv_2022,lazygcn_2020_neuips,wholegraph_SC_2022,TurboGNN_TOC_2023}.
% 增加的引用：
% PYG_corr_2019， DBLP:journals/pvldb/ZhuZYLZALZ19（aligraph),  Seastar_EuroSys_2021,  p3_osdi_2021,  Ginex_vldb_2022, ByteGNN_VLDB_2022,  SANCUS_VLDB_2022, PCG_VLDB_2022, Sgs_2022, ducati
% Full-batch GNN training divides the large graph into partitions and trains GNN models concurrently with many workers.
%NeuGraph,DistDGL,roc,DGCL,neutronstar.
% \Paragraph{Full-batch GNN training.} A set of GNN systems \cite{neugraph_atc_2019,neutron_sigmod_2022,DGCL_eurosys_2021,distdgl_ieee_2020,ROC_mlsys_2020} adopt full-batch GNN training to guarantee high accuracy. NeuGraph \cite{neugraph_atc_2019} defines a new, flexible SAGA-NN model to express GNNs. It fuses graph operations into TensorFlow and supports training on multiple GPUs. %To reduce and balance the communication, DistDGL \cite{distdgl_ieee_2020} leverages METIS partitioning algorithms and asynchronous GNN training.
% ROC \cite{ROC_mlsys_2020} optimizes graph partitioning with online learning while managing memory with dynamic programming. For effective GNN training on several GPUs, DGCL \cite{DGCL_eurosys_2021} designs a communication planning algorithm that avoids conflict on various links. Neutronstar \cite{neutron_sigmod_2022} designs hybrid dependency management for a more balanced use of communication and computational resources.

%GraphSage,pagraph,PyTorch-direct,DGL, GNNlab,data tiering,mariusGNN,lazyGCN,whole graph,turboGNN By using a mini-batcher, sample-based GNN training can scale to large graphs and has shown accuracy competitive with full-batch training   in GNN, such as sum, max, pool, average, and LSTM\cite{ST_ICML_2018,sampling_caa_2022,GNNSampling_sigops_2021}.
\Paragraph{Sample-based mini-batch GNN training.}  GraphSage \cite{Graphsage_2017} first introduces the fixed-number vertex sampling and proposes the general message aggregation methods. Past works mainly focused on solving the sampling and gathering bottleneck in sample-based GNN training. For the sampling step, LazyGCN \cite{lazygcn_2020_neuips} proposes a sample graph reusing method to reduce sampling overhead and maintain accuracy. MariusGNN \cite{Marius++_arxiv_2022} design the DENSE data structure to minimize redundant computation in multi-hop sampling. On the other hand, Nextdoor \cite{Nextdoor_eurosys_2021}, Wholegraph \cite{wholegraph_SC_2022}, and TurboGNN \cite{TurboGNN_TOC_2023} propose the GPU-based sampling method to accelerate the sampling step. For the gathering step, a series of feature and graph caching strategies are proposed. Data tiering \cite{DATATIRING_VLDB_2022}, and Quiver \cite{quiver_2023_arxiv} design various efficient GPU feature cache strategies to minimize CPU-GPU communication. Ducati \cite{ducati_sigmod_2023} further designs the graph topology cache method to fully utilize the GPU memory.
% Pytorch-direct \cite{PYTDIRECT_VLDB_2021} makes GPU threads directly access sparse features in host memory through on-demand zero-copy accesses. 


% \Paragraph{Historical embeddings reusing.}
% Existing works \cite{ST_ICML_2018,SANCUS_VLDB_2022,GNNautoscale_icml_2021,refresh_arxiv_2023} have shown the benefits of historical embedding (HE) reuse for training acceleration without compromising accuracy. VR-GCN \cite{ST_ICML_2018} uses HE in GNN training to reduce the variance and further control the number of nodes to sample. Refresh \cite{refresh_arxiv_2023} uses a combination of gradient-based and staleness criteria to reduce estimation errors. SANCUS \cite{SANCUS_VLDB_2022} reuses HE for boundary vertices to reduce communication in distributed GNN training. These frameworks focus on optimizing scenarios where computation is exclusively deployed on GPUs. In contrast, our work extends HE reuse to the CPU-GPU environment, achieving both high performance and scalability.

% GNNLab \cite{GNNlab_EUROSYS_2022} proposes a pre-sampling-based cache policy to accommodate different graph datasets and sampling algorithms. Data tiering \cite{DATATIRING_VLDB_2022} considers the multi-hop neighbor information of the vertex by the Reverse-PageRank when selecting cache vertices. 


\Paragraph{Task orchestrating in DNN training.} 
Zero-Infinity DeepSpeed (ZID) \cite{ZID_2020} incorporates CPU offload techniques \cite{ZID_1_SC_2020,ZID_2_SC_2021,ZID_ATC_2021} to optimize the DNN training under heterogeneous environments. Although ZID and our design share similarities in scheduling training tasks across CPU and GPUs to improve scalability, their optimization objectives are different.
ZID primarily focuses on reducing the memory consumption of model parameters, reducing memory consumption by offloading model parameters to the CPU. In DNNs, model parameters which consist of dense matrices that can be disjointly partitioned. These slices can be efficiently communicated between the CPU and GPUs because of their dense and regular data access patterns. 
In contrast, NeutronOrch primarily focuses on the efficiency of sample-based GNN training, minimizing communication and computation by offloading historical embedding computation to the CPU. 
Sample-based GNN training involves multiple data preparation stages (sample and gather). It exhibits irregular vertex access patterns due to the inherent complexity of graph structures, which pose new challenges in optimizing the data I/O between CPU and GPUs. ZID didn't address these challenges. However, NeutronOrch effectively resolves these problems by its task orchestrating method, transferring and reusing historical embedding for frequently accessed vertices. 

\vspace{-0.2cm}
\section{Conclusion}
\label{sec7}
We present \system, a scalable and efficient GNN training system that fully utilizes CPU and GPUs. 
\system leverages two key components to achieve its performance, including a hotness-aware layer-based task orchestrating method that combines CPU computation offloading with historical embeddings reuse to optimize computation and communication and a super-batch pipeline training method that utilizes CPU-GPU pipelining to achieve efficient and staleness-bounded version control. 
Our experiments demonstrate that \system efficiently accelerates mini-batch GNN training with an accuracy loss of no more than 1\%.

% for training sample-based mini-batch GNNs under CPU-GPU heterogeneous environments
% The effectiveness of \system and its high performance are contributed by several system components, including a layer-based task orchestrating method, hotness-aware embedding reusing, and super-batch pipeline scheduling.
% on billion-scale graphs using just 4 GPUs by fully utilizing CPU, GPU, and interconnects.


% Compared with existing mini-batch GNN systems DGL \cite{DGL_CORR_2019}, PaGraph \cite{PAGRAPH_SOCC_2020}, GNNAutoScale \cite{GNNautoscale_icml_2021}, GNNLab \cite{GNNlab_EUROSYS_2022}, and DSP \cite{DSP_PPOPP_2023}, the speedups of \system range from 1.20 $\times$ to 4.91$\times$ by achieving a balanced computation resource utilization.
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].

\begin{thebibliography}{52}

%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command.  Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}.  The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip}   % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip}           % plain TeX syntax
%%%
%%% ====================================================================

\ifx \showCODEN    \undefined \def \showCODEN     #1{\unskip}     \fi
\ifx \showDOI      \undefined \def \showDOI       #1{#1}\fi
\ifx \showISBNx    \undefined \def \showISBNx     #1{\unskip}     \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii  #1{\unskip}     \fi
\ifx \showISSN     \undefined \def \showISSN      #1{\unskip}     \fi
\ifx \showLCCN     \undefined \def \showLCCN      #1{\unskip}     \fi
\ifx \shownote     \undefined \def \shownote      #1{#1}          \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1}   \fi
\ifx \showURL      \undefined \def \showURL       {\relax}        \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}

\bibitem[\protect\citeauthoryear{Backstrom, Huttenlocher, Kleinberg, and
  Lan}{Backstrom et~al\mbox{.}}{2006}]%
        {LIVE_KDD_2006}
\bibfield{author}{\bibinfo{person}{Lars Backstrom}, \bibinfo{person}{Daniel~P.
  Huttenlocher}, \bibinfo{person}{Jon~M. Kleinberg}, {and}
  \bibinfo{person}{Xiangyang Lan}.} \bibinfo{year}{2006}\natexlab{}.
\newblock \showarticletitle{Group formation in large social networks:
  membership, growth, and evolution}. In \bibinfo{booktitle}{\emph{Proceedings
  of the Twelfth {ACM} {SIGKDD} International Conference on Knowledge Discovery
  and Data Mining, KDD'06, Philadelphia, PA, USA}}. \bibinfo{pages}{44--54}.
\newblock


\bibitem[\protect\citeauthoryear{Bastings, Titov, Aziz, Marcheggiani, and
  Sima'an}{Bastings et~al\mbox{.}}{2017}]%
        {NMT_EMNLP_2017}
\bibfield{author}{\bibinfo{person}{Jasmijn Bastings}, \bibinfo{person}{Ivan
  Titov}, \bibinfo{person}{Wilker Aziz}, \bibinfo{person}{Diego Marcheggiani},
  {and} \bibinfo{person}{Khalil Sima'an}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Graph Convolutional Encoders for Syntax-aware
  Neural Machine Translation}. In \bibinfo{booktitle}{\emph{Proceedings of the
  2017 Conference on Empirical Methods in Natural Language Processing,
  EMNLP'17, Copenhagen, Denmark}}. \bibinfo{publisher}{Association for
  Computational Linguistics}, \bibinfo{pages}{1957--1967}.
\newblock


\bibitem[\protect\citeauthoryear{Cai, Zhou, Yan, Zheng, Song, Zheng, Cheng, and
  Karypis}{Cai et~al\mbox{.}}{2023}]%
        {DSP_PPOPP_2023}
\bibfield{author}{\bibinfo{person}{Zhenkun Cai}, \bibinfo{person}{Qihui Zhou},
  \bibinfo{person}{Xiao Yan}, \bibinfo{person}{Da Zheng},
  \bibinfo{person}{Xiang Song}, \bibinfo{person}{Chenguang Zheng},
  \bibinfo{person}{James Cheng}, {and} \bibinfo{person}{George Karypis}.}
  \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{{DSP:} Efficient {GNN} Training with Multiple
  GPUs}. In \bibinfo{booktitle}{\emph{Proceedings of the 28th {ACM} {SIGPLAN}
  Annual Symposium on Principles and Practice of Parallel Programming,
  PPoPP'23, Montreal, QC, Canada}}. \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{392--404}.
\newblock


\bibitem[\protect\citeauthoryear{Chen, Ma, and Xiao}{Chen
  et~al\mbox{.}}{2018a}]%
        {FASTGcn_iclr_2018}
\bibfield{author}{\bibinfo{person}{Jie Chen}, \bibinfo{person}{Tengfei Ma},
  {and} \bibinfo{person}{Cao Xiao}.} \bibinfo{year}{2018}\natexlab{a}.
\newblock \showarticletitle{FastGCN: Fast Learning with Graph Convolutional
  Networks via Importance Sampling}. In \bibinfo{booktitle}{\emph{6th
  International Conference on Learning Representations, ICLR'18, Vancouver, BC,
  Canada}}. \bibinfo{publisher}{OpenReview.net}.
\newblock


\bibitem[\protect\citeauthoryear{Chen, Zhu, and Song}{Chen
  et~al\mbox{.}}{2018b}]%
        {ST_ICML_2018}
\bibfield{author}{\bibinfo{person}{Jianfei Chen}, \bibinfo{person}{Jun Zhu},
  {and} \bibinfo{person}{Le Song}.} \bibinfo{year}{2018}\natexlab{b}.
\newblock \showarticletitle{Stochastic Training of Graph Convolutional Networks
  with Variance Reduction}. In \bibinfo{booktitle}{\emph{Proceedings of the
  35th International Conference on Machine Learning, ICML'18,
  Stockholmsm{\"{a}}ssan, Stockholm, Sweden}}
  \emph{(\bibinfo{series}{Proceedings of Machine Learning Research})},
  Vol.~\bibinfo{volume}{80}. \bibinfo{publisher}{{PMLR}},
  \bibinfo{pages}{941--949}.
\newblock


\bibitem[\protect\citeauthoryear{El{-}Kishky, Bronstein, Xiao, and
  Haghighi}{El{-}Kishky et~al\mbox{.}}{2022}]%
        {Recom_kdd_22}
\bibfield{author}{\bibinfo{person}{Ahmed El{-}Kishky},
  \bibinfo{person}{Michael~M. Bronstein}, \bibinfo{person}{Ying Xiao}, {and}
  \bibinfo{person}{Aria Haghighi}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Graph-based Representation Learning for Web-scale
  Recommender Systems}. In \bibinfo{booktitle}{\emph{The 28th {ACM} {SIGKDD}
  Conference on Knowledge Discovery and Data Mining, KDD'22, Washington, DC,
  USA}}, \bibfield{editor}{\bibinfo{person}{Aidong Zhang} {and}
  \bibinfo{person}{Huzefa Rangwala}} (Eds.). \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{4784--4785}.
\newblock


\bibitem[\protect\citeauthoryear{Fan, Ma, Li, He, Zhao, Tang, and Yin}{Fan
  et~al\mbox{.}}{2019}]%
        {REC_WWW_2019}
\bibfield{author}{\bibinfo{person}{Wenqi Fan}, \bibinfo{person}{Yao Ma},
  \bibinfo{person}{Qing Li}, \bibinfo{person}{Yuan He},
  \bibinfo{person}{Yihong~Eric Zhao}, \bibinfo{person}{Jiliang Tang}, {and}
  \bibinfo{person}{Dawei Yin}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Graph Neural Networks for Social Recommendation}.
  In \bibinfo{booktitle}{\emph{The World Wide Web Conference, WWW'19, San
  Francisco, CA, USA}}. \bibinfo{publisher}{{ACM}}, \bibinfo{pages}{417--426}.
\newblock


\bibitem[\protect\citeauthoryear{Fey and Lenssen}{Fey and Lenssen}{2019}]%
        {PYG_corr_2019}
\bibfield{author}{\bibinfo{person}{Matthias Fey} {and}
  \bibinfo{person}{Jan~Eric Lenssen}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Fast Graph Representation Learning with PyTorch
  Geometric}.
\newblock \bibinfo{journal}{\emph{CoRR}}  \bibinfo{volume}{abs/1903.02428}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Fey, Lenssen, Weichert, and Leskovec}{Fey
  et~al\mbox{.}}{2021}]%
        {GNNautoscale_icml_2021}
\bibfield{author}{\bibinfo{person}{Matthias Fey}, \bibinfo{person}{Jan~Eric
  Lenssen}, \bibinfo{person}{Frank Weichert}, {and} \bibinfo{person}{Jure
  Leskovec}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{GNNAutoScale: Scalable and Expressive Graph Neural
  Networks via Historical Embeddings}. In \bibinfo{booktitle}{\emph{Proceedings
  of the 38th International Conference on Machine Learning, {ICML} 2021, 18-24
  July 2021, Virtual Event}} \emph{(\bibinfo{series}{Proceedings of Machine
  Learning Research})}, \bibfield{editor}{\bibinfo{person}{Marina Meila} {and}
  \bibinfo{person}{Tong Zhang}} (Eds.), Vol.~\bibinfo{volume}{139}.
  \bibinfo{publisher}{{PMLR}}, \bibinfo{pages}{3294--3304}.
\newblock


\bibitem[\protect\citeauthoryear{Gong, Ji, Yao, Fletcher, Hughes, and
  Torrellas}{Gong et~al\mbox{.}}{2022}]%
        {graphite_isca_2022}
\bibfield{author}{\bibinfo{person}{Zhangxiaowen Gong},
  \bibinfo{person}{Houxiang Ji}, \bibinfo{person}{Yao Yao},
  \bibinfo{person}{Christopher~W. Fletcher}, \bibinfo{person}{Christopher~J.
  Hughes}, {and} \bibinfo{person}{Josep Torrellas}.}
  \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Graphite: optimizing graph neural networks on CPUs
  through cooperative software-hardware techniques}. In
  \bibinfo{booktitle}{\emph{The 49th Annual International Symposium on Computer
  Architecture, ISCA'22, New York, USA}}. \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{916--931}.
\newblock


\bibitem[\protect\citeauthoryear{Hamilton, Ying, and Leskovec}{Hamilton
  et~al\mbox{.}}{2017a}]%
        {rep_ieee_2017}
\bibfield{author}{\bibinfo{person}{William~L. Hamilton}, \bibinfo{person}{Rex
  Ying}, {and} \bibinfo{person}{Jure Leskovec}.}
  \bibinfo{year}{2017}\natexlab{a}.
\newblock \showarticletitle{Representation Learning on Graphs: Methods and
  Applications}.
\newblock \bibinfo{journal}{\emph{{IEEE} Data Eng. Bull.}}
  \bibinfo{volume}{40}, \bibinfo{number}{3} (\bibinfo{year}{2017}),
  \bibinfo{pages}{52--74}.
\newblock


\bibitem[\protect\citeauthoryear{Hamilton, Ying, and Leskovec}{Hamilton
  et~al\mbox{.}}{2017b}]%
        {Graphsage_2017}
\bibfield{author}{\bibinfo{person}{William~L. Hamilton},
  \bibinfo{person}{Zhitao Ying}, {and} \bibinfo{person}{Jure Leskovec}.}
  \bibinfo{year}{2017}\natexlab{b}.
\newblock \showarticletitle{Inductive Representation Learning on Large Graphs}.
  In \bibinfo{booktitle}{\emph{Advances in Neural Information Processing
  Systems 30: Annual Conference on Neural Information Processing Systems 2017,
  NeurIPS'17 Long Beach, CA, {USA}}}. \bibinfo{pages}{1024--1034}.
\newblock


\bibitem[\protect\citeauthoryear{Ho, Cipar, Cui, Lee, Kim, Gibbons, Gibson,
  Ganger, and Xing}{Ho et~al\mbox{.}}{2013}]%
        {ssp_2013_nips}
\bibfield{author}{\bibinfo{person}{Qirong Ho}, \bibinfo{person}{James Cipar},
  \bibinfo{person}{Henggang Cui}, \bibinfo{person}{Seunghak Lee},
  \bibinfo{person}{Jin~Kyu Kim}, \bibinfo{person}{Phillip~B. Gibbons},
  \bibinfo{person}{Garth~A. Gibson}, \bibinfo{person}{Gregory~R. Ganger}, {and}
  \bibinfo{person}{Eric~P. Xing}.} \bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{More Effective Distributed {ML} via a Stale
  Synchronous Parallel Parameter Server}. In \bibinfo{booktitle}{\emph{Advances
  in Neural Information Processing Systems 26: 27th Annual Conference on Neural
  Information Processing Systems 2013. NeurIPS'12, Lake Tahoe, Nevada, United
  States}}. \bibinfo{pages}{1223--1231}.
\newblock


\bibitem[\protect\citeauthoryear{Hu, Fey, Zitnik, Dong, Ren, Liu, Catasta, and
  Leskovec}{Hu et~al\mbox{.}}{2020}]%
        {ogb_2020_neurIPS}
\bibfield{author}{\bibinfo{person}{Weihua Hu}, \bibinfo{person}{Matthias Fey},
  \bibinfo{person}{Marinka Zitnik}, \bibinfo{person}{Yuxiao Dong},
  \bibinfo{person}{Hongyu Ren}, \bibinfo{person}{Bowen Liu},
  \bibinfo{person}{Michele Catasta}, {and} \bibinfo{person}{Jure Leskovec}.}
  \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Open Graph Benchmark: Datasets for Machine Learning
  on Graphs}. In \bibinfo{booktitle}{\emph{Advances in Neural Information
  Processing Systems 33: Annual Conference on Neural Information Processing
  Systems 2020, NeurIPS'20, December 6-12}}.
\newblock


\bibitem[\protect\citeauthoryear{Huang, Jiang, Wang, Xiao, Wipf, Song, Gan,
  Huang, Zhai, and Zhang}{Huang et~al\mbox{.}}{2023}]%
        {refresh_arxiv_2023}
\bibfield{author}{\bibinfo{person}{Kezhao Huang}, \bibinfo{person}{Haitian
  Jiang}, \bibinfo{person}{Minjie Wang}, \bibinfo{person}{Guangxuan Xiao},
  \bibinfo{person}{David Wipf}, \bibinfo{person}{Xiang Song},
  \bibinfo{person}{Quan Gan}, \bibinfo{person}{Zengfeng Huang},
  \bibinfo{person}{Jidong Zhai}, {and} \bibinfo{person}{Zheng Zhang}.}
  \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{ReFresh: Reducing Memory Access from Exploiting
  Stable Historical Embeddings for Graph Neural Network Training}.
\newblock \bibinfo{journal}{\emph{CoRR}}  \bibinfo{volume}{abs/2301.07482}
  (\bibinfo{year}{2023}).
\newblock


\bibitem[\protect\citeauthoryear{Huang, Zhai, Zheng, Yi, and Shen}{Huang
  et~al\mbox{.}}{2021a}]%
        {GNNGPU_PPOPP_2021}
\bibfield{author}{\bibinfo{person}{Kezhao Huang}, \bibinfo{person}{Jidong
  Zhai}, \bibinfo{person}{Zhen Zheng}, \bibinfo{person}{Youngmin Yi}, {and}
  \bibinfo{person}{Xipeng Shen}.} \bibinfo{year}{2021}\natexlab{a}.
\newblock \showarticletitle{Understanding and bridging the gaps in current
  {GNN} performance optimizations}. In \bibinfo{booktitle}{\emph{26th {ACM}
  {SIGPLAN} Symposium on Principles and Practice of Parallel Programming, PPoPP
  '21, Virtual Event, Republic of Korea}}. \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{119--132}.
\newblock


\bibitem[\protect\citeauthoryear{Huang, Zhai, Zheng, Yi, and Shen}{Huang
  et~al\mbox{.}}{2021b}]%
        {gnngaps_ppopp_2021}
\bibfield{author}{\bibinfo{person}{Kezhao Huang}, \bibinfo{person}{Jidong
  Zhai}, \bibinfo{person}{Zhen Zheng}, \bibinfo{person}{Youngmin Yi}, {and}
  \bibinfo{person}{Xipeng Shen}.} \bibinfo{year}{2021}\natexlab{b}.
\newblock \showarticletitle{Understanding and bridging the gaps in current
  {GNN} performance optimizations}. In \bibinfo{booktitle}{\emph{26th {ACM}
  {SIGPLAN} Symposium on Principles and Practice of Parallel Programming,
  Virtual Event, PPoPP '21, Republic of Korea}}. \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{119--132}.
\newblock


\bibitem[\protect\citeauthoryear{Ilyas, Rekatsinas, Konda, Pound, Qi, and
  Soliman}{Ilyas et~al\mbox{.}}{2022}]%
        {SAGA_SIGMOD_2022}
\bibfield{author}{\bibinfo{person}{Ihab~F. Ilyas}, \bibinfo{person}{Theodoros
  Rekatsinas}, \bibinfo{person}{Vishnu Konda}, \bibinfo{person}{Jeffrey Pound},
  \bibinfo{person}{Xiaoguang Qi}, {and} \bibinfo{person}{Mohamed~A. Soliman}.}
  \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Saga: {A} Platform for Continuous Construction and
  Serving of Knowledge at Scale}. In \bibinfo{booktitle}{\emph{International
  Conference on Management of Data, SIGMOD'22, Philadelphia, PA, USA}}.
  \bibinfo{publisher}{{ACM}}, \bibinfo{pages}{2259--2272}.
\newblock


\bibitem[\protect\citeauthoryear{Jangda, Polisetty, Guha, and Serafini}{Jangda
  et~al\mbox{.}}{2021}]%
        {Nextdoor_eurosys_2021}
\bibfield{author}{\bibinfo{person}{Abhinav Jangda}, \bibinfo{person}{Sandeep
  Polisetty}, \bibinfo{person}{Arjun Guha}, {and} \bibinfo{person}{Marco
  Serafini}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Accelerating graph sampling for graph machine
  learning using GPUs}. In \bibinfo{booktitle}{\emph{Sixteenth European
  Conference on Computer Systems, EuroSys '21, Online Event, United Kingdom}},
  \bibfield{editor}{\bibinfo{person}{Antonio Barbalace},
  \bibinfo{person}{Pramod Bhatotia}, \bibinfo{person}{Lorenzo Alvisi}, {and}
  \bibinfo{person}{Cristian Cadar}} (Eds.). \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{311--326}.
\newblock


\bibitem[\protect\citeauthoryear{Jia, Lin, Gao, Zaharia, and Aiken}{Jia
  et~al\mbox{.}}{2020}]%
        {ROC_mlsys_2020}
\bibfield{author}{\bibinfo{person}{Zhihao Jia}, \bibinfo{person}{Sina Lin},
  \bibinfo{person}{Mingyu Gao}, \bibinfo{person}{Matei Zaharia}, {and}
  \bibinfo{person}{Alex Aiken}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Improving the Accuracy, Scalability, and
  Performance of Graph Neural Networks with Roc}. In
  \bibinfo{booktitle}{\emph{Proceedings of Machine Learning and Systems 2020,
  MLSys'20, Austin, TX, USA}}, \bibfield{editor}{\bibinfo{person}{Inderjit~S.
  Dhillon}, \bibinfo{person}{Dimitris~S. Papailiopoulos}, {and}
  \bibinfo{person}{Vivienne Sze}} (Eds.). \bibinfo{publisher}{mlsys.org}.
\newblock


\bibitem[\protect\citeauthoryear{Kipf and Welling}{Kipf and Welling}{2017}]%
        {GCN_iclr_2017}
\bibfield{author}{\bibinfo{person}{Thomas~N. Kipf} {and} \bibinfo{person}{Max
  Welling}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Semi-Supervised Classification with Graph
  Convolutional Networks}. In \bibinfo{booktitle}{\emph{5th International
  Conference on Learning Representations, ICLR'17, Toulon, France, Conference
  Track Proceedings}}. \bibinfo{publisher}{OpenReview.net}.
\newblock


\bibitem[\protect\citeauthoryear{Kunegis}{Kunegis}{2013}]%
        {konect_www_2013}
\bibfield{author}{\bibinfo{person}{J{\'{e}}r{\^{o}}me Kunegis}.}
  \bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{{KONECT:} the Koblenz network collection}. In
  \bibinfo{booktitle}{\emph{22nd International World Wide Web Conference,
  WWW'13, Rio de Janeiro, Brazil}}. \bibinfo{publisher}{International World
  Wide Web Conferences Steering Committee / {ACM}},
  \bibinfo{pages}{1343--1350}.
\newblock


\bibitem[\protect\citeauthoryear{Lin, Li, Miao, Liu, and Xu}{Lin
  et~al\mbox{.}}{2020}]%
        {PAGRAPH_SOCC_2020}
\bibfield{author}{\bibinfo{person}{Zhiqi Lin}, \bibinfo{person}{Cheng Li},
  \bibinfo{person}{Youshan Miao}, \bibinfo{person}{Yunxin Liu}, {and}
  \bibinfo{person}{Yinlong Xu}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{PaGraph: Scaling {GNN} training on large graphs via
  computation-aware caching}. In \bibinfo{booktitle}{\emph{{ACM} Symposium on
  Cloud Computing, SoCC'20, Virtual Event, USA}}. \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{401--415}.
\newblock


\bibitem[\protect\citeauthoryear{Maass, Min, Kashyap, Kang, Kumar, and
  Kim}{Maass et~al\mbox{.}}{2017}]%
        {MOSAIC_EUROSYS_2017}
\bibfield{author}{\bibinfo{person}{Steffen Maass}, \bibinfo{person}{Changwoo
  Min}, \bibinfo{person}{Sanidhya Kashyap}, \bibinfo{person}{Woon{-}Hak Kang},
  \bibinfo{person}{Mohan Kumar}, {and} \bibinfo{person}{Taesoo Kim}.}
  \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Mosaic: Processing a Trillion-Edge Graph on a
  Single Machine}. In \bibinfo{booktitle}{\emph{Proceedings of the Twelfth
  European Conference on Computer Systems, EuroSys'17, Belgrade, Serbia}}.
  \bibinfo{publisher}{{ACM}}, \bibinfo{pages}{527--543}.
\newblock


\bibitem[\protect\citeauthoryear{Microsoft}{Microsoft}{2020}]%
        {ZID_2020}
\bibfield{author}{\bibinfo{person}{Microsoft}.}
  \bibinfo{year}{2020}\natexlab{}.
\newblock \bibinfo{title}{Extreme-scale model training for everyone.}
\newblock
  \bibinfo{howpublished}{\url{https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scalemodel-training-for-everyone}}.
\newblock


\bibitem[\protect\citeauthoryear{Min, Wu, Hidayetoglu, Xiong, Song, and
  Hwu}{Min et~al\mbox{.}}{2022}]%
        {DATATIRING_VLDB_2022}
\bibfield{author}{\bibinfo{person}{Seungwon Min}, \bibinfo{person}{Kun Wu},
  \bibinfo{person}{Mert Hidayetoglu}, \bibinfo{person}{Jinjun Xiong},
  \bibinfo{person}{Xiang Song}, {and} \bibinfo{person}{Wen{-}Mei Hwu}.}
  \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Graph Neural Network Training and Data Tiering}. In
  \bibinfo{booktitle}{\emph{The 28th {ACM} {SIGKDD} Conference on Knowledge
  Discovery and Data Mining, KDD'22, Washington, DC, USA}}.
  \bibinfo{publisher}{{ACM}}, \bibinfo{pages}{3555--3565}.
\newblock


\bibitem[\protect\citeauthoryear{Min, Wu, Huang, Hidayetoglu, Xiong, Ebrahimi,
  Chen, and Hwu}{Min et~al\mbox{.}}{2021}]%
        {PYTDIRECT_VLDB_2021}
\bibfield{author}{\bibinfo{person}{Seungwon Min}, \bibinfo{person}{Kun Wu},
  \bibinfo{person}{Sitao Huang}, \bibinfo{person}{Mert Hidayetoglu},
  \bibinfo{person}{Jinjun Xiong}, \bibinfo{person}{Eiman Ebrahimi},
  \bibinfo{person}{Deming Chen}, {and} \bibinfo{person}{Wen{-}mei~W. Hwu}.}
  \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Large Graph Convolutional Network Training with
  GPU-Oriented Data Communication Architecture}.
\newblock \bibinfo{journal}{\emph{Proc. {VLDB} Endow.}} \bibinfo{volume}{14},
  \bibinfo{number}{11} (\bibinfo{year}{2021}), \bibinfo{pages}{2087--2100}.
\newblock


\bibitem[\protect\citeauthoryear{Mohoney, Waleffe, Xu, Rekatsinas, and
  Venkataraman}{Mohoney et~al\mbox{.}}{2021}]%
        {Marius_OSDI_2021}
\bibfield{author}{\bibinfo{person}{Jason Mohoney}, \bibinfo{person}{Roger
  Waleffe}, \bibinfo{person}{Henry Xu}, \bibinfo{person}{Theodoros Rekatsinas},
  {and} \bibinfo{person}{Shivaram Venkataraman}.}
  \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Marius: Learning Massive Graph Embeddings on a
  Single Machine}. In \bibinfo{booktitle}{\emph{15th {USENIX} Symposium on
  Operating Systems Design and Implementation, OSDI'21}}.
  \bibinfo{publisher}{{USENIX} Association}, \bibinfo{pages}{533--549}.
\newblock


\bibitem[\protect\citeauthoryear{NVIDIA}{NVIDIA}{2022}]%
        {DGX1}
\bibfield{author}{\bibinfo{person}{NVIDIA}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \bibinfo{title}{DGX Systems.}
\newblock
  \bibinfo{howpublished}{\url{https://www.nvidia.com/en-sg/data-center/dgx-systems/dgx-1}}.
\newblock


\bibitem[\protect\citeauthoryear{Peng, Chen, Shao, Shen, Chen, and Cao}{Peng
  et~al\mbox{.}}{2022}]%
        {SANCUS_VLDB_2022}
\bibfield{author}{\bibinfo{person}{Jingshu Peng}, \bibinfo{person}{Zhao Chen},
  \bibinfo{person}{Yingxia Shao}, \bibinfo{person}{Yanyan Shen},
  \bibinfo{person}{Lei Chen}, {and} \bibinfo{person}{Jiannong Cao}.}
  \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{{SANCUS:} Staleness-Aware Communication-Avoiding
  Full-Graph Decentralized Training in Large-Scale Graph Neural Networks}.
\newblock \bibinfo{journal}{\emph{Proc. {VLDB} Endow.}} \bibinfo{volume}{15},
  \bibinfo{number}{9} (\bibinfo{year}{2022}), \bibinfo{pages}{1937--1950}.
\newblock


\bibitem[\protect\citeauthoryear{Rajbhandari, Rasley, Ruwase, and
  He}{Rajbhandari et~al\mbox{.}}{2020}]%
        {ZID_1_SC_2020}
\bibfield{author}{\bibinfo{person}{Samyam Rajbhandari}, \bibinfo{person}{Jeff
  Rasley}, \bibinfo{person}{Olatunji Ruwase}, {and} \bibinfo{person}{Yuxiong
  He}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{ZeRO: memory optimizations toward training trillion
  parameter models}. In \bibinfo{booktitle}{\emph{Proceedings of the
  International Conference for High Performance Computing, Networking, Storage
  and Analysis, SC'20, Virtual Event / Atlanta, Georgia, USA}}.
  \bibinfo{publisher}{{IEEE/ACM}}, \bibinfo{pages}{20}.
\newblock


\bibitem[\protect\citeauthoryear{Rajbhandari, Ruwase, Rasley, Smith, and
  He}{Rajbhandari et~al\mbox{.}}{2021}]%
        {ZID_2_SC_2021}
\bibfield{author}{\bibinfo{person}{Samyam Rajbhandari},
  \bibinfo{person}{Olatunji Ruwase}, \bibinfo{person}{Jeff Rasley},
  \bibinfo{person}{Shaden Smith}, {and} \bibinfo{person}{Yuxiong He}.}
  \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{ZeRO-infinity: breaking the {GPU} memory wall for
  extreme scale deep learning}. In \bibinfo{booktitle}{\emph{International
  Conference for High Performance Computing, Networking, Storage and Analysis,
  SC'21, St. Louis, Missouri, USA}}. \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{59}.
\newblock


\bibitem[\protect\citeauthoryear{Ramezani, Cong, Mahdavi, Sivasubramaniam, and
  Kandemir}{Ramezani et~al\mbox{.}}{2020}]%
        {lazygcn_2020_neuips}
\bibfield{author}{\bibinfo{person}{Morteza Ramezani}, \bibinfo{person}{Weilin
  Cong}, \bibinfo{person}{Mehrdad Mahdavi}, \bibinfo{person}{Anand
  Sivasubramaniam}, {and} \bibinfo{person}{Mahmut~T. Kandemir}.}
  \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{{GCN} meets {GPU:} Decoupling "When to Sample" from
  "How to Sample"}. In \bibinfo{booktitle}{\emph{Advances in Neural Information
  Processing Systems 33: Annual Conference on Neural Information Processing
  Systems 2020, NeurIPS'20, virtual}}, \bibfield{editor}{\bibinfo{person}{Hugo
  Larochelle}, \bibinfo{person}{Marc'Aurelio Ranzato}, \bibinfo{person}{Raia
  Hadsell}, \bibinfo{person}{Maria{-}Florina Balcan}, {and}
  \bibinfo{person}{Hsuan{-}Tien Lin}} (Eds.).
\newblock


\bibitem[\protect\citeauthoryear{Ren, Rajbhandari, Aminabadi, Ruwase, Yang,
  Zhang, Li, and He}{Ren et~al\mbox{.}}{2021}]%
        {ZID_ATC_2021}
\bibfield{author}{\bibinfo{person}{Jie Ren}, \bibinfo{person}{Samyam
  Rajbhandari}, \bibinfo{person}{Reza~Yazdani Aminabadi},
  \bibinfo{person}{Olatunji Ruwase}, \bibinfo{person}{Shuangyan Yang},
  \bibinfo{person}{Minjia Zhang}, \bibinfo{person}{Dong Li}, {and}
  \bibinfo{person}{Yuxiong He}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{ZeRO-Offload: Democratizing Billion-Scale Model
  Training}. In \bibinfo{booktitle}{\emph{2021 {USENIX} Annual Technical
  Conference, ATC’21}}. \bibinfo{publisher}{{USENIX} Association},
  \bibinfo{pages}{551--564}.
\newblock


\bibitem[\protect\citeauthoryear{Sun, Su, Shi, Shen, Wang, Wang, Zhang, Li, Yu,
  Zhou, and Wu}{Sun et~al\mbox{.}}{2023}]%
        {legion_atc_2023}
\bibfield{author}{\bibinfo{person}{Jie Sun}, \bibinfo{person}{Li Su},
  \bibinfo{person}{Zuocheng Shi}, \bibinfo{person}{Wenting Shen},
  \bibinfo{person}{Zeke Wang}, \bibinfo{person}{Lei Wang}, \bibinfo{person}{Jie
  Zhang}, \bibinfo{person}{Yong Li}, \bibinfo{person}{Wenyuan Yu},
  \bibinfo{person}{Jingren Zhou}, {and} \bibinfo{person}{Fei Wu}.}
  \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Legion: Automatically Pushing the Envelope of
  Multi-GPU System for Billion-Scale {GNN} Training}. In
  \bibinfo{booktitle}{\emph{{USENIX} Annual Technical Conference, {USENIX}
  {ATC} 2023, Boston, MA, USA, July 10-12, 2023}},
  \bibfield{editor}{\bibinfo{person}{Julia Lawall} {and} \bibinfo{person}{Dan
  Williams}} (Eds.). \bibinfo{publisher}{{USENIX} Association},
  \bibinfo{pages}{165--179}.
\newblock


\bibitem[\protect\citeauthoryear{Tan, Yuan, He, Sit, Li, Liu, Ai, Zeng,
  Pietzuch, and Mai}{Tan et~al\mbox{.}}{2023}]%
        {quiver_2023_arxiv}
\bibfield{author}{\bibinfo{person}{Zeyuan Tan}, \bibinfo{person}{Xiulong Yuan},
  \bibinfo{person}{Congjie He}, \bibinfo{person}{Man{-}Kit Sit},
  \bibinfo{person}{Guo Li}, \bibinfo{person}{Xiaoze Liu},
  \bibinfo{person}{Baole Ai}, \bibinfo{person}{Kai Zeng},
  \bibinfo{person}{Peter~R. Pietzuch}, {and} \bibinfo{person}{Luo Mai}.}
  \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Quiver: Supporting GPUs for Low-Latency,
  High-Throughput {GNN} Serving with Workload Awareness}.
\newblock \bibinfo{journal}{\emph{CoRR}}  \bibinfo{volume}{abs/2305.10863}
  (\bibinfo{year}{2023}).
\newblock


\bibitem[\protect\citeauthoryear{Velickovic, Cucurull, Casanova, Romero,
  Li{\`{o}}, and Bengio}{Velickovic et~al\mbox{.}}{2018}]%
        {GAT_ICLR_2018}
\bibfield{author}{\bibinfo{person}{Petar Velickovic}, \bibinfo{person}{Guillem
  Cucurull}, \bibinfo{person}{Arantxa Casanova}, \bibinfo{person}{Adriana
  Romero}, \bibinfo{person}{Pietro Li{\`{o}}}, {and} \bibinfo{person}{Yoshua
  Bengio}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Graph Attention Networks}. In
  \bibinfo{booktitle}{\emph{6th International Conference on Learning
  Representations, ICLR'18, Vancouver, BC, Canada, Conference Track
  Proceedings}}. \bibinfo{publisher}{OpenReview.net}.
\newblock


\bibitem[\protect\citeauthoryear{Waleffe, Mohoney, Rekatsinas, and
  Venkataraman}{Waleffe et~al\mbox{.}}{2022}]%
        {Marius++_arxiv_2022}
\bibfield{author}{\bibinfo{person}{Roger Waleffe}, \bibinfo{person}{Jason
  Mohoney}, \bibinfo{person}{Theodoros Rekatsinas}, {and}
  \bibinfo{person}{Shivaram Venkataraman}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Marius++: Large-Scale Training of Graph Neural
  Networks on a Single Machine}.
\newblock \bibinfo{journal}{\emph{CoRR}}  \bibinfo{volume}{abs/2202.02365}
  (\bibinfo{year}{2022}).
\newblock


\bibitem[\protect\citeauthoryear{Wang, Yu, Zheng, Gan, Gai, Ye, Li, Zhou,
  Huang, Ma, Huang, Guo, Zhang, Lin, Zhao, Li, Smola, and Zhang}{Wang
  et~al\mbox{.}}{2019}]%
        {DGL_CORR_2019}
\bibfield{author}{\bibinfo{person}{Minjie Wang}, \bibinfo{person}{Lingfan Yu},
  \bibinfo{person}{Da Zheng}, \bibinfo{person}{Quan Gan}, \bibinfo{person}{Yu
  Gai}, \bibinfo{person}{Zihao Ye}, \bibinfo{person}{Mufei Li},
  \bibinfo{person}{Jinjing Zhou}, \bibinfo{person}{Qi Huang},
  \bibinfo{person}{Chao Ma}, \bibinfo{person}{Ziyue Huang},
  \bibinfo{person}{Qipeng Guo}, \bibinfo{person}{Hao Zhang},
  \bibinfo{person}{Haibin Lin}, \bibinfo{person}{Junbo Zhao},
  \bibinfo{person}{Jinyang Li}, \bibinfo{person}{Alexander~J. Smola}, {and}
  \bibinfo{person}{Zheng Zhang}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Deep Graph Library: Towards Efficient and Scalable
  Deep Learning on Graphs}.
\newblock \bibinfo{journal}{\emph{CoRR}}  \bibinfo{volume}{abs/1909.01315}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Wang, Chen, Wong, and He}{Wang
  et~al\mbox{.}}{2023}]%
        {hongtu_SIGMOD_2024}
\bibfield{author}{\bibinfo{person}{Qiange Wang}, \bibinfo{person}{Yao Chen},
  \bibinfo{person}{Weng{-}Fai Wong}, {and} \bibinfo{person}{Bingsheng He}.}
  \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{HongTu: Scalable Full-Graph {GNN} Training on
  Multiple GPUs (via communication-optimized {CPU} data offloading)}.
\newblock \bibinfo{journal}{\emph{CoRR}}  \bibinfo{volume}{abs/2311.14898}
  (\bibinfo{year}{2023}).
\newblock


\bibitem[\protect\citeauthoryear{Wu, Li, Luo, and Nejdl}{Wu
  et~al\mbox{.}}{2021a}]%
        {link_www_2021}
\bibfield{author}{\bibinfo{person}{Wei Wu}, \bibinfo{person}{Bin Li},
  \bibinfo{person}{Chuan Luo}, {and} \bibinfo{person}{Wolfgang Nejdl}.}
  \bibinfo{year}{2021}\natexlab{a}.
\newblock \showarticletitle{Hashing-Accelerated Graph Neural Networks for Link
  Prediction}. In \bibinfo{booktitle}{\emph{The Web Conference 2021, WWW'21,
  Virtual Event / Ljubljana, Slovenia}}. \bibinfo{publisher}{{ACM} / {IW3C2}},
  \bibinfo{pages}{2910--2920}.
\newblock


\bibitem[\protect\citeauthoryear{Wu, Shi, He, and Jin}{Wu
  et~al\mbox{.}}{2023}]%
        {TurboGNN_TOC_2023}
\bibfield{author}{\bibinfo{person}{Wenchao Wu}, \bibinfo{person}{Xuanhua Shi},
  \bibinfo{person}{Ligang He}, {and} \bibinfo{person}{Hai Jin}.}
  \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{TurboGNN: Improving the End-to-End Performance for
  Sampling-Based GNN Training on GPUs}.
\newblock \bibinfo{journal}{\emph{IEEE Trans. Comput.}} (\bibinfo{year}{2023}).
\newblock


\bibitem[\protect\citeauthoryear{Wu, Pan, Chen, Long, Zhang, and Yu}{Wu
  et~al\mbox{.}}{2021b}]%
        {SURVEY_TNNLS_2021}
\bibfield{author}{\bibinfo{person}{Zonghan Wu}, \bibinfo{person}{Shirui Pan},
  \bibinfo{person}{Fengwen Chen}, \bibinfo{person}{Guodong Long},
  \bibinfo{person}{Chengqi Zhang}, {and} \bibinfo{person}{Philip~S. Yu}.}
  \bibinfo{year}{2021}\natexlab{b}.
\newblock \showarticletitle{A Comprehensive Survey on Graph Neural Networks}.
\newblock \bibinfo{journal}{\emph{{IEEE} Trans. Neural Networks Learn. Syst.}}
  \bibinfo{volume}{32}, \bibinfo{number}{1} (\bibinfo{year}{2021}),
  \bibinfo{pages}{4--24}.
\newblock


\bibitem[\protect\citeauthoryear{Yang, Liu, Qi, and Lai}{Yang
  et~al\mbox{.}}{2022a}]%
        {wholegraph_SC_2022}
\bibfield{author}{\bibinfo{person}{Dongxu Yang}, \bibinfo{person}{Junhong Liu},
  \bibinfo{person}{Jiaxing Qi}, {and} \bibinfo{person}{Junjie Lai}.}
  \bibinfo{year}{2022}\natexlab{a}.
\newblock \showarticletitle{WholeGraph: {A} Fast Graph Neural Network Training
  Framework with Multi-GPU Distributed Shared Memory Architecture}. In
  \bibinfo{booktitle}{\emph{International Conference for High Performance
  Computing, Networking, Storage and Analysis, SC'22, Dallas, TX, USA}}.
  \bibinfo{publisher}{{IEEE}}, \bibinfo{pages}{1--14}.
\newblock


\bibitem[\protect\citeauthoryear{Yang and Leskovec}{Yang and Leskovec}{2015}]%
        {ORUKT_KIS_2015}
\bibfield{author}{\bibinfo{person}{Jaewon Yang} {and} \bibinfo{person}{Jure
  Leskovec}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Defining and evaluating network communities based
  on ground-truth}.
\newblock \bibinfo{journal}{\emph{Knowl. Inf. Syst.}} \bibinfo{volume}{42},
  \bibinfo{number}{1} (\bibinfo{year}{2015}), \bibinfo{pages}{181--213}.
\newblock


\bibitem[\protect\citeauthoryear{Yang, Tang, Song, Wang, Yin, Chen, Yu, and
  Zhou}{Yang et~al\mbox{.}}{2022b}]%
        {GNNlab_EUROSYS_2022}
\bibfield{author}{\bibinfo{person}{Jianbang Yang}, \bibinfo{person}{Dahai
  Tang}, \bibinfo{person}{Xiaoniu Song}, \bibinfo{person}{Lei Wang},
  \bibinfo{person}{Qiang Yin}, \bibinfo{person}{Rong Chen},
  \bibinfo{person}{Wenyuan Yu}, {and} \bibinfo{person}{Jingren Zhou}.}
  \bibinfo{year}{2022}\natexlab{b}.
\newblock \showarticletitle{GNNLab: a factored system for sample-based {GNN}
  training over GPUs}. In \bibinfo{booktitle}{\emph{Seventeenth European
  Conference on Computer Systems, EuroSys '22, Rennes, France}}.
  \bibinfo{publisher}{{ACM}}, \bibinfo{pages}{417--434}.
\newblock


\bibitem[\protect\citeauthoryear{Ying, He, Chen, Eksombatchai, Hamilton, and
  Leskovec}{Ying et~al\mbox{.}}{2018}]%
        {GCNWS_SIGKDD_2018}
\bibfield{author}{\bibinfo{person}{Rex Ying}, \bibinfo{person}{Ruining He},
  \bibinfo{person}{Kaifeng Chen}, \bibinfo{person}{Pong Eksombatchai},
  \bibinfo{person}{William~L. Hamilton}, {and} \bibinfo{person}{Jure
  Leskovec}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Graph Convolutional Neural Networks for Web-Scale
  Recommender Systems}. In \bibinfo{booktitle}{\emph{Proceedings of the 24th
  {ACM} {SIGKDD} International Conference on Knowledge Discovery {\&} Data
  Mining, KDD'18, London, UK}}. \bibinfo{publisher}{{ACM}},
  \bibinfo{pages}{974--983}.
\newblock


\bibitem[\protect\citeauthoryear{Zhang and Chen}{Zhang and Chen}{2018}]%
        {link_neurips_2018}
\bibfield{author}{\bibinfo{person}{Muhan Zhang} {and} \bibinfo{person}{Yixin
  Chen}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Link Prediction Based on Graph Neural Networks}. In
  \bibinfo{booktitle}{\emph{Advances in Neural Information Processing Systems
  31: Annual Conference on Neural Information Processing Systems 2018,
  NeurIPS'18, Montr{\'{e}}al, Canada}}. \bibinfo{pages}{5171--5181}.
\newblock


\bibitem[\protect\citeauthoryear{Zhang, Shen, Shao, and Chen}{Zhang
  et~al\mbox{.}}{2023}]%
        {ducati_sigmod_2023}
\bibfield{author}{\bibinfo{person}{Xin Zhang}, \bibinfo{person}{Yanyan Shen},
  \bibinfo{person}{Yingxia Shao}, {and} \bibinfo{person}{Lei Chen}.}
  \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{{DUCATI:} {A} Dual-Cache Training System for Graph
  Neural Networks on Giant Graphs with the {GPU}}.
\newblock \bibinfo{journal}{\emph{Proc. {ACM} Manag. Data}}
  \bibinfo{volume}{1}, \bibinfo{number}{2} (\bibinfo{year}{2023}),
  \bibinfo{pages}{166:1--166:24}.
\newblock


\bibitem[\protect\citeauthoryear{Zhang, Cui, and Zhu}{Zhang
  et~al\mbox{.}}{2022}]%
        {DLG_TKDE_2022}
\bibfield{author}{\bibinfo{person}{Ziwei Zhang}, \bibinfo{person}{Peng Cui},
  {and} \bibinfo{person}{Wenwu Zhu}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Deep Learning on Graphs: {A} Survey}.
\newblock \bibinfo{journal}{\emph{{IEEE} Trans. Knowl. Data Eng.}}
  \bibinfo{volume}{34}, \bibinfo{number}{1} (\bibinfo{year}{2022}),
  \bibinfo{pages}{249--270}.
\newblock


\bibitem[\protect\citeauthoryear{Zheng, Ma, Wang, Zhou, Su, Song, Gan, Zhang,
  and Karypis}{Zheng et~al\mbox{.}}{2020}]%
        {distdgl_ieee_2020}
\bibfield{author}{\bibinfo{person}{Da Zheng}, \bibinfo{person}{Chao Ma},
  \bibinfo{person}{Minjie Wang}, \bibinfo{person}{Jinjing Zhou},
  \bibinfo{person}{Qidong Su}, \bibinfo{person}{Xiang Song},
  \bibinfo{person}{Quan Gan}, \bibinfo{person}{Zheng Zhang}, {and}
  \bibinfo{person}{George Karypis}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{DistDGL: Distributed Graph Neural Network Training
  for Billion-Scale Graphs}. In \bibinfo{booktitle}{\emph{10th {IEEE/ACM}
  Workshop on Irregular Applications: Architectures and Algorithms, IA3'20,
  Atlanta, GA, USA}}. \bibinfo{publisher}{{IEEE}}, \bibinfo{pages}{36--44}.
\newblock


\bibitem[\protect\citeauthoryear{Zhou, Cui, Hu, Zhang, Yang, Liu, Wang, Li, and
  Sun}{Zhou et~al\mbox{.}}{2020}]%
        {GNNlayer-2020}
\bibfield{author}{\bibinfo{person}{Jie Zhou}, \bibinfo{person}{Ganqu Cui},
  \bibinfo{person}{Shengding Hu}, \bibinfo{person}{Zhengyan Zhang},
  \bibinfo{person}{Cheng Yang}, \bibinfo{person}{Zhiyuan Liu},
  \bibinfo{person}{Lifeng Wang}, \bibinfo{person}{Changcheng Li}, {and}
  \bibinfo{person}{Maosong Sun}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Graph neural networks: {A} review of methods and
  applications}.
\newblock \bibinfo{journal}{\emph{{AI} Open}}  \bibinfo{volume}{1}
  (\bibinfo{year}{2020}), \bibinfo{pages}{57--81}.
\newblock


\end{thebibliography}