FedOptima: Optimizing Resource Utilization in Federated Learning
Extracted source text · arXiv 2504.13850
\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{subfigure}
\usepackage{pifont}
\usepackage{url}
\usepackage{hyperref}
\usepackage[ruled,linesnumbered]{algorithm2e}
\usepackage{multirow}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\usepackage{makecell}
\usepackage{tabu}
\usepackage{tablefootnote}
\usepackage{bm}
% Disable colorful text
% \usepackage[monochrome]{xcolor}
% Enable colorful text
\usepackage{xcolor}
\usepackage{tcolorbox}
\tcbset{
sharp corners,
colback = white,
before skip = 0.3cm, % add extra space before the box
after skip = 0.25cm % add extra space after the box
} % setting global options for tcolorbox
\newtcolorbox{boxH}{
colback = gray!8,
colframe = black!60,
boxrule = 0.5pt,
leftrule = 3pt,
left=2pt,
right=2pt,
top=2pt,
bottom=2pt
}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}
\renewcommand\ttdefault{cmvtt}
\newcommand{\MethodName}{\texttt{FedOptima}}
%\title{\texttt{FedOptima}: Optimizing Resource\\Utilization in Federated Learning}
\title{Resource Utilization Optimized Federated Learning}
\author{\IEEEauthorblockN{Zihan Zhang}
\IEEEauthorblockA{\textit{University of St Andrews, UK}
%\\zz66@st-andrews.ac.uk
}
\and
\IEEEauthorblockN{Leon Wong}
\IEEEauthorblockA{\textit{Rakuten Mobile, Inc., Japan}
%\\leon.wong@rakuten.com
}
\and
\IEEEauthorblockN{Blesson Varghese}
\IEEEauthorblockA{\textit{University of St Andrews, UK}
%\\blesson@st-andrews.ac.uk
\thanks{Corresponding authors: \{zz66, blesson\}@st-andrews.ac.uk}}
}
%\author{\IEEEauthorblockN{Anonymous Authors}
%}
\maketitle
\thispagestyle{plain}
\pagestyle{plain}
\begin{abstract}
\input{sections/abstract}
\end{abstract}
\begin{IEEEkeywords}
federated learning, distributed system, resource utilization, idle time, edge computing
\end{IEEEkeywords}
\section{Introduction}
\input{sections/introduction}
\section{Background and Motivation}
\label{sec:motivation}
\input{sections/motivation}
\section{Overview}
\label{sec:overview}
\input{sections/overview}
\section{System Design}
\label{sec:design}
\input{sections/design}
\section{Implementation and Setup}
\label{sec:implement}
\input{sections/implement}
\section{Results}
\label{sec:results}
\input{sections/results}
\section{Related Work}
\label{sec:rw}
\input{sections/related_work}
\section{Conclusion}
\label{sec:conclusion}
\input{sections/conclusion}
\section*{Acknowledgment}
This research is funded by Rakuten Mobile, Inc., Japan, and supported by the UKRI grant EP/Y028813/1.
% \bibliographystyle{IEEEtran}
% \bibliography{ref.bib}
\input{main.bbl}
\end{document}
Federated learning (FL) systems facilitate distributed machine learning across a server and multiple devices. However, FL systems have low resource utilization limiting their practical use in the real world. This inefficiency primarily arises from two types of idle time: (i) task dependency between the server and devices, and (ii) stragglers among heterogeneous devices. This paper introduces \MethodName, a resource-optimized FL system designed to simultaneously minimize both types of idle time; existing systems do not eliminate or reduce both at the same time.
\MethodName\ offloads the training of certain layers of a neural network from a device to server using three innovations.
First, devices operate independently of each other using asynchronous aggregation to eliminate straggler effects, and independently of the server by utilizing auxiliary networks to minimize idle time caused by task dependency. Second, the server performs centralized training using a task scheduler that ensures balanced contributions from all devices, improving model accuracy. Third, an efficient memory management mechanism on the server increases scalability of the number of participating devices.
Four state-of-the-art offloading-based and asynchronous FL methods are chosen as baselines. Experimental results show that compared to the best results of the baselines on convolutional neural networks and transformers on multiple lab-based testbeds, \MethodName\ (i) achieves higher or comparable accuracy, (ii) accelerates training by 1.9$\times$ to 21.8$\times$, (iii) reduces server and device idle time by up to 93.9\% and 81.8\%, respectively, and (iv) increases throughput by 1.1$\times$ to 2.0$\times$.
Federated learning (FL)~\cite{fedavg, DBLP:journals/corr/KonecnyMR15,DBLP:journals/corr/KonecnyMRR16} offers distributed training across user devices as an alternative to traditional centralized machine training. In FL, a server and multiple devices iteratively conduct local training and aggregation. Devices train a deep neural network (DNN) on their data and send model parameters to the server. The server aggregates these into a global model, which is then distributed to the devices for the next round. Thus, FL utilizes insight from user data via local models to train a global model without sharing original data.
Sub-optimal resource utilization is a critical problem in FL that results in \textbf{two types of idle time} on the server and devices (see Section~\ref{subsec:problem}). The \textit{first is due to task dependency between server and devices}~- the server is idle for considerable periods when aggregating local models from devices as it waits for on-device training to complete, which is usually time-consuming. The \textit{second is due to hardware heterogeneity}~- stragglers or slower devices require more time to train than faster devices that idle while waiting for the stragglers.
Two categories of methods are considered in the existing literature for reducing idle time. Firstly, offloading-based FL (OFL) methods~\cite{splitfed,pipar,cse-fsl,fedgkt,10631047} improve server utilization by splitting the neural network and offloading the training of certain layers from the device to the server. Secondly, asynchronous FL (AFL) methods~\cite{fedasync,fedbuff,10631034} reduce the impact of stragglers by aggregating local models whenever the server receives them so that devices work independent of each other.
However, the above categories reduce idle time either on the server or devices, but not simultaneously (see Section~\ref{subsec:problem}). Simply combining OFL and AFL methods is not sufficient as the combination inherits the limitations of OFL and AFL. As such, it introduces large communication overheads, is not scalable, and certain devices disproportionately contribute to training, which increases idle time and reduces model accuracy (Section~\ref{subsec:challenge}). To address these challenges, this paper proposes a resource utilization-optimized FL system, \MethodName.
\textbf{Our solution}: \MethodName\ is novel in synergistically combining asynchronous distributed training on devices and centralized training on server to simultaneously reduce the server and device idle time. Each device trains a part of the DNN consisting of initial layers, which is made independent of the server by using an auxiliary network that computes a local loss. The on-device DNNs are aggregated asynchronously, ensuring that training on any device is independent of other devices. The server trains the rest of the DNN in a centralized manner to improve server utilization and model accuracy.
Evaluation on a range of models and datasets highlights that compared to the best results of four OFL and AFL baselines on multiple testbeds, \MethodName\ achieves higher or comparable accuracy, accelerates training by 1.9$\times$ to 21.8$\times$, reduces server and device idle time by up to 93.9\% and 81.8\%, respectively, and increases throughput by 1.1$\times$ to 2.0$\times$.
The research contributions of this paper are:
1) An \textbf{idle time analysis} to precisely identify bottlenecks in FL, thereby informing targeted optimizations.
2) \textbf{Independent asynchronous training} on each device and server to reduce network communication and idle time.
3) \textbf{Balanced task scheduling} to ensure devices contribute in a balanced way to centralized training on the server, which improves model accuracy.
4) \textbf{Efficient memory management} on the server via a novel flow control mechanism to enhance scalability of the number of devices that can collaborate with the server.
The rest of this paper is organized as follows. Section~\ref{sec:motivation} discusses two types of idle time in FL and the challenges of addressing them simultaneously. Section~\ref{sec:overview} provides an overview of \MethodName\ and Section~\ref{sec:design} presents the system design. Section~\ref{sec:implement} describes the implementation and experimental setup. Section~\ref{sec:results} highlights the effectiveness of \MethodName\ compared to baselines. Related work is discussed in Section~\ref{sec:rw}, and conclusions are drawn in Section~\ref{sec:conclusion}.
This section presents an idle time analysis followed by the challenges and our solution to simultaneously reduce server and device idle time.
\subsection{Idle Time Analysis}
\label{subsec:problem}
Assume $K$ devices participate in FL training with their local dataset $\{\mathcal{D}_k;k\in [K]\}$. The objective of FL~\cite{fedavg} is:
\begin{equation}
\label{eq:fl}
\underset{\bm{\theta}}{\min} \mathcal{F}(\bm{\theta}) = \frac{1}{k} \sum_{k=1}^{K} \mathbb{E}_{\zeta_k \sim \mathcal{D}_k} f_{k} (\bm{\theta})
\end{equation}
where $\bm{\theta}$ denotes the parameters of the global model and $\{f_k(\bm{\theta});k\in [K]\}$ is the local loss function.
The FL training steps are as follows: Step 1: \textit{Initialize} the global model on the server and distribute it to the devices; Step 2: \textit{Train} the models on the devices using local data; Step 3: \textit{Upload} the local models (not the data) from the devices to the server; Step 4: \textit{Aggregate} the local models on the server to obtain the global model; Step 5: \textit{Distribute} the global model to the devices for the next round of training.
The above steps are repeated over multiple iterations to improve model accuracy.
However, the iterative and distributed nature of FL results in low resource utilization, which introduces idle time and impacts training efficiency. Two types of idle time are discussed below.
\begin{figure*}[tp]
\centering
\subfigure[Classic FL]{
\includegraphics[width=0.23\linewidth]{figs/fl.pdf}
\label{subfig:fl}
}
\subfigure[Offloading-Based FL]{
\includegraphics[width=0.23\linewidth]{figs/ofl.pdf}
\label{subfig:ofl}
}
\subfigure[Asynchronous FL]{
\includegraphics[width=0.23\linewidth]{figs/afl.pdf}
\label{subfig:afl}
}
\subfigure[\MethodName]{
\includegraphics[width=0.23\linewidth]{figs/timeline.pdf}
\label{subfig:new-timeline}
}
\caption{Training timeline for various federated learning (FL) methods with one server and two devices (Device 1 is assumed to be faster than Device 2). For simplicity, only one transfer of the model and activations per device is shown for \MethodName.}
\label{fig:timeline}
\end{figure*}
\begin{boxH}
\textbf{Idle time due to task dependency} -
Server aggregation and device training are interdependent. Idle time on the server or device when waiting during model transfer between training and aggregation is named \textbf{Type~I idle time}.
\end{boxH}
In classic FL, aggregation requires local models, while device training requires the aggregated model. This dependency results in idle time on the server, as it waits for all devices to complete training and send their models (Figure~\ref{subfig:fl}).
Offloading-based FL (OFL) methods, such as SplitFed~\cite{splitfed} and PiPar~\cite{pipar}, mitigate this problem by offloading training from the device to the server. In OFL, a DNN is partitioned, and the initial layers are trained on the device (referred to as device-side network), while the rest on the server (server-side network).
The devices execute the forward pass of the device-side network and send intermediate results (or activations) to the server (Figure~\ref{subfig:ofl}). The server trains one server-side network corresponding to each device and sends the gradients back to the device to continue the backward pass and update the initial layers. This split architecture reduces Type~I idle time on the server. However, Type~I idle time increases on the device since it requires gradients from the server. Sending intermediate results between the devices and server increases the communication overhead. Although OFL improves server utilization, it does not address task dependency.
\begin{boxH}
\textbf{Idle time due to stragglers} -
Hardware heterogeneity among participating devices results in devices that slow down overall training. Faster devices wait for the stragglers to complete local training, giving rise to \textbf{Type~II idle time}.
\end{boxH}
As shown in Figure~\ref{subfig:fl}, Device~1 waits for Device~2 to complete local training. Asynchronous FL (AFL) methods, such as FedAsync~\cite{fedasync} and FedBuff~\cite{fedbuff}, are proposed to address this. Aggregation in AFL requires one local model instead of models from all devices. The global model is the weighted average of the current global model and a given local model. Devices send trained models to the server at their own pace, reducing Type~II idle time on the devices (Figure~\ref{subfig:afl}). However, faster devices exchange models with the server more frequently, which increases the communication overhead.
Asynchronous model updates introduce a further problem - staleness. The local model may be stale when aggregated into the global model, because the latter may have already been updated by local models from other devices. AFL reduces the weight of the local model during aggregation as its staleness increases but the final accuracy is lower than classic FL.
\subsection{Challenges in Addressing Idle Time}
\label{subsec:challenge}
Addressing Type~I and Type~II idle time simultaneously in FL is challenging. A straightforward solution might combine OFL and AFL methods. However, this combination inherits the limitations of the individual methods rather than mitigates them. This is demonstrated by developing an offloading-based asynchronous FL (OAFL) method that combines OFL and AFL. The OAFL implementation is based on SplitFed~\cite{splitfed} and uses the asynchronous aggregation of FedAsync~\cite{fedasync}.
A MobileNetV3-Large~\cite{mobilenet} is trained on the Tiny ImageNet dataset~\cite{tinyimagenet} using OAFL on a testbed of 16 devices with heterogeneous computational resources (see full details of the model, dataset and testbed in Section~\ref{subsec:setup}). Three challenges are encountered that motivate a new FL system. \MethodName\ is developed in response that combines the strengths of OFL and AFL and redesigns training to address the challenges.
\begin{figure}[t]
\begin{minipage}[b]{0.47\linewidth}
\centering\includegraphics[width=\linewidth]{figs/comm_all.png}
\caption{Communication volume per round.}
\label{fig:comm-mot}
\end{minipage}
\hspace{0.25cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/memory_final.png}
\caption{Memory required on server for varying no. of devices.}
\label{fig:memory-mot}
\end{minipage}
\end{figure}
\begin{boxH}
\textbf{Challenge 1: Large Communication Overheads}~- Offloading-based FL requires the exchange of activations and gradients between each device and the server. The communication further increases when offloading-based FL is combined with asynchronous FL, because the latter increases the frequency of communication by allowing all devices to continue training without waiting for stragglers.
\end{boxH}
The communication volume per round\footnote{A training round refers to training on $D$ data samples, where $D$ is the total size of datasets on all devices.} between devices and the server is measured during training. Compared to OFL, OAFL has a 24\% increase in communication (Figure~\ref{fig:comm-mot}).
\textit{Our system:} Most communication in OFL and OAFL arises from transferring activations and gradients. \MethodName\ eliminates the need for a device to receive gradients from the server as each device trains its model independently with a local loss function (Section~\ref{subsec:device-design}). The activations are only transferred to the server upon request, rather than in every iteration, which further reduces the communication volume (Section~\ref{subsec:act-control}). Figure~\ref{fig:comm-mot} highlights that \MethodName\ has lower communication than both OFL and OAFL. Specifically, \MethodName\ reduces communication of OAFL by 59\%.
\begin{boxH}
\textbf{Challenge 2: Lack of Scalability} - The amount of memory available on the server is the factor limiting the scalability of offloading-based FL, and consequently its combination with asynchronous FL. The maximum number of devices that can be trained depends on the number of server counterparts of the partitioned model that can be held in memory. As the number of devices increases, more server memory is required to accommodate their activations.
\end{boxH}
The memory required $\mu$ by the server is as follows:
\begin{equation}
\mu = (K+1) * \mu^{model} + K * \mu^{act}
\end{equation}
where $K$ denotes the number of devices, and $\mu^{model}$ and $\mu^{act}$ are sizes of model and activations, respectively. The first term of the equation indicates the server memory required for $K$ partitioned models and one global model. The second term represents the memory required by $K$ batches of activations.
Figure~\ref{fig:memory-mot} shows the server memory required by OAFL for different number of devices. Memory required linearly increases with the number of devices; OAFL is not scalable for a large number of devices. A server with 8 GB memory can only work with at most 26 devices. Note that MobileNetV3-Large is a memory-efficient convolutional neural network (CNN), and using other CNNs, such as VGG-16~\cite{vgg} or ResNet-101~\cite{resnet}, will further reduce the number of devices.
\textit{Our system:} In \MethodName, the server hosts a single model that receives activations from all devices (Section~\ref{subsec:server-design}). The server memory required is substantially lower than OAFL. The server has a predefined cap $\omega$ on the number of activations. Devices send activations only when the server allows (Section~\ref{subsec:act-control}). As a result, \MethodName\ operates on fixed memory budgets $\mu$ regardless of $K$, as shown in Equation~\ref{eq:memory-fedoptima} and Figure~\ref{fig:memory-mot}. Since the server model is trained in a centralized way (no aggregation required), \MethodName\ achieves comparable or higher accuracy than FL (Section~\ref{subsec:efficiency}).
\begin{equation}
\label{eq:memory-fedoptima}
\mu = \mu^{model} + \omega * \mu^{act}
\end{equation}
\begin{boxH}
\textbf{Challenge 3: Disproportionate Contribution to Training}~- When the server asynchronously aggregates local models in offloading-based asynchronous FL, faster devices that complete more training iterations contribute disproportionately to the global model than slower devices.
\end{boxH}
The above affects the final model accuracy in two ways. Firstly, local models from slower devices may be stale as the global model may already be updated by local models from faster devices. In OAFL, staleness exists in both server and device models. Secondly, local models from slower devices are sent less frequently to the server, thereby making a relatively smaller contribution to the global model. Since local data on devices in the real-world is non-IID (non-independent and identically distributed), the imbalanced contributions from different datasets may lower the accuracy of the global model.
The impact on accuracy is shown in Table~\ref{tab:homo-hetero}. We use two testbeds - one comprising homogeneous devices and the other heterogeneous devices. The latter is simulated using homogeneous devices by changing the default GPU frequency (see Section~
\ref{subsec:setup}). As shown in Table~\ref{tab:homo-hetero}, using OAFL, the accuracy of the model trained on the heterogeneous device platform is only 20.02\%, which is lower than the model accuracy on homogeneous devices (24.03\%).
\textit{Our system:} Since \MethodName\ trains only one global model it is never stale (Section~\ref{subsec:server-design}). The activations from different local models are selectively scheduled to the global model, thereby balancing the contribution of activations from different devices. Since the global model is trained centrally, the accuracy is higher than OAFL. Table~\ref{tab:homo-hetero} shows that models trained using \MethodName\ have comparable accuracy on homogeneous (39.75\%) and heterogeneous (40.05\%) devices, which is higher than the accuracy achieved by OAFL, suggesting that the model accuracy of \MethodName\ is not impacted by the disproportionate contribution of heterogeneous devices.
\textbf{Addressing idle time in our system}:
In \MethodName, each device operates independently and periodically transfers activations to the server. The server places the activations into a queue and uses them to train a single server model. The device models are also placed into a queue and asynchronously aggregated in sequence. Finally, as shown in Figure~\ref{subfig:new-timeline}, \MethodName\ allows each device to work continuously without waiting for the server or other devices. The server works until the activation queue is empty. This minimizes both types of idle time, which improves the overall training performance.
\begin{table}[t]
\centering
\caption{The accuracy of MobileNet3-Large trained on Tiny ImageNet using homogeneous and heterogeneous devices.}
\begin{tabular}{ccc}
\Xhline{2\arrayrulewidth}
\multirow{2}{*}{FL Method} & \multicolumn{2}{c}{Model Accuracy} \\
\cline{2-3}
& Homogeneous & Heterogeneous \\
\Xhline{2\arrayrulewidth}
OAFL & 24.03 & 20.02 \\
\hline
\MethodName & 39.75 & 40.05 \\
\Xhline{2\arrayrulewidth}
\end{tabular}
\label{tab:homo-hetero}
\end{table}
\begin{figure*}[htp]
\subfigure[Initialization]{
\includegraphics[width=0.47\textwidth]{figs/init.pdf}
\label{subfig:init}
}
\subfigure[Training]{
\includegraphics[width=0.52\textwidth]{figs/overview.pdf}
\label{subfig:training}
}
\caption{Overview of \MethodName.}
\label{fig:overview}
\end{figure*}
\MethodName\ is a novel framework designed to enhance FL training by reducing both Type~I and Type~II idle times, thereby maximizing training performance. \MethodName\ operates on a server and $K$ devices. \MethodName\ works for any sequential DNN, including convolutional neural networks and transformers. Figure~\ref{fig:overview} provides an overview of the \MethodName\ system comprising an initialization and a training stage.
During \textit{initialization} (Figure~\ref{subfig:init}), the devices send information, such as network bandwidth and floating operations per second to the server. The DNN $M$ is initialized on the server. A Pre-processor module (Section~\ref{subsec:preprocessor}) uses the device information to partition $M$ into $M_d$, the initial layers for the device, and $M_s$, the remaining layers to be offloaded to the server. This module also generates an auxiliary network $\tilde{M}_d$ that is attached to the last layer of $M_d$. $M_d$ and $\tilde{M}_d$ sent to each device ($M_{d_k}$ and $\tilde{M}_{d_k}, k\in [K]$) are called the device-side network and auxiliary network, respectively, while $M_s$ on the server is the server-side network.
$\tilde{M}_{d_k}$ will be used for local training on devices. The server also maintains the parameters of $M_d$ and $\tilde{M}_d$, denoted by $\bm{\theta}_d$ and $\bm{\tilde{\theta}}_d$, respectively.
The \textit{training} stage of \MethodName\ is shown in Figure~\ref{subfig:training}. On any device $k$, \textbf{device-side FedOptima} has a Compute Engine that comprises a Trainer and an Aggregator, and a Communicator (Section~\ref{subsec:device-design}). In repeated iterations, the Trainer takes a mini-batch of local data and generates activations as the output of the last layer of $M_{d_k}$. The activations are uploaded to the server and used to train $M_s$. Concurrently, the Trainer uses the activations to compute the gradients using $\tilde{M}_{d_k}$ and the local loss function $f_d$. Local training generates gradients instead of receiving them from the server, thereby halving the communication and mitigating \textit{Challenge~1}. After a predetermined number of training iterations, the Aggregator forwards the local models $\bm{\theta}_{d_k}$ (the parameters of $M_{d_k}$) to the server for aggregation. The received global model $\bm{\theta}_d$ (the parameters of $M_d$) replaces the local model.
\textbf{Server-side FedOptima} has three modules, namely a Communicator, a Task Scheduler and a Compute Engine (Section~\ref{subsec:server-design}). The activations are received from all devices and used for training the server-side network $M_s$, which is akin to centralized training but using activations instead of raw data. The server model\footnote{We use `model' to refer to the parameters of a neural network.} $\bm{\theta}_s$ (the parameters of $M_s$) is iteratively updated by minimizing the server-side loss function $f_s$. To mitigate the problem of disproportionate contribution discussed in \textit{Challenge~3}, a Task Scheduler is designed to select the activations for each iteration. \MethodName\ maintains only one server-side network and processes activations from the Task Scheduler in sequence, which mitigates \textit{Challenge~2}. The Task Scheduler also dispatches local models $\{\bm{\theta}_{d_k};k\in [K]\}$ to the Aggregator. The Aggregator uses an asynchronous algorithm to aggregate local models to the global model $\bm{\theta}_d$ and then sends it to the corresponding devices. Although aggregation occurs asynchronously, staleness will exist in the device-side model rather than the entire model since only the server-side model is trained in a centralized way.
The overall objective of \MethodName\ can be separated as a device objective and a server objective, which are formulated as Equation~\ref{eq:fedoptima-device} and Equation~\ref{eq:fedoptima-server}, respectively. The local datasets on the $K$ devices are denoted as $\{\mathcal{D}_k;k\in [K]\}$, and the activation set on the server is denoted as $\mathcal{A}$.
\begin{equation}
\label{eq:fedoptima-device}\underset{\bm{\theta}_d,\bm{\tilde{\theta}}_d}{\min} \mathcal{F}_d(\bm{\theta}_d,\bm{\tilde{\theta}}_d) = \frac{1}{k} \sum_{k=1}^{K} \mathbb{E}_{\zeta_k \sim \mathcal{D}_k} f_d (\bm{\theta}_d,\bm{\tilde{\theta}}_d; \zeta_k)
\end{equation}
\begin{equation}
\label{eq:fedoptima-server}
\underset{\bm{\theta}_s}{\min} \mathcal{F}_s(\bm{\theta}_s) = \mathbb{E}_{\xi \sim \mathcal{A}} f_s(\bm{\theta}_s; \xi)
\end{equation}
\textit{Privacy Considerations}:
FL, AFL and OFL inherently enhance privacy by transferring models or activations instead of the original data. Established techniques, such as differential privacy~\cite{dp, pixeldp}, mitigate the potential risk of reconstructing data from models or activations.
Privacy-preserving techniques are orthogonal to our contributions on reducing idle time and can be integrated in the proposed system as needed.
The initialization and training stages and the memory and device management strategy of \MethodName\ are presented.
\subsection{Initialization Stage}
\label{subsec:preprocessor}
In the initialization stage, the devices share information, namely network bandwidth $b_k$ and CPU/GPU floating-point operations per second $o_k$, with the server. The \textbf{Pre-processor} uses the device information to (1) split the DNN $M$ into $M_d$ and $M_s$, and (2) generate an auxiliary network.
\textbf{Splitting the DNN}:
When splitting $M$, the Pre-processor only selects sequential neural network layer connections as split points and does not cut off the branch network. In selecting the split point, the trade-off between device training time $t^{train}$ and activation transfer time $t^{transfer}$ must be considered. $M$ is profiled by the Pre-processor to obtain the floating operations $O_l$ and the output sizes of each layer $\{S_l, l\in [L]\}$, where $L$ is the layer number of $M$. Then $t^{train}$ and $t^{transfer}$ with split point $l$ for device $k$ are estimated by Equation~\ref{eq:train_time} and Equation~\ref{eq:transfer_time}.
\begin{equation}
\label{eq:train_time}
t^{train}_k(l) = \sum_i^l O_l/o_k
\end{equation}
\begin{equation}
\label{eq:transfer_time}
t^{transfer}_k(l) = S_l/b_k
\end{equation}
The selection of the split point $l$ is formulated as the following optimization problem:
\begin{equation}
l = \argmin_{l=1}^L \max_{k=1}^K \max\{t^{train}_k(l), t^{transfer}_k(l)\}
\end{equation}
\textbf{Designing the Auxiliary Network}:
The auxiliary network $\tilde{M}_{d_k}$ serves as an extension to the local network $M_{d_k}$, specifically designed to adapt the output of the network for local objectives $f_d$ to provide gradients for model updates. Compared to existing offloading FL methods, using an auxiliary network allows the device to update the model without receiving gradients from the server, thus significantly reducing communication traffic. $\tilde{M}_{d_k}$ usually consists of fewer layers than $M_s$. The default option of $\tilde{M}_{d_k}$ is one layer of the same type as the last layer of $M_{d_k}$, followed by a dense layer as the classifier. We use the default option in experiments. The layer number of $\tilde{M}_{d_k}$ is a tunable hyperparameter.
The overhead of the initialization stage is under 5 seconds.
\subsection{Training Stage}
\subsubsection{Device-Side \MethodName}
\label{subsec:device-design}
\MethodName\ on each device consists of two modules that run in parallel: the Compute Engine and the Communicator.
The \textbf{Communicator} uses a Sender and Receiver in parallel that maintains a sending queue for outgoing messages and a receiving queue for incoming messages. The Sender queues activations and local models for dispatch. Incoming messages queued in the Receiver are retrieved by the Compute Engine.
\begin{algorithm}[t]
\caption{Compute Engine on device $k$.}
\label{alg:device-compute}
\KwIn{Device-side network $M_{d_k}$ with initial parameters $\bm{\theta}_{d_k}$; auxiliary network $\tilde{M}_{d_k}$ with initial parameters $\bm{\tilde{\theta}}_{d_k}$; local dataset $\mathcal{D}_k$; maximum local round number $E_{d_k}$; iteration number per round $H$; device learning rate $\gamma_d$}
\KwOut{Device-side model $\bm{\theta}_{d_k}$}
\BlankLine
Let local model version $t_k \gets 0$
\For{$e \gets 0$ to $E_{d_k}$}{
\tcc{Local training iterations.}
\For{$h \gets 0$ \KwTo $H$}{
Get a mini-batch of data $\zeta_k^h$ from $\mathcal{D}_k$
Execute forward pass of $M_{d_k}$ on $\zeta_k^h$
Get activations $\xi_k^h$ and put into the Sender
Execute forward pass of $\tilde{M}_{d_k}$ on $\xi_k^h$
Get local loss $f_d (\bm{\theta}_{d_k},\bm{\tilde{\theta}}_{d_k}; \zeta_k^h)$
Execute backward pass of $\tilde{M}_{d_k}$ and $M_{d_k}$
Let $\bm{\theta}_d \gets \bm{\theta}_{d_k} - \gamma_d \partial f_d / \partial \bm{\theta}_{d_k}$
Let $\bm{\tilde{\theta}}_d \gets \bm{\tilde{\theta}}_d - \gamma_d \partial f_d / \partial \bm{\tilde{\theta}}_{d_k}$
}
\tcc{Aggregation.}
Put $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_k\}$ into the Sender
Get $\{\bm{\theta}_d, \bm{\tilde{\theta}}_d, t\}$ from the Receiver
Let $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_k\} \gets \{\bm{\theta}_d, \bm{\tilde{\theta}}_d, t$\}
\uIf{Receive STOP}{
\textit{break}
}
}
\textbf{Return} $\bm{\theta}_{d_k}$
\end{algorithm}
The \textbf{Compute Engine} has two submodules: a Trainer for iteratively training the local device-side model and an Aggregator for periodically aggregating the local model.
As shown in Algorithm~\ref{alg:device-compute}, the device-side network is trained in multiple rounds ($E_{d_k}$ rounds if no stop signal is received), each consisting of $H$ iterations (Lines~2-18). The device-side model version $t_k$ is initialized as 0 at the start (Line~1). Within each iteration (Lines~3-12), the Trainer processes a mini-batch of data through $M_{d_k}$ (Line~4). After the forward pass (Line~5), the activations are obtained and placed with the Sender for server-side training (Line~6). Meanwhile, the activations are used to compute the local loss through $\tilde{M}_{d_k}$ (Lines~7-8). Backpropagation follows (Line~9), leveraging the local loss to calculate gradients that update the model parameters (Lines~10-11) required for the next iteration.
The Aggregator works at the end of each round. The device-side model and the auxiliary model along with their version are put into the Sender (Line~13). After the aggregation on the server completes, the new models are received from the server (Line~14) and replace the old ones (Line~15).
Training completes when a stop signal is received from the server (Line~16) or after $E_{d_k}$ rounds are completed.
\subsubsection{Server-Side \MethodName}
\label{subsec:server-design}
The server trains the server-side model using activations from devices and aggregates device-side models. The server has three modules: the Communicator, the Task Scheduler, and the Compute Engine.
All devices are concurrently connected to the \textbf{Communicator}. The Sender and Receiver maintain a sending queue and a receiving queue, respectively, accessed by the Task Scheduler.
The \textbf{Task Scheduler} handles two message types: local models and activations. The $put()$ and $get()$ algorithms re-order received messages from the Communicator before dispatching them to the Compute Engine when requested.
\begin{algorithm}[t]
\caption{$put()$ algorithm of the Task Scheduler.}
\label{alg:put}
\KwIn{Received message $m$; model queue $Q^{model}$; activation queues $\{Q^{act}_k; k\in [K]\}$
}
\BlankLine
\uIf{$m$.type == ``model"}{
Let $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_k\} \gets m.content$
Put $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_k\}$ to $Q^M$
}
\Else{
Let $\xi \gets m.content$
Let $k \gets m.origin$
Put $\xi$ to $Q^{act}_k$
}
\textbf{Return}
\end{algorithm}
The Task Scheduler maintains a model queue $Q^{model}$ to store models and $K$ activation queues $\{Q^{act}_k, k=1, \cdots, K\}$ to store activations received from each device. As shown in Algorithm~\ref{alg:put}, the Task Scheduler recognizes whether a message from the Communicator contains a combination of the local models (Line~1) or a batch of activations (Line~4). The combination of local models and activation batches are then put into the corresponding queues (Lines~2-3, 5-7).
\begin{algorithm}[t]
\caption{$get()$ algorithm of the Task Scheduler.}
\label{alg:get}
\KwIn{Model queue $Q^{model}$; activation queues $\{Q^{act}_k; k\in [K]\}$; activation counter $\{c_k; k\in [K]\}$
}
\KwOut{Activations $\xi$ or local models $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_{d_k}\}$}
\BlankLine
\uIf{$Q^{model}$ is not empty}{
Get $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_k\}$ from $Q^{model}$
\textbf{Return} $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_k\}$
}
\Else{
Let $k \gets \argmin_k c_k$
Get $\xi$ from $Q^{act}_k$
Let $c_k \gets c_k+1$
\textbf{Return} $\xi$
}
\end{algorithm}
When a request is received from the Compute Engine, the Task Scheduler gets a batch of activations for training or a combination of local models for aggregation. As shown in Algorithm~\ref{alg:get}, a higher priority is given for retrieving models (Line~1) to the Compute Engine over the activations. If there are no pending models in the queue, the Task Scheduler retrieves activations for the Trainer to balance training continuity and timely model aggregation (Line~4). The FIFO policy is used to obtain local model combinations from the queue (Lines~2-3). To address Challenge~3, a counter-based sorting algorithm is implemented to get activations (Lines~5-8). The counter $\{c_k; k\in [K]\}$ tallies the activations used for training from each device and is used to prioritize devices that contributed fewer activations. Activations received from slower devices have a higher priority, thereby balancing the contribution of devices for server-side model training.
\begin{algorithm}[t]
\caption{Compute Engine on server.}
\label{alg:server-compute}
\KwIn{Device number $K$; server-side network $M_s$ with initial parameters $\bm{\theta}_s$; Task Scheduler $\mathcal{S}$; maximum global round number $E_s$; server-side learning rate $\gamma_s$; maximum delay $D$; aggregation weight $\alpha$}
\KwOut{Server-side model $\bm{\theta}_s$}
\BlankLine
Let global model version $t \gets 0$
\For{$e \gets 0$ \KwTo $E_s$}{
\While{$m \gets \mathcal{S}.get()$}{
\uIf{$m$.type == ``activation"}{
\tcc{Global training iterations.}
Let activations $\xi \gets m.content$
Execute forward pass of $M_s$
Get global loss $f_s (\bm{\theta}_s; \xi)$
Execute backward pass of $M_s$
Get gradients $\nabla f_s (\bm{\theta}_s; \xi)$
$\bm{\theta}_s \gets \bm{\theta}_s - \gamma_s \nabla f_s (\bm{\theta}_s; \xi)$
}
\Else{
\tcc{Aggregation.}
Let $\{\bm{\theta}_{d_k}, \bm{\tilde{\theta}}_{d_k}, t_k\} \gets m.content$
\If{$t - t_k > D$}{
\textit{continue}
}
Let $\alpha \gets \frac{1}{t-t_k+1}$
Let $\bm{\theta}_d \gets \alpha\bm{\theta}_{d_k} + (1-\alpha)\bm{\theta}_d$
Let $\bm{\tilde{\theta}}_d \gets \alpha\bm{\tilde{\theta}}_{d_k} + (1-\alpha)\bm{\tilde{\theta}}_d$
Let $t \gets t + 1$
Put $\{\bm{\theta}_d, \bm{\tilde{\theta}}_d, t\}$ into the Sender
\uIf{$t \% K == 0$}{
\textit{break}
}
}
}
\uIf{$\bm{\theta}_s$ has converged}{
\textit{break}
}
}
Send \textit{STOP} to all devices
\textbf{Return} $\bm{\theta}_s$
\end{algorithm}
The \textbf{Compute Engine} requests inputs continuously from the Task Scheduler to carry out training and aggregation.
As shown in Algorithm~\ref{alg:server-compute}, training consists of at most $E_s$ rounds. The global device-side model version is set to 0 at the beginning (Line~1). Within each training round (Lines~2-28), the Compute Engine requests a message repeatedly from the Task Scheduler (Line~3). When activations are received (Line~4), one training iteration of the server-side network $M_s$ is executed using stochastic gradient descent (Lines~5-10), analogous to centralized training but using activations instead of the original data. This addresses server-side model staleness. Upon receiving a local model combination (Line~11), the Compute Engine performs aggregation (Lines~12-19) using FedAsync~\cite{fedasync}. To reduce the impact of staleness aggregation does not occur if the difference between the versions of the local model and the global model is higher than the predefined maximum delay $D$ (Lines~13-14). Otherwise, the aggregation weight $\alpha$ is updated based on the difference of the version (Line~16), and the global model, global auxiliary model and global model version are updated (Lines~17-19) and put into the Sender (Line~20). A global training round is completed after finishing $K$ aggregation (Lines~21-22). Then the next round commences.
After the server-side model converges or $E_s$ rounds, the training completes (Lines~25-27) and a stop signal is sent to all devices (Line~28).
\subsection{Memory and Device Management}
\label{subsec:act-control}
\textbf{Activation Flow Control}: Efficient memory management on the server is required in \MethodName\ due to the continuous transfer of activations from devices to the server.
Each device maintains a \textit{Sender Status} to control the transfer of activations to the server. The server manages activation queues for each device, denoted as $Q^{act}_k$ for device $k$, with a predefined maximum queue length $\omega$ to avoid memory overflow. The activation flow control is as follows (see Figure~\ref{fig:act-flow}).
\textit{Device-side Flow Control}: After every local training iteration, each device evaluates its \textit{Sender Status} before sending activations. If the Sender is active, the device sends activations to the server and then deactivates the Sender, until receiving a `turn-on' signal from the server; otherwise, the device proceeds to the next iteration without sending activations.
\textit{Server-side Flow Control}: The server adds the activations received to the queue of the corresponding device $Q^{act}_k$. When activations are enqueued to or dequeued from $Q^{act}_k$, the server checks whether $Q^{act}_k$ is full. If not, it dispatches a `turn-on' signal to the corresponding device, reactivating its Sender.
The above ensures that the server memory is used efficiently and within a defined budget. Since the activation queues on the server have fixed lengths regardless of device number, an increase in the number of devices will not cause contention for server resources. It also optimizes the network bandwidth by eliminating the transmission of redundant activations.
\begin{figure}[tp]
\centering
\includegraphics[width=\linewidth]{figs/activation_flow.pdf}
\caption{The activation flow control between a device and the server. A device only sends activations to the server when the Sender is active.}
\label{fig:act-flow}
\end{figure}
\textbf{Devices Joining and Leaving}:
Devices may not be always available in real-world FL. In synchronous FL, devices that participate are selected before a training round to reduce delays due to failed or dropped devices~\cite{fedavg,oort,refl}. Since local device models are aggregated asynchronously in asynchronous FL, devices that join or leave do not delay training~\cite{fedasync,fedbuff}.
However, the trade-off between system and statistical performance needs to be considered when selecting participating devices~\cite{oort,refl}.
Faster devices are more likely to be selected for improving system performance, but the data on slower devices will be less represented in the global model. This results in requiring a larger number of training rounds to achieve a target model accuracy. A wider coverage of data is required for improving statistical performance but requires slower devices to be selected for training, which will inevitably increase the training time per round.
In \MethodName, since devices work independently they join or leave without interrupting training. When a device leaves training, the other devices and server will continue training to maintain system performance. Activation flow control discussed above ensures that new devices that join do not contend for server resources. The Task Scheduler in \MethodName\ balances data across different devices, thereby improving statistical performance in FL. In short, \MethodName\ balances system performance and statistical performance.
The implementation of \MethodName\ is considered in Section~\ref{subsec:implement} and the experimental setup, including the testbed, models, datasets and baselines are presented in Section~\ref{subsec:setup}.
\subsection{Implementation}
\label{subsec:implement}
\MethodName\ is developed using PyTorch to facilitate DNN partitioning. TCP/IP is used in server and device communication shown in Figure~\ref{subfig:training} to prevent packet loss.
\MethodName\ uses concurrent threads to implement the modules/submodules on the server and device. Four threads are created on the \textbf{\textit{server}}: for the Sender, Receiver, Task Scheduler, and Compute Engine. The Sender and Receiver connect to all devices. The Compute Engine places messages that are tagged with a destination into the sending queue of the Sender. The Sender checks the queue and sends messages to the intended device. The Receiver continuously collects messages from devices and puts them in a receiving queue. The Task Scheduler monitors the receiving queue and categorizes incoming messages and directs them to either the model or activation queues. The Compute Engine retrieves models and activations from the Task Scheduler used for aggregation and training.
Each \textit{\textbf{device}} has three threads: for the Sender, Receiver, and Compute Engine. The Sender and Receiver establish the connection with the server. The Compute Engine executes local training and puts activations into the sending queue. At the end of each training iteration, the Compute Engine places the local model in the sending queue and retrieves the global model from the receiving queue to complete aggregation.
\subsection{Experimental Setup}
\label{subsec:setup}
\textbf{Testbeds}:
\MethodName\ is evaluated on two distinct testbeds:
(1) Testbed A consists of a CPU server and 8 Raspberry Pis with no hardware acceleration. This represents a resource-constrained FL system with edge devices, such as low-end smartphones. (2) Testbed B includes a GPU server and 16 Jetson Nanos with GPU-acceleration representative of high-end smartphones. The devices used are akin to those used in cross-device FL systems presented in the literature~\cite{10.1145/3485730.3493444,9762360,fedadapt}.
The frequency of the device CPU/GPU is adjusted to achieve computational heterogeneity. The devices are divided into four groups, and devices within each group have the same compute and memory resources. The technical specification is provided in Table~\ref{tab:testbed}. The bandwidth between the server and the devices is 50 Mbps and 100 Mbps for Testbed~A and Testbed~B, respectively, unless stated otherwise.
\begin{table*}[tp]
\centering
\caption{Specification of the testbed. *Frequency adjusted to achieve computational heterogeneity.}
\begin{tabular}{ccccccc}
\Xhline{2\arrayrulewidth}
\textbf{Testbed} & \textbf{} & \textbf{Group} & \textbf{Platform} & \textbf{CPU} & \textbf{GPU} & \textbf{Memory} \\
\Xhline{2\arrayrulewidth}
\multirow{5}{*}{A} & Server & & Dell Laptop & Intel i7-11850H & N/A & 16 GB \\
\cline{2-7}
& \multirow{4}{*}{Device} & a & Raspberry Pi 3B $\times$ 2 & Broadcom BCM2837 600 MHz* & N/A & 1 GB \\
& & b & Raspberry Pi 3B $\times$ 2 & Broadcom BCM2837 1.2 GHz & N/A & 1 GB \\
& & c & Raspberry Pi 4B $\times$ 2 & Broadcom BCM2711 1.2 GHz* & N/A & 4 GB \\
& & d & Raspberry Pi 4B $\times$ 2 & Broadcom BCM2711 1.8 GHz & N/A & 4 GB \\
\Xhline{2\arrayrulewidth}
\multirow{5}{*}{B} & Server & & Workstation & AMD EPYC 7713p & NVIDIA RTX A6000 & 16 GB \\
\cline{2-7}
& \multirow{4}{*}{Device} & a & Jeston Nano $\times$ 4 & ARM Cortex-A57 1.7 GHz & NVIDIA GM20B 240 MHz* & 4 GB \\
& & b & Jeston Nano $\times$ 4 & ARM Cortex-A57 1.7 GHz & NVIDIA GM20B 320 MHz* & 4 GB \\
& & c & Jeston Nano $\times$ 4 & ARM Cortex-A57 1.7 GHz & NVIDIA GM20B 640 MHz* & 4 GB \\
& & d & Jeston Nano $\times$ 4 & ARM Cortex-A57 1.7 GHz & NVIDIA GM20B 921 MHz & 4 GB \\
\Xhline{2\arrayrulewidth}
\end{tabular}
\label{tab:testbed}
\end{table*}
\textbf{Tasks, DNNs and Datasets}:
Two tasks, namely image classification and sentiment analysis, are considered.
For \textit{image classification}, Testbed~A uses a 5-layer VGG~\cite{vgg} (VGG-5) and CIFAR-10 dataset~\cite{cifar10,cifar10-2} that consists of 50,000 images with 10 classes for training and 10,000 images for testing. On Testbed~B, the MobileNetV3-Large~\cite{mobilenet} for mobile vision apps and the Tiny ImageNet dataset~\cite{tinyimagenet} are chosen. The training and test datasets contain 100,000 and 10,000 images, respectively, with 200 classes.
For sentiment analysis on Testbed A, a transformer~\cite{transformer} (Transformer-6) that consists of 6 transformer encoder layers is trained on the SST-2 dataset~\cite{sst2}, a standard dataset for sentiment analysis comprising 67,349 movie reviews (48,000 for training and 19,349 for testing) with binary sentiment labels. On Testbed B, a larger 12-layer transformer~\cite{transformer} (Transformer-12) is used to recognize complex linguistic patterns. The dataset is IMDB~\cite{imdb} that contains movie reviews with longer sentences. The training and test sets contain 25,000 samples.
The architectures of the DNNs are shown in Table~\ref{table:model-arch}. For VGG-5, `CONV-A-B' represents a convolutional layer of A$\times$A kernel size and of B output channels. `FC-A' represents a fully connected layer with the output size A. For MobileNet-Large, `BNECK-A-B' denotes a bottleneck residual block that consists of an expansion layer, a convolutional layer with the kernel size A$\times$A and a projection layer with the output channel number B. For both Transformer-6 and Transformer-12, `EMB-A' is a work embedding layer with output dimension A, and `ENC-A-B-C' is a transformer encoder layer with embedding dimension A, B heads and hidden dimension C. The number of classes is X. The activation, batch normalization and pooling layers are not shown for simplicity.
The dataset is split in a non-IID manner across devices using the Dirichlet distribution with 0.5 prior based on the literature~\cite{acar2021federated}. Each device is assigned a vector with the size of the number of classes and its values are drawn from a Dirichlet distribution. These vectors represent the class distribution for each device. For each device, a label is randomly selected based on its corresponding vector, and a data point with this label is sampled without replacement. This continues until every data sample is allocated to a device.
\begin{table}[tp]
\centering
\caption{Neural network architectures chosen for this paper.}
\begin{tabular}{ccc}
\Xhline{2\arrayrulewidth}
\textbf{VGG-5} & \textbf{MobileNetV3-Large} & \textbf{Transformer-6} \\
\Xhline{2\arrayrulewidth}
CONV-3-32 & CONV-3-16 & EMB-100 \\
\hline
CONV-3-64 $\times$ 2 & BNECK-3-16 & ENC-100-5-100 $\times$ 6 \\
\hline
FC-128 & BNECK-3-24 $\times$ 2 & FC-X \\
\hline
FC-X & BNECK-5-40 $\times$ 3 \\
\cline{1-2}
& BNECK-3-80 $\times$ 4 & \\
\cline{1-2}
& BNECK-3-112 $\times$ 2 & \textbf{Transformer-12} \\
\hline
& BNECK-5-160 $\times$ 3 & EMB-100 \\
\hline
& CONV-1-960 & ENC-100-50-100 $\times$ 12 \\
\cline{2-3}
& CONV-1-1280 & FC-X \\
\cline{2-3}
& FC-X & \\
\Xhline{2\arrayrulewidth}
\end{tabular}
\label{table:model-arch}
\end{table}
\textbf{Baselines}: The performance of \MethodName\ is compared against six baselines: (a) classic FL~\cite{fedavg}, (b) two offloading-based FL methods, namely SplitFed~\cite{splitfed} and PiPar~\cite{pipar}, (c) two asynchronous FL methods, namely FedAsync~\cite{fedasync} and FedBuff~\cite{fedbuff}, and (d) a hybrid of offloading and asynchronous FL, OAFL (see Section~\ref{subsec:challenge} for details).
The baselines are state-of-the-art methods and are further discussed in Section~\ref{sec:rw}.
The results demonstrate that \MethodName\ has superior training efficiency (Section~\ref{subsec:efficiency}), lower server and device idle time (Section~\ref{subsec:idle-time}), and higher throughput (Section~\ref{subsec:throughput}).
\subsection{Training Efficiency}
\label{subsec:efficiency}
\begin{figure}[tp]
\centering
\subfigure[Testbed A]{
\includegraphics[width=0.22\textwidth]{figs/cpu_acc_non_iid.png}
\label{subfig:cpu_cv_acc}
}
\subfigure[Testbed B]{
\includegraphics[width=0.22\textwidth]{figs/gpu_acc_non_iid.png}
\label{subfig:gpu_cv_acc}
}
\caption{Accuracy (higher is better) versus training time (lower is better) for image classification.}
\label{fig:train-time-cv}
\end{figure}
\begin{figure}[tp]
\centering
\subfigure[Testbed A]{
\includegraphics[width=0.22\textwidth]{figs/cpu_nlp_acc_non_iid.png}
\label{subfig:cpu_nlp_acc}
}
\subfigure[Testbed B]{
\includegraphics[width=0.22\textwidth]{figs/gpu_nlp_acc_non_iid.png}
\label{subfig:gpu_nlp_acc}
}
\caption{Accuracy (higher is better) versus training time (lower is better) for sentiment analysis.}
\label{fig:train-time-nlp}
\end{figure}
End-to-end training for the two tasks, namely image classification and sentiment analysis, is considered for \MethodName\ and the baselines. The wall-clock training time, model convergence rate and final accuracy are measured. The accuracy curves are shown in Figure~\ref{fig:train-time-cv} and Figure~\ref{fig:train-time-nlp}. Each curve shows the validation accuracy versus time for model convergence. A model is considered to have converged when validation accuracy does not improve after 30 epochs.
Figure~\ref{fig:train-time-cv} highlights that on both testbeds, \MethodName\ accelerates model convergence for image classification over all baselines. On Testbed~A, \MethodName\ achieves 82.4\% accuracy in 2.95 hours. FedAsync and OAFL do not achieve comparable accuracy; SplitFed, PiPar, FedBuff and FL achieve comparable accuracy but require 5.4$\times$, 4.3$\times$, 2.6$\times$ and 11.7$\times$ more time. On Testbed~B, the efficiency of \MethodName\ over the baselines is more clear since more layers are trained on the server. \MethodName\ achieves higher accuracy than all baselines at any time during training. Only FL and PiPar achieve comparable final accuracy (42.3\%) to \MethodName\ but require 18.7$\times$ and 10.0$\times$ more time. The final accuracy of \MethodName\ is higher than all baselines.
The higher efficiency of \MethodName\ is also seen for sentiment analysis, as shown in Figure~\ref{fig:train-time-nlp}. On both testbeds, the model in \MethodName\ converges faster than all baselines. On Testbed~A, \MethodName\ achieves 92.84\% accuracy in 1.71 hours which FedAsync, FedBuff and OAFL do not achieve. SplitFed, PiPar and FL achieve comparable final accuracy but require 1.9$\times$, 2.1$\times$ and 10.1$\times$ more time than \MethodName, respectively. On Testbed~B, \MethodName\ outperforms all baselines for convergence time (0.31 hours) and final accuracy (83.4\%). FedBuff achieves the next highest accuracy (81.8\%) but requires 21.8$\times$ more time than \MethodName.
\begin{boxH}
\textbf{Observation 1:} Models trained using \MethodName\ converge faster than all baselines, and achieve final accuracy that is higher or comparable to the best baseline.
\end{boxH}
\subsection{Idle Time Analysis}
\label{subsec:idle-time}
\begin{figure}[tp]
\subfigure[Testbed A - Server Side]{
\includegraphics[width=0.46\linewidth]{figs/cpu_server_idle_non_iid.png}
\label{subfig:cpu_cv_server_idle}
}
\subfigure[Testbed A - Device Side]{
\includegraphics[width=0.46\linewidth]{figs/cpu_device_idle_non_iid.png}
\label{subfig:cpu_cv_device_idle}
}
\subfigure[Testbed B - Server Side]{
\includegraphics[width=0.46\linewidth]{figs/gpu_server_idle_non_iid.png}
\label{subfig:gpu_cv_server_idle}
}
\subfigure[Testbed B - Device Side]{
\includegraphics[width=0.46\linewidth]{figs/gpu_device_idle_non_iid.png}
\label{subfig:gpu_cv_device_idle}
}
\caption{Server and device idle time (lower is better) of image classification.}
\label{fig:idle-time-cv}
\end{figure}
\begin{figure}[tp]
\subfigure[Testbed A - Server Side]{
\includegraphics[width=0.46\linewidth]{figs/cpu_nlp_server_idle_non_iid.png}
\label{subfig:cpu_nlp_server_idle}
}
\subfigure[Testbed A - Device Side]{
\includegraphics[width=0.46\linewidth]{figs/cpu_nlp_device_idle_non_iid.png}
\label{subfig:cpu_nlp_device_idle}
}
\subfigure[Testbed B - Server Side]{
\includegraphics[width=0.46\linewidth]{figs/gpu_nlp_server_idle_non_iid.png}
\label{subfig:gpu_nlp_server_idle}
}
\subfigure[Testbed B - Device Side]{
\includegraphics[width=0.46\linewidth]{figs/gpu_nlp_device_idle_non_iid.png}
\label{subfig:gpu_nlp_device_idle}
}
\caption{Server and device idle time (lower is better) of sentiment analysis.}
\label{fig:idle-time-nlp}
\end{figure}
The idle time of running the two tasks is shown in Figure~\ref{fig:idle-time-cv} and Figure~\ref{fig:idle-time-nlp}. Device idle time is the average of all devices. \MethodName\ has the lowest idle time in all cases. Only FedAsync has a comparable device idle time to \MethodName, but has larger server idle time. Compared to the baseline with the lowest idle time in all cases, \MethodName\ reduces idle time by up to 93.9\% on the server and 81.8\% on the devices.
\begin{boxH}
\textbf{Observation 2:} \MethodName\ has the lowest server and device idle time compared to all baselines.
\end{boxH}
\subsection{System Throughput}
\label{subsec:throughput}
\begin{figure}[tp]
\centering
\subfigure[Testbed A]{
\includegraphics[width=0.22\textwidth]{figs/cpu_throughput_non_iid.png}
\label{subfig:cpu_cv_throughput}
}
\subfigure[Testbed B]{
\includegraphics[width=0.22\textwidth]{figs/gpu_throughput_non_iid.png}
\label{subfig:gpu_cv_throughput}
}
\caption{System throughput (higher is better) for image classification.}
\label{fig:throughput-cv}
\end{figure}
\begin{figure}[tp]
\centering
\subfigure[Testbed A]{
\includegraphics[width=0.22\textwidth]{figs/cpu_nlp_throughput_non_iid.png}
\label{subfig:cpu_nlp_throughput}
}
\subfigure[Testbed B]{
\includegraphics[width=0.22\textwidth]{figs/gpu_nlp_throughput_non_iid.png}
\label{subfig:gpu_nlp_throughput}
}
\caption{System throughput (higher is better) for sentiment analysis.}
\label{fig:throughput-nlp}
\end{figure}
The number of data samples processed per unit time (throughput) is another indicator of system performance. In this section, we compare the throughput of \MethodName\ and baselines in stable and unstable environments.
%\subsubsection{Stable Network Environment}
%\label{subsubsec:stable}
\textbf{Stable Environment}:
Image classification throughput is shown in Figure~\ref{fig:throughput-cv}. \MethodName\ has the highest throughput on both testbeds. On Testbed~A, FedAsync achieves 70\% throughput of \MethodName, while other baselines have a lower throughput. On Testbed~B, \MethodName\ has a 2.0$\times$ throughput gain than the next best, FedAsync. For sentiment analysis, \MethodName\ has the highest throughput. PiPar achieves 92\% throughput of \MethodName\ on Testbed~A. On Testbed B, \MethodName\ outperforms the best baseline (PiPar) by 1.4$\times$.
\textbf{Unstable Environment}:
The throughput of the FL system is affected by an unstable network in which the bandwidth between server and devices changes and devices join and leave at any time. We create an unstable network by randomly and periodically adjusting the bandwidth and adding/removing the devices. The network condition for each device changes every 10 minutes. When the bandwidth changes, the probability of a device leaving the network is $p$, where $0 \leq p \leq 0.5$; otherwise, its bandwidth is randomly set. The bandwidth follows a uniform distribution ranging from 25 Mbps to 50 Mbps for Testbed~A and from 50 Mbps to 100 Mbps for Testbed~B. FedAsync and PiPar are the baselines for image classification and sentiment analysis, respectively, because they have higher throughput than other methods under stable settings.
\begin{figure}[tp]
\centering
\subfigure[Testbed A]{
\includegraphics[width=0.22\textwidth]{figs/cpu_throughput_unstable.png}
\label{subfig:cpu_cv_throughput_unstable}
}
\subfigure[Testbed B]{
\includegraphics[width=0.22\textwidth]{figs/gpu_throughput_unstable.png}
\label{subfig:gpu_cv_throughput_unstable}
}
\caption{Throughput (higher is better) in an unstable network environment with different probabilities of device leaving ($p$) for image classification.}
\label{fig:throughput-cv-unstable}
\end{figure}
\begin{figure}[tp]
\centering
\subfigure[Testbed A]{
\includegraphics[width=0.22\textwidth]{figs/cpu_nlp_throughput_unstable.png}
\label{subfig:cpu_nlp_throughput_unstable}
}
\subfigure[Testbed B]{
\includegraphics[width=0.22\textwidth]{figs/gpu_nlp_throughput_unstable.png}
\label{subfig:gpu_nlp_throughput_unstable}
}
\caption{Throughput (higher is better) in an unstable network environment with different probabilities of device leaving ($p$) for sentiment analysis.}
\label{fig:throughput-nlp-unstable}
\end{figure}
Figure~\ref{fig:throughput-cv-unstable} and Figure~\ref{fig:throughput-nlp-unstable} show the system throughput changes with different $p$ for image classification and sentiment analysis, respectively. The horizontal dashed line is the throughput for \MethodName\ and the baselines in a stable network. In the unstable environment with no devices leaving ($p=0$), \MethodName\ has higher throughput than the baselines in both testbeds and tasks. For image classification (Figure~\ref{fig:throughput-cv-unstable}), with increasing $p$, the throughput of \MethodName\ decreases but is still consistently higher than the baseline (FedAsync). The throughput gap is wider on the GPU-accelerated Testbed~B. For sentiment analysis (Figure~\ref{fig:throughput-cv-unstable}) similar trends are noted.
Throughput reduces for PiPar with increasing $p$, which implies it is more sensitive to devices leaving. A leaving device blocks training in PiPar, but in \MethodName, it does not affect the operation of others.
\begin{boxH}
\textbf{Observation 3:} \MethodName\ achieves the highest throughput compared to all baselines and is less affected by network bandwidth variations and device dropouts.
\end{boxH}
In classic FL~\cite{fedavg}, the entire neural network is trained on a device, which results in server idle time. Offloading-based FL methods improve server utilization. Split learning (SL)~\cite{sl,slh} was the first work to offload a part of the neural network to the server. The first device trains the initial layers and sends activations to the server that trains the subsequent layers. Then the device-side network is sent to the next device. Training performance is low since devices work in a round-robin.
SplitFed~\cite{splitfed} combines SL and FL to split the neural network across the server and devices and train all devices in parallel. Although it improves server utilization, the dependency between server and device training results in idle time. PiPar~\cite{pipar} further reduces server idle time by using pipeline parallelism to optimize computational efficiency of FL. CSE-FSL~\cite{cse-fsl} and SplitGP~\cite{splitgp} use the auxiliary network and local loss on devices to mitigate task dependency. Since devices compute gradients, the server does not send gradients to devices, which reduces the communication overhead. FedGKT~\cite{fedgkt} transfers logit tensors between server and devices to guide training on both sides. An incentive mechanism is proposed to encourage client participation and enhance model accuracy.~\cite{10631047}. However, these methods do not address device stragglers.
To address the straggler problem, FedAsync~\cite{fedasync} employs asynchronous aggregation in FL. Aggregation occurs when any device completes a training round, and a device obtains the global model without waiting for other devices. However, asynchronous aggregation introduces staleness that impacts model accuracy. FedBuff~\cite{fedbuff} balances stragglers and staleness using a buffering mechanism. The server aggregates updates from a subset of devices rather than all devices. Nevertheless, FedBuff also faces model staleness. Libra~\cite{10631034} improves client scheduling in asynchronous FL to mitigate staleness. However, the above methods do not improve server utilization.
In contrast, \MethodName\ simultaneously addresses the task dependency and straggler problems to reduce idle time for both the server and devices while minimizing model staleness and improving model accuracy.
Classic federated learning under-utilizes computational resources, resulting in server and device idle times and low training efficiency. Two types of idle time stem from server-device task dependency (Type~I) and device stragglers (Type~II). \MethodName, a novel resource-optimized federated learning system is developed to simultaneously minimize both types of idle time; existing research can reduce either but not both at the same time.
\MethodName\ redesigns the distributed training of a DNN. Devices train the initial layers of the DNN, and the rest of the layers is offloaded to the server. Using an auxiliary network and asynchronous aggregation, training on a device is decoupled from the server and other devices, which eliminates Type~II idle time. The server undertakes centralized training on the activations received from devices for reducing Type~I idle time. A task scheduler is used to balance the contribution of heterogeneous devices. The server memory is efficiently managed by controlling the activation flow to enhance the scalability of the system.
Compared to the best results of four state-of-the-art offloading-based and asynchronous FL baselines, \MethodName\ achieves higher or comparable accuracy, accelerates training by 1.9$\times$ to 21.8$\times$, reduces server and device idle time by up to 93.9\% and 81.8\%, respectively, and increases throughput by 1.1$\times$ to 2.0$\times$.
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{fedavg}
B.~McMahan, E.~Moore, D.~Ramage, S.~Hampson, and B.~A.~y. Arcas, ``{Communication-Efficient Learning of Deep Networks from Decentralized Data},'' in \emph{20th International Conference on Artificial Intelligence and Statistics}, vol.~54, 2017, pp. 1273--1282.
\bibitem{DBLP:journals/corr/KonecnyMR15}
J.~Kone{\v{c}}n{\'y}, B.~McMahan, and D.~Ramage, ``{Federated Optimization: Distributed Optimization Beyond the Datacenter},'' \emph{8th NIPS Workshop on Optimization for Machine Learning}, 2015.
\bibitem{DBLP:journals/corr/KonecnyMRR16}
J.~Kone{\v{c}}n{\'y}, H.~B. McMahan, D.~Ramage, and P.~Richt{\'{a}}rik, ``{Federated Optimization: Distributed Machine Learning for On-Device Intelligence},'' \emph{CoRR}, vol. abs/1610.02527, 2016.
\bibitem{splitfed}
C.~Thapa, M.~A.~P. Chamikara, and S.~Camtepe, ``{SplitFed: When Federated Learning Meets Split Learning},'' \emph{AAAI Conference on Artificial Intelligence}, vol. 36(8), pp. 8485--8493, 2022.
\bibitem{pipar}
Z.~Zhang, P.~Rodgers, P.~Kilpatrick, I.~Spence, and B.~Varghese, ``{PiPar: Pipeline Parallelism for Collaborative Machine Learning},'' \emph{Journal of Parallel and Distributed Computing}, vol. 193, p. 104947, 2024.
\bibitem{cse-fsl}
Y.~Mu and C.~Shen, ``{Communication and Storage Efficient Federated Split Learning},'' in \emph{IEEE International Conf. on Communications}, 2023.
\bibitem{fedgkt}
C.~He, M.~Annavaram, and S.~Avestimehr, ``{Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge},'' in \emph{34th International Conference on Neural Information Processing Systems}, 2020.
\bibitem{10631047}
P.~Han, C.~Huang, X.~Shi, J.~Huang, and X.~Liu, ``{Incentivizing Participation in SplitFed Learning: Convergence Analysis and Model Versioning},'' in \emph{2024 IEEE 44th International Conference on Distributed Computing Systems}, 2024, pp. 846--856.
\bibitem{fedasync}
C.~Xie, O.~Koyejo, and I.~Gupta, ``{Asynchronous Federated Optimization},'' in \emph{12th Workshop on Optimization for Machine Learning}, 2023.
\bibitem{fedbuff}
J.~Nguyen, K.~Malik, H.~Zhan, A.~Yousefpour, M.~Rabbat, M.~Malek, and D.~Huba, ``{Federated Learning with Buffered Asynchronous Aggregation},'' in \emph{Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, vol. 151, 2022, pp. 3581--3607.
\bibitem{10631034}
C.~Wang, H.~Huang, R.~Li, J.~Liu, T.~Cai, and Z.~Zheng, ``{Libra: A Fairness-Guaranteed Framework for Semi-Asynchronous Federated Learning},'' in \emph{2024 IEEE 44th International Conference on Distributed Computing Systems}, 2024, pp. 797--808.
\bibitem{mobilenet}
A.~Howard, M.~Sandler, G.~Chu, L.~Chen, B.~Chen, M.~Tan, W.~Wang, Y.~Zhu, R.~Pang, V.~Vasudevan, Q.~V. Le, and H.~Adam, ``{Searching for MobileNetV3},'' \emph{IEEE/CVF International Conference on Computer Vision}, pp. 1314--1324, 2019.
\bibitem{tinyimagenet}
O.~Russakovsky, J.~Deng, H.~Su, J.~Krause, S.~Satheesh, S.~Ma, Z.~Huang, A.~Karpathy, A.~Khosla, M.~S. Bernstein, A.~C. Berg, and L.~Fei{-}Fei, ``{ImageNet Large Scale Visual Recognition Challenge},'' \emph{CoRR}, vol. abs/1409.0575, 2014.
\bibitem{vgg}
K.~Simonyan and A.~Zisserman, ``{Very Deep Convolutional Networks for Large-scale Image Recognition},'' \emph{3rd International Conference on Learning Representations}, p. 1–14, 2015.
\bibitem{resnet}
K.~He, X.~Zhang, S.~Ren, and J.~Sun, ``{Deep Residual Learning for Image Recognition},'' in \emph{IEEE Conference on Computer Vision and Pattern Recognition}, 2016, pp. 770--778.
\bibitem{dp}
M.~Abadi, A.~Chu, I.~Goodfellow, H.~B. McMahan, I.~Mironov, K.~Talwar, and L.~Zhang, ``{Deep Learning with Differential Privacy},'' in \emph{Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security}, 2016, p. 308–318.
\bibitem{pixeldp}
M.~Lecuyer, V.~Atlidakis, R.~Geambasu, D.~Hsu, and S.~Jana, ``{Certified Robustness to Adversarial Examples with Differential Privacy},'' in \emph{2019 IEEE Symposium on Security and Privacy (SP)}, 2019, pp. 656--672.
\bibitem{oort}
F.~Lai, X.~Zhu, H.~V. Madhyastha, and M.~Chowdhury, ``{Oort: Efficient Federated Learning via Guided Participant Selection},'' in \emph{15th {USENIX} Symposium on Operating Systems Design and Implementation}, 2021.
\bibitem{refl}
A.~M. Abdelmoniem, A.~N. Sahu, M.~Canini, and S.~A. Fahmy, ``{REFL: Resource-Efficient Federated Learning},'' in \emph{Proceedings of the Eighteenth European Conference on Computer Systems}, 2023, p. 215–232.
\bibitem{10.1145/3485730.3493444}
T.~Zhang, C.~He, T.~Ma, L.~Gao, M.~Ma, and S.~Avestimehr, ``{Federated Learning for Internet of Things},'' in \emph{19th ACM Conference on Embedded Networked Sensor Systems}, 2021, p. 413–419.
\bibitem{9762360}
Y.~Jiang, S.~Wang, V.~Valls, B.~J. Ko, W.-H. Lee, K.~K. Leung, and L.~Tassiulas, ``{Model Pruning Enables Efficient Federated Learning on Edge Devices},'' \emph{IEEE Transactions on Neural Networks and Learning Systems}, vol.~34, no.~12, pp. 10\,374--10\,386, 2023.
\bibitem{fedadapt}
D.~Wu, R.~Ullah, P.~Harvey, P.~Kilpatrick, I.~Spence, and B.~Varghese, ``{FedAdapt: Adaptive Offloading for IoT Devices in Federated Learning},'' \emph{IEEE Internet of Things Journal}, vol.~9, no.~21, pp. 20\,889--20\,901, 2022.
\bibitem{cifar10}
A.~Krizhevsky, V.~Nair, and G.~Hinton, ``{CIFAR-10 (Canadian Institute for Advanced Research)},'' 2009.
\bibitem{cifar10-2}
A.~Krizhevsky and G.~Hinton, ``{Learning Multiple Layers of Features from Tiny Images},'' \emph{Master's thesis, Department of Computer Science, University of Toronto}, 2009.
\bibitem{transformer}
A.~Vaswani, N.~Shazeer, N.~Parmar, J.~Uszkoreit, L.~Jones, A.~N. Gomez, L.~u. Kaiser, and I.~Polosukhin, ``{Attention is All you Need},'' in \emph{Advances in Neural Information Processing Systems}, vol.~30, 2017.
\bibitem{sst2}
R.~Socher, A.~Perelygin, J.~Wu, J.~Chuang, C.~D. Manning, A.~Ng, and C.~Potts, ``{Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank},'' in \emph{Proceedings of the Conference on Empirical Methods in Natural Language Processing}, 2013, pp. 1631--1642.
\bibitem{imdb}
A.~L. Maas, R.~E. Daly, P.~T. Pham, D.~Huang, A.~Y. Ng, and C.~Potts, ``{Learning Word Vectors for Sentiment Analysis},'' in \emph{Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies}, June 2011, pp. 142--150.
\bibitem{acar2021federated}
D.~A.~E. Acar, Y.~Zhao, R.~Matas, M.~Mattina, P.~Whatmough, and V.~Saligrama, ``{Federated Learning Based on Dynamic Regularization},'' in \emph{International Conference on Learning Representations}, 2021.
\bibitem{sl}
O.~Gupta and R.~Raskar, ``{Distributed Learning of Deep Neural Network over Multiple Agents},'' \emph{Journal of Network and Computer Applications}, vol. 116, pp. 1--8, 2018.
\bibitem{slh}
P.~Vepakomma, O.~Gupta, T.~Swedish, and R.~Raskar, ``{Split Learning For Health: Distributed Deep Learning without Sharing Raw Patient Data},'' in \emph{ICLR Workshop on AI for Social Good}, 2019.
\bibitem{splitgp}
D.-J. Han, D.-Y. Kim, M.~Choi, C.~G. Brinton, and J.~Moon, ``{SplitGP: Achieving Both Generalization and Personalization in Federated Learning},'' in \emph{IEEE Conference on Computer Communications}, 2023.
\end{thebibliography}