NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients
Extracted source text · arXiv 2308.07761
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2022
% ready for submission
% \usepackage[nonatbib]{neurips_2022}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2022}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2022}
% to avoid loading the natbib package, add option nonatbib:
\usepackage[nonatbib, preprint]{neurips_2022}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{multirow,graphicx}
\usepackage{color, colortbl}
\usepackage{multirow, makecell}
\usepackage{rotating}
\usepackage{subcaption}
\usepackage{wrapfig}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage{amsmath}
\definecolor{Gray}{gray}{0.9}
\usepackage{bm}
\makeatletter
\def\algbackskip{\hskip-\ALG@thistlm}
\makeatother
\title{NeFL: Nested Federated Learning \\for Heterogeneous Clients}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Honggu ~Kang \\
% \thanks{Use footnote for providing further information
% about author (webpage, alternative address)---\emph{not} for acknowledging
% funding agencies.} \\
KAIST\\
\texttt{khg13@kaist.ac.kr} \\
% examples of more authors
\And
Seohyeon ~Cha \\
KAIST \\
\texttt{kaitjgus@kaist.ac.kr} \\
\And
Jinwoo ~Shin \\
KAIST \\
\texttt{jinwoos@kaist.ac.kr} \\
\And
Jongmyeong ~Lee \\
KAIST \\
\texttt{whenstyle07@kaist.ac.kr} \\
\And
Joonhyuk ~Kang \\
KAIST \\
\texttt{jkang@kaist.ac.kr} \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
% Recently, distributed learning and privacy concerns have become increasingly important in the field of machine learning. Federated learning (FL) has emerged as a promising approach to address these challenges.
Federated learning (FL) is a promising approach in distributed learning keeping privacy.
However, during the training pipeline of FL, slow or incapable clients (i.e., stragglers) slow down the total training time and degrade performance.
System heterogeneity, including heterogeneous computing and network bandwidth, has been addressed to mitigate the impact of stragglers. Previous studies split models to tackle the issue, but with less degree-of-freedom in terms of model architecture. We propose {\it nested federated learning (NeFL)}, a generalized framework that efficiently divides a model into submodels using both depthwise and widthwise scaling. NeFL is implemented by interpreting models as solving ordinary differential equations (ODEs) with adaptive step sizes. To address the {\it inconsistency} that arises when training multiple submodels with different architecture, we decouple a few parameters.
NeFL enables resource-constrained clients to effectively join the FL pipeline and the model to be trained with a larger amount of data. Through a series of experiments, we demonstrate that NeFL leads to significant gains, especially for the worst-case submodel (e.g., \textbf{8.33\%} improvement on CIFAR-10). Furthermore, we demonstrate NeFL aligns with recent studies in FL.
% Somewhat interestingly, by investigating how to utilize pre-trained models within the NeFL, we found that Vision Transformers (ViTs) hold promise in addressing both system and statistical heterogeneity.
% We also provide insights into utilizing
%. Based on the insights,
%Our code is available at \hyperlink{https://github.com/will-be-released-after-blind-review-process}{here}.
% We conducted a number of experiments to evaluate the performance of NeFL and observed that NeFL leads to significant gains especially on the worst-case submodel compared to baselines.
% In addition, we conducted experiments to investigate the NeFL for transfer learning. Specifically, we attempted to figure out whether NeFL could effectively leverage pretrained models to improve the performance of FL.
% The results show that depthwise scaling is more effective than (i) widthwise or (ii) integrally depthwise and widthwise.
% To go further, we studied the results of NeFL under statistical heterogeneity and found that NeFL with transformers are robust to the statistical heterogeneity.
% Overall, our work offers a remarkable general approach that resolves the heterogeneity challenges of FL.
% url{https://github.com/will-be-released-after-blind-review-process}.
% while providing a more generalized framework.
% With the surging interest on distributed learning and privacy issues, federated learning (FL) has been received attention. FL must tolerate stragglers that degrades the performance as well as the training time. The stragglers could be mitigated by tackling the system heterogeneity (e.g., computing and network bandwidth). There have been rigorous studies on mitigating the system heterogeneity, but models are restrictively splitted. Restrictively splitted models cause model-size-inefficiency that careful splitting is needed.
% In this work, we propose {\it nested federated learning (NeFL)}, a generalized framework of efficiently splitting a global model into submodels for embracing resource-constrained clients in the FL pipeline. The submodels are extracted from a global model without retraining.
% We further studied on the performance of NeFL for the statistical heterogeneity and transfer learning. The results show that vision transformers (ViTs) are robust to the statistical heterogeneity and depthwise scaling is more effective for transfer learning. Extensive evaluation across diverse methods shows NeFL leads to performance gain as well as providing a generalized framework over previous work.
\end{abstract}
\section{Introduction}
% The development of deep learning has been explosively increased.
The success of deep learning owes much to large-scale computing infrastructures and vast amounts of training data where a large amount of data comes from mobile devices and internet-of-things (IoT) devices. Recently, privacy regulations regarding data collection have gained attention, potentially impeding development \cite{europeData, dou2021federated}.
A distributed machine learning framework, federated learning (FL) is getting attention to address the privacy concerns. FL enables model training by collaboratively leveraging the vast amount of data on clients while preserving data privacy. Rather than centralizing raw data, FL collects trained model weights from clients, which are aggregated on a server by a method (e.g., FedAvg) \cite{mcmahan2017communication}. FL has shown its potential, and several studies have explored to utilize it more practically \cite{liang2020think, hong2022efficient, lin2020ensemble, he2020group, haddadpour2021federated, makhija2022federated, zhuang2022divergenceaware, he2021ssfl, horvath2021fjord, kim2023depthfl}.
% -- Research regarding to FL --
Despite the promising perspective, FL faces challenges of systems-related heterogeneity \cite{kairouz2021advances,Li2020federated}. Clients with heterogeneous resources, such as computing power, communication bandwidth, and memory, introduce stragglers that lead to longer delays and training times during the FL pipeline. The FL server must make decisions on whether to wait for or drop out stragglers. The exclusion of stragglers can result in performance degradation, such as reduced prediction accuracy. Therefore, FL must accommodate clients with heterogeneous resources.
Mobile and embedded devices, with their limited memory, storage, and network bandwidth, often struggle when training a large model. These resource-poor clients risk becoming stragglers, leading to a biased model trained predominantly on data from resource-rich clients. Otherwise, reducing the global model size to accommodate poor clients can degrade performance due to limited model capacity. To this end, FL with a single global model may not be efficient for heterogeneous clients.
In this paper, we propose {\it Nested Federated Learning (NeFL)}, a method that embraces existing studies of federated learning in a nested manner \cite{horvath2021fjord,diao2021heterofl,kim2023depthfl}. While FL trains a model with a fixed size and structure, NeFL trains several submodels of adaptive sizes to meet clients' dynamic requirements (e.g., memory, computing and bandwidth dynamics). We propose to scale down model into submodels generally (by widthwise or/and depthwise). The scaling is motivated by interpreting a model forwarding as solving ordinary differential equations (ODEs). The interpretation also motivated us to propose the submodels to have {\it learnable step size parameters} and the concept of {\it inconsistency}. The scaling method provides more degree-of-freedom (DoF) to scale down a model than previous studies \cite{horvath2021fjord,diao2021heterofl,kim2023depthfl}. The increased DoF make submodels to be more efficient in size \cite{mingxing2019efficient} and provides more flexibility on model size and computing cost.
We also propose a parameter averaging method for NeFL: {\it Nested Federated Averaging (NeFedAvg)} for averaging {\it consistent parameters} and FedAvg for averaging {\it inconsistent parameters} of submodels.
% Depthwise-scaling is implemented by residual connections \cite{he2016deep} using deterministic depth, ODEs, and adaptive depth, while width-scaling is achieved through contiguous pruning.
Notably, NeFL outperforms the Top-1 accuracy of baselines by \textbf{8.33\%} for the worst-case submodel of ResNet110.
Additionally, we verify if NeFL aligns with the recently proposed ideas in FL: (i) pre-trained models improve the performance of FL in both identically independently distributed (IID) and non-IID settings \cite{chen2023on} and (ii) simply rethinking the model architecture improves the perofrmance, especially in non-IID settings \cite{qu2022rethinking}.
% we provide guidelines for employing pre-trained models with large dataset in FL to address heterogeneity.
% While pre-trained models significantly improve the FL performance, depthwise scaling demonstrates better results than widthwise scaling since the pre-trained models are not trained by structured contiguous pruning. With the fact that NeFL works well with pre-trained models, we evaluate the performance of NeFL with Vision Transformers (ViTs) \cite{dosovitskiy2021an} under statistical heterogeneity.
We observe that NeFL with ViTs shows \textbf{22.7\%} better performance than previous widthwise scaling based method with ViTs for IID dataset and \textbf{24.49\%} for non-IID dataset with submodel employing 50\% parameters of a global model (to be revised into worst-case or avergae performance).
Through a series of experiments we observe that NeFL outperforms baselines sharing the advantages of recent studies.
The main contributions of this study can be summarized as follows:
\begin{itemize}
\item We propose a general model scaling method employing the concept of ODE solver to deal with the system heterogeneity.
\item We propose a parameter averaging method of the generally scaled submodels by NeFedAvg and FedAvg.
\item We evaluate the performance of NeFL through a series of experiments and verify the applicability of NeFL over recent studies.
\end{itemize}
\section{Related Works}
\paragraph{Knowledge distillation}
Knowledge distillation (KD) aims to compress models by transferring knowledge from a large teacher model to a smaller student model \cite{hinton2015distilling}. Several studies have explored the integration of knowledge distillation within the context of federated learning \cite{seo2020federated, wu2022fedkd, afonin2022towards, lin2020ensemble, zhu2021datafree, he2020fedgkt}. These studies investigate the use of KD to address (i) reducing the model size and communication bottleneck associated with transmitting model weights and (ii) fusing the knowledge of several models with different architectures.
FedKD \cite{wu2022fedkd} proposes an adaptive mutual distillation where distillation is controlled based on prediction performance. FedGKT \cite{he2020fedgkt} presents a edge-computing method that employs knowledge distillation to train resource-constrained devices.
The method partitions the model, and clients transfer intermediate features to an offloading server for task offloading. FedDF \cite{lin2020ensemble} introduces a model fusion technique that employs ensemble distillation to combine models with different architecture.
However, it is worth noting that knowledge distillation-based FL requires shared data or shared generative models to get the knowledge distillation loss. Alternatively, clients can transfer trained models to the other clients \cite{afonin2022towards}.
% This may violate the benefits of federated learning of privacy. by keeping raw data at local clients.
\paragraph{Compression and sparsification}
Several studies have focused to deal with the communication bottleneck \cite{rothchild2020fetchsgd, haddadpour2021federated}. The literature studies gradient compression (e.g., quantization or sparsification) and reducing communication rounds \cite{haddadpour2021federated}.
FetchSGD \cite{rothchild2020fetchsgd} proposes a method that compresses model updates using a Count Sketch and merge them. Some approaches aim to represent the weights of a larger network using a smaller network \cite{ha2017hypernetworks, cristian2006model}. However, these studies primarily address heterogeneous communication bandwidth. Pruning is another technique that can be used to compress models. However, it typically determines the submodel based on the worst case of resource-constrained clients, which means it does not fully leverage the capabilities of the resource-rich clients \cite{lee2020signal, jiang2022pruningFL, Mugunthan2022fedltn}.
\paragraph{Dynamic runtime}
The concept of interpreting neural networks with residual connections as numerically solving ODEs was introduced in NeuralODE
\cite{chen2018node}. By using numerical techniques such as computing with less steps, the ODE can be solved with reduced computational requirements. Otherwise, with more computing, the ODE can be solved achieving less numerical errors \cite{ODEbook}.
This approach offers a way to dynamically optimize computation.
Split learning \cite{vepakomma2018split, thapa2022splitfed, he2020fedgkt} is another technique that addresses the issue of resource-constrained clients by leveraging richer computing resources, such as cloud or edge servers. The methods provide solutions for clients with heterogeneous computing power or runtime dynamics.
However, these methods primarily studies on improving the efficiency of resource allocation or computational distribution, rather than explicitly addressing system heterogeneity.
\paragraph{Model splitting}
Various approaches have been proposed to address the heterogeneity of clients by splitting the global network based on their capabilities \cite{horvath2021fjord, diao2021heterofl, kim2023depthfl}. LG-FedAvg \cite{liang2020think} introduced a method to split the model and decouple the layers into global and local layers, reducing the number of parameters that need to be communicated.
While FjORD \cite{horvath2021fjord} and HeteroFL \cite{diao2021heterofl} split a global model widthwise, DepthFL \cite{kim2023depthfl} split a global model depthwise. Unlike widthwise scaling, DepthFL incorporates an additional bottleneck layer and an independent classifier for every submodel.
However, it has been observed that {\it deep and narrow} models as well as {\it shallow and wide} models may result in inefficient performance in terms of the number of parameters or floating-point operations (FLOPs). Prior studies have shown that carefully balancing the depth and width of a network can lead to improved performance \cite{sergey2016wide,mingxing2019efficient}. Therefore, a balanced network is expected to contribute to performance gains in the context of FL.
% The skip connections allows to train very deep networks but at the same time, gradients might bypass through the skip connections so that the residual blocks cannot useful representations.
\begin{figure}[!t]
\centering
% \includegraphics[width=5cm, height=4cm]{fig/nefl.png}
\includegraphics[width=5.5in]{fig/NeFL_main.pdf}
\caption{\textbf{FL with generally split submodels}. NeFL aggregates weights of submodels, where the submodels are scaled by both widthwise and depthwise. The weights include consistent and inconsistent parameters.}
\label{fig:head}
\end{figure}
\section{Background}\label{background}
\begin{figure}[!t]
\centering
% \includegraphics[width=5cm, height=4cm]{fig/nefl.png}
\includegraphics[width=5.5in]{fig/nefl_graphs2.pdf}
\caption{\textbf{Model scaling inspired by ODE solver}}
\label{fig:graph}
\end{figure}
We propose to scale down the models inspired by solving ODEs in a numerical way (e.g., Euler method). Modern deep neural networks stack residual blocks that contain identity skip-connections that bypass the residual layers. A residual block is written as $\mathbf{Y}_{j+1} = \mathbf{Y}_{j} + F_j(\mathbf{Y}_j, \phi_j)$, where $\mathbf{Y}_j$ is the feature map at the $j$th layer, $\phi_j$ denotes the $j$th block’s network parameters, and $F_j$ represents a residual module of the $j$th block. These networks can be interpreted as solving ODEs by numerical analysis \cite{chang2018multilevel,he2016deep}.
Consider an initial value problem to find $y$ at $t$ given $\frac{dy}{dt}=f(t,y)$ and $y(t_0)=y_0$. We can obtain $y$ at any point by integration: $y = y_0 + \int_{t_0}^t f(t,y) dt$. It can be approximated by Taylor’s expansion as $ y = y_0 + f(t_0,y_0) \left(t-t_0\right)$ and it can also be approximated with more steps as follows: % + \mathcal{O}\left(\left(t-t_0 \right)^2 \right)
\begin{equation}
y_{n+1} = y_n + hf(t_n, y_n)= y_0 + hf(t_0, y_0) + \cdots + hf(t_{n-1}, y_{n-1}) + hf(t_n, y_n),\label{eqn:euler}
\end{equation}
where $h$ denotes the step size. An ODE solver can design to compute less steps with larger step size for example as follows: $y_{n+1} = y_0 + 2hf(t_0, y_0) + \cdots + 2hf(t_{n-1}, y_{n-1})$ when $n$ is odd. The results would be numerically less accurate than fully computing with a smaller step size.
Note that the equation looks like the equation of residual connections.
An output of a residual block is rewritten as follows:
\begin{equation}
\mathbf{Y}_{j+1} = \mathbf{Y}_{j} + F_j(\mathbf{Y}_j, \phi_j) = \mathbf{Y}_{0} + F_0(\mathbf{Y}_0, \phi_0) + \cdots + F_{j-1}(\mathbf{Y}_{j-1}, \phi_{j-1}) + F_{j}(\mathbf{Y}_j, \phi_j).
\end{equation}
Motivated by interpreting the neural networks as solving ODEs, we proposed that few residual blocks can be omitted during forward propagation. For example, $\mathbf{Y}_3=\mathbf{Y}_0+F_0+F_1+F_2$ could be approximated by $\mathbf{Y}_3=\mathbf{Y}_0+F_0+2F_1$ omitting the $F_2$ block. Here, we also propose {\it learnable step size parameters} \cite{touvron2021going, bachlechner2021rezero}. Instead of pre-determining the step size parameters ($h$’s in the equation \ref{eqn:euler}), we let step size parameters be trained with the network parameters. For example, omitting $F_2$ block, $\mathbf{Y}_3=\mathbf{Y}_0+s_0 F_0+ s_1 F_1$ where $s_i$’s are also optimized.
We scale the network by this ODE interpretation.
Furthermore, it is possible to scale each residual block by adjusting its width (e.g., the size of filters in convolutional layers). The theoretical background behind width-wise scaling is provided by the Eckart-Young-Mirsky theorem \cite{eckart1936approximation}. A width-wise scaled residual block represents the optimal $k$-rank approximation of the original (full) block \cite{horvath2021fjord}.
Referring to Figure \ref{fig:graph}, the black line represents the function to approximate, while the red line represents a discretized ode solver that is a full neural network model. The blue color line represents the depthwise-scaled submodel. The model omitted to compute block $F_2(\cdot)$ at the point $\mathbf{Y}_2$ and instead, larger step size for computing $F_1(\mathbf{Y}_1,\phi_1)$ can compensate the omitted block \cite{chang2018multilevel}. The pink colored line that represents the widthwise-scaled submodel. Following the theorem that a model of less parameters are approximation of the a model of more parameters \cite{eckart1936approximation}, a widthwise scaled model with fewer parameters for each block outputs a result, but with numerical error.
% The discretized function is obtained by taking gradients of an original function $f=\frac{\partial g}{\partial t}$ at certain points $t_0,\dots,t_k$ with step sizes $h_t$.
% By discretization, we can approximate the function given an initial point ($g(T) = g_0+\sum_t h_t f(t)$).
\section{NeFL: Nested Federated Learning}
\label{sec:method}
% We introduce methods to split a model. We propose to split model depthwise by deterministic depth, ODE depth and adaptive depth while we split widthwise orderly.
{\it NeFL} implements FL that accomodate resource-poor clients. Instead of requiring every client to train a single global model, NeFL allows resource-poor clients to train submodels that are scaled in both widthwise and depthwise, based on the dynamic nature of their environments. This flexibility enables more clients, even with constrained resources, to pariticipate in the FL pipeline, thereby providing the model with a larger and more diverse dataset.
In the NeFL pipeline, heterogeneous clients have the ability to determine their submodel every iteration, allowing them to adapt to varying computing or communication bottlenecks in their respective environments. The NeFL server then aggregates the parameters from these different submodels, resulting in a collaborative and comprehensive model.
Furthermore, during test time, clients have the flexibility to select an appropriate submodel based on their specific background process burdens, such as runtime memory constraints or CPU/GPU bandwidth dynamics. This allows clients to balance between performance and factors like memory usage or latency, tailoring to their individual needs and constraints.
To this end, NeFL consists of method for scaling a model into submodels and aggregating the parameters of submodels.
NeFL scales a global model $\mathbf{F}_G$ into $N_s$ submodels $\mathbf{F}_1,\dots,\mathbf{F}_{N_s}$ with corresponding weights $\theta_1,\dots,\theta_{N_s}$. Without loss of generality, we suppose $\mathbf{F}_G=\mathbf{F}_{N_s}$. The scaling method is described in following Section \ref{subsec:model_scaling}. During each communication round $t$, the clients in subset $\mathcal{C}_t$ (which is subset of $M$ clients) train one of the $N_s$ submodels based on their respective environments by the local epochs $E$. Then, clients transmit their trained weights $\{\mathbf{w}_i\}_{i \in \mathcal{C}_t}$ to the NeFL server which aggregates them into $\{\theta_{\textrm{c},N_s},\theta_{\textrm{ic},1},\dots\theta_{\textrm{ic},N_s}\}$ where $\theta_{\textrm{c},k}$ and $\theta_{\textrm{ic},k}$ denote {\it consistent} and {\it inconsistent} parameters of a submodel $k$. The aggregated weights are then distributed to the clients in $\mathcal{C}_{t+1}$, and this process continues for the total number of communication rounds.
\begin{algorithm}[t]
\caption{NeFL: Nested Federated Learning}\label{alg:NeFL}
{\footnotesize
% \hspace*{\algorithmicindent}
\textbf{Input:} Submodels $\{\mathbf{F}_k\}_{k=1}^{N_s}$, total communication round $T$, the number of local epochs $E$
% \hspace*{\algorithmicindent} \textbf{Output}
\begin{algorithmic}[1]
% \Procedure{MyProcedure}{}
% \Procedure{MyProcedure}{$x,y$}
% % Input:
% \Comment{Input: x}
% % Output:
% \Comment{Output:y}
\For{$t \gets 0$ to $T-1$} // Global iterations %\Comment{Global iterations}
\State NeFL server broadcasts the weights $\{\theta_{\textrm{c},N_s},\theta_{\textrm{ic},1},\dots\theta_{\textrm{ic},N_s}\}$ to clients in $\mathcal{C}_t$
\State $\mathbf{w}_i \gets \theta_j\in\{\theta_1,\dots,\theta_{N_s}\}$ $\forall i \in \mathcal{C}_t$
% \Comment{Client $i$ determines what submodel to train according to the environments $\forall i \in \mathcal{C}_t$}
\For{$k \gets 0$ to $E-1$} // Local iterations %\Comment{Local iterations}
\State Client $i$ updates ${\mathbf{w}_i}$ $\forall i \in \mathcal{C}_t$
\EndFor
\State Client $i$ transmits the weights $\mathbf{w}_{i}$ to the NeFL server $\forall i \in \mathcal{C}_t$
\State $\{\theta_{\textrm{c},N_s}, \theta_{\textrm{ic},1}, \dots\theta_{\textrm{ic},N_s}\} \gets \texttt{ParameterAverage}(\{\mathbf{w}_i \}_{i \in \mathcal{C}_t}$) \Comment{Algorithm \ref{alg:NeFedAvg}}
\EndFor
\end{algorithmic}}
\end{algorithm}
\subsection{Model Scaling} \label{subsec:model_scaling}
We propose a global network slimming method which is a combined method combined of widthwise scaling and depthwise scaling. We scale the model widthwise by ratio of $\gamma_W$ and depthwise by ratio of $\gamma_D$. For example, a global model is represented as $\gamma=\gamma_W \gamma_D=1$ and a submodel that has 25\% parameters of a global model can be scaled by $\gamma_W=0.5$ and $\gamma_D=0.5$. Unless otherwise indicated, we split the model by $\gamma_W = \gamma_D$ \cite{mingxing2019efficient}.
We empirically observed that comparatively shallow and wide models are better to employ widthwise splitting while deep and narrow models are better to employ depthwise splitting for from-the-scratch training. Furthermore, flexible widthwise/depthwise scaling provide more degree of freedom to split the models. We further describe on the slimming strategies in following sections.
% \footnote{ResNet18 has enough width (channels) but deficient length to be split resulting in deficient degree of freedom to split depthwise.}
\subsubsection{Depthwise Scaling} \label{subsec:ode_depth}
% However, it has also been noted that residual connections prevent blocks to learn useful representations \cite{mingxing2019efficient}.
Residual networks, such as ResNets \cite{he2016deep} and ViTs \cite{dosovitskiy2021an}, have gained popularity due to their ability to train {\it deep} networks with significant performance improvements. The output of a residual block can be seen as a function multiplied by a step size as $ \mathbf{Y}_{j+1} = \mathbf{Y}_{j} + s_j F(\mathbf{Y}_j, \phi_j)=\mathbf{Y}_{0} + \sum_{j} s_j F(\mathbf{Y}_j, \phi_j)$. Note that ViTs forwards with skip connections using either self-attention (SA) layers or feed-forward networks (FFN) that $F$ could be either SA or FFN:
\begin{equation}
\mathbf{Y}_{j+1} = \mathbf{Y}_{j} + s_j SA(\mathbf{Y}_j, \phi_j),\quad
\mathbf{Y}_{j+2} = \mathbf{Y}_{j+1} + s_{j+1} FFN(\mathbf{Y}_{j+1}, \phi_{j+1}).
\label{eqn:vit}
\end{equation}
For example, ResNets (e.g., ResNet18, ResNet34) have a downsampling layer at the first and following residual blocks have two convolutional layers respectively and ViTs let embedding patches input into following transformer encoder blocks.
ResNet34 having 16 residual blocks or ViT/B-16 have 12 encoder blocks and depthwise scaling can be implemented by skipping any of these blocks. Each block could have different number of the parameters or computing cost that it could be designed for modeling a submodel.
We can observe that skipping a few blocks in a network still allows it to operate effectively \cite{chang2018multilevel}. This observation is in line with the concept of stochastic depth proposed in \cite{huang2016stochasticDepth}, where a subset of blocks is randomly bypassed during training and the all of the blocks are used during testing.
% Figure \ref{fig:l2norm} presents the L2-norm of the output of layers. The flat line across layer denotes that the submodel does not have blocks so that its output has same value until the forwarding existing blocks. We can observe the L2-norm value does not significantly vary without blocks. (Details on the figure should be added)
The step size parameters $s$'s can be viewed as dynamically training how much forward pass should be made at each step. They allow each block to adaptively contribute useful representations, and larger step sizes are multiplied to blocks that provide valuable information. Referring to Figure \ref{fig:ode_solver} and numerical analysis methods (e.g., Euler method, Runge-Kutta methods \cite{ODEbook}), we can observe that adaptive step sizes, rather than uniformly spaced steps, can effectively reduce numerical errors.
{\it Note that DepthFL \cite{kim2023depthfl} is a special case of the proposed depthwise scaling.}\footnote{As a baseline, we implemented DepthFL without auxiliary bottleneck layers and self-distillation and implemented by pruning depthwise non-contiguously for fair comparison and ablation study that provides insights. The details are described in the Appendix.}
% It has been observed that deeper networks can suffer from decreased performance. The issue can be resolved by introducing additional (step size) parameters that are multiplied with the output values of blocks such as $F(\cdot)$, $SA(\cdot)$, $FFN(\cdot)$ \cite{touvron2021going, bachlechner2021rezero}. These multiplied parameters can be either per-block parameters \cite{bachlechner2021rezero} or per-channel parameters \cite{touvron2021going}. The introduced parameters led to performance improvement as well as facilitation of the training deep networks.
\subsubsection{Widthwise Scaling} \label{subsec:width_scaling}
% \begin{equation}
% \mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+F_2(t_2).
% \end{equation}
% \subsubsection{NeFL Adaptive Depth} \label{subsec:ada_depth}
% \begin{wrapfigure}{r}{0.3\textwidth}
% \begin{center}
% \includegraphics[width=0.28\textwidth]{fig/ode_ad.png}
% \end{center}
% \caption{Step size parameters learns how much to forward in perspective of solving ODEs.}
% \label{fig:ODE_AD}
% \vspace{-0.4cm}
% \end{wrapfigure}
% NeFL adaptive depth ({\it NeFL-AD}) forwards as
% \begin{equation} \label{eqn:step_size}
% \mathbf{Y}_{j+1}=\mathbf{Y}_j+s_j F_j.
% \end{equation}
% A $j+2$-th output of reduced model without block $j+1$ is computed as
% % \begin{equation}
% $\mathbf{Y}_{j+2}=\mathbf{Y}_j+s_{j+1} F_{j+1}.$
% % \end{equation}
% Hereafter, we denote NeFL-AD with step size parameters initialized with deterministic depth as {\it NeFL-AD\textsubscript{D}}, and initialized with ODE depth as {\it NeFL-AD\textsubscript{O}}. Figure \ref{fig:steps} presents the step sizes of trained submodels. We can observe that step sizes show similar trend, but slightly different. The slightly different step sizes could compensate for the potential numerical error in perspective of ODE (Figure \ref{fig:head}c).
% It was also observed that the identity mapping contributes more than the residual blocks so that a model can work well without few blocks \cite{chang2018multilevel}.
% \begin{figure}[t]
% \centering
% \includegraphics[width=10cm]{fig/submodels.png}
% \caption{Widthwise splitting.}
% \label{fig:submodels}
% \end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.49\textwidth}
\centering
\includegraphics[width=0.9\textwidth]{fig/resnet18_NeFLADD.pdf}
\caption{The trained submodels have different step sizes.}
% NeFL-D (AD) adds trainable step-size parameters that trains how much to forward at each step (i.e., $h$ in the equation (\ref{eqn:ode})). NeFL-D (AD) computes as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_n F_n(t_t)+s_{n+1} F_{n+1}(t_{n+1})$ while a shrinked model without block $n$ is computed as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_{n+1} F_{n+1}(t_{n+1})$.
\label{fig:steps}
\end{subfigure}
% \begin{subfigure}{0.32\textwidth}
% \centering
% \includegraphics[width=0.9\textwidth]{fig/l2norm3.png}
% \caption{L2 norm of layer output of submodels trained with NeFL-AD.}
% % NeFL-D (AD) adds trainable step-size parameters that trains how much to forward at each step (i.e., $h$ in the equation (\ref{eqn:ode})). NeFL-D (AD) computes as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_n F_n(t_t)+s_{n+1} F_{n+1}(t_{n+1})$ while a shrinked model without block $n$ is computed as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_{n+1} F_{n+1}(t_{n+1})$.
% \label{fig:l2norm}
% \end{subfigure}
\begin{subfigure}{0.49\textwidth}
\centering
\includegraphics[width=0.9\textwidth]{fig/resnet18_NeFLW.pdf}
% \hspace{0.1cm}
\caption{Average L1 norm of weights of submodels trained.}
% For a model consisting of block $0$, block $1$ and block $2$, depthwise model shrinking without block $1$ is computed as follows. NeFL-D (OD): $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+2F_1(t_1)$ and NeFL-D (DD): $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+F_2(t_2)$, while the original model is $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+F_1(t_1)+F_2(t_2)$.
\label{fig:l1norm}
\end{subfigure}
% \hfill
\caption{Weights of trained five submodels by NeFL}
\label{fig:pruning}
\end{figure}
Previous studies have been conducted on reducing the model size in the width dimension \cite{li2017pruning, Liu2017learning}. In the context of NeFL, widthwise scaling is employed to slim down the network by utilizing structured contiguous pruning (ordered dropout in \cite{horvath2021fjord}) along with learnable step sizes.
We apply contiguous channel-based pruning for convolutional networks \cite{he2017channel} and node removal for fully connected layers. The parameters of narrow-width submodel constitute a subset of the parameters of the wide-width model. Consequently, parameters of the slimmest model are trained by every client while the parameters of the larger model is trained less frequently. This approach ensures that the parameters of the slimmest model capture the most useful representations.
We can obtain the further insights by toy example. Suppose a neural network that is linearly represented and data from a uniform distribution are given. Then, widthwise-scaled optimal submodel is the best $k$-rank approximation by Eckart–Young–Mirsky theorem \cite{horvath2021fjord, eckart1936approximation}. Figure \ref{fig:l1norm} presents the L1 norm of weights averaged by the number of weights at each layer of five submodels \cite{li2017pruning}. (Details would be added). We observe that L1-norm averaged by the number of weights are large enough for submodel 1 (the slimmest model) and shows the slight difference with wider submodels. The slimmest model might have learned the most useful representation and the additional parameters for larger model still gets less useful information.
% as $\mathbf{F}: \mathbb{R}^n\rightarrow\mathbb{R}^m$
% and rest of the parameters to learn less but still useful representation
% Green line in Figure \ref{fig:head}c presents the slimmed discretized function that approximates the representations. Due to the slimmed blocks, the performance to output useful representation would be worse. It is worth noting that NeFL-W also incorporates adaptive step size parameters ($s$ in Equation \ref{eqn:step_size}) to enhance its performance and flexibility.
% \subsection{NeFL-WD: General Splitting}
\subsection{Parameter Averaging} \label{sec:NeFedAvg}
\paragraph{Inconsitency}
% NeFedAvg: Nested Federated Averaging
% NeFL addresses the issue of inconsistency that arises when training multiple submodels, which can be attributed to differences in the model architecture or the heterogeneity of data of local clients \cite{acar2021federated,chen2022when,Li2020fedprox,santurkar2018how}.
% Inconsistent submodels may exhibit variants in the loss landscape \cite{chen2022when, santurkar2018how}, meaning that minimizing the empirical loss on a global model does not necessarily lead to minimizing the empirical loss on a submodel and vice versa \cite{acar2021federated, Li2020fedprox}.
% There have been studies on decoupling few layers to deal with the inconsistency between clients \cite{liang2020think, arivazhagan2019federated}.
% To overcome these challenges, NeFL separates the inconsistent parameters by submodels and designates them as separate entities (Figure \ref{fig:head}b) \cite{liang2020think, arivazhagan2019federated}. This includes separating step size parameters and normalization layers (e.g., batch normalization (BN), layer normalization(LN)) for different submodels. Note that we observed through experiments that treating LN parameters as consistent parameters yielded better performance compared to treating them as inconsistent parameters. The separated inconsistent parameters not only improved performance but also enhanced convergence.
% It is worth highlighting that BN can improve Lipschitzness of loss \cite{santurkar2018how} and the convergence of FL is sensitive to the Lipschitzness \cite{Li2020On}.
% In NeFL, the submodel parameters are denoted as $\theta_1=\{\theta_{{\textrm{c}},1},\theta_{{\textrm{ic}},1}\}, \dots ,\theta_{N_s}=\{\theta_{{\textrm{c}},{N_s}},\theta_{{\textrm{ic}},{N_s}}\}$ where $\theta_c$ denotes consistent parameters and $\theta_{{\textrm{ic}}}$ denotes inconsistent parameters. Note that the global parameters $\mathbf{w}_{\mathbf{F}_G}$, which are broadcasted to clients for the next FL iteration, encompass $\theta_{{\textrm{c}},1},\theta_{{\textrm {ic}},{1}},\dots,\theta_{{\textrm{ic}},{N_s}}$.
NeFL enables training several submodels on local data for subsequent aggregation. However, submodels of different model architectures (i.e., different width and depth) have different characteristics \cite{acar2021federated,chen2022when,Li2020fedprox,santurkar2018how}.
Referring to Figure \ref{fig:graph}, depthwise-scaled submodel omitted to compute a block can be compensated by near steps and widthwise-scaled submodel can have different step sizes to compensate for the numerical error.
We can infer that each submodel requires different step sizes to compensate for the numerical errors according to its respective model architecture.
Furthermore, submodels with different model architectures have different loss landscapes. Consider the situation where losses are differentiable. A stationary point of a global model is not necessarily the stationary points of individual submodels \cite{acar2021federated, Li2020fedprox}. The trainability of neural networks, which varies based on network architecture, may lead the averaged submodels to become non-stationary and deviate from the optima.
The motivation led us to introduce decoupled set of parameters \cite{liang2020think, arivazhagan2019federated} for respective submodels that require different parameter averaging method. We refer to these different characteristics between submodels as {\it inconsistency}.
% and we decoupled few parameters as {\it inconsistent parameters} from respective submodel.
% There have been studies that deal with heterogeneous distribution of local dataset by adding a proximal loss term [g9, g10].
To this end, we proposed the concept of separating a few parameters, referred to as {\it inconsistent parameters}, from individual submodels. We addressed step size parameters and batch normalization layers as inconsistent parameters. Note that the batch normalization can improve Lipschitzness of loss, which is sensitive to the convergence of FL \cite{santurkar2018how, Li2020On}. We further provided an ablation study to assess the effectiveness of inconsistent parameters.
Meanwhile, {\it consistent parameters} are averaged across submodels. However, the weights of submodels are subset of the largest submodel that should be addressed differently with conventional FL.
The parameters of the submodel are denoted as $\theta_1=\{\theta_{{\textrm{c}},1},\theta_{{\textrm{ic}},1}\}, \dots ,\theta_{N_s}=\{\theta_{{\textrm{c}},{N_s}},\theta_{{\textrm{ic}},{N_s}}\}$ where $\theta_c$ denotes consistent parameters and $\theta_{{\textrm{ic}}}$ denotes inconsistent parameters. Note that the global parameters, which are broadcasted to clients for the next FL iteration, encompass $\theta_{{\textrm{c}},1},\theta_{{\textrm {ic}},{1}},\dots,\theta_{{\textrm{ic}},{N_s}}$.
\begin{algorithm}[t]
{\footnotesize
\caption{\texttt{ParameterAverage}}\label{alg:NeFedAvg}
% \hspace*{\algorithmicindent}
\textbf{Input:} Trained weights from clients $\mathbf{W}=\{\mathbf{w}_i \}_{i \in \mathcal{C}_t}$
% \hspace*{\algorithmicindent} \textbf{Output}
\begin{algorithmic}[1]
% \Procedure{MyProcedure}{}
% \Procedure{MyProcedure}{$x,y$}
% % Input:
% \Comment{Input: x}
% % Output:
% \Comment{Output:y}
\For{$k$ in $\{1,\dots,N_s\}$}
\State $\mathcal{M}_k \gets \{\mathbf{w}_i | \mathbf{w}_i=\theta_k\}_{i\in\mathcal{C}_t}$
\EndFor
\For{block $\phi_j$ in $\theta_{\textrm{c},N_s}$} // \texttt{NeFedAvg} \Comment{Consistent parameters}
\State $\mathcal{M}' \gets \mathcal{M}=\{\mathcal{M}_k\}_{k=1}^{N_s}$
\For{$k$ in $\{1,\dots,N_s\}$}
\State $\mathcal{M}' \gets \mathcal{M}'\setminus \mathcal{M}_k$ if $\phi_j \notin \theta_{\textrm{c},k}$
\EndFor
\State $k' \gets 0$
\For{$k$ in $\{k|\mathcal{M}_k \in \mathcal{M}'\}$}
\State $ \phi_{j,k} \setminus \phi_{j,k'} \gets \sum_{\{i|\mathbf{w}_i \in \bigcup_{l\geq k, \mathcal{M}_l\in \mathcal{M}'}\mathcal{M}_l\} } {\phi_{j,k}^i} \setminus \phi_{j,k'}^i /\sum_{l\geq k, \mathcal{M}_l\in \mathcal{M}'}{|\mathcal{M}_l|} $
\State $k' \gets k$
\EndFor
% \State $\phi_j \gets \bigcup_{\{k|\mathcal{M}_k \in \mathcal{M}'\}} \phi_{j,k}$
\EndFor
\State $\theta_{\textrm{c},N_s} \gets \bigcup_j \phi_j$
\For{$k$ in $\{1,\dots,N_s\}$} // \texttt{FedAvg} \cite{mcmahan2017communication} \Comment{Inconsistent parameters}
\State $\theta_{\textrm{ic},k} \gets \sum_{\{i|\mathbf{w}_i \in \mathcal{M}_k\}} \theta_{\textrm{ic},k}^i /|\mathcal{M}_k|$
% \State \texttt{FedAvg(${\mathbf{W}}^{l_{\textrm{ic}}}$)} \Comment{\cite{mcmahan2017communication}}
% \For{$i \gets 1$ to $N_s$}
% \State $\mathbf{w}_{\mathbf{F}_G}^{l_{ic},i} \gets \sum_{n_i}{\mathbf{w}^{l_{ic},i}/n_i}$
% \EndFor
\EndFor
% \Procedure{NeAvg}{$\mathcal{N},\mathbf{w}^{l}$}
% \For{$i \gets 1$ to $N_s$}
% \State $\mathbf{w}^{l,i} \setminus \mathbf{w}^{l,i-1}=\sum_{n_i}{\mathbf{w}^{l,i} \setminus \mathbf{w}^{l,i-1}/n_i}$
% \Comment{$\mathbf{w}^{l,0}=\{\}$}
% \EndFor
% \EndProcedure
\end{algorithmic}}
\end{algorithm}
\paragraph{Parameter Averaging} For averaging the uploaded parameters of submodels from clients, we propose Algorithm \ref{alg:NeFedAvg}. Locally trained weights $\mathbf{W}=\{\mathbf{w}_i\}_{i\in\mathcal{C}_t}$ from clients are provided as input. The NeFL server sorts the uploaded weights from clients by submodels in $\mathcal{M}$ where $\mathcal{M} = \{ \mathcal{M}_1, \dots, \mathcal{M}_{N_s}\}$ is set of uploaded weights sorted by submodels. For consistent parameters, the parameters are averaged by {\it Nested Federated Averaging (NeFedAvg)}. In short, parameters are averaged by parameters that are updated in a nested manner at each client. For nested averaging of parameters, the server accesses parameters block by block. For each block, the server checks which submodel has the block (line 6 in Algorithm \ref{alg:NeFedAvg}). Then, parameters are averaged in a nested manner by width. The parameters of a block with the smallest width are included in the parameters of a block with larger width. Hence, the parameters are averaged by weights of clients whose submodels have the block. Meanwhile, the parameters of a block with the largest width is only contained in a submodel that has the largest width. Thus, the parameters are averaged by clients whose submodels have the block with the largest width.
Then, FedAvg \cite{mcmahan2017communication} is employed for averaging inconsistent parameters. Each submodel has the same size of inconsistent parameters that the inconsistent parameters are averaged for respective submodels.
% First of all, NeFL server sorts and counts how many times respective submodels were trained by clients.
% For each consistent parameters $\theta_c$, whether each submodel contain the layer by $\mathcal{M}$ and
% Then, it averages the weights by blockwise while inconsistent parameters are averaged by FedAvg \cite{mcmahan2017communication}.
% Inconsistent parameters are decoupled by submodels and averaged by FedAvg by submodels, respectively.
Consider an example with five submodels. Let us suppose that a convolutional layer from the first block is included in submodel 1, 3, and 5. Assume that submodel 1 and submodel 5 are trained twice (selected by two clients), while submodel 3 is trained three times at a communication round. Then, we have $|\mathcal{M}_2|=|\mathcal{M}_4|=0$, $|\mathcal{M}_1|=|\mathcal{M}_5|=2$, and $|\mathcal{M}_3|=3$. Now, delving into the parameter averaging process, the parameters exclusive to submodel 5 ($\phi_{1,5}\setminus \phi_{1,3}$) are averaged using two updated weights ($\mathcal{M}_5$). Likewise, the parameters possessed by submodel 3 but not by submodel 1 ($\phi_{1,3}\setminus\phi_{1,1}$) are averaged using five weights ($\mathcal{M}_5\cup \mathcal{M}_3$). Finally, the parameters of submodel 1 $\phi_{1,1}$, that is trained seven times, are averaged using seven weights ($\mathcal{M}_5\cup \mathcal{M}_3 \cup \mathcal{M}_1$). This approach ensures that consistent parameters are appropriately averaged, taking into account their depthwise inclusion and the widthwise number of occurrences across different submodels.
\section{Experiment} \label{sec:experiment}
In this section, we demonstrate the performance of our proposed NeFL over baselines. We provide the experiment results of NeFL that trains submodels from scratch. We also demonstrate whether NeFL aligns with the recently proposed ideas in FL \cite{chen2023on,qu2022rethinking}: experiments that verify (i) the performance of NeFL with initial weights loaded from pre-trained model and (ii) the performance of NeFL with ViTs in non-IID settings.
We conduct experiments using the CIFAR-10 dataset \cite{cifar10} for image classification with ResNets. The experiments are evaluated with 5 submodels ($N_s=5$ where $\gamma_1=0.2,\gamma_2=0.4,\gamma_3=0.6,\gamma_4=0.8,\gamma_5=1$) over a total 500 communication rounds ($T$) with 100 clients ($M$). At each round, a fraction rate of $0.1$ is used, indicating that 10 clients ($|\mathcal{C}_t|=10$) transmit their weights to the server. During the training process of clients, local batch size of $32$ and a local epoch of $E=5$ are used. For training ResNets from scratch, we employ SGD optimizer \cite{ruder2016overview} without momentum and weight decay. The initial learning rate is set to $0.1$ and decreases by a factor of $\frac{1}{10}$ at the halfway point and $\frac{3}{4}$ of the total communication rounds. Pre-trained models we use for evaluation are trained on the ImageNet-1k dataset\cite{deng2009imagenet, pretrainedModels}. To incorporate system heterogeneity, each client is assigned one of the submodels in each iteration and statistical heterogeneity was implemented by label distribution skew following Dirichlet distribution of concentration parameter $0.5$ \cite{yurochkin2019bayesian, li2021federated}.
% \subsection{Performance Evaluation}
\begin{table}[t]
\caption{Results of NeFL with five submodels for CIFAR-10 dataset under \textbf{IID} settings. We report Top-1 classification accuracies (\%) for the worst-case submodel and the average of the performance of five submodels.} \label{tab:resnet}
\centering
\resizebox*{!}{0.23\textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{6}{*}{ResNet18}
& HeteroFL \cite{diao2021heterofl} & 80.62 & 84.26 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 85.12 & 87.32 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 64.80 & 82.44 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{86.86} & \textbf{87.88} \\ \midrule
\multirow{6}{*}{ResNet34}
& HeteroFL \cite{diao2021heterofl} & 79.51 & 83.16 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 85.12 & 87.36 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 25.73 & 75.30 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{87.71} & \textbf{89.02} \\
\bottomrule
\end{tabular}
}
\resizebox*{!}{0.23 \textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirowcell{6}{Pre-trained \\ ResNet18}
& HeteroFL \cite{diao2021heterofl} & 78.26 & 84.06 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 86.37 & 88.91 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 47.76 & 82.86 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{88.61} & \textbf{89.6} \\ \midrule
\multirowcell{6}{Pre-trained \\ ResNet34}
& HeteroFL \cite{diao2021heterofl} & 79.97 & 84.34 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 87.08 & 89.37 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 52.08 & 83.63 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{88.36} & \textbf{91.14} \\
\bottomrule
\end{tabular}
}
\end{table}
% \resizebox{.497\textwidth}{!}{
% \begin{tabular}{@{}clccccc@{}}
% \toprule
% \multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
% & & Worst {\footnotesize ($\gamma=0.2$)} & Avg \\ \midrule \midrule
% \multirow{7}{*}{ResNet56}
% & LG-FedAvg \cite{liang2020think} & 76.14 {\scriptsize ($\pm 6.63$)} & 76.14 {\scriptsize ($\pm 6.63$)} \\ \cmidrule(l){2-4}
% & HeteroFL \cite{diao2021heterofl}
% & 74.56 {\scriptsize ($\pm 0.93$)} & 80.11 {\scriptsize ($\pm 9.10$)} \\ \cmidrule(l){2-4}
% & FjORD \cite{horvath2021fjord}
% & 83.51 {\scriptsize ($\pm 0.04$)} & 86.66 {\scriptsize ($\pm 2.81$)} \\ \cmidrule(l){2-4}
% & DepthFL \cite{kim2023depthfl} {\footnotesize (NeFL-DD)} & 85.73 {\scriptsize ($\pm 0.30$)} & 87.14 {\scriptsize ($\pm 0.59$)} \\ \cmidrule(l){2-4}
% & \textbf{NeFL (ours)} & \textbf{86.76} {\scriptsize ($\pm 0.00$)} & \textbf{87.41} {\scriptsize ($\pm 0.15$)} \\ \midrule
% \multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
% & & Worst {\footnotesize (${\gamma}={0.04}$)} & Avg \\ \midrule \midrule
% \multirow{4}{*}{ResNet110} & HeteroFL \cite{diao2021heterofl}
% & 61.01 {\scriptsize ($\pm 4.55$)} & 75.10 {\scriptsize ($\pm 59.76$)} \\ \cmidrule(l){2-4}
% & FjORD \cite{horvath2021fjord}
% & 74.69 {\scriptsize ($\pm 0.23$)} & 84.20 {\scriptsize ($\pm 26.41$)} \\ \cmidrule(l){2-4}
% & \textbf{NeFL (ours)}
% & \textbf{82.7} {\scriptsize ($\pm 0.02$)} & \textbf{85.61} {\scriptsize ($\pm 3.23 $)} \\
% \bottomrule
% \end{tabular}}
% \resizebox{.497\textwidth}{!}{
% \begin{tabular}{@{}clccccc@{}}
% \toprule
% \multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
% & & Worst & Avg \\ \midrule \midrule
% \multirow{7}{*}{ResNet56}
% & LG-FedAvg \cite{liang2020think} & 69.72 & 69.72 \\ \cmidrule(l){2-4}
% & HeteroFL \cite{diao2021heterofl} & 65.09 & 74.13 \\ \cmidrule(l){2-4}
% & FjORD \cite{horvath2021fjord} & 81.38 & 84.77 \\ \cmidrule(l){2-4}
% & DepthFL \cite{kim2023depthfl} & 72.94 & 82.78 \\ \cmidrule(l){2-4}
% & \textbf{NeFL (ours)} & \textbf{84.38} & \textbf{86.13} \\
% \midrule
% \multirow{7}{*}{ResNet110}
% & LG-FedAvg \cite{liang2020think} & 69.72 & 69.72 \\ \cmidrule(l){2-4}
% & HeteroFL \cite{diao2021heterofl} & 54.83 & 67.33 \\ \cmidrule(l){2-4}
% & FjORD \cite{horvath2021fjord} & 81.70 & 85.16 \\ \cmidrule(l){2-4}
% & DepthFL \cite{kim2023depthfl} & 73.56 & 86.01 \\ \cmidrule(l){2-4}
% & \textbf{NeFL (ours)} & \textbf{84.41} & \textbf{86.28} \\
% \bottomrule
% \end{tabular}}
\begin{table}[t]
\caption{Results of NeFL with five submodels for CIFAR-10 dataset under {\textbf{non-IID}} settings. We report Top-1 classification accuracies (\%) for the worst-case submodel and the average of the performance of five submodels.} \label{tab:resnet-niid}
\centering
% \resizebox{.485\textwidth}{!}{
\resizebox*{!}{0.23\textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{6}{*}{ResNet18}
& HeteroFL \cite{diao2021heterofl} & 76.25 & 80.11 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 75.81 & 77.99 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 59.61 & 76.89 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{81.26} & \textbf{81.71} \\ \midrule
\multirow{6}{*}{ResNet34}
& HeteroFL \cite{diao2021heterofl} & 76.03 & 79.63 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 74.70 & 76.01 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 30.42 & 70.76 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{80.76} & \textbf{83.31} \\
\bottomrule
\end{tabular}
}
\resizebox*{!}{0.23 \textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
% \multirow{7}{*}{Pre-trained ResNet18}
\multirowcell{6}{Pre-trained \\ ResNet18}
& HeteroFL \cite{diao2021heterofl} & 71.95 & 76.17 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 81.81 & 81.96 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 39.78 & 67.71 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{82.91} & \textbf{85.85} \\ \midrule
\multirowcell{6}{Pre-trained \\ ResNet34}
& HeteroFL \cite{diao2021heterofl} & 72.33 & 78.2 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 78.2 & 78.89 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 42.09 & 79.86 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{83.62} & \textbf{86.48} \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[t]
\caption{Results of NeFL with three submodels for CIFAR-10 dataset under IID (left) and non-IID (right) settings. A initial weights for global model was given with the pre-trained model with ImageNet-1k. We report Top-1 classification accuracies (\%) for the submodels.} \label{tab:vit-niid}
\centering
% \resizebox{.485\textwidth}{!}{
\resizebox*{!}{0.11\textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Param. \#} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
{ViT} & 86.4M & 93.02 & 95.95 \\ \midrule
\multirowcell{2}{Pre-trained \\ Wide ResNet101 \cite{sergey2016wide}}
& \multirow{2}{*}{124.8M} & \multirow{2}{*}{90.9} & \multirow{2}{*}{91.35} \\ \\ \bottomrule
\end{tabular}}
\resizebox*{!}{0.11\textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Param. \#} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
{ViT} & 86.4M & 87.56 & 93.50 \\ \midrule
\multirowcell{2}{Pre-trained \\ Wide ResNet101 \cite{sergey2016wide}}
& \multirow{2}{*}{124.8M} & \multirow{2}{*}{87.17} & \multirow{2}{*}{87.74} \\ \\ \bottomrule
\end{tabular}}
\end{table}
% We begin by evaluating the performance of various NeFL methods and comparing them to models that are split widthwise or depthwise. The global model is split into $N_s=5$ submodels, where each submodel $i$ has a splitting ratio of approximately $\gamma_{W_i}\approx \gamma_{D_i}$, except for ResNet18 which has fewer degrees of freedom in terms of depthwise splitting. When we implement depthwise scaling, we heuristically determine which blocks to skip while ensuring the first blocks not to be skipped when the channel size was changed.
As illustrated in Table \ref{tab:resnet}, we observed that NeFL outperforms baselines for both the worst-case submodel ($\gamma=0.2$) and the average performance of the five submodels. We compared with other baselines by adjusting similar number of the parameters for each submodel. The details including computing costs (i.e., FLOPs) are described in the Appendix.
% It is worth noting that our implementation of DepthFL \cite{kim2023depthfl}, which has independent bottleneck layers and classifiers for different model sizes, was done without bottleneck layers in our study, representing a special case of our approach (NeFL-DD).
% For the ResNet architecture, we employ a downsampling layer based on `option B' \cite{he2016deep} and replace the first convolutional layer with a kernel size of $3\times3$, a stride of 1, and padding of 1.
% The performance analysis reveals that NeFL shows the best performance both in the worst-case submodel and the average performance of the five submodels for ResNet18 and ResNet34.
% The submodels of ResNet 18 are split widthwise except for the worst-case submodel due to its shallow architecture.
% Meanwhile, with deeper but narrower architecture (ResNet56 and ResNet110), depthwise scaling (NeFL-AD\textsubscript{D}) shows the best performance for the smallest model size ($\gamma=0.2$).
% The results for another scaling setting of submodel sizes ($\gamma=0.04, 0.16, 0.36, 0.64, 1$) with ResNet110 present that NeFL-WD significantly increases the performance compared to widthwise-only and depthwise-only scaling. Due to its excessively reduced model ($\gamma=0.04$), a general scaling shows more performance gains, especially for the worst model. This performance gain for the worst model is particularly important, as significant performance difference between the worst and best models may discourage resource-constrained clients from participating in the NeFL pipeline.
% \subsection{NeFL with Pre-trained Models}
We investigate the performance of incorporating pre-trained models into NeFL. Recent studies on FL figured out that FL gets benefits from pre-trained models even more than centralized learning \cite{kolesnikov2020big, chen2023on}. It motivates us to evaluate the performance of NeFL on pre-trained models. The pre-trained model that is trained in a common way with ImageNet-1k is loaded from PyTorch \cite{pytorch}. Even if pre-trained models are trained regardless of NeFL, the performance of NeFL was improved with pre-trained models. Furthermore, the results in Table \ref{tab:resnet} and Table \ref{tab:resnet-niid} show that the performance of NeFL gets improved by pre-training for both IID and non-IID settings following the results of the recent studies. Meanwhile, baselines such as HeteroFL and DepthFL that do not have inconsistent parameters, when trained with pre-trained models, showed performance comparable to that of models trained from scratch.
% The recent studies have shown significant performance gains by employing pre-trained models \cite{kolesnikov2020big, chen2023on}.
% Therefore, we provide an analysis on how to incorporate pre-trained models into NeFL, noting that the models are pre-trained without any widthwise/depthwise scaling. %, which is employed in NeFL.
% As a result, it is crucial to rigorously evaluate their performance.
% Table \ref{tab:pretrained} demonstrates that pre-trained models lead to a significant increase in performance. Despite not being pre-trained specifically for NeFL, pre-trained models are still useful in NeFL framework. Upon analyzing ResNet18, we note that the worst model of NeFL-W exhibits a performance gain of 1.9\%, while the best model achieves a gain of 2.63\%. Similarly, for ResNet18, NeFL-WD shows gains of 2.53\% for the worst model and 3.03\% for the best model. Moving on to ResNet34 with NeFL-AD\textsubscript{D}, we observe gains of 2.12\% and 3.65\% for the worst and best model, respectively. The performance gain was particularly notable with NeFL-AD\textsubscript{D} compared to NeFL-WD or NeFL-W. We can attribute these interesting results to the slimming method. %Pre-trained models are not trained with contiguous pruning of widthwise splitting\footnote{Employing pre-trained model was still useful for NeFL-WD and NeFL-W.}.
% Although models are not pre-trained with contiguous pruning of widthwise splitting, employing pre-trained model was still effective in NeFL-WD and NeFL-W. On the other hand, NeFL-AD\textsubscript{D} with pre-trained models achieves even greater performance gains by skipping certain blocks to reduce the model size, which allows it to be less affected by the pre-training method. {\it Consequently, NeFL-AD\textsubscript{D} yields the best performance across all submodels. We have thus discovered that pre-trained models are useful in NeFL, offering significant performance gains, particularly when employing depthwise scaling.}
%Given these findings, we can now consider utilizing ViTs which require pre-training.
% Based on these findings, it is now worth considering the use of ViTs, which necessitate pre-training \cite{dosovitskiy2021an}.
% \subsection{ViTs \& Statistical Heterogeneity}
We now present an experiment using ViTs and Wide ResNet \cite{sergey2016wide}. The pre-trained weights trained on ImageNet-1k dataset \cite{deng2009imagenet} are loaded on the initial global models and subsequently NeFL was performed. The three submodels ($N_s=3$) of $\gamma_1=0.5,\gamma_2=0.75,\gamma_3=1$ are given and the number of clients is $M=10$, all of whom participate in the NeFL pipeline (with a fraction rate of $1$). The experiment consists of $T=100$ communication rounds, and each client performs local training for a single epoch ($E=1$). We use a cosine annealing learning rate scheduling \cite{loshchilov2017sgdr} with 500 steps of warmup and an initial learning rate 0.03. Note that utilizing layer normalization layers as consistent parameters, as opposed to BN layers that are inconsistent parameters, yields better performance.
% The images are resized to a size of $256$ and randomly cropped to a size of $224$ with a padding size of $28$.
% Meanwhile, the presence of statistical heterogeneity in FL, where non-IID data is distributed across clients, poses a challenge \cite{kairouz2021advances}. This statistical heterogeneity leads to an increased divergence in FL, resulting in significant performance degradation.
Previous studies have examined the efficacy of ViTs in FL scenarios and found that ViTs can effectively alleviate the adverse effects of statistical heterogeneity due to their inherent robustness on distribution shifts \cite{qu2022rethinking}. Building upon this line of research,
% we further evaluate the performance of NeFL under the conditions of non-IID data across clients. To model the statistical heterogeneity, we implemented class distribution skew by partitioning the data based on classes and allocating it to each client.
% \footnote{For instance, in a dataset with $10$ classes, an average data partition number of $4$ means that a client could have class $0$, class $1$, class $2$, class $3$ with same quantity (e.g., $100$ samples per each class) or class $0$ and class $2$ with twice the quantity (e.g., $200$ samples per each class).}.
Table \ref{tab:vit-niid} demonstrates that ViTs outperform ResNets in both IID and non-IID settings, despite the larger number of parameters. Particularly in non-IID settings, ViTs exhibit significantly less performance degradation for average performance compared to IID setting across all model sizes.
% When employing NeFL with ViTs, we observe degradation of 0.35\% (8 partitions), 0.79\% (4 partitions) for the global model, and 0.87\% (8 partitions), 9.97\% (4 partitions) for a model with scaling factor $\gamma=0.75$. It is worth mentioning that widthwise scaling, as seen in the findings on pre-trained models (Table \ref{tab:pretrained}), is not as effective as depthwise scaling.
\section{Conclusion}
%In this work, we have introduced NeFL, a generalized federated learning framework for tackling heterogeneous clients. Our experiments present that introducing NeFL can be effective for more clients to participate the FL pipeline without being stragglers.
In this work, we have introduced {\it Nested Federated Learning (NeFL)}, a generalized FL framework that addresses the challenges of system heterogeneity. By leveraging proposed depthwise and widthwise scaling, NeFL efficiently divides models into submodels employing the concept of ODE solver, leading to improved performance and enhanced compatibility with resource-constrained clients. We propose to decouple few parameters as inconsistent parameters for respective submodels that FedAvg are employed for averaging inconsistent parameters while NeFedAvg was utilized for averaging consistent parameters.
Our experimental results highlight the significant performance gains achieved by NeFL, particularly for the worst-case submodel.
Furthermore, we also explore NeFL in line with recent studies of FL such as pretraining and statistical heterogeneity.
% Furthermore, we observed that depthwise scaling effectively incorporates pre-trained models, and NeFL with ViTs is robust to statistical heterogeneity.
% These findings contribute to the advancement of FL techniques and demonstrate the potential of NeFL for privacy-preserving machine learning in distributed environments.
% A future direction can be
% Our findings contribute to ~, offering insights into ~
\newpage
% \begin{ack}
% Use unnumbered first level headings for the acknowledgments. All acknowledgments
% go at the end of the paper before the list of references. Moreover, you are required to declare
% funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
% More information about this disclosure can be found at: \url{https://neurips.cc/Conferences/2022/PaperInformation/FundingDisclosure}.
% Do {\bf not} include this section in the anonymized submission, only in the final paper. You can use the \texttt{ack} environment provided in the style file to autmoatically hide this section in the anonymized submission.
% \end{ack}
\medskip
\pagebreak
\bibliography{HGref}
\bibliographystyle{ieee_fullname}
% \bibliographystyle{ieeetr} % alpha
% \section*{References}
% References follow the acknowledgments. Use unnumbered first-level heading for
% the references. Any choice of citation style is acceptable as long as you are
% consistent. It is permissible to reduce the font size to \verb+small+ (9 point)
% when listing the references.
% Note that the Reference section does not count towards the page limit.
% \medskip
% {
% \small
% [1] Alexander, J.A.\ \& Mozer, M.C.\ (1995) Template-based algorithms for
% connectionist rule extraction. In G.\ Tesauro, D.S.\ Touretzky and T.K.\ Leen
% (eds.), {\it Advances in Neural Information Processing Systems 7},
% pp.\ 609--616. Cambridge, MA: MIT Press.
% [2] Bower, J.M.\ \& Beeman, D.\ (1995) {\it The Book of GENESIS: Exploring
% Realistic Neural Models with the GEneral NEural SImulation System.} New York:
% TELOS/Springer--Verlag.
% [3] Hasselmo, M.E., Schnell, E.\ \& Barkai, E.\ (1995) Dynamics of learning and
% recall at excitatory recurrent synapses and cholinergic modulation in rat
% hippocampal region CA3. {\it Journal of Neuroscience} {\bf 15}(7):5249-5262.
% }
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
{\Large \textbf{Supplementary Material}}
\section{Additional Experiments}
\subsection{Other Dataset}
We evaluate the performance for other dataset such as CIFAR-100 \cite{cifar10}, CINIC-10 \cite{darlow2018cinic10} SVHN \cite{svhn} dataset and we observe a similar trend to reported results in terms of Top-1 accuracy of the worst-case submodel and average accuracy over submodels. Note that we set total communication round $T=100$ for training SVHN. The results are presented in Table \ref{tab:other-dataset}.
\begin{table}[h]
\caption{Results of NeFL with five submodels for \textbf{CIFAR-100} (left), \textbf{CINIC10} (right) and \textbf{SVHN} (bottom) dataset under {IID} settings. We report Top-1 classification accuracies (\%) for the worst-case submodel and the average of the performance of five submodels.} \label{tab:other-dataset}
\centering
% \resizebox{.485\textwidth}{!}{
\resizebox*{!}{0.25\textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multicolumn{4}{c}{CIFAR-100} & & \\ \midrule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{6}{*}{ResNet18}
& HeteroFL \cite{diao2021heterofl} & 41.33 & 47.09 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 49.29 & 52.67 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 31.68 & 49.56 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{52.63} & \textbf{53.62} \\ \midrule
\multirow{6}{*}{ResNet34}
& HeteroFL \cite{diao2021heterofl} & 34.96 & 39.75 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 47.59 & 50.7 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 14.51 & 46.79 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{55.22} & \textbf{56.26} \\ \bottomrule
\end{tabular}}
\resizebox*{!}{0.25 \textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multicolumn{4}{c}{CINIC-10} & & \\ \midrule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
% \multirow{7}{*}{Pre-trained ResNet18}
\multirowcell{6}{ResNet18}
& HeteroFL \cite{diao2021heterofl} & 67.55 & 70.40 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 71.95 & 74.98 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 54.51 & 71.42 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{74.16} & \textbf{75.29} \\ \midrule
\multirowcell{6}{ResNet34}
& HeteroFL \cite{diao2021heterofl} & 67.39 & 69.62 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 71.58 & 74.19 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 32.05 & 67.04 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{75.02} & \textbf{76.68} \\ \bottomrule
\end{tabular}}
\resizebox*{!}{0.25 \textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multicolumn{4}{c}{SVHN} & & \\ \midrule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
% \multirow{7}{*}{Pre-trained ResNet18}
\multirowcell{6}{ResNet18}
& HeteroFL \cite{diao2021heterofl} & 91.82 & 93.46 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 94.31 & 93.97 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 91.54 & 93.97 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{94.45} & \textbf{94.94} \\ \midrule
\multirowcell{6}{ResNet34}
& HeteroFL \cite{diao2021heterofl} & 89.86 & 92.39 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 93.83 & 94.63 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 74.33 & 89.96 \\ \cmidrule(l){2-4}
& \textbf{NeFL (ours)} & \textbf{94.72} & \textbf{95.22} \\ \bottomrule
\end{tabular}}
\end{table}
\subsection{Different Number of Clients}
We conducted further experiments on various numbers of clients while keeping the total amount of dataset. In Table \ref{tab:other-client-num}, We could observe that as the number of clients increases, the performance of NeFL as well as baselines degrades. The results align with previous studies \cite{kim2023depthfl,thapa2022splitfed, Wang2020Federated}. The more the number of clients, training data deviates further from being IID. While the IID sampling of the training data ensures the stochastic gradient to be an unbiased estimate of the full gradient, the non-IID sampling leads to non-guaranteed convergence and model weight divergence in FL \cite{Li2020fedprox, li2021model, zhao2018federated}.
In this regard, our proposed algorithm remains effective across different numbers of clients; however, the performance (e.g., accuracy and convergence) degrades due to changes in data distribution among clients as their number increases.
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[h]
\caption{Results of NeFL of five submodels with a global model ResNet18 for CIFAR-10 dataset under IID settings across different number of clients. We report Top-1 classification accuracies (\%) for the worst-case submodel and the average of the performance of five submodels..} \label{tab:other-client-num}
\centering
\begin{tabular}{llcccc}
\toprule
\multicolumn{1}{r}{\multirow{2}{*}{\# of Clients}} & \multirow{2}{*}{Model size} & \multicolumn{4}{c}{Method} \\ \cmidrule{3-6}
\multicolumn{1}{r}{} & & \multicolumn{1}{c}{\textbf{NeFL (ours)}} & \multicolumn{1}{c}{FjORD \cite{horvath2021fjord}} & \multicolumn{1}{c}{HeteroFL \cite{diao2021heterofl}} & DepthFL \cite{kim2023depthfl} \\ \midrule\midrule
\multirow{2}{*}{100} & Worst & \multicolumn{1}{c}{\textbf{86.86}} & \multicolumn{1}{c}{85.12} & \multicolumn{1}{c}{80.62} & 64.8 \\ \cmidrule{2-6}
& Avg & \multicolumn{1}{c}{\textbf{87.88}} & \multicolumn{1}{c}{87.32} & \multicolumn{1}{c}{84.62} & 82.44 \\ \midrule
\multirow{2}{*}{50} & Worst & \multicolumn{1}{c}{\textbf{88.42}} & \multicolumn{1}{c}{86.19} & \multicolumn{1}{c}{84.67} & 52.07 \\ \cmidrule{2-6}
& Avg & \multicolumn{1}{c}{\textbf{89.14}} & \multicolumn{1}{c}{88.43} & \multicolumn{1}{c}{87.23} & 82.04 \\ \midrule
\multirow{2}{*}{20} & Worst & \multicolumn{1}{c}{\textbf{89.2}} & \multicolumn{1}{c}{87.76} & \multicolumn{1}{c}{88.74} & 24.94 \\ \cmidrule{2-6}
& Avg & \multicolumn{1}{c}{\textbf{89.88}} & \multicolumn{1}{c}{89.6} & \multicolumn{1}{c}{88.71} & 76.54 \\ \bottomrule
\end{tabular}
\end{table}
\subsection{Ablation Study}
\begin{table}[!t]
\caption{Ablation study by NeFL with five submodels for CIFAR-10 dataset under IID settings. We report Top-1 classification accuracies (\%) for the worst-case submodel and the average of the performance of five submodels.} \label{tab:ablation}
\centering
\resizebox*{!}{0.27\textheight}{
% \resizebox{.45\textwidth}{!}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{14}{*}{ResNet18}
& HeteroFL \cite{diao2021heterofl} & 80.62 & 84.26 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 85.12 & 87.32 \\ \cmidrule(l){2-4}
& \textbf{NeFL-W} & \textbf{85.13} & \textbf{87.36} \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 64.80 & 82.44 \\ \cmidrule(l){2-4}
& {NeFL-D (N/L)} & {86.29} & {88.12} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O} (N/L)} & {86.24} & {88.22} \\ \cmidrule(l){2-4}
& \textbf{NeFL-D} & \textbf{86.06} & \textbf{87.94} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O}} & {85.98} & {88.20} \\ \cmidrule(l){2-4}
& \textbf{NeFL-WD} & \textbf{86.86} & \textbf{87.88} \\ \cmidrule(l){2-4}
& {NeFL-WD (N/L)} & {86.85} & {88.21} \\
\bottomrule
\end{tabular}
}
\resizebox*{!}{0.27\textheight}{
% \resizebox{.45\textwidth}{!}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{14}{*}{ResNet34}
& HeteroFL \cite{diao2021heterofl} & 79.51 & 83.16 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 85.12 & 87.36 \\ \cmidrule(l){2-4}
& \textbf{NeFL-W} & \textbf{85.65} & \textbf{87.97} \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 25.73 & 75.30 \\ \cmidrule(l){2-4}
& {NeFL-D (N/L)} & {87.40} & {89.12} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O} (N/L)} & {86.47} & {88.49} \\ \cmidrule(l){2-4}
& \textbf{NeFL-D} & \textbf{87.71} & \textbf{89.02} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O}} & {87.06} & {88.71} \\ \cmidrule(l){2-4}
& \textbf{NeFL-WD} & \textbf{86.73} & \textbf{88.42} \\ \cmidrule(l){2-4}
& {NeFL-WD (N/L)} & {86.2} & {88.16} \\
\bottomrule
\end{tabular}
}
\resizebox*{!}{0.27\textheight}{
% \resizebox{.45\textwidth}{!}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirowcell{14}{Pre-trained\\ResNet18}
& HeteroFL \cite{diao2021heterofl} & 78.26 & 84.06 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 86.37 & 88.91 \\ \cmidrule(l){2-4}
& \textbf{NeFL-W} & \textbf{86.1} & \textbf{89.13} \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 47.76 & 82.85 \\ \cmidrule(l){2-4}
& {NeFL-D (N/L)} & {86.95} & {89.77} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O} (N/L)} & {86.24} & {89.76} \\ \cmidrule(l){2-4}
& \textbf{NeFL-D} & \textbf{87.13} & \textbf{90.00} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O}} & {87.02} & {89.72} \\ \cmidrule(l){2-4}
& \textbf{NeFL-WD} & \textbf{88.61} & \textbf{89.60} \\ \cmidrule(l){2-4}
& {NeFL-WD (N/L)} & {88.57} & {89.70} \\
\bottomrule
\end{tabular}
}
\resizebox*{!}{0.27\textheight}{
% \resizebox{.45\textwidth}{!}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirowcell{14}{Pre-trained\\ResNet34}
& HeteroFL \cite{diao2021heterofl} & 79.97 & 84.34 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 87.08 & 89.37 \\ \cmidrule(l){2-4}
& \textbf{NeFL-W} & \textbf{87.41} & \textbf{89.75} \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 52.08 & 83.63 \\ \cmidrule(l){2-4}
& {NeFL-D (N/L)} & {87.95} & {90.79} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O} (N/L)} & {87.44} & {90.58} \\ \cmidrule(l){2-4}
& \textbf{NeFL-D} & \textbf{88.36} & \textbf{91.14} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O}} & {87.86} & {90.90} \\ \cmidrule(l){2-4}
& \textbf{NeFL-WD} & \textbf{87.69} & \textbf{90.18} \\ \cmidrule(l){2-4}
& {NeFL-WD (N/L)} & {87.37} & {89.78} \\
\bottomrule
\end{tabular}
}
\resizebox*{!}{0.27\textheight}{
% \resizebox{.45\textwidth}{!}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{14}{*}{ResNet56}
& HeteroFL \cite{diao2021heterofl} & 65.09 & 74.13 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 81.38 & 84.77 \\ \cmidrule(l){2-4}
& \textbf{NeFL-W} & \textbf{82.05} & \textbf{85.48} \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 72.94 & 86.19 \\ \cmidrule(l){2-4}
& {NeFL-D (N/L)} & {84.38} & {86.13} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O} (N/L)} & {83.08} & {85.59} \\ \cmidrule(l){2-4}
& \textbf{NeFL-D} & \textbf{84.38} & \textbf{86.13} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O}} & {81.97} & {85.37} \\ \cmidrule(l){2-4}
& \textbf{NeFL-WD} & \textbf{83.92} & \textbf{86.00} \\ \cmidrule(l){2-4}
& {NeFL-WD (N/L)} & {83.68} & {85.85} \\
\bottomrule
\end{tabular}
}
\resizebox*{!}{0.27\textheight}{
% \resizebox{.45\textwidth}{!}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{14}{*}{ResNet110}
& HeteroFL \cite{diao2021heterofl} & 54.83 & 67.33 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 81.70 & 85.16 \\ \cmidrule(l){2-4}
& \textbf{NeFL-W} & \textbf{81.67} & \textbf{85.32} \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 73.56 & 82.42 \\ \cmidrule(l){2-4}
& {NeFL-D (N/L)} & {85.23} & {86.34} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O} (N/L)} & {84} & {85.97} \\ \cmidrule(l){2-4}
& \textbf{NeFL-D} & \textbf{85.96} & \textbf{86.66} \\ \cmidrule(l){2-4}
& {NeFL-D\textsubscript{O}} & {82.74} & {85.66} \\ \cmidrule(l){2-4}
& \textbf{NeFL-WD} & \textbf{84.41} & \textbf{86.28} \\ \cmidrule(l){2-4}
& {NeFL-WD (N/L)} & {83.58} & {85.73} \\
\bottomrule
\end{tabular}
}
\end{table}
For the ease of understanding for ablation study we refer {\it NeFL-W} that all submodels are scaled widthwise, {\it NeFL-D} that all submodels are scaled depthwise and {\it NeFL-WD} that submodels are scaled both widthwise and depthwise.
We further refer to NeFL-D\textsubscript{O} that has different initial step sizes with NeFL-D.
Referring to Table \ref{tab:resnet56detail}, NeFL-D\textsubscript{O} has larger magnitude step sizes compared to NeFL-D\textsubscript{D}, aligning with the principles of ODE solver.
NeFL-D\textsubscript{D} scales submodels by skipping a subset of blocks of a global model, thus reducing the depth of the model. NeFL-D\textsubscript{D} does not compensate for the skipped blocks by using larger step sizes. Given the initial step sizes as $s_0=1, s_1=1, s_2=0$, output after Block $2$ without Block $2$ is $\mathbf{Y}_3=\mathbf{Y}_0+F_0+F_1$.
NeFL-D\textsubscript{O} reduces the size of the global model by skipping a subset of block functions $F(\cdot)$ and gives larger initial step sizes to compensate it. The step sizes are determined based on the number of blocks that are skipped. A model without Block $2$, output after Block $2$ is initially computed as $\mathbf{Y}_3=\mathbf{Y}_0+F_0+2F_1$.
The performance comparison between NeFL-W and FjORD \cite{horvath2021fjord} provides the effectiveness of learnable step sizes and comparison between NeFL-W and HeteroFL \cite{diao2021heterofl} provides the effectiveness of inconsistent parameters including learnable step sizes.
Similarly, the comparison between NeFL-D and NeFL-D with no learnable step sizes (N/L; constant step sizes with given intial values) provides the effectiveness of learnables step sizes and comparison between NeFL-D and DepthFL \cite{kim2023depthfl} (NeFL-WD and NeFL-WD (N/L)) provides the effectiveness of inconsistent parameters.
We summarized the NeFL with various scaled submodels in Table \ref{tab:NeFLsummary}. We also provide the parameter sizes and average FLOPs of submodels by scaling in Table \ref{tab:flop}.
Insights can be derived from the ablation studies (Table \ref{tab:ablation}) conducted on the NeFL framework.
We observe that NeFL-D outperforms NeFL-D\textsubscript{O} in most cases. The rationale comes from the trained step sizes are not as large as initial value for NeFL-D\textsubscript{O} that large values for NeFL-D\textsubscript{O} degrades the trainability of depthwise-scaled submodels. The performance improvement of NeFL-WD over NeFL-WD, NeFL-D over NeFL-D (N/L) and NeFL-W over FjORD \cite{horvath2021fjord} provides the effectiveness of learnable step sizes. The effectiveness of the proposed parameter averaging for consistent parameters and inconsistent parameters is also verified by NeFL-D over DepthFL \cite{kim2023depthfl} and NeFL-W over HeteroFL \cite{diao2021heterofl}.
Furthermore, We observe that NeFL-D and NeFL-WD have better performance over widthwise scaling. The performance gap of depthwise scaling over widthwise scaling gets larger for narrow and deeper networks.
We evaluated the experiments by similar number of parameters for several scaling methods and depthwise scaling requires slightly more FLOPs than widthwise scaling for ResNet18, ResNet34 and ResNet110 and less FLOPs for ResNet56.
Note that beyond the performance improvement, NeFL provides the more degree-of-freedom for widthwise/depthwise scaling that can be determined by the requirements of clients while NeFL-W also outperforms baselines in Table \ref{tab:ablation}. Also refer to Table \ref{tab:110-04} that has different scaling ratio. Note that in this case, FjORD \cite{horvath2021fjord} outperforms NeFL-W. In this case with severe scaling factors (the worst model has 4\% parameters of a global model), step sizes could not compensate the limited number of parameters and degraded the trainability with auxiliary parameters.
% \begin{wraptable}{r}{7cm}
\begin{table}[]
\centering
{ %\scriptsize
\caption{Summarization of NeFL and baselines for ablation study}
\label{tab:NeFLsummary}
\begin{tabular}{lccc}
\toprule
& Depthwise scaling & Widthwise scaling & Adaptive step sizes \\ \midrule \midrule
% NeFL-OD & \checkmark & & \\ \hline
% NeFL-DD & \checkmark & & \\ \hline
DepthFL & \checkmark & & \\ \midrule
FjORD, HeteroFL & & \checkmark & \\ \midrule
NeFL-D & \checkmark & & \checkmark \\ \midrule
NeFL-W & & \checkmark & \checkmark \\ \midrule
NeFL-WD & \checkmark & \checkmark & \checkmark \\ \bottomrule
\end{tabular}}
\end{table}
% \end{wraptable}
\begin{table}[]
\centering
\caption{Details of average FLOPs of submodels of $\bm{\gamma}=[0.2,0.4,0.6,0.8,1]$}
\label{tab:flop}
\begin{tabular}{llccc}
\toprule
\multirow{2}{*}{Model} & \multirow{2}{*}{Metric} & \multicolumn{3}{c}{Method} \\ \cmidrule{3-5}
& & \multicolumn{1}{l}{Width/Depthwise scaling} & \multicolumn{1}{l}{Widthwise scaling} & \multicolumn{1}{l}{Depthwise scaling} \\ \midrule\midrule
\multirow{3}{*}{ResNet18} & Param \# & \multicolumn{1}{c}{6.71M} & \multicolumn{1}{c}{6.71M} & 6.68M \\ \cmidrule{2-5}
& FLOPs & \multicolumn{1}{c}{87.8M} & \multicolumn{1}{c}{85M} & 102M \\ \midrule
\multirow{3}{*}{ResNet34} & Param \# & \multicolumn{1}{c}{12.6M} & \multicolumn{1}{c}{12.8M} & 12.9M \\ \cmidrule{2-5}
& FLOPs & \multicolumn{1}{c}{181M} & \multicolumn{1}{c}{176M} & 193M \\ \midrule
\multirow{3}{*}{ResNet56} & Param \# & \multicolumn{1}{c}{0.51M} & \multicolumn{1}{c}{0.52M} & 0.51M \\ \cmidrule{2-5}
& FLOPs & \multicolumn{1}{c}{530M} & \multicolumn{1}{c}{534M} & 526M \\ \midrule
\multirow{3}{*}{ResNet110} & Param \# & \multicolumn{1}{c}{1.05M} & \multicolumn{1}{c}{1.06M} & 1.04M \\ \cmidrule{2-5}
& FLOPs & \multicolumn{1}{c}{158M} & \multicolumn{1}{c}{159M} & 234M \\ \bottomrule
\end{tabular}
\end{table}
% Insights can be derived from the ablation studies (Table \ref{tab:ablation1} and Table \ref{tab:ablation2}) conducted on the NeFL framework. While interpreting nerual networks as solving ODEs provides valuable insights for NeFL, it is observed that NeFL-DD outperforms NeFL-OD, and NeFL-AD\textsubscript{D} outperforms NeFL-AD\textsubscript{O}. The rationale behind these results can be understood by examining the learned step sizes. It is found that the learned step sizes generally fall within the range of $[0, 2]$, but NeFL-AD\textsubscript{O} and NeFL-OD initialize the step sizes too high. NeFL-AD\textsubscript{D} performs well not only for pre-trained models but also for models trained from scratch, demonstrating its effectiveness. However, for ResNet18 with shallow depth or with different scaling factor (Table \ref{tab:ablation2}), NeFL-WD proves to be more effective. It should be noted that depthwise scaling has limited flexibility when it comes to splitting models for shallow architectures. Therefore, determining the most suitable NeFL method requires further investigation and study. Overall, the ablation studies shed light on the performance and characteristics of different NeFL methods, highlighting the importance of carefully selecting the appropriate method based on the specific model architecture and requirements.
% \begin{table}[t]
% \caption{Genral splitting}
% \label{tab:noniid2}
% \centering
% \resizebox{.8\textwidth}{!}{
% \begin{tabular}{clccc}
% \toprule
% \multirow{2}{*}{Avg. data partition \#} & \multirow{2}{*}{Model} & \multicolumn{3}{c}{Model size} \\ \cmidrule(l){3-5}
% & & 0.5 & 0.75 & 1 \\ \midrule \midrule
% \multirow{2}{*}{10} & ViT-B/16 (86.4M) & 93.02 (-) & 97.07 (-) & 97.77 (-)\\ \cmidrule(l){2-5}
% & \cellcolor[HTML]{EFEFEF} Wide ResNet101\_2 (124.8M) & \cellcolor[HTML]{EFEFEF} 87.83 (-) & \cellcolor[HTML]{EFEFEF} 88.52 (-) & \cellcolor[HTML]{EFEFEF} 89.54 (-) \\ \midrule
% \multirow{2}{*}{8} & ViT-B/16 (86.4M) & 82.76 (-10.26) & 96.2 (-0.87) & 97.42 (-0.35) \\ \cmidrule(l){2-5}
% & \cellcolor[HTML]{EFEFEF} Wide ResNet101\_2 (124.8M) & \cellcolor[HTML]{EFEFEF} 77.88 (-9.95) & \cellcolor[HTML]{EFEFEF} 85.43 (-3.09) & \cellcolor[HTML]{EFEFEF} 86.33 (-3.21) \\ \midrule
% \multirow{2}{*}{4} & ViT-B/16 (86.4M) & 71.48 (-11.28) & 87.1 (-9.1) & 96.98 (-0.44) \\ \cmidrule(l){2-5}
% & \cellcolor[HTML]{EFEFEF} Wide ResNet101\_2 (124.8M) & \cellcolor[HTML]{EFEFEF} 64.82 (-13.06) & \cellcolor[HTML]{EFEFEF} 66.66 (-18.77) & \cellcolor[HTML]{EFEFEF} 69.60 (-16.73) \\ \midrule
% \multirow{2}{*}{2} & ViT-B/16 (86.4M) & 45.81 (-25.67) & 47.92 (-39.18) & 58.48 (-38.5) \\ \cmidrule(l){2-5}
% & \cellcolor[HTML]{EFEFEF} Wide ResNet101\_2 (124.8M) & \cellcolor[HTML]{EFEFEF} 32.81 (-32.01) & \cellcolor[HTML]{EFEFEF} 25.98 (-40.68) & \cellcolor[HTML]{EFEFEF} 35.24 (-34.36) \\
% \bottomrule
% \end{tabular}}
% \end{table}
\begin{table}[t] % Different Scaling Ratio $\gamma$
\caption{Results of NeFL with five submodels ($\bm{\gamma}=[0.04,0.16,0.36,0.64,1]$) for CIFAR-10 dataset on ResNet110. Results of NeFL with five submodels for CIFAR-10 dataset under IID settings. We report Top-1 classification accuracies (\%) for the worst-case submodel and the average of the performance of five submodels.} \label{tab:110-04}
\centering
% \resizebox*{!}{0.23\textheight}{
\begin{tabular}{@{}clccccc@{}}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Method} & \multicolumn{2}{c}{Model size} \\ \cmidrule(l){3-4}
& & Worst & Avg \\ \midrule \midrule
\multirow{6}{*}{ResNet110}
& HeteroFL \cite{diao2021heterofl} & 46.58 & 63.62 \\ \cmidrule(l){2-4}
& FjORD \cite{horvath2021fjord} & 69.61 & 81.46 \\ \cmidrule(l){2-4}
& \textbf{NeFL-W} & 68.27 & 80.98 \\ \cmidrule(l){2-4}
& DepthFL \cite{kim2023depthfl} & 11.00 & 53.91 \\ \cmidrule(l){2-4}
& \textbf{NeFL-D} & \textbf{75.4} & \textbf{84.31} \\ \cmidrule(l){2-4}
& \textbf{NeFL-WD} & \textbf{76.60} & \textbf{84.02} \\ \bottomrule
\end{tabular}
% }
\end{table}
% \begin{table}[h]
% \caption{Results of NeFL with five submodels ($\bm{\gamma}=[0.04,0.16,0.36,0.64,1]$) for CIFAR-10 dataset on ResNet110.}
% \label{tab:ablation2}
% \centering
% \resizebox{.97\textwidth}{!}{
% \begin{tabular}{@{}clcccccc@{}}
% \toprule
% \multirow{2}{*}{Model} & \multirow{2}{*}{Algorithm} & \multicolumn{6}{c}{Model size} \\ \cmidrule(l){3-8}
% & & 0.04 & 0.16 & 0.36 & 0.64 & 1 &Avg. \\ \midrule \midrule
% \multirow{7}{*}{ResNet110} & HeteroFL \cite{diao2021heterofl}
% & 61.01 {\scriptsize ($\pm 4.55$)} & 74.12 {\scriptsize ($\pm 0.29$)} & 80.13 {\scriptsize ($\pm 0.17$)} & 80.13 {\scriptsize ($\pm 0.17$)} & 80.13 {\scriptsize ($\pm 0.17$)} & 75.10 {\scriptsize ($\pm 59.76$)}\\ \cmidrule(l){2-8}
% & FjORD \cite{horvath2021fjord}
% & 74.69 {\scriptsize ($\pm 0.23$)} & 83.90 {\scriptsize ($\pm 0.01$)} & 87.09 {\scriptsize ($\pm 0.12$)} & 87.69 {\scriptsize ($\pm 0.08$)} & 87.64 {\scriptsize ($\pm 0.08$)} & 84.20 {\scriptsize ($\pm 26.41$)} \\ \cmidrule(l){2-8}
% & \textbf{NeFL-WD} & 82.7 {\scriptsize ($\pm 0.02$)} & 86.21 {\scriptsize ($\pm 0.21$)} & 87.04 {\scriptsize ($\pm 0.16$)} & 86.84 {\scriptsize ($\pm 0.09$)} & 85.29 {\scriptsize ($\pm 3.38$)} & 85.61 {\scriptsize ($\pm 3.23$)} \\ \cmidrule(l){2-8}
% & NeFL-W & 75.18 {\scriptsize ($\pm 0.32$)} & 84.34 {\scriptsize ($\pm 0.16$)} & 87.25 {\scriptsize ($\pm 0.09$)} & 87.71 {\scriptsize ($\pm 0.11$)} & 87.67 {\scriptsize ($\pm 0.21$)} & 84.43 {\scriptsize ($\pm 24.71$)}\\ \cmidrule(l){2-8}
% & NeFL-AD\textsubscript{D} & 80.9 {\scriptsize ($\pm 0.08$)}& 87.49 {\scriptsize ($\pm 0.09$)}& 87.56 {\scriptsize ($\pm 0.00$)}& 86.94 {\scriptsize ($\pm 0.18$)}& 86.57 {\scriptsize ($\pm 0.61$)} & 85.89 {\scriptsize ($\pm 6.96$)}\\
% \bottomrule
% \end{tabular}}
% \end{table}
% \begin{figure}[!t]
% \centering
% \begin{subfigure}{0.49\textwidth}
% \centering
% \includegraphics[width=0.95\textwidth]{fig/ODE.png}
% % \hspace{0.1cm}
% \caption{A model depthwise shrinks by skipping a subset of blocks.}
% % For a model consisting of block $0$, block $1$ and block $2$, depthwise model shrinking without block $1$ is computed as follows. NeFL-D (OD): $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+2F_1(t_1)$ and NeFL-D (DD): $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+F_2(t_2)$, while the original model is $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+F_1(t_1)+F_2(t_2)$.
% \label{fig:skipBlock}
% \end{subfigure}
% % \hfill
% \begin{subfigure}{0.49\textwidth}
% \centering
% \includegraphics[width=0.95\textwidth]{fig/AD.png}
% \caption{Trainable step size parameters are multiplied to output of blocks.}
% % NeFL-D (AD) adds trainable step-size parameters that trains how much to forward at each step (i.e., $h$ in the equation (\ref{eqn:ode})). NeFL-D (AD) computes as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_n F_n(t_t)+s_{n+1} F_{n+1}(t_{n+1})$ while a shrinked model without block $n$ is computed as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_{n+1} F_{n+1}(t_{n+1})$.
% \label{fig:learnableSteps}
% \end{subfigure}
% \caption{Neural networks with skip connections as ODE solver.}
% \label{fig:DepthBlocks}
% \end{figure}
% \begin{figure}[!t]
% \centering
% \begin{subfigure}{0.49\textwidth}
% \centering
% \includegraphics[width=0.99\textwidth]{fig/r18step.png}
% % \hspace{0.1cm}
% \caption{Trained step sizes on ResNet18.}
% % For a model consisting of block $0$, block $1$ and block $2$, depthwise model shrinking without block $1$ is computed as follows. NeFL-D (OD): $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+2F_1(t_1)$ and NeFL-D (DD): $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+F_2(t_2)$, while the original model is $\mathbf{Y}_3=\mathbf{Y}_0+F_0(t_0)+F_1(t_1)+F_2(t_2)$.
% % \label{fig:skipBlock}
% \end{subfigure}
% % \hfill
% \begin{subfigure}{0.49\textwidth}
% \centering
% \includegraphics[width=0.99\textwidth]{fig/pr18step.png}
% \caption{Trained Step sizes on pre-trained ResNet18.}
% % NeFL-D (AD) adds trainable step-size parameters that trains how much to forward at each step (i.e., $h$ in the equation (\ref{eqn:ode})). NeFL-D (AD) computes as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_n F_n(t_t)+s_{n+1} F_{n+1}(t_{n+1})$ while a shrinked model without block $n$ is computed as $\mathbf{Y}_{n+2}=\mathbf{Y}_n+s_{n+1} F_{n+1}(t_{n+1})$.
% % \label{fig:learnableSteps}
% \end{subfigure}
% \caption{Step sizes of trained submodels by NeFL-WD.}
% \label{fig:stepSizes}
% \end{figure}
% \section{Limitations}
% \begin{table}[t]
% \caption{Results of NeFL with three submodels for CIFAR-10 dataset with the statistical heterogeneity under the pretrained global models with ImageNet-1k. We report Top-1 classification accuracies (\%) for the submodels.}
% \label{tab:noniid3}
% \centering
% \resizebox{.98\textwidth}{!}{
% \begin{tabular}{clcccccc}
% \toprule % \multirow{2}{*}{Avg. data \newline partition \#}
% \multirowcell{3}{Avg. \\data partition \# \\(Degree of non-IID)}
% & \multirow{3}{*}{Method}
% & \multicolumn{3}{c}{Model size}
% & \multicolumn{3}{c}{Model size}\\
% & & &{\small (ViT-B/16 (86.4M))} & & & {\small (Wide ResNet101\_2 (124.8M))} \\ \cmidrule(l){3-8}
% & & 0.5 & 0.75 & 1 & 0.5 & 0.75 & 1 \\ \midrule \midrule
% \multirow{2}{*}{10} & Widthwise scaling \cite{horvath2021fjord, diao2021heterofl} & 70.32 & 72.94 & 97.43\\ \cmidrule(l){2-8}
% & \cellcolor[HTML]{EFEFEF} \textbf{NeFL (ours)} & \cellcolor[HTML]{EFEFEF} 93.02 & \cellcolor[HTML]{EFEFEF} 97.07 & \cellcolor[HTML]{EFEFEF} 97.77 & \cellcolor[HTML]{EFEFEF} 87.83 & \cellcolor[HTML]{EFEFEF} 88.52 & \cellcolor[HTML]{EFEFEF} 89.54
% \\ \midrule
% \multirow{2}{*}{8} & Widthwise scaling \cite{horvath2021fjord, diao2021heterofl} & 60.94 & 65.00 & 96.66 \\ \cmidrule(l){2-8}
% & \cellcolor[HTML]{EFEFEF} \textbf{NeFL (ours)}
% & \cellcolor[HTML]{EFEFEF} 82.76 & \cellcolor[HTML]{EFEFEF} 96.2 & \cellcolor[HTML]{EFEFEF} 97.42
% & \cellcolor[HTML]{EFEFEF} 77.88 & \cellcolor[HTML]{EFEFEF} 85.43 & \cellcolor[HTML]{EFEFEF} 86.33
% \\ \midrule
% \multirow{2}{*}{4} & Widthwise scaling \cite{horvath2021fjord, diao2021heterofl} & 46.99 & 52.26 & 87.01 \\ \cmidrule(l){2-8}
% & \cellcolor[HTML]{EFEFEF} \textbf{NeFL (ours)} &
% \cellcolor[HTML]{EFEFEF} 71.48 & \cellcolor[HTML]{EFEFEF} 87.1 & \cellcolor[HTML]{EFEFEF} 96.98 &
% \cellcolor[HTML]{EFEFEF} 64.82 & \cellcolor[HTML]{EFEFEF} 66.66 & \cellcolor[HTML]{EFEFEF} 69.60\\ \midrule
% \multirow{2}{*}{2} & FjORD \cite{horvath2021fjord} & 33.66 & 34.31 & 58.28 \\ \cmidrule(l){2-8}
% & \cellcolor[HTML]{EFEFEF} \textbf{NeFL (ours)} & \cellcolor[HTML]{EFEFEF} 45.81 & \cellcolor[HTML]{EFEFEF} 47.92 & \cellcolor[HTML]{EFEFEF} 58.48
% & \cellcolor[HTML]{EFEFEF} 32.81 & \cellcolor[HTML]{EFEFEF} 25.98 & \cellcolor[HTML]{EFEFEF} 35.24
% \\
% \bottomrule
% \end{tabular}}
% \end{table}
\clearpage
\section{Experimental Details}
\subsection{Model Architectures}
For the experiments presented in Table \ref{tab:resnet}, Table \ref{tab:resnet-niid}, Table \ref{tab:other-dataset}, Table \ref{tab:other-client-num} and Table \ref{tab:ablation}, we consider five submodels with $\bm{\gamma} =\left[\gamma_1, \gamma_2, \gamma_3, \gamma_4, \gamma_5\right] =\left[0.2,0.4,0.6,0.8,1\right]$ and $\bm{\gamma} =\left[0.04,0.16,0.36,0.64,1\right]$ for Table \ref{tab:110-04}. Additionally, for Table \ref{tab:vit-niid}, we use three submodels with $\bm{\gamma} =\left[\gamma_1, \gamma_2, \gamma_3\right] =\left[0.5,0.75,1\right]$.
The ResNet18 architecture (detailed in Table \ref{tab:resnet18detail}) and ResNet34 architecture (detailed in Table \ref{tab:resnet34detail}) consist of four layers, while ResNet56 (detailed in Table \ref{tab:resnet56detail}) and ResNet110 (detailed in Table \ref{tab:resnet110detail}) have three layers. These layers are composed of blocks with different channel sizes, specifically (64, 128, 256, 512) for ResNet18/32 and (16, 32, 64) for ResNet56/110. Wide ResNet101\_2 (detailed in Table \ref{tab:wresnet101detail}) comprises four layers of bottleneck blocks with channel sizes (128, 256, 512, 1024) \cite{he2016deep, sergey2016wide}.
The ViT-B/16 architecture consists of twelve layers, with each layer containing blocks comprising self-attention (SA) and feed-forward networks (FFN) \cite{dosovitskiy2021an}. The widthwise splitting for ViT models are implemented by varying the embedding dimension ($D$ in \cite{dosovitskiy2021an}).
In the tables, 1's and 0's denote the step sizes. A step size of zero indicates that a submodel does not include the corresponding block. Unlike ResNets, which have a step size for each block, ViTs have different learnable step size parameters to be multiplied with SA and FFN.
NeFL-W is characterized by $\bm{\gamma}_D=[1,\dots,1]$ and $\bm{\gamma}_W$ with a target size, NeFL-D is characterized by $\bm{\gamma}_W=[1,\dots,1]$ and $\bm{\gamma}_D$ with a target size and NeFL-WD is characterized by target size $\bm{\gamma}_W \bm{\gamma}_D$.
NeFL on Table \ref{tab:resnet} and Table \ref{tab:resnet-niid} follows NeFL-WD (for ResNet18 and ResNet110) and NeFL-D\textsubscript{D} (for ResNet 34 and ResNet 56). Corresponding number of parameters and FLOPs is provided in Table \ref{tab:flop}.
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\subsection{Dataset}
\paragraph{CIFAR10/100} The CIFAR10 dataset consists of 60000 images (train dataset consists of 50000 samples and test dataset consists of 10000 samples). $32\times32\times3$ color images are categorized by 10 classes, with 6000 images per class \cite{cifar10}. For FL with each client has 500 data samples for $M=100$, and 5000 data samples for $M=10$. We perform data augmentation and pre-processing of random cropping (by $32\times32$ with padding of 4), random horizontal flip, and normalization
by mean of $(0.4914, 0.4822, 0.4465)$ and standard deviation of $(0.2023, 0.1994, 0.2010)$.
\paragraph{CINIC10} The CIFAR10 dataset consists of 270000 $32\times32\times3$ color images (train dataset consists of 90000 samples and validation and test dataset consists of 90000 samples respectively) in 10 classes \cite{darlow2018cinic10}. It is constructed from ImageNet and CIFAR10. We perform data augmentation and pre-processing of random cropping (by $32\times32$ with padding of 4), random horizontal flip, and normalization
by mean of $(0.47889522, 0.47227842, 0.43047404)$ and standard deviation of $(0.24205776, 0.23828046, 0.25874835)$.
\paragraph{SVHN} The SVHN dataset consists of 73257 digits for training and 26032 digits for testing of $32\times32\times3$ color images \cite{svhn}. The deataset is obtained from house numbers in Google Stree view images in 10 classes (digit `0' to digit `9'). For FL for $M=100$ each client has 732 samples. We perform data augmentation and pre-processing of random cropping (by $32\times32$ with padding of 2), color jitter (by brightness of 63/255, saturation=[0.5, 1.5] and contrast=[0.2, 1.8] implmented by \texttt{torchvision.transforms.ColorJitter}), and normalization by mean of $(0.4376821, 0.4437697, 0.47280442)$ and standard deviation of $(0.19803012, 0.20101562, 0.19703614)$.
% \subsection{NeFedAvg} \label{appx:NeFedAvg}
% We present the details of NeFedAvg (Algorithm \ref{alg:NeFedAvg}). Inconsistent parameters are decoupled by submodels that the parameters are averaged by FedAvg \cite{mcmahan2017communication} by submodels, respectively. Meanwhile, consistent parameters of widthwise-nested structure and separate depthwise structure require a distinct averaging strategy.
% For consistent parameters, the NeFL server first assess the depthwise inclusion within submodels and calculate the frequency of their occurence in uploaded weights. Subsequently, with regard to the width, these parameters are averaged in a nested manner.
% Consider an example with five submodels. Let us suppose that a convolutional layer from the first block of the first layer is included in submodel 1, 3, and 5.
% Furthermore, submodel 1 and submodel 5 are trained twice, while submodel 3 is trained three times. In this case, we have $|\mathcal{M}_2|=|\mathcal{M}_4|=0$, $|\mathcal{M}_1|=|\mathcal{M}_5|=2$, and $|\mathcal{M}_3|=3$.
% Now, delving into the parameter averaging process, the parameters exclusive to submodel 5 has ($\theta_{{\textrm c},5}\setminus\theta_{{\textrm c},3}$) are averaged using two updated weights ($\mathcal{M}_5$). Likewise, the parameters possessed by submodel 3 but not by submodel 1 ($\theta_{{\textrm c},3}\setminus\theta_{{\textrm c},1}$) are averaged using five weights ($\mathcal{M}_5\cup \mathcal{M}_3$). Finally, the parameters of submodel 1 $\theta_{{\textrm c},1}$, that is trained seven times, are averaged using seven weights ($\mathcal{M}_5\cup \mathcal{M}_3 \cup \mathcal{M}_1$). This approach ensures that consistent parameters are appropriately averaged, taking into account their depthwise inclusion and the number of occurrences across different submodels.
\subsection{Baselines}
% \paragraph{LG-FedAvg \cite{liang2020think}} The model is split into two parts. The first split part is trained locally (i.e., the parameters are not shared) and the following split part is trained globally by FedAvg \cite{mcmahan2017communication}. For the ease of the implementation, we split the models by the first layer (i.e., the first convolutional layers and following BN layers for ResNets) and the rest of the models. We can interpret the splitting that deals with inconsistency as stated in Section \ref{sec:NeFedAvg}.
\paragraph{DepthFL\cite{kim2023depthfl}} The model is split depthwise, and an auxiliary bottleneck layer is included as an independent classifier.
We implement DepthFL without separate bottleneck layers to be a special case of NeFL-DD with contiguous skipping blocks. In this special case, each submodels is a subset of the larger submodels.
Our DepthFL models incorporate downsampling layers that adjust the feature size to match the input sizes of the classifier. It is important to note that the auxiliary bottleneck layers for submodels in DepthFL can be interpreted as parameter decoupling, as discussed in Section \ref{sec:NeFedAvg}. Furthermore, we enhance DepthFL also by incorporating our proposed parameter decoupling techniques (i.e., BN layers decoupling). This addresses the issue of inconsistency and further improves the performance of DepthFL.
\paragraph{HeteroFL \& FjORD \cite{diao2021heterofl,horvath2021fjord}} HeteroFL \cite{diao2021heterofl} and FjORD \cite{horvath2021fjord} are two widthwise splitting methods designed to address the challenges posed by client heterogeneity in FL. While both methods aim to mitigate the impact of heterogeneity, these are several key differences between them. Firstly, HeteroFL does not utilize separate (i.e., inconsistent) parameters for BN layers in its submodels, whereas FjROD incorporates distinct BN layer for each submodel. This difference in handling BN layers can impact the learning dynamics and model performance.
Secondly, HeteroFL employs static batch normalization, where BN statistics are updated using the entire dataset after the training process. On the other hand, FjORD updates BN statistics during training.
Lastly, HeteroFL utilizes a masked cross-entropy loss to address the statistical heterogeneity among clients. This loss function helps to mitigate the impact of clients with the statistical heterogeneity. However, in our implementation of HeteroFL, the masked cross-entropy loss is not utilized.
\subsection{Pre-trained models}
The pre-trained models on Table \ref{tab:vit-niid} and Table \ref{tab:resnet-niid} are trained on ImageNet-1k \cite{deng2009imagenet} as following recipes \cite{pretrainedModels}:
\paragraph{ResNet18/34} The models are trained by epochs of 90, batch size of 32, SGD optimizer \cite{ruder2016overview}, learning rate of 0.1 with momentum of 0.9 and weight decay of 0.0001, where learning rate is decreased by a factor of 0.1 every 30 epochs.
\paragraph{Wide ResNet101\_2} The model is trained by epochs of 90, batch size of 32, SGD optimizer \cite{ruder2016overview}, learning rate of 0.1 with momentum of 0.9 and weight decay of 0.0001, where the learning scheduler is cosine learning rate \cite{loshchilov2017sgdr} and warming up restarts for 256 epochs.
\paragraph{ViT-B/16} The model is trained by epochs of 300, batch size of 512, AdamW optimizer \cite{loshchilov2018decoupled} with learning rate of 0.003 and weight decay of 0.3. The learning scheduler is cosine annealing \cite{loshchilov2017sgdr} after linear warmup method with decay of 0.033 for 30 epochs. Additionally, the random augmentation \cite{cubuk2020randaugment}, random mixup with $\alpha=0.2$ \cite{zhang2018mixup}, cutmix of $\alpha=1$ \cite{yun2019cutmix}, repeated augmentation, label smoothing of 0.11 \cite{szegedy2016rethinking}, clipping gradient norm to 1, model exponential moving average (EMA) are employed.
\subsection{Dynamic environment}
We simulate a dynamic environment by randomly selecting which model to train for each client during every communication round. In our experiments for Table \ref{tab:resnet} and Table \ref{tab:pretrained}, we have an equal number of five tiers of clients ($M/N_s=20$ for all tiers of clients). The resource-constrained clients (tier 1) randomly select models between $\bm{\gamma}=0.2, 0.4, 0.6$, clients in tier 2 randomly select models from the set $\bm{\gamma}=0.2, 0.4, 0.6, 0.8$, clients in tier 3 randomly select models from the set $\bm{\gamma}=0.2, 0.4, 0.6, 0.8, 1$, clients in tier 4 randomly select models from the set $\bm{\gamma}=0.4, 0.6, 0.8, 1$, and the resource-richest clients (tier 5) randomly select models from the set $\bm{\gamma}=0.6, 0.8,1$. In our experiments for Table \ref{tab:noniid} involving three submodels and 10 clients, the tier 1 clients (3 out of 10 total clients) select $\bm{\gamma}=0.5$, tier 2 clients (3 out of 10 total clients) select $\bm{\gamma}=0.75$ and tier 3 clients (4 out of 10 total clients) select $\bm{\gamma}=1$. By allowing clients to randomly choose from the available submodels, our setup reflects the dynamic nature in which clients may encounter communication computing bottlenecks during each iteration.
\newpage
\begin{table}[b]
\centering
\caption{Details of $\bm{\gamma}$ of NeFL on ResNet18}
\label{tab:resnet18detail}
\resizebox{.8\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} & \multirow{2}{*}{$\gamma_W$} & \multirow{2}{*}{$\gamma_D$} & \multicolumn{4}{c}{NeFL-D (ResNet18)} \\ \cline{4-7}
& & & \multicolumn{1}{c|}{Layer 1 (64)} & \multicolumn{1}{c|}{Layer 2 (128)} & \multicolumn{1}{c|}{Layer3 (256)} & Layer 4 (512)\\ \hline
0.2 & 1 & 0.20 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{0,0} & \multicolumn{1}{c|}{1,1} & 0,0 \\ \hline
0.4 & 1 & 0.38 & \multicolumn{1}{c|}{1,0} & \multicolumn{1}{c|}{0,0} & \multicolumn{1}{c|}{1,0} & 1,0 \\ \hline
0.6 & 1 & 0.57 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & 1,0 \\ \hline
0.8 & 1 & 0.81 & \multicolumn{1}{c|}{1,0} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{0,0} & 1,1 \\ \hline
1 & 1 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \bottomrule
\end{tabular}
}
\resizebox{.8\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} & \multirow{2}{*}{$\gamma_W$} & \multirow{2}{*}{$\gamma_D$} & \multicolumn{4}{c}{NeFL-WD (ResNet18)} \\ \cline{4-7}
& & & \multicolumn{1}{c|}{Layer 1 (64)} & \multicolumn{1}{c|}{Layer 2 (128)} & \multicolumn{1}{c|}{Layer3 (256)} & Layer 4 (512)\\ \hline
0.2 & 0.34 & 0.58 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,0 \\ \hline
0.4 & 0.4 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \hline
0.6 & 0.6 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \hline
0.8 & 0.8 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \hline
1 & 1 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \bottomrule
\end{tabular}
}
% \resizebox{.72\textwidth}{!}{
% \begin{tabular}{ccc|cccc}
% \toprule
% \multirowcell{2}{Model size\\$\gamma$} & \multirow{2}{*}{$\gamma_W$} & \multirow{2}{*}{$\gamma_D$} & \multicolumn{4}{c}{NeFL-W (ResNet18)} \\ \cline{4-7}
% & & & \multicolumn{1}{c|}{Layer 1} & \multicolumn{1}{c|}{Layer 2} & \multicolumn{1}{c|}{Layer3} & Layer 4 \\ \hline
% 0.2 & 0.2 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \hline
% 0.4 & 0.4 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \hline
% 0.6 & 0.6 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \hline
% 0.8 & 0.8 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \hline
% 1 & 1 & 1 & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1,1} & \multicolumn{1}{c|}{1, 1} & 1,1 \\ \bottomrule
% \end{tabular}
% }
\end{table}
% \clearpage
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% \begin{table}[]
% \centering
% \begin{tabular}{l|l|l|ccccc}
% \toprule
% \multicolumn{1}{l}{\multirow{2}{*}{Model}} & \multicolumn{1}{l}{\multirow{2}{*}{Method}} & \multicolumn{1}{c}{\multirow{2}{*}{$\bm{\gamma}$}} & \multicolumn{5}{c}{Model size} \\ \cline{4-8}\noalign{\vspace{0.4ex}} \multicolumn{1}{c}{}
% & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{0.2} & \multicolumn{1}{c}{0.4} & \multicolumn{1}{c}{0.6} & \multicolumn{1}{c}{0.8} & 1 \\ \noalign{\vspace{-0.4ex}}\midrule \midrule
% \multirow{4}{*}{ResNet18} & \multirow{2}{*}{NeFL-WD} & $\bm{\gamma}_W$ & \multicolumn{1}{c|}{0.35} & \multicolumn{1}{c|}{0.4} & \multicolumn{1}{c|}{0.6} & \multicolumn{1}{c|}{0.8} & 1 \\ \cline{3-8}
% & & $\bm{\gamma}_D$ & \multicolumn{1}{c|}{0.58} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{2-8}
% & \multirow{2}{*}{NeFL-W} & $\bm{\gamma}_W$ & \multicolumn{1}{c|}{0.2} & \multicolumn{1}{c|}{0.4} & \multicolumn{1}{c|}{0.6} & \multicolumn{1}{c|}{0.8} & 1 \\ \cline{3-8}
% & & $\bm{\gamma}_D$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \hline
% \multirow{6}{*}{ResNet56} & \multirow{2}{*}{NeFL-WD} & $\gamma_W$ & \multicolumn{1}{c|}{0.46} & \multicolumn{1}{c|}{0.61} & \multicolumn{1}{c|}{0.78} & \multicolumn{1}{c|}{0.90} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{0.43} & \multicolumn{1}{c|}{0.66} & \multicolumn{1}{c|}{0.77} & \multicolumn{1}{c|}{0.89} & 1 \\ \cline{2-8}
% & \multirow{2}{*}{NeFL-W} & $\gamma_W$ & \multicolumn{1}{c|}{0.2} & \multicolumn{1}{c|}{0.4} & \multicolumn{1}{c|}{0.6} & \multicolumn{1}{c|}{0.8} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{2-8}
% & \multirow{2}{*}{NeFL-D} & $\gamma_W$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{0.20} & \multicolumn{1}{c|}{0.40} & \multicolumn{1}{c|}{0.60} & \multicolumn{1}{c|}{0.80} & 1 \\ \hline
% \multirow{6}{*}{ResNet110} & \multirow{2}{*}{NeFL-WD} & $\gamma_W$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{0.2} & \multicolumn{1}{c|}{0.4} & \multicolumn{1}{c|}{0.6} & \multicolumn{1}{c|}{0.8} & 1 \\ \cline{2-8}
% & \multirow{2}{*}{NeFL-W} & $\gamma_W$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{0.2} & \multicolumn{1}{c|}{0.4} & \multicolumn{1}{c|}{0.6} & \multicolumn{1}{c|}{0.8} & 1 \\ \cline{2-8}
% & \multirow{2}{*}{NeFL-D} & $\gamma_W$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{0.20} & \multicolumn{1}{c|}{0.40} & \multicolumn{1}{c|}{0.60} & \multicolumn{1}{c|}{0.80} & 1 \\ \bottomrule
% \end{tabular}
% \end{table}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% \begin{table}[]
% \centering
% \begin{tabular}{l|l|l|ccccc}
% \toprule
% \multirow{2}{*}{Model} & \multirow{2}{*}{Method} & \multirow{2}{*}{$\gamma$} & \multicolumn{5}{c|}{Model size} \\ \cline{4-8}
% & & & \multicolumn{1}{c|}{0.04} & \multicolumn{1}{c|}{0.16} & \multicolumn{1}{c|}{0.36} & \multicolumn{1}{c|}{0.64} & 1 \\ \midrule\midrule
% \multirow{6}{*}{ResNet110} & \multirow{2}{*}{NeFL-WD} & $\gamma_W$ & \multicolumn{1}{c|}{0.26} & \multicolumn{1}{c|}{0.42} & \multicolumn{1}{c|}{0.59} & \multicolumn{1}{c|}{0.77} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{0.16} & \multicolumn{1}{c|}{0.38} & \multicolumn{1}{c|}{0.61} & \multicolumn{1}{c|}{0.83} & 1 \\ \cline{2-8}
% & \multirow{2}{*}{NeFL-W} & $\gamma_W$ & \multicolumn{1}{c|}{0.2} & \multicolumn{1}{c|}{0.4} & \multicolumn{1}{c|}{0.6} & \multicolumn{1}{c|}{0.8} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{2-8}
% & \multirow{2}{*}{NeFL-D} & $\gamma_W$ & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \cline{3-8}
% & & $\gamma_D$ & \multicolumn{1}{c|}{0.04} & \multicolumn{1}{c|}{0.16} & \multicolumn{1}{c|}{0.37} & \multicolumn{1}{c|}{0.65} & 1 \\ \bottomrule
% \end{tabular}
% \end{table}
\begin{table}[]
\centering
\caption{Details of $\bm{\gamma}$ of NeFL on ResNet34}
\label{tab:resnet34detail}
\resizebox{.8\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} & \multirow{2}{*}{$\gamma_W$} & \multirow{2}{*}{$\gamma_D$} & \multicolumn{4}{c}{NeFL-D (ResNet34)} \\ \cline{4-7}
& & & \multicolumn{1}{c|}{Layer 1 (64)} & \multicolumn{1}{c|}{Layer 2 (128)} & \multicolumn{1}{c|}{Layer3 (256)} & Layer 4 (512)\\ \hline
0.2 & 1 & 0.23 & \multicolumn{1}{c|}{1,0,0} & \multicolumn{1}{c|}{1,0,0,0} & \multicolumn{1}{c|}{1,0,0,0,0,0} & 1,0,0 \\ \hline
0.4 & 1 & 0.39 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1,1,0,0,0,1} & 1,0,0 \\ \hline
0.6 & 1 & 0.61 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1,1,0,0,0,1} & 1,0,1 \\ \hline
0.8 & 1 & 0.81 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,0,0,1} & \multicolumn{1}{c|}{1,1,0,0,0,1} & 1,1,1 \\ \hline
1 & 1 & 1 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1,1,1,1,1,1} & 1,1,1 \\ \bottomrule
\end{tabular}
}
\resizebox{.8\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} & \multirow{2}{*}{$\gamma_W$} & \multirow{2}{*}{$\gamma_D$} & \multicolumn{4}{c}{NeFL-WD (ResNet34)} \\ \cline{4-7}
& & & \multicolumn{1}{c|}{Layer 1 (64)} & \multicolumn{1}{c|}{Layer 2 (128)} & \multicolumn{1}{c|}{Layer3 (256)} & Layer 4 (512)\\ \hline
0.2 & 0.38 & 0.53 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,0,0,1} & \multicolumn{1}{c|}{1,0,0,0,0,1} & 1,0,1 \\ \hline
0.4 & 0.63 & 0.64 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,0,0,1} & \multicolumn{1}{c|}{1,1,1,0,0,1} & 1,0,1 \\ \hline
0.6 & 0.77 & 0.78 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1,1,1,1,0,1} & 1,0,1 \\ \hline
0.8 & 0.90 & 0.89 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1,1,1,0,0,1} & 1,1,1 \\ \hline
1 & 1 & 1 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1,1,1,1,1,1} & 1,1,1 \\ \bottomrule
\end{tabular}
}
\end{table}
\begin{table}[]
\centering
\caption{Details of $\bm{\gamma}$ of NeFL on ResNet56}
\label{tab:resnet56detail}
\resizebox{.85\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} & \multirow{2}{*}{$\gamma_W$} & \multirow{2}{*}{$\gamma_D$} & \multicolumn{3}{c}{NeFL-D (ResNet56)} \\ \cline{4-6} & & & \multicolumn{1}{c|}{Layer 1 (16)} & \multicolumn{1}{c|}{Layer 2 (32)} & \multicolumn{1}{c}{Layer3 (64)} \\ \hline
0.2 & 1 & 0.2 & \multicolumn{1}{c|}{1, 1, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.4 & 1 & 0.4 & \multicolumn{1}{c|}{1, 1, 1, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 0, 0, 0, 0, 0} \\ \hline
0.6 & 1 & 0.6 & \multicolumn{1}{c|}{1, 1, 1, 1, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 0, 0, 0} \\ \hline
0.8 & 1 & 0.8 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 0, 0} \\ \hline
1 & 1 & 1 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1} \\ \bottomrule
\end{tabular}
}
\resizebox{.85\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} & \multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{3}{c}{NeFL-D\textsubscript{O} (ResNet56)} \\ \cline{4-6} & &
& \multicolumn{1}{c|}{Layer 1 (16)} & \multicolumn{1}{c|}{Layer 2 (32)} & \multicolumn{1}{c}{Layer3 (64)} \\ \hline
0.2 &1 &0.2 & \multicolumn{1}{c|}{1, 8, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 8, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 8, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.4 &1 &0.4 & \multicolumn{1}{c|}{1, 1, 7, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 7, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 6, 0, 0, 0, 0, 0} \\ \hline
0.6 &1 &0.6 & \multicolumn{1}{c|}{1, 1, 1, 6, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 6, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 4, 0, 0, 0} \\ \hline
0.8 &1 &0.8 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 2, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 3, 0, 0} \\ \hline
1 &1 &1 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1} \\ \bottomrule
\end{tabular}
}
\resizebox{.85\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} & \multirow{2}{*}{$\gamma_D$} & \multicolumn{3}{c}{NeFL-WD (ResNet56)} \\ \cline{4-6} & &
& \multicolumn{1}{c|}{Layer 1 (16)} & \multicolumn{1}{c|}{Layer 2 (32)} & \multicolumn{1}{c}{Layer3 (64)}\\ \hline
0.2 & 0.46 & 0.43 & \multicolumn{1}{c|}{1, 1, 1, 1, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 0, 0, 0, 0, 0} \\ \hline
0.4 & 0.61 & 0.66 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 0, 0, 0} \\ \hline
0.6 & 0.77 & 0.77 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 0, 0} \\ \hline
0.8 & 0.90 & 89 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 0} \\ \hline
1 & 1 & 1 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1} \\ \bottomrule
\end{tabular}
}
\end{table}
\begin{table}[]
\centering
\caption{Details of $\bm{\gamma}$ of NeFL on ViT-B/16}
\resizebox{.45\textwidth}{!}{
\begin{tabular}{ccc|c}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{1}{c}{NeFL-D (ViT-B/16)}
\\ \cline{4-4} & &
& \multicolumn{1}{c}{Block} \\ \hline
0.5 &1 &0.50 & \multicolumn{1}{c}{1,1,1,1,1,1,0,0,0,0,0,0} \\ \hline
0.75 &1 &0.75 & \multicolumn{1}{c}{1,1,1,1,1,1,1,1,1,0,0,0} \\ \hline
1 &1 &1 & \multicolumn{1}{c}{1,1,1,1,1,1,1,1,1,1,1,1} \\ \bottomrule
\end{tabular}
}
\resizebox{.45\textwidth}{!}{
\begin{tabular}{ccc|c}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{1}{c}{NeFL-W (ViT-B/16)}
\\ \cline{4-4} & &
& \multicolumn{1}{c}{Block} \\ \hline
0.5 &0.5 &1 & \multicolumn{1}{c}{1,1,1,1,1,1,1,1,1,1,1,1} \\ \hline
0.75 &0.75 &1 & \multicolumn{1}{c}{1,1,1,1,1,1,1,1,1,1,1,1} \\ \hline
1 &1 &1 & \multicolumn{1}{c}{1,1,1,1,1,1,1,1,1,1,1,1} \\ \bottomrule
\end{tabular}
}
\end{table}
\begin{sidewaystable} % rotating
\centering
\caption{Details of $\bm{\gamma}$ of NeFL on ResNet110}
\label{tab:resnet110detail}
\resizebox{1\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{3}{c}{NeFL-D (ResNet110)} \\ \cline{4-6} & &
& \multicolumn{1}{c|}{Layer 1 (16)} & \multicolumn{1}{c|}{Layer 2 (32)} & \multicolumn{1}{c}{Layer3 (64)}\\ \hline
0.2 &1 &0.20 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.4 &1 &0.40 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.6 &1 &0.60 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.8 &1 &0.80 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0} \\ \hline
1 &1 &1 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} \\ \bottomrule
\end{tabular}
}
\resizebox{1\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{3}{c}{NeFL-WD (ResNet110)} \\ \cline{4-6} & &
& \multicolumn{1}{c|}{Layer 1 (16)} & \multicolumn{1}{c|}{Layer 2 (32)} & \multicolumn{1}{c}{Layer3 (64)} \\ \hline
0.2 &0.46 &0.44 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1} \\ \hline
0.4 &0.60 &0.66 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1} \\ \hline
0.6 &0.77 &0.77 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1} \\ \hline
0.8 &0.90 &0.89 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1} \\ \hline
1 &1 &1 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} \\ \bottomrule
\end{tabular}
}
\resizebox{1\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{3}{c}{NeFL-D (ResNet110)} \\ \cline{4-6} & &
& \multicolumn{1}{c|}{Layer 1 (16)} & \multicolumn{1}{c|}{Layer 2 (32)} & \multicolumn{1}{c}{Layer3 (64)} \\ \hline
0.04 & 1& 0.04 & \multicolumn{1}{c|}{1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.16 & 1& 0.16 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.36 & 1& 0.37 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.64 & 1& 0.65 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0} \\ \hline
1 & 1& 1 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} \\ \bottomrule
\end{tabular}
}
\resizebox{1\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{3}{c}{NeFL-WD (ResNet110)} \\ \cline{4-6} & &
& \multicolumn{1}{c|}{Layer 1 (16)} & \multicolumn{1}{c|}{Layer 2 (32)} & \multicolumn{1}{c}{Layer3 (64)} \\ \hline
0.04 & 0.26 & 0.16 & \multicolumn{1}{c|}{1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.16 & 0.42 & 0.38 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.36 & 0.59 & 0.61 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0} \\ \hline
0.64 & 0.77 & 0.83 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0} \\ \hline
1 & 1 & 1 & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & \multicolumn{1}{c}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} \\ \bottomrule
\end{tabular}
}
\vspace{0.3in}
\caption{Details of $\bm{\gamma}$ of NeFL on Wide ResNet101\_2}
\label{tab:wresnet101detail}
\resizebox{.9\textwidth}{!}{
\begin{tabular}{ccc|cccc}
\toprule
\multirowcell{2}{Model size\\$\gamma$} &\multirow{2}{*}{$\gamma_W$} &\multirow{2}{*}{$\gamma_D$} & \multicolumn{4}{c}{NeFL-D (Wide ResNet101\_2)}
\\ \cline{4-7} & &
& \multicolumn{1}{c|}{Layer 1 (128)} & \multicolumn{1}{c|}{Layer 2 (256)} & \multicolumn{1}{c|}{Layer3 (512)} & Layer 4 (1024)\\ \hline
0.5 &1 &0.51 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} & 1,1,0 \\ \hline
0.75 &1 &0.75 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0} & 1,1,1 \\ \hline
1 &1 &1 & \multicolumn{1}{c|}{1,1,1} & \multicolumn{1}{c|}{1,1,1,1} & \multicolumn{1}{c|}{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} & 1,1,1 \\ \bottomrule
\end{tabular}}
\end{sidewaystable}
\clearpage
\end{document}
\begin{thebibliography}{10}\itemsep=0pt
\bibitem{europeData}
Data protection: Rules for the protection of personal data inside and outside
the {EU}, 2022.
\newblock \url{https://commission.europa.eu/law/law-topic/data-protection_en}
[Accessed: (2023/5/10)].
\bibitem{pretrainedModels}
Pytorch models and pre-trained weights, 2023.
\newblock \url{https://pytorch.org/vision/stable/models.html} [Accessed:
(2023/5/16)].
\bibitem{acar2021federated}
Durmus Alp~Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough,
and Venkatesh Saligrama.
\newblock Federated learning based on dynamic regularization.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2021.
\bibitem{afonin2022towards}
Andrei Afonin and Sai~Praneeth Karimireddy.
\newblock Towards model agnostic federated learning using knowledge
distillation.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2022.
\bibitem{arivazhagan2019federated}
Manoj~Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya~Kumar Singh, and Sunav
Choudhary.
\newblock Federated learning with personalization layers.
\newblock {\em arXiv preprint arXiv:1912.00818}, 2019.
\bibitem{bachlechner2021rezero}
Thomas Bachlechner, Bodhisattwa~Prasad Majumder, Henry Mao, Gary Cottrell, and
Julian McAuley.
\newblock Rezero is all you need: fast convergence at large depth.
\newblock In {\em Conference on Uncertainty in Artificial Intelligence (UAI)},
2021.
\bibitem{cristian2006model}
Cristian Bucilu\v{a}, Rich Caruana, and Alexandru Niculescu-Mizil.
\newblock Model compression.
\newblock In {\em ACM SIGKDD International Conference on Knowledge Discovery
and Data Mining (KDD)}, 2006.
\bibitem{chang2018multilevel}
Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert.
\newblock Multi-level residual networks from dynamical systems view.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2018.
\bibitem{chen2023on}
Hong-You Chen, Cheng-Hao Tu, Ziwei Li, Han~Wei Shen, and Wei-Lun Chao.
\newblock On the importance and applicability of pre-training for federated
learning.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2023.
\bibitem{chen2018node}
Ricky T.~Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud.
\newblock Neural ordinary differential equations.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2018.
\bibitem{chen2022when}
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong.
\newblock When vision transformers outperform resnets without pre-training or
strong data augmentations.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2022.
\bibitem{cubuk2020randaugment}
Ekin~Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le.
\newblock Randaugment: Practical automated data augmentation with a reduced
search space.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2020.
\bibitem{darlow2018cinic10}
Luke~N. Darlow, Elliot~J. Crowley, Antreas Antoniou, and Amos~J. Storkey.
\newblock Cinic-10 is not imagenet or cifar-10.
\newblock {\em arXiv preprint arXiv:1810.03505}, 2018.
\bibitem{deng2009imagenet}
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
\newblock {ImageNet}: A large-scale hierarchical image database.
\newblock In {\em IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2009.
\bibitem{diao2021heterofl}
Enmao Diao, Jie Ding, and Vahid Tarokh.
\newblock Hetero{FL}: Computation and communication efficient federated
learning for heterogeneous clients.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2021.
\bibitem{dosovitskiy2021an}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at
scale.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2021.
\bibitem{dou2021federated}
Qi Dou, Tiffany~Y So, Meirui Jiang, Quande Liu, Varut Vardhanabhuti, Georgios
Kaissis, Zeju Li, Weixin Si, Heather~HC Lee, Kevin Yu, et~al.
\newblock Federated deep learning for detecting {COVID}-19 lung abnormalities
in {CT}: {A} privacy-preserving multinational validation study.
\newblock {\em NPJ digital medicine}, 4(1):60, 2021.
\bibitem{eckart1936approximation}
Carl Eckart and Gale Young.
\newblock The approximation of one matrix by another of lower rank.
\newblock {\em Psychometrika}, 1(3):211--218, 1936.
\bibitem{ha2017hypernetworks}
David Ha, Andrew~M. Dai, and Quoc~V. Le.
\newblock Hypernetworks.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2017.
\bibitem{haddadpour2021federated}
Farzin Haddadpour, Mohammad~Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi.
\newblock Federated learning with compression: Unified analysis and sharp
guarantees.
\newblock In {\em International Conference on Artificial Intelligence and
Statistics (AISTATS)}, 2021.
\bibitem{ODEbook}
E. Hairer, S.P. N{\o}rsett, and G. Wanner.
\newblock {\em Solving Ordinary Differential Equations {I} Nonstiff problems}.
\newblock Springer, Berlin, second edition, 2000.
\bibitem{he2020group}
Chaoyang He, Murali Annavaram, and Salman Avestimehr.
\newblock Group knowledge transfer: Federated learning of large cnns at the
edge.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2020.
\bibitem{he2020fedgkt}
Chaoyang He, Murali Annavaram, and Salman Avestimehr.
\newblock Group knowledge transfer: Federated learning of large cnns at the
edge.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2020.
\bibitem{he2021ssfl}
Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and
Salman Avestimehr.
\newblock {SSFL}: Tackling label deficiency in federated learning via
personalized self-supervision.
\newblock {\em arXiv preprint arXiv:2110.02470}, 2021.
\bibitem{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In {\em IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2016.
\bibitem{he2017channel}
Yihui He, Xiangyu Zhang, and Jian Sun.
\newblock Channel pruning for accelerating very deep neural networks.
\newblock In {\em IEEE International Conference on Computer Vision (ICCV)},
2017.
\bibitem{hinton2015distilling}
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
\newblock Distilling the knowledge in a neural network.
\newblock {\em arXiv preprint arXiv:1503.02531}, 2015.
\bibitem{hong2022efficient}
Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou.
\newblock Efficient split-mix federated learning for on-demand and in-situ
customization.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2022.
\bibitem{horvath2021fjord}
Samuel Horv\'{a}th, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis,
Stylianos Venieris, and Nicholas Lane.
\newblock {FjORD}: Fair and accurate federated learning under heterogeneous
targets with ordered dropout.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2021.
\bibitem{huang2016stochasticDepth}
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger.
\newblock Deep networks with stochastic depth.
\newblock In {\em European Conference on Computer Vision (ECCV)}, 2016.
\bibitem{jiang2022pruningFL}
Yuang Jiang, Shiqiang Wang, Víctor Valls, Bong~Jun Ko, Wei-Han Lee, Kin~K.
Leung, and Leandros Tassiulas.
\newblock Model pruning enables efficient federated learning on edge devices.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems
(TNNLS)}, pages 1--13, 2022.
\bibitem{kairouz2021advances}
Peter Kairouz, H.~Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi
Bennis, Arjun~Nitin Bhagoji, Kallista Bonawit, Zachary Charles, Graham
Cormode, Rachel Cummings, Rafael G.~L. D’Oliveira, Hubert Eichner, Salim
El~Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón,
Badih Ghazi, Phillip~B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He,
Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi,
Gauri Joshi, Mikhail Khodak, Jakub Konecný, Aleksandra Korolova, Farinaz
Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal,
Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel
Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian~U.
Stich, Ziteng Sun, Ananda Theertha~Suresh, Florian Tramèr, Praneeth
Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix~X. Yu, Han Yu,
and Sen Zhao.
\newblock {\em Advances and Open Problems in Federated Learning}.
\newblock Now Foundations and Trends, 2021.
\bibitem{kim2023depthfl}
Minjae Kim, Sangyoon Yu, Suhyun Kim, and Soo-Mook Moon.
\newblock Depth{FL} : Depthwise federated learning for heterogeneous clients.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2023.
\bibitem{kolesnikov2020big}
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung,
Sylvain Gelly, and Neil Houlsby.
\newblock Big transfer {(BiT)}: General visual representation learning.
\newblock In {\em European Conference on Computer Vision (ECCV)}, 2020.
\bibitem{cifar10}
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.
\newblock {CIFAR}-10 ({Canadian Institute for Advanced Research}).
\newblock \url{http://www.cs.toronto.edu/~kriz/cifar.html}.
\bibitem{lee2020signal}
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip~HS Torr.
\newblock A signal propagation perspective for pruning neural networks at
initialization.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2020.
\bibitem{li2017pruning}
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans~Peter Graf.
\newblock Pruning filters for efficient convnets.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2017.
\bibitem{li2021federated}
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He.
\newblock Federated learning on non-iid data silos: An experimental study.
\newblock {\em arXiv preprint arXiv:2102.02079}, 2021.
\bibitem{li2021model}
Qinbin Li, Bingsheng He, and Dawn Song.
\newblock Model-contrastive federated learning.
\newblock In {\em IEEE/CVF Conference on Computer Vision and Pattern
Recognition}, 2021.
\bibitem{Li2020federated}
Tian Li, Anit~Kumar Sahu, Ameet Talwalkar, and Virginia Smith.
\newblock Federated learning: Challenges, methods, and future directions.
\newblock {\em {IEEE} Signal Processing Magazine}, 37(3):50--60, May 2020.
\bibitem{Li2020fedprox}
Tian Li, Anit~Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and
Virginia Smith.
\newblock Federated optimization in heterogeneous networks.
\newblock In {\em Machine Learning and Systems (MLSys)}, 2020.
\bibitem{Li2020On}
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang.
\newblock On the convergence of fedavg on non-iid data.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2020.
\bibitem{liang2020think}
Paul~Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and
Louis-Philippe Morency.
\newblock Think locally, act globally: Federated learning with local and global
representations.
\newblock {\em arXiv preprint arXiv:2001.01523}, 2020.
\bibitem{lin2020ensemble}
Tao Lin, Lingjing Kong, Sebastian~U Stich, and Martin Jaggi.
\newblock Ensemble distillation for robust model fusion in federated learning.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2020.
\bibitem{Liu2017learning}
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui
Zhang.
\newblock Learning efficient convolutional networks through network slimming.
\newblock In {\em IEEE International Conference on Computer Vision (ICCV)},
2017.
\bibitem{loshchilov2017sgdr}
Ilya Loshchilov and Frank Hutter.
\newblock {SGDR}: Stochastic gradient descent with warm restarts.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2017.
\bibitem{loshchilov2018decoupled}
Ilya Loshchilov and Frank Hutter.
\newblock Decoupled weight decay regularization.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2019.
\bibitem{makhija2022federated}
Disha Makhija, Nhat Ho, and Joydeep Ghosh.
\newblock Federated self-supervised learning for heterogeneous clients.
\newblock {\em arXiv preprint arXiv:2205.12493}, 2022.
\bibitem{mcmahan2017communication}
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera~y
Arcas.
\newblock Communication-efficient learning of deep networks from decentralized
data.
\newblock In {\em International Conference on Artificial Intelligence and
Statistics (AISTATS)}, 2017.
\bibitem{Mugunthan2022fedltn}
Vaikkunth Mugunthan, Eric Lin, Vignesh Gokul, Christian Lau, Lalana Kagal, and
Steve Pieper.
\newblock {FedLTN}: Federated learning for sparse and personalized lottery
ticket networks.
\newblock In {\em European Conference on Computer Vision (ECCV)}, 2022.
\bibitem{svhn}
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew~Y.
Ng.
\newblock Reading digits in natural images with unsupervised feature learning.
\newblock In {\em NIPS Workshop on Deep Learning and Unsupervised Feature
Learning}, 2011.
\bibitem{pytorch}
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban
Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan
Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith
Chintala.
\newblock Pytorch: An imperative style, high-performance deep learning library.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2019.
\bibitem{qu2022rethinking}
Liangqiong Qu, Yuyin Zhou, Paul~Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli,
Li Fei-Fei, and Daniel Rubin.
\newblock Rethinking architecture design for tackling data heterogeneity in
federated learning.
\newblock In {\em IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2022.
\bibitem{rothchild2020fetchsgd}
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica,
Vladimir Braverman, Joseph Gonzalez, and Raman Arora.
\newblock Fetch{SGD}: Communication-efficient federated learning with
sketching.
\newblock In {\em International Conference on Machine Learning (ICML)}, 2020.
\bibitem{ruder2016overview}
Sebastian Ruder.
\newblock An overview of gradient descent optimization algorithms.
\newblock {\em arXiv preprint arXiv:1609.04747}, 2016.
\bibitem{santurkar2018how}
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.
\newblock How does batch normalization help optimization?
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)},
2018.
\bibitem{seo2020federated}
Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim.
\newblock Federated knowledge distillation.
\newblock {\em arXiv preprint arXiv:2011.02367}, 2020.
\bibitem{szegedy2016rethinking}
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Wojna.
\newblock Rethinking the inception architecture for computer vision.
\newblock In {\em IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, pages 2818--2826, 2016.
\bibitem{mingxing2019efficient}
Mingxing Tan and Quoc Le.
\newblock {E}fficient{N}et: Rethinking model scaling for convolutional neural
networks.
\newblock In {\em International Conference on Machine Learning (ICML)}, 2019.
\bibitem{thapa2022splitfed}
Chandra Thapa, Pathum~Chamikara Mahawaga~Arachchige, Seyit Camtepe, and Lichao
Sun.
\newblock Splitfed: When federated learning meets split learning.
\newblock {\em AAAI Conference on Artificial Intelligence (AAAI)}, 2022.
\bibitem{touvron2021going}
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and
Herv\'e J\'egou.
\newblock Going deeper with image transformers.
\newblock In {\em IEEE International Conference on Computer Vision (ICCV)},
2021.
\bibitem{vepakomma2018split}
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar.
\newblock Split learning for health: Distributed deep learning without sharing
raw patient data.
\newblock {\em arXiv preprint arXiv:1812.00564}, 2018.
\bibitem{Wang2020Federated}
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and
Yasaman Khazaeni.
\newblock Federated learning with matched averaging.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2020.
\bibitem{wu2022fedkd}
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie.
\newblock Communication-efficient federated learning via knowledge
distillation.
\newblock {\em Nature Communications}, 13(1), Apr. 2022.
\bibitem{yun2019cutmix}
Sangdoo Yun, Dongyoon Han, Seong~Joon Oh, Sanghyuk Chun, Junsuk Choe, and
Youngjoon Yoo.
\newblock Cutmix: Regularization strategy to train strong classifiers with
localizable features.
\newblock In {\em IEEE International Conference on Computer Vision (ICCV)},
2019.
\bibitem{yurochkin2019bayesian}
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia
Hoang, and Yasaman Khazaeni.
\newblock {B}ayesian nonparametric federated learning of neural networks.
\newblock In {\em International Conference on Machine Learning (ICML)}, 2019.
\bibitem{sergey2016wide}
Sergey Zagoruyko and Nikos Komodakis.
\newblock Wide residual networks.
\newblock In {\em British Machine Vision Conference (BMVC)}, 2016.
\bibitem{zhang2018mixup}
Hongyi Zhang, Moustapha Cisse, Yann~N. Dauphin, and David Lopez-Paz.
\newblock mixup: Beyond empirical risk minimization.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2018.
\bibitem{zhao2018federated}
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.
\newblock Federated learning with non-iid data.
\newblock {\em arXiv preprint arXiv:1806.00582}, 2018.
\bibitem{zhu2021datafree}
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou.
\newblock Data-free knowledge distillation for heterogeneous federated
learning.
\newblock {\em arXiv preprint arXiv:2105.10056}, 2021.
\bibitem{zhuang2022divergenceaware}
Weiming Zhuang, Yonggang Wen, and Shuai Zhang.
\newblock Divergence-aware federated self-supervised learning.
\newblock In {\em International Conference on Learning Representations (ICLR)},
2022.
\end{thebibliography}