Fedstellar: A Platform for Decentralized Federated Learning
Extracted source text · arXiv 2306.09750
\documentclass[5p, review]{elsarticle}
\makeatletter
\def\ps@pprintTitle{%
\let\@oddhead\@empty
\let\@evenhead\@empty
\let\@evenfoot\@oddfoot}
\makeatother
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[english]{babel}
\usepackage[babel=true]{csquotes}
\usepackage[fleqn]{amsmath}
\usepackage{amsthm}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{amssymb}
\usepackage{float}
\usepackage{hyperref}
\usepackage[english]{cleveref}
\usepackage{subcaption}
\usepackage{caption}
\usepackage{natbib}
\usepackage{soul}
\usepackage{color}
\usepackage[dvipsnames,table]{xcolor}
\usepackage{pifont}
\usepackage{graphicx}
\usepackage{svg}
\usepackage[inline]{enumitem}
\usepackage{array}
\usepackage{longtable}
\usepackage{tabularx}
\usepackage{graphicx}
\usepackage{makecell}
\usepackage{lscape}
\usepackage{threeparttable}
\usepackage{xurl}
\usepackage{wasysym}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage[
separate-uncertainty = true,
multi-part-units = repeat
]{siunitx}
\usepackage{tablefootnote}
\usepackage{footnote}
\makesavenoteenv{tabular}
\makesavenoteenv{table}
\makesavenoteenv{table*}
\setstcolor{blue}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\newtoggle{finalPaper}
\togglefalse{finalPaper} % Change \toggletrue <-> \togglefalse
%\toggletrue{finalPaper} % Change \toggletrue <-> \togglefalse
\iftoggle{finalPaper} {
\newcommand{\addtxt}[1]{#1}
\newcommand{\addtable}[1]{#1}
\newcommand{\change}[2]{#2}
\newcommand{\rmvtxt}[1]{}
} {
\newcommand{\addtxt}[1]{\textcolor{blue}{#1}}
\newcommand{\addtable}[1]{\color{blue}{#1}}
\newcommand{\change}[2]{\st{#1}\textcolor{blue}{#2}}
\newcommand{\rmvtxt}[1]{\st{#1}}
}
\definecolor{ao}{rgb}{0.0, 0.5, 0.0}
\definecolor{amber}{rgb}{1.0, 0.49, 0.0}
\newcommand{\yestick}{{\color{ao}\ding{51}}}
\newcommand{\notick}{{\color{red}\ding{55}}}
\newcommand{\maybetick}{{\color{amber}$\approx$}}
\newcommand{\specialcell}[2][c]{%
\begin{tabular}[#1]{@{}c@{}}#2\end{tabular}}
\newcommand\sbullet[1][.5]{\mathbin{\vcenter{\hbox{\scalebox{#1}{$\bullet$}}}}}
%\usepackage{lineno}
%\modulolinenumbers[5]
\definecolor{gray45}{gray}{.45}
\definecolor{gray75}{gray}{.75}
\definecolor{orange-fig}{HTML}{C55A11}
\hyphenation{data-sets}
\journal{Expert Systems with Applications}
%\bibliographystyle{elsarticle-num}
%% APA style - Expert Systems with Applications
% https://www.elsevier.com/journals/expert-systems-with-applications/0957-4174/guide-for-authors
\providecommand{\URLprefix}{Retrieved from }
\bibliographystyle{model5-names}\biboptions{authoryear}
\begin{document}
\begin{frontmatter}
\title{Fedstellar: A Platform for Decentralized Federated Learning}
\author[1]{Enrique Tom\'as Mart\'inez Beltr\'an\corref{cor1}}\ead{enriquetomas@um.es}
\author[2]{\'Angel Luis Perales G\'omez}\ead{angelluis.perales@um.es}
\author[3]{Chao Feng}\ead{cfeng@ifi.uzh.ch}
\author[1]{Pedro Miguel S\'anchez S\'anchez}\ead{pedromiguel.sanchez@um.es}
\author[1]{Sergio L\'opez Bernal}\ead{slopez@um.es}
\author[4]{G\'er\^ome Bovet}\ead{gerome.bovet@armasuisse.ch}
\author[1]{Manuel Gil P\'erez}\ead{mgilperez@um.es}
\author[1]{Gregorio Mart\'inez P\'erez}\ead{gregorio@um.es}
\author[3]{Alberto Huertas Celdr\'an}\ead{huertas@ifi.uzh.ch}
\cortext[cor1]{Corresponding author.}
\address[1]{Department of Information and Communications Engineering, University of Murcia, 30100 Murcia, Spain}
\address[2]{Department of Computers Engineering and Technology, University of Murcia, 30100 Murcia, Spain}
\address[3]{Communication Systems Group CSG, Department of Informatics IfI, University of Zurich UZH, CH--8050 Zürich, Switzerland}
\address[4]{Cyber-Defence Campus, Armasuisse Science and Technology, 3602 Thun, Switzerland}
\begin{abstract}
In 2016, Google proposed Federated Learning (FL) as a novel paradigm to train Machine Learning (ML) models across the participants of a federation while preserving data privacy. Since its birth, Centralized FL (CFL) has been the most used approach, where a central entity aggregates participants' models to create a global one. However, CFL presents limitations such as communication bottlenecks, single point of failure, and reliance on a central server. Decentralized Federated Learning (DFL) addresses these issues by enabling decentralized model aggregation and minimizing dependency on a central entity. Despite these advances, current platforms training DFL models struggle with key issues such as managing heterogeneous federation network topologies, adapting the FL process to virtualized or physical deployments, and using a limited number of metrics to evaluate different federation scenarios for efficient implementation. To overcome these challenges, this paper presents Fedstellar, a novel platform designed to train FL models in a decentralized, semi-decentralized, and centralized fashion across diverse federations of physical or virtualized devices. Fedstellar allows users to create federations by customizing parameters like the number and type of devices training FL models, the network topology connecting them, the machine and deep learning algorithms, or the datasets of each participant, among others. Additionally, it offers real-time monitoring of model and network performance. The Fedstellar implementation encompasses a web application with an interactive graphical interface, a controller for deploying federations of nodes using physical or virtual devices, and a core deployed on each device which provides the logic needed to train, aggregate, and communicate in the network. The effectiveness of the platform has been demonstrated in two scenarios: a physical deployment involving single-board devices such as Raspberry Pis for detecting cyberattacks, and a virtualized deployment comparing various FL approaches in a controlled environment using MNIST and CIFAR-10 datasets. In both scenarios, Fedstellar demonstrated consistent performance and adaptability, achieving $F_{1} \ scores$ of 91\%, 98\%, and 91.2\% using DFL for detecting cyberattacks and classifying MNIST and CIFAR-10, respectively, reducing training time by 32\% compared to centralized approaches.
\end{abstract}
\begin{keyword}
Decentralized Federated Learning \sep Deep Learning \sep Collaborative Training \sep Communication Mechanisms
\end{keyword}
\end{frontmatter}
%\linenumbers
\input{1_introduction}
\input{2_relatedwork}
%\input{3_statement}
\input{4_platform}
\input{5_validation}
\input{6_results}
\input{7_conclusion}
\section*{Acknowledgments}
\section*{Funding}
This work has been partially supported by \textit{(a)} 21629/FPI/21, Seneca Foundation - Science and Technology Agency of the Region of Murcia (Spain), \textit{(b)} the Strategic Cybersecurity Project in Spain entitled CDL-TALENTUM (Development of Professionals and Researchers in Cybersecurity, Cyberdefence and Data Science) with the support of INCIBE and the European Mechanism for Recovery and Resilience (MRR) and as part of the measures of the Recovery, Transformation and Resilience Plan, \textit{(c)} the Swiss Federal Office for Defense Procurement (armasuisse) with the DEFENDIS and CyberForce projects (CYD-C-2020003), \textit{(d)} the University of Zürich UZH, \textit{(e)} MCIN/AEI/ 10.13039/501100011033, NextGenerationEU/PRTR, UE, under grant TED 2021-129300B-I00, and \textit{(f)} MCIN/AEI/ 10.13039/501100011033/FEDER, UE, under grant PID2021-122466OB-I00.
\bibliography{references}
\end{document}
\section{Introduction}\label{sec:introduction}
In the digital era, a remarkable proliferation of Internet of Things (IoT) devices is transforming various sectors, ranging from wearable technologies and connected vehicles to smart homes and cities. The rapid evolution of these devices has resulted in an enormous volume, variety, and speed of data generated by IoT devices, amplifying the demand for advanced computation techniques \citep{idc:iot_number:2022}. With their heavy reliance on centralized data collection and processing, traditional Machine Learning (ML) and Deep Learning (DL) techniques increasingly face privacy, scalability, computational efficiency, and latency issues \citep{Paleyes:ml-challenges:2022}.
Federated Learning (FL), an emerging and promising Artificial Intelligence (AI) paradigm, offers a potential solution to these limitations \citep{McMahan:FL_google:2016}. It facilitates the training of models across numerous devices or nodes, preserving data privacy by conducting computations locally on each device, thus avoiding the need to share raw data directly. Furthermore, FL promotes a distributed learning process across the federation, mitigating some of the primary limitations of traditional ML, such as scalability and the requirement for central data repositories. While the majority of current tools, frameworks, and platforms for training FL models are based on Centralized Federated Learning (CFL), wherein a single participant receives models from others and performs the aggregation, this centralization introduces potential issues. These include a single point of failure and communication bottlenecks, which can negatively impact system performance and reliability \citep{Hard:google_centralized_to_dfl:2021}.
Decentralized Federated Learning (DFL) emerged to overcome these limitations. DFL enhances decentralization by facilitating model aggregation at multiple nodes, drastically reducing dependence on a single central server \citep{MartinezBeltran:DFL_survey:2022}. This innovative structure promises to deliver substantial improvements in scalability, robustness, and efficiency, aligning seamlessly with the dynamic and distributed nature of IoT applications \citep{Wang:dfl_edge:2022}. Despite these promising attributes, the adoption and practical implementation of DFL poses challenges. Current platforms and frameworks designed for DFL face limitations such as inadequate support for various aggregation algorithms, models, datasets, or network topologies, along with constraints in asynchronous message exchange during federation \citep{Wang:edge_communication_optimization:2021}. Many are also specifically tailored for certain applications, thereby lacking in versatility. Furthermore, they do not facilitate deployment on physical devices. While these platforms have certainly made strides in the field of DFL, there is still a pressing need for innovative solutions.
To address these challenges and contribute to the advancement of DFL, this paper presents the following contributions:
\begin{itemize}
\item The design and implementation of \textit{Fedstellar}, a comprehensive platform geared towards training FL models in a decentralized fashion (source code publicly available in \citep{MartinezBeltran:fedstellar_git:2023}). Fedstellar supports the establishment of federations comprising diverse devices, network topologies, and algorithms. It also provides sophisticated federation management tools and performance metrics to facilitate efficient learning process monitoring. This is achieved through extensible modules that offer data storage and asynchronous capabilities alongside efficient mechanisms for model training, communication, and comprehensive analysis for federation monitoring. The platform incorporates a modular architecture comprising three elements: a user-friendly frontend for experiment setup and monitoring, a controller for effective orchestration of operations, and a core component deployed in each device for model training and communication.
\item The deployment of Fedstellar on two distinct federated scenarios: \textit{(i)} a physical scenario composed of five Raspberry Pi 4 and three Rock64 boards affected by cyberattacks that need to be detected, and \textit{(ii)} a virtualized scenario deployed on a server where twenty nodes used the MNIST and CIFAR-10 datasets to perform image classification tasks in a decentralized manner. In the first scenario, a fully connected topology was employed with DFL and FedAvg, using an autoencoder as the federated model. In the second scenario, different FL architectures have been benchmarked within Fedstellar, including DFL, Semi-Decentralized Federated Learning (SDFL), and CFL, under a controlled and reproducible setting. Different network topologies such as fully connected, star, and ring were employed for this comparison.
\item The evaluation of the platform performance in the previous scenarios. For this purpose, various Key Performance Indicators (KPIs) were employed, including model $F_{1} \ score$, training time, communication overhead, and resource usage. In the physical scenario, Fedstellar achieved an $F_{1} \ score$ of 91\% in detecting cyberattacks using DFL with a fully connected topology. In the virtualized scenario, Fedstellar obtained an $F_{1} \ score$ of 98\% using DFL and 97.3\% using SDFL with the MNIST dataset \citep{Deng:MNIST:2012} while reducing the training time for model convergence by 32\% compared to centralized architectures. Furthermore, the effectiveness of the platform was emphasized by its adaptability to various network structures, including fully connected, star, and ring topologies.
\end{itemize}
The remainder of the paper is structured as follows. Section~\ref{sec:relatedwork} reviews the related work, discussing existing platforms and frameworks for DFL, SDFL, and CFL. After that, Section~\ref{sec:platform} describes the design and implementation of the Fedstellar platform. Section~\ref{sec:validation} validates the solution with a physical and virtualized deployment following a well-defined protocol. Subsequently, Section~\ref{sec:results} details the experiments and performance of the Fedstellar with different settings. Finally, Section \ref{sec:conclusion} presents some conclusions and future work.
\section{Related Work}\label{sec:relatedwork}
As the landscape of FL continuously evolves, numerous platforms and frameworks have emerged to adapt to diverse scenarios, addressing intricate challenges like decentralization, network topologies, scalability, and extensibility. \tablename~\ref{tab:compare-funcionality} provides the summary of some notable works, marking significant advances in the field.
\begin{table*}[ht!]
\caption{Comparison of functionality provided by literature solutions} \label{tab:compare-funcionality}
\resizebox{\textwidth}{!}{
\centering
\begin{threeparttable}
\begin{tabular}{c c c c c c c c c c}
\hline
\textbf{Reference} & \textbf{FL Architecture} & \textbf{Topology} & \textbf{Scalability} & \textbf{Extensibility} & \textbf{Libraries} & \textbf{Deployment Mode} & \textbf{Application Scenario} & \textbf{Metrics} & \textbf{Open Source} \\
\hline
TFF \citep{google:tff:2019} & CFL & Star & \maybetick & \maybetick & TensorFlow & Simulated & Smart grid systems & \makecell{Federated models\\Computation time} & \yestick \\
\hline
FATE \citep{fedai:fate:2021} & CFL & Star & \maybetick & \yestick & \makecell{TensorFlow, PyTorch,\\C++, Java for comms.} & Simulated & \makecell{Finance\\ Credit Scoring} & Federated models & \yestick \\
\hline
FedML \citep{He:fedml:2020} & CFL & Star & \yestick & \notick & TensorFlow, PyTorch & Hybrid & Mobile services & \makecell{Federated models\\Convergence rate} & \yestick \\
\hline
BrainTorrent \citep{Roy:braintorrent:2019} & DFL & Fully connected & \textit{N/S} & \textit{N/S} & \textit{N/S} & Simulated & Healthcare & \makecell{Federated models\\CPU usage} & \notick \\
\hline
Scatterbrained \citep{Wilt:scatterbrained:2021} & DFL & Fully connected & \notick & \yestick & \makecell{PyTorch,\\Synchronous sockets} & Simulated & Academic research & Federated models & \yestick \\
\hline
FL-SEC \citep{Qu:fl-sec:2022} & DFL & Fully connected & \notick & \maybetick & PyTorch and blockchain & Simulated & AIoT & \makecell{Federated models} & \notick \\
\hline
P4L \citep{Arapakis:P4L:2023} & DFL & Fully connected & \yestick & \maybetick & PyTorch with HE & Simulated & \textit{N/S} & \makecell{Federated models\\Privacy leakage level} & \notick \\
\hline
2DF-IDS \citep{Friha:2df-ids:2023} & DFL & \makecell{Random, Ring} & \maybetick & \maybetick & \makecell{Key exchange protocol\\DP gradient exchange} & Simulated & IIoT & \makecell{Federated models} & \notick \\
\hline
DEFEAT \citep{Lu:DEFEAT:2023} & \makecell{DFL\\SDFL} & Fully connected & \notick & \notick & \textit{N/S} & Simulated & Industry & \makecell{Federated models\\CPU usage} & \notick \\
\hline
DFedSN \citep{Chen:DFedSN:2023} & \makecell{DFL\\SDFL} & Fully connected & \maybetick & \yestick & \makecell{TensorFlow\\Affine Transformations} & Simulated & Social Networks & \makecell{Federated models} & \notick \\
\hline
\textbf{Fedstellar} (This work) & \textbf{\makecell{DFL\\SDFL\\CFL}} & \textbf{\makecell{Fully connected\\Star, Ring, Random,\\Custom}} & \yestick & \yestick & \textbf{\makecell{PyTorch\\Asynchronous sockets\\Docker containers}} & Hybrid & \textbf{Multiple} & \textbf{\makecell{Federated models\\Resources usage\\Communications}} & \yestick \\
\hline
\end{tabular}
\begin{tablenotes}
\item \yestick\space high,\space\maybetick\space moderate, \space\notick\space not addressed by the solution. \textit{N/S} (Not Specified) by the authors.
\end{tablenotes}
\end{threeparttable}}
\end{table*}
Starting from CFL architectures, TensorFlow Federated (TFF) served as the cornerstone in FL, offering a toolkit for federated computations \citep{google:tff:2019}. The TFF primarily aids server-based FL, a distinct characteristic of CFL, an approach highly beneficial for specific use cases such as smart grid systems adopting a star topology. In contrast, Federated AI Technology Enabler (FATE) successfully connected and integrated elements of CFL and distributed computation in centralized networks, a feature typically associated with DFL \citep{fedai:fate:2021}. However, FATE enables server-based FL by integrating communications optimization and secure computation. Finally, FedML emerged as a prominent FL platform oriented to academic purposes, offering support for various architectures and providing APIs for researchers \citep{He:fedml:2020}. It not only offers flexibility and scalability but also emphasizes maintaining network connections, a feature crucial for IoT devices where network access may occasionally be unreliable. However, none of these solutions fully support DFL, highlighting their struggle to adapt to diverse network configurations beyond the star topology.
After these CFL solutions, DFL marked its emergence with BrainTorrent, a pioneering solution in healthcare applications \citep{Roy:braintorrent:2019}. It functions in a Peer-to-Peer (P2P) setting, facilitating direct interaction between entities without a central coordinator. It is a valuable strategy when establishing a central trusted entity is challenging or a central server failure could halt the training process. BrainTorrent has proven successful, specifically for whole-brain segmentation of Magnetic Resonance Imaging (MRI) scans, presenting performance similar to traditional server-based FL and models trained on pooled data. However, the scope of BrainTorrent remains restricted, specifically tuned for healthcare applications, and it lacks the flexibility to adapt to various use cases and network topologies. Extending this P2P approach, the Scatterbrained framework expands the scope of DFL \citep{Wilt:scatterbrained:2021}. The framework separates the ML model from the FL business logic, reducing complicated tasks for developers and allowing the use of various learning tools. Its Python library simplifies the development process, providing an easy-to-use API to customize the parameter-sharing behavior. Although it provides substantial support for academic research, the primary focus on this area may restrict its practical applicability in real-world IoT environments.
In an increasingly privacy-conscious world, FL-SEC, a private and DFL architecture that excels in the Artificial Intelligence of Things (AIoT), has made significant advances \citep{Qu:fl-sec:2022}. It employs a unique blockchain structure to achieve decentralization, eliminates single points of failure, and protects against poisoning attacks. It also has a personalized incentive mechanism that encourages participation and reduces communications in many scenarios, like smart homes with fully connected topologies. Despite its advancements, the reliance of FL-SEC on blockchain technology may introduce complexity and scalability issues that Fedstellar mitigates with its scalable design. Further enhancing privacy is P4L, a privacy-preserving P2P learning system that uses multiple heterogeneous devices \citep{Arapakis:P4L:2023}. P4L utilizes partial Homomorphic Encryption (HE) to uphold the confidentiality of shared gradients, thereby circumventing information leakage attacks. Notably, the system operates without a centralized federation, Public Key Infrastructure (PKI), or internet connection, offering a cost-effective, privacy-preserving alternative to CFL architectures. Its performance has been evaluated through analytical simulations using CIFAR-10, Avito, and IMDB datasets under a fully connected topology, demonstrating that P4L can deliver federated models with competitive performance. However, it lacks the flexibility to incorporate new datasets easily and is restricted to a specific network topology. Furthermore, it does not provide sufficient metrics for monitoring the performance and progress of the learning process, which could be a limitation when used in more complex, real-world scenarios compared to Fedstellar.
Recognizing the need for security in smart industrial facilities, 2DF-IDS introduces a secure, decentralized, and Differentially Private (DP) FL-based Intrusion Detection System (IDS) \citep{Friha:2df-ids:2023}. It showcases its strength in smart industrial facilities utilizing random and ring topologies. It comprises three elements: a key exchange protocol, a DP gradient exchange scheme, and a decentralized FL approach. Its effectiveness in identifying various cyberattacks on an Industrial Internet of Things (IIoT) system using different datasets demonstrates its impressive capabilities. However, the solution has limitations, such as a lack of flexibility due to the limited customization of parameters such as aggregation algorithms, models, datasets, or network topologies. In addition, it does not facilitate actual deployments on physical devices. The DEFEAT framework, in its quest to defend against gradient attacks, uses a P2P network to transmit model parameters, enabling multiple connected clients to train a personalized model jointly \citep{Lu:DEFEAT:2023}. It balances communication costs and model training accuracy by using SDFL approaches with different schemes regarding message exchange. Evaluations on real datasets showed that DEFEAT performs better than other solutions in model training accuracy and effectively mitigates gradient attacks while maintaining high accuracy in the resulting models. As for the architecture of the solution, it does not present a modular structure and, therefore, lacks interdependence between the features of the solution. This limits the customization potential of the solution. In addition, the communications presented by the participants do not guarantee an asynchronous exchange of messages during federation, unlike Fedstellar, which keeps it implicit and transparent to the device.
Lastly, the DFedSN framework comes forth as an innovative solution specifically adapted for different data types in social networks \citep{Chen:DFedSN:2023}. This framework adopts SDFL with a strategy of establishing data differences via affine transformation. It considers that all data users transmit as a unified image collection, fostering shared image classifications. This approach ensures that large data differences between users will not lead to performance problems during the federation. Tested on TensorFlow with two standard image datasets (MNIST and CIFAR-10) and implemented on three standard network architectures (AlexNet, Inception-Net, and Mini-ResNet) and a fully connected topology, DFedSN obtained results close to 95\% accuracy in the federated models, even with large differences in the distribution of user data.
In conclusion, while the solutions discussed herein contribute significantly to FL, they also present noteworthy limitations. These range from a lack of modular architecture and inadequate support for asynchronous message exchange during federation to limited customization and flexibility, such as a lack of support for different aggregation algorithms, models, datasets, or network topologies. Additionally, many of these DFL solutions are not open source, limiting their accessibility and adaptability. Furthermore, several solutions are restricted in scope to specific applications or fail to perform deployments on physical devices. Another significant limitation lies in the inability of many solutions to monitor real-time metrics, such as network traffic, which is crucial in decentralized scenarios. While these solutions have undoubtedly advanced the field, there is still a need for novel platforms covering these important limitations.
\section{Fedstellar Platform}\label{sec:platform}
Fedstellar is an innovative platform that facilitates the training of FL models in a decentralized fashion across many physical and virtualized devices. In particular, Fedstellar offers the following key features:
\begin{itemize}
\item Creation and management of federations comprising diverse types and numbers of real and/or virtualized devices.
\item Generation and deployment of complex network topologies to link within a federation, including DFL, SDFL, and CFL, each catering to specific roles of the participants.
\item Usage of different ML/DL models and datasets to tackle heterogeneous FL-based problems.
\item Incorporation of user-friendly features that enable flexible customization and efficient monitoring of training processes.
\item Monitoring of KPIs across multiple levels, including:
\begin{itemize}
\item \textit{Resource KPIs.} Monitoring of computational load, memory usage, and local model accuracy.
\item \textit{Communication KPIs.} Focus on communication latency, data transmission speed, and successful data transfer rate.
\item \textit{Federated model KPIs.} Assessment of model accuracy, precision, recall, and $F_{1} \ score$, training time, and convergence speed of each local model.
\end{itemize}
\end{itemize}
With these extensive features and integrated KPIs, Fedstellar effectively manages various learning scenarios and facilitates continuous performance assessment and optimization.
\subsection{Overall Architecture}
The architecture of Fedstellar unifies numerous functional components to facilitate the deployment of different FL architectures across physical and virtual devices through an intricate design encompassing advanced data handling, efficient training mechanisms, and communication protocols. The choice of Python as the underlying programming language is fundamental to the design, highlighting its capabilities in supporting diverse network topologies, managing efficient communication, and offering a rich ecosystem of ML and data processing libraries. As shown in Figure~\ref{fig:overview}, the architecture highlights four main components:
\begin{itemize}
\item \textit{User.} It has the responsibility of managing the platform operation. Through the intuitive interface of the frontend, users can establish, configure, and monitor federated scenarios, tailoring the system to the requirements of the learning process.
\item \textit{Frontend.} This interactive interface is where the user designs and oversees learning scenarios. Its user-focused design simplifies system configuration and supports ongoing KPIs performance tracking.
\item \textit{Controller.} The controller serves as the orchestration center of the platform. It interprets users' commands from the frontend, manages the entire federated scenario, assigns learning algorithms and datasets, and configures network topologies to ensure an efficient and effective FL process.
\item \textit{Core.} It is deployed on each device of the federation and is responsible for the execution of FL tasks. It manages model training, data preprocessing, secure communication among devices, and storage of the federated models. Additionally, the core supervises the calculation of KPIs and conveys this information back to the frontend for performance monitoring.
\end{itemize}
\begin{figure}[!htb]
\includegraphics[width=\columnwidth]{overview.pdf}
\centering
\caption{Components of the Fedstellar platform: user, frontend, controller, and core.}
\label{fig:overview}
\end{figure}
The various components of Fedstellar are meticulously intertwined, fostering a customizable and dynamic federated scenario. As shown in \figurename~\ref{fig:sequence-diagram}, the deployment process commences with the configuration of the scenario. Upon receiving the configuration from the frontend, the controller generates the network topology and establishes core instances for physical and virtualized devices. Then, it securely transfers the previously defined configuration to each device. This ensures a consistent and secure provision across all devices. The process culminates with the execution of the FL operation while monitoring and processing critical metrics that furnish insights into the performance and efficiency of the scenario.
\begin{figure}[!htb]
\includegraphics[width=\columnwidth]{sequence_diagram.pdf}
\centering
\caption{Sequence diagram showing the interaction of Fedstellar components}
\label{fig:sequence-diagram}
\end{figure}
\subsection{Fedstellar Frontend}
This component is designed to be user-friendly and intuitive, serving as the user's primary point of interaction with Fedstellar. As illustrated in the upper level of \figurename~\ref{fig:implementation}, users can define federated scenarios according to their needs, monitor their progress, and track the physical locations of devices within the topology.
\begin{figure*}[!htb]
\includegraphics[width=\textwidth]{implementation.pdf}
\centering
\caption{Overall architecture of the Fedstellar platform}
\label{fig:implementation}
\end{figure*}
\subsubsection{Scenario Definition}
\label{sec:scenario-definition}
A scenario within Fedstellar represents a unique execution of an FL experiment, distinguished by specific parameters, computational resource allocation, network topology, and targeted applications or devices. Defining a scenario involves setting parameters, extending beyond the basic naming and description for reference purposes. For instance, users can specify the deployment mode (physical or virtualized devices) and choose the federation architecture (DFL, SDFL, or CFL). Additionally, a key feature allows users to graphically design and interconnect nodes, which the system then converts into the corresponding adjacency matrix for managing device communication during the FL process. This feature is powered by D3.js, a JavaScript library known for its robust data-driven document manipulation and visualization capabilities \citep{Bostock:d3js:2021}.
In an advanced configuration view, users can define the specific dataset to be utilized in the learning process, catering to various tasks, including image recognition, natural language processing, or predictive analytics. Users also have the flexibility to select the learning model, from traditional ML models to advanced DL models. Fedstellar employs a dynamic form creation system using a form generator based on JSON Schema to accommodate these diverse and intricate user inputs, ensuring adaptability and resilience across various FL scenarios.
\subsubsection{Real-time Monitoring}
The real-time monitoring functionality within Fedstellar plays a critical role, allowing users to track the performance and progress of ongoing federated scenarios. This feature equips users with up-to-the-minute updates on a range of metrics generated by each device within the network topology (see Section~\ref{sec:monitoring}). By offering this real-time insight, users can make timely, informed decisions, intervene when needed, and dynamically gauge the effectiveness of the experiment.
An important aspect of this capability is its customized implementation of TensorBoard, a suite extensively used for visualizing ML experiments \citep{google:tensorboard:2019}. In this sense, the suite has been modified to display individual device metrics and handle simultaneous updates from multiple devices. Additionally, users can export the generated data in universally compatible formats, such as CSV or JSON. This functionality enables seamless data integration with various data analysis and visualization tools, granting users the opportunity for more detailed exploration and understanding of the performance of their scenarios.
\subsubsection{Geolocation}
This functionality elevates the scope of experimentation and offers insightful context about the devices engaged in a federated scenario. This dynamic geolocation capability is further enhanced using Leaflet, an open-source JavaScript library known for its user-friendly interactive maps \citep{Agafonkin:leaflet:2021}. It enables the visualization of the distribution of the nodes and provides essential information such as each IP address and the cluster to which it belongs. This function maps out the precise geographical locations of physical devices, an essential feature in various analyses and applications where device positioning significantly impacts the outcome. This is particularly relevant for scenarios involving mobility or location-based factors, such as mobile networks or vehicle-to-vehicle communication, which are inherent to the study.
When using virtualized devices, the platform permits users to manually assign specific coordinates to each one, thereby simulating virtual device geolocation. This feature introduces the ability to emulate node mobility dynamically, a critical aspect for certain applications where functionality may depend on factors like the distance between nodes or the ability to adapt to network topology changes in real time. For example, this could be used to explore the performance of FL algorithms in mobile networks or other distributed systems where nodes are not static.
%%%%%%%%%%%%%%%%%%%%%
\subsection{Fedstellar Controller}
As an integral component of Fedstellar, the controller serves as a bridge, enabling seamless interaction between the frontend and deploying core modules in physical or virtualized devices. Corresponding to the second level in \figurename~\ref{fig:implementation}, it includes the deployment and resource provision, network topology, federation architecture, and federated functionality, each playing a vital role in orchestrating the FL process.
\subsubsection{Deployment and Resource Provision}
This functionality adeptly blends state-of-the-art device management and virtualization technologies to accommodate both physical devices and virtual settings (see \figurename~\ref{fig:backend-deployment}). Beginning with physical devices, Fedstellar harnesses the powerful functionalities of Mender, a renowned open-source software purpose-built updater for embedded Linux devices \citep{Northern:mender:2021}. This strategic integration sets the way for a robust, efficient, and fail-safe software deployment process, smoothing the progression from a theoretical, experimental setup to a tangible FL application. Focusing on virtualized scenarios, Fedstellar taps into the capabilities of Docker containers. These containers are renowned for their innate potential for isolation, portability, and maintaining application consistency. They enable the encapsulation of applications and their dependencies into standardized units, ensuring a streamlined, replicable deployment process. In addition, Docker-level virtualization enables Fedstellar to deploy virtualized devices on a wide range of operating systems, including Windows, Linux, and MacOS.
\begin{figure}[!htb]
\includegraphics[width=\columnwidth]{backend-deploy.pdf}
\centering
\caption{Types of deployment of Fedstellar: virtualized and physical scenarios}
\label{fig:backend-deployment}
\end{figure}
Following the scenario definition, Fedstellar brings into play its resource provision management functionality. This feature is instrumental in the reasonable allocation and regulation of computational resources for each participant, regardless they are physical or virtual. Each participant is presented with a custom YAML configuration file to aid this, enabling them to tweak their internal operations individually. Additionally, Fedstellar employs REST APIs, thus enabling users to designate CPU, GPU, and RAM usage caps. This feature gives users unprecedented control over resource management, making Fedstellar a versatile tool in DFL.
\subsubsection{Network Topology}
The creation of network topologies commences through a well-structured user-oriented process. As commented in Section~\ref{sec:scenario-definition}, users define the initial topology within the intuitive frontend interface. In order to facilitate this process, the user can select preestablished topologies including, but not limited to, fully connected, partially connected (star, ring, random), or clustering configurations. Each selected topology corresponds to a specific configuration, and its selection activates a particular set of inter-device communication protocols. Once selected, Fedstellar creates a virtual internal network mirroring the user's selection, assigning distinct sockets to each participating device to enable seamless collaboration.
Fedstellar is designed to detect and integrate new connections into its platform, accommodating the dynamic nature of federated networks. Devices can join the existing federation by connecting to one or more devices already part of the network, thereby expanding the scope and capabilities of the scenario. Additionally, Fedstellar ensures robust resilience in the face of possible network disruptions. A key feature of real-world FL scenarios is the unpredictable availability of devices, with nodes entering, exiting, or occasionally disconnecting from the network.
\subsubsection{Federation Architecture}
Concerning the federation architecture, users can select among DFL, SDFL, or CFL, as illustrated in \figurename~\ref{fig:federation-architectures}. DFL facilitates a distributed environment where devices learn collectively, avoiding a single point of failure and ensuring scalability, thereby making it suitable for extensive networks. SDFL represents a hybrid model that integrates a central server with decentralized communications. In this architecture, one or more rotating aggregators play a leadership role, passing the aggregation functionality within the topology after each participant's internal round. In contrast, CFL employs a central server for model aggregation, providing an efficient and simplified structure for less intricate network structures.
\begin{figure}[!htb]
\includegraphics[width=\columnwidth]{fedarch.pdf}
\centering
\caption{Federation architectures implemented in Fedstellar: DFL, SDFL, and CFL}
\label{fig:federation-architectures}
\end{figure}
In terms of implementation, Fedstellar operates by associating a specific functionality to each participant based on the selected federation architecture during the deployment phase. For a DFL scenario, each participant is programmed with an independent thread for aggregating model parameters. Additionally, Fedstellar automatically generates leadership transfer messages using a custom-built Python class in SDFL. This class keeps track of the current leader and uses socket programming features to send leadership transfer messages to the next designated aggregator. In contrast, in CFL, each participant is assigned to a unique set of thread lockers. These thread lockers, managed by the Python threading library, control the synchronization between the central server and participant devices, ensuring data integrity and orderliness in communications.
\subsubsection{Federated Functionality}
The setup process in Fedstellar commences with the user configuring various parameters from the frontend. This process entails selecting a suitable model structure from a comprehensive collection of ML/DL models, including architectures such as Deep Neural Network (DNN) and Convolutional Neural Network (CNN). These models are supported by well-established libraries such as PyTorch and Scikit-learn, granting users access to a broad array of algorithms.
Subsequently, the training dataset is defined, with Fedstellar offering users the flexibility to utilize standardized benchmark datasets, such as MNIST or CIFAR-10, and also import custom ones. Additionally, users can dictate the data distribution strategy, choosing between Independent and Identically Distributed (IID) or Non-IID options. This choice has significant repercussions on partitioning the dataset among the devices. To ensure effective data distribution, the platform first demarcates distinct portions of the dataset for each device by the selected distribution configuration. Subsequently, devices are directed to the dataset location, enabling them to load their designated portion based on pre-defined indices.
Once these configurations are in place, Fedstellar initiates its federated functions. It recognizes three primary roles in the federated context: trainers, aggregators, and proxies. Trainers are responsible for local model training, aggregators for merging model updates, and proxies for collecting updates from trainers, alleviating the communication load on aggregators. Furthermore, users can select from an array of aggregation algorithms. The options span from Federated Averaging (FedAvg) to more specialized alternatives such as Krum, Trimmed Mean, and Median.
\subsection{Fedstellar Core}
This core is a fundamental component of the Fedstellar platform, deployed by the controller component into physical or virtualized devices. This element is responsible for managing most of the computation and communication required for the successful operation of FL. It is subdivided into several components: (i) the monitoring module for establishing the operational environment, (ii) the training module for handling the learning process, and (iii) the communications module for facilitating interactions between federated participants and with the frontend.
\subsubsection{Monitoring}
\label{sec:monitoring}
The monitoring capability of Fedstellar embodies a crucial component, facilitating systematic and periodic data collection about the ongoing federated scenario. This data collection process is executed every five seconds by default but can be customized to align with specific experimental requirements. With the aid of auxiliary libraries such as \textit{psutil} and \textit{pynvml}, Fedstellar proficiently gathers system-level and GPU-related metrics, respectively. The platform gathers metrics categorized into three primary groups:
\begin{itemize}
\item \textit{Federated model.} This category includes metrics related to the model size, learning progress (encompassing loss, accuracy, precision, recall, and $F_{1}$ score), model synchronization frequency, and the number of training rounds.
\item \textit{Resources usage.} It comprises parameters such as CPU usage, GPU usage, memory utilization, network bytes, and component temperatures, providing insights into the computational and storage resources consumed throughout the scenario.
\item \textit{Communications status.} This category captures metrics related to communication operations, such as status updates, latency, active connections, and data transmission rates, illuminating network performance during federation.
\end{itemize}
In addition to the aforementioned aspects, Fedstellar takes responsibility for creating and managing log files generated during the execution of the FL process. These logs are a vital asset for potential troubleshooting, offering detailed insights into the operational attributes of the system. Additionally, they support the replication of scenarios and validation of results, reinforcing the scientific credibility of the conducted scenarios.
\subsubsection{Training}
This module is responsible for data management and model training within FL scenarios. It utilizes a comprehensively organized directory, housing files integral to the execution of the specific federated scenario. Each of these files plays a critical role in outlining the dataset parameters, determining whether the data source is local or remote, orchestrating data transformations, and dictating the operation of the data loader. To further refine the data quality, preprocessing techniques like normalization, feature extraction, and data augmentation are deployed. These procedures play a pivotal role in processing raw data into a quality resource ready for model training, also providing a template to accommodate future data processing techniques.
Regarding model training, Fedstellar fosters a highly adaptable and expandable environment that eases the definition and modification of learning models. By leveraging Lightning with PyTorch as a training backend, the platform simplifies complex ML logistics, accommodating diverse learning requirements corresponding to various datasets \citep{Falcon:PyTorch-Lightning:2019}. Regarding aggregation algorithms, Fedstellar offers a potent suite of options, providing an extensible design to integrate additional aggregation algorithms, whether the user can define or import from external libraries. This flexibility sets a favorable stage for those engaged in research or testing innovative or custom aggregation strategies, thus supporting scientific exploration within the FL landscape.
\subsubsection{Communications}
This module primarily communicates and transmits necessary information within the network, including the frontend for status updates and the neighboring nodes for the asynchronous exchange of model parameters. \tablename~\ref{tab:comms_messages} outlines the messages deployed in Fedstellar, providing a detailed account of the communication process. Following this exposition, as depicted in \figurename~\ref{fig:steps}, the communication process has four steps: (i) connecting all nodes with their neighbors; (ii) initiating the federation setup; (iii) executing the training process with decentralized aggregation; and (iv) monitoring and alerting.
\begin{table}[ht!]
\caption{Communication messages implemented in Fedstellar}
\label{tab:comms_messages}
\resizebox{\columnwidth}{!}{
\centering
\begin{tabular}{l l}
\hline
Message & Description \\
\hline
\multicolumn{2}{c}{\textit{Messages with forwarding}}
\\ \hline
START\_LEARNING & Initiates the FL process \\
STOP\_LEARNING & Halts the ongoing FL process \\
PARAMS & Exchange of model parameters among participants \\
STOP & Signals termination of the current federation scenario \\
MODELS\_READY & Indicates readiness of locally trained models \\
MODELS\_AGGREGATED & Informs about the completion of the aggregation \\
\hline
\multicolumn{2}{c}{\textit{Messages without forwarding}}
\\ \hline
CONNECT\_TO & Requests a node to establish a connection with another \\
BEAT & Validates the connection and liveliness of a node \\
ROLE & Assigns a role (trainer, aggregator, or proxy) to a node \\
METRICS & Transfers collected monitoring data for analysis \\
LEADERSHIP & Transfers the aggregation functionality in SDFL \\
\hline
\end{tabular}}
\end{table}
\begin{figure}[!htb]
\includegraphics[width=\columnwidth]{steps.pdf}
\centering
\caption{Schematic representation of the communication process in Fedstellar}
\label{fig:steps}
\end{figure}
During the initial step, each device within the federation creates a communication link with its immediate neighbors, storing these connections through open sockets. This step ensures the efficient and prompt exchange of information throughout the federation. Significantly, these communications are executed atop an abstract layer, offering the versatility to employ other technologies for data exchange. In the second step, the federation initiation commences with a designated node broadcasting a message outlining the definition, including the federation rounds, which refer to the cyclical process of local model training and model update sharing within the federation. On receiving this message, nodes tailor their operations to align with their assigned roles and responsibilities within the federation. While the current design involves all available nodes, future enhancements could fine-tune node inclusion based on available resources or network characteristics, including latency and delay.
The third step of the process is the key to the FL operation: model training, decentralized aggregation, and asynchronous exchange of model parameters. In this step, each node employs its respective data to train local models independently. An implementation leveraging a thread-based system ensures the transparent, asynchronous receipt, forwarding, and sending of messages between nodes. This system employs incoming and outgoing message queues to maintain a structured flow of communication, storing and processing incoming and outgoing messages in an orderly manner. It facilitates the decentralized aggregation approach whereby the trained model parameters are asynchronously exchanged among neighboring nodes. The platform integrates an intuitive mechanism to signal when a device should perform aggregation. This action is triggered when all neighboring nodes have transmitted their parameters or after a predetermined timeout period of five minutes commences at the onset of the process. This ensures a smooth and timely workflow. The types of messages employed in this process are outlined in \tablename~\ref{tab:comms_messages}.
The concluding step of the process concerns monitoring and alerting. The federated participants actively assess and report on any disruptions, such as link failures or devices leaving the topology. In this way, nodes can remove their affected neighboring nodes from their internal queue of open sockets. Furthermore, the nodes maintain unidirectional connections with the frontend through secure WebSockets, facilitating the continuous transmission of metric data and ensuring constant federation monitoring. Moreover, Algorithm~\ref{alg:cycle} provides a comprehensive exposition of the internal cycle of each participant, aligning with the steps mentioned above.
\begin{algorithm}[ht!]
\caption{Federated participant cycle in Fedstellar}
\label{alg:cycle}
\footnotesize
\begin{algorithmic}[1]
\Require{$R$: local round, $\alpha$: learning rate, $\lambda$: regularization parameter, $S_j$: socket to neighbor j, $D$: local dataset}
\State $D_{\text{Train}}, D_{\text{Test}} \gets split(D)$
\For{$r$ in $R$}
\State $\text{Initialize Local Model with Parameters } \theta$
\For{each $(x, y)$ in $D_{\text{Train}}$}
\State $\theta \gets \theta - \alpha (\nabla_\theta J(\theta, x, y) + \lambda \theta)$ \Comment{\textbf{Train}}
\EndFor
\For{$j$ in $N$} \Comment{\textbf{Send}}
\State $\text{Model Parameters to } j \text{ via } S_j$
\EndFor
\While{$not \; Timeout$}
\For{$j$ in $N$} \Comment{\textbf{Receive}}
\State $RP_j \gets \text{Model Parameters from } j
\text{ via } S_j$
\EndFor
\EndWhile
\State $\theta \gets \frac{1}{|N|+1} (\theta + \sum_{j \in N} RP_j)$ \Comment{\textbf{Aggregate} (FedAvg)}
\State $\text{Update Local Model with } \theta$
\EndFor
\For{each $(x, y)$ in $D_{\text{\textbf{Test}}}$}
\State $y_{pred} \gets \text{Predict with Local Model on } x$ \Comment{\textbf{Test}}
\State $L \gets \frac{1}{|D_{\text{Test}}|}\sum_{i=1}^{|D_{\text{Test}}|} l(y, y_{pred})$ \Comment{\textbf{Compute Loss}}
\EndFor
\State $\text{Send metrics to frontend}$ \Comment{\textbf{Report Metrics}}
\end{algorithmic}
\end{algorithm}
\section{Validation Scenarios}
\label{sec:validation}
The Fedstellar platform has been deployed on two federated scenarios: a physical scenario, which showcases the real-world applicability of Fedstellar, highlighting its performance in a real scenario with tangible resource-constrained devices such as Raspberry Pis; and a virtualized scenario, which illustrates the scalability of the platform and its ability to handle a wide variety of configurations. \tablename~\ref{tab:validation-scenarios} compares the characteristics of both scenarios.
\begin{table}[ht]
\caption{Validation scenarios of Fedstellar using physical and virtualized devices}
\label{tab:validation-scenarios}
\resizebox{\columnwidth}{!}{
\centering
\begin{tabular}{c c c}
\hline
\textbf{Characteristic} & \textbf{Physical Scenario} & \textbf{Virtualized Scenario} \\
\hline
\makecell{Participant\\Characteristics} & 8 (5 Raspberry Pi 4 / 3 Rock64) & 20 (docker containers) \\
\hline
Dataset & Syscalls \citep{Huertas:syscall-dataset:2023} & \makecell{MNIST \citep{Deng:MNIST:2012} /\\CIFAR-10 \citep{krizhevsky:cifar10-100:2009}} \\
\hline
\makecell{Federated\\Model} & Autoencoder & LeNet5 / MobileNet \\
\hline
\makecell{Network\\Topology} & Fully connected & \makecell{Fully connected\\Star, Ring} \\
\hline
\makecell{Federation\\Architecture} & DFL & DFL, SDFL, CFL \\
\hline
\end{tabular}}
\end{table}
\subsection{Overview}
The physical scenario leveraged the IoT spectrum sensors from ElectroSense, an open-source, real-world crowdsensing platform \citep{Rajendran:electrosense:2018}. These low-cost sensors were employed to collect data from the radio frequency spectrum. Each sensor was calibrated to monitor ``normal'' behavior. Further manipulations of the source code led to the execution of eight types of Spectrum Sensing Data Falsification (SSDF) attacks per sensor: Delay, Confusion, Freeze, Hop, Mimic, Noise, Repeat, and Spoof \citep{Huertas:ssdf-attacks:2023}. These attacks were designed to alter the spectral data observed by the sensors, allowing for the detection and identification of malicious behavior.
In contrast, the virtualized scenario was focused on image recognition. Given the immense volume of unlabeled image data ubiquitously present on devices, this serves as a significant scenario for deploying FL capabilities. The widely-used MNIST and CIFAR-10 datasets served as resources for this scenario, enabling the validation of the Fedstellar platform in a frequently explored academic and industry context. This emphasizes the application of FL in enhancing the accuracy and efficiency of image classification tasks, a significant step forward in developing AI technologies.
\subsection{Participant Characteristics}
In the physical scenario, the federation comprised eight devices, five of which were Raspberry Pi 4 boards and the remaining three were Rock64 boards, all in the same private local network. The Raspberry Pi 4 devices, equipped with a 1.5GHz quad-core 64-bit ARM Cortex-A72 CPU and 2GB of RAM, presented an optimal combination of size, affordability, and processing power. Alternatively, the Rock64 boards enhanced the heterogeneity by offering a slightly inferior processing capacity, characterized by a 64-bit ARM Cortex-A53 with a 1.5 GHz clock speed and up to 2GB RAM.
In contrast, the virtualized scenario constituted twenty virtual devices as Docker containers. These containers were deployed on a host machine with an Intel Core i7-10700F processor that operates at a base frequency of 2.9 GHz (up to 4.8 GHz), 128GB RAM, 2TB storage, and dual NVIDIA RTX 3080 GPUs. The allocation of GPUs to virtual participants was achieved through a random selection scheme, where each GPU served ten participants.
\subsection{Dataset and Federated Model}
In the physical scenario, a dataset containing syscalls monitored under SSDF attacks was employed \citep{Huertas:syscall-dataset:2023}. Syscalls are a potent source of data for detecting various types of cyberattacks due to their ability to capture the system internal actions and process anomalies \citep{Sanchez:fingerprinting:2021}. The dataset contains syscalls from the spectrum collection service and its processes over 60-second intervals. The federated model employed was an autoencoder configured with a DNN architecture and trained via the FedAvg aggregation algorithm. The autoencoder, comprised of 64, 16, 8, 16, and 64 neurons across its hidden layers, used the ReLU activation function for non-linear transformation. The model learning defined an anomaly recognition threshold set at the $95^{th}$ percentile of the reconstruction error of the training dataset. Thus, any deviation beyond this point is flagged as a potential threat.
For the virtualized scenario, Fedstellar was validated against the MNIST and CIFAR-10 datasets, both being well-established for benchmarking. The MNIST dataset, a robust collection of handwritten digits, is frequently used for training and validating models within the ML domain \citep{Deng:MNIST:2012}. Similarly, the CIFAR-10 dataset comprises images distributed across ten distinct classes with a more complex classification task \citep{krizhevsky:cifar10-100:2009}. To address the specific needs of these datasets, Fedstellar employed two different models: a LeNet5 model for the MNIST dataset \citep{Lecun:lenet5:1998}, and a MobileNet model for the CIFAR-10 dataset \citep{Howard:mobilenet:2017}, both lightweight yet powerful models. LeNet5, a pioneering Convolutional Neural Network (CNN), consists of several layers that recognize image patterns. MobileNet is a lightweight model for mobile and embedded vision applications. The aggregation of the federated models was executed using the FedAvg algorithm, similar to the physical scenario. Notably, the data used in physical and virtualized scenarios were Non-IID, reflecting real-world conditions where data are often unevenly distributed across devices. Additionally, the participants employed ten local rounds where they executed the federation process iteratively (see Algorithm~\ref{alg:cycle}), and the local models underwent twenty epochs during training.
\subsection{Network Topology}
A fully connected topology served as the network backbone in the physical scenario. This choice motivates the complete intercommunication capability, granting each device in the federation direct access to all others. This configuration ensures optimal information flow, a critical requirement for the implementation of DFL. The interconnectivity of this network structure guarantees an optimal choice for scenarios where the federation comprises a moderate number of devices.
The virtualized scenario, in contrast, leveraged multiple topologies: fully connected, star, and ring. This decision highlights the broad compatibility with various network configurations, comparing the adaptability to varying federated contexts. The fully connected topology ensured comprehensive information flow and quick consensus achievement in federations of modest size. Nonetheless, the star topology demonstrated how Fedstellar could respond to situations requiring a centralized node for information routing. Meanwhile, the ring topology showcased the ability to operate efficiently in larger federations, where partial connectivity can minimize communication overhead and enhance scalability.
\subsection{Federation Architecture}
The DFL architecture was selected for the physical scenario due to its resilience and robustness. The DFL approach facilitates decentralization, eliminating potential bottlenecks and vulnerabilities tied to a central server. Its adoption offers valuable insights for future real-world implementations, particularly where intermittent connectivity and device failures are prevalent.
Conversely, the virtual scenario exploited the benefits of all three federation architectures: DFL, SDFL, and CFL. The variety was chosen to test the proficiency of Fedstellar under various conditions. SDFL allows Fedstellar to demonstrate its ability to balance decentralization and strategic centralization for optimal learning outcomes. The inclusion of CFL, on the other hand, underscores the ability of the platform to handle scenarios that demand a fully centralized structure, emphasizing its versatility.
\section{Results}\label{sec:results}
The evaluation of Fedstellar performance, encompassing both physical and virtualized deployments, focused on key indicators such as the $F_{1} \ score$ for federated models, the percentage of CPU and RAM usage, network traffic in megabytes $(\text{MB})$, and model convergence time. These measurements provide comprehensive insights across varying federated architectures, topologies, datasets, and ML/DL models.
\subsection{Physical Scenario}
Delving into the details of the physical deployment, \figurename~\ref{fig:results_physical} illustrates the outcomes of Fedstellar running on eight devices, each labeled as $d=\text{\{1-8\}}$. These devices, encompassing five Raspberry Pi 4 units (1-5) and three Rock64 units (6-8), operate in an intricately orchestrated network, showcasing the full capability of Fedstellar in real-world scenarios.
\begin{figure*}[htb!]
\centering
\begin{subfigure}{.25\textwidth}
\centering
\includegraphics[width=\linewidth]{f1_physical.pdf}
\caption{Federated models $(F_{1} \ score)$}
\label{fig:results_physical_f1}
\end{subfigure}%
\begin{subfigure}{.25\textwidth}
\centering
\includegraphics[width=\linewidth]{cpu_physical.pdf}
\caption{CPU usage $(\%)$}
\label{fig:results_physical_cpu}
\end{subfigure}%
\begin{subfigure}{.25\textwidth}
\centering
\includegraphics[width=\linewidth]{ram_physical.pdf}
\caption{RAM usage $(\%)$}
\label{fig:results_physical_ram}
\end{subfigure}%
\begin{subfigure}{.25\textwidth}
\centering
\includegraphics[width=\linewidth]{net_physical.pdf}
\caption{Network usage $(\text{MB})$}
\label{fig:results_physical_net}
\end{subfigure}
\caption{Performance of Fedstellar in a physical deployment with eight devices using a syscall dataset during 60 minutes}
\label{fig:results_physical}
\end{figure*}
\figurename~\ref{fig:results_physical_f1} shows that within 25 minutes of training, the $F_{1} \ score$ rises consistently to an acceptable 85 \%. This denotes an efficient learning process triggered by the decentralization of model parameters. Beyond the 25-minute mark, the $F_{1} \ score$ continues to rise gradually, reaching a pinnacle of 91 \% at the 60-minute mark. Even though the eight devices used for the experiment have different hardware capabilities, they demonstrate similar performance patterns. Notwithstanding the asynchronous nature of the DFL process, this uniformity of performance demonstrates the platform adaptability. However, this asynchronicity also led to subtle variances, with some devices ending the 60-minute experimentation period with slightly lower $F_{1} \ score$ than others.
Moreover, \figurename~\ref{fig:results_physical_cpu} reveals the patterns of CPU usage, specifically reflecting the highest utilized core across the eight devices involved. The patterns seem to oscillate consistently, with cycles of CPU usage increases lasting 5-10 minutes, during which the usage reaches a mean of 80 \%, with an occasional peak of up to 90.7 \%. Each of these surges is then followed by a drop, reducing the CPU usage to roughly 20-30 \%. The correlation of these trends with the computational cycle outlined in \figurename~\ref{fig:steps} and Algorithm~\ref{alg:cycle} is remarkable. More in detail, intense CPU utilization coincides with periods of model training, and a notable reduction happens as the trained models undergo parameter exchanges with neighboring nodes.
Moving into RAM usage, \figurename~\ref{fig:results_physical_ram} demonstrates a more irregular pattern. The inconsistency likely stems from the variation in data volume derived from syscalls in the system and, thus, the dataset creation. On average, there is a 20 \% utilization of RAM, with sporadic spikes that hover between 30 \% and 31.5 \%. It is plausible to relate the RAM usage to the data handling demands of the DFL process, where each device processes its local dataset, adding to the memory footprint. Finally, \figurename~\ref{fig:results_physical_net} illustrates the network usage. An increase in data exchanged by all devices is observed, culminating in a staggering 1190 MB at the end of the 60-minute window. This increasing network activity can be attributed to the fully connected topology instituted among the devices. With each node exchanging model parameters with all other nodes in the network, network traffic rises significantly.
In conclusion, these results collectively validate the performance of the Fedstellar platform, demonstrating its capability to navigate the complexities of a decentralized IoT environment. It showcases a successful adaptation to asynchronous processes, efficient resource management, and the adept handling of extensive network activity. This suggests that Fedstellar is well-positioned to facilitate effective and scalable FL solutions in real-world IoT scenarios.
\subsection{Virtualized Scenario}
In this scenario, Fedstellar illustrates its proficiency in adapting to varying configurations. With the deployment of twenty docker containers, the system handles different federation architectures and network topologies, each tested with the MNIST and CIFAR-10 datasets. The performance analysis, as shown in \tablename~\ref{tab:metrics-MNIST} and \tablename~\ref{tab:metrics-CIFAR10}, offers a detailed view of the average metrics obtained across the twenty containers over 60 minutes.
\begin{table}[ht!]
\caption{Average metrics obtained by twenty virtualized participants using MNIST dataset during 60 minutes}
\label{tab:metrics-MNIST}
\centering
\resizebox{\columnwidth}{!}{
\begin{threeparttable}
\begin{tabular}{c c c c c c c}
\hline
\textbf{\makecell{Federation\\Architecture}} & \textbf{\makecell{Network\\Topology}} & \makecell[r]{\textbf{Model}\\$(F_{1} \ score)$} & \makecell[r]{\textbf{CPU}\\$(\%)$} & \makecell[r]{\textbf{RAM}\\$(\%)$} & \makecell[r]{\textbf{Network}\\$(\text{MB})$} & \makecell[r]{\textbf{Time*}\\$(min.)$} \\
\hline
\multirow{3}{*}{DFL} & Fully & \makecell{0.987 $\pm0.009$} & \makecell{78 $\pm15\ \%$} & \makecell{29 $\pm6 \ \%$} & \makecell{$\approx 1243 \ \text{MB}$} & \makecell{$\approx 28$} \\
& Star & \makecell{0.955 $\pm0.012$} & \makecell{72 $\pm13\ \%$} & \makecell{28 $\pm5\ \%$} & \makecell{$\approx 1165 \ \text{MB}$} & \makecell{$\approx 35$} \\
& Ring & \makecell{0.917 $\pm0.019$} & \makecell{70 $\pm14\ \%$} & \makecell{26 $\pm4\ \%$} & \makecell{$\approx 1089 \ \text{MB}$} & \makecell{$\approx 41$} \\
\hline
\multirow{3}{*}{SDFL} & Fully & \makecell{0.973 $\pm0.015$} & \makecell{69 $\pm12\ \%$} & \makecell{28 $\pm5\ \%$} & \makecell{$\approx 1148 \ \text{MB}$} & \makecell{$\approx 32$} \\
& Star & \makecell{0.938 $\pm0.020$} & \makecell{66 $\pm11\ \%$} & \makecell{27 $\pm4\ \%$} & \makecell{$\approx 1065 \ \text{MB}$} & \makecell{$\approx 38$} \\
& Ring & \makecell{0.901 $\pm0.027$} & \makecell{64 $\pm13\ \%$} & \makecell{25 $\pm4\ \%$} & \makecell{$\approx 1023 \ \text{MB}$} & \makecell{$\approx 45$} \\
\hline
CFL & Star & \makecell{0.992 $\pm0.010$} & \makecell{58 $\pm10\ \%$} & \makecell{26 $\pm3\ \%$} & \makecell{$\approx 985 \ \text{MB}$} & \makecell{$\approx 40$} \\
\hline
\end{tabular}
\begin{tablenotes}
\item * Overall time to reach model $F_{1} \ score \geq 90\ \%$
\end{tablenotes}
\end{threeparttable}}
\end{table}
\begin{table}[ht!]
\caption{Average metrics obtained by twenty virtualized participants using CIFAR-10 dataset during 60 minutes}
\label{tab:metrics-CIFAR10}
\centering
\resizebox{\columnwidth}{!}{
\begin{threeparttable}
\begin{tabular}{c c c c c c c}
\hline
\textbf{\makecell{Federation\\Architecture}} & \textbf{\makecell{Network\\Topology}} & \makecell[r]{\textbf{Model}\\$(F_{1} \ score)$} & \makecell[r]{\textbf{CPU}\\$(\%)$} & \makecell[r]{\textbf{RAM}\\$(\%)$} & \makecell[r]{\textbf{Network}\\$(\text{MB})$} & \makecell[r]{\textbf{Time*}\\$(min.)$} \\
\hline
\multirow{3}{*}{DFL} & Fully & \makecell{0.912 $\pm0.012$} & \makecell{80 $\pm16\ \%$} & \makecell{32 $\pm6\ \%$} & \makecell{$\approx 1280 \ \text{MB}$} & \makecell{$\approx 33$} \\
& Star & \makecell{0.889 $\pm0.015$} & \makecell{76 $\pm14\ \%$} & \makecell{30 $\pm6\ \%$} & \makecell{$\approx 1202 \ \text{MB}$} & \makecell{$\approx 38$} \\
& Ring & \makecell{0.862 $\pm0.020$} & \makecell{74 $\pm15\ \%$} & \makecell{28 $\pm5\ \%$} & \makecell{$\approx 1134 \ \text{MB}$} & \makecell{$\approx 43$} \\
\hline
\multirow{3}{*}{SDFL} & Fully & \makecell{0.895 $\pm0.016$} & \makecell{72 $\pm13\ \%$} & \makecell{29 $\pm6\ \%$} & \makecell{$\approx 1156 \ \text{MB}$} & \makecell{$\approx 36$} \\
& Star & \makecell{0.873 $\pm0.020$} & \makecell{70 $\pm14\ \%$} & \makecell{28 $\pm5\ \%$} & \makecell{$\approx 1090 \ \text{MB}$} & \makecell{$\approx 40$} \\
& Ring & \makecell{0.852 $\pm0.027$} & \makecell{67 $\pm12\ \%$} & \makecell{27 $\pm6\ \%$} & \makecell{$\approx 1050 \ \text{MB}$} & \makecell{$\approx 46$} \\
\hline
CFL & Star & \makecell{0.924 $\pm0.013$} & \makecell{62 $\pm11\ \%$} & \makecell{28 $\pm4\ \%$} & \makecell{$\approx 1020 \ \text{MB}$} & \makecell{$\approx 43$} \\
\hline
\end{tabular}
\begin{tablenotes}
\item * Overall time to reach model $F_{1} \ score \geq 85\ \%$
\end{tablenotes}
\end{threeparttable}}
\end{table}
\tablename~\ref{tab:metrics-MNIST} details the performance with the MNIST dataset, where the CFL configuration reports the highest $F_{1} \ score$ of 99.2 \%. Although this high score emphasizes the efficiency of a centralized approach, it is essential to acknowledge the trade-offs involved. Despite reporting a slightly lower $F_{1} \ score$ of 98.7 \%, the DFL with a fully connected topology presents considerable advantages. A key highlight is its accelerated convergence, taking approximately 28 minutes to reach an $F_{1} \ score$ of 80 \%, a considerable reduction in time compared to the 40 minutes of CFL. This acceleration in achieving model performance indicates an increased efficiency integral to the DFL platform, despite a slight trade-off in $F_{1} \ score$. Simultaneously, the average CPU usage across all configurations remains relatively high, highlighting its efficiency in utilizing available computational resources.
Considering the CPU usage, DFL with a fully connected topology stands at 78 \%. This observation can be linked to the high communication requirements and the corresponding processing needed for maintaining such a topology. In contrast, the RAM usage remains comparably stable among the other configurations, indicating the balanced resource allocation within the DFL architecture. This evidence further supports the notion that Fedstellar is a memory-efficient system with minimal impact on processing requirements from changes in federation architecture.
The SDFL architectures also demonstrate positive results, despite a slightly lower $F_{1} \ score$ compared to DFL and CFL. SDFL exhibits efficient resource consumption, reducing network usage to approximately 1148 MB, CPU usage to 69 \%, and RAM to 28 \% when compared to the corresponding values under DFL. These numbers indicate an excellent balance between resource usage and performance. This balance, coupled with an adequate $F_{1} \ score$, makes SDFL an appealing option, synthesizing the benefits of both centralized and decentralized architectures. When moving to the CIFAR-10 dataset, as seen in \tablename~\ref{tab:metrics-CIFAR10}, there is a noticeable decrease in performance under similar configurations. This drop reflects the increased complexity of the CIFAR-10 dataset compared to MNIST. Despite this, the DFL architecture with fully connected topology remains robust, achieving the highest $F_{1} \ score$ of 91.2 \% within the DFL configuration. Though slightly lower than CFL with 92.4 \%, this score is attained in approximately 33 minutes, about 10 minutes faster than CFL.
This quick convergence highlights the adaptability of the proposed platform with more complex datasets. The increase in CPU usage of 80 \% and network usage of 1280 MB can be attributed to the complexity and size of the CIFAR-10 dataset, which has color images compared to grayscale images of MNIST. Furthermore, the $F_{1} \ scores$ for SDFL configurations indicate a small decrease ranging from 97.3 \% to 89.5 \%, which could be improved with more robust selection techniques or using multiple aggregators simultaneously.
In conclusion, the results from the virtualized scenario underscore the versatile capabilities of Fedstellar across diverse federated architectures and dataset complexities. While CFL demonstrates impressive performance, DFL, in conjunction with a fully connected topology, offers a compelling blend of efficiency, rapid convergence, and robust performance, even when dealing with the more complex CIFAR-10 dataset. This demonstrates the platform readiness to handle a variety of AI tasks effectively, making it a promising tool for flexible and efficient FL applications.
\section{Conclusion}\label{sec:conclusion}
This work presents Fedstellar, an innovative platform enabling the decentralized training of FL models across diverse physical and virtualized devices. It facilitates the creation and management of DFL, SDFL, and CFL architectures, deployment of complex network topologies, and usage of assorted ML/DL models and datasets, all while enabling user-friendly customization and efficient training process monitoring. The platform incorporates a modular architecture comprising an intuitive frontend for experiment setup and monitoring, a controller for effective operations orchestration, and a core component deployed in each device for federated training and communication. This design makes Fedstellar a comprehensive solution for diverse network topologies and IoT scenarios. Fedstellar was tested across two federated scenarios: (i) a physical one, composed of eight resource-constrained devices, such as Raspberry Pis and Rock64s, acting as spectrum sensors subjected to cyberattacks, and (ii) a virtualized scenario dealing with twenty Docker containers using the well-known MNIST and CIFAR-10 datasets. These tests demonstrated their flexibility and performance, with $F_{1} \ score$ of 91\% in the physical deployment using DFL with a fully connected topology, while 98\% and 97.3\% were obtained using DFL and SDFL with the MNIST dataset. Similarly, the SDFL architecture showcased an approximate 8\% reduction in network usage, a slight trade-off for federated model performance, thus underscoring the balance between resource optimization and communication efficiency.
As future work, there remains a multitude of avenues for exploration. Further research could expand these evaluations to various data types and network topologies. Moreover, additional investigation into the specific performance dynamics of DFL and SDFL could be insightful, particularly in developing techniques for improving its $F_{1} \ score$ without significantly increasing network overhead. Other open research challenges include optimizing the federated training process, exploring more robust selection techniques, and using multiple simultaneous aggregators. By addressing these challenges, further progress can be made in advancing the performance and adaptability of Fedstellar.
\begin{thebibliography}{31}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\DOIprefix}{}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{https://doi.org/#1}{https://doi.org/#1}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Misc
\bibitem[{Agafonkin(2021)}]{Agafonkin:leaflet:2021}
\bibinfo{author}{Agafonkin, V.} (\bibinfo{year}{2021}).
\newblock \bibinfo{title}{Leaflet}.
\newblock \URLprefix \url{https://leafletjs.com}. Accessed 2023-06-03.
%Type = Article
\bibitem[{Arapakis et~al.(2023)Arapakis, Papadopoulos, Katevas \&
Perino}]{Arapakis:P4L:2023}
\bibinfo{author}{Arapakis, I.}, \bibinfo{author}{Papadopoulos, P.},
\bibinfo{author}{Katevas, K.}, \& \bibinfo{author}{Perino, D.}
(\bibinfo{year}{2023}).
\newblock \bibinfo{title}{{P4L}: Privacy preserving peer-to-peer learning for
infrastructureless setups}.
\newblock {\it \bibinfo{journal}{arXiv preprint arXiv:2302.13438}\/}, .
%Type = Misc
\bibitem[{Bostock(2021)}]{Bostock:d3js:2021}
\bibinfo{author}{Bostock, M.} (\bibinfo{year}{2021}).
\newblock \bibinfo{title}{D3.js - data-driven documents}.
\newblock \URLprefix \url{https://d3js.org}. Accessed 2023-05-20.
%Type = Article
\bibitem[{Chen et~al.(2023)Chen, Liang \& Gao}]{Chen:DFedSN:2023}
\bibinfo{author}{Chen, Y.}, \bibinfo{author}{Liang, L.}, \&
\bibinfo{author}{Gao, W.} (\bibinfo{year}{2023}).
\newblock \bibinfo{title}{{DFedSN}: Decentralized federated learning based on
heterogeneous data in social networks}.
\newblock {\it \bibinfo{journal}{World Wide Web}\/}, .
\DOIprefix\doi{10.1007/s11280-023-01152-4}.
%Type = Article
\bibitem[{Deng(2012)}]{Deng:MNIST:2012}
\bibinfo{author}{Deng, L.} (\bibinfo{year}{2012}).
\newblock \bibinfo{title}{The {MNIST} database of handwritten digit images for
machine learning research}.
\newblock {\it \bibinfo{journal}{IEEE Signal Processing Magazine}\/}, {\it
\bibinfo{volume}{29}\/}, \bibinfo{pages}{141--142}.
\DOIprefix\doi{10.1109/MSP.2012.2211477}.
%Type = Misc
\bibitem[{Falcon(2019)}]{Falcon:PyTorch-Lightning:2019}
\bibinfo{author}{Falcon, W.} (\bibinfo{year}{2019}).
\newblock \bibinfo{title}{{PyTorch Lightning}}.
\newblock \URLprefix \url{https://github.com/Lightning-AI/lightning}. Accessed
2023-06-10. \DOIprefix\doi{10.5281/zenodo.3828935}.
%Type = Article
\bibitem[{Friha et~al.(2023)Friha, Ferrag, Benbouzid, Berghout, Kantarci \&
Choo}]{Friha:2df-ids:2023}
\bibinfo{author}{Friha, O.}, \bibinfo{author}{Ferrag, M.~A.},
\bibinfo{author}{Benbouzid, M.}, \bibinfo{author}{Berghout, T.},
\bibinfo{author}{Kantarci, B.}, \& \bibinfo{author}{Choo, K.-K.~R.}
(\bibinfo{year}{2023}).
\newblock \bibinfo{title}{{2DF-IDS}: Decentralized and differentially private
federated learning-based intrusion detection system for industrial iot}.
\newblock {\it \bibinfo{journal}{Computers \& Security}\/}, {\it
\bibinfo{volume}{127}\/}, \bibinfo{pages}{103097}.
\DOIprefix\doi{10.1016/j.cose.2023.103097}.
%Type = Misc
\bibitem[{Google(2019{\natexlab{a}})}]{google:tensorboard:2019}
\bibinfo{author}{Google} (\bibinfo{year}{2019}{\natexlab{a}}).
\newblock \bibinfo{title}{Tensorboard}.
\newblock \URLprefix \url{https://www.tensorflow.org/tensorboard}. Accessed
2023-06-02.
%Type = Misc
\bibitem[{Google(2019{\natexlab{b}})}]{google:tff:2019}
\bibinfo{author}{Google} (\bibinfo{year}{2019}{\natexlab{b}}).
\newblock \bibinfo{title}{{TensorFlow Federated}}.
\newblock \URLprefix \url{https://www.tensorflow.org/federated}. Accessed
2023-05-05.
%Type = Inproceedings
\bibitem[{Hard et~al.(2021)Hard, Partridge, Mathews \&
Augenstein}]{Hard:google_centralized_to_dfl:2021}
\bibinfo{author}{Hard, A.}, \bibinfo{author}{Partridge, K.},
\bibinfo{author}{Mathews, R.}, \& \bibinfo{author}{Augenstein, S.}
(\bibinfo{year}{2021}).
\newblock \bibinfo{title}{Jointly learning from decentralized (federated) and
centralized data to mitigate distribution shift}.
\newblock In {\it \bibinfo{booktitle}{Proceedings of {NeurIPS} {Workshop on
Distribution Shifts}}\/}.
%Type = Article
\bibitem[{He et~al.(2020)He, Li, So, Zhang, Wang, Wang, Vepakomma, Singh, Qiu,
Shen, Zhao, Kang, Liu, Raskar, Yang, Annavaram \& Avestimehr}]{He:fedml:2020}
\bibinfo{author}{He, C.}, \bibinfo{author}{Li, S.}, \bibinfo{author}{So, J.},
\bibinfo{author}{Zhang, M.}, \bibinfo{author}{Wang, H.},
\bibinfo{author}{Wang, X.}, \bibinfo{author}{Vepakomma, P.},
\bibinfo{author}{Singh, A.}, \bibinfo{author}{Qiu, H.},
\bibinfo{author}{Shen, L.}, \bibinfo{author}{Zhao, P.},
\bibinfo{author}{Kang, Y.}, \bibinfo{author}{Liu, Y.},
\bibinfo{author}{Raskar, R.}, \bibinfo{author}{Yang, Q.},
\bibinfo{author}{Annavaram, M.}, \& \bibinfo{author}{Avestimehr, S.}
(\bibinfo{year}{2020}).
\newblock \bibinfo{title}{{FedML}: A research library and benchmark for
federated machine learning}.
\newblock {\it \bibinfo{journal}{Advances in Neural Information Processing
Systems, Best Paper Award at Federate Learning Workshop}\/}, .
%Type = Article
\bibitem[{Howard et~al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang, Weyand,
Andreetto \& Adam}]{Howard:mobilenet:2017}
\bibinfo{author}{Howard, A.~G.}, \bibinfo{author}{Zhu, M.},
\bibinfo{author}{Chen, B.}, \bibinfo{author}{Kalenichenko, D.},
\bibinfo{author}{Wang, W.}, \bibinfo{author}{Weyand, T.},
\bibinfo{author}{Andreetto, M.}, \& \bibinfo{author}{Adam, H.}
(\bibinfo{year}{2017}).
\newblock \bibinfo{title}{Mobilenets: Efficient convolutional neural networks
for mobile vision applications}.
\newblock {\it \bibinfo{journal}{arXiv preprint arXiv:1704.04861}\/}, .
%Type = Article
\bibitem[{Huertas~Celdrán et~al.(2023{\natexlab{a}})Huertas~Celdrán,
Sánchez~Sánchez, Bovet, Martínez~Pérez \&
Stiller}]{Huertas:ssdf-attacks:2023}
\bibinfo{author}{Huertas~Celdrán, A.}, \bibinfo{author}{Sánchez~Sánchez,
P.~M.}, \bibinfo{author}{Bovet, G.}, \bibinfo{author}{Martínez~Pérez, G.},
\& \bibinfo{author}{Stiller, B.} (\bibinfo{year}{2023}{\natexlab{a}}).
\newblock \bibinfo{title}{{CyberSpec}: Behavioral fingerprinting for
intelligent attacks detection on crowdsensing spectrum sensors}.
\newblock {\it \bibinfo{journal}{IEEE Transactions on Dependable and Secure
Computing}\/}, (pp. \bibinfo{pages}{1--14}).
\DOIprefix\doi{10.1109/TDSC.2023.3252918}.
%Type = Article
\bibitem[{Huertas~Celdrán et~al.(2023{\natexlab{b}})Huertas~Celdrán,
Sánchez~Sánchez, Feng, Bovet, Pérez \&
Stiller}]{Huertas:syscall-dataset:2023}
\bibinfo{author}{Huertas~Celdrán, A.}, \bibinfo{author}{Sánchez~Sánchez,
P.~M.}, \bibinfo{author}{Feng, C.}, \bibinfo{author}{Bovet, G.},
\bibinfo{author}{Pérez, G.~M.}, \& \bibinfo{author}{Stiller, B.}
(\bibinfo{year}{2023}{\natexlab{b}}).
\newblock \bibinfo{title}{Privacy-preserving and syscall-based intrusion
detection system for iot spectrum sensors affected by data falsification
attacks}.
\newblock {\it \bibinfo{journal}{IEEE Internet of Things Journal}\/}, {\it
\bibinfo{volume}{10}\/}, \bibinfo{pages}{8408--8415}.
\DOIprefix\doi{10.1109/JIOT.2022.3213889}.
%Type = Misc
\bibitem[{Krizhevsky(2009)}]{krizhevsky:cifar10-100:2009}
\bibinfo{author}{Krizhevsky, A.} (\bibinfo{year}{2009}).
\newblock \bibinfo{title}{Learning multiple layers of features from tiny
images}.
\newblock \URLprefix
\url{https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf}.
Accessed 2023-04-20.
%Type = Article
\bibitem[{Lecun et~al.(1998)Lecun, Bottou, Bengio \&
Haffner}]{Lecun:lenet5:1998}
\bibinfo{author}{Lecun, Y.}, \bibinfo{author}{Bottou, L.},
\bibinfo{author}{Bengio, Y.}, \& \bibinfo{author}{Haffner, P.}
(\bibinfo{year}{1998}).
\newblock \bibinfo{title}{Gradient-based learning applied to document
recognition}.
\newblock {\it \bibinfo{journal}{Proceedings of the IEEE}\/}, {\it
\bibinfo{volume}{86}\/}, \bibinfo{pages}{2278--2324}.
\DOIprefix\doi{10.1109/5.726791}.
%Type = Article
\bibitem[{Liu et~al.(2021)Liu, Fan, Chen, Xu \& Yang}]{fedai:fate:2021}
\bibinfo{author}{Liu, Y.}, \bibinfo{author}{Fan, T.}, \bibinfo{author}{Chen,
T.}, \bibinfo{author}{Xu, Q.}, \& \bibinfo{author}{Yang, Q.}
(\bibinfo{year}{2021}).
\newblock \bibinfo{title}{{FATE}: An industrial grade platform for
collaborative learning with data protection}.
\newblock {\it \bibinfo{journal}{Journal of Machine Learning Research}\/},
{\it \bibinfo{volume}{22}\/}, \bibinfo{pages}{1--6}. \URLprefix
\url{http://jmlr.org/papers/v22/20-815.html}.
%Type = Article
\bibitem[{Lu et~al.(2023)Lu, Xiong, Li, Mohammad, Li \& Li}]{Lu:DEFEAT:2023}
\bibinfo{author}{Lu, G.}, \bibinfo{author}{Xiong, Z.}, \bibinfo{author}{Li,
R.}, \bibinfo{author}{Mohammad, N.}, \bibinfo{author}{Li, Y.}, \&
\bibinfo{author}{Li, W.} (\bibinfo{year}{2023}).
\newblock \bibinfo{title}{{DEFEAT}: A decentralized federated learning against
gradient attacks}.
\newblock {\it \bibinfo{journal}{High-Confidence Computing}\/}, (p.
\bibinfo{pages}{100128}). \DOIprefix\doi{10.1016/j.hcc.2023.100128}.
%Type = Misc
\bibitem[{Mart{\'i}nez~Beltr{\'a}n et~al.(2023)Mart{\'i}nez~Beltr{\'a}n,
Perales~G\'omez, Feng, S{\'a}nchez~S{\'a}nchez, L{\'o}pez~Bernal, Bovet,
Gil~P{\'e}rez, Mart{\'i}nez~P{\'e}rez \&
Huertas~Celdr{\'a}n}]{MartinezBeltran:fedstellar_git:2023}
\bibinfo{author}{Mart{\'i}nez~Beltr{\'a}n, E.~T.},
\bibinfo{author}{Perales~G\'omez, A.~L.}, \bibinfo{author}{Feng, C.},
\bibinfo{author}{S{\'a}nchez~S{\'a}nchez, P.~M.},
\bibinfo{author}{L{\'o}pez~Bernal, S.}, \bibinfo{author}{Bovet, G.},
\bibinfo{author}{Gil~P{\'e}rez, M.}, \bibinfo{author}{Mart{\'i}nez~P{\'e}rez,
G.}, \& \bibinfo{author}{Huertas~Celdr{\'a}n, A.} (\bibinfo{year}{2023}).
\newblock \bibinfo{title}{{enriquetomasmb/fedstellar}}.
\newblock \URLprefix \url{https://github.com/enriquetomasmb/fedstellar}.
%Type = Article
\bibitem[{Mart{\'i}nez~Beltr{\'a}n et~al.(2022)Mart{\'i}nez~Beltr{\'a}n,
Quiles~P{\'e}rez, S{\'a}nchez~S{\'a}nchez, L{\'o}pez~Bernal, Bovet,
Gil~P{\'e}rez, Mart{\'i}nez~P{\'e}rez \&
Huertas~Celdr{\'a}n}]{MartinezBeltran:DFL_survey:2022}
\bibinfo{author}{Mart{\'i}nez~Beltr{\'a}n, E.~T.},
\bibinfo{author}{Quiles~P{\'e}rez, M.},
\bibinfo{author}{S{\'a}nchez~S{\'a}nchez, P.~M.},
\bibinfo{author}{L{\'o}pez~Bernal, S.}, \bibinfo{author}{Bovet, G.},
\bibinfo{author}{Gil~P{\'e}rez, M.}, \bibinfo{author}{Mart{\'i}nez~P{\'e}rez,
G.}, \& \bibinfo{author}{Huertas~Celdr{\'a}n, A.} (\bibinfo{year}{2022}).
\newblock \bibinfo{title}{{Decentralized Federated Learning: Fundamentals,
State-of-the-art, Frameworks, Trends, and Challenges}}.
\newblock {\it \bibinfo{journal}{arXiv preprint arXiv:2211.08413}\/}, .
%Type = Article
\bibitem[{McMahan et~al.(2016)McMahan, Moore, Ramage, Hampson \&
Arcas}]{McMahan:FL_google:2016}
\bibinfo{author}{McMahan, H.~B.}, \bibinfo{author}{Moore, E.},
\bibinfo{author}{Ramage, D.}, \bibinfo{author}{Hampson, S.}, \&
\bibinfo{author}{Arcas, B. A.~y.} (\bibinfo{year}{2016}).
\newblock \bibinfo{title}{Communication-efficient learning of deep networks
from decentralized data}.
\newblock {\it \bibinfo{journal}{arXiv preprint arXiv:1602.05629}\/}, .
\DOIprefix\doi{10.48550/ARXIV.1602.05629}.
%Type = Misc
\bibitem[{Northern.tech(2022)}]{Northern:mender:2021}
\bibinfo{author}{Northern.tech} (\bibinfo{year}{2022}).
\newblock \bibinfo{title}{Mender: Open source over-the-air software updates for
linux devices}.
\newblock \URLprefix \url{https://mender.io}. Accessed 2023-06-10.
%Type = Article
\bibitem[{Paleyes et~al.(2022)Paleyes, Urma \&
Lawrence}]{Paleyes:ml-challenges:2022}
\bibinfo{author}{Paleyes, A.}, \bibinfo{author}{Urma, R.-G.}, \&
\bibinfo{author}{Lawrence, N.~D.} (\bibinfo{year}{2022}).
\newblock \bibinfo{title}{Challenges in deploying machine learning: A survey of
case studies}.
\newblock {\it \bibinfo{journal}{ACM Comput. Surv.}\/}, {\it
\bibinfo{volume}{55}\/}. \DOIprefix\doi{10.1145/3533378}.
%Type = Article
\bibitem[{Qu et~al.(2022)Qu, Xu, Gao, Xiang \& Yu}]{Qu:fl-sec:2022}
\bibinfo{author}{Qu, Y.}, \bibinfo{author}{Xu, C.}, \bibinfo{author}{Gao, L.},
\bibinfo{author}{Xiang, Y.}, \& \bibinfo{author}{Yu, S.}
(\bibinfo{year}{2022}).
\newblock \bibinfo{title}{{FL-SEC}: Privacy-preserving decentralized federated
learning using signsgd for the internet of artificially intelligent things}.
\newblock {\it \bibinfo{journal}{IEEE Internet of Things Magazine}\/}, {\it
\bibinfo{volume}{5}\/}, \bibinfo{pages}{85--90}.
\DOIprefix\doi{10.1109/IOTM.001.2100173}.
%Type = Article
\bibitem[{Rajendran et~al.(2018)Rajendran, Calvo-Palomino, Fuchs, Van~den
Bergh, Cordobes, Giustiniano, Pollin \&
Lenders}]{Rajendran:electrosense:2018}
\bibinfo{author}{Rajendran, S.}, \bibinfo{author}{Calvo-Palomino, R.},
\bibinfo{author}{Fuchs, M.}, \bibinfo{author}{Van~den Bergh, B.},
\bibinfo{author}{Cordobes, H.}, \bibinfo{author}{Giustiniano, D.},
\bibinfo{author}{Pollin, S.}, \& \bibinfo{author}{Lenders, V.}
(\bibinfo{year}{2018}).
\newblock \bibinfo{title}{Electrosense: Open and big spectrum data}.
\newblock {\it \bibinfo{journal}{IEEE Communications Magazine}\/}, {\it
\bibinfo{volume}{56}\/}, \bibinfo{pages}{210--217}.
\DOIprefix\doi{10.1109/MCOM.2017.1700200}.
%Type = Misc
\bibitem[{Reinsel et~al.(2018)Reinsel, Gantz \& Rydnin}]{idc:iot_number:2022}
\bibinfo{author}{Reinsel, D.}, \bibinfo{author}{Gantz, J.}, \&
\bibinfo{author}{Rydnin, J.} (\bibinfo{year}{2018}).
\newblock \bibinfo{title}{The digitization of the world from edge to core}.
\newblock \URLprefix
\url{https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf}.
Accessed 2023-04-20.
%Type = Article
\bibitem[{Roy et~al.(2019)Roy, Siddiqui, P{\"o}lsterl, Navab \&
Wachinger}]{Roy:braintorrent:2019}
\bibinfo{author}{Roy, A.~G.}, \bibinfo{author}{Siddiqui, S.},
\bibinfo{author}{P{\"o}lsterl, S.}, \bibinfo{author}{Navab, N.}, \&
\bibinfo{author}{Wachinger, C.} (\bibinfo{year}{2019}).
\newblock \bibinfo{title}{Braintorrent: A peer-to-peer environment for
decentralized federated learning}.
\newblock {\it \bibinfo{journal}{arXiv preprint arXiv:1905.06731}\/}, .
%Type = Article
\bibitem[{Sánchez~Sánchez et~al.(2021)Sánchez~Sánchez, Jorquera~Valero,
Huertas~Celdrán, Bovet, Gil~Pérez \&
Martínez~Pérez}]{Sanchez:fingerprinting:2021}
\bibinfo{author}{Sánchez~Sánchez, P.~M.}, \bibinfo{author}{Jorquera~Valero,
J.~M.}, \bibinfo{author}{Huertas~Celdrán, A.}, \bibinfo{author}{Bovet, G.},
\bibinfo{author}{Gil~Pérez, M.}, \& \bibinfo{author}{Martínez~Pérez, G.}
(\bibinfo{year}{2021}).
\newblock \bibinfo{title}{A survey on device behavior fingerprinting: Data
sources, techniques, application scenarios, and datasets}.
\newblock {\it \bibinfo{journal}{IEEE Communications Surveys \& Tutorials}\/},
{\it \bibinfo{volume}{23}\/}, \bibinfo{pages}{1048--1077}.
\DOIprefix\doi{10.1109/COMST.2021.3064259}.
%Type = Article
\bibitem[{Wang et~al.(2022)Wang, Xu, Xu, Chen \& Huang}]{Wang:dfl_edge:2022}
\bibinfo{author}{Wang, L.}, \bibinfo{author}{Xu, Y.}, \bibinfo{author}{Xu, H.},
\bibinfo{author}{Chen, M.}, \& \bibinfo{author}{Huang, L.}
(\bibinfo{year}{2022}).
\newblock \bibinfo{title}{Accelerating decentralized federated learning in
heterogeneous edge computing}.
\newblock {\it \bibinfo{journal}{IEEE Transactions on Mobile Computing}\/},
(pp. \bibinfo{pages}{1--1}). \DOIprefix\doi{10.1109/TMC.2022.3178378}.
%Type = Article
\bibitem[{Wang et~al.(2021)Wang, Liu, Zheng, Dai, Jia \&
Xie}]{Wang:edge_communication_optimization:2021}
\bibinfo{author}{Wang, T.}, \bibinfo{author}{Liu, Y.}, \bibinfo{author}{Zheng,
X.}, \bibinfo{author}{Dai, H.-N.}, \bibinfo{author}{Jia, W.}, \&
\bibinfo{author}{Xie, M.} (\bibinfo{year}{2021}).
\newblock \bibinfo{title}{Edge-based communication optimization for distributed
federated learning}.
\newblock {\it \bibinfo{journal}{IEEE Transactions on Network Science and
Engineering}\/}, (pp. \bibinfo{pages}{1--1}).
\DOIprefix\doi{10.1109/TNSE.2021.3083263}.
%Type = Article
\bibitem[{Wilt et~al.(2021)Wilt, Matelsky \&
Gearhart}]{Wilt:scatterbrained:2021}
\bibinfo{author}{Wilt, M.}, \bibinfo{author}{Matelsky, J.~K.}, \&
\bibinfo{author}{Gearhart, A.~S.} (\bibinfo{year}{2021}).
\newblock \bibinfo{title}{Scatterbrained: A flexible and expandable pattern for
decentralized machine learning}.
\newblock {\it \bibinfo{journal}{arXiv preprint arXiv:2112.07718}\/}, .
\end{thebibliography}