Federated Sinkhorn
Extracted source text · arXiv 2502.07021
\documentclass[lettersize,journal]{IEEEtran}
\usepackage{amsmath,amsfonts}
\usepackage{algorithmic}
\usepackage{array}
\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage{textcomp}
\usepackage{stfloats}
\usepackage{url}
\usepackage{verbatim}
\usepackage{balance}
\usepackage{tikz}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{url}
\usepackage{multicol}
\usepackage{multirow}
\usepackage[ruled]{algorithm2e}
\usepackage{wrapfig}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{amsthm}
\usepackage{authblk}
\usepackage{subcaption}% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\usepackage{optidef}
\input{math_commands.tex}
\newtheorem{theorem}{Theorem}
\newtheorem{remark}{Remark}%
\newtheorem{assumption}{Assumption}
\newtheorem{corollary}{Corollary}
\newtheorem{definition}{Definition}%
\newtheorem{proposition}{Proposition}
\title{Federated Sinkhorn}
\author[2]{Jeremy Kulcsar}
\author[1]{Vyacheslav Kungurtsev}
\affil[1]{\small Department of Computer Science, Czech Technical University in Prague, Czechia}
\author[1,3,5]{Georgios Korpas}
\author[4]{Giulio Giaconi}
\author[4]{William Shoosmith}
\affil[2]{\small HSBC Quantum Technologies Group, Innovation \& Ventures, HSBC, Hong Kong}
\affil[3]{\small HSBC Quantum Technologies Group, Innovation \& Ventures, HSBC, Singapore}
\affil[4]{\small HSBC Quantum Technologies Group, Innovation \& Ventures, HSBC, United Kingdom}
\affil[5]{\small Archimedes Research Unit on AI, Data Science and Algorithms, Marousi, Greece}
%\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
In this work we, investigate the potential of solving the discrete Optimal Transport (OT) problem with entropy regularization in a federated learning setting. Recall that the celebrated Sinkhorn algorithm transforms the classical OT linear program into strongly convex constrained optimization, facilitating first order methods for otherwise intractably large problems.
%We can consider that a natural process of scaling up the problem further would be
A common contemporary setting that remains an open problem as far as the application of Sinkhorn is the presence of data spread across clients with distributed inter-communication, either due to clients whose privacy is a concern, or simply by necessity of processing and memory hardware limitations.
In this work we investigate various natural procedures, which we refer to as Federated Sinkhorn, that handle distributed environments where data is partitioned across multiple clients. We formulate the problem as minimizing the transport cost with an entropy regularization term, subject to marginal constraints, where block components of the source and target distribution vectors are locally known to clients corresponding to each block. We consider both synchronous and asynchronous variants as well as all-to-all and server-client communication topology protocols. Each procedure allows clients to compute local operations on their data partition while periodically exchanging information with others. We provide theoretical guarantees on convergence for the different variants under different possible conditions. We empirically demonstrate the algorithms' performance on synthetic datasets and a real-world financial risk assessment application. The investigation highlights the subtle tradeoffs associated with computation and communication time in different settings and how they depend on problem size and sparsity. %Our work opens new avenues for applying optimal transport techniques in federated learning scenarios, with potential applications in privacy-preserving data analysis and distributed machine learning.
\end{abstract}
%\section{Introduction}
%We are interested in the problem
%\begin{mini}|l|
% {P\in\mathbb{R}^{n\times n}}{ \langle P, C\rangle + \epsilon \sum\limits_{i,j=1}^N P_{ij} (\log (P_{ij})-1) }{}{}\label{eq:otprob}
% \addConstraint{b}{ = P a}{}
% \end{mini}
%Here, $\langle P, C\rangle$ represents the transport cost. Each matrix component $P_{ij}$ defines how much is moved from point $i$ to point $j$. The cost matrix $C$ is fixed. The objective component $ \epsilon \sum_{i,j} P_{ij} (\log (P_{ij})-1)$ is an entropy regularization term, transforming the LP into a strongly convex optimization problem for which Sinkhorn was designed for. The vectors $a,b\in\mathbb{R}^{n}$ are the source and target distributions and, furthermore, $n=cm$, wherein there are $c$ clients each with $m$ components of $a$ and $b$. This is the celebrated optimal transport problem in discrete form~\cite{ambrosio2021lectures}, wherein the distributions of $a$ are transported at minimal distance, with minimal defined by $C$, to $b$, where the transport map is defined by $P$
%Our aim is to consider the celebrated Sinkhorn algorithm \cite{peyre2019computational}, a dual gradient approach to solving~\eqref{eq:otprob}, however now performed on a federated platform. While the algorithm has been sped up effectively with GPUs \cite{NIPS2013_af21d0c9} surprisingly little work has been done as far as distributed memory settings. \\
%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
%%%%%%%%%%%%%%%%%%%%%%
Optimal Transport (OT) theory provides a powerful mathematical framework for comparing probability distributions, with applications spanning machine learning, economics, physics, and more \cite{Villani2009}, \cite{peyre2020computationaloptimaltransport}. The fundamental problem of OT is to find the most cost-efficient way to transform one probability distribution into another, given a predefined cost function. In discrete settings, the OT problem can be formulated as a linear program, but solving it directly is computationally demanding for large instances. For this reason, one may relax the OT problem using entropy regularization. Specifically, we are interested in addressing the following optimization problem:
\begin{mini}|l|
{P\in\mathbb{R}^{n\times n}}{ \langle P, C\rangle + \epsilon \sum_{i,j=1}^n P_{ij} (\log P_{ij} - 1) }{}{}
\addConstraint{P \mathbf{1}}{= a}{}
\addConstraint{P^\top \mathbf{1}}{= b}{}
\addConstraint{P}{\geq 0.}{}
\label{eq:otprob}
\end{mini}
Here, $ P $ is the transportation plan matrix, with $ P_{ij} $ denoting the amount of mass moved from source point $ i $ to target point $ j $. The cost matrix $ C \in \mathbb{R}^{n \times n} $ is predefined, where $ C_{ij} $ represents the cost of transporting unit mass from $ i $ to $ j $. The term $ \langle P, C \rangle = \sum_{i,j} P_{ij} C_{ij} $ computes the total transportation cost. The vectors $ a, b \in \mathbb{R}^n $ are the source and target distributions, respectively, satisfying $ a, b \geq 0 $ and $ \mathbf{1}^\top a = \mathbf{1}^\top b $, where $ \mathbf{1} $ is the vector of ones.
The entropy regularization term $ \epsilon \sum_{i,j} P_{ij} (\log P_{ij} - 1) $ with $ \epsilon > 0 $ ensures that the optimization problem is strictly convex \cite{NIPS2013_af21d0c9}. This transforms the original linear program into one amenable to solution by large scale first order methods, expanding the potential problem size that can be considered in practice.
The Sinkhorn algorithm, also known as the Sinkhorn-Knopp’s fixed point iteration \cite{sinkhorn1967concerning}, is an iterative method developed by Sinkhorn and Knopp which was used to solve the entropy-regularized OT problem \cite{NIPS2013_af21d0c9}. This method gained widespread adoption due to its simplicity and computational efficiency, especially when accelerated using modern GPUs. It operates by alternately scaling the rows and columns of a kernel matrix $K= e^{-C/\epsilon}$ to enforce the marginal constraints. The optimal transportation plan is given in a diagonal scaling form:
$$
P^*=\operatorname{diag}(u) K \operatorname{diag}(v),
$$
where vectors $u, v \in \mathbb{R}_{+}^n$ are the asymptotic fixed points of the iteration:
$$
u^{(t+1)}=\frac{a}{K v^{(t)}}, \quad v^{(t+1)}=\frac{b}{K^{\top} u^{(t+1)}}.
$$
In many practical applications, data is distributed across multiple clients due to privacy regulations or proprietary constraints— a common scenario in finance \cite{he2020fedmlresearchlibrarybenchmark}. For instance, different banks may wish to collaborate to understand collective risk exposure without sharing sensitive customer data. Federated learning \cite{mcmahan2017communication} provides a framework for such collaborations by enabling model training across decentralized data while preserving privacy.
In this work, we introduce a federated version of the Sinkhorn algorithm to solve \eqref{eq:otprob} in settings where $ n = c m $, with $ c $ clients each holding local distributions $ a^{(k)}, b^{(k)} \in \mathbb{R}^m $ for $ k = 1, \dots, c $, and $ a = [a^{(1)}; \dots; a^{(c)}] $, $ b = [b^{(1)}; \dots; b^{(c)}] $. Our federated Sinkhorn algorithm allows each client to compute local updates, coordinating with a server to enforce the global marginal constraints without exchanging raw data.
\subsection{Related Work}
Optimal Transport (OT) has seen a surge in prominence since the introduction of the entropic regularization approach by Cuturi, which significantly reduced computational complexity and paved the way for scalable solutions~\cite{cuturi2013sinkhorn}. Building on this method, often referred to as the Sinkhorn–Knopp algorithm, subsequent works have explored various techniques to increase efficiency, including multi-GPU parallelization, low-rank approximations, and stochastic gradient approaches for large-scale OT problems~\cite{genevay2016stochastic, altschuler2019kmeans}. %These developments underscore the need for distributed and federated frameworks, especially as data size and dimensionality grow.
Federated learning frameworks, meanwhile, have garnered attention for decentralizing data storage and computation, thereby maintaining privacy while leveraging distributed computational resources~\cite{konecny2016federated}. The primary focus in early federated learning literature was on classical supervised methods such as gradient descent for linear models~\cite{mcmahan2017communication}, but more recent works have broadened the scope to tasks like matrix factorizations, kernel methods, and even generative modeling. However, the intersection of federated paradigms with OT problems—particularly in entropic regularization settings—remains relatively unexplored.
In the context of distributed OT, challenges such as ensuring consistency across partial marginals and mitigating stale updates when clients operate at different speeds have spurred investigations into both synchronous and asynchronous update schemes. The synchronous case typically involves each client finishing its local computation step before a global synchronization, thus guaranteeing a uniform iteration count. In contrast, asynchronous protocols allow each client to update and transmit partial information at different times, potentially reducing idle time but introducing complexities in convergence analysis~\cite{recht2011hogwild}. More recent advances in asynchronous parallel computing~\cite{lian2018asynchronous} have laid theoretical groundwork for analyzing possible convergence rates, even under communication delays and partial updates.
Bringing these themes together, our work aligns with the growing body of research seeking to balance communication efficiency, fault tolerance, and local GPU-accelerated processing factors that are particularly salient for large-scale, real-world datasets (see for example ~\cite{chetlur2014cudnn}). By federating the Sinkhorn–Knopp algorithm, we merge the dual goals of (i) scaling OT computations beyond the memory constraints of a single machine or data center node, and (ii) respecting constraints where data remain partitioned across geographically or administratively distinct clients. Our investigation also complements broader efforts toward privacy-preserving analytics (e.g., differential privacy layers applied to local distribution vectors) and domain-specific distributed setups (such as financial risk modeling). Consequently, this work situates itself at the intersection of large-scale OT computations, federated learning, and parallel algorithm design, contributing theoretical and empirical insights on how to effectively adapt Sinkhorn iterations to distributed environments.
\subsection{Contributions}
Our contributions are:
\begin{enumerate}
\item \emph{Federated Sinkhorn algorithm}. We develop four variant algorithms, across all possibilities regarding synchronicity, asynchronicity, all-to-all connectedness and star connectedness, that extends the Sinkhorn iterations to a federated environment, allowing for distributed computation of the optimal transport plan $ P $ while maintaining data privacy.
\item \emph{Convergence Analysis}. We provide the required Algorithmic features ensuring theoretical guarantees that the algorithms converge to the global optimal solution of the entropy-regularized OT \eqref{eq:otprob} problem under standard assumptions.
\item \emph{Numerical Results.} We present a comprehensive comparison of three variants of algorithms across a range of data size and number of computing nodes, showcasing the variation in convergence time and scaling considerations across settings.
\item \emph{Numerical Demonstration:} We provide a proof of concept for all of the methods on real world problems by demonstrating their computation on establishing robustness properties with financial allocation.
\end{enumerate}
This paper is organised as follows: In section \ref{sec:alg} we introduce the four variants of the federated Sinkhorn algorithm, and discuss the convergence properties of each. In section \ref{sec:nums} we demonstrate the performance of the algorithms on experiments based on synthetic data. In section \ref{sec:distrib}, we introduce the specifics of the environment and discuss the properties of distributed computation. In section \ref{sec:numt} we demonstrate that the algorithm converges in the context of solving a financial problem that is measuring the worst-case expected loss of a portfolio. Finally we conclude and summarise in section \ref{sec:conclusion}.
\section{Algorithms}\label{sec:alg}
Let $K\coloneqq e^{-C/\epsilon}$ be the Gibbs kernel associated with the cost matrix $C$. Next, recalling that $n=cm$, we block partition $K$ and assume each client knows all of its $m$ columns and rows, that is, $K_j=K[(j-1)m+1:jm,:]$ and, with abuse of notation $K_j^T = K[:,(j-1)m+1:jm]^T$ (using the standard pseudocode notation).
\begin{figure}[h]
\centering
\includegraphics[width=0.8\linewidth]{figures/sliced_inputs.pdf}
\caption{Illustration of how $a$, $b$ and $K$ are sliced in the case of $n=4$ and $c=2$ (which implies $m=2$).}
\label{fig:sliced_inputs}
\end{figure}
Locally, we denote the matrix with the $j$-th $m$-block of rows in as $K_j$ and similarly $K_j^T$ and the vector components $a_j$ and $b_j$, wherein $j \in \mathcal{C}$ where $\mathcal{C}$ is the set of clients $c$.
The solution is represented by the form $P=uv^T$. We consider that each client $j \in \mathcal{C}$ has a local copy $u_j$ and $v_j$, as well as intermediate algorithmic vectors $r_j$ and $q_j$, all of dimension $m$. Furthermore, we denote $u_{jj}$ to be the $j$-th column of $m$ components of $u_j$ and $u_{-j}$ to be the remaining matrix of dimension $n\times n-m$. The vectors $v_{jj}$, $v_{-j}$, $r_{jj}$, $r_{-j}$, $q_{jj}$, and $q_{-j}$ are defined accordingly.
We will consider repeated single iterations of computation and communication, as well as \emph{local} variants, as analogous to Local SGD~\cite{lin2019don}. To this effect, we introduce the parameter $w\in\mathbb{N}$, which aim to set a fixed number of location iteration before each communication. That is, if $w=1$, then communication takes place every round, and if $w=2$ every other round, etc.
\subsection{Privacy Regime 1 - "All-to-All" federation}
This first privacy regime is similar to a peer-to-peer communication. Each node of the network has its own sufficient local information, namely $a_j$, $b_j$ and $K_j$, and shares a connection with every other node of the network. They cannot share the original data due to network restrictions, either legal or technical. They can however share intermediate results of the computations via a process which, as we demonstrate in section~\ref{sec:con}, still has the same convergence properties of the centralised algorithm. This type of privacy regime can be applied for instance when multiple locations of a retail company want to perform price alignment across locations, but cannot share the prices directly. It will solve a problem where the output gives the new prices, and those prices will be harmonised across the locations due to their ability to be shared as the result of the computation.
\subsubsection{Synchronous Federated Sinkhorn All-to-All}
The All-to-All variant is a peer-to-peer approach to federating the Sinkhorn-Knopp algorithm across multiple nodes, where each node has the same role as the others.
The first step of this algorithm is to slice the inputs in equal parts, and share those part across nodes. Each node will have its local slices (cf. Figure 1 to see how the elements are sliced), which will not be communicated during the entirety of the process.
Then, a full $u$ and $v$ are initialised in every node as vectors of ones. For instance, $u=(1,1,1,1)$ for a problem where $n=4$ (cf. our illustrated example in Figure ~\ref{fig:sliced_inputs}).
In the first step of the iteration, the local slice $K_j$ will be used along with $v$ to compute $q_j$. Then, the local slice $a_j$ will be used with $q_j$ to generate $u_j$. $u_j$ is exactly the node's local slice of $u$, and has the same size as $a_j$. Namely, every node $j$ will only have a local slice $u_j$ that corresponds to $a_j$'s size.
Once every node is done computing its local iteration vector $u_j$, a consistent broadcast will be performed among the nodes and every $u_j$ will be shared through the $AllGather$ MPI function and then concatenated. More information about MPI and how it is used can be found in subsection~\ref{subsec:environment}. Every node will have the same resulting vector $u=[u_1,...,u_c]$, which is the full updated iteration vector $u$.
In the second step of the iteration, the local slice $K^T_j$ will be used along with the updated $u$ to compute $r_j$. Then, the local slice $b_j$ will be used with $r_j$ to generate $v_j$.
The exact same process as before will be performed for the local slices $v_j$ to gather a full updated $v$ via consistent broadcasting.
These two steps are performed until a convergence criteria is met: e.g., a set maximum number of iterations or the value of the marginal error being below a set threshold.
The resulting $u$ and $v$ will be used to compute the optimal transport plan $P$.
See Algorithm~\ref{alg:fedsinksyncall} for a detailed implementation.
\begin{algorithm}[t]
\DontPrintSemicolon
\caption{Synchronous Federated Sinkhorn All-to-All}
\label{alg:fedsinksyncall}
\small
\KwIn{Each client receives a set of local vectors $\{a_i\},\{b_i\}$ and cost matrix $\{K_i\}$. Communication frequency $w$}
\KwOut{Complete Transport Plan $P$}
\SetKwBlock{Begin}{function}{end function}{
\textbf{For Each Client} $j\in\mathcal{C}$, initialize $u_j,v_j$\;
\For{$k$ in $1...K$ iterations}{
\For{Each Client $i$}{
\If{$\mod(k,w)=0$}{Client $i$ does \emph{AllGather} $\{v_{-i}\}$ with $[v_i]_j = v_{jj}$ for all $j\in [c]$}
%\For{Each Client $j$}{
%Client $i$ \emph{Receives} $\{v_{jj},i\}$ and stores it in the $j$'th block of $v_{-i}$}
Multiply $q_i=K_i v_i$\;
Compute $u_{ii}=a_i/q_i$ with component-wise division\;
\If{$\mod(k,w)=0$}{Client $i$ does \emph{AllGather} $\{u_{-i}\}$ with $[u_i]_j = u_{jj}$ for all $j\in [c]$}
Multiply $r_i=K_i^T u_i$\;
Compute $v_{ii}=b_i/r_i$ with component-wise division\;
}
}
}
\end{algorithm}
\subsubsection{Asynchronous Federated Sinkhorn All-to-All}
The Asynchronous Federated Sinkhorn All-to-All works similarly to its synchronous counterpart but operates without a global lockstep. More precisely, the algorithm does not wait until every node has finished computing its local $u_j$ for broadcasting to be performed. Once a node completes its computation on $u_j$, it broadcasts the updated slice to other nodes on the network. Nodes that are currently waiting for communication will update their local copy of $u$ with the received slice. If a node is still performing its computation when the broadcast occurs, it will pick up the message once it finishes its current step. This results in a lag or delay in the information received, and each node will maintain a slightly different copy of the full iteration vector $u$ due to the asynchrony. The same consideration applies to $v$. More details on delays can be found in Section~\ref{subsubsec:delays_in_async}.
To improve stability and mitigate the effects of stale updates, the algorithm can include a step size parameter $\alpha \in [0, 1]$. This step size modifies the update rule for $u$ and $v$ as follows:
$$
u^{(t+1)} = \alpha \frac{a}{K v^{(t)}} + (1 - \alpha)u^{(t)},
$$
$$
v^{(t+1)} = \alpha \frac{b}{K^{\top} u^{(t+1)}} + (1 - \alpha)v^{(t)},
$$
where $\alpha = 1$ corresponds to the standard Sinkhorn-Knopp algorithm (no damping), and $\alpha = 0$ means no updates are applied to the vectors. Intuitively, the step size acts as a control knob balancing stability and progress. By scaling the magnitude of updates, it prevents large jumps in the optimization trajectory, dampens the effect of variability in update arrival times, and improves robustness under asynchronous conditions. Without a step size (i.e., $\alpha = 1$), the algorithm can diverge due to stale updates, especially in heterogeneous network settings. More details on the stepsize can be found in Section~\ref{subsubsec:stepsizesec}.
Convergence criteria can either be a set number of iterations being exceeded or whether a node achieves a marginal error below a set threshold. At the end of the process, a consistent broadcast ensures that all nodes have the same fully updated $u$ and $v$ vectors, which can then be used to compute the optimal transport plan $P$.
\begin{figure}[h!]
\centering % Center the figure
\begin{tikzpicture}
% Include the PDF page as the background
\node[anchor=south west, inner sep=0] (image) at (0,0)
{\includegraphics[width=\linewidth]{figures/all-to-all-pdf.pdf}};
% Define the coordinate system based on the image
\begin{scope}[x={(image.south east)}, y={(image.north west)}]
\node[font=\bfseries, text=black] at (0.2155, 0.743) {$u_1$};
\node[font=\bfseries, text=black] at (0.412, 0.743) {$u_2$};
\node[font=\bfseries, text=black] at (0.605, 0.743) {$u_3$};
\node[font=\bfseries, text=black] at (0.8, 0.743) {$u_4$};
\node[font=\bfseries, text=white] at (0.2155, 0.612) {$b_1, K^T_1$};
\node[font=\bfseries, text=white] at (0.412, 0.612) {$b_2, K^T_2$};
\node[font=\bfseries, text=white] at (0.605, 0.612) {$b_3, K^T_3$};
\node[font=\bfseries, text=white] at (0.8, 0.612) {$b_4, K^T_4$};
\node[font=\bfseries, text=black] at (0.2155, 0.488) {$v_1$};
\node[font=\bfseries, text=black] at (0.412, 0.488) {$v_2$};
\node[font=\bfseries, text=black] at (0.605, 0.488) {$v_3$};
\node[font=\bfseries, text=black] at (0.8, 0.488) {$v_4$};
\node[font=\bfseries, text=white] at (0.2155, 0.356) {$a_1, K_1$};
\node[font=\bfseries, text=white] at (0.412, 0.356) {$a_2, K_2$};
\node[font=\bfseries, text=white] at (0.605, 0.356) {$a_3, K_3$};
\node[font=\bfseries, text=white] at (0.8, 0.356) {$a_4, K_4$};
\node[font=\bfseries, text=black] at (0.2155, 0.233) {$u_1$};
\node[font=\bfseries, text=black] at (0.412, 0.233) {$u_2$};
\node[font=\bfseries, text=black] at (0.605, 0.233) {$u_3$};
\node[font=\bfseries, text=black] at (0.8, 0.233) {$u_4$};
\end{scope}
\end{tikzpicture}
\caption{Illustration of a full step in the iterative All-to-All process. Each node $i$ receives slices of $u$ from other nodes, concatenates them into a global $u$, uses its local $K^T_i$ and $b_i$ to compute $r_i=K^T_i u$ and $v_i=b_i / r_i$, and sends slice $v_i$ to other nodes. Then, each node does the same respective operations with $v$, $K_i$ and $a_i$.}
\label{fig:all-to-all}
\end{figure}
\begin{algorithm}[t]
\DontPrintSemicolon
\caption{Asynchronous Federated Sinkhorn All-to-All}
\label{alg:fedsinkasyncallstepsize}
\small
\KwIn{Each client receives a set of local vectors $\{a_i\}, \{b_i\}$ and cost matrix $\{K_i\}$}
\KwOut{Complete Transport Plan $P$}
\SetKwBlock{Begin}{function}{end function}{
\textbf{For Each Client} $j\in[c]$, initialize $u_j, v_j$\;
\For{index in 1...$K$ iterations}{
\For{Each Client $i$}{
\emph{Inconsistently Broadcast} $\{v_{ii},i\}$
\For{Each Client $j$}{
Check if a message from $j$ has been sent, in which case \emph{Inconsistent Read} $\{v_{jj},j\}$ and store it in the $j$'th block of $v_{-i}$}
Multiply $q_i = K_i v_i$\;
Compute $u_{ii} = \alpha a_i / q_i + (1 - \alpha)u_{ii}$ with component-wise division\;
\emph{Inconsistent Broadcast} $\{u_{ii},i\}$\;
\For{Each Client $j$}{
Check if a message from $j$ has been sent, in which case \emph{Inconsistent Read} $\{u_{jj},j\}$ and store it in the $j$'th block of $v_{-i}$}
Multiply $r_i = K_i^\top u_i$\;
Compute $v_{ii} = \alpha b_i / r_i + (1 - \alpha)v_{ii}$ with component-wise division\;
}
}
}
\end{algorithm}
See Algorithm~\ref{alg:fedsinkasyncallstepsize} for a detailed implementation.
\subsection{Privacy Regime 2 - ``Star``-Network Federation}
In this second privacy regime, each client shares the communication only with a centralised server which runs the network and therefore has the cost information. For instance, a multinational insurance company which needs a centralised report at the group level, but cannot access the country-level data due to jurisdictions in those countries. It will possess the central cost matrix $C$ (namely $K$) while the country-level offices will possess their local slices $a_j$ and $b_j$.
\noindent Clients have $a_i$, $b_i$, Server has $K$.
\noindent In this case we present a \emph{Synchronous Federated Star-Network Sinkhorn} approach.
Unlike the All-to-All approach, the Star-Network approach works with a hierarchy between the nodes. A main node called the server has the role of gathering all the updated slices $u_j$ (resp. $v_j$), concatenating them into full a iteration vector $u$ (resp. $v$), and computing $q=Kv$ (resp. $r=Ku$) in their respective order. The server scatters the intermediate vectors $q$ (resp. $r$) in the same vein as $u$ (resp. $v$) are sliced across the other nodes. Meanwhile, the other nodes called the clients receive the intermediate vectors $q_j$ (resp. $r_j$) from the server, and use them along with local slices of the inputs $a_j$ and $b_j$ to compute the local iteration vectors $u_j$ and $b_j$ before sending them to back the server.
The server waits until every client has sent its updated local slice $u_j$ (resp. $v_j$) to compute the full intermediate vectors $q$ (resp. $r$), and then performs a scatter to make sure every node works with the same updated data.
Note that in the Star Network case, the cost matrix $K$ must be stored at the server level, so the All-to-All is necessary if we must store $K_j$ blocks locally.
\begin{figure}[h!]
\centering % Center the figure
\begin{tikzpicture}
% Include the PDF page as the background
\node[anchor=south west, inner sep=0] (image) at (0,0)
{\includegraphics[width=\linewidth]{figures/star-network-pdf.pdf}};
% Define the coordinate system based on the image
\begin{scope}[x={(image.south east)}, y={(image.north west)}]
\node[font=\bfseries, text=black] at (0.307, 0.787) {\fontsize{7}{10}\selectfont $r_1$};
\node[font=\bfseries, text=black] at (0.432, 0.8235) {\fontsize{7}{10}\selectfont $r_2$};
\node[font=\bfseries, text=black] at (0.5785, 0.8235) {\fontsize{7}{10}\selectfont $r_3$};
\node[font=\bfseries, text=black] at (0.712, 0.787) {\fontsize{7}{10}\selectfont $r_4$};
\node[font=\bfseries, text=black] at (0.425, 0.662) {\fontsize{7}{10}\selectfont $u_1$};
\node[font=\bfseries, text=black] at (0.48, 0.662) {\fontsize{7}{10}\selectfont $u_2$};
\node[font=\bfseries, text=black] at (0.535, 0.662) {\fontsize{7}{10}\selectfont $u_3$};
\node[font=\bfseries, text=black] at (0.59, 0.662) {\fontsize{7}{10}\selectfont $u_4$};
\node[font=\bfseries, text=black] at (0.62, 0.585) {\fontsize{7}{10}\selectfont $u$};
\node[font=\bfseries, text=black] at (0.62, 0.488) {\fontsize{7}{10}\selectfont $K$};
\node[font=\bfseries, text=black] at (0.62, 0.375) {\fontsize{7}{10}\selectfont $q$};
\node[font=\bfseries, text=black] at (0.425, 0.311) {\fontsize{7}{10}\selectfont $q_1$};
\node[font=\bfseries, text=black] at (0.48, 0.311) {\fontsize{7}{10}\selectfont $q_2$};
\node[font=\bfseries, text=black] at (0.535, 0.311) {\fontsize{7}{10}\selectfont $q_3$};
\node[font=\bfseries, text=black] at (0.59, 0.311) {\fontsize{7}{10}\selectfont $q_4$};
\node[font=\bfseries, text=black] at (0.289, 0.196) {\fontsize{7}{10}\selectfont $v_1$};
\node[font=\bfseries, text=black] at (0.407, 0.158) {\fontsize{7}{10}\selectfont $v_2$};
\node[font=\bfseries, text=black] at (0.603, 0.158) {\fontsize{7}{10}\selectfont $v_3$};
\node[font=\bfseries, text=black] at (0.728, 0.196) {\fontsize{7}{10}\selectfont $v_4$};
\end{scope}
\end{tikzpicture}
\caption{Illustration of a step in the iterative Star-Network process. The clients receive slices of $r$ from the previous iteration step, use them to compute the $u_j$ and send them to the server node. The server then uses the global $K$ to compute $q$ and scatter it to the nodes, so they can compute the slices $v_j$, and so on.}
\label{fig:star-network}
\end{figure}
See Algorithm~\ref{alg:fedsinksyncstar} for a detailed implementation, and Figure~\ref{fig:star-network} for an illustration.
%\subsubsection{Asynchronous Federated Star-Network Sinkhorn}
%In the same vein as Algorithm 3, the Asynchronous Federated Star-Network works similarly to its synchronous analog, but with inconsistent broadcasts.
\begin{algorithm}[!htb]\label{alg:2}
\DontPrintSemicolon
\caption{Synchronous Federated Star-Network Sinkhorn}
\label{alg:fedsinksyncstar}
\small
\KwIn{Each client receives a set of local vectors $\{a_i\},\{b_i\}$ and cost matrix $\{K_i\}$. Communication frequency $w$}
\KwOut{Complete Transport Plan $P$}
\SetKwBlock{Begin}{function}{end function}{
\textbf{Server} initializes $u,v$\;
\For{index in 1...$K$ iterations}{
\textbf{Each Client} $i$ \emph{Sends} $\{v_{ii},i\}$ to the \textbf{Server}
\For{Each Client $j$}{
\textbf{Server} \emph{Receives} $\{v_{jj},j\}$ and stores it in the $j$'th block of $v$.}
\textbf{Server} Computes $q=K v$\;
\textbf{Server} \emph{Scatters $q$}
\textbf{Each Client} $i$ computes $u_{ii}=a_i/q_i$ with component-wise division\;
\textbf{Each Client} $i$ \emph{Sends} $\{u_{ii},i\}$ to \textbf{Server}\;
\For{Each \textbf{Client} $j$}{
\textbf{Server} \emph{Receives} $\{u_{jj},j\}$ and stores it in the $j$'th block of $u$}
\textbf{Server} Computes $r=K^T u$\;
\textbf{Server} \emph{Scatters $r$}
\textbf{Each Client} $i$ Computes $v_{ii}=b_i/r_i$ with component-wise division\;
}
}
\end{algorithm}
\begin{remark}
We also attempted to perform so called ``local iterations'', akin to the famed local SGD / federated averaging, as the canonical federated training method for standard deep learning (see, e.g.~\cite{lin2019don} for a formulation and analysis). We implemented and tested variants of both synchronous and asynchronous All-to-All Federated Sinkhorn Algorithms that performed the blocks of computation steps (e.g., $q_i=K_iv_i$) multiple times before performing a communication step. We found that multiple local iterations were unequivocally detrimental to the performance. See Appendix B for experimental results on this approach for both synchronous and asynchronous settings.
\end{remark}
%\newpage
\subsection{Convergence Properties}\label{sec:con}
In this Section we present the theoretical guarantees of the Federated Sinkhorn algorithm for different circumstances.
For the variants with Synchronous communication, it can be observed that the sequence of iterations for the conventional centralized Sinkhorn algorithm is ultimately the same as the sequence generated by both the all-to-all and star network communication synchronous federated Sinkhorn algorithms. The convergence follows from the strong existing results, e.g.~\cite{genevay2016stochastic}.
\begin{proposition}[Synchronous Federated Sinkhorn convergence]\label{prop:sync}
Let $c\in \mathbb{N}_+$ be the number of clients, each holding local positive probability vectors $a^{(i)}, b^{(i)} \in$ $\mathbb{R}_{+}^m$, such that $a=\left[a^{(1)} ; \ldots ; a^{(c)}\right] \in \mathbb{R}_{+}^n$ and $b=\left[b^{(1)} ; \ldots ; b^{(c)}\right] \in \mathbb{R}_{+}^n$, where $n=c m$. Let $C \in \mathbb{R}_{+}^{n \times n}$ be the cost matrix with finite entries, and let $\epsilon>0$ be the regularization parameter. Assume that the Gibbs kernel $K=$ $\exp \left(-\frac{C}{\epsilon}\right) \in \mathbb{R}_{+}^{n \times n}$ has strictly positive entries. Then, the synchronous Federated Sinkhorn Algorithm \ref{alg:fedsinkasyncall} converges linearly to a unique fixed point $\left(u^*, v^*\right)$ such that the resulting transport plan $P^*=\operatorname{diag}\left(u^*\right) K \operatorname{diag}\left(v^*\right)$ is the unique solution to the entropy-regularized optimal transport problem of \eqref{eq:otprob}.
\end{proposition}
For the Asynchronous variant, the same guarantees hold, but only for sufficiently small step-size.
\begin{proposition}[Asynchronous Federated Sinkhorn all-to-all algorithm convergence]
Consider the same setting and assumptions as in Proposition~\ref{prop:sync}. Then for $0<\alpha<1$ sufficiently small, the same conclusions can be reached, that is, linear convergence to the unique solution to the entropy regularized problem.
\end{proposition}
\begin{proof}
This result can be seen as the straightforward application of~\cite{peng2016arock}, Theorems 3 and 4.
\end{proof}
\section{Experiments on Synthetic Datasets}\label{sec:nums}
In this section, we show that the federated implementation of the Sinkhorn-Knopp algorithm converges to the same results as the standard algorithm. For this, we set synthetic inputs and observe the convergence behaviour (i.e. what has an impact on it, and how many minimal iterations are needed for proper convergence).
\subsection{Study of the regularisation term $\epsilon$}
\label{epsilon_study}
For computational simplicity, we choose to observe the impact of the regularisation term on a relatively small input. The observed behaviour of the convergence should help build an intuition the impact $\epsilon$ has on convergence when considering larger inputs. Moreover, everything that follows in this section is done on the centralised setting.
We pick two distributions $a = [0.3, 0.2, 0.1, 0.4]$ and $b = [0.2, 0.3, 0.3, 0.2]$, and a cost matrix \[C=\begin{pmatrix}
0 & 1 & 2 & 3 \\
1 & 0 & 3 & 2 \\
2 & 3 & 0 & 1 \\
3 & 2 & 1 & 0 \\
\end{pmatrix}
\]
Multiple graphs are plotted in Figure~\ref{fig:errors_epsilon} to show the marginal errors and the objective function's behaviour over a specific number of iterations for different values of $\epsilon$. The figure shows that the objective function's evolution is inversely proportional to that of the marginal errors. This is not surprising, as enforcing constraints over the marginals will have an impact on the minimal value the objective function can reach.
\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{figures/errors_epsilon_0.005.png}
\includegraphics[width=\linewidth]{figures/errors_epsilon_0.001.png}
\includegraphics[width=\linewidth]{figures/errors_epsilon_0.0005.png}
\includegraphics[width=\linewidth]{figures/errors_epsilon_0.0001.png}
\caption{Marginal error and objective function value against number of iterations. The smaller epsilon is, the higher the number of iterations to reach convergence. Note that the marginal errors on $a$ and $b$ differ due to the algorithm starting with $u$.}
\label{fig:errors_epsilon}
\end{figure}
We also notice that as $\epsilon$ gets smaller, a higher number of iterations is necessary to make both marginal errors converge to a minimal value (which is around $10^{-17}$ and $10^{-51}$ on a and b, respectively).
We can therefore note the (approximate) minimal number of iteration, that we call $I_{min}$ for the objective function to converge for various value of $\epsilon$:
\begin{itemize}
\item $\epsilon = 5.10^{-3}$, $I_{min} = 300$
\item $\epsilon = 10^{-3}$, $I_{min} = 1300$
\item $\epsilon = 10^{-4}$, $I_{min} = 13,000$
\item $\epsilon = 10^{-5}$, $I_{min} = 130,000$
\end{itemize}
It seems that, at least at this scale, the number of iterations is simply inversely proportional to $\epsilon$ (i.e. dividing $\epsilon$ by $10$ requires $10$ times more iterations to converge).
Due to rounding errors, the case $\epsilon = 10^{-6}$ does not allow convergence even after $1,300,000$ and more iterations, because the iteration vectors $u$ and $v$ simply see some of their values rounded down to 0, which gets transposed into the marginals and the resulting transport map $P$.
We can see the behaviour of the objective function's minimal value for multiple values of $\epsilon$ on Figure 2. The figure shows that the minimal value seems to be converging towards 0.3 as $\epsilon$ gets smaller. This is probably because we're getting closer to the "real" value of $\langle P, C\rangle$ without the regularisation term, which we can assume here is 0.3.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/objfunc_epsilon}
\caption{Logarithmic scale is used to visualise the behaviour of the function.}
\label{fig:objfunc_epsilon}
\end{figure}
In conclusion, the regularisation term has a direct impact on the convergence speed of the algorithm in terms of iterations. As the topic of this study isn't finding the closest result to the "real" value of $\langle P, C\rangle$ but rather comparing the performance of each setting (i.e. centralised, federated synchronous and federated asynchronous), the value of $\epsilon$ won't have much importance as long as it remains the same across the comparisons.
\section{Properties of Distributed Computation}\label{sec:distrib}
\subsection{Environment}
\label{subsec:environment}
The Federated Sinkhorn-Knopp algorithm was implemented using Python, mpi4py and PyTorch.
The mpi4py library was optimised to integrate the Message Passing Interface (MPI) standard as a Python framework to enable distributed computing tasks. It supports parallel computing across multiple nodes in clusters, facilitating communication, synchronization of computations, and data exchange.
PyTorch is a widely-used deep learning framework that has been optimized to streamline the training and inference of machine learning models within Python environments. The Federated Sinkhorn algorithm, which involves multiple matrix multiplications, can benefit from PyTorch's tensor computation-focused optimization which enhances the computational performance and scalability of GPU computations.
We remark on an important consideration we encountered as far as implementation: MPI and GPU-native message passing aren't necessarily directly compatible, and special procedures are required for direct GPU-to-GPU communication. MPI works as a communication interface which is used for efficient inter-node communication and synchronization, meanwhile the GPU-native message passing leverages the parallel processing capabilities of GPUs for intra-device communication and data exchange. Moreover, even though some extensions of MPI such as CUDA-aware MPI allow for GPU-to-GPU data transfers, MPI only natively supports CPU-to-CPU communication.
For our experiments, objects are stored with a float64 precision, except for the experiments on $\epsilon$ where the precision is set to the 50th decimal.
We used a cluster of NVIDIA A100 GPU, allocating one per MPI process. The cluster is hosted by the Czech Technical University.
The code was written with a clear distinction between the two approaches:
\begin{itemize}
\item Synchronous, each sending node waits for the receiver node to be ready to receive the message.
\item Asynchronous, allow nodes to initiate communication operations and continue with other computation tasks without waiting for the communication to complete.
\end{itemize}
MPI blocking functions such as \texttt{allgather} were used for the synchronous federation, while non-blocking functions such as \texttt{Isend} and \texttt{Ireceive} were used for the asynchronous federation.
\subsection{Details on the synchronous federation}
\subsubsection{Coherence check}
Using mpi4py, the code can run on a given number of nodes. Each node will have a slice of the input vectors $a_{slice}$ and $b_{slice}$ and perform a local version of the Sinkhorn-Knopp algorithm on its respective slice, building local iteration vectors $u_{slice}$ and $v_{slice}$. Then, the updated slices will be shared among nodes to update all copies of the full vectors $u$ and $v$ across nodes, which are necessary to compute $Kv$ and $K^{T}u$.
Since each node will have the same updated copy of the full input vectors and the full cost matrix, they will return the same result which is the optimal matrix $P$.
After testing the algorithm for different numbers of nodes {$1$, $2$, $4$} with the same values for $\epsilon$ and number of iterations, the objective function returns the exact same values as in the section \nameref{epsilon_study}, which shows stability of the algorithm after synchronous federation and confirms the theoretical results.
\subsubsection{Computation and communication times}
We compared the computation time, which is the cumulative time spent on the matrix multiplications and the communication overhead, which is the time spent for communication between nodes.
For this, we fixed a number of iterations to run, and measured the different times as a function of the number of nodes. We observe in Figure~\ref{fig:sync_node_times} that even though the computation times are always smaller than the centralised computation time, the communication times are always greater than it, and increase on average with the number of nodes. The communication time increases with the number of nodes. See~\cite{quinn2004parallel} or~\cite{sterling2011hpc} for more details on how the network complexity can affect the communication times.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/sync_node_times.png}
\caption{Execution times for different number of nodes at $n=10000$ and a set number of 250 iterations. Each dot represent an individual node. Experiments performed on a cluster, allowing up to 8 local MPI processes.}
\label{fig:sync_node_times}
\end{figure}
\subsubsection{Comparison between serial and vectorised resolutions}
\cite{cuturi2013sinkhorn} showed that the Sinkhorn-Knopp algorithm could be vectorised and parallely solved for $N$ target vectors, which they refer to as target histograms. In our case, the target vector $b$ becomes a matrix, and each target histogram $b_1$, $\dots$, $b_N$ is a column of $b$.
When $N=1$, we recover the normal Sinkhorn-Knopp algorithm we introduced. When $N>1$, the computations for $N$ target histograms can be simultaneously carried out by updating a single matrix of scaling factors $u \in \mathbb{R}^{n\times N}_{+}$ instead of updating a scaling vector $u \in \mathbb{R}^{n}_{+}$ (similar for $v$).
This allows to utilise the GPU's capabilites, which is optimised for large-scale matrix operations.
A synthetic dataset with $a \in \mathbb{R}^{5000}_{+}$, $b \in \mathbb{R}^{5000 \times 500}_{+}$, $C \in \mathbb{R}^{5000 \times 5000}_{+}$ was built. $b$ having 500 columns means that we are soluving 500 OT problems. Experiments showed that convergence was met after 10 iterations. Consequently, we set a fixed number of iterations that is 15. We then isolated the computation time, and observed the time required to solve all problems.
What we observed is:
\begin{itemize}
\item It takes 0.32 seconds to run 15 iterations for 1 OT problem $b_i$
\item It takes 0.31 seconds to run 15 iterations for 500 OT problems in parallel
\item It takes 11.56 seconds to run 15 iterations for 500 OT problems sequentially
\end{itemize}
This observation confirms the expected advantage of solving multiple OT problems in parallel.
\subsubsection{Performance for different values of $N$}
\label{subsubsec:dif_val_N}
In this paragraph, we dropped the precision to float32 to allow for bigger matrices to be loaded and computed. A float64 precision doesn't allow for 5000x100000 matrices to be loaded as it saturates the memory.
Figure~\ref{fig:compute_time_multitarget} shows that for a large enough scale, it becomes advantageous to federate the computation across multiple nodes in terms of computation time. This is most likely due to the centralised setting saturating the GPU parallelism and losing in performance, which is preserved when the matrix is sliced across different nodes.
\begin{figure}[h]
\centering
\includegraphics[width=1.02\linewidth]{figures/compute_time_multitarget.png}
\caption{Isolated compute time for $N \in [1, 1000, 5000, 10000, 50000, 75000, 100000]$, compared across different synchronous federated settings.}
\label{fig:compute_time_multitarget}
\end{figure}
This result is promising as it shows that for larger scales, federating matrix computations will potentially tackle hardware limitations and allow to process bigger computations that would initially not work on a single computer.
That being said, with the current configuration, the communication time increases non-linearly with the data object's size, and completely overpasses the centralised total execution time. This has an impact on the overall performance and execution time of the algorithm.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/communication_time_multitarget.png}
\caption{Isolated communication time for $N \in [1, 1000, 5000, 10000, 50000, 75000, 100000]$, compared across different synchronous federated settings.}
\label{fig:communication_time_multitarget}
\end{figure}
However, the communication time is very sensitive to the network configuration and to how the data is transferred across nodes. At this scale, the network's state at time of execution can have a non-deterministic impact on communication smoothness, which is likely why we can observe a larger communication time for a smaller number of nodes in Figure~\ref{fig:communication_time_multitarget}. Optimising network configuration and communication could be the object of future research work.
\subsection{Details on the asynchronous federation}
\subsubsection{Non-determinism}
The asynchronous federation operates similarly to the synchronous federation but runs without a global lockstep. Each node independently sends and receives updated slices to and from other nodes in the network while continuing to work on its local slice without waiting for communication to complete.
This asynchronous behavior introduces non-determinism, as nodes may return different results depending on the network and computer resources used. It is also possible for nodes to converge at different times. However, this is not necessarily problematic because each node can meet its convergence criteria independently and return the optimal transport plan without needing to wait for others.
In Figure~\ref{fig:async_multirun}, we illustrate the non-determinism of this approach by running the asynchronous federated algorithm 15 times on a randomly generated synthetic dataset (modulo constraints) with $a$, $b$, and $C$ of size 10,000. For each run, the marginal error on $a$, defined as $\sum_{i=1}^n \left( \left(P \mathbf{1}\right)_{i} - a_{i} \right)$, was plotted against the iteration number.
This experiment demonstrates that straightforward asynchronous approaches might not currently provide a clear performance gain but can still converge in systems where regular synchronization is infeasible. This federation approach could serve as a foundation for future research on non-deterministic communication across network environments.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/async_multirun.png}
\caption{Marginal error on $a$ at the first node, from an asynchronous federation across 2 nodes for 2000 iterations. Mean value at last step is $9.10^{-5}$, and standard deviation at $2.10^{-4}$. Meanwhile, convergence was considered at $10^{-10}$. Out of 15 runs, 5 reached an asymptote at $10^{-17}$, 1 went below the convergence threshold, and 9 stayed above the threshold.}
\label{fig:async_multirun}
\end{figure}
\subsubsection{Influence of Step Size ($\alpha$)}
\label{subsubsec:stepsizesec}
The asynchronous algorithm was tested both with and without a step size parameter ($\alpha$). The absence of a step size represents a special case of the asynchronous algorithm, corresponding to $\alpha = 1$. Experiments with step size were conducted for various values of $\alpha$, while keeping initial conditions constant. For consistency, these experiments were performed on CPUs to minimize the effect of variable communication times on the total runtime (see Subsection~\ref{subsec:cpu} for CPU settings).
The previous subsection showed that, without a step size (\emph{i.e., $\alpha = 1$}), the asynchronous algorithm exhibited instability, with some runs diverging entirely. In the current subsection, we show that a step size ($\alpha < 1$) improves stability, ensuring a higher convergence likelihood tested scenarios. However, even with a step size, non-determinism persisted, as convergence speed varied significantly between runs, even under identical initial conditions. Figures~\ref{fig:times_and_error_cpu_async_stepsize10}, \ref{fig:times_and_error_cpu_async_stepsize25}, and \ref{fig:times_and_error_cpu_async_stepsize50} highlight this behavior, showing that, despite consistent convergence, the time required to reach convergence can vary widely.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/times_and_error_cpu_async_stepsize10_2.png}
\includegraphics[width=\linewidth]{figures/times_and_error_cpu_async_stepsize10_3.png}
\caption{Two runs at $k=0.1$ with identical initial conditions. We observe that convergence speeds vary a lot, especially for the 2-node setting.}
\label{fig:times_and_error_cpu_async_stepsize10}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/times_and_error_cpu_async_stepsize25_2.png}
\includegraphics[width=\linewidth]{figures/times_and_error_cpu_async_stepsize25_3.png}
\caption{Two runs at $k=0.25$ with identical initial conditions. We observe again that the 2-node setting is the setting with the highest variability.}
\label{fig:times_and_error_cpu_async_stepsize25}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/times_and_error_cpu_async_stepsize50_2.png}
\includegraphics[width=\linewidth]{figures/times_and_error_cpu_async_stepsize50_3.png}
\caption{Two runs at $k=0.5$ with identical initial conditions. This time, both the 2-node and 8-node settings show high variability. The set limit of 8000 maximum iterations cut the 2-node run, but it shows a converging trend.}
\label{fig:times_and_error_cpu_async_stepsize50}
\end{figure}
Table~\ref{tab:convg_stepsizes} shows a summary of convergence times for different settings, as well as different values of $\alpha$ for an input of size $n=10000$, ran on the CPU. We observe a clear reduction of convergence time with the increasing stepsize, which confirms the intuition.
\begin{table}[h!]
\centering
\begin{tabular}{cccc}
\toprule
& $\alpha=0.1$ & $\alpha=0.25$ & $\alpha=0.5$ \\
\midrule
2 nodes & 105.02 & 20.41 & 13.19 \\
4 nodes & 24.11 & 10.04 & 6.62 \\
8 nodes & 33.45 & 40.84 & 15.12 \\
\bottomrule
\end{tabular}
\caption{Time to convergence in seconds for different settings and values of $\alpha$. For each setting, 15 simulations were ran and the convergence times were averaged.}
\label{tab:convg_stepsizes}
\end{table}
To further investigate, an experiment was conducted using randomized inputs. While the problem size remained fixed at $n=10,000$, new random inputs were generated for each simulation. The experiment measured convergence times, the percentage of convergence, and the percentage of divergence. These observations were used to analyze the convergence behavior (in terms of time and stability) across various settings.
The experiment was divided into categories based on the following parameter values:
\begin{itemize}
\item \textbf{Convergence threshold}: Two threshold levels were considered — "Loose," where the threshold was set to $10^{-5}$, and "Tight," where the threshold was set to $10^{-12}$.
\item \textbf{Timeout limit}: Two timeout configurations were used — "Fast convergence," where the timeout was set at 10 seconds, and "Slow convergence," where the timeout was set at 1200 seconds.
\item \textbf{Divergence criterion}: If the algorithm failed to converge within 3000 iterations, it was classified as divergent.
\end{itemize}
Note that only the Asynchronous run with the best performing stepsize $\alpha$ is shown in the tables.
\begin{table}[h!]
\centering
\textbf{Synchronous All-To-All}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.77 & 1.01 \\
\textbf{tight} & 1.10 & 1.38 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 96.77 & 100.0 \\
\textbf{tight} & 97.87 & 100.0 \\
\end{tabular} \\
\hline
\textbf{\% of timeout} & \textbf{\% of divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 3.22 & 0.0 \\
\textbf{tight} & 2.13 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} \\
\hline
\end{tabular}
\vspace{0.2cm}
\textbf{Synchronous Star-Network}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 1.00 & 0.66 \\
\textbf{tight} & 1.01 & 1.69 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 95.83 & 100.0 \\
\textbf{tight} & 100.0 & 100.0 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 4.17 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} \\
\hline
\end{tabular}
\vspace{0.2cm}
\textbf{Asynchronous, $\alpha=0.5$}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.77 & 1.01 \\
\textbf{tight} & 1.10 & 1.38 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 96.77 & 100.0 \\
\textbf{tight} & 97.87 & 100.0 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 3.23 & 0.0 \\
\textbf{tight} & 2.13 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} \\
\hline
\end{tabular}
\caption{Summary table of experiments for 2 nodes.}
\label{tab:summary_stability_2}
\end{table}
% ================================================================================
\begin{table}[h!]
\centering
\textbf{Synchronous All-To-All}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 9.71 & 3.36 \\
\textbf{tight} & 4.58 & 4.89 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 66.67 & 100.0 \\
\textbf{tight} & 80.0 & 100.0 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 33.33 & 0.0 \\
\textbf{tight} & 20.0 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} \\
\hline
\end{tabular}
\vspace{0.2cm}
\textbf{Synchronous Star-Network}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 12.86 & 15.51 \\
\textbf{tight} & 16.79 & 16.31 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 66.67 & 100.0 \\
\textbf{tight} & 66.67 & 100.0 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 33.33 & 0.0 \\
\textbf{tight} & 33.33 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} \\
\hline
\end{tabular}
\vspace{0.2cm}
\textbf{Asynchronous, $\alpha=0.5$}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 4.48 & 11.21 \\
\textbf{tight} & 6.20 & 45.58 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 66.67 & 100.0 \\
\textbf{tight} & 66.67 & 66.67 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 33.33 & 0.0 \\
\textbf{tight} & 33.33 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 33.33 \\
\end{tabular} \\
\hline
\end{tabular}
\caption{Summary table of experiments for 4 nodes}
\label{tab:summary_stability_4}
\end{table}
% ================================================================================
\begin{table}[h!]
\centering
\textbf{Synchronous All-To-All}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 1.19 & 1.94 \\
\textbf{tight} & 1.79 & 1.36 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 100.0 & 100.0 \\
\textbf{tight} & 100.0 & 100.0 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} \\
\hline
\end{tabular}
\vspace{0.2cm}
\textbf{Synchronous Star-Network}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.48 & 0.49 \\
\textbf{tight} & 0.95 & 0.95 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 100.0 & 100.0 \\
\textbf{tight} & 100.0 & 100.0 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 0.0 \\
\end{tabular} \\
\hline
\end{tabular}
\vspace{0.2cm}
\textbf{Asynchronous, $\alpha=0.5$}
\\[0.1cm]
\begin{tabular}{|c|c|}
\hline
\textbf{Avg Time per Execution (s)} & \textbf{\% of Convergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 2.97 & 6.96 \\
\textbf{tight} & 4.39 & 16.75 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 80.0 & 100.0 \\
\textbf{tight} & 80.0 & 80.0 \\
\end{tabular} \\
\hline
\textbf{\% of Timeout} & \textbf{\% of Divergence} \\
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 20.0 & 0.0 \\
\textbf{tight} & 20.0 & 0.0 \\
\end{tabular} &
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Limit} & \textbf{fast} & \textbf{slow} \\
\midrule
\textbf{loose} & 0.0 & 0.0 \\
\textbf{tight} & 0.0 & 20.0 \\
\end{tabular} \\
\hline
\end{tabular}
\caption{Summary table of experiments for 8 nodes}
\label{tab:summary_stability_8}
\end{table}
Figure~\ref{fig:robust} shows the convergence robustness for different values of $\alpha$, ranging from $0.001$ to $0.5$, considering the slow and loose crietria.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/rob_2.png}
\includegraphics[width=\linewidth]{figures/rob_4.png}
\includegraphics[width=\linewidth]{figures/rob_8.png}
\caption{Percentage of simulations that converged within the experiment, for different values of $\alpha$.}
\label{fig:robust}
\end{figure}
From the experimental results presented in Table~\ref{tab:summary_stability_2}, Table~\ref{tab:summary_stability_4}, and Table~\ref{tab:summary_stability_8}, the following conclusions can be drawn:
\textbf{Synchronous All-To-All}
\begin{itemize}
\item The percentage of convergence is consistently high across all configurations, with values close to or equal to 100\%. This demonstrates that the algorithm is highly stable under synchronous all-to-all communication.
\item Execution times increase slightly for tighter thresholds compared to loose thresholds, as expected due to stricter convergence criteria. For example, execution times range from 0.77s (fast-loose) to 1.38s (slow-tight).
\item Timeout occurrences are minimal (less than 4\%) in the fast configuration and nonexistent in the slow configuration. No divergence is observed under any setting.
\end{itemize}
\textbf{Synchronous Star-Network}
\begin{itemize}
\item For the loose threshold, the convergence percentage is slightly lower in the fast configuration (95.83\%), but the tight threshold achieves 100\% convergence across all settings.
\item Execution times remain low overall, with minor increases for tighter thresholds and slow configurations (e.g., 1.69s for slow-tight).
\item Timeout occurrences are slightly higher in the fast-loose configuration (4.17\%) but are absent in other configurations. No divergence is observed in any scenario.
\end{itemize}
\textbf{Asynchronous Communication (Varying $\alpha$)}
The performance of the asynchronous approach varies significantly across step sizes ($\alpha$). For smaller step sizes ($\alpha = 0.001$ and $\alpha = 0.005$), no or minimal convergence is observed in most configurations. Specifically, $\alpha = 0.001$ fails entirely, with 100\% timeouts in the fast configuration and 100\% divergence in the slow configuration. A small improvement is noted for $\alpha = 0.005$ in the slow-loose configuration, achieving 27.27\% convergence; however, the remaining configurations still exhibit timeouts or divergence. As $\alpha$ increases to 0.2, convergence improves significantly, particularly under loose thresholds, with 72.73\% convergence in the fast-loose configuration and 100\% in the slow-loose configuration. However, tighter thresholds result in reduced convergence rates (e.g., 66.67\% in the slow-tight configuration) and occasional divergence (33.33\% in the slow-tight configuration). For $\alpha = 0.5$, convergence is consistently high across most configurations, with values near or equal to 100\%. This trend confirms that larger step sizes are more effective in achieving stability for asynchronous communication.
The best performance is observed for $\alpha = 0.5$, which achieves near-optimal convergence rates (e.g., 96.77\% for fast-loose and 100\% for slow-loose) while maintaining execution times that are comparable to or better than synchronous settings (e.g., 4.48s for fast-loose and 6.20s for fast-tight). Timeout occurrences are minimal (e.g., 3.23\% for fast-loose), and divergence is generally absent except for a small percentage (33.33\%) in the slow-tight configuration. Compared to the synchronous variant, the asynchronous approach with $\alpha = 0.5$ achieves similar stability while benefiting from reduced execution times in some configurations, particularly when thresholds are loose. This demonstrates the potential of asynchronous communication to balance stability and efficiency when the step size is appropriately chosen.
\textbf{General Observations and Insights}
\begin{itemize}
\item \textbf{Synchronous Communication:} Both all-to-all and star-network synchronous communication models exhibit robust and stable performance, achieving high convergence rates (close to or equal to 100\%) and low execution times across all configurations. Timeout and divergence are negligible, making synchronous models highly effective for achieving stability.
\item \textbf{Impact of Tight Thresholds:} For synchronous configurations, tighter thresholds lead to slightly longer execution times while maintaining high convergence rates. In asynchronous settings, however, tighter thresholds exacerbate divergence and timeout issues, particularly for smaller step sizes ($\alpha=0.001$ and $\alpha=0.005$).
\item \textbf{Influence of Step Size ($\alpha$) in Asynchronous Settings:} Smaller step sizes result in poor convergence performance, with frequent timeouts and divergence. Larger step sizes ($\alpha=0.2$ and $\alpha=0.5$) improve convergence rates and reduce execution times, bringing performance closer to synchronous models. However, divergence remains a challenge for slower nodes under tight thresholds.
\item \textbf{Overall Stability of Asynchronous Communication:} While asynchronous communication with properly tuned parameters ($\alpha=0.5$) can achieve convergence rates comparable to synchronous configurations, it is far more sensitive to parameter tuning. Poorly chosen step sizes (e.g., $\alpha=0.001$) lead to significant instability, with frequent timeouts and divergence. Additionally, asynchronous settings generally exhibit higher execution times compared to synchronous models, and no asynchronous configuration outperformed synchronous settings in terms of convergence time.
\item \textbf{Equilibrium Point for $\alpha$:} The results suggest that an optimal step size for $\alpha$ exists, balancing fast convergence and stability. For example, $\alpha=0.5$ performs well in most configurations, while $\alpha=1$ (normal asynchronous Sinkhorn) introduces a highly non-deterministic behavior, and smaller $\alpha$ values result in excessive timeouts and divergence. Proper tuning of $\alpha$ is therefore critical to achieving efficient and stable convergence in asynchronous models.
\end{itemize}
\subsubsection{Computation and communication times}
Similarly to the synchronous setting, computation and communication times were observed for a fixed amount of iterations.
Figure~\ref{fig:async_node_times} shows that the communication time is still taking over the computation time, although the computation time seems to be decreasing on average with the number of nodes. However it is always above the baseline time. This is most likely due to the non-determinism aspect of this setting.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/async_node_times.png}
\caption{Execution times for different number of nodes at $n=10000$ and a set number of 250 iterations. Experiments performed on a cluster, allowing up to 8 local MPI processes.}
\label{fig:async_node_times}
\end{figure}
\subsubsection{Delays in iterations}
\label{subsubsec:delays_in_async}
The asynchronous federation is subject to delays between messages due to each node's individual compute power and the network speed. It is possible that after a node sends an update to other nodes of the network, those nodes will have performed one or more local iterations before receiving the first node's message.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/delay_count_illustration.png}
\caption{Illustration of how the delays are counted. In this example, node B has the time to perform 3 local iterations before receiving node A's message, making it 3 iterations old. We use $\tau$ to denote the message's age.}
\label{fig:delay_count_illustration}
\end{figure}
This section is about observing these delays across multiple settings (2 nodes, 4 nodes and 8 nodes) for $n=10000$, and only one OT problem (i.e. no vectorisation).
The asynchronous federated Sinkhorn algorithm was ran 1000 times for a fixed number of iterations $T$. Each delay was recorded at the node level, as a receiver of messages from every other node. This means that the number of recorded delays from all simulations can be as high as $1000\times n_{nodes}\times n_{nodes} \times T$ (e.g. in the theoretical scenario that every node does 1 local iteration before receiving the message from every other node). Figure~\ref{fig:delay_count_illustration} gives an illustration of how the delays are counted.
Figures~\ref{fig:async_delays_to_50} and ~\ref{fig:async_delays_from_50} show that as the number of nodes increases, the distribution of delays becomes more concentrated, with the kernel density estimation (KDE) plot showing a sharper peak and shifting to the left. This indicates that the delays are becoming less variable and shorter on average as the number of nodes grows. The observed reduction in variance further supports this observation, as it signifies a decrease in the randomness and spread of delay values. This behavior can be attributed to improved efficiency or reduced contention in communication as the system scales, potentially due to better resource utilization or optimized network behavior. However, the decreasing variability also highlights a trade-off, as it suggests that the system may be converging towards a predictable performance profile, which might mask occasional extreme delays that are not captured in the narrower distribution.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/async_delays_2_nodes_u.png}
\includegraphics[width=\linewidth]{figures/async_delays_2_nodes_v.png}
\includegraphics[width=\linewidth]{figures/async_delays_4_nodes_u.png}
\includegraphics[width=\linewidth]{figures/async_delays_4_nodes_v.png}
\includegraphics[width=\linewidth]{figures/async_delays_8_nodes_u.png}
\includegraphics[width=\linewidth]{figures/async_delays_8_nodes_v.png}
\caption{KDE plot of $\tau$ for $T=500$ across different settings for 1000 simulations, for $u$ and $v$ messages. The density is plotted for $\tau \in \{1,...,50\}$.}
\label{fig:async_delays_to_50}
\end{figure}
More precisely, figure~\ref{fig:async_delays_to_50}, where we only plot the KDE for $\tau \in \{1,...,50\}$, shows that delays are common in the asynchronous federation, although most of them are close to 1 iteration (0 iteration would mean no delay). We see that the higher the number of nodes, the bigger the delay distribution skewness, but also the lower the number of iterations without delays.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/async_delays_2_nodes_u_from_50.png}
\includegraphics[width=\linewidth]{figures/async_delays_2_nodes_v_from_50.png}
\includegraphics[width=\linewidth]{figures/async_delays_4_nodes_u_from_50.png}
\includegraphics[width=\linewidth]{figures/async_delays_4_nodes_v_from_50.png}
\includegraphics[width=\linewidth]{figures/async_delays_8_nodes_u_from_50.png}
\includegraphics[width=\linewidth]{figures/async_delays_8_nodes_v_from_50.png}
\caption{KDE plot of $\tau$ for $T=500$ across different settings for 1000 simulations, for $u$ and $v$ messages. The density is plotted for values of $\tau$ greater than $50$. Note that the y-axis scale is much different from previous figure.}
\label{fig:async_delays_from_50}
\end{figure}
On the other hand, figure~\ref{fig:async_delays_from_50}, where we observe the tail of the KDE, shows that higher values of $\tau$ can appear, although at a lower frequency, which shows how sensitive the federation is to the network's connection state. Another observation is that the higher the number of node, the lower the maximum delay across iterations.
These delays significantly contribute to the non-deterministic nature of the asynchronous federation. Our experiments further reveal that, even with identical hardware, some nodes experience more delay than others. This variability in delays extends the convergence time of the asynchronous federation, as local iterations performed on inconsistently updated vectors $u$ and $v$ can mislead the algorithm, driving it away from the intended optimization trajectory.
\begin{table}[h]
\centering
\begin{tabular}{lcccc}
\toprule
Number of nodes & $\tau^{max}$ & $\tau^{min}$ & $\tau^{mean}$ & $\tau^{std}$ \\
\midrule
2 & 2477 & 1 & 2.35 & 24.89 \\
4 & 1494 & 1 & 1.18 & 10.53 \\
8 & 500 & 1 & 1.01 & 0.72 \\
\bottomrule
\end{tabular}
\caption{Statistics on delays for different number of nodes. The delays were averaged over $u$ and $v$, across 1000 simulations.}
\label{table:delays_num_nodes}
\end{table}
The observations in Table~\ref{table:delays_num_nodes} show that the higher the number of nodes, the smaller $\tau^{max}$ gets. This means that the complexity of the network potentially has a positive impact on reducing delays for the Asynchronous Federated Sinkhorn.
\subsection{Performance tests}
In order to observe the performance of the Federated Sinkhorn, the different settings were ran (when possible) with a variety of inputs, with the iteration to taken to converge and runtimes recorded.
The different inputs were:
\begin{itemize}
\item Dimension $n \in \{1k, 5k, 10k, 25k\}$
\item Number of target histograms $N \in \{1, 100, 1k, 10k\}$
\item Off-diagonal block sparsity $s \in \{0, 0.5, 0.9, 1\}$
\item Condition number $c$, in 3 classes: well conditioned, medium conditioned, ill conditioned
\end{itemize}
The stopping threshold was set at $10^{-15}$. Namely, the convergence criteria is considered met when the marginal error on $a$ crosses this value.
Tables in appendix~\ref{subsec:table_perf} give details on performance tests.
For the centralised and synchronous settings, we observe that despite the high variation in the input value, the convergence behaviour remains stable in terms of time and iterations before convergence. This is potentially due to the straightforwardness of the Sinkhorn-Knopp algorithm, which doesn't use complex matrix operations throughout the iterations.
A $\chi^2$-test was performed for different sizes, choosing the type of algorithm, the number of nodes and the condition $c$ as covariates. Table~\ref{table:chisquaretable} shows the results, which seem to indicate that there is no real trend or variation among the different settings for a GPU setting.
\begin{table}[h]
\centering
\begin{tabular}{lcc}
\toprule
Size of input & $chi^2$ statistic & $p-value$ \\
\midrule
1000 & 21459.9 & 0.44 \\
5000 & 13129.9 & 0.43 \\
10000 & 8786.0 & 0.43 \\
\bottomrule
\end{tabular}
\caption{$\chi^2$-test on the total time of execution of the algorithm for different input sizes. Covariates are type of algorithm, number of nodes and condition $c$.}
\label{table:chisquaretable}
\end{table}
However, recall in Figure~\ref{fig:compute_time_multitarget} that there is an order of magnitude increase in the communication time for 100,000 dimensional data. However, after multiple attempts, we found that the matrix storage, even in sparse settings, became infeasible for the resources available for comprehensive experiments with 1e6, 2e6 and larger dimension size. Thus, it could be that for much larger problem size, the communication time scales such as to provide an advantage to asynchronous methods, however, significantly more resources would have to be devoted to such an investigation. We can conclude that for most common settings, the pattern observed is that asynchronous communication for Federated Sinkhorn can converge if one must use such a structure by necessity, but it offers no computational advantages. However, for terabyte cost matrix data instances, this may be subject to change. For illustrative purposes, we perform similar tests on CPU computation.
\subsection{Experiments on a CPU, and potential for speed-ups in future work}
\label{subsec:cpu}
There is reason to support intuition that after a given scale for $n$ is passed, the algorithm will benefit from the federation in runtime as the computation time will be distributed across multiple nodes. With our current resources, the scale where this happen couldn't be reached yet.
Potential evidence can be seen in Figure~\ref{fig:compute_time_multitarget} from paragraph~\ref{subsubsec:dif_val_N}, where the federated setting performs better than the centralised one after 75k histograms for $b$. However, the communication time still takes over and the total runtime for the federated setting is worse than the centralised one.
Another potential evidence is if we try to run the experiments with a CPU, which slows down the computations to a point that makes the communication times smaller compared to the local computation times.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/cpu_node_times.png}
\includegraphics[width=\linewidth]{figures/cpu_node_times_25.png}
\caption{Comparison of different times across multiple synchronous settings with a set number of 250 iterations, this time ran on CPUs instead of GPUs.}
\label{fig:cpu_node_times}
\end{figure}
Figure~\ref{fig:cpu_node_times} shows that there is clearly a time benefit in federating the algorithm, as the computation time decreases with the number of nodes.
Moreover, Figures ~\ref{fig:sync_cpu_cvg} and ~\ref{fig:sync_cpu_cvg_offset} show that in the synchronous case, the marginal error's convergence speed is very sensitive to the number of nodes. However, experiments showed that the first iteration has a variable time, which is potentially due to the GPU availability when initialising and loading the variables, as well as MPI behaviour when opening the communication for the first time. As a remedy, another graph was plotted where the first recorded points were offset to the same starting value, making it easier to visualise the convergence speed between different values of nodes.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/sync_cpu_cvg.png}
\caption{Marginal error on $a$ for different synchronous settings, plotted against the elapsed time in seconds.}
\label{fig:sync_cpu_cvg}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/sync_cpu_cvg_offset.png}
\includegraphics[width=\linewidth]{figures/sync_cpu_cvg_25k.png}
\caption{Starting points where equalised to allow for a better observation of the convergence behaviour. Additionally, the algorithm was run with $n=25000$, where the same trend between node number and convergence speed is observed.}
\label{fig:sync_cpu_cvg_offset}
\end{figure}
Similar experiments on the convergence where done for the asynchronous settings. Figure~\ref{fig:async_cpu_cvg_10k} and Figure~\ref{fig:async_cpu_cvg_10k} show that even though the setting is still non-deterministic as multiple runs might have drastically different behaviours, the convergence still seems more stable than with the GPU as there are less cases of divergence observed. The computation time isn't negligible anymore compared to communication time, so there is less sensitivity to the network's state at the time of execution.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/async_cpu_cvg.png}
\includegraphics[width=\linewidth]{figures/async_cpu_cvg_2.png}
\includegraphics[width=\linewidth]{figures/async_cpu_cvg_4.png}
\caption{Three examples of runs showing the marginal error on $a$ for different asynchronous settings, plotted against the elapsed time in seconds. We see that the convergence behaviour varies drastically from one run to another. We also see in the third example that the 2-node setting cannot converge even after a time more than 20 times longer than what it took the other settings to converge. This shows again the non-deterministic aspect of the asynchronous setting.}
\label{fig:async_cpu_cvg_10k}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/async_cpu_cvg_25k_1.png}
\includegraphics[width=\linewidth]{figures/async_cpu_cvg_25k_2.png}
\caption{With $n=25000$, two runs with identical initial conditions might show completely different behaviours. The non-determinism remains observed. The 8-node run reached the set limit number of iterations before converging, but the curve shows that it would have converged eventually.}
\label{fig:async_cpu_cvg_25k}
\end{figure}
Lastly, we observe in Figures ~\ref{fig:cpu_dist_comp} and ~\ref{fig:cpu_dist_comm} that with the CPU, the computation and communication times are rather stable, except for one case of communication time that took more than 7 times the average communication. This shows that the setting is still sensitive to the network's health, although it is less impactful when the computation time isn't negligible compared to the communication time.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/cpu_dist_comp_2.png}
\includegraphics[width=\linewidth]{figures/cpu_dist_comp_4.png}
\includegraphics[width=\linewidth]{figures/cpu_dist_comp_8.png}
\caption{Distribution of computation times per node for different CPU synchronised settings. Besides from a few outliers, the computation time seems stable across multiple runs.}
\label{fig:cpu_dist_comp}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/cpu_dist_comm_2.png}
\includegraphics[width=\linewidth]{figures/cpu_dist_comm_4.png}
\includegraphics[width=\linewidth]{figures/cpu_dist_comm_8.png}
\caption{Distribution of communication times per node for different CPU synchronised settings. We notice an outlier for the 4-node setting, where a run ran more than 7 times longer than the other ones.}
\label{fig:cpu_dist_comm}
\end{figure}
By how the Sinkhorn-Knopp algorithm works, the bigger the matrix, the faster it is to converge in iteration number. If we run the problem for bigger matrices, we will observe the central and synchronised settings converge in only a few iterations, which is hardly beatable by the asynchronous one. On the other hand, if we use smaller matrices, we will need more iterations to converge, but the network impact will be greater on the convergence for the asynchronous setting, which will increase convergence time.
We can therefore develop the intuition that for more complex problems and an optimised network where the computation time surpasses the communication time, speedups can be observed for the synchronous setting. However, due to the non-deterministic nature of the asynchronous setting, we do not observe, e.g., linear or any other specific quantitative scaling law with respect to the number of nodes.
\section{Experiments on Financial Time Series}\label{sec:numt}
We consider an application in finance using the risk formulation given in~\cite{blanchet2019quantifying}
The objective of this application will be to compute the worst-case return of a portfolio, given a set of financial data spread across multiple nodes that cannot share it (but share intermediate results) using the Federated Sinkhorn algorithm.
\subsection{Blanchet and Murphy formulation to an optimal transport problem}
\subsubsection{Ambiguity Set}
~\cite{blanchet2019quantifying} introduces the concept of Ambiguity Set, that can be written as follows:
\[
\mathcal{P}_\delta = \left\{ P \in \mathcal{P}(\mathbb{R}^d) \mid W_c(P, \hat{P}) \leq \delta \right\}
\]
Where:
\begin{itemize}
\item \(\hat{P}\) is the empirical distribution obtained from historical financial data.
\item \(\mathcal{P}(\mathbb{R}^d)\) is the set of all probability distributions on \(\mathbb{R}^d\).
\item \(W_c(P, \hat{P})\) is the Wasserstein distance between \(P\) and \(\hat{P}\) with cost function \(c(x, y)\).
\item \(\delta \geq 0\) specifies the maximum allowable deviation from the empirical distribution \(\hat{P}\).
\end{itemize}
In other words, the ambiguity set is defined as the Wasserstein Ball, centered around the the empirical distribution \(\hat{P}\) and of radius \(\delta\).
\subsubsection{Worst-Case expected loss}
The worst-case expected loss is defined as:
\[
\rho_{\text{worst}} = \sup_{P \in \mathcal{P}_\delta} \mathbb{E}_P[l(X)]
\]
Where \(l:\mathbb{R}^d \rightarrow \mathbb{R}\) is a loss function (in our case, a negative portfolio return).
The worst-case expected loss \(\rho_{\text{worst}}\) is the maximum expected loss over all distributions \(P\) within the Wasserstein ball of radius \(\delta\) around \(\hat{P}\).
\subsubsection{Reformulating the Worst-Case Expected Loss}
Using duality principles from optimal transport theory as discussed in ~\cite{blanchet2019quantifying}, the worst-case expected loss can be expressed as:
\[
\rho_{\text{worst}} = \inf_{\lambda \geq 0} \left\{ \lambda \delta + \mathbb{E}_{\hat{P}} \left[ h_\lambda(X) \right] \right\}
\]
Where:
\begin{itemize}
\item \(\lambda \geq 0\) is a dual variable associated with the Wasserstein constraint.
\item \(h_\lambda(x) = \sup_{x' \in \mathbb{R}^d} \left\{ l(x') - \lambda c(x, x') \right\}\).
\item \(c(x, x')\) is the cost function used in the Wasserstein distance.
\end{itemize}
The dual formulation shifts the problem from maximizing over distributions \(P\) to minimizing over the scalar variable \(\lambda\), thus transforming the problem into a more tractable optimization.
\subsubsection{Discretising towards optimal transport}
The term \(\mathbb{E}_{\hat{P}} \left[ h_\lambda(X) \right]\) involves a supremum over \(x'\), which can be interpreted within the optimal transport framework when discretized.
Suppose we have \(n\) data points \(\{ x_i \}_{i=1}^n\) from \(\hat{P}\), each with equal probability \(\hat{p}_i = \dfrac{1}{n}\), and \(m\) potential outcomes \(\{ x'_j \}_{j=1}^m\) (these could be the same as \(\{ x_i \}\)).
For each \(x_i\), we can write:
\[
h_\lambda(x_i) = \max_{x'_j} \left\{ l(x'_j) - \lambda c(x_i, x'_j) \right\}
\]
\subsubsection{Introducing the Optimal Transport Plan}
We can introduce a joint probability distribution \(P_{ij}\) representing the transport plan, that is the amount of probability mass transported from \(x_j\) to \(x'_j\):
\begin{itemize}
\item Positivity: \(P_{ij} \geq 0\)
\item Source marginal constraint: \(\sum_{j=1}^m P_{ij} = a_{i} = \hat{p}_i = \dfrac{1}{n}, \forall\) \(i\)
\item Target marginal constraint: \(\sum_{i=1}^n P_{ij} = b_j, \forall\) \(j\)
\item Wasserstein cost constraint: \( \sum_{i=1}^n \sum_{j=1}^m P_{ij} c(x_i, x'_j) \leq \delta \)
\end{itemize}
We can reinterpret \(h_\lambda(x_i)\) as the optimal value of an optimization problem over \(P_{ij}\), by writing the following:
\[
h^{OT}_\lambda(x_i) = \max_{P_{ij} \geq 0, \sum_j P_{ij} = \hat{p}_i} \left\{ \sum_{j=1}^m P_{ij} \left( l(x'_j) - \lambda c(x_i, x'_j) \right) \right\}
\]
For a fixed \(x_i\), we maximize over \(P_{ij}\) (subject to the constraint that the total mass leaving \(x_i\) is \(\hat{p}_i\)).
The objective function \(\sum_{j=1}^m P_{ij} \left( l(x'_j) - \lambda c(x_i, x'_j) \right) \) represents the total "adjusted" loss for \(x_i\) when distributing its mass to \(x'_j\) with consideration of the cost.
Since \(\hat{p}_i\) is fixed and we are maximizing a linear function over \(P_{ij}\) with a constraint \(\sum_j P_{ij} = \hat{p}_i\), the optimal solution will allocate all mass to the \(x'_j\) that maximizes \( l(x'_j) - \lambda c(x_i, x'_j) \).
Therefore, we can write for this specific case:
\[
h^{OT}_\lambda(x_i) = \hat{p}_i \max_{x'_j} \left\{ l(x'_j) - \lambda c(x_i, x'_j) \right\}
\]
However, \(\hat{p}_i = \dfrac{1}{n}\), so:
\[
h^{OT}_\lambda(x_i) = \dfrac{1}{n} \max_{x'_j} \left\{ l(x'_j) - \lambda c(x_i, x'_j) \right\} = \dfrac{1}{n} h_\lambda(x_i)
\]
Which shows that the transport plan \(P_{ij}\) directly relates to the original problem \(h_\lambda(x_i)\).
\subsubsection{Rewriting the objective function}
Now, we consider the combined objective over all \(x_i\):
\[
\sum_j \hat{p}_{i} h_\lambda(x_i) = \sum_j \hat{p}_{i} \max_{x'_j} \left\{ l(x'_j) - \lambda c(x_i, x'_j) \right\}
\]
Instead of treating them as separate optimization problems, we can combine them into one unique optimization problem over \(P_{ij}\) for all \(i\) and \(j\).
The objective function can be rewritten:
\[
\sum_{i=1}^n \sum_{j=1}^m P_{ij} \left( l(x'_j) - \lambda c(x_i, x'_j) \right)
\]
However, everything up to now has been a maximization problem. The optimal transport problem is a minimization one. To solve this, we just need to consider the negative of the objective function, which gives an equivalent minimization problem. Thus, the optimal transport problem of our case can be written:
\[
\min_{P_{ij}} \left\{ \sum_{i=1}^n \sum_{j=1}^m P_{ij} \left( \lambda c(x_i, x'_j) - l(x'_j) \right) \right\}
\]
Subject to the constraints written in subsection 5.1.5.
\subsubsection{Rewriting the optimal transport cost}
In our context, the loss function \(l(x'_j)\) introduces an additional term in the objective. We can think of \(-l(x'_j)\) as a cost associated with transporting mass to \(x'_j\). The higher the loss at \(x'_j\), the more favorable it is to transport mass there (since our ultimate goal is to compute the worst-case expected loss).
Thus, we can introduce a new combined cost matrix defined by:
\[
C_{ij} = \lambda c(x_i, x'_j) - l(x'_j)
\]
The problem becomes a conventional optimal transport problem:
\[
\min_{P_{ij}} \left\{ \sum_{i=1}^n \sum_{j=1}^m P_{ij} C_{ij} \right\}
\]
Subject to the constraints written in subsection 5.1.5.
\subsubsection{Entropic Regularization and Skinhorn-Knopp}
To solve this problem using the Sinkhorn-Knopp algorithm, we add an entropic regularisation term:
\[
\min_{P_{ij}} \left\{ \sum_{i=1}^n \sum_{j=1}^m P_{ij} C_{ij} + \epsilon \sum_{i=1}^n \sum_{j=1}^m P_{ij} ( \log P_{ij} -1 ) \right\}
\]
Subject to the constraints written in subsection 5.1.5., where \(\epsilon > 0\) is the regularisation parameter.
If we assume \(n=m\) and name \({P}_\delta\) the ambiguity set on \(\mathcal{P}(\mathbb{R}^{n\times n})\), then it can be rewritten as the following:
\begin{mini}|l|
{P\in\mathcal{P}_\delta}{ \langle P, C\rangle + \epsilon \sum_{i,j=1}^n P_{ij} (\log P_{ij} - 1) }{}{}
\addConstraint{(P \mathbf{1})_{i}}{= a_i = \frac{1}{n}}{}
\addConstraint{P^\top \mathbf{1}}{= b}{}
\addConstraint{P}{\geq 0}{}
\end{mini}
Which is very close to the original problem of this paper, at the exception that \(P\in\mathcal{P}_\delta\) instead of \(P\in\mathbb{R}^{n\times n}\). This difference will simply require the addition of an update rule after each iteration to ensure the solution belongs to the constrained set.
\subsubsection{Enforcing the Wasserstein constraint with \(\delta\)}
The Wasserstein constraint at optimality enforces the following:
\[
\langle P^\ast, c\rangle = \sum_{i=1}^n \sum_{j=1}^m P^\ast_{ij} c(x_i, x'_j) = \delta
\]
The goal is to find \(\lambda^\ast\) such that the total cost under \(P^\ast\) is equal to \(\delta\).
Since \(C\) is a function of \(\lambda\), then \(P^\ast\) needs to be recomputed after each update of \(lambda\). Here are the iterative steps:
\begin{enumerate}
\item Use the Sinkhorn-Knopp algorithm to find \(P^\ast\)
\item Compute the total cost \(\langle P^\ast, c\rangle\)
\item Adjust \(\lambda\) accordingly:
\begin{itemize}
\item If \(\text{total cost} > \delta\), increase \(\lambda\) to penalize transport cost more
\item If \(\text{total cost} < \delta\), decrease \(\lambda\)
\item If \(\text{total cost} = \delta\), stop the loop
\end{itemize}
\item Repeat steps 1-3 until step 3 breaks the loop
\end{enumerate}
This will return a set containing \(\lambda^\ast\) and \(P^\ast\), providing the transport plan complying to both the optimal transport and the Wasserstein constraints.
\subsection{Application to find the worst-case expected loss}
\subsubsection{Setting up a theoretical example}
Consider a portfolio of assets (e.g. stocks) with historical return data. In a multi-node environment (e.g. computers from different geographies), each node will hold a subset of the data.
Let's set the portfolio loss as the loss function:
\[
l(x) = w^T x
\]
Where:
\begin{itemize}
\item \(w\) is the portfolio weight vector
\item \(x\) is the asset return vector
\end{itemize}
For the cost function, we can simply set the squared euclidean distance:
\[
c(x_i, x'_j) = (x_i - x'_j)^2
\]
Thus, we can establish the consolidated cost matrix as:
\[
C_{ij} = \lambda (x_i - x'_j)^2 + \frac{1}{n}w^T x
\]
Where $w^T x$ is divided by the dimension to ensure it doesn't overtake the first term.
If now, we:
\begin{itemize}
\item assume a vector of empirical data points \(x\) with a uniform probability distribution \(a\) (i.e. \( \forall i, a_i = \frac{1}{n} \)), distributed across nodes
\item assume target outcomes \(x'\) with a target distribution \(b\)
\item set values for \(\lambda\), \(\delta\), \(\epsilon\)
\item set weight values for \(w\), either distributed across nodes or shared
\item use the above to compute each node's local slice of \(C\)
\end{itemize}
Then we can solve the problem with the Federated Sinkhorn-Knopp algorithm to find the optimal transport plan \(P^\ast\) which allows to find the worst-case expected loss.
\subsubsection{Solution verification}
Let's suppose that we found the optimal transport plan \(P^\ast\) with \(\lambda^\ast\) such that \(\langle P^\ast, c\rangle \approx \delta\). Then we can first explicitly compute the worst-case expected loss:
\[
\rho_{\text{worst}} = \mathbb{E}_{P^\ast}[l(X)] = \sum_{i,j} P^\ast_{ij}l(x_{j})
\]
Let's show that we find the same result using the dual formulation introduced by ~\cite{blanchet2019quantifying}:
\begin{equation}
\rho_{\text{worst}} = \lambda^\ast \delta + \sum_{i} \hat{p}_{i}h_{\lambda^\ast}(x_i) \\
= \lambda^\ast \delta + \sum_{i,j} P^\ast_{ij}l(x_{j}) - \lambda^\ast \sum_{i,j} P^\ast_{ij} c(x_{i}, x'_{j})
\end{equation}
However, we showed that \( \sum_{i,j} P^\ast_{ij} c(x_{i}, x'_{j}) = \delta \) at optimality. Thus:
\[
\rho_{\text{worst}} = \lambda^\ast \delta + \sum_{i,j} P^\ast_{ij}l(x_{j}) - \lambda^\ast \delta = \sum_{i,j} P^\ast_{ij}l(x_{j})
\]
Which shows equality between both results, and confirms that a representation of the Blanchet \& Murthy risk formulation can be solved using the Siknhorn-Knopp algorithm.
\subsubsection{Conclusion of the theoretical example}
In this simple example, we showed that we can find the worst-case expected loss of a portfolio by reducing the Blanchet \& Murthy risk formulation to an optimal transport problem, and applying the Federated Sinkhorn-Knopp algorithm.
We can conclude that this approach can solve cases where the worst-case expected loss, or any other metric of interest, cannot be explicitly calculated from \(\mathbb{E}_{P}[l(X)]\).
In practice, this could be used to gauge the potential downside risk that an investor might face under adverse scenarios. Quantifying the worst-case expected loss can have usages in risk assessment, portfolio optimisation or stress testing.
The federated component takes its importance in practice, when the loss function is spread over multiple nodes. For instance, a bank with multiple sources that cannot communicate their raw data about the portfolio weights.
\subsubsection{Numerical example}
Let's apply this example to a portfolio of 3 assets, each representing a stock in tech.
We observe the past daily return for each of them:
\begin{itemize}
\item \$TICK1, with a return of \(-0.51\%\)
\item \$TICK2, with a return of \(-0.66\%\)
\item \$TICK3, with a return of \(+4.34\%\)
\end{itemize}
\noindent Which gives the asset return vector:
\[
x = \left[ -0.51,-0.66,4.34 \right]
\]
\noindent These stocks will be distributed accordingly:
\[
w = \left[ \frac{2}{5},\frac{1}{10},\frac{1}{2} \right]
\]
\noindent For this example, we set the following parameters:
\begin{itemize}
\item $n = 3 \text{ (number of index stocks in the portfolio)}$
\item $\lambda = 0.1 \text{ (Blanchet \& Murphy regularisation parameter)}$
\item $\delta = 0.01 \text{ (tolerance)} $
\item $\epsilon = 0.01 \text{ (Sinkhorn-Knopp regularisation parameter)}$
\end{itemize}
Note that in practice, the parameters should be picked under business motivations, and $\lambda$ and $\delta$ should verify the conditions exposed in the previous section.
Let's say that a team of analyst did thorough market research, and propose the following target return vector, which describes the expected return for next day:
\[
x' = \left[ 0.43,-0.8,3.86 \right]
\]
To run the algorithm, the inputs $x$ and $x'$ first need to be rescaled so that the sum of their coordinates is equal to 1. However, since the coordinates in the algorithm are typically required to be non-negative, we first perform a shift by adding a constant \( k \) to all entries of \( x \) and \( x' \) such that all elements become positive. Let:
\[
k = \max(|\min(x)|, |\min(x')|) + \epsilon
\]
where \( \epsilon > 0 \) is a small constant to ensure positivity. In this case:
\[
k = \max(0.66, 0.8) + 0.01 = 0.81
\]
Adding \( k \) to both \( x \) and \( x' \), we get the shifted vectors:
\[
x_{\text{shifted}} = x + k = \left[ 0.3, 0.15, 5.15 \right]
\]
\[
x'_{\text{shifted}} = x' + k = \left[ 1.24, 0.01, 4.67 \right]
\]
Now we can normalize these shifted vectors:
\[
\tilde{x}_{\text{shifted}, i} = \frac{x_{\text{shifted}, i}}{\sum_{j=1}^3 x_{\text{shifted}, j}}, \quad \tilde{x}'_{\text{shifted}, i} = \frac{x'_{\text{shifted}, i}}{\sum_{j=1}^3 x'_{\text{shifted}, j}}
\]
The sums are:
\[
\sum_{j=1}^3 x_{\text{shifted}, j} = 5.6, \quad \sum_{j=1}^3 x'_{\text{shifted}, j} = 5.92
\]
The final normalized vectors are:
\[
\tilde{x}_{\text{shifted}} = \left[ 0.054, 0.027, 0.912 \right]
\]
\[
\tilde{x}'_{\text{shifted}} = \left[ 0.2095, 0.0017, 0.7888 \right]
\]
This leads to the following cost matrix:
\[
C =
\begin{bmatrix}
0.164 & 0.163 & 0.214 \\
0.163 & 0.161 & 0.232 \\
0.214 & 0.232 & 0.163
\end{bmatrix}
\]
Now, let's suppose that the problem is spread across 3 offices of a bank which cannot communicate the original return data for privacy reasons, but they all have access to the analysts' predictions because they come from a global team:
\begin{itemize}
\item Office 1 has access to $x_1$ and $x'$
\item Office 2 has access to $x_2$ and $x'$
\item Office 3 has access to $x_3$ and $x'$
\end{itemize}
\noindent In terms of cost matrix, it means that the offices have the following accesses:
\begin{itemize}
\item Office 1 has access to $C_1 = \left[ C_{11}, C_{12}, C_{13} \right]$
\item Office 2 has access to $C_2 = \left[ C_{21}, C_{22}, C_{23} \right]$
\item Office 3 has access to $C_3 = \left[ C_{31}, C_{32}, C_{33} \right]$
\end{itemize}
\noindent Moreover, the cost matrix is symmetrical, which means the offices have respective access for the partial cost matrices $C^{T}_j$.
The Federated Sinkhorn-Knopp algorithm provides a functional way to compute the worst-case expected loss, which would translate how much money could be lost if we choose to believe these analysts and invest in the portfolio we defined earlier in this section.
After convergence, the algorithm returns the following optimal transport matrix:
\[
P^{*} =
\begin{bmatrix}
1.40 \times 10^{-1} & 6.78 \times 10^{-4} & 7.45 \times 10^{-38} \\
6.94 \times 10^{-2} & 1.02 \times 10^{-3} & 1.19 \times 10^{-46} \\
3.95 \times 10^{-8} & 6.19 \times 10^{-19} & 7.89 \times 10^{-1}
\end{bmatrix}
\]
Which helps us find the following result:
\[
\rho_{\text{worst}} = - \sum_{i,j} P^\ast_{ij} \left(w^T x\right)_{j} = - w^T x \sum_{i,j} P^\ast_{ij} = -0.48
\]
Namely, in this current example, listening to the analysts would put us at a risk of losing approximately 48\% of our portfolio value. This result, although extreme, shouldn't be surprising as this example has a suboptimal setting due to the arbitrary choices in parameters and target return vector.
Additionally, figure~\ref{fig:example_cvg} shows the convergence time for each setting applied to the example. We can see that all three of them converge in less than half of a second, although at different speed. This shows the usability of the Federated Sinkhorn algorithm for our real-life example.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/example_sync.png}
\includegraphics[width=\linewidth]{figures/example_sync_star.png}
\includegraphics[width=\linewidth]{figures/example_async.png}
\caption{Time of convergence for the three settings applied to this example. The Synchronous All-to-All's marginal error drops to 0 after a few iterations, probably due to rounding.}
\label{fig:example_cvg}
\end{figure}
\section{Discussion and Conclusion}\label{sec:conclusion}
We investigated the performance potential of the celebrated Sinkhorn algorithm applied on federated computing platforms. While this could change for much larger data instances than have been used in practice, that is for data vectors of size more than 1e5, we did not observe speedup in end to end solution time with increasing numbers of nodes or with asynchronous communication. However, across a variety of settings, convergence was observed to be consistent and reliable, suggesting if privacy or memory dictates the necessity of federation, then federated Sinkhorn is an acceptable procedure.
\section*{DISCLAIMER}
This paper was prepared for information purposes and is not a product of HSBC Bank Plc. or its affiliates. Neither HSBC Bank Plc. nor any of its affiliates make any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, but not limited to, the completeness, accuracy, reliability of information contained herein and the potential legal, compliance, tax or accounting effects thereof. Copyright HSBC Group 2025.
\bibliographystyle{plain}
\bibliography{refs}
\appendix
\section{Appendix}
\subsection{Local iterations}
A parameter was added in the iteration loop to allow for $w$ local iteration before broadcasting. Then, the marginal error on $a$ was plotted for 1000 iterations to see the effects on the synchronous federation.
Experiments on Figure~\ref{fig:bcast_sync} and Figure~\ref{fig:bcast_async} seem to show that adding local iterations before broadcasting only delay the convergence.
\begin{figure}[h]
\includegraphics[width=\linewidth]{figures/rank_0_bcast_sync_1.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_sync_2.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_sync_5.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_sync_10.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_sync_20.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_sync_50.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_sync_100.png}
\caption{Effect of local iterations before broadcast for different values of $w$ for synchronous federated setting, against number of iterations.}
\label{fig:bcast_sync}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/rank_0_bcast_async_1.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_async_2.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_async_5.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_async_10.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_async_20.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_async_50.png}
\includegraphics[width=\linewidth]{figures/rank_0_bcast_async_100.png}
\caption{Effect of local iterations before broadcast for different values of $w$ for asynchronous federated setting, against number of iterations.}
\label{fig:bcast_async}
\end{figure}
Moreover, by plotting the marginal error against elapsed time for different values of $w$ on Figure~\ref{fig:bcast_async}, we observe that it worsens the convergence not only in terms of iterations, but also in terms of convergence time in seconds.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/bcast_times.png}
\caption{Effect of local iterations before broadcast for different values of $w$ for synchronous federated setting, against time elapsed.}
\label{fig:bcast_time}
\end{figure}
\subsection{Tables on performance}
\label{subsec:table_perf}
Note: Some of the table might have partial data due to resource limitations while running the experiments.
\begin{table}[h]
\centering
\begin{tabular}{ccccc}
\toprule
$n$ & $s$ & $N$ & Runtime (s) & Iterations \\
\midrule
1000 & 0.0 & 1 & 0.036 & 4 \\
1000 & 0.5 & 1 & 0.090 & 4 \\
1000 & 0.9 & 1 & 0.025 & 4 \\
1000 & 1.0 & 1 & 0.027 & 4 \\
1000 & 0.0 & 100 & 0.038 & 4 \\
1000 & 0.5 & 100 & 0.082 & 4 \\
1000 & 0.9 & 100 & 0.028 & 4 \\
1000 & 1.0 & 100 & 0.078 & 4 \\
1000 & 0.0 & 1000 & 0.021 & 4 \\
1000 & 0.5 & 1000 & 0.021 & 4 \\
1000 & 0.9 & 1000 & 0.021 & 4 \\
1000 & 1.0 & 1000 & 0.155 & 4 \\
1000 & 0.0 & 10000 & 0.022 & 4 \\
1000 & 0.5 & 10000 & 0.081 & 4 \\
1000 & 0.9 & 10000 & 0.020 & 4 \\
1000 & 1.0 & 10000 & 0.023 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.022 & 3 \\
5000 & 0.5 & 1 & 0.021 & 3 \\
5000 & 0.9 & 1 & 0.024 & 3 \\
5000 & 1.0 & 1 & 0.024 & 3 \\
5000 & 0.0 & 100 & 0.019 & 3 \\
5000 & 0.5 & 100 & 0.020 & 3 \\
5000 & 0.9 & 100 & 0.019 & 3 \\
5000 & 1.0 & 100 & 0.020 & 3 \\
5000 & 0.0 & 1000 & 0.019 & 3 \\
5000 & 0.5 & 1000 & 0.019 & 3 \\
5000 & 0.9 & 1000 & 0.019 & 3 \\
5000 & 1.0 & 1000 & 0.019 & 3 \\
5000 & 0.0 & 10000 & 0.029 & 3 \\
5000 & 0.5 & 10000 & 0.020 & 3 \\
5000 & 0.9 & 10000 & 0.028 & 3 \\
5000 & 1.0 & 10000 & 0.027 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.021 & 3 \\
10000 & 0.5 & 1 & 0.026 & 3 \\
10000 & 0.9 & 1 & 0.057 & 3 \\
10000 & 1.0 & 1 & 0.166 & 3 \\
10000 & 0.0 & 100 & 0.020 & 3 \\
10000 & 0.5 & 100 & 0.072 & 3 \\
10000 & 0.9 & 100 & 0.129 & 3 \\
10000 & 1.0 & 100 & 0.094 & 3 \\
10000 & 0.0 & 1000 & 0.020 & 3 \\
10000 & 0.5 & 1000 & 0.022 & 3 \\
10000 & 0.9 & 1000 & 0.022 & 3 \\
10000 & 1.0 & 1000 & 0.022 & 3 \\
10000 & 0.0 & 10000 & 0.025 & 3 \\
10000 & 0.5 & 10000 & 0.022 & 3 \\
10000 & 0.9 & 10000 & 0.125 & 3 \\
10000 & 1.0 & 10000 & 0.024 & 3 \\
\bottomrule
\end{tabular}
\caption{Time of execution and number of iterations to convergence for $c=10$, centralised Sinkhorn.}
\label{table:big_table_cent_c10}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccc}
\toprule
$n$ & $s$ & $N$ & Runtime (s) & Iterations \\
\midrule
1000 & 0.0 & 1 & 0.024 & 4 \\
1000 & 0.5 & 1 & 0.135 & 4 \\
1000 & 0.9 & 1 & 0.025 & 4 \\
1000 & 1.0 & 1 & 0.087 & 4 \\
1000 & 0.0 & 100 & 0.028 & 4 \\
1000 & 0.5 & 100 & 0.033 & 4 \\
1000 & 0.9 & 100 & 0.066 & 4 \\
1000 & 1.0 & 100 & 0.084 & 4 \\
1000 & 0.0 & 1000 & 0.021 & 4 \\
1000 & 0.5 & 1000 & 0.022 & 4 \\
1000 & 0.9 & 1000 & 0.021 & 4 \\
1000 & 1.0 & 1000 & 0.055 & 4 \\
1000 & 0.0 & 10000 & 0.065 & 4 \\
1000 & 0.5 & 10000 & 0.020 & 4 \\
1000 & 0.9 & 10000 & 0.020 & 4 \\
1000 & 1.0 & 10000 & 0.020 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.021 & 3 \\
5000 & 0.5 & 1 & 0.022 & 3 \\
5000 & 0.9 & 1 & 0.022 & 3 \\
5000 & 1.0 & 1 & 0.022 & 3 \\
5000 & 0.0 & 100 & 0.019 & 3 \\
5000 & 0.5 & 100 & 0.020 & 3 \\
5000 & 0.9 & 100 & 0.023 & 3 \\
5000 & 1.0 & 100 & 0.020 & 3 \\
5000 & 0.0 & 1000 & 0.020 & 3 \\
5000 & 0.5 & 1000 & 0.019 & 3 \\
5000 & 0.9 & 1000 & 0.024 & 3 \\
5000 & 1.0 & 1000 & 0.024 & 3 \\
5000 & 0.0 & 10000 & 0.020 & 3 \\
5000 & 0.5 & 10000 & 0.020 & 3 \\
5000 & 0.9 & 10000 & 0.021 & 3 \\
5000 & 1.0 & 10000 & 0.072 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.033 & 3 \\
10000 & 0.5 & 1 & 0.021 & 3 \\
10000 & 0.9 & 1 & 0.021 & 3 \\
10000 & 1.0 & 1 & 0.021 & 3 \\
10000 & 0.0 & 100 & 0.019 & 3 \\
10000 & 0.5 & 100 & 0.164 & 3 \\
10000 & 0.9 & 100 & 0.023 & 3 \\
10000 & 1.0 & 100 & 0.022 & 3 \\
10000 & 0.0 & 1000 & 0.025 & 3 \\
10000 & 0.5 & 1000 & 0.020 & 3 \\
10000 & 0.9 & 1000 & 0.022 & 3 \\
10000 & 1.0 & 1000 & 0.022 & 3 \\
10000 & 0.0 & 10000 & 0.022 & 3 \\
10000 & 0.5 & 10000 & 0.133 & 3 \\
10000 & 0.9 & 10000 & 0.022 & 3 \\
10000 & 1.0 & 10000 & 0.023 & 3 \\
\bottomrule
\end{tabular}
\caption{Time of execution and number of iterations to convergence for $c=1000$, centralised Sinkhorn.}
\label{table:big_table_cent_c1k}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{cccccc}
\toprule
$n$ & $s$ & $c$ & $N$ & Runtime (s) & Iterations \\
\midrule
1000 & 0.0 & 1 & 0.031 & 4 \\
1000 & 0.5 & 1 & 0.024 & 4 \\
1000 & 0.9 & 1 & 0.071 & 4 \\
1000 & 1.0 & 1 & 0.024 & 4 \\
1000 & 0.0 & 100 & 0.028 & 4 \\
1000 & 0.5 & 100 & 0.028 & 4 \\
1000 & 0.9 & 100 & 0.029 & 4 \\
1000 & 1.0 & 100 & 0.027 & 4 \\
1000 & 0.0 & 1000 & 0.099 & 4 \\
1000 & 0.5 & 1000 & 0.116 & 4 \\
1000 & 0.9 & 1000 & 0.021 & 4 \\
1000 & 1.0 & 1000 & 0.021 & 4 \\
1000 & 0.0 & 10000 & 0.021 & 4 \\
1000 & 0.5 & 10000 & 0.081 & 4 \\
1000 & 0.9 & 10000 & 0.020 & 4 \\
1000 & 1.0 & 10000 & 0.020 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.023 & 3 \\
5000 & 0.5 & 1 & 0.021 & 3 \\
5000 & 0.9 & 1 & 0.021 & 3 \\
5000 & 1.0 & 1 & 0.023 & 3 \\
5000 & 0.0 & 100 & 0.021 & 3 \\
5000 & 0.5 & 100 & 0.022 & 3 \\
5000 & 0.9 & 100 & 0.022 & 3 \\
5000 & 1.0 & 100 & 0.022 & 3 \\
5000 & 0.0 & 1000 & 0.020 & 3 \\
5000 & 0.5 & 1000 & 0.019 & 3 \\
5000 & 0.9 & 1000 & 0.021 & 3 \\
5000 & 1.0 & 1000 & 0.019 & 3 \\
5000 & 0.0 & 10000 & 0.022 & 3 \\
5000 & 0.5 & 10000 & 0.021 & 3 \\
5000 & 0.9 & 10000 & 0.027 & 3 \\
5000 & 1.0 & 10000 & 0.024 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.118 & 3 \\
10000 & 0.5 & 1 & 0.115 & 3 \\
10000 & 0.9 & 1 & 0.021 & 3 \\
10000 & 1.0 & 1 & 0.021 & 3 \\
10000 & 0.0 & 100 & 0.020 & 3 \\
10000 & 0.5 & 100 & 0.021 & 3 \\
10000 & 0.9 & 100 & 0.019 & 3 \\
10000 & 1.0 & 100 & 0.020 & 3 \\
10000 & 0.0 & 1000 & 0.023 & 3 \\
10000 & 0.5 & 1000 & 0.026 & 3 \\
10000 & 0.9 & 1000 & 0.028 & 3 \\
10000 & 1.0 & 1000 & 0.071 & 3 \\
10000 & 0.0 & 10000 & 0.023 & 3 \\
10000 & 0.5 & 10000 & 0.035 & 3 \\
10000 & 0.9 & 10000 & 0.023 & 3 \\
10000 & 1.0 & 10000 & 0.024 & 3 \\
\bottomrule
\end{tabular}
\caption{Time of execution and number of iterations to convergence for $c=1000000$, centralised Sinkhorn.}
\label{table:big_table_cent_c1M}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.098 & 0.076 & 0.174 & 4 \\
1000 & 0.0 & 100 & 0.098 & 0.066 & 0.164 & 4 \\
1000 & 0.0 & 1000 & 0.161 & 0.140 & 0.301 & 4 \\
1000 & 0.0 & 10000 & 1.672 & 1.652 & 3.324 & 4 \\
1000 & 0.5 & 1 & 0.087 & 0.065 & 0.152 & 4 \\
1000 & 0.5 & 100 & 0.130 & 0.103 & 0.233 & 4 \\
1000 & 0.5 & 1000 & 0.157 & 0.136 & 0.294 & 4 \\
1000 & 0.5 & 10000 & 1.661 & 1.642 & 3.303 & 4 \\
1000 & 0.9 & 1 & 0.098 & 0.075 & 0.172 & 4 \\
1000 & 0.9 & 100 & 0.102 & 0.075 & 0.177 & 4 \\
1000 & 0.9 & 1000 & 0.160 & 0.139 & 0.299 & 4 \\
1000 & 0.9 & 10000 & 1.905 & 1.886 & 3.791 & 5 \\
1000 & 1.0 & 1 & 0.106 & 0.083 & 0.188 & 4 \\
1000 & 1.0 & 100 & 0.122 & 0.096 & 0.218 & 4 \\
1000 & 1.0 & 1000 & 0.153 & 0.132 & 0.284 & 4 \\
1000 & 1.0 & 10000 & 1.621 & 1.602 & 3.223 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.110 & 0.090 & 0.201 & 3 \\
5000 & 0.0 & 100 & 0.358 & 0.337 & 0.695 & 3 \\
5000 & 0.0 & 1000 & 1.606 & 1.588 & 3.194 & 3 \\
5000 & 0.0 & 10000 & 5.689 & 5.669 & 11.358 & 3 \\
5000 & 0.5 & 1 & 0.623 & 0.603 & 1.225 & 3 \\
5000 & 0.5 & 100 & 0.221 & 0.201 & 0.422 & 3 \\
5000 & 0.5 & 1000 & 1.423 & 1.402 & 2.825 & 3 \\
5000 & 0.5 & 10000 & 6.065 & 6.047 & 12.112 & 3 \\
5000 & 0.9 & 1 & 0.524 & 0.504 & 1.028 & 3 \\
5000 & 0.9 & 100 & 0.539 & 0.520 & 1.059 & 3 \\
5000 & 0.9 & 1000 & 1.490 & 1.472 & 2.962 & 3 \\
5000 & 0.9 & 10000 & 6.047 & 6.029 & 12.076 & 3 \\
5000 & 1.0 & 1 & 0.596 & 0.576 & 1.172 & 3 \\
5000 & 1.0 & 100 & 0.127 & 0.107 & 0.233 & 3 \\
5000 & 1.0 & 1000 & 1.330 & 1.311 & 2.641 & 3 \\
5000 & 1.0 & 10000 & 6.416 & 6.397 & 12.812 & 3 \\
\midrule
10000 & 0.0 & 1 & 2.746 & 2.723 & 5.469 & 3 \\
10000 & 0.0 & 100 & 3.516 & 3.489 & 7.005 & 3 \\
10000 & 0.0 & 1000 & 7.526 & 7.508 & 15.034 & 3 \\
10000 & 0.0 & 10000 & 20.533 & 20.474 & 41.007 & 3 \\
10000 & 0.5 & 1 & 4.512 & 4.492 & 9.003 & 3 \\
10000 & 0.5 & 100 & 4.662 & 4.641 & 9.304 & 3 \\
10000 & 0.5 & 1000 & 7.544 & 7.526 & 15.071 & 3 \\
10000 & 0.5 & 10000 & 18.917 & 18.898 & 37.815 & 3 \\
10000 & 0.9 & 1 & 3.577 & 3.557 & 7.134 & 3 \\
10000 & 0.9 & 100 & 3.811 & 3.794 & 7.605 & 3 \\
10000 & 0.9 & 1000 & 7.695 & 7.634 & 15.329 & 3 \\
10000 & 0.9 & 10000 & 18.175 & 18.155 & 36.331 & 3 \\
10000 & 1.0 & 100 & 4.954 & 4.934 & 9.889 & 3 \\
10000 & 1.0 & 1000 & 6.245 & 6.226 & 12.471 & 3 \\
10000 & 1.0 & 10000 & 21.689 & 21.671 & 43.361 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 2-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync2_c10}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.086 & 0.064 & 0.150 & 4 \\
1000 & 0.0 & 100 & 0.117 & 0.090 & 0.207 & 4 \\
1000 & 0.0 & 1000 & 0.194 & 0.173 & 0.366 & 4 \\
1000 & 0.0 & 10000 & 1.670 & 1.651 & 3.320 & 4 \\
1000 & 0.5 & 1 & 0.145 & 0.123 & 0.268 & 4 \\
1000 & 0.5 & 100 & 0.084 & 0.059 & 0.143 & 4 \\
1000 & 0.5 & 1000 & 0.176 & 0.154 & 0.330 & 4 \\
1000 & 0.5 & 10000 & 1.492 & 1.473 & 2.965 & 4 \\
1000 & 0.9 & 1 & 0.067 & 0.042 & 0.108 & 4 \\
1000 & 0.9 & 100 & 0.069 & 0.044 & 0.113 & 4 \\
1000 & 0.9 & 1000 & 0.148 & 0.127 & 0.275 & 4 \\
1000 & 0.9 & 10000 & 1.688 & 1.669 & 3.356 & 4 \\
1000 & 1.0 & 1 & 0.113 & 0.090 & 0.204 & 4 \\
1000 & 1.0 & 100 & 0.139 & 0.110 & 0.249 & 4 \\
1000 & 1.0 & 1000 & 0.153 & 0.132 & 0.285 & 4 \\
1000 & 1.0 & 10000 & 1.550 & 1.532 & 3.082 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.567 & 0.546 & 1.113 & 3 \\
5000 & 0.0 & 100 & 0.530 & 0.511 & 1.040 & 3 \\
5000 & 0.0 & 1000 & 1.313 & 1.292 & 2.606 & 3 \\
5000 & 0.0 & 10000 & 6.392 & 6.373 & 12.765 & 3 \\
5000 & 0.5 & 1 & 0.467 & 0.446 & 0.913 & 3 \\
5000 & 0.5 & 100 & 0.227 & 0.207 & 0.434 & 3 \\
5000 & 0.5 & 1000 & 1.588 & 1.569 & 3.156 & 3 \\
5000 & 0.5 & 10000 & 7.178 & 7.160 & 14.338 & 3 \\
5000 & 0.9 & 1 & 0.527 & 0.507 & 1.033 & 3 \\
5000 & 0.9 & 100 & 0.432 & 0.412 & 0.844 & 3 \\
5000 & 0.9 & 1000 & 1.441 & 1.421 & 2.862 & 3 \\
5000 & 0.9 & 10000 & 5.894 & 5.876 & 11.770 & 3 \\
5000 & 1.0 & 1 & 0.281 & 0.261 & 0.542 & 3 \\
5000 & 1.0 & 100 & 0.296 & 0.275 & 0.572 & 3 \\
5000 & 1.0 & 1000 & 1.457 & 1.439 & 2.896 & 3 \\
5000 & 1.0 & 10000 & 5.925 & 5.906 & 11.830 & 3 \\
\midrule
10000 & 0.0 & 1 & 5.404 & 5.380 & 10.784 & 3 \\
10000 & 0.0 & 100 & 4.955 & 4.934 & 9.889 & 3 \\
10000 & 0.0 & 1000 & 8.058 & 8.039 & 16.098 & 3 \\
10000 & 0.0 & 10000 & 17.819 & 17.800 & 35.619 & 3 \\
10000 & 0.5 & 1 & 3.841 & 3.818 & 7.659 & 3 \\
10000 & 0.5 & 100 & 4.479 & 4.461 & 8.940 & 3 \\
10000 & 0.5 & 1000 & 10.026 & 10.008 & 20.034 & 3 \\
10000 & 0.5 & 10000 & 18.052 & 18.034 & 36.086 & 3 \\
10000 & 0.9 & 1 & 2.668 & 2.644 & 5.312 & 3 \\
10000 & 0.9 & 100 & 3.750 & 3.729 & 7.479 & 3 \\
10000 & 0.9 & 1000 & 7.775 & 7.756 & 15.531 & 3 \\
10000 & 0.9 & 10000 & 20.619 & 20.597 & 41.216 & 3 \\
10000 & 1.0 & 1 & 5.223 & 5.200 & 10.423 & 3 \\
10000 & 1.0 & 100 & 5.022 & 5.002 & 10.024 & 3 \\
10000 & 1.0 & 1000 & 7.029 & 7.010 & 14.039 & 3 \\
10000 & 1.0 & 10000 & 20.436 & 20.380 & 40.816 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 2-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync2_c1k}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.093 & 0.070 & 0.163 & 4 \\
1000 & 0.0 & 100 & 0.075 & 0.049 & 0.124 & 4 \\
1000 & 0.0 & 1000 & 0.176 & 0.155 & 0.331 & 4 \\
1000 & 0.0 & 10000 & 1.608 & 1.589 & 3.197 & 4 \\
1000 & 0.5 & 1 & 0.087 & 0.061 & 0.148 & 4 \\
1000 & 0.5 & 100 & 0.114 & 0.087 & 0.201 & 4 \\
1000 & 0.5 & 1000 & 0.166 & 0.145 & 0.311 & 4 \\
1000 & 0.5 & 10000 & 1.493 & 1.474 & 2.967 & 4 \\
1000 & 0.9 & 1 & 0.077 & 0.054 & 0.131 & 4 \\
1000 & 0.9 & 100 & 0.134 & 0.108 & 0.242 & 4 \\
1000 & 0.9 & 1000 & 0.171 & 0.150 & 0.321 & 4 \\
1000 & 0.9 & 10000 & 1.670 & 1.651 & 3.321 & 4 \\
1000 & 1.0 & 1 & 0.083 & 0.060 & 0.143 & 4 \\
1000 & 1.0 & 100 & 0.129 & 0.103 & 0.232 & 4 \\
1000 & 1.0 & 1000 & 0.145 & 0.125 & 0.271 & 4 \\
1000 & 1.0 & 10000 & 1.803 & 1.785 & 3.588 & 4 \\
\midrule
5000 & 0.0 & 1 & 1.401 & 1.381 & 2.782 & 3 \\
5000 & 0.0 & 100 & 0.543 & 0.519 & 1.062 & 3 \\
5000 & 0.0 & 1000 & 0.945 & 0.927 & 1.872 & 3 \\
5000 & 0.0 & 10000 & 6.029 & 6.009 & 12.038 & 3 \\
5000 & 0.5 & 1 & 0.476 & 0.456 & 0.931 & 3 \\
5000 & 0.5 & 100 & 0.519 & 0.501 & 1.020 & 3 \\
5000 & 0.5 & 1000 & 1.570 & 1.552 & 3.122 & 3 \\
5000 & 0.5 & 10000 & 5.819 & 5.800 & 11.619 & 3 \\
5000 & 0.9 & 1 & 0.740 & 0.721 & 1.461 & 3 \\
5000 & 0.9 & 100 & 0.406 & 0.388 & 0.794 & 3 \\
5000 & 0.9 & 1000 & 1.800 & 1.779 & 3.579 & 3 \\
5000 & 0.9 & 10000 & 6.721 & 6.703 & 13.424 & 3 \\
5000 & 1.0 & 1 & 0.416 & 0.396 & 0.812 & 3 \\
5000 & 1.0 & 100 & 0.335 & 0.317 & 0.652 & 3 \\
5000 & 1.0 & 1000 & 1.370 & 1.349 & 2.719 & 3 \\
5000 & 1.0 & 10000 & 6.114 & 6.095 & 12.209 & 3 \\
\midrule
10000 & 0.0 & 1 & 2.897 & 2.877 & 5.774 & 3 \\
10000 & 0.0 & 100 & 4.538 & 4.517 & 9.055 & 3 \\
10000 & 0.0 & 1000 & 7.681 & 7.662 & 15.343 & 3 \\
10000 & 0.0 & 10000 & 19.500 & 19.480 & 38.980 & 3 \\
10000 & 0.5 & 1 & 5.731 & 5.711 & 11.442 & 3 \\
10000 & 0.5 & 100 & 4.811 & 4.790 & 9.601 & 3 \\
10000 & 0.5 & 1000 & 8.068 & 8.051 & 16.119 & 3 \\
10000 & 0.5 & 10000 & 20.603 & 20.582 & 41.185 & 3 \\
10000 & 0.9 & 1 & 2.684 & 2.659 & 5.343 & 3 \\
10000 & 0.9 & 100 & 4.478 & 4.458 & 8.936 & 3 \\
10000 & 0.9 & 1000 & 7.503 & 7.485 & 14.988 & 3 \\
10000 & 0.9 & 10000 & 19.658 & 19.601 & 39.258 & 3 \\
10000 & 1.0 & 1 & 4.508 & 4.485 & 8.993 & 3 \\
10000 & 1.0 & 100 & 3.969 & 3.951 & 7.920 & 3 \\
10000 & 1.0 & 1000 & 7.103 & 7.048 & 14.151 & 3 \\
10000 & 1.0 & 10000 & 17.528 & 17.477 & 35.004 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 2-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync2_c1M}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.110 & 0.086 & 0.195 & 4 \\
1000 & 0.0 & 100 & 0.121 & 0.088 & 0.209 & 4 \\
1000 & 0.0 & 1000 & 0.270 & 0.247 & 0.517 & 4 \\
1000 & 0.0 & 10000 & 1.968 & 1.946 & 3.914 & 4 \\
1000 & 0.5 & 1 & 0.096 & 0.072 & 0.169 & 4 \\
1000 & 0.5 & 100 & 0.110 & 0.083 & 0.192 & 4 \\
1000 & 0.5 & 1000 & 0.216 & 0.193 & 0.410 & 4 \\
1000 & 0.5 & 10000 & 1.925 & 1.901 & 3.826 & 4 \\
1000 & 0.9 & 1 & 0.269 & 0.241 & 0.510 & 4 \\
1000 & 0.9 & 100 & 0.269 & 0.240 & 0.509 & 4 \\
1000 & 0.9 & 1000 & 0.240 & 0.217 & 0.457 & 4 \\
1000 & 0.9 & 10000 & 2.299 & 2.278 & 4.577 & 4 \\
1000 & 1.0 & 1 & 0.803 & 0.778 & 1.582 & 4 \\
1000 & 1.0 & 100 & 0.114 & 0.087 & 0.201 & 4 \\
1000 & 1.0 & 1000 & 0.274 & 0.251 & 0.524 & 4 \\
1000 & 1.0 & 10000 & 1.770 & 1.743 & 3.512 & 4 \\
\midrule
5000 & 0.0 & 1 & 1.499 & 1.468 & 2.967 & 3 \\
5000 & 0.0 & 100 & 2.078 & 2.054 & 4.132 & 3 \\
5000 & 0.0 & 1000 & 2.021 & 2.001 & 4.022 & 3 \\
5000 & 0.0 & 10000 & 19.034 & 19.012 & 38.046 & 3 \\
5000 & 0.5 & 1 & 2.066 & 2.042 & 4.107 & 3 \\
5000 & 0.5 & 100 & 1.556 & 1.531 & 3.087 & 3 \\
5000 & 0.5 & 1000 & 1.978 & 1.955 & 3.933 & 3 \\
5000 & 0.5 & 10000 & 24.656 & 24.634 & 49.290 & 3 \\
5000 & 0.9 & 1 & 1.738 & 1.713 & 3.451 & 3 \\
5000 & 0.9 & 100 & 1.784 & 1.758 & 3.542 & 3 \\
5000 & 0.9 & 1000 & 1.595 & 1.575 & 3.170 & 3 \\
5000 & 0.9 & 10000 & 13.070 & 13.051 & 26.122 & 3 \\
5000 & 1.0 & 1 & 1.834 & 1.807 & 3.641 & 3 \\
5000 & 1.0 & 100 & 1.644 & 1.591 & 3.235 & 3 \\
5000 & 1.0 & 1000 & 1.726 & 1.706 & 3.433 & 3 \\
5000 & 1.0 & 10000 & 9.291 & 9.259 & 18.550 & 3 \\
\midrule
10000 & 0.0 & 1 & 12.451 & 12.428 & 24.879 & 3 \\
10000 & 0.0 & 100 & 0.977 & 0.955 & 1.932 & 3 \\
10000 & 0.0 & 1000 & 5.382 & 5.354 & 10.736 & 3 \\
10000 & 0.0 & 10000 & 22.154 & 22.134 & 44.288 & 3 \\
10000 & 0.5 & 1 & 11.759 & 11.735 & 23.494 & 3 \\
10000 & 0.5 & 100 & 12.595 & 12.563 & 25.158 & 3 \\
10000 & 0.5 & 1000 & 10.570 & 10.547 & 21.116 & 3 \\
10000 & 0.5 & 10000 & 69.328 & 69.307 & 138.635 & 3 \\
10000 & 0.9 & 1 & 9.753 & 9.730 & 19.483 & 3 \\
10000 & 0.9 & 100 & 11.401 & 11.380 & 22.781 & 3 \\
10000 & 0.9 & 1000 & 13.461 & 13.439 & 26.900 & 3 \\
10000 & 0.9 & 10000 & 33.294 & 33.271 & 66.565 & 3 \\
10000 & 1.0 & 1 & 3.688 & 3.650 & 7.338 & 3 \\
10000 & 1.0 & 100 & 13.574 & 13.553 & 27.127 & 3 \\
10000 & 1.0 & 1000 & 6.861 & 6.842 & 13.703 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 4-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync4_c10}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.205 & 0.180 & 0.385 & 4 \\
1000 & 0.0 & 100 & 0.225 & 0.127 & 0.352 & 4 \\
1000 & 0.0 & 1000 & 0.205 & 0.182 & 0.387 & 4 \\
1000 & 0.0 & 10000 & 2.118 & 2.097 & 4.215 & 4 \\
1000 & 0.5 & 1 & 0.156 & 0.129 & 0.285 & 4 \\
1000 & 0.5 & 100 & 0.208 & 0.181 & 0.389 & 4 \\
1000 & 0.5 & 1000 & 0.285 & 0.262 & 0.547 & 4 \\
1000 & 0.5 & 10000 & 1.794 & 1.767 & 3.562 & 4 \\
1000 & 0.9 & 1 & 0.097 & 0.073 & 0.170 & 4 \\
1000 & 0.9 & 100 & 0.105 & 0.077 & 0.182 & 4 \\
1000 & 0.9 & 1000 & 0.231 & 0.209 & 0.440 & 4 \\
1000 & 0.9 & 10000 & 1.661 & 1.638 & 3.300 & 4 \\
1000 & 1.0 & 1 & 0.135 & 0.111 & 0.246 & 4 \\
1000 & 1.0 & 100 & 0.218 & 0.160 & 0.378 & 4 \\
1000 & 1.0 & 1000 & 0.209 & 0.186 & 0.396 & 4 \\
1000 & 1.0 & 10000 & 1.519 & 1.497 & 3.016 & 4 \\
\midrule
5000 & 0.0 & 1 & 1.725 & 1.695 & 3.420 & 3 \\
5000 & 0.0 & 100 & 1.936 & 1.912 & 3.848 & 3 \\
5000 & 0.0 & 1000 & 2.356 & 2.336 & 4.692 & 3 \\
5000 & 0.0 & 10000 & 19.369 & 19.335 & 38.704 & 3 \\
5000 & 0.5 & 1 & 1.696 & 1.668 & 3.364 & 3 \\
5000 & 0.5 & 100 & 3.211 & 3.110 & 6.321 & 3 \\
5000 & 0.5 & 1000 & 2.128 & 2.107 & 4.236 & 3 \\
5000 & 0.5 & 10000 & 26.559 & 26.538 & 53.096 & 3 \\
5000 & 0.9 & 1 & 1.697 & 1.668 & 3.365 & 3 \\
5000 & 0.9 & 100 & 2.508 & 2.438 & 4.946 & 3 \\
5000 & 0.9 & 1000 & 1.676 & 1.655 & 3.332 & 3 \\
5000 & 0.9 & 10000 & 9.214 & 9.195 & 18.409 & 3 \\
5000 & 1.0 & 1 & 2.107 & 2.078 & 4.185 & 3 \\
5000 & 1.0 & 100 & 1.827 & 1.736 & 3.563 & 3 \\
5000 & 1.0 & 1000 & 2.015 & 1.974 & 3.989 & 3 \\
5000 & 1.0 & 10000 & 15.136 & 15.115 & 30.251 & 3 \\
\midrule
10000 & 0.0 & 1 & 12.257 & 12.231 & 24.488 & 3 \\
10000 & 0.0 & 100 & 13.500 & 13.477 & 26.977 & 3 \\
10000 & 0.0 & 1000 & 6.464 & 6.437 & 12.901 & 3 \\
10000 & 0.0 & 10000 & 38.702 & 38.681 & 77.383 & 3 \\
10000 & 0.5 & 1 & 11.657 & 11.635 & 23.292 & 3 \\
10000 & 0.5 & 100 & 12.881 & 12.859 & 25.741 & 3 \\
10000 & 0.5 & 1000 & 12.901 & 12.881 & 25.782 & 3 \\
10000 & 0.5 & 10000 & 65.977 & 65.863 & 131.840 & 3 \\
10000 & 0.9 & 1 & 11.255 & 11.233 & 22.488 & 3 \\
10000 & 0.9 & 100 & 13.261 & 13.240 & 26.500 & 3 \\
10000 & 0.9 & 1000 & 10.479 & 10.454 & 20.933 & 3 \\
10000 & 0.9 & 10000 & 44.303 & 44.282 & 88.586 & 3 \\
10000 & 1.0 & 1 & 2.677 & 2.652 & 5.329 & 3 \\
10000 & 1.0 & 100 & 12.384 & 12.365 & 24.749 & 3 \\
10000 & 1.0 & 1000 & 8.559 & 8.535 & 17.094 & 3 \\
10000 & 1.0 & 10000 & 98.092 & 97.981 & 196.074 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 4-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync4_c1k}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.102 & 0.077 & 0.179 & 4 \\
1000 & 0.0 & 100 & 0.420 & 0.391 & 0.811 & 4 \\
1000 & 0.0 & 1000 & 0.230 & 0.204 & 0.433 & 4 \\
1000 & 0.0 & 10000 & 1.846 & 1.822 & 3.668 & 4 \\
1000 & 0.5 & 1 & 0.131 & 0.106 & 0.236 & 4 \\
1000 & 0.5 & 100 & 0.150 & 0.120 & 0.271 & 4 \\
1000 & 0.5 & 1000 & 0.220 & 0.197 & 0.418 & 4 \\
1000 & 0.5 & 10000 & 1.447 & 1.426 & 2.873 & 4 \\
1000 & 0.9 & 1 & 0.160 & 0.135 & 0.295 & 4 \\
1000 & 0.9 & 100 & 0.278 & 0.151 & 0.429 & 4 \\
1000 & 0.9 & 1000 & 0.237 & 0.214 & 0.451 & 4 \\
1000 & 0.9 & 10000 & 1.695 & 1.674 & 3.368 & 4 \\
1000 & 1.0 & 1 & 0.225 & 0.201 & 0.427 & 4 \\
1000 & 1.0 & 100 & 0.117 & 0.090 & 0.207 & 4 \\
1000 & 1.0 & 1000 & 0.220 & 0.197 & 0.417 & 4 \\
1000 & 1.0 & 10000 & 2.056 & 2.014 & 4.070 & 4 \\
\midrule
5000 & 0.0 & 1 & 1.683 & 1.652 & 3.334 & 3 \\
5000 & 0.0 & 100 & 1.939 & 1.916 & 3.855 & 3 \\
5000 & 0.0 & 1000 & 2.927 & 2.903 & 5.831 & 3 \\
5000 & 0.0 & 10000 & 11.470 & 11.450 & 22.920 & 3 \\
5000 & 0.5 & 1 & 2.591 & 2.566 & 5.157 & 3 \\
5000 & 0.5 & 100 & 1.677 & 1.651 & 3.328 & 3 \\
5000 & 0.5 & 1000 & 3.053 & 3.033 & 6.086 & 3 \\
5000 & 0.5 & 10000 & 16.155 & 16.127 & 32.282 & 3 \\
5000 & 0.9 & 1 & 1.509 & 1.484 & 2.993 & 3 \\
5000 & 0.9 & 100 & 2.821 & 2.796 & 5.617 & 3 \\
5000 & 0.9 & 1000 & 2.052 & 2.032 & 4.084 & 3 \\
5000 & 0.9 & 10000 & 8.732 & 8.713 & 17.445 & 3 \\
5000 & 1.0 & 1 & 2.032 & 2.007 & 4.040 & 3 \\
5000 & 1.0 & 100 & 1.911 & 1.841 & 3.752 & 3 \\
5000 & 1.0 & 1000 & 2.832 & 2.812 & 5.644 & 3 \\
5000 & 1.0 & 10000 & 20.591 & 20.570 & 41.162 & 3 \\
\midrule
10000 & 0.0 & 1 & 13.201 & 13.178 & 26.379 & 3 \\
10000 & 0.0 & 100 & 10.727 & 10.704 & 21.431 & 3 \\
10000 & 0.0 & 1000 & 6.312 & 6.289 & 12.601 & 3 \\
10000 & 0.0 & 10000 & 55.411 & 55.387 & 110.799 & 3 \\
10000 & 0.5 & 1 & 9.128 & 9.080 & 18.208 & 3 \\
10000 & 0.5 & 100 & 12.913 & 12.878 & 25.790 & 3 \\
10000 & 0.5 & 1000 & 12.414 & 12.390 & 24.804 & 3 \\
10000 & 0.5 & 10000 & 36.378 & 36.356 & 72.734 & 3 \\
10000 & 0.9 & 1 & 2.198 & 2.177 & 4.375 & 3 \\
10000 & 0.9 & 100 & 2.387 & 2.364 & 4.751 & 3 \\
10000 & 0.9 & 1000 & 14.285 & 14.265 & 28.550 & 3 \\
10000 & 0.9 & 10000 & 66.135 & 66.112 & 132.247 & 3 \\
10000 & 1.0 & 1 & 3.172 & 3.144 & 6.315 & 3 \\
10000 & 1.0 & 100 & 2.951 & 2.921 & 5.872 & 3 \\
10000 & 1.0 & 1000 & 14.615 & 14.595 & 29.210 & 3 \\
10000 & 1.0 & 10000 & 93.768 & 93.745 & 187.513 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 4-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync4_c1M}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.586 & 0.552 & 1.139 & 4 \\
1000 & 0.0 & 100 & 0.213 & 0.182 & 0.396 & 4 \\
1000 & 0.0 & 1000 & 0.335 & 0.309 & 0.643 & 4 \\
1000 & 0.0 & 10000 & 1.999 & 1.981 & 3.980 & 4 \\
1000 & 0.5 & 1 & 0.256 & 0.229 & 0.485 & 4 \\
1000 & 0.5 & 100 & 0.212 & 0.188 & 0.400 & 4 \\
1000 & 0.5 & 1000 & 0.401 & 0.374 & 0.775 & 4 \\
1000 & 0.5 & 10000 & 2.015 & 1.997 & 4.012 & 4 \\
1000 & 0.9 & 1 & 0.643 & 0.616 & 1.259 & 4 \\
1000 & 0.9 & 100 & 0.550 & 0.525 & 1.075 & 4 \\
1000 & 0.9 & 1000 & 0.604 & 0.578 & 1.182 & 4 \\
1000 & 0.9 & 10000 & 2.127 & 2.108 & 4.235 & 4 \\
1000 & 1.0 & 1 & 0.205 & 0.178 & 0.383 & 4 \\
1000 & 1.0 & 100 & 0.216 & 0.191 & 0.407 & 4 \\
1000 & 1.0 & 1000 & 0.428 & 0.403 & 0.831 & 4 \\
1000 & 1.0 & 10000 & 1.326 & 1.308 & 2.634 & 4 \\
\midrule
5000 & 0.0 & 1 & 1.834 & 1.810 & 3.643 & 3 \\
5000 & 0.0 & 100 & 3.054 & 3.032 & 6.086 & 3 \\
5000 & 0.0 & 1000 & 1.309 & 1.291 & 2.600 & 3 \\
5000 & 0.0 & 10000 & 14.011 & 13.993 & 28.004 & 3 \\
5000 & 0.5 & 1 & 1.735 & 1.711 & 3.447 & 3 \\
5000 & 0.5 & 100 & 2.477 & 2.455 & 4.932 & 3 \\
5000 & 0.5 & 1000 & 1.760 & 1.742 & 3.503 & 3 \\
5000 & 0.5 & 10000 & 10.628 & 10.610 & 21.238 & 3 \\
5000 & 0.9 & 1 & 1.860 & 1.836 & 3.696 & 3 \\
5000 & 0.9 & 100 & 2.441 & 2.419 & 4.860 & 3 \\
5000 & 0.9 & 1000 & 1.382 & 1.361 & 2.743 & 3 \\
5000 & 0.9 & 10000 & 10.762 & 10.744 & 21.505 & 3 \\
5000 & 1.0 & 1 & 0.970 & 0.946 & 1.915 & 3 \\
5000 & 1.0 & 100 & 2.605 & 2.583 & 5.188 & 3 \\
5000 & 1.0 & 1000 & 1.357 & 1.339 & 2.695 & 3 \\
5000 & 1.0 & 10000 & 12.865 & 12.844 & 25.709 & 3 \\
\midrule
10000 & 0.0 & 1 & 43.513 & 43.483 & 86.997 & 3 \\
10000 & 0.5 & 1 & 3.147 & 3.123 & 6.269 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 8-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync8_c10}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.173 & 0.146 & 0.320 & 4 \\
1000 & 0.0 & 100 & 0.466 & 0.441 & 0.907 & 4 \\
1000 & 0.0 & 1000 & 0.497 & 0.471 & 0.967 & 4 \\
1000 & 0.0 & 10000 & 1.865 & 1.846 & 3.711 & 4 \\
1000 & 0.5 & 1 & 0.375 & 0.347 & 0.722 & 4 \\
1000 & 0.5 & 100 & 0.465 & 0.440 & 0.905 & 4 \\
1000 & 0.5 & 1000 & 0.794 & 0.768 & 1.562 & 4 \\
1000 & 0.5 & 10000 & 1.825 & 1.806 & 3.631 & 4 \\
1000 & 0.9 & 1 & 0.219 & 0.192 & 0.411 & 4 \\
1000 & 0.9 & 100 & 0.310 & 0.284 & 0.594 & 4 \\
1000 & 0.9 & 1000 & 0.644 & 0.618 & 1.261 & 4 \\
1000 & 0.9 & 10000 & 2.136 & 2.118 & 4.254 & 4 \\
1000 & 1.0 & 1 & 0.208 & 0.181 & 0.389 & 4 \\
1000 & 1.0 & 100 & 0.590 & 0.566 & 1.156 & 4 \\
1000 & 1.0 & 1000 & 0.891 & 0.865 & 1.756 & 4 \\
1000 & 1.0 & 10000 & 1.848 & 1.830 & 3.679 & 4 \\
\midrule
5000 & 0.0 & 1 & 2.587 & 2.562 & 5.149 & 3 \\
5000 & 0.0 & 100 & 2.506 & 2.483 & 4.990 & 3 \\
5000 & 0.0 & 1000 & 1.612 & 1.594 & 3.206 & 3 \\
5000 & 0.0 & 10000 & 12.214 & 12.195 & 24.409 & 3 \\
5000 & 0.5 & 1 & 1.854 & 1.831 & 3.685 & 3 \\
5000 & 0.5 & 100 & 2.470 & 2.448 & 4.918 & 3 \\
5000 & 0.5 & 1000 & 1.567 & 1.549 & 3.116 & 3 \\
5000 & 0.5 & 10000 & 10.333 & 10.315 & 20.648 & 3 \\
5000 & 0.9 & 1 & 2.039 & 2.012 & 4.051 & 3 \\
5000 & 0.9 & 100 & 2.278 & 2.255 & 4.533 & 3 \\
5000 & 0.9 & 1000 & 2.693 & 2.675 & 5.368 & 3 \\
5000 & 0.9 & 10000 & 10.148 & 10.130 & 20.278 & 3 \\
5000 & 1.0 & 1 & 0.922 & 0.898 & 1.821 & 3 \\
5000 & 1.0 & 100 & 1.032 & 1.010 & 2.042 & 3 \\
5000 & 1.0 & 1000 & 1.479 & 1.459 & 2.938 & 3 \\
5000 & 1.0 & 10000 & 6.895 & 6.876 & 13.771 & 3 \\
\midrule
10000 & 0.0 & 1 & 10.853 & 10.827 & 21.681 & 3 \\
10000 & 0.5 & 1 & 3.550 & 3.520 & 7.071 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 8-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync8_c1k}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.303 & 0.277 & 0.580 & 4 \\
1000 & 0.0 & 100 & 0.418 & 0.391 & 0.809 & 4 \\
1000 & 0.0 & 1000 & 0.734 & 0.709 & 1.443 & 4 \\
1000 & 0.0 & 10000 & 2.062 & 2.043 & 4.105 & 4 \\
1000 & 0.5 & 1 & 1.144 & 1.118 & 2.262 & 4 \\
1000 & 0.5 & 100 & 0.200 & 0.176 & 0.376 & 4 \\
1000 & 0.5 & 1000 & 0.564 & 0.539 & 1.103 & 4 \\
1000 & 0.5 & 10000 & 1.950 & 1.931 & 3.881 & 4 \\
1000 & 0.9 & 1 & 0.283 & 0.257 & 0.540 & 4 \\
1000 & 0.9 & 100 & 0.214 & 0.184 & 0.397 & 4 \\
1000 & 0.9 & 1000 & 0.707 & 0.681 & 1.388 & 4 \\
1000 & 0.9 & 10000 & 1.964 & 1.945 & 3.909 & 4 \\
1000 & 1.0 & 1 & 0.297 & 0.270 & 0.566 & 4 \\
1000 & 1.0 & 100 & 0.187 & 0.162 & 0.349 & 4 \\
1000 & 1.0 & 1000 & 0.614 & 0.588 & 1.202 & 4 \\
1000 & 1.0 & 10000 & 1.969 & 1.951 & 3.921 & 4 \\
\midrule
5000 & 0.0 & 1 & 1.692 & 1.663 & 3.355 & 3 \\
5000 & 0.0 & 100 & 3.058 & 3.032 & 6.090 & 3 \\
5000 & 0.0 & 1000 & 1.762 & 1.744 & 3.505 & 3 \\
5000 & 0.0 & 10000 & 10.239 & 10.221 & 20.459 & 3 \\
5000 & 0.5 & 1 & 1.929 & 1.905 & 3.833 & 3 \\
5000 & 0.5 & 100 & 2.396 & 2.374 & 4.770 & 3 \\
5000 & 0.5 & 1000 & 1.499 & 1.477 & 2.976 & 3 \\
5000 & 0.5 & 10000 & 10.144 & 10.125 & 20.270 & 3 \\
5000 & 0.9 & 1 & 1.796 & 1.772 & 3.568 & 3 \\
5000 & 0.9 & 100 & 2.894 & 2.872 & 5.766 & 3 \\
5000 & 0.9 & 1000 & 1.592 & 1.571 & 3.163 & 3 \\
5000 & 0.9 & 10000 & 10.386 & 10.367 & 20.752 & 3 \\
5000 & 1.0 & 1 & 2.244 & 2.220 & 4.464 & 3 \\
5000 & 1.0 & 100 & 1.668 & 1.645 & 3.313 & 3 \\
5000 & 1.0 & 1000 & 1.694 & 1.675 & 3.369 & 3 \\
5000 & 1.0 & 10000 & 21.526 & 21.508 & 43.034 & 3 \\
\midrule
10000 & 0.0 & 1 & 3.419 & 3.389 & 6.808 & 3 \\
10000 & 0.5 & 1 & 3.666 & 3.637 & 7.304 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 8-node synchronous All-to-All federated Sinkhorn. Only node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_sync8_c1M}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.066 & 0.035 & 0.101 & 4 \\
1000 & 0.5 & 1 & 0.056 & 0.030 & 0.085 & 5 \\
1000 & 0.9 & 1 & 0.054 & 0.028 & 0.082 & 5 \\
1000 & 1.0 & 1 & 0.085 & 0.059 & 0.144 & 5 \\
1000 & 0.0 & 100 & 0.085 & 0.045 & 0.130 & 4 \\
1000 & 0.5 & 100 & 0.088 & 0.053 & 0.141 & 5 \\
1000 & 0.9 & 100 & 0.083 & 0.048 & 0.132 & 5 \\
1000 & 1.0 & 100 & 0.084 & 0.046 & 0.130 & 5 \\
1000 & 0.0 & 1000 & 0.214 & 0.162 & 0.376 & 4 \\
1000 & 0.5 & 1000 & 0.224 & 0.172 & 0.396 & 5 \\
1000 & 0.9 & 1000 & 0.235 & 0.178 & 0.412 & 5 \\
1000 & 1.0 & 1000 & 0.236 & 0.180 & 0.417 & 5 \\
1000 & 0.0 & 10000 & 1.678 & 1.405 & 3.083 & 4 \\
1000 & 0.5 & 10000 & 2.160 & 1.823 & 3.984 & 5 \\
1000 & 0.9 & 10000 & 2.051 & 1.715 & 3.766 & 5 \\
1000 & 1.0 & 10000 & 2.062 & 1.728 & 3.791 & 5 \\
\midrule
5000 & 0.0 & 1 & 0.046 & 0.017 & 0.062 & 3 \\
5000 & 0.5 & 1 & 0.045 & 0.016 & 0.061 & 3 \\
5000 & 0.9 & 1 & 0.045 & 0.016 & 0.061 & 3 \\
5000 & 1.0 & 1 & 0.046 & 0.017 & 0.063 & 3 \\
5000 & 0.0 & 100 & 0.177 & 0.138 & 0.315 & 3 \\
5000 & 0.5 & 100 & 0.339 & 0.295 & 0.634 & 3 \\
5000 & 0.9 & 100 & 0.091 & 0.052 & 0.143 & 3 \\
5000 & 1.0 & 100 & 0.100 & 0.061 & 0.161 & 3 \\
5000 & 0.0 & 1000 & 0.589 & 0.422 & 1.011 & 3 \\
5000 & 0.5 & 1000 & 0.584 & 0.424 & 1.008 & 3 \\
5000 & 0.9 & 1000 & 0.590 & 0.412 & 1.001 & 3 \\
5000 & 1.0 & 1000 & 0.681 & 0.517 & 1.199 & 3 \\
5000 & 0.0 & 10000 & 6.179 & 4.915 & 11.094 & 3 \\
5000 & 0.5 & 10000 & 6.217 & 4.943 & 11.160 & 3 \\
5000 & 0.9 & 10000 & 6.269 & 4.996 & 11.265 & 3 \\
5000 & 1.0 & 10000 & 6.254 & 4.975 & 11.229 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.065 & 0.022 & 0.087 & 3 \\
10000 & 0.5 & 1 & 0.065 & 0.022 & 0.087 & 3 \\
10000 & 0.9 & 1 & 0.065 & 0.023 & 0.087 & 3 \\
10000 & 1.0 & 1 & 0.064 & 0.022 & 0.086 & 3 \\
10000 & 0.0 & 100 & 3.302 & 3.221 & 6.524 & 3 \\
10000 & 0.5 & 100 & 0.185 & 0.114 & 0.298 & 3 \\
10000 & 0.9 & 100 & 0.182 & 0.107 & 0.289 & 3 \\
10000 & 1.0 & 100 & 0.181 & 0.105 & 0.285 & 3 \\
10000 & 0.0 & 1000 & 1.108 & 0.750 & 1.858 & 3 \\
10000 & 0.5 & 1000 & 1.205 & 0.802 & 2.007 & 3 \\
10000 & 0.9 & 1000 & 1.084 & 0.737 & 1.820 & 3 \\
10000 & 1.0 & 1000 & 1.139 & 0.793 & 1.932 & 3 \\
10000 & 0.0 & 10000 & 11.987 & 8.691 & 20.679 & 3 \\
10000 & 0.5 & 10000 & 14.919 & 11.640 & 26.559 & 3 \\
10000 & 0.9 & 10000 & 12.055 & 8.753 & 20.808 & 3 \\
10000 & 1.0 & 10000 & 12.374 & 9.079 & 21.453 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 2-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar2_c10}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.059 & 0.034 & 0.092 & 4 \\
1000 & 0.5 & 1 & 0.052 & 0.027 & 0.079 & 4 \\
1000 & 0.9 & 1 & 0.055 & 0.028 & 0.083 & 5 \\
1000 & 1.0 & 1 & 0.062 & 0.036 & 0.098 & 5 \\
1000 & 0.0 & 100 & 0.080 & 0.046 & 0.126 & 4 \\
1000 & 0.5 & 100 & 0.085 & 0.047 & 0.133 & 5 \\
1000 & 0.9 & 100 & 0.078 & 0.045 & 0.123 & 4 \\
1000 & 1.0 & 100 & 0.084 & 0.046 & 0.129 & 5 \\
1000 & 0.0 & 1000 & 0.201 & 0.149 & 0.350 & 4 \\
1000 & 0.5 & 1000 & 0.223 & 0.167 & 0.390 & 5 \\
1000 & 0.9 & 1000 & 0.252 & 0.196 & 0.448 & 5 \\
1000 & 1.0 & 1000 & 0.219 & 0.167 & 0.386 & 5 \\
1000 & 0.0 & 10000 & 1.631 & 1.361 & 2.992 & 4 \\
1000 & 0.5 & 10000 & 2.038 & 1.705 & 3.743 & 5 \\
1000 & 0.9 & 10000 & 2.063 & 1.727 & 3.790 & 5 \\
1000 & 1.0 & 10000 & 2.125 & 1.788 & 3.913 & 5 \\
\midrule
5000 & 0.0 & 1 & 0.045 & 0.016 & 0.061 & 3 \\
5000 & 0.5 & 1 & 0.045 & 0.017 & 0.062 & 3 \\
5000 & 0.9 & 1 & 0.045 & 0.016 & 0.061 & 3 \\
5000 & 1.0 & 1 & 0.608 & 0.580 & 1.188 & 3 \\
5000 & 0.0 & 100 & 0.585 & 0.542 & 1.128 & 3 \\
5000 & 0.5 & 100 & 0.101 & 0.062 & 0.163 & 3 \\
5000 & 0.9 & 100 & 0.378 & 0.339 & 0.718 & 3 \\
5000 & 1.0 & 100 & 0.099 & 0.052 & 0.151 & 3 \\
5000 & 0.0 & 1000 & 0.566 & 0.411 & 0.977 & 3 \\
5000 & 0.5 & 1000 & 0.564 & 0.408 & 0.972 & 3 \\
5000 & 0.9 & 1000 & 0.612 & 0.445 & 1.056 & 3 \\
5000 & 1.0 & 1000 & 0.571 & 0.414 & 0.984 & 3 \\
5000 & 0.0 & 10000 & 5.847 & 4.580 & 10.427 & 3 \\
5000 & 0.5 & 10000 & 6.234 & 4.969 & 11.203 & 3 \\
5000 & 0.9 & 10000 & 6.172 & 4.900 & 11.072 & 3 \\
5000 & 1.0 & 10000 & 6.057 & 4.789 & 10.847 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.065 & 0.023 & 0.088 & 3 \\
10000 & 0.5 & 1 & 0.061 & 0.023 & 0.084 & 3 \\
10000 & 0.9 & 1 & 0.065 & 0.023 & 0.088 & 3 \\
10000 & 1.0 & 1 & 0.065 & 0.023 & 0.088 & 3 \\
10000 & 0.0 & 100 & 0.185 & 0.108 & 0.293 & 3 \\
10000 & 0.5 & 100 & 3.804 & 3.727 & 7.531 & 3 \\
10000 & 0.9 & 100 & 0.184 & 0.109 & 0.293 & 3 \\
10000 & 1.0 & 100 & 0.185 & 0.110 & 0.295 & 3 \\
10000 & 0.0 & 1000 & 1.163 & 0.795 & 1.958 & 3 \\
10000 & 0.5 & 1000 & 6.933 & 6.520 & 13.453 & 3 \\
10000 & 0.9 & 1000 & 6.262 & 5.887 & 12.149 & 3 \\
10000 & 1.0 & 1000 & 1.165 & 0.793 & 1.958 & 3 \\
10000 & 0.0 & 10000 & 12.009 & 8.719 & 20.729 & 3 \\
10000 & 0.5 & 10000 & 12.021 & 8.717 & 20.738 & 3 \\
10000 & 0.9 & 10000 & 12.445 & 9.130 & 21.575 & 3 \\
10000 & 1.0 & 10000 & 17.483 & 14.175 & 31.658 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 2-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar2_c1k}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.054 & 0.029 & 0.082 & 4 \\
1000 & 0.5 & 1 & 0.054 & 0.029 & 0.083 & 5 \\
1000 & 0.9 & 1 & 0.060 & 0.034 & 0.093 & 5 \\
1000 & 1.0 & 1 & 0.105 & 0.079 & 0.184 & 4 \\
1000 & 0.0 & 100 & 0.093 & 0.060 & 0.153 & 4 \\
1000 & 0.5 & 100 & 0.118 & 0.084 & 0.202 & 4 \\
1000 & 0.9 & 100 & 0.080 & 0.044 & 0.125 & 5 \\
1000 & 1.0 & 100 & 0.086 & 0.047 & 0.133 & 5 \\
1000 & 0.0 & 1000 & 0.185 & 0.136 & 0.321 & 4 \\
1000 & 0.5 & 1000 & 0.220 & 0.167 & 0.387 & 5 \\
1000 & 0.9 & 1000 & 0.218 & 0.167 & 0.385 & 5 \\
1000 & 1.0 & 1000 & 0.229 & 0.172 & 0.400 & 5 \\
1000 & 0.0 & 10000 & 1.704 & 1.432 & 3.136 & 4 \\
1000 & 0.5 & 10000 & 2.131 & 1.795 & 3.926 & 5 \\
1000 & 0.9 & 10000 & 1.609 & 1.337 & 2.946 & 4 \\
1000 & 1.0 & 10000 & 2.523 & 2.185 & 4.709 & 5 \\
\midrule
5000 & 0.0 & 1 & 0.045 & 0.016 & 0.061 & 3 \\
5000 & 0.5 & 1 & 0.045 & 0.016 & 0.061 & 3 \\
5000 & 0.9 & 1 & 0.045 & 0.016 & 0.061 & 3 \\
5000 & 1.0 & 1 & 0.981 & 0.949 & 1.929 & 3 \\
5000 & 0.0 & 100 & 0.101 & 0.063 & 0.164 & 3 \\
5000 & 0.5 & 100 & 0.469 & 0.426 & 0.896 & 3 \\
5000 & 0.9 & 100 & 0.118 & 0.079 & 0.197 & 3 \\
5000 & 1.0 & 100 & 0.089 & 0.050 & 0.139 & 3 \\
5000 & 0.0 & 1000 & 0.590 & 0.442 & 1.032 & 3 \\
5000 & 0.5 & 1000 & 0.567 & 0.410 & 0.977 & 3 \\
5000 & 0.9 & 1000 & 0.559 & 0.386 & 0.945 & 3 \\
5000 & 1.0 & 1000 & 0.577 & 0.409 & 0.986 & 3 \\
5000 & 0.0 & 10000 & 7.968 & 6.704 & 14.671 & 3 \\
5000 & 0.5 & 10000 & 6.218 & 4.954 & 11.172 & 3 \\
5000 & 0.9 & 10000 & 6.200 & 4.937 & 11.138 & 3 \\
5000 & 1.0 & 10000 & 7.291 & 6.026 & 13.318 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.065 & 0.023 & 0.087 & 3 \\
10000 & 0.5 & 1 & 0.061 & 0.023 & 0.084 & 3 \\
10000 & 0.9 & 1 & 0.065 & 0.022 & 0.087 & 3 \\
10000 & 1.0 & 1 & 0.065 & 0.023 & 0.088 & 3 \\
10000 & 0.0 & 100 & 0.177 & 0.102 & 0.278 & 3 \\
10000 & 0.5 & 100 & 5.179 & 5.104 & 10.283 & 3 \\
10000 & 0.9 & 100 & 0.170 & 0.100 & 0.269 & 3 \\
10000 & 1.0 & 100 & 0.182 & 0.107 & 0.289 & 3 \\
10000 & 0.0 & 1000 & 1.075 & 0.718 & 1.793 & 3 \\
10000 & 0.5 & 1000 & 1.026 & 0.692 & 1.719 & 3 \\
10000 & 0.9 & 1000 & 2.348 & 2.010 & 4.358 & 3 \\
10000 & 1.0 & 1000 & 1.136 & 0.800 & 1.936 & 3 \\
10000 & 0.0 & 10000 & 12.061 & 8.757 & 20.818 & 3 \\
10000 & 0.5 & 10000 & 12.453 & 9.166 & 21.619 & 3 \\
10000 & 0.9 & 10000 & 15.436 & 12.126 & 27.562 & 3 \\
10000 & 1.0 & 10000 & 12.413 & 9.127 & 21.540 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 2-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar2_c1M}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.069 & 0.037 & 0.106 & 4 \\
1000 & 0.5 & 1 & 0.058 & 0.033 & 0.091 & 5 \\
1000 & 0.9 & 1 & 0.062 & 0.037 & 0.099 & 4 \\
1000 & 1.0 & 1 & 0.074 & 0.046 & 0.119 & 4 \\
1000 & 0.0 & 100 & 0.084 & 0.047 & 0.130 & 4 \\
1000 & 0.5 & 100 & 0.080 & 0.048 & 0.128 & 4 \\
1000 & 0.9 & 100 & 0.079 & 0.047 & 0.126 & 4 \\
1000 & 1.0 & 100 & 0.085 & 0.053 & 0.138 & 4 \\
1000 & 0.0 & 1000 & 0.222 & 0.174 & 0.396 & 4 \\
1000 & 0.5 & 1000 & 0.211 & 0.165 & 0.376 & 4 \\
1000 & 0.9 & 1000 & 0.241 & 0.191 & 0.432 & 5 \\
1000 & 1.0 & 1000 & 0.209 & 0.166 & 0.375 & 4 \\
1000 & 0.0 & 10000 & 1.800 & 1.531 & 3.331 & 4 \\
1000 & 0.5 & 10000 & 1.902 & 1.636 & 3.538 & 4 \\
1000 & 0.9 & 10000 & 2.000 & 1.726 & 3.725 & 4 \\
1000 & 1.0 & 10000 & 1.881 & 1.608 & 3.489 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.191 & 0.165 & 0.355 & 3 \\
5000 & 0.5 & 1 & 0.314 & 0.288 & 0.602 & 3 \\
5000 & 0.9 & 1 & 0.269 & 0.243 & 0.512 & 3 \\
5000 & 1.0 & 1 & 0.181 & 0.155 & 0.336 & 3 \\
5000 & 0.0 & 100 & 0.157 & 0.120 & 0.278 & 3 \\
5000 & 0.5 & 100 & 0.141 & 0.103 & 0.244 & 3 \\
5000 & 0.9 & 100 & 0.136 & 0.098 & 0.233 & 3 \\
5000 & 1.0 & 100 & 0.133 & 0.096 & 0.229 & 3 \\
5000 & 0.0 & 1000 & 0.716 & 0.559 & 1.275 & 3 \\
5000 & 0.5 & 1000 & 0.665 & 0.521 & 1.186 & 3 \\
5000 & 0.9 & 1000 & 0.952 & 0.805 & 1.756 & 3 \\
5000 & 1.0 & 1000 & 0.860 & 0.712 & 1.572 & 3 \\
5000 & 0.0 & 10000 & 7.067 & 5.811 & 12.879 & 3 \\
5000 & 0.5 & 10000 & 7.115 & 5.846 & 12.960 & 3 \\
5000 & 0.9 & 10000 & 8.035 & 6.772 & 14.807 & 3 \\
5000 & 1.0 & 10000 & 7.686 & 6.428 & 14.114 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.744 & 0.707 & 1.451 & 3 \\
10000 & 0.5 & 1 & 0.627 & 0.590 & 1.217 & 3 \\
10000 & 0.9 & 1 & 0.078 & 0.042 & 0.120 & 3 \\
10000 & 1.0 & 1 & 0.553 & 0.516 & 1.069 & 3 \\
10000 & 0.0 & 100 & 1.081 & 1.021 & 2.102 & 3 \\
10000 & 0.5 & 100 & 0.705 & 0.641 & 1.346 & 3 \\
10000 & 0.9 & 100 & 0.228 & 0.158 & 0.385 & 3 \\
10000 & 1.0 & 100 & 0.681 & 0.619 & 1.300 & 3 \\
10000 & 0.0 & 1000 & 2.253 & 1.930 & 4.183 & 3 \\
10000 & 0.5 & 1000 & 3.183 & 2.853 & 6.036 & 3 \\
10000 & 0.9 & 1000 & 2.794 & 2.467 & 5.261 & 3 \\
10000 & 1.0 & 1000 & 3.255 & 2.906 & 6.161 & 3 \\
10000 & 0.0 & 10000 & 16.015 & 12.737 & 28.752 & 3 \\
10000 & 0.5 & 10000 & 17.009 & 13.725 & 30.734 & 3 \\
10000 & 0.9 & 10000 & 17.973 & 14.679 & 32.652 & 3 \\
10000 & 1.0 & 10000 & 16.177 & 12.894 & 29.071 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 4-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar4_c10}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.056 & 0.031 & 0.087 & 4 \\
1000 & 0.5 & 1 & 0.057 & 0.033 & 0.090 & 4 \\
1000 & 0.9 & 1 & 0.080 & 0.057 & 0.137 & 4 \\
1000 & 1.0 & 1 & 0.063 & 0.039 & 0.102 & 4 \\
1000 & 0.0 & 100 & 0.078 & 0.047 & 0.125 & 4 \\
1000 & 0.5 & 100 & 0.082 & 0.047 & 0.129 & 4 \\
1000 & 0.9 & 100 & 0.080 & 0.048 & 0.128 & 4 \\
1000 & 1.0 & 100 & 0.097 & 0.064 & 0.162 & 4 \\
1000 & 0.0 & 1000 & 0.216 & 0.169 & 0.385 & 4 \\
1000 & 0.5 & 1000 & 0.248 & 0.194 & 0.441 & 5 \\
1000 & 0.9 & 1000 & 0.218 & 0.173 & 0.391 & 4 \\
1000 & 1.0 & 1000 & 0.214 & 0.168 & 0.382 & 4 \\
1000 & 0.0 & 10000 & 1.893 & 1.622 & 3.515 & 4 \\
1000 & 0.5 & 10000 & 1.763 & 1.491 & 3.254 & 4 \\
1000 & 0.9 & 10000 & 1.739 & 1.472 & 3.211 & 4 \\
1000 & 1.0 & 10000 & 1.872 & 1.582 & 3.454 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.209 & 0.183 & 0.392 & 3 \\
5000 & 0.5 & 1 & 0.184 & 0.158 & 0.341 & 3 \\
5000 & 0.9 & 1 & 0.062 & 0.037 & 0.099 & 3 \\
5000 & 1.0 & 1 & 0.365 & 0.340 & 0.705 & 3 \\
5000 & 0.0 & 100 & 0.142 & 0.105 & 0.247 & 3 \\
5000 & 0.5 & 100 & 0.136 & 0.098 & 0.234 & 3 \\
5000 & 0.9 & 100 & 0.139 & 0.101 & 0.240 & 3 \\
5000 & 1.0 & 100 & 0.136 & 0.097 & 0.233 & 3 \\
5000 & 0.0 & 1000 & 0.641 & 0.474 & 1.114 & 3 \\
5000 & 0.5 & 1000 & 0.674 & 0.510 & 1.184 & 3 \\
5000 & 0.9 & 1000 & 0.623 & 0.477 & 1.100 & 3 \\
5000 & 1.0 & 1000 & 0.713 & 0.567 & 1.280 & 3 \\
5000 & 0.0 & 10000 & 6.959 & 5.701 & 12.661 & 3 \\
5000 & 0.5 & 10000 & 6.904 & 5.647 & 12.551 & 3 \\
5000 & 0.9 & 10000 & 7.470 & 6.216 & 13.686 & 3 \\
5000 & 1.0 & 10000 & 7.489 & 6.228 & 13.717 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.080 & 0.043 & 0.123 & 3 \\
10000 & 0.5 & 1 & 0.080 & 0.041 & 0.121 & 3 \\
10000 & 0.9 & 1 & 0.693 & 0.647 & 1.340 & 3 \\
10000 & 1.0 & 1 & 0.078 & 0.042 & 0.120 & 3 \\
10000 & 0.0 & 100 & 0.657 & 0.593 & 1.250 & 3 \\
10000 & 0.5 & 100 & 0.222 & 0.158 & 0.380 & 3 \\
10000 & 0.9 & 100 & 0.854 & 0.790 & 1.645 & 3 \\
10000 & 1.0 & 100 & 0.277 & 0.213 & 0.490 & 3 \\
10000 & 0.0 & 1000 & 1.583 & 1.218 & 2.800 & 3 \\
10000 & 0.5 & 1000 & 2.630 & 2.277 & 4.907 & 3 \\
10000 & 0.9 & 1000 & 2.062 & 1.737 & 3.799 & 3 \\
10000 & 1.0 & 1000 & 2.850 & 2.513 & 5.363 & 3 \\
10000 & 0.0 & 10000 & 16.317 & 13.034 & 29.351 & 3 \\
10000 & 0.5 & 10000 & 15.562 & 12.275 & 27.837 & 3 \\
10000 & 0.9 & 10000 & 16.018 & 12.731 & 28.750 & 3 \\
10000 & 1.0 & 10000 & 16.551 & 13.265 & 29.816 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 4-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar4_c1k}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.060 & 0.035 & 0.096 & 4 \\
1000 & 0.5 & 1 & 0.057 & 0.031 & 0.088 & 4 \\
1000 & 0.9 & 1 & 0.056 & 0.032 & 0.088 & 4 \\
1000 & 1.0 & 1 & 0.058 & 0.033 & 0.091 & 4 \\
1000 & 0.0 & 100 & 0.110 & 0.072 & 0.182 & 4 \\
1000 & 0.5 & 100 & 0.079 & 0.047 & 0.126 & 4 \\
1000 & 0.9 & 100 & 0.075 & 0.045 & 0.121 & 4 \\
1000 & 1.0 & 100 & 0.082 & 0.049 & 0.131 & 4 \\
1000 & 0.0 & 1000 & 0.225 & 0.179 & 0.404 & 4 \\
1000 & 0.5 & 1000 & 0.222 & 0.176 & 0.399 & 4 \\
1000 & 0.9 & 1000 & 0.217 & 0.170 & 0.387 & 4 \\
1000 & 1.0 & 1000 & 0.222 & 0.176 & 0.398 & 4 \\
1000 & 0.0 & 10000 & 1.955 & 1.687 & 3.642 & 4 \\
1000 & 0.5 & 10000 & 1.759 & 1.491 & 3.250 & 4 \\
1000 & 0.9 & 10000 & 1.839 & 1.575 & 3.413 & 4 \\
1000 & 1.0 & 10000 & 1.973 & 1.704 & 3.676 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.318 & 0.293 & 0.611 & 3 \\
5000 & 0.5 & 1 & 0.200 & 0.175 & 0.374 & 3 \\
5000 & 0.9 & 1 & 0.235 & 0.209 & 0.444 & 3 \\
5000 & 1.0 & 1 & 0.061 & 0.035 & 0.096 & 3 \\
5000 & 0.0 & 100 & 0.136 & 0.099 & 0.235 & 3 \\
5000 & 0.5 & 100 & 0.137 & 0.102 & 0.239 & 3 \\
5000 & 0.9 & 100 & 0.128 & 0.093 & 0.221 & 3 \\
5000 & 1.0 & 100 & 0.140 & 0.101 & 0.242 & 3 \\
5000 & 0.0 & 1000 & 0.678 & 0.519 & 1.197 & 3 \\
5000 & 0.5 & 1000 & 0.647 & 0.501 & 1.149 & 3 \\
5000 & 0.9 & 1000 & 0.804 & 0.654 & 1.458 & 3 \\
5000 & 1.0 & 1000 & 0.790 & 0.630 & 1.420 & 3 \\
5000 & 0.0 & 10000 & 6.906 & 5.643 & 12.549 & 3 \\
5000 & 0.5 & 10000 & 7.059 & 5.801 & 12.860 & 3 \\
5000 & 0.9 & 10000 & 7.015 & 5.756 & 12.771 & 3 \\
5000 & 1.0 & 10000 & 7.835 & 6.576 & 14.411 & 3 \\
\midrule
10000 & 0.0 & 1 & 0.089 & 0.053 & 0.142 & 3 \\
10000 & 0.5 & 1 & 1.506 & 1.469 & 2.975 & 3 \\
10000 & 0.9 & 1 & 0.340 & 0.303 & 0.643 & 3 \\
10000 & 1.0 & 1 & 1.024 & 0.987 & 2.011 & 3 \\
10000 & 0.0 & 100 & 0.223 & 0.157 & 0.379 & 3 \\
10000 & 0.5 & 100 & 0.321 & 0.255 & 0.576 & 3 \\
10000 & 0.9 & 100 & 0.750 & 0.685 & 1.435 & 3 \\
10000 & 1.0 & 100 & 0.222 & 0.157 & 0.378 & 3 \\
10000 & 0.0 & 1000 & 2.890 & 2.566 & 5.456 & 3 \\
10000 & 0.5 & 1000 & 1.274 & 0.926 & 2.201 & 3 \\
10000 & 0.9 & 1000 & 2.326 & 2.000 & 4.326 & 3 \\
10000 & 1.0 & 1000 & 2.281 & 1.957 & 4.238 & 3 \\
10000 & 0.0 & 10000 & 16.918 & 13.649 & 30.567 & 3 \\
10000 & 0.5 & 10000 & 15.648 & 12.365 & 28.013 & 3 \\
10000 & 0.9 & 10000 & 15.462 & 12.179 & 27.641 & 3 \\
10000 & 1.0 & 10000 & 18.830 & 15.537 & 34.368 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 4-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar4_c1M}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.079 & 0.046 & 0.125 & 4 \\
1000 & 0.5 & 1 & 0.061 & 0.038 & 0.099 & 4 \\
1000 & 0.9 & 1 & 0.109 & 0.085 & 0.194 & 4 \\
1000 & 1.0 & 1 & 0.062 & 0.038 & 0.100 & 4 \\
1000 & 0.0 & 100 & 0.519 & 0.483 & 1.001 & 4 \\
1000 & 0.5 & 100 & 0.660 & 0.628 & 1.288 & 4 \\
1000 & 0.9 & 100 & 0.123 & 0.091 & 0.213 & 4 \\
1000 & 1.0 & 100 & 0.144 & 0.113 & 0.257 & 4 \\
1000 & 0.0 & 1000 & 0.234 & 0.188 & 0.421 & 4 \\
1000 & 0.5 & 1000 & 0.242 & 0.192 & 0.433 & 4 \\
1000 & 0.9 & 1000 & 0.250 & 0.207 & 0.457 & 4 \\
1000 & 1.0 & 1000 & 0.310 & 0.265 & 0.575 & 4 \\
1000 & 0.0 & 10000 & 2.556 & 2.286 & 4.842 & 4 \\
1000 & 0.5 & 10000 & 2.189 & 1.926 & 4.114 & 4 \\
1000 & 0.9 & 10000 & 1.950 & 1.686 & 3.636 & 4 \\
1000 & 1.0 & 10000 & 2.319 & 2.054 & 4.373 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.693 & 0.670 & 1.363 & 3 \\
5000 & 0.5 & 1 & 1.265 & 1.240 & 2.505 & 3 \\
5000 & 0.9 & 1 & 0.647 & 0.624 & 1.271 & 3 \\
5000 & 1.0 & 1 & 0.827 & 0.800 & 1.627 & 3 \\
5000 & 0.0 & 100 & 0.364 & 0.327 & 0.692 & 3 \\
5000 & 0.5 & 100 & 0.583 & 0.546 & 1.130 & 3 \\
5000 & 0.9 & 100 & 0.591 & 0.555 & 1.146 & 3 \\
5000 & 1.0 & 100 & 0.633 & 0.596 & 1.229 & 3 \\
5000 & 0.0 & 1000 & 1.078 & 0.928 & 2.006 & 3 \\
5000 & 0.5 & 1000 & 1.109 & 0.955 & 2.064 & 3 \\
5000 & 0.9 & 1000 & 1.827 & 1.675 & 3.503 & 3 \\
5000 & 1.0 & 1000 & 0.989 & 0.829 & 1.818 & 3 \\
5000 & 0.0 & 10000 & 9.261 & 8.029 & 17.290 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 8-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar8_c10}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.070 & 0.047 & 0.117 & 4 \\
1000 & 0.5 & 1 & 0.064 & 0.040 & 0.104 & 4 \\
1000 & 0.9 & 1 & 0.063 & 0.039 & 0.102 & 4 \\
1000 & 1.0 & 1 & 0.490 & 0.465 & 0.955 & 4 \\
1000 & 0.0 & 100 & 0.551 & 0.520 & 1.070 & 4 \\
1000 & 0.5 & 100 & 0.554 & 0.521 & 1.075 & 4 \\
1000 & 0.9 & 100 & 0.121 & 0.090 & 0.211 & 4 \\
1000 & 1.0 & 100 & 0.100 & 0.070 & 0.170 & 4 \\
1000 & 0.0 & 1000 & 0.329 & 0.286 & 0.615 & 4 \\
1000 & 0.5 & 1000 & 0.279 & 0.231 & 0.510 & 4 \\
1000 & 0.9 & 1000 & 0.253 & 0.207 & 0.459 & 4 \\
1000 & 1.0 & 1000 & 0.293 & 0.248 & 0.541 & 4 \\
1000 & 0.0 & 10000 & 2.269 & 2.005 & 4.275 & 4 \\
1000 & 0.5 & 10000 & 2.009 & 1.746 & 3.754 & 4 \\
1000 & 0.9 & 10000 & 1.945 & 1.680 & 3.625 & 4 \\
1000 & 1.0 & 10000 & 2.035 & 1.773 & 3.808 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.472 & 0.449 & 0.921 & 3 \\
5000 & 0.5 & 1 & 0.610 & 0.584 & 1.194 & 3 \\
5000 & 0.9 & 1 & 0.467 & 0.443 & 0.910 & 3 \\
5000 & 1.0 & 1 & 0.727 & 0.704 & 1.431 & 3 \\
5000 & 0.0 & 100 & 0.622 & 0.585 & 1.207 & 3 \\
5000 & 0.5 & 100 & 1.214 & 1.177 & 2.392 & 3 \\
5000 & 0.9 & 100 & 0.551 & 0.514 & 1.065 & 3 \\
5000 & 1.0 & 100 & 0.548 & 0.511 & 1.059 & 3 \\
5000 & 0.0 & 1000 & 0.976 & 0.811 & 1.787 & 3 \\
5000 & 0.5 & 1000 & 1.056 & 0.898 & 1.954 & 3 \\
5000 & 0.9 & 1000 & 1.161 & 1.006 & 2.167 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 8-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar8_c1k}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{ccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. \\
\midrule
1000 & 0.0 & 1 & 0.068 & 0.044 & 0.112 & 4 \\
1000 & 0.5 & 1 & 0.106 & 0.083 & 0.190 & 4 \\
1000 & 0.9 & 1 & 0.063 & 0.041 & 0.104 & 4 \\
1000 & 1.0 & 1 & 0.562 & 0.538 & 1.100 & 4 \\
1000 & 0.0 & 100 & 0.515 & 0.485 & 1.000 & 4 \\
1000 & 0.5 & 100 & 0.737 & 0.706 & 1.442 & 4 \\
1000 & 0.9 & 100 & 0.108 & 0.078 & 0.186 & 4 \\
1000 & 1.0 & 100 & 0.089 & 0.057 & 0.147 & 4 \\
1000 & 0.0 & 1000 & 0.244 & 0.199 & 0.443 & 4 \\
1000 & 0.5 & 1000 & 0.300 & 0.256 & 0.556 & 4 \\
1000 & 0.9 & 1000 & 0.403 & 0.359 & 0.762 & 4 \\
1000 & 1.0 & 1000 & 0.288 & 0.243 & 0.530 & 4 \\
1000 & 0.0 & 10000 & 2.023 & 1.761 & 3.784 & 4 \\
1000 & 0.5 & 10000 & 2.131 & 1.868 & 3.999 & 4 \\
1000 & 0.9 & 10000 & 2.108 & 1.842 & 3.950 & 4 \\
1000 & 1.0 & 10000 & 2.049 & 1.788 & 3.837 & 4 \\
\midrule
5000 & 0.0 & 1 & 0.913 & 0.889 & 1.802 & 3 \\
5000 & 0.5 & 1 & 1.052 & 1.022 & 2.074 & 3 \\
5000 & 0.9 & 1 & 0.518 & 0.494 & 1.012 & 3 \\
5000 & 1.0 & 1 & 0.613 & 0.586 & 1.199 & 3 \\
5000 & 0.0 & 100 & 0.880 & 0.844 & 1.724 & 3 \\
5000 & 0.5 & 100 & 0.642 & 0.604 & 1.246 & 3 \\
5000 & 0.9 & 100 & 0.575 & 0.536 & 1.111 & 3 \\
5000 & 1.0 & 100 & 0.655 & 0.618 & 1.272 & 3 \\
5000 & 0.0 & 1000 & 3.106 & 2.949 & 6.055 & 3 \\
5000 & 0.5 & 1000 & 1.265 & 1.110 & 2.375 & 3 \\
5000 & 0.9 & 1000 & 1.102 & 0.938 & 2.041 & 3 \\
5000 & 1.0 & 1000 & 1.319 & 1.165 & 2.484 & 3 \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 8-node synchronous-star federated Sinkhorn. Only the server node's times were recorded.}
\label{table:big_table_syncstar8_c1M}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 1.397 & 0.729 & 2.126 & 1500 & no \\
1000 & 0.0 & 100 & 3.038 & 2.331 & 5.369 & 1500 & no \\
1000 & 0.0 & 1000 & 4.599 & 4.183 & 8.782 & 618 & yes \\
1000 & 0.0 & 10000 & 173.335 & 167.579 & 340.914 & 1500 & no \\
1000 & 0.5 & 1 & 1.398 & 0.733 & 2.131 & 1500 & no \\
1000 & 0.5 & 100 & 3.265 & 2.515 & 5.779 & 1494 & yes \\
1000 & 0.5 & 1000 & 4.145 & 3.793 & 7.937 & 521 & yes \\
1000 & 0.5 & 10000 & 87.395 & 84.454 & 171.848 & 762 & yes \\
1000 & 0.9 & 1 & 1.397 & 0.734 & 2.131 & 1500 & no \\
1000 & 0.9 & 100 & 3.105 & 2.405 & 5.509 & 1500 & no \\
1000 & 0.9 & 1000 & 6.245 & 5.676 & 11.921 & 825 & yes \\
1000 & 0.9 & 10000 & 171.369 & 165.579 & 336.948 & 1500 & no \\
1000 & 1.0 & 1 & 1.455 & 0.761 & 2.216 & 1500 & no \\
1000 & 1.0 & 100 & 1.536 & 1.127 & 2.663 & 825 & yes \\
1000 & 1.0 & 1000 & 12.562 & 11.543 & 24.105 & 1495 & yes \\
1000 & 1.0 & 10000 & 81.919 & 79.154 & 161.073 & 714 & yes \\
\midrule
5000 & 0.0 & 1 & 4.121 & 2.127 & 6.247 & 1500 & no \\
5000 & 0.0 & 100 & 8.815 & 6.950 & 15.765 & 1500 & no \\
5000 & 0.0 & 1000 & 56.895 & 45.880 & 102.776 & 1500 & no \\
5000 & 0.0 & 10000 & 242.187 & 212.427 & 454.614 & 428 & yes \\
5000 & 0.5 & 1 & 3.693 & 1.916 & 5.609 & 1500 & no \\
5000 & 0.5 & 100 & 5.321 & 4.009 & 9.330 & 1055 & yes \\
5000 & 0.5 & 100 & 6.002 & 4.665 & 10.667 & 1076 & yes \\
5000 & 0.5 & 1000 & 58.740 & 47.649 & 106.389 & 1500 & no \\
5000 & 0.5 & 10000 & 458.226 & 401.225 & 859.451 & 819 & yes \\
5000 & 0.9 & 1 & 3.945 & 2.044 & 5.989 & 1500 & no \\
5000 & 0.9 & 100 & 9.122 & 7.262 & 16.383 & 1500 & no \\
5000 & 0.9 & 1000 & 54.584 & 43.563 & 98.147 & 1500 & no \\
5000 & 0.9 & 10000 & 765.251 & 671.046 & 1436.297 & 1368 & yes \\
5000 & 1.0 & 1 & 3.658 & 1.893 & 5.551 & 1500 & no \\
5000 & 1.0 & 100 & 10.405 & 8.512 & 18.917 & 1500 & no \\
5000 & 1.0 & 1000 & 50.329 & 39.259 & 89.589 & 1500 & no \\
5000 & 1.0 & 10000 & 597.742 & 523.844 & 1121.585 & 1074 & yes \\
\midrule
10000 & 0.0 & 1 & 6.681 & 3.518 & 10.199 & 1500 & no \\
10000 & 0.0 & 100 & 17.742 & 12.175 & 29.916 & 1500 & no \\
10000 & 0.0 & 1000 & 67.588 & 46.930 & 114.518 & 744 & yes \\
10000 & 0.0 & 1000 & 63.972 & 42.191 & 106.163 & 784 & yes \\
10000 & 0.0 & 10000 & 1808.974 & 1400.411 & 3209.385 & 1500 & no \\
10000 & 0.5 & 1 & 6.488 & 3.351 & 9.839 & 1500 & no \\
10000 & 0.5 & 100 & 18.430 & 12.856 & 31.286 & 1500 & no \\
10000 & 0.5 & 1000 & 114.829 & 79.610 & 194.440 & 1268 & yes \\
10000 & 0.5 & 1000 & 112.672 & 76.566 & 189.238 & 1301 & yes \\
10000 & 0.5 & 10000 & 1812.257 & 1403.917 & 3216.175 & 1500 & no \\
10000 & 0.9 & 1 & 6.424 & 3.302 & 9.726 & 1500 & no \\
10000 & 0.9 & 100 & 18.950 & 13.401 & 32.351 & 1500 & no \\
10000 & 0.9 & 1000 & 52.134 & 35.419 & 87.553 & 603 & yes \\
10000 & 0.9 & 10000 & 481.781 & 373.866 & 855.647 & 401 & yes \\
10000 & 1.0 & 1 & 6.736 & 3.494 & 10.230 & 1500 & no \\
10000 & 1.0 & 100 & 18.687 & 13.135 & 31.821 & 1500 & no \\
10000 & 1.0 & 1000 & 65.106 & 45.562 & 110.668 & 703 & yes \\
10000 & 1.0 & 10000 & 708.022 & 548.387 & 1256.409 & 593 & yes \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 2-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async2_c10}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 1.377 & 0.720 & 2.096 & 1500 & no \\
1000 & 0.0 & 100 & 3.182 & 2.488 & 5.670 & 1500 & no \\
1000 & 0.0 & 1000 & 5.591 & 5.079 & 10.670 & 753 & yes \\
1000 & 0.0 & 10000 & 84.939 & 82.084 & 167.023 & 741 & yes \\
1000 & 0.5 & 1 & 1.381 & 0.721 & 2.101 & 1500 & no \\
1000 & 0.5 & 100 & 3.071 & 2.381 & 5.451 & 1500 & no \\
1000 & 0.5 & 1000 & 3.502 & 3.129 & 6.631 & 545 & yes \\
1000 & 0.5 & 10000 & 97.722 & 94.434 & 192.156 & 854 & yes \\
1000 & 0.9 & 1 & 1.385 & 0.727 & 2.113 & 1500 & no \\
1000 & 0.9 & 100 & 1.458 & 1.019 & 2.477 & 944 & yes \\
1000 & 0.9 & 100 & 1.585 & 1.118 & 2.703 & 951 & yes \\
1000 & 0.9 & 1000 & 12.914 & 11.895 & 24.809 & 1500 & no \\
1000 & 0.9 & 10000 & 62.029 & 59.932 & 121.960 & 541 & yes \\
1000 & 1.0 & 1 & 1.384 & 0.725 & 2.109 & 1500 & no \\
1000 & 1.0 & 100 & 1.855 & 1.400 & 3.255 & 920 & yes \\
1000 & 1.0 & 100 & 1.910 & 1.464 & 3.374 & 930 & yes \\
1000 & 1.0 & 1000 & 4.214 & 3.843 & 8.058 & 531 & yes \\
1000 & 1.0 & 10000 & 63.811 & 61.652 & 125.463 & 557 & yes \\
\midrule
5000 & 0.0 & 1 & 3.663 & 1.878 & 5.541 & 1500 & no \\
5000 & 0.0 & 100 & 3.764 & 2.889 & 6.653 & 706 & yes \\
5000 & 0.0 & 1000 & 59.707 & 48.607 & 108.315 & 1500 & no \\
5000 & 0.0 & 10000 & 840.959 & 737.715 & 1578.674 & 1500 & no \\
5000 & 0.5 & 1 & 3.727 & 1.926 & 5.653 & 1500 & no \\
5000 & 0.5 & 100 & 9.515 & 7.449 & 16.964 & 1500 & no \\
5000 & 0.5 & 1000 & 56.476 & 45.399 & 101.875 & 1500 & no \\
5000 & 0.5 & 10000 & 217.551 & 191.305 & 408.856 & 378 & yes \\
5000 & 0.9 & 1 & 3.733 & 1.937 & 5.670 & 1500 & no \\
5000 & 0.9 & 100 & 8.536 & 6.675 & 15.211 & 1500 & no \\
5000 & 0.9 & 1000 & 55.345 & 44.317 & 99.662 & 1500 & no \\
5000 & 0.9 & 10000 & 654.884 & 573.716 & 1228.600 & 1168 & yes \\
5000 & 1.0 & 1 & 3.694 & 1.916 & 5.610 & 1500 & no \\
5000 & 1.0 & 100 & 8.687 & 6.790 & 15.477 & 1500 & no \\
5000 & 1.0 & 1000 & 56.245 & 45.204 & 101.449 & 1500 & no \\
5000 & 1.0 & 10000 & 215.236 & 189.466 & 404.701 & 373 & yes \\
\midrule
10000 & 0.0 & 1 & 6.617 & 3.434 & 10.051 & 1500 & no \\
10000 & 0.0 & 100 & 18.829 & 13.259 & 32.088 & 1500 & no \\
10000 & 0.0 & 1000 & 44.697 & 30.128 & 74.826 & 524 & yes \\
10000 & 0.0 & 1000 & 51.543 & 36.778 & 88.320 & 528 & yes \\
10000 & 0.0 & 10000 & 849.924 & 658.616 & 1508.540 & 709 & yes \\
10000 & 0.5 & 1 & 6.644 & 3.451 & 10.095 & 1500 & no \\
10000 & 0.5 & 100 & 18.265 & 12.695 & 30.960 & 1500 & no \\
10000 & 0.5 & 1000 & 51.114 & 34.712 & 85.826 & 590 & yes \\
10000 & 0.5 & 10000 & 1814.280 & 1405.834 & 3220.114 & 1500 & no \\
10000 & 0.9 & 1 & 6.701 & 3.491 & 10.192 & 1500 & no \\
10000 & 0.9 & 100 & 17.924 & 12.358 & 30.283 & 1500 & no \\
10000 & 0.9 & 1000 & 77.099 & 52.264 & 129.363 & 894 & yes \\
10000 & 0.9 & 10000 & 409.419 & 316.052 & 725.471 & 347 & yes \\
10000 & 1.0 & 1 & 6.450 & 3.325 & 9.775 & 1500 & no \\
10000 & 1.0 & 100 & 17.614 & 12.043 & 29.657 & 1500 & no \\
10000 & 1.0 & 1000 & 61.384 & 43.419 & 104.804 & 646 & yes \\
10000 & 1.0 & 10000 & 593.192 & 455.372 & 1048.564 & 512 & yes \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 2-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async2_c1k}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 1.385 & 0.727 & 2.111 & 1500 & no \\
1000 & 0.0 & 100 & 2.453 & 1.799 & 4.251 & 1500 & no \\
1000 & 0.0 & 1000 & 11.968 & 10.992 & 22.960 & 1500 & no \\
1000 & 0.0 & 10000 & 106.692 & 103.088 & 209.780 & 936 & yes \\
1000 & 0.5 & 1 & 1.384 & 0.723 & 2.107 & 1500 & no \\
1000 & 0.5 & 100 & 3.077 & 2.379 & 5.456 & 1500 & no \\
1000 & 0.5 & 1000 & 12.795 & 11.778 & 24.573 & 1500 & no \\
1000 & 0.5 & 10000 & 173.523 & 167.757 & 341.280 & 1500 & no \\
1000 & 0.9 & 1 & 1.386 & 0.725 & 2.112 & 1500 & no \\
1000 & 0.9 & 100 & 2.933 & 2.255 & 5.188 & 1500 & no \\
1000 & 0.9 & 1000 & 11.493 & 10.516 & 22.009 & 1500 & no \\
1000 & 0.9 & 10000 & 118.520 & 114.517 & 233.037 & 1040 & yes \\
1000 & 1.0 & 1 & 1.395 & 0.733 & 2.128 & 1500 & no \\
1000 & 1.0 & 100 & 2.839 & 2.159 & 4.998 & 1500 & no \\
1000 & 1.0 & 1000 & 12.499 & 11.482 & 23.981 & 1500 & no \\
1000 & 1.0 & 10000 & 153.224 & 148.101 & 301.325 & 1330 & yes \\
\midrule
5000 & 0.0 & 1 & 3.704 & 1.913 & 5.618 & 1500 & no \\
5000 & 0.0 & 100 & 9.307 & 7.451 & 16.758 & 1500 & no \\
5000 & 0.0 & 1000 & 55.617 & 44.581 & 100.198 & 1500 & no \\
5000 & 0.0 & 10000 & 610.388 & 533.825 & 1144.213 & 1109 & yes \\
5000 & 0.5 & 1 & 3.704 & 1.924 & 5.628 & 1500 & no \\
5000 & 0.5 & 100 & 8.363 & 6.511 & 14.874 & 1500 & no \\
5000 & 0.5 & 1000 & 50.917 & 39.896 & 90.813 & 1500 & no \\
5000 & 0.5 & 10000 & 842.439 & 739.238 & 1581.677 & 1500 & no \\
5000 & 0.9 & 1 & 3.650 & 1.900 & 5.551 & 1500 & no \\
5000 & 0.9 & 100 & 4.368 & 3.526 & 7.894 & 672 & yes \\
5000 & 0.9 & 1000 & 59.591 & 48.491 & 108.082 & 1500 & no \\
5000 & 0.9 & 10000 & 724.270 & 635.492 & 1359.762 & 1290 & yes \\
5000 & 1.0 & 1 & 3.649 & 1.884 & 5.533 & 1500 & no \\
5000 & 1.0 & 100 & 8.867 & 7.000 & 15.867 & 1500 & no \\
5000 & 1.0 & 1000 & 56.672 & 45.656 & 102.328 & 1500 & no \\
5000 & 1.0 & 10000 & 223.448 & 196.673 & 420.120 & 388 & yes \\
\midrule
10000 & 0.0 & 1 & 6.489 & 3.356 & 9.845 & 1500 & no \\
10000 & 0.0 & 100 & 17.615 & 12.049 & 29.664 & 1500 & no \\
10000 & 0.0 & 1000 & 109.527 & 75.092 & 184.619 & 1241 & yes \\
10000 & 0.0 & 10000 & 1809.662 & 1401.539 & 3211.200 & 1500 & no \\
10000 & 0.5 & 1 & 6.715 & 3.542 & 10.257 & 1500 & no \\
10000 & 0.5 & 100 & 18.375 & 12.759 & 31.134 & 1500 & no \\
10000 & 0.5 & 1000 & 68.272 & 46.704 & 114.976 & 777 & yes \\
10000 & 0.5 & 10000 & 1799.303 & 1391.844 & 3191.147 & 1500 & no \\
10000 & 0.9 & 1 & 6.507 & 3.347 & 9.854 & 1500 & no \\
10000 & 0.9 & 100 & 18.222 & 12.656 & 30.878 & 1500 & no \\
10000 & 0.9 & 1000 & 92.171 & 64.727 & 156.898 & 988 & yes \\
10000 & 0.9 & 10000 & 982.597 & 758.822 & 1741.419 & 831 & yes \\
10000 & 1.0 & 1 & 6.498 & 3.337 & 9.836 & 1500 & no \\
10000 & 1.0 & 100 & 17.731 & 12.172 & 29.902 & 1500 & no \\
10000 & 1.0 & 1000 & 104.670 & 72.824 & 177.494 & 1146 & yes \\
10000 & 1.0 & 10000 & 1324.072 & 1021.963 & 2346.036 & 1112 & yes \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 2-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async2_c1M}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 1.866 & 1.333 & 3.199 & 1500 & no \\
1000 & 0.0 & 100 & 0.412 & 0.320 & 0.732 & 210 & yes \\
1000 & 0.0 & 1000 & 1.582 & 1.460 & 3.042 & 225 & yes \\
1000 & 0.0 & 10000 & 25.417 & 24.345 & 49.761 & 543 & yes \\
1000 & 0.5 & 1 & 1.874 & 1.349 & 3.223 & 1500 & no \\
1000 & 0.5 & 100 & 0.960 & 0.778 & 1.738 & 487 & yes \\
1000 & 0.5 & 1000 & 1.290 & 1.192 & 2.482 & 181 & yes \\
1000 & 0.5 & 10000 & 7.660 & 7.313 & 14.973 & 164 & yes \\
1000 & 0.9 & 1 & 1.840 & 1.323 & 3.162 & 1500 & no \\
1000 & 0.9 & 100 & 1.007 & 0.813 & 1.820 & 512 & yes \\
1000 & 0.9 & 1000 & 1.958 & 1.819 & 3.777 & 287 & yes \\
1000 & 0.9 & 10000 & 25.201 & 24.078 & 49.279 & 554 & yes \\
1000 & 1.0 & 1 & 1.859 & 1.335 & 3.194 & 1500 & no \\
1000 & 1.0 & 100 & 1.007 & 0.801 & 1.808 & 502 & yes \\
1000 & 1.0 & 1000 & 1.761 & 1.638 & 3.398 & 260 & yes \\
1000 & 1.0 & 10000 & 7.434 & 7.095 & 14.530 & 162 & yes \\
\midrule
5000 & 0.0 & 1 & 4.127 & 3.041 & 7.168 & 1500 & no \\
5000 & 0.0 & 100 & 7.816 & 6.832 & 14.648 & 1294 & yes \\
5000 & 0.0 & 1000 & 19.430 & 16.689 & 36.118 & 709 & yes \\
5000 & 0.0 & 10000 & 99.088 & 90.916 & 190.005 & 200 & yes \\
5000 & 0.5 & 1 & 4.133 & 3.047 & 7.180 & 1500 & no \\
5000 & 0.5 & 100 & 3.895 & 3.344 & 7.239 & 702 & yes \\
5000 & 0.5 & 1000 & 11.788 & 10.126 & 21.914 & 430 & yes \\
5000 & 0.5 & 10000 & 125.166 & 115.148 & 240.314 & 251 & yes \\
5000 & 0.9 & 1 & 4.602 & 3.411 & 8.013 & 1500 & no \\
5000 & 0.9 & 100 & 5.733 & 5.030 & 10.763 & 967 & yes \\
5000 & 0.9 & 1000 & 11.853 & 10.181 & 22.035 & 435 & yes \\
5000 & 0.9 & 10000 & 107.786 & 99.962 & 207.748 & 219 & yes \\
5000 & 1.0 & 1 & 4.133 & 3.082 & 7.215 & 1500 & no \\
5000 & 1.0 & 100 & 8.913 & 7.828 & 16.740 & 1500 & no \\
5000 & 1.0 & 1000 & 24.361 & 21.046 & 45.406 & 863 & yes \\
5000 & 1.0 & 10000 & 178.782 & 163.922 & 342.703 & 372 & yes \\
\midrule
10000 & 0.0 & 1 & 7.092 & 5.284 & 12.377 & 1500 & no \\
10000 & 0.0 & 100 & 17.037 & 14.152 & 31.189 & 1500 & no \\
10000 & 0.0 & 1000 & 91.540 & 70.179 & 161.719 & 1500 & no \\
10000 & 0.0 & 10000 & 248.559 & 212.725 & 461.284 & 259 & yes \\
10000 & 0.5 & 1 & 7.373 & 5.449 & 12.821 & 1500 & no \\
10000 & 0.5 & 100 & 16.647 & 13.691 & 30.339 & 1500 & no \\
10000 & 0.5 & 1000 & 88.522 & 68.186 & 156.707 & 1428 & yes \\
10000 & 0.5 & 10000 & 254.946 & 217.545 & 472.491 & 270 & yes \\
10000 & 0.9 & 1 & 7.030 & 5.244 & 12.274 & 1500 & no \\
10000 & 0.9 & 100 & 18.186 & 15.248 & 33.434 & 1500 & no \\
10000 & 0.9 & 1000 & 93.278 & 71.939 & 165.217 & 1500 & no \\
10000 & 0.9 & 10000 & 568.965 & 486.996 & 1055.961 & 591 & yes \\
10000 & 1.0 & 1 & 7.043 & 5.248 & 12.290 & 1500 & no \\
10000 & 1.0 & 100 & 17.181 & 14.228 & 31.410 & 1500 & no \\
10000 & 1.0 & 1000 & 91.766 & 70.413 & 162.179 & 1500 & no \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 4-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async4_c10}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 1.846 & 1.327 & 3.173 & 1500 & no \\
1000 & 0.0 & 100 & 0.990 & 0.784 & 1.774 & 500 & yes \\
1000 & 0.0 & 1000 & 1.793 & 1.664 & 3.457 & 257 & yes \\
1000 & 0.0 & 10000 & 10.152 & 9.697 & 19.849 & 218 & yes \\
1000 & 0.5 & 1 & 1.842 & 1.323 & 3.165 & 1500 & no \\
1000 & 0.5 & 100 & 0.599 & 0.468 & 1.067 & 300 & yes \\
1000 & 0.5 & 1000 & 4.097 & 3.845 & 7.942 & 532 & yes \\
1000 & 0.5 & 10000 & 16.522 & 15.797 & 32.319 & 353 & yes \\
1000 & 0.9 & 1 & 1.848 & 1.325 & 3.173 & 1500 & no \\
1000 & 0.9 & 100 & 1.249 & 1.013 & 2.263 & 648 & yes \\
1000 & 0.9 & 1000 & 2.827 & 2.628 & 5.456 & 421 & yes \\
1000 & 0.9 & 10000 & 7.997 & 7.629 & 15.627 & 174 & yes \\
1000 & 1.0 & 1 & 1.850 & 1.329 & 3.179 & 1500 & no \\
1000 & 1.0 & 100 & 0.484 & 0.379 & 0.863 & 244 & yes \\
1000 & 1.0 & 1000 & 2.135 & 1.978 & 4.112 & 319 & yes \\
1000 & 1.0 & 10000 & 8.691 & 8.297 & 16.988 & 191 & yes \\
\midrule
5000 & 0.0 & 1 & 4.369 & 3.262 & 7.631 & 1500 & no \\
5000 & 0.0 & 100 & 3.514 & 3.045 & 6.559 & 588 & yes \\
5000 & 0.0 & 1000 & 18.310 & 15.778 & 34.089 & 649 & yes \\
5000 & 0.0 & 10000 & 83.805 & 77.481 & 161.286 & 170 & yes \\
5000 & 0.5 & 1 & 4.208 & 3.109 & 7.317 & 1500 & no \\
5000 & 0.5 & 100 & 7.026 & 6.153 & 13.178 & 1193 & yes \\
5000 & 0.5 & 1000 & 10.014 & 8.585 & 18.598 & 366 & yes \\
5000 & 0.5 & 10000 & 115.646 & 106.254 & 221.899 & 230 & yes \\
5000 & 0.9 & 1 & 4.155 & 3.064 & 7.218 & 1500 & no \\
5000 & 0.9 & 100 & 5.699 & 4.949 & 10.648 & 961 & yes \\
5000 & 0.9 & 1000 & 15.458 & 13.284 & 28.742 & 564 & yes \\
5000 & 0.9 & 10000 & 91.883 & 84.458 & 176.341 & 183 & yes \\
5000 & 1.0 & 1 & 4.205 & 3.108 & 7.314 & 1500 & no \\
5000 & 1.0 & 100 & 9.002 & 7.819 & 16.821 & 1500 & no \\
5000 & 1.0 & 1000 & 5.995 & 5.130 & 11.125 & 221 & yes \\
5000 & 1.0 & 10000 & 124.074 & 114.160 & 238.235 & 257 & yes \\
\midrule
10000 & 0.0 & 1 & 6.991 & 5.166 & 12.157 & 1500 & no \\
10000 & 0.0 & 100 & 17.233 & 14.338 & 31.571 & 1500 & no \\
10000 & 0.0 & 1000 & 64.511 & 49.090 & 113.601 & 1085 & yes \\
10000 & 0.0 & 10000 & 251.879 & 214.877 & 466.756 & 270 & yes \\
10000 & 0.5 & 1 & 6.961 & 5.172 & 12.133 & 1500 & no \\
10000 & 0.5 & 100 & 17.093 & 14.147 & 31.240 & 1500 & no \\
10000 & 0.5 & 1000 & 91.462 & 70.109 & 161.571 & 1500 & no \\
10000 & 0.5 & 10000 & 234.762 & 200.086 & 434.847 & 253 & yes \\
10000 & 0.9 & 1 & 7.033 & 5.229 & 12.262 & 1500 & no \\
10000 & 0.9 & 100 & 16.695 & 13.756 & 30.451 & 1500 & no \\
10000 & 0.9 & 1000 & 91.453 & 70.097 & 161.550 & 1500 & no \\
10000 & 1.0 & 1 & 6.939 & 5.169 & 12.108 & 1500 & no \\
10000 & 1.0 & 100 & 16.821 & 13.875 & 30.696 & 1500 & no \\
10000 & 1.0 & 1000 & 92.146 & 70.802 & 162.948 & 1500 & no \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 4-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async4_c1k}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 1.871 & 1.345 & 3.216 & 1500 & no \\
1000 & 0.0 & 100 & 0.738 & 0.596 & 1.333 & 372 & yes \\
1000 & 0.0 & 1000 & 1.242 & 1.146 & 2.388 & 178 & yes \\
1000 & 0.0 & 10000 & 10.024 & 9.577 & 19.602 & 212 & yes \\
1000 & 0.5 & 1 & 1.848 & 1.329 & 3.177 & 1500 & no \\
1000 & 0.5 & 100 & 0.493 & 0.390 & 0.883 & 248 & yes \\
1000 & 0.5 & 1000 & 1.704 & 1.578 & 3.282 & 251 & yes \\
1000 & 0.5 & 10000 & 8.382 & 8.010 & 16.392 & 182 & yes \\
1000 & 0.9 & 1 & 1.829 & 1.315 & 3.144 & 1500 & no \\
1000 & 0.9 & 100 & 0.488 & 0.381 & 0.870 & 233 & yes \\
1000 & 0.9 & 1000 & 1.555 & 1.440 & 2.995 & 225 & yes \\
1000 & 0.9 & 10000 & 8.833 & 8.435 & 17.267 & 193 & yes \\
1000 & 1.0 & 1 & 1.859 & 1.338 & 3.197 & 1500 & no \\
1000 & 1.0 & 100 & 1.503 & 1.227 & 2.730 & 797 & yes \\
1000 & 1.0 & 1000 & 1.552 & 1.437 & 2.989 & 226 & yes \\
1000 & 1.0 & 10000 & 8.505 & 8.120 & 16.625 & 181 & yes \\
\midrule
5000 & 0.0 & 1 & 4.223 & 3.113 & 7.336 & 1500 & no \\
5000 & 0.0 & 100 & 3.041 & 2.626 & 5.667 & 501 & yes \\
5000 & 0.0 & 1000 & 19.273 & 16.593 & 35.866 & 693 & yes \\
5000 & 0.0 & 10000 & 105.070 & 97.417 & 202.487 & 218 & yes \\
5000 & 0.5 & 1 & 4.209 & 3.107 & 7.316 & 1500 & no \\
5000 & 0.5 & 100 & 3.268 & 2.823 & 6.091 & 579 & yes \\
5000 & 0.5 & 1000 & 12.786 & 11.006 & 23.792 & 452 & yes \\
5000 & 0.5 & 10000 & 111.482 & 102.554 & 214.036 & 228 & yes \\
5000 & 0.9 & 1 & 4.207 & 3.111 & 7.318 & 1500 & no \\
5000 & 0.9 & 100 & 4.268 & 3.666 & 7.935 & 766 & yes \\
5000 & 0.9 & 1000 & 7.506 & 6.439 & 13.945 & 268 & yes \\
5000 & 0.9 & 10000 & 71.164 & 65.615 & 136.779 & 144 & yes \\
5000 & 1.0 & 1 & 4.212 & 3.112 & 7.324 & 1500 & no \\
5000 & 1.0 & 100 & 5.653 & 4.912 & 10.565 & 955 & yes \\
5000 & 1.0 & 1000 & 34.229 & 29.443 & 63.672 & 1239 & yes \\
5000 & 1.0 & 10000 & 103.946 & 95.293 & 199.239 & 212 & yes \\
\midrule
10000 & 0.0 & 1 & 7.050 & 5.237 & 12.287 & 1500 & no \\
10000 & 0.0 & 100 & 18.440 & 15.500 & 33.940 & 1500 & no \\
10000 & 0.0 & 1000 & 88.234 & 67.797 & 156.031 & 1436 & yes \\
10000 & 0.0 & 10000 & 307.776 & 262.527 & 570.303 & 330 & yes \\
10000 & 0.5 & 1 & 7.017 & 5.212 & 12.229 & 1500 & no \\
10000 & 0.5 & 100 & 17.177 & 14.243 & 31.420 & 1500 & no \\
10000 & 0.5 & 1000 & 67.206 & 51.253 & 118.459 & 1117 & yes \\
10000 & 0.5 & 10000 & 490.283 & 419.140 & 909.422 & 513 & yes \\
10000 & 0.9 & 1 & 7.200 & 5.361 & 12.561 & 1500 & no \\
10000 & 0.9 & 100 & 17.303 & 14.413 & 31.716 & 1500 & no \\
10000 & 0.9 & 1000 & 91.953 & 70.600 & 162.553 & 1500 & no \\
10000 & 1.0 & 1 & 7.924 & 5.904 & 13.828 & 1500 & no \\
10000 & 1.0 & 100 & 16.964 & 14.071 & 31.035 & 1500 & no \\
10000 & 1.0 & 1000 & 73.372 & 55.968 & 129.341 & 1224 & yes \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 4-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async4_c1M}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 2.912 & 2.405 & 5.317 & 1500 & no \\
1000 & 0.0 & 100 & 1.560 & 1.348 & 2.908 & 581 & yes \\
1000 & 0.0 & 1000 & 1.603 & 1.514 & 3.117 & 200 & yes \\
1000 & 0.0 & 10000 & 12.984 & 12.628 & 25.612 & 311 & yes \\
1000 & 0.5 & 1 & 2.883 & 2.416 & 5.299 & 1500 & no \\
1000 & 0.5 & 100 & 1.541 & 1.337 & 2.877 & 579 & yes \\
1000 & 0.5 & 1000 & 2.453 & 2.325 & 4.778 & 299 & yes \\
1000 & 0.5 & 10000 & 23.691 & 23.083 & 46.774 & 554 & yes \\
1000 & 0.9 & 1 & 2.844 & 2.372 & 5.216 & 1500 & no \\
1000 & 0.9 & 100 & 1.480 & 1.297 & 2.777 & 559 & yes \\
1000 & 0.9 & 1000 & 2.126 & 2.007 & 4.132 & 259 & yes \\
1000 & 0.9 & 10000 & 17.548 & 17.081 & 34.629 & 395 & yes \\
1000 & 1.0 & 1 & 2.829 & 2.367 & 5.195 & 1500 & no \\
1000 & 1.0 & 100 & 4.103 & 3.609 & 7.712 & 1500 & no \\
1000 & 1.0 & 1000 & 4.080 & 3.876 & 7.956 & 506 & yes \\
1000 & 1.0 & 10000 & 19.168 & 18.693 & 37.861 & 431 & yes \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=10$, 8-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async8_c10}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 4.012 & 3.412 & 7.425 & 1500 & no \\
1000 & 0.0 & 100 & 2.603 & 2.297 & 4.900 & 985 & yes \\
1000 & 0.0 & 1000 & 7.363 & 6.969 & 14.332 & 901 & yes \\
1000 & 0.0 & 10000 & 19.071 & 18.552 & 37.623 & 460 & yes \\
1000 & 0.5 & 1 & 2.875 & 2.410 & 5.284 & 1500 & no \\
1000 & 0.5 & 100 & 1.053 & 0.910 & 1.963 & 379 & yes \\
1000 & 0.5 & 1000 & 2.815 & 2.669 & 5.483 & 347 & yes \\
1000 & 0.5 & 10000 & 13.687 & 13.314 & 27.001 & 321 & yes \\
1000 & 0.9 & 1 & 2.836 & 2.369 & 5.205 & 1500 & no \\
1000 & 0.9 & 100 & 4.084 & 3.594 & 7.678 & 1500 & no \\
1000 & 0.9 & 1000 & 6.164 & 5.871 & 12.035 & 744 & yes \\
1000 & 0.9 & 10000 & 16.355 & 15.940 & 32.295 & 376 & yes \\
1000 & 1.0 & 1 & 2.845 & 2.384 & 5.229 & 1500 & no \\
1000 & 1.0 & 100 & 4.136 & 3.641 & 7.777 & 1500 & no \\
1000 & 1.0 & 1000 & 4.450 & 4.195 & 8.645 & 530 & yes \\
1000 & 1.0 & 10000 & 9.680 & 9.433 & 19.113 & 218 & yes \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000$, 8-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async8_c1k}
\end{table}
\begin{table}[h]
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{cccccccc}
\toprule
$n$ & $s$ & $N$ & Comp. (s) & Comm. (s) & Tot. (s) & Iter. & Cvg. \\
\midrule
1000 & 0.0 & 1 & 3.012 & 2.542 & 5.554 & 1500 & no \\
1000 & 0.0 & 100 & 3.472 & 3.051 & 6.523 & 1265 & yes \\
1000 & 0.0 & 1000 & 6.707 & 6.381 & 13.088 & 840 & yes \\
1000 & 0.0 & 10000 & 26.509 & 25.798 & 52.307 & 638 & yes \\
1000 & 0.5 & 1 & 2.888 & 2.396 & 5.284 & 1500 & no \\
1000 & 0.5 & 100 & 4.400 & 3.944 & 8.344 & 1500 & no \\
1000 & 0.5 & 1000 & 1.471 & 1.379 & 2.850 & 172 & yes \\
1000 & 0.5 & 10000 & 14.479 & 14.102 & 28.581 & 337 & yes \\
1000 & 0.9 & 1 & 2.938 & 2.469 & 5.407 & 1500 & no \\
1000 & 0.9 & 100 & 4.043 & 3.548 & 7.591 & 1500 & no \\
1000 & 0.9 & 1000 & 12.232 & 11.679 & 23.911 & 1500 & no \\
1000 & 0.9 & 10000 & 9.758 & 9.504 & 19.261 & 220 & yes \\
1000 & 1.0 & 1 & 2.859 & 2.398 & 5.257 & 1500 & no \\
1000 & 1.0 & 100 & 1.587 & 1.378 & 2.965 & 588 & yes \\
1000 & 1.0 & 1000 & 1.620 & 1.533 & 3.153 & 204 & yes \\
\bottomrule
\end{tabular}
\caption{Computation time, Communication time, Total time of execution, and number of iterations to convergence for $c=1000000$, 8-node asynchronous federated Sinkhorn. Only the node with the highest total execution time was kept, as others need to wait for the slowest one.}
\label{table:big_table_async8_c1M}
\end{table}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{10}
\bibitem{altschuler2019kmeans}
Jason Altschuler, Francis Bach, and Alessandro Rudi.
\newblock Approximating the k-means objective in sublinear time.
\newblock In {\em Proceedings of the International Conference on Machine
Learning (ICML)}, page n/a, 2019.
\bibitem{blanchet2019quantifying}
Jose Blanchet and Karthyek Murthy.
\newblock Quantifying distributional model risk via optimal transport.
\newblock {\em Mathematics of Operations Research}, 44(2):565--600, 2019.
\bibitem{chetlur2014cudnn}
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John
Tran, Bryan Catanzaro, and Evan Shelhamer.
\newblock cudnn: Efficient primitives for deep learning.
\newblock {\em arXiv preprint arXiv:1410.0759}, 2014.
\bibitem{NIPS2013_af21d0c9}
Marco Cuturi.
\newblock Sinkhorn distances: Lightspeed computation of optimal transport.
\newblock In C.J. Burges, L.~Bottou, M.~Welling, Z.~Ghahramani, and K.Q.
Weinberger, editors, {\em Advances in Neural Information Processing Systems},
volume~26. Curran Associates, Inc., 2013.
\bibitem{cuturi2013sinkhorn}
Marco Cuturi.
\newblock Sinkhorn distances: Lightspeed computation of optimal transport.
\newblock In {\em Advances in Neural Information Processing Systems (NIPS)},
pages 2292--2300, 2013.
\bibitem{genevay2016stochastic}
Aude Genevay, Gabriel Peyré, and Marco Cuturi.
\newblock Stochastic optimization for large-scale optimal transport.
\newblock In {\em Advances in Neural Information Processing Systems (NIPS)},
pages 3440--3448, 2016.
\bibitem{he2020fedmlresearchlibrarybenchmark}
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi~Zhang, Hongyi Wang, Xiaoyang
Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Xinghua Zhu, Jianzong
Wang, Li~Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang,
Murali Annavaram, and Salman Avestimehr.
\newblock Fedml: A research library and benchmark for federated machine
learning, 2020.
\bibitem{konecny2016federated}
Jakub Konečnỳ, H.~Brendan McMahan, Daniel Ramage, and Peter Richtárik.
\newblock Federated learning: Strategies for improving communication
efficiency.
\newblock In {\em NIPS Workshop on Optimization for Machine Learning}, 2016.
\bibitem{lian2018asynchronous}
Xiangru Lian, Ce~Zhang, Huan Zhang, Cho-Jui Hsieh, and Ji~Liu.
\newblock Asynchronous decentralized parallel stochastic gradient descent.
\newblock In {\em Proceedings of the International Conference on Machine
Learning (ICML)}, page n/a, 2018.
\bibitem{lin2019don}
Tao Lin, Sebastian~Urban Stich, Kumar~Kshitij Patel, and Martin Jaggi.
\newblock Don't use large mini-batches, use local sgd.
\newblock In {\em Proceedings of the 8th International Conference on Learning
Representations}, 2019.
\bibitem{mcmahan2017communication}
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise~Aguera
y~Arcas.
\newblock Communication-efficient learning of deep networks from decentralized
data.
\newblock In {\em Artificial intelligence and statistics}, pages 1273--1282.
PMLR, 2017.
\bibitem{peng2016arock}
Zhimin Peng, Yangyang Xu, Ming Yan, and Wotao Yin.
\newblock Arock: an algorithmic framework for asynchronous parallel coordinate
updates.
\newblock {\em SIAM Journal on Scientific Computing}, 38(5):A2851--A2879, 2016.
\bibitem{peyre2020computationaloptimaltransport}
Gabriel Peyré and Marco Cuturi.
\newblock Computational optimal transport, 2020.
\bibitem{quinn2004parallel}
Michael~J. Quinn.
\newblock {\em Parallel Programming in C with MPI and OpenMP}.
\newblock McGraw-Hill Education, 2004.
\bibitem{recht2011hogwild}
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu.
\newblock Hogwild!: A lock-free approach to parallelizing stochastic gradient
descent.
\newblock In {\em Advances in Neural Information Processing Systems (NIPS)},
pages 693--701, 2011.
\bibitem{sinkhorn1967concerning}
Richard Sinkhorn and Paul Knopp.
\newblock Concerning nonnegative matrices and doubly stochastic matrices.
\newblock {\em Pacific Journal of Mathematics}, 21(2):343--348, 1967.
\bibitem{sterling2011hpc}
Thomas Sterling, Matthew Anderson, and Maciej Brodowicz.
\newblock {\em High Performance Computing: Modern Systems and Practices}.
\newblock Morgan Kaufmann, 2011.
\bibitem{Villani2009}
Cédric Villani.
\newblock {\em Optimal Transport}.
\newblock Springer Berlin Heidelberg, 2009.
\end{thebibliography}