Birch SGD: A Tree Graph Framework for Local and Asynchronous SGD Methods
Extracted source text · arXiv 2505.09218
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
% \input{math_commands.tex}
\usepackage{amsthm}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage[hypertexnames=false]{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{subcaption}
\usepackage{adjustbox}
\DeclareSymbolFont{extraup}{U}{zavm}{m}{n}
\DeclareMathSymbol{\varheart}{\mathalpha}{extraup}{86}
\DeclareMathSymbol{\vardiamond}{\mathalpha}{extraup}{87}
\input{macros}
% \theoremstyle{definition}
% \newtheorem{theorem}{Theorem}[section]
% \newtheorem{lemma}[theorem]{Lemma}
% \newtheorem{corollary}[theorem]{Corollary}
% \newtheorem{assumption}[theorem]{Assumption}
% \newtheorem{definition}[theorem]{Definition}
% \newtheorem{remark}[theorem]{Remark}
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem*{theorem*}{Theorem}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\newtheorem{example}[theorem]{Example}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\hypersetup{
colorlinks = true, %Colours links instead of ugly boxes
urlcolor = blue, %Colour for external hyperlinks
linkcolor = {red!75!black}, %Colour of internal links
citecolor = {blue!50!black} %Colour of citations
}
\usepackage{tcolorbox}
\usepackage{tikz}
\usetikzlibrary{arrows.meta}
\pgfdeclarelayer{nodelayer}
\pgfdeclarelayer{edgelayer}
\pgfsetlayers{edgelayer,nodelayer,main}
\allowdisplaybreaks
% \newtheorem{definitionhighlighted}{Definition}
\newcommand{\alexander}[1]{\todo[inline]{\textbf{Alexander: }#1}}
\newcommand{\danil}[1]{\todo[inline]{\textbf{Danil: }#1}}
\definecolor{mydarkred}{RGB}{192,25,25}
\definecolor{mydarkgreen}{RGB}{36,124,4}
\definecolor{mydarkblue}{RGB}{25,25,192}
\definecolor{mydarkorange}{RGB}{255,153,51}
\title{Birch SGD: A Tree Graph Framework for \\ Local and Asynchronous SGD Methods}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{%
Alexander Tyurin \\
AXXX, Moscow, Russia \\
Applied AI Institute, Moscow, Russia
\And
Danil Sivtsov \\
AXXX, Moscow, Russia \\
Applied AI Institute, Moscow, Russia
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\usepackage{tcolorbox}
\newtcolorbox{theorembox}{
colback=gray!20,
colframe=gray!20,
boxrule=0.8pt,
before skip=7pt,
after skip=7pt,
boxsep=-1mm,
}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
We propose a new unifying framework, \algname{Birch SGD}, for analyzing and designing distributed \algname{SGD} methods. The central idea is to represent each method as a weighted directed tree, referred to as a \emph{computation tree}. Leveraging this representation, we introduce a general theoretical result that reduces convergence analysis to studying the geometry of these trees. This perspective yields a purely graph-based interpretation of optimization dynamics, offering a new and intuitive foundation for method development. Using \algname{Birch SGD}, we design eight new methods and analyze them alongside previously known ones, with at least six of the new methods shown to have optimal computational time complexity. Our research leads to two key insights: (i) all methods share the same ``iteration rate'' of $\mathcal{O}\left(\nicefrac{(R + 1) L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$, where $R$ the maximum ``tree distance'' along the main branch of a tree; and (ii) different methods exhibit different trade-offs---for example, some update iterates more frequently, improving practical performance, while others are more communication-efficient or focus on other aspects. \algname{Birch SGD} serves as a unifying framework for navigating these trade-offs. We believe these results provide a unified foundation for understanding, analyzing, and designing efficient asynchronous and parallel optimization methods.
\end{abstract}
\section{Introduction}
Optimization is central to machine learning (ML), data science (DS), and federated learning (FL) \citep{konevcny2016federated, bottou2018optimization, kairouz2021advances}. In these domains, stochastic optimization techniques such as stochastic gradient descent (\algname{SGD}) \citep{robbins1951stochastic} and its adaptive variants (\algname{Adam}, \algname{AdamW}, etc) \citep{kingma2014adam, loshchilov2017decoupled} have become the standard approach for tackling large-scale problems \citep{schmidt2021descending}. Due to the rising computational demands of modern functions, the theoretical foundation of distributed algorithms supporting a large number of workers (e.g., CPUs, GPUs, servers) is important \citep{mayer2020scalable, kairouz2021advances, douillard2023diloco}.
% Nonetheless, these algorithms come with their own set of challenges \citep{li2020federateda}. There is the issue of \emph{computation and communication heterogeneity}: while multiple workers operate in parallel, they often do so at different speeds with varying performance, and they may experience intermittent disconnections.
% Additionally, communication channels themselves can be slow and unreliable.
% Nonetheless, these algorithms come with their own set of challenges \citep{li2020federateda}. There is the issue computation heterogeneity, slow communications,
% while multiple workers operate in parallel, they often do so at different speeds with varying performance, and they may experience intermittent disconnections.
% \subsection{Problem Setup}
We consider distributed optimization problems with smooth nonconvex optimization functions:
\begin{align}
\label{eq:main_problem}
\textstyle \min\limits_{x \in \R^d} f(x),
\end{align}
In nonconvex settings, the goal is to find an $\varepsilon$--stationary point, meaning we want to find a random vector $\bar{x}$ such that
$\mathbb{E}\left[\|\nabla f(\bar{x})\|^2\right] \leq \varepsilon$ \citep{nemirovskij1983problem,murty1985some}. The function $f:\R^d \rightarrow \R$ satisfies the following standard assumptions:
\begin{assumption}
\label{ass:lipschitz_constant}
$f$ is differentiable and $L$--smooth: $\norm{\nabla f(x) - \nabla f(y)} \leq L \norm{x - y}$ $\forall x, y \in \R^d.$
\end{assumption}
\begin{assumption}
\label{ass:lower_bound}
There exist $f^* \in \R$ such that $f(x) \geq f^*$ for all $x \in \R^d$.
\end{assumption}
We focus on problems where workers are limited to computing stochastic gradients. Each worker has access to unbiased stochastic gradients, denoted by $\nabla f(x;\xi)$, whose variance is bounded by $\sigma^2$. In the context of ML, this implies that all workers can access the same data, which is practical when training large language and computer vision models. In such scenarios, privacy is not a critical concern, and devices can sample data from the Internet or shared datasets \citep{goodfellow2016deep}.
\begin{assumption}
\label{ass:stochastic_variance_bounded}
For all $x \in \R^d,$ stochastic gradients $\nabla f(x;\xi)$ are unbiased and $\sigma^2$-variance-bounded, i.e., ${\rm \mathbb{E}}_{\xi}[\nabla f(x;\xi)] = \nabla f(x)$ and
${\rm \mathbb{E}}_{\xi}[\|\nabla f(x;\xi) - \nabla f(x)\|^2] \leq \sigma^2,$ where $\sigma^2 \geq 0.$
\end{assumption}
\subsection{Related work}
\label{sec:related_work}
\textbf{One worker and optimal oracle complexity.} With a single worker, the most standard optimization method is the \algname{Vanilla SGD} algorithm, which updates the iterate as
$w^{k+1} = w^k - \gamma \nabla f(w^k; \eta^k),$ where $\{\eta^k\}$ are i.i.d., $w^0 \in \mathbb{R}^d$ is a starting point, $\gamma$ is a step size, and $\Delta \eqdef f(w^0) - f^*.$ \citet{arjevani2022lower,carmon2020lower} showed that \algname{Vanilla SGD} is \emph{optimal} in terms of oracle complexity, which is given by
$\Theta\left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$ for finding an $\varepsilon$-stationary point.
% There are many practical versions of \algname{Vanilla SGD} such as \algname{Adam} \citep{kingma2014adam}. However, first, such methods are not better than \algname{Vanilla SGD} theoretically due to the lower bound. Second, they require either strict assumptions or have much worse theoretical guarantees.
\textbf{Multiple workers and optimal time complexity.} Consider now that we have $n$ workers computing stochastic gradients asynchronously and in parallel. In this setup, there are numerous ways to construct a distributed \algname{SGD} method. The most well-known celebrated and recent approaches include \algname{Synchronized SGD} (\algname{Minibatch SGD}), \algname{Local SGD} \citep{zinkevich2010parallelized,stich2018local}, \algname{Asynchronous SGD} \citep{recht2011hogwild}, \algname{Picky SGD} \citep{cohen2021asynchronous}, \algname{Rennala SGD} \citep{tyurin2023optimal}, and \algname{Ringmaster ASGD} \citep{maranjyan2025ringmaster}. The multi-worker setup is rich and versatile, offering numerous ways to design distributed \algname{SGD} methods.
% \alexander{Draft:}
% One may naturally ask which method offers the best theoretical performance. Note that in the distributed setting, the standard oracle complexity measure becomes less relevant as workers obtain stochastic gradients (through the oracle) in parallel, and their computation times can vary.
% One may naturally ask which method offers the best theoretical performance. Note that in the distributed setting, the standard oracle complexity measure becomes less informative, as workers obtain stochastic gradients in parallel, and their computation times can vary. One way to compare distributed \algname{SGD} methods is through the \emph{$h_i$-fixed computation model} \citep{mishchenko2022asynchronous}. In this model, it is assumed that worker~$i$ requires at most $h_i$ seconds to compute a single stochastic gradient, for all $i \in [n] \eqdef \{1, \dots, n\}.$ Under this setting, \citet{mishchenko2022asynchronous,koloskova2022sharper} showed that \algname{Asynchronous SGD} provably converges faster than \algname{Synchronized SGD}. Surprisingly, the time complexity of \algname{Asynchronous SGD} can be further improved by \algname{Rennala SGD} \citep{tyurin2023optimal} and \algname{Ringmaster ASGD}\footnote{which is \algname{Asynchronous SGD} with an important key modification. See Alg.~\ref{alg:ringmasternew}.} \citep{maranjyan2025ringmaster}, both of which are optimal---not only under the $h_i$-fixed computation model but also under the more general \emph{universal computation model} \citep{tyurin2024tight} (detailed comparison is provided in Section~\ref{sec:related_work_fixed}). However, as we will discuss in more detail later, other factors come into play, such as communication complexity, support for \texttt{AllReduce}, peak bandwidth, and model update frequency.
One may naturally ask which method offers the best theoretical performance. In distributed settings, the standard oracle complexity becomes less informative, as workers compute stochastic gradients in parallel with varying speeds. A more suitable comparison uses the \emph{$h_i$-fixed computation model} \citep{mishchenko2022asynchronous}, where each worker~$i$ needs at most $h_i$ seconds to compute a gradient. In this model, \citet{mishchenko2022asynchronous,koloskova2022sharper} showed that \algname{Asynchronous SGD} outperforms \algname{Synchronized SGD}. Its time complexity is further improved by \algname{Rennala SGD} \citep{tyurin2023optimal} and \algname{Ringmaster ASGD}\footnote{\algname{Asynchronous SGD} with a key modification; see Alg.\ref{alg:ringmasternew}.} \citep{maranjyan2025ringmaster}, both optimal under this and the more general \emph{universal computation model} \citep{tyurin2024tight} (see Section~\ref{sec:related_work_fixed}). However, as we will discuss in more detail later, other factors come into play, such as communication complexity, support for \texttt{AllReduce}, peak bandwidth, and model update frequency.
% \makeatletter
% \newcommand{\setword}[2]{%
% \phantomsection
% #1\def\@currentlabel{\unexpanded{#1}}\label{#2}%
% }
% \makeatother
% These recent developments raise several intriguing questions. We know that \algname{Rennala SGD} and \algname{Ringmaster ASGD} are optimal methods which, as we will see later, differ in both design and structure, each offering its own advantages and trade-offs.
% This naturally leads to our main questions:
% \emph{Are there other optimal methods? Can we develop a unified framework that encompasses all distributed \algname{SGD} methods and offers theoretical guidance? What fundamental properties make these methods optimal? And, given different system constraints, which method should one choose?}
These developments raise several important questions. \algname{Rennala SGD} and \algname{Ringmaster ASGD} are known to be optimal, yet differ in design and structure, each with distinct advantages and trade-offs. This leads to our central questions: \emph{Are there other optimal methods? Can we develop a unified framework that encompasses all distributed \algname{SGD} methods and offers theoretical guidance? What fundamental properties make these methods optimal? And, given different system constraints, which method should one choose?}
% They also considered \algname{Local SGD}, where each worker computes a stochastic gradient and updates the iterate independently. This method converges after $\cO\left(\max_{i \in [n]} h_i \times \left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)\right)$ seconds. Finally, they considered \algname{Asynchronous SGD}, where each worker computes a stochastic gradient and updates the iterate independently. This method converges after $\cO\left(\max_{i \in [n]} h_i \times \left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)\right)$ seconds.
% This method converges after $\operatorname{O}\left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{n \varepsilon^2}\right)$ iterations \citep{cotter2011better, goyal2017accurate, gower2019sgd} and after
% \begin{align}
% \label{eq:mini_batch_compl}
% \textstyle \operatorname{O}\left(\max\limits_{i \in [n]} h_i \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
% \end{align} seconds because this method waits for the slowest worker with the time $\max_{i \in [n]} h_i.$
\subsection{Contributions}
\label{sec:contr}
$\spadesuit$ \textbf{New framework: \algnamebf{Birch SGD} (Section~\ref{sec:birch}).} We propose \algname{Birch SGD}, a unifying framework that captures a wide range of distributed \algname{SGD} methods. The key idea is that \algname{SGD} methods can be represented using weighted directed trees, which we refer to as \emph{computation trees} (see Figure~\ref{fig:threeimages}). We develop a new theoretical result, Theorem~\ref{thm:main}, that reduces the analysis of \algname{SGD} methods to analyzing of the structure of these computation trees. The proofs become purely geometric and topological in nature, offering geometric intuition for the design of new methods. Moreover, this geometric viewpoint leads to tighter time complexity guarantees even for \algname{Local SGD} (\algname{FedAvg}) approaches \citep{mcmahan2017communication}, as we illustrate in Section~\ref{sec:compare}.
$\clubsuit$ \textbf{Eight new methods (Table~\ref{tab:methods_summary_check} and Section~\ref{sec:alg}).} Using \algname{Birch SGD}, we identify eight new methods in addition to those already known. For the first time, we prove that at least \textbf{six of these newly discovered methods are computationally optimal}, matching the lower bound~\citep{tyurin2023optimal}. We compare all methods across several dimensions, including computational and communication complexity, \texttt{AllReduce} compatibility, peak bandwidth, and model update frequency. Our improvements: i) our newly developed \algname{Async-Local SGD} and \algname{Async-Batch SGD} provably improve the communication complexity of \algname{Ringmaster ASGD} while preserving asynchronicity; ii) we introduce \algname{Cycle SGD}, which provably reduces peak bandwidth compared to all existing methods; iii) we propose a key modification to the family of local methods and design \algname{Local SGD} and \algname{Dual-Process SGD} that, for the first time in the literature, achieve the optimal time complexities within this family and improve upon the classical approach (see Section~\ref{sec:compare}); iv) for multi-cluster settings, we introduce \algname{Local-Async SGD} and \algname{Nested Local-Async SGD}, incorporating a carefully designed synchronization mechanism that guarantees optimality in computational time complexity; v) we develop a flexible meta-algorithm, \algname{Meta Local SGD}, which supports arbitrary synchronization strategies, while incorporating a ``Hard Sync'' mechanism to guarantee convergence rates and to temper overly chaotic synchronization. As a byproduct, we prove that frequent model updates of fully asynchronous methods can lead to faster convergence and improve optimal \algname{Rennala SGD}.
$\vardiamond$ \textbf{Insights and Guidelines (Section~\ref{sec:conlusion}).} We observe that there is no silver bullet—each method has its own advantages and disadvantages. Some methods update the iterates more frequently, making them more appealing in practice, while others prioritize communication efficiency, support \texttt{AllReduce}, or focus on different aspects. Through our new framework, we uncover insights that provide deeper intuition and a simpler perspective on asynchronous, local, and parallel optimization methods.
\section{\algname{Birch SGD}: A General View of \algname{SGD} Methods}
\label{sec:birch}
We begin our work by observing that various \algname{SGD} methods, including \algname{Vanilla SGD}, \algname{Asynchronous SGD}, \algname{Local SGD}, among others, can be constructed in the manner described in Algorithm~\ref{alg:framework}.
% \begin{algorithm}[H]
% \caption{\algname{Birch SGD} framework}
% \label{alg:framework}
% \begin{algorithmic}
% \STATE \textbf{Input:} starting point $w^{0} \in \R^d$, step size $\gamma \geq 0$
% \STATE Initialize the set of computed points: $V = \{w^{0}\}$
% \STATE (and the set of edges $E = \emptyset$)
% \FOR{$k = 0, 1, 2, \dots$}
% \STATE Choose any point $w_{\textnormal{base}} \in V$ from which to compute a new point
% \STATE Choose any point $w_{\textnormal{grad}} \in V$ at which to compute a stochastic gradient
% \STATE Compute the new point\footnotemark: $w^{k + 1} = w_{\textnormal{base}} - \gamma \nabla f(w_{\textnormal{grad}}; \eta),$ \qquad $\eta \sim \mathcal{D}_{\eta}$
% \STATE Add $w^{k+1}$ to the set of computed points $V$
% \STATE (and add the edge with weight $(w_{\textnormal{base}}, w^{k + 1}, \nabla f(w_{\textnormal{grad}}; \eta))$ to the set of edges $E$)
% \ENDFOR
% \end{algorithmic}
% \end{algorithm}
% \begin{figure}[t]
% \centering
% \begin{minipage}[t]{.76\textwidth}
% \begin{algorithm}[H]
% \caption{\algname{Birch SGD} framework}
% \label{alg:framework}
% \begin{algorithmic}
% \STATE \textbf{Input:} starting point $w^{0} \in \R^d$, step size $\gamma \geq 0$
% \STATE Initialize the set of computed points: $V = \{w^{0}\}$
% \STATE (and the set of edges $E = \emptyset$)
% \FOR{$k = 0, 1, 2, \dots$}
% \STATE Choose any point $w_{\textnormal{base}} \in V$ from which to compute a new point
% \STATE Choose any point $w_{\textnormal{grad}} \in V$ at which to compute a stochastic gradient
% \STATE Compute the new point\footnotemark: $$w^{k + 1} = w_{\textnormal{base}} - \gamma \nabla f(w_{\textnormal{grad}}; \eta), \quad \eta \sim \mathcal{D}_{\xi}$$
% \STATE \vspace{-0.4cm}Add $w^{k+1}$ to the set of computed points $V$
% \STATE (and add the edge with weight $(w_{\textnormal{base}}, w^{k + 1}, \nabla f(w_{\textnormal{grad}}; \eta))$ to the set of edges $E$)
% \ENDFOR
% \end{algorithmic}
% \end{algorithm}
% \end{minipage}\hfill
% \begin{minipage}[t]{.23\textwidth}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.84\linewidth]{Grabar_February_azure_1904_gtg.jpg}
% \caption*{\phantom{a}\emph{February Azure}, \\\phantom{a}Igor Grabar. 1904. \\\phantom{a}Tretyakov Gallery.}
% \end{figure}
% \end{minipage}
% \end{figure}
\begin{figure}[t]
\centering
\begin{minipage}[t]{.76\textwidth}
\begin{algorithm}[H]
\caption{\algname{Birch SGD} framework}
\label{alg:framework}
\begin{algorithmic}
\STATE \textbf{Input:} starting point $w^{0} \in \R^d$, step size $\gamma \geq 0$
\STATE Initialize the set of computed points: $V = \{w^{0}\}$
\STATE (and the set of edges $E = \emptyset$)
\FOR{$k = 0, 1, 2, \dots$}
\STATE Choose any point $w_{\textnormal{base}} \in V$ from which to compute a new point
\STATE Choose any point $w_{\textnormal{grad}} \in V$ at which to compute a stochastic gradient
% \STATE Compute the new point\footnotemark: $$w^{k + 1} = w_{\textnormal{base}} - \gamma \nabla f(w_{\textnormal{grad}}; \eta), \quad \eta \sim \mathcal{D}_{\xi}$$
\STATE Compute the new point\footnotemark: $w^{k + 1} = w_{\textnormal{base}} - \gamma \nabla f(w_{\textnormal{grad}}; \eta), \quad \eta \sim \mathcal{D}_{\xi}$
% \STATE \vspace{-0.4cm}Add $w^{k+1}$ to the set of computed points $V$
\STATE Add $w^{k+1}$ to the set of computed points $V$
\STATE (and add the edge with weight $(w_{\textnormal{base}}, w^{k + 1}, \nabla f(w_{\textnormal{grad}}; \eta))$ to the set of edges $E$)
\ENDFOR
\end{algorithmic}
\end{algorithm}
\end{minipage}\hfill
\begin{minipage}[t]{.23\textwidth}
\begin{figure}[H]
\centering
% \includegraphics[width=0.84\linewidth]{Grabar_February_azure_1904_gtg.jpg}
\includegraphics[width=0.8\linewidth]{Grabar_February_azure_1904_gtg.jpg}
\vspace{-0.2cm}
\caption*{\scriptsize \phantom{aaa}\emph{February Azure}, \\\phantom{aaa}Igor Grabar. 1904.}
% \\\phantom{a}Tretyakov Gallery.}
\end{figure}
\end{minipage}
\end{figure}
Let us explain it. Initially, any \algname{SGD} method starts at some point $w^{0} \in \R^d$, computes a stochastic gradient at $w^{0}$, and then finds a new point $w^{1} = w^{0} - \gamma \nabla f(w^{0}; \cdot),$ which is added to the set $V$ of computed points. In the next step, there are four options for choosing the subsequent point $w^{2}$: $w^{2} = w^{i} - \gamma \nabla f(w^{j}; \cdot)$ for $i, j \in \{0, 1\}.$ This process continues indefinitely, and the number of possible choices, and hence methods, grows exponentially (see an example in Figure~\ref{fig:threeimages}).
\begin{figure}[t]
\centering
\hspace*{\fill}%
\begin{subfigure}[c]{0.12\textwidth}
\centering
\includegraphics[page=4,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}%
\hspace*{\fill}%
\begin{subfigure}[c]{0.12\textwidth}
\centering
\includegraphics[page=5,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}%
\hspace*{\fill}%
\begin{subfigure}[c]{0.16\textwidth}
\centering
\includegraphics[page=6,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}%
\hspace*{\fill}%
\begin{subfigure}[c]{0.16\textwidth}
\centering
\includegraphics[page=7,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}%
\hspace*{\fill}%
\begin{subfigure}[c]{0.05\textwidth}
\centering
\Large $\ldots$
\end{subfigure}%
\hspace*{\fill}%
\begin{subfigure}[c]{0.22\textwidth}
\centering
\includegraphics[page=8,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}%
\hspace*{\fill}
\caption{A possible computation tree $G$ for \algname{SGD} method after four steps and beyond.}
\label{fig:threeimages}
\end{figure}
Note that any instance of Algorithm~\ref{alg:framework}, after any steps, can be represented by a weighted directed tree $G = (V, E),$ called a \emph{computation tree}, where $V$ is the set of computed points and $E$ is the set of edges with weights given by the stochastic gradients used to compute the new points. Our main idea now is to take any computation tree $G$ and analyze its structure to provide convergence guarantees for the corresponding \algname{SGD} method. Intuitively, the structure of the tree, e.g., number of branches, length of branches, the tree distance between $w_{\textnormal{grad}}$ and $w_{\textnormal{base}}$ in Alg.~\ref{alg:framework} when we calculate a new point should be related to the convergence speed of the method.
\textbf{Example.} Consider \algname{Local SGD} from Figure~\ref{fig:local_sgd_graph}. There, we illustrate two global steps of the method with $2$ workers. In the first round, they compute $M_1 = 2$ and $M_2 = 2$ local steps (first figure in Figure~\ref{fig:local_sgd_graph}). During these steps, worker $1$ first calculates $\nabla f(x^0;\eta^{0,0}_{1}),$ finds $z^{0,1}_{1} = x^0 - \gamma \nabla f(x^0;\eta^{0,0}_{1}),$ then calculates $\nabla f(z^{0,1}_{1};\eta^{0,1}_{1})$ and $z^{0,2}_{1} = x^0 - \gamma \nabla f(z^{0,1}_{1};\eta^{0,1}_{1}).$ Similar steps are performed by worker $2.$ Then, via a parameter-server or \texttt{AllReduce}, \algname{LocalSGD} aggregates the stochastic gradients and performs the global step $x^0 - \gamma (\nabla f(x^0;\eta^{0,0}_{1}) + \nabla f(z^{0,1}_{1};\eta^{0,1}_{1}) + \nabla f(x^0;\eta^{0,0}_{2}) + \nabla f(z^{0,1}_{2};\eta^{0,1}_{2}))$ to obtain the new global point $x^4,$ from which the second global steps will start. The step of finding $x^4$ is equivalent to the steps $x^1 = x^0 - \gamma \nabla f(x^0;\eta^{0,0}_{1}),$ $x^2 = x^1 - \gamma \nabla f(z^{0,1}_{1};\eta^{0,1}_{1}),$ $x^3 = x^2 - \gamma \nabla f(z^{0,1}_{1};\eta^{0,1}_{1}),$ and $x^4 = x^3 - \gamma \nabla f(z^{0,1}_{2};\eta^{0,1}_{2}).$ This is how we construct the second figure in Figure~\ref{fig:local_sgd_graph}, which is a geometric representation of the first global step. Then, the workers compute $M_1 = 1$ and $M_2 = 3$ local steps, accordingly, and synchronize again (third figure in Figure~\ref{fig:local_sgd_graph}) to find $x^8,$ from which the third global steps will start.
% after which they synchronize (second figure). In the second round, they compute $M_1 = 1$ and $M_2 = 3$ local steps and synchronize again (third figure).
% \textbf{Previous frameworks.}
% Given an arbitrary computation tree $G$, our goal is to analyze the convergence speed of the corresponding \algname{SGD} method.
% \begin{wrapfigure}{r}{5.5cm}
% \caption{White Winter. Rooks’ nests. 1904. \emph{State Tretyakov Gallery} !!!Improve quality}
% \includegraphics[width=0.4\textwidth]{Grabar_White_winter_Rooks_nests_1904_gtg.jpg}
% \label{wrap-fig:1}
% \end{wrapfigure}
\subsection{Main theoretical result on convergence rates}
\label{sec:main}
Before we state our main theorem, we need to introduce sequences and definitions that characterize the structure of computation trees $G$.
% \subsubsection{Main branch and auxiliary sequence}
\begin{definition}[\emph{Main Branch} and \emph{Auxiliary Sequence}]
\label{def:branch}
For a given computation tree $G$, we call a sequence $\{x^k\}_{k \geq 0}$ a \emph{main branch} if it forms a path in $G$ starting at the initial node $w^0 \equiv x^0$. That is, for each $k \geq 0$, the node $x^{k+1}$ is a direct successor of $x^k$ in $G$. By the construction of tree $G,$ if $\{x^k\}_{k \geq 0}$ is a \emph{main branch}, then for each $k \geq 0$ there exists a unique pair $(z^k, \xi^k),$ where $z^k \in V$ and $\xi^k \sim \mathcal{D}_{\xi},$ such that $x^{k+1} = x^k - \gamma \nabla f(z^k; \xi^k).$ The sequence $\{(z^k, \xi^k)\}_{k \geq 0}$, which generates the main branch $\{x^k\}_{k \geq 0}$, is called an \emph{auxiliary sequence}.
\end{definition}
Although there may be several possible choices and any of them can be chosen in general, the selection of the \emph{main branch} is typically unique and straightforward in all reasonable \algname{SGD} methods, as it forms the backbone of the tree\footnote{A fitting analogy is the Git distributed version control system, which also has a central main branch.}.
% \begin{figure}[H]
% \centering
% \begin{minipage}{.6\textwidth}
% Let us consider an example. In Figure~\ref{fig:threeimages}, we can take a main branch $\{x^k\}_{k \geq 0}$ as follows:
% % $x^0 = w^0,$ $x^1 = w^2,$ $x^2 = w^3,$ $x^3 = w^8,$ $x^4 = w^9,$ $x^5 = w^{10}.$
% % \begin{align*}
% % &x^0 = w^0, x^1 = w^2, x^2 = w^3, \\
% % &x^3 = w^8, x^4 = w^9, x^5 = w^{10}.
% % \end{align*}
% $x^0 = w^0, x^1 = w^2, x^2 = w^3, x^3 = w^8, x^4 = w^9, x^5 = w^{10}.$
% Accordingly, the auxiliary sequence is given by $(z^0,\xi^0) = (w^0,\eta^0),$ $(z^1,\xi^1) = (w^1,\eta^1),$ $(z^2,\xi^2) = (w^2,\eta^2),$ $(z^3,\xi^3) = (w^4,\eta^6),$ $(z^4,\xi^4) = (w^1,\eta^3).$ See Figure~\ref{fig:main_branch}.
% \end{minipage}\hfill
% \begin{minipage}{.38\textwidth}
% \centering
% \includegraphics[page=9,width=0.7\linewidth]{./graphs/graphs_new.pdf}
% % \caption{The main branch $\{x^k\}_{k \geq 0}$ and the corresponding auxiliary sequence $\{(z^k, \xi^k)\}_{k \geq 0}$ for the computation tree $G$ in Figure~\ref{fig:threeimages}.}
% % \caption{Visualization of the main branch.}
% \caption{Visualization.}
% \label{fig:main_branch}
% \end{minipage}
% \end{figure}
% \begin{figure}[H]
% \centering
% \begin{minipage}{.6\textwidth}
% Let us consider an example. In Figure~\ref{fig:threeimages}, we can take a main branch $\{x^k\}_{k \geq 0}$ as follows:
% $x^0 = w^0, x^1 = w^2, x^2 = w^3, x^3 = w^8, x^4 = w^9, x^5 = w^{10}.$
% Accordingly, the auxiliary sequence is given by $(z^0,\xi^0) = (w^0,\eta^0),$ $(z^1,\xi^1) = (w^1,\eta^1),$ $(z^2,\xi^2) = (w^2,\eta^2),$ $(z^3,\xi^3) = (w^4,\eta^6),$ $(z^4,\xi^4) = (w^1,\eta^3).$ See Figure~\ref{fig:main_branch}.
% \end{minipage}\hfill
% \begin{minipage}{.38\textwidth}
% \centering
% \includegraphics[page=9,width=0.7\linewidth]{./graphs/graphs_new.pdf}
% \caption{Visualization.}
% \label{fig:main_branch}
% \end{minipage}
% \end{figure}
% In order to bound
% Recall that $x^{k+1} = x^k - \gamma \nabla f(z^k; \xi^k)$
\noindent
\begin{minipage}{.45\textwidth}
% Let us consider an example. In Figure~\ref{fig:main_branch}, we take a main branch $\{x^k\}_{k \geq 0}$ as:
% $x^0 = w^0,\ x^1 = w^2,\ x^2 = w^3,\ x^3 = w^8,\ x^4 = w^9,\ x^5 = w^{10}.$
% Accordingly, the auxiliary sequence is
% $(z^0,\xi^0) = (w^0,\eta^0),$
% $(z^1,\xi^1) = (w^1,\eta^1),$
% $(z^2,\xi^2) = (w^2,\eta^2),$
% $(z^3,\xi^3) = (w^4,\eta^6),$
% $(z^4,\xi^4) = (w^1,\eta^3).$
Let us consider an example. In Figure~\ref{fig:threeimages}, we can take a main branch $\{x^k\}_{k \geq 0}$ as follows:
$x^0 = w^0, x^1 = w^2, x^2 = w^3, x^3 = w^8, x^4 = w^9, x^5 = w^{10}.$ Accordingly, the auxiliary sequence is given by $(z^0,\xi^0) = (w^0,\eta^0),$ $(z^1,\xi^1) = (w^1,\eta^1),$ $(z^2,\xi^2) = (w^2,\eta^2),$ $(z^3,\xi^3) = (w^4,\eta^6),$ $(z^4,\xi^4) = (w^1,\eta^3).$ See Figure~\ref{fig:main_branch}.
\end{minipage}%
\hfill
\begin{minipage}{.55\textwidth}
\centering
\includegraphics[page=9,width=0.8\linewidth]{./graphs/graphs_new.pdf}
\captionof{figure}{Visualization.}
\label{fig:main_branch}
\end{minipage}
% Intuitively, the convergence rate should depend on a measure of the distance between $x^k$ and $z^k$. When these two points are close, for example, $x^k = z^k$, the stochastic gradient is computed almost at the update point, expecting a descent (on average). Conversely, if the points are far apart, the gradient computed at $z^k$ may not accurately represent the local curvature of $f$ at $x^k$, and the descent direction might become largely irrelevant. Therefore, it is essential to design a suitable distance metric that can be easily evaluated for any pair of points and is directly related to the convergence speed of the \algname{SGD} method. We propose to use the following distance:
Intuitively, the convergence rate should depend on the distance between $x^k$ and $z^k$. When these points are close (e.g., $x^k = z^k$), the stochastic gradient is computed near the update point, typically yielding descent on average. In contrast, if they are far apart, the gradient at $z^k$ may poorly approximate the local behavior of $f$ at $x^k$, making the update direction irrelevant. Thus, it is crucial to define a suitable distance metric that is both easy to evaluate for any point pair and directly related to the convergence speed of the \algname{SGD} method. We propose the following:
\begin{definition}
\label{def:dist}
% For all $y, z \in V,$ the distance $\textnormal{dist}(y, z)$ between $y$ and $z$ is the number of different stochastic gradients applied to $w^0$ to get $y$ and $z$.
For all $y, z \in V,$ the tree distance $\textnormal{dist}(y, z)$ between $y$ and $z$ is the maximum number of edges to the common closest ancestor of $y$ and $z$.
% More formally, for all $y, z \in V,$ there exist $w_{\textnormal{common}} \in V$ such that $y = w_{\textnormal{common}} - \gamma \sum_{i=1}^{p} \nabla f(w_i, \xi_i)$ and $z = w_{\textnormal{common}} - \gamma \sum_{i=1}^{q} \nabla f(\bar{w}_i, \bar{\xi}_i)$ for some\footnote{$w_1, \dots, w_p, \bar{w}_1, \dots, \bar{w}_q \in V,$ and $\xi_1, \dots, \xi_p, \bar{\xi}_1, \dots, \bar{\xi}_q \sim \mathcal{D}_{\xi}.$} $p, q \geq 0.$ Then $\textnormal{dist}(y, z) \eqdef \max\{p, q\}.$
\end{definition}
As an example, consider Figure~\ref{fig:main_branch}, where $\textnormal{dist}(w^9, w^4) = \max\{4, 2\} = 4,$ because the common ancestor is $w^0,$ the number of edges from $w^9$ to $w^0$ is $4,$ and the number of edges from $w^4$ to $w^0$ is $2.$ It is left to define the \emph{representation} of a point $y \in V.$
\begin{definition}
For all $y \in V,$ the representation $\textnormal{repr}(y)$ is the multiset of stochastic gradients applied to $w^0$ to get $y.$ In other words, there exist $\{(m^{1}, \kappa^{1}), \dots, (m^{p}, \kappa^{p})\} =: \textnormal{repr}(y)$ for some $p \geq 0$ such that $y = w^0 - \gamma \sum_{j=1}^{p} \nabla f(m^{j}, \kappa^{j}).$
\end{definition}
We define the representation of points to understand how all points are related. An important relation that we need is that $\textnormal{repr}(x) \subseteq \textnormal{repr}(y),$ which essentially means that all stochastic gradients used to compute $x$ are also used to compute $y.$ For instance, in Figure~\ref{fig:main_branch}, $\textnormal{repr}(w^9) = \{(w^0, \eta^0), (w^1, \eta^1), (w^2, \eta^2), (w^4, \eta^6)\}$ and $\textnormal{repr}(w^4) = \{(w^0, \eta^0), (w^2, \eta^2)\},$ which allows to track the path from from the starting point $w^0$ to $w^9$ and $w^4,$ and show that $\textnormal{repr}(w^4) \subseteq \textnormal{repr}(w^9).$
% Both $\textnormal{repr}(x)$ and $\textnormal{dist}(y, z)$ can be easily evaluated and maintained for any pair of points $x, y \in V.$
% \begin{theorem}
% Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold.
% We consider any \algname{SGD} method represented by the weighted directed acyclic tree $G = (V, E).$ Let $\{x^k\}$ be any \emph{main branch} of $G$ (see Def.~\ref{def:branch}) such that
% \begin{enumerate}
% \item there exists $R \in [0, \infty]$ such that $\textnormal{dist}(x^k, z^k) \leq R$ for all $k \geq 0,$
% \item and $\textnormal{repr}(z^k) \subseteq \textnormal{repr}(x^k).$
% \end{enumerate}
% Then $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\norm{\nabla f(x^k)}^2} \leq \varepsilon$ for all
% \begin{align*}
% K \geq \frac{R L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{\varepsilon^2}.
% \end{align*}
% \end{theorem}
% \subsection{}
% We are ready to present our main result.
\begin{theorembox}
\begin{restatable}[Main Theorem]{theorem}{MAINTHEOREM}
\label{thm:main}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider any \algname{SGD} method represented by \emph{computation tree} $G = (V, E)$. Let $\{x^k\}_{k \geq 0}$ be a \emph{main branch} of $G$ and $\{(z^k, \xi^k)\}_{k \geq 0}$ be the corresponding \emph{auxiliary sequence} (see Def.~\ref{def:branch}) that satisfy the following conditions: \\
\textnormal{\bf Condition 1:} For all $k \geq 0,$ $\xi^k$ is statistically independent of
$\{(x^{i+1}, z^{i+1}, \xi^i)\}_{i=0}^{k-1}.$ \\
% $(x^k, z^k, \xi^0, \dots, \xi^{k-1}).$ \\
\textnormal{\bf Condition 2:} The representation of $z^k$ is contained within that of $x^k$, i.e.,
$
\textstyle \textnormal{repr}(z^k) \subseteq \textnormal{repr}(x^k)
$
for all $k \geq 0.$ Equivalently, all stochastic gradients used in the computation of $z^k$ are also utilized in calculating $x^k$. \\
\textnormal{\bf Condition 3:} There exists a constant $R \in [0, \infty]$ such that
$
\textstyle \textnormal{dist}(x^k, z^k) \le R
$
for all $k \geq 0.$
% Equivalently, we can define $R \eqdef \sup_{k \geq 0} \textnormal{dist}(x^k, z^k).$ \\
Then $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
% \begin{align}
% \label{eq:CjSqSVYxtGcm}
$K \geq \frac{4 (R + 1) L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}$
% \end{align}
with step size $\gamma = \min\{\frac{1}{2 L}, \frac{1}{2 R L}, \frac{\varepsilon}{4 \sigma^2 L}\},$ where $\Delta = f(x^0) - f^*.$
\end{restatable}
\end{theorembox}
% \alexander{Add picture to visulize every condition?}
Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} are well-known and standard in the analysis of stochastic optimization methods \citep{lan2020first,arjevani2022lower}. Let us explain the conditions of the theorem.
% \textbf{Condition 1.} The first condition states that $\xi^k$ is independent of the pair $(x^k, z^k)$, which is a weak assumption. Recall that in \algname{Vanilla SGD}, where the updates follow $x^{k+1} = x^k - \gamma \nabla f(x^k;\xi^k)$ for all $k \geq 0$, it is typically assumed that $\xi^k$ is a newly generated random variable (data point) that is independent of all previous samples $\{\xi_i\}_{i < k}$. Our assumption is simply a natural extension of this condition to all other \algname{SGD} methods. This condition ensures that no stochastic gradient $\nabla f(\cdot;\xi^k)$ used in the calculations of $x^k$ and $z^k$ is reused, $x^k$ and $z^k$ do not depend on $\xi^k.$ Note that there will consider many \algname{SGD} methods where the same stochastic gradient is reused (for instance, \algname{Local SGD}), but the condition is still satisfied.
% \textbf{Condition 1.} The first condition states that $\xi^k$ is independent of $\{(x^{i+1}, z^{i+1}, \xi^i)\}_{i=0}^{k-1}$, which is a weak assumption. Recall that in \algname{Vanilla SGD}, where the updates follow $x^{k+1} = x^k - \gamma \nabla f(x^k;\xi^k)$ for all $k \geq 0$, it is typically assumed that $\xi^k$ is a newly generated random variable (data point) that is independent of all previous samples $\{\xi_i\}_{i < k}$. Our assumption is simply a natural extension of this condition to all other \algname{SGD} methods. This condition ensures that no stochastic gradient $\nabla f(\cdot;\xi^k)$ used in the calculations of $x^k$ and $z^k$ for all $k \geq 0.$ Note that we will see many \algname{SGD} methods where the same stochastic gradient is reused (e.g., \algname{Local SGD}), but the condition will still be satisfied.
\textbf{Condition 1.} The first condition condition requires that $\xi^k$ is independent of $\{(x^{i+1}, z^{i+1}, \xi^i)\}_{i=0}^{k-1},$ which is a weak assumption. In \algname{Vanilla SGD}, where $x^{k+1} = x^k - \gamma \nabla f(x^k;\xi^k)$, it is standard to assume that each $\xi^k$ is an independent sample. Our condition generalizes this to other \algname{SGD} variants. It guarantees that the stochastic gradient $\nabla f(\cdot;\xi^k)$ is not used in computing $x^k$ or $z^k$. Notably, this remains true even in methods like \algname{Local SGD}, where gradients may be reused.
\textbf{Condition 2.} The second condition is also weak in any \emph{reasonable and effective} \algname{SGD} method.
% To illustrates this, consider the figure bellow:
\begin{figure}[t]
\centering
\includegraphics[page=1,width=0.9\textwidth]{./graphs/graphs_new}
\caption{A general representation of the step $x^{k+1} = x^k - \gamma \nabla f(z^k; \xi^k)$ that shows how $x^k$ and $z^k$ are graph-geometrically related.}
\label{fig:fork}
\end{figure}
Figure~\ref{fig:fork} illustrates that there exists $p \geq 0$ such that \\
% \begin{align*}
% \textstyle z^{k} = x^0 - {\color{mydarkorange}\gamma \sum\limits_{i=0}^{p - 1} \nabla f(z^i; \xi^i)} - {\color{mydarkred} \gamma \sum\limits_{(w,\xi) \in S^k} \nabla f(w; \xi)},
% \end{align*}
$\textstyle z^{k} = x^0 - {\color{mydarkorange}\gamma \sum\limits_{i=0}^{p - 1} \nabla f(z^i; \xi^i)} - {\color{mydarkred} \gamma \sum\limits_{(w,\xi) \in S^k} \nabla f(w; \xi)},
% \begin{align*}
% \textstyle x^{k} = x^0 - {\color{mydarkorange}\gamma \sum\limits_{i=0}^{p - 1} \nabla f(z^i; \xi^i)} - {\color{mydarkblue} \gamma \sum\limits_{i=p}^{k-1} \nabla f(z^i; \xi^i)},
% \end{align*}
% $\textstyle
x^{k} = x^0 - {\color{mydarkorange}\gamma \sum\limits_{i=0}^{p - 1} \nabla f(z^i; \xi^i)} - {\color{mydarkblue} \gamma \sum\limits_{i=p}^{k-1} \nabla f(z^i; \xi^i)},$
where $S^k$ is the set of points and random variables used to compute $z^k$ starting from $x^{p}$.
Computing each stochastic gradient is time-consuming, so it is desirable to utilize as many computed gradients as possible, including $\{\nabla f(w; \xi)\}_{(w,\xi) \in S^k}$. Once $\nabla f(z^k; \xi^k)$ has been used to compute $x^{k+1}$, the first condition prevents further use of $\{\nabla f(w; \xi)\}_{(w, \xi) \in S^k}$ in subsequent iterations because $z^k$ depends on $\xi$ for all $(w, \xi) \in S^k.$ Thus, it is reasonable to assume that if an \algname{SGD} method employs the stochastic gradient $\nabla f(z^k; \xi^k)$ to compute $x^{k+1}$, then it has already used the gradients $\{\nabla f(w; \xi)\}_{(w, \xi) \in S^k}$ in previous iterations to fully leverage all available information. In other words, all stochastic gradients used in the computation of $z^k$ are also utilized in calculating $x^k$.
This is equivalent to the second condition $\textnormal{repr}(z^k) \subseteq \textnormal{repr}(x^k).$
\textbf{Condition 3.} This condition is arguably the most important in Theorem~\ref{thm:main} because it determines the \emph{iteration rate} of the main branch $\{x^k\}_{k \geq 0}$. In fact, \emph{iteration rate}
$\cO\left(\nicefrac{(R + 1) L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$
depends on $R \eqdef \sup_{k \geq 0} \textnormal{dist}(x^k, z^k).$
\algnamebf{Vanilla SGD} (Section~\ref{sec:vanilla_sgd}).
% \citep{robbins1951stochastic}.
For instance, consider the simplest method, the classical stochastic gradient descent (\algname{Vanilla SGD}) method: $w^{k+1} = w^{k} - \gamma \nabla f(w^k;\eta^k),$
where $w^0$ is a starting point and are $\{\eta^k\}$ are i.i.d. random variables. Taking $x^k = z^k = w^k$ and $\xi^k = \eta^k$ for all $k \geq 0.$ Clearly, all conditions of Theorem~\ref{thm:main} are satisfied: $\xi^k$ is independent of $\{(x^{i+1}, z^{i+1}, \xi^i)\}_{i=0}^{k-1},$ $\textnormal{repr}(x^k) = \textnormal{repr}(z^k)$ for all $k \geq 0,$ and $R = 0.$ We get the \emph{iteration rate} $\cO\left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right).$ The corresponding tree is in Figure~\ref{fig:sgd_graph}.
% For the most conservative approach---the \algname{Vanilla SGD} method---where $z^k = x^k$ (so that $R = 0$), the iteration rate is
% $\cO\left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right),$
% as expected \citep{lan2020first,arjevani2022lower}.
Conversely, if an \algname{SGD} method is overly non-conservative, leading to a large tree distance $R$ between $x^k$ and $z^k$, the \emph{iteration rate} correspondingly increases. The further the maximum tree distance $R$ between $x^k$ and $z^k,$ the more iterations are required to achieve the desired accuracy $\varepsilon.$
\textbf{Proof novelties.} In Section~\ref{sec:novel}, we outline the key novelties, challenges, and the intuition guiding our choice of conditions. Although our proof in Section~\ref{sec:proof} is compact—which we view as a strength rather than a limitation—it unifies a broad class of methods and provides new insights. Notably, right at the beginning, we introduce a distinct approach to handling the staleness term $\|x^k - z^k\|,$ which naturally arises from the update $x^{k+1} = x^k - \gamma \nabla f(z^k; \xi^k)$ in asynchronous and local methods. This treatment fundamentally differs from prior work, as it analyzes staleness through geometric graph reasoning. Moreover, using our framework, we later present our version of \algname{Local SGD}, which yields tighter guarantees compared to the classical \algname{Local SGD} (see Sections~\ref{sec:alg} and \ref{sec:compare}), further validating both our framework and proof technique.
% But we still have to bound the number of terms in $\bar{S}^k$
% Intuitively, in order to get a ``good'' bound to the norm, $x^k$ and $z^k$ should be relatively close.
% This is the place where we use Conditions 2 that gives the identity $x^k - z^k = - \gamma \sum_{j \in \bar{S}^k} \nabla f(z^i;\xi^i),$
% The natural and geometric intuition is important.
% \subsection{Discussion}
% The second condition is a natural requirement for the convergence of \algname{SGD} methods. The third condition is a new requirement that ensures the convergence of \algname{SGD} methods. It is satisfied by many \algname{SGD} methods, including \algname{Vanilla SGD}, \algname{Delayed SGD}, and \algname{Asynchronous SGD}. The theorem states that the convergence speed of \algname{SGD} methods depends on the distance between the computed points and the points at which the stochastic gradients are computed. The smaller the distance, the faster the convergence. The theorem provides a bound on the number of iterations required to achieve a given accuracy $\varepsilon.$
% Let us consider examples.
\section{Existing and New Algorithms: Summary and Comparison}
\label{sec:alg}
% In this section, we consider examples of stochastic optimization methods.
In this section, we consider examples of distributed methods.
% Some of the following methods are well-known, while others are new.
We will show that all of them can be represented by computation trees and analyzed using Theorem~\ref{thm:main}. The detailed analysis of each method is provided in Section~\ref{sec:algorithms}.
% Here, we offer a slightly less detailed explanation.
% \subsection{Linear graphs}
% \begin{figure}[t]
% \centering
% \begin{subfigure}[t]{\textwidth}
% \centering
% \begin{tikzpicture}[->,>=Stealth, auto, node distance=2.0cm]
% \node[draw, circle, minimum size=0.8cm] (x0) {$x^0$};
% \node[draw, circle, right=of x0] (x1) {$x^1$};
% \node[draw, circle, right=of x1] (x2) {$x^2$};
% \node[right=of x2] (dots1) {$\cdots$};
% \node[draw, circle, right=of dots1] (xk) {$x^{k+1}$};
% \draw (x0) edge node[above] {$\nabla f(x^0;\xi^0)$} (x1);
% \draw (x1) edge node[above] {$\nabla f(x^1;\xi^1)$} (x2);
% \draw (x2) edge node[above] {$\nabla f(x^2;\xi^2)$} (dots1);
% \draw (dots1) edge node[above] {$\nabla f(x^k;\xi^k)$} (xk);
% \end{tikzpicture}
% \caption{\algname{Vanilla SGD}}
% \label{fig:sgd_graph}
% \end{subfigure}
% \quad
% \begin{subfigure}[t]{\textwidth}
% \centering
% \begin{tikzpicture}[->,>=Stealth, auto, node distance=2.0cm]
% \node[draw, circle, minimum size=0.8cm] (x0) {$x^0$};
% \node[draw, circle, right=of x0] (x1) {$x^1$};
% \node[draw, circle, right=of x1] (x2) {$x^2$};
% \node[right=of x2] (dots1) {$\cdots$};
% \node[draw, circle, right=of dots1] (xk) {$x^{k+1}$};
% \draw (x0) edge node[above] {$\nabla f(x^{0 - \delta};\xi^0)$} (x1);
% \draw (x1) edge node[above] {$\nabla f(x^{1 - \delta};\xi^1)$} (x2);
% \draw (x2) edge node[above] {$\nabla f(x^{2 - \delta};\xi^2)$} (dots1);
% \draw (dots1) edge node[above] {$\nabla f(x^{k-\delta};\xi^k)$} (xk);
% \end{tikzpicture}
% \caption{\algname{Delayed SGD}}
% \label{fig:delayed_sgd_graph}
% \end{subfigure}
% \quad
% \begin{subfigure}[t]{\textwidth}
% \centering
% \begin{tikzpicture}[->,>=Stealth, auto, node distance=2.0cm]
% \node[draw, circle, minimum size=0.8cm] (x0) {$x^0$};
% \node[draw, circle, right=of x0] (x1) {$x^1$};
% \node[draw, circle, right=of x1] (x2) {$x^2$};
% \node[right=of x2] (dots1) {$\cdots$};
% \node[draw, circle, right=of dots1] (xk) {$x^{k+1}$};
% \draw (x0) edge node[above] {$\nabla f(x^{0 - \delta^0};\xi^0)$} (x1);
% \draw (x1) edge node[above] {$\nabla f(x^{1 - \delta^1};\xi^1)$} (x2);
% \draw (x2) edge node[above] {$\nabla f(x^{2 - \delta^2};\xi^2)$} (dots1);
% \draw (dots1) edge node[above] {$\nabla f(x^{k-\delta^k};\xi^k)$} (xk);
% \end{tikzpicture}
% \caption{\algname{Asynchronous SGD}}
% \label{fig:async_sgd_graph}
% \end{subfigure}
% \caption{Graph representations of different ``linear'' \algname{SGD} variants. (a) \algname{Vanilla SGD}; (b) \algname{Delayed SGD} with a fixed delay $\delta$; (c) \algname{Asynchronous SGD} with variable delays.}
% \label{fig:combined_sgd}
% \end{figure}
\algnamebf{Rennala SGD} (Section~\ref{sec:rennala_sgd}).
% The next method is \algname{Rennala SGD}, which can be written as
Consider \algname{Rennala SGD}, which can be written as
\begin{align}
\textstyle w^{k+1} = w^{k} - \gamma \sum\limits_{i=1}^{B} \nabla f(w^k;\eta^{k,i}),
\label{eq:BxYrFsyLoVzhayw}
\end{align}
% \footnote{Typically, the batch is also normalized by the batch size, which is ``hidden'' in the step size $\gamma$ in our case.}:
where $n$ workers collaboratively calculate the batch of size $B$ (see Alg.~\ref{alg:rennala}).
% (as it is done in \algname{Rennala SGD}).
This method produces a computation tree constructed as follows:
% \begin{alignat}{2}
% x^{1} &= x^{0} - \gamma \nabla f(x^0;\xi^{0}), \quad &\dots,\quad & x^{B} = x^{B-1} - \gamma \nabla f(x^0;\xi^{B-1}), \nonumber \\
% x^{B+1} &= x^{B} - \gamma \nabla f(x^{B};\xi^{B}), \quad &\dots,\quad & x^{2 B} = x^{2 B-1} - \gamma \nabla f(x^B;\xi^{2 B - 1}), \quad \dots, \label{eq:OPxkGkCos}
% \end{alignat}
% \begin{alignat}{2}
$x^{1} = x^{0} - \gamma \nabla f(x^0;\xi^{0}), \dots, x^{B} = x^{B-1} - \gamma \nabla f(x^0;\xi^{B-1}),
x^{B+1} = x^{B} - \gamma \nabla f(x^{B};\xi^{B}), \dots, x^{2 B} = x^{2 B-1} - \gamma \nabla f(x^B;\xi^{2 B - 1}), \dots, $
% \end{alignat}
where $B$ is a batch size (see Figure~\ref{fig:minibatch_sgd_graph}) and $\{\xi^k\}$ are i.i.d.~from $\mathcal{D}_{\xi}.$ Notice that the computation tree is equivalent to \eqref{eq:BxYrFsyLoVzhayw} because $x^{B} = w^1, x^{2B} = w^2,$ etc. Here, all conditions of Theorem~\ref{thm:main} are satisfied for the main branch $\{x^{k}\}$ with the auxiliary sequence $\{(z^k, \xi^k)\}$ such that $z^0 = \dots = z^{B - 1} = x^0,$ $z^B = \dots = z^{2 B - 1} = x^B,$ etc, and $\xi^0 = \eta^{0,0}, \dots, \xi^{B-1} = \eta^{0,B - 1}, \xi^{B} = \eta^{1,0}, $ etc. However, unlike \algname{Vanilla SGD}, $R = B - 1$ because $\textnormal{dist}(x^{0}, z^{0}) = 0, \textnormal{dist}(x^{1}, z^{1}) = 1, \dots, \textnormal{dist}(x^{B - 1}, z^{B - 1}) = B - 1, \textnormal{dist}(x^{B}, z^{B}) = 0,$ etc. Thus, the \emph{iteration rate} is $\cO\left(\nicefrac{B L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right).$
% Actually, it does not matter how the batch is computed—it may be done by one worker, by $B$ workers (as it is done in \algname{Synchronized SGD}), or by $n$ workers collaborating on a batch of size $B$ (as it is done in \algname{Rennala SGD}). This method produces a computation tree constructed as follows:
% \begin{alignat*}{2}
% x^{1} &= x^{0} - \gamma \nabla f(x^0;\xi^{0}), \quad &\dots,\quad & x^{B} = x^{B-1} - \gamma \nabla f(x^0;\xi^{B-1}), \\
% x^{B+1} &= x^{B} - \gamma \nabla f(x^{B};\xi^{B}), \quad &\dots,\quad & x^{2 B} = x^{2 B-1} - \gamma \nabla f(x^B;\xi^{2 B - 1}), \quad \dots,
% \end{alignat*}
% where $B$ is a batch size (see Figure~\ref{fig:minibatch_sgd_graph}). Here, all conditions of Theorem~\ref{thm:main} are satisfied for the main branch $\{x^{k}\}$ with the auxiliary sequence $\{(z^k, \xi^k)\}$ such that $z^0 = \dots = z^{B - 1} = x^0,$ $z^B = \dots = z^{2 B - 1} = x^B,$ etc. However, unlike \algname{Vanilla SGD}, $R = B - 1$ because $\textnormal{dist}(x^{0}, z^{0}) = 0, \textnormal{dist}(x^{1}, z^{1}) = 1, \dots, \textnormal{dist}(x^{B - 1}, z^{B - 1}) = B - 1, \textnormal{dist}(x^{B}, z^{B}) = 0,$ etc. Thus, the \emph{iteration rate} is $\cO\left(\nicefrac{B L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right).$
% \begin{figure}[H]
% \centering
% \includegraphics[page=11,width=0.6\textwidth]{./graphs/graphs_new.pdf}
% \caption{\algname{Rennala SGD}}
% \label{fig:minibatch_sgd_graph}
% \end{figure}
\begin{table}[t]
\caption{\textbf{Summary of distributed optimization methods from Sections~\ref{sec:alg} and \ref{sec:algorithms}.} In this table, we compare methods across different metrics. A {\color{mydarkgreen} \ding{51}} indicates a favorable property in the corresponding metric. As can be seen, each method has its own advantages and disadvantages. Therefore, for any practical setup, one should choose the most suitable method based on the specific requirements. For all methods, we use the parameters from the theorems of Section~\ref{sec:proof_compt} when deriving the metrics.}
\label{tab:methods_summary_check}
\centering
\scriptsize
\begin{adjustbox}{width=1.0\columnwidth,center}
\begin{threeparttable}
\begin{tabular}[t]{ccccccc}
\toprule
\textbf{Method} (Sec.~\ref{sec:algorithms}) & \makecell{\textbf{Optimal} \textbf{Computational} \\ \textbf{Complexity} (Sec.~\ref{sec:proof_compt})}
% & \makecell{\textbf{Communication}\textsuperscript{{\color{blue}(g)}} \\ \textbf{Complexity} (Sec.~\ref{sec:proof_communication})}
% & \makecell{\textbf{Communication}\textsuperscript{{\color{blue}(g)}} \\ \textbf{Complexity} (Sec.~\ref{sec:proof_communication})}
& \makecell{\textbf{Communication Com-}\textsuperscript{{\color{blue}(g)}} \\ \textbf{plexity with Equal Times}}
& \makecell{\textbf{Optimal} \textbf{Total} \\ \textbf{Complexity} (Sec.~\ref{sec:proof_communication_general})}
& \textbf{AllReduce}\textsuperscript{{\color{blue}(a)}}
& \makecell{\phantom{\textsuperscript{{\color{blue}(d)}}}\textbf{Update}\textsuperscript{{\color{blue}(d)}} \\ \textbf{Frequency}} & \makecell{\phantom{\textsuperscript{{\color{blue}(e)}}}\textbf{Peak}\textsuperscript{{\color{blue}(e)}} \\ \textbf{Bandwidth}} \\ \midrule
\makecell{\algname{Rennala SGD} (Alg.~\ref{alg:rennala}) \\ \citep{tyurin2023optimal}} &
{\color{mydarkgreen} \ding{51}}
& $\tau \frac{L \Delta}{\varepsilon}$ {\color{mydarkgreen} \ding{51}}
& \ding{55}
& {\color{mydarkgreen} \ding{51}}
& $\uparrow$
& $n$ \\
\midrule
\makecell{\algname{Ringmaster ASGD} (Alg.~\ref{alg:ringmasternew}) \\ \citep{maranjyan2025ringmaster}} &
{\color{mydarkgreen} \ding{51}}
& $\tau \left({\color{red}\frac{\sigma^2 L \Delta}{n \varepsilon^2}} \vee \frac{L \Delta}{\varepsilon}\right)$
& \ding{55}
& \ding{55}
& {\phantom{\ding{51}}} $\uparrow\uparrow\uparrow$ {\color{mydarkgreen} \ding{51}}
& $n$ \\
\midrule
\makecell{\algname{Local SGD} (Alg.~\ref{alg:local_sgd})
\textbf{(new)}\textsuperscript{{\color{blue}(f)}}} &
{\color{mydarkgreen} \ding{51}}
& $\tau \frac{L \Delta}{\varepsilon}$ {\color{mydarkgreen} \ding{51}}
& \ding{55}
& {\color{mydarkgreen} \ding{51}}
& $\uparrow\uparrow$
& $n$ \\
\midrule
\makecell{\algname{Cycle SGD} (Alg.~\ref{alg:circular_local_sgd}) \textbf{(new)}} &
\ding{55}
& $\tau \left({\color{red}\frac{\sigma^2 L \Delta}{n \varepsilon^2}} \vee \frac{L \Delta}{\varepsilon}\right)$
& \ding{55}
& ---
& $\uparrow\uparrow$
& \phantom{\color{mydarkgreen} \ding{51}} $\frac{n^2\varepsilon}{\sigma^2} \vee 1$ {\color{mydarkgreen} \ding{51}} \\
\midrule
\makecell{\algname{Async-Local/Batch SGD}\\ (Alg.~\ref{alg:async_plus_local} and Sec.~\ref{sec:async_batch}) \textbf{(new)}} &
{\color{mydarkgreen} \ding{51}}
& $\tau \frac{L \Delta}{\varepsilon}$ {\color{mydarkgreen} \ding{51}}
& ---
& \ding{55}
& $\uparrow\uparrow$
& $n$ \\
\midrule
\makecell{\algname{(Nested) Local-Async SGD} \\ (Alg.~\ref{alg:grouped_async_local_sgd} and \ref{alg:grouped_async_local_sgd_2}) \textbf{(new)}} &
{\color{mydarkgreen} \ding{51}}
& \phantom{\textsuperscript{{\color{blue}(c)}}}---\textsuperscript{{\color{blue}(c)}}
& \phantom{\textsuperscript{{\color{blue}(c)}}}---\textsuperscript{{\color{blue}(c)}}
& ---
& $\uparrow\uparrow$
& \phantom{\textsuperscript{{\color{blue}(c)}}}---\textsuperscript{{\color{blue}(c)}} \\
\midrule
\makecell{\algname{Dual-Process SGD} \\ (Alg.~\ref{alg:bi_local_sgd}) \textbf{(new)}} &
{\color{mydarkgreen} \ding{51}}
& $\tau \frac{L \Delta}{\varepsilon}$ {\color{mydarkgreen} \ding{51}}
& {\color{mydarkgreen} \ding{51}}
& \ding{55}
& $\uparrow\uparrow$
& $n$ \\
\midrule
\makecell{\algname{Meta Local SGD}\textsuperscript{{\color{blue}(b)}} \\ (Alg.~\ref{alg:adaptive_local_sgd}) \textbf{(new)}} &
---
& ---
& ---
& ---
& ---
& --- \\
\bottomrule
\end{tabular}
\begin{tablenotes}
\scriptsize
\item [{\color{blue}(a)}] Does a method support \texttt{AllReduce}? Asynchronous SGD-like methods do not support it due to their greedy update nature. \algname{Cycle SGD} synchronizes only a subset of workers; thus, we cannot say definitively. We also cannot say definitively in the case of \algname{Local-Async SGD} because the local asynchronous steps cannot be implemented with \texttt{AllReduce}, while the global steps can be.
\item [{\color{blue}(b)}] This is an abstract method where all metrics (Computational Complexity, Communication Complexity, etc) depend on the chosen strategy.
\item [{\color{blue}(c)}] Similar to \texttt{AllReduce}, here we also can not say definitely since \algname{Local-Async SGD} and \algname{Nested Local-Async SGD} are specially designed multi-cluster learning methods.
\item [{\color{blue}(d)}] This is a slightly less formal metric that indicates how often an algorithm updates its iterate/model. \algname{Rennala SGD} asks the workers to compute stochastic gradients at the same point; thus, it updates the iterates less frequently. In contrast, \algname{Ringmaster ASGD} updates the iterates immediately. All other methods fall somewhere in between. See the discussion in Section~\ref{sec:alg}.
\item [{\color{blue}(e)}] In the \algname{Rennala SGD}, \algname{Ringmaster ASGD}, \algname{Local SGD}, and \algname{Async-Local SGD} methods, all workers can start communication simultaneously; thus, their peak bandwidth is $\cO(n)$ when $n \leq \nicefrac{\sigma^2}{\varepsilon}$ In the \algname{Cycle SGD} method, the workers communicate in a circular manner, so the peak bandwidth is $\cO(\nicefrac{n^2\varepsilon}{\sigma^2})$ when $n \leq \nicefrac{\sigma^2}{\varepsilon}$, which is smaller.
\item [{\color{blue}(f)}] While we recognize that \algname{Local SGD} is well-known in the literature, what makes our version novel is the better time complexity compared to the classical version (Sec.~\ref{sec:compare}), the stopping criterion $\sum_{i=1}^{n} M_i = B$ in Alg.~\ref{alg:local_sgd}, and the analysis in Sec.~\ref{sec:local_sgd}, \ref{sec:proof_compt}, and \ref{sec:proof_communication}, which leads to the optimal computational time complexities with a proper choice of $B.$
\item [{\color{blue}(g)}] We report the terms w.r.t. communication time $\tau$ under the \emph{$(h,\tau)$-fixed computation model} from Section~\ref{sec:proof_communication}.
\end{tablenotes}
\end{threeparttable}
\end{adjustbox}
\end{table}
\algnamebf{Ringmaster ASGD} (Section~\ref{sec:ringmaster_asgd}).
This an \algname{Asynchronous SGD} method with the update rule
\begin{align}
\textstyle w^{k+1} = w^k - \gamma \nabla f(w^{k - \delta^k}; \eta^{k - \delta^k}_i),
\label{eq:async_sgd}
\end{align}
where $\delta^k$ is a delay such that $\delta^k \leq G - 1,$ where $G \geq 1$ is a hyperparameter (see Alg.~\ref{alg:ringmasternew}).
% \alexander{Draft}
We take $x^k = w^k$ for all $k \geq 0.$ Thus, the corresponding auxiliary sequence is defined by $z^k = x^{k - \delta^k} \equiv w^{k - \delta^k}$ and $\xi^k = \eta^{k - \delta^k}_i$ for all $k \geq 0.$ Constructing the computation tree (Figure~\ref{fig:async_sgd_graph}), we can show that the conditions of Theorem~\ref{thm:main} hold with $R = \max_{k \geq 0} \delta^k \leq G - 1$ and the \emph{iteration rate} is $\cO\left(\nicefrac{G L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right).$
% The main idea of \algname{Ringmaster ASGD} to get the optimal computational complexity was to ignore stochastic gradients computed at $w^{k - \delta^k}$ such that $\delta^k \geq B,$ where $B \geq 1$ is a hyperparameter. Thus, the iteration rate of \algname{Ringmaster ASGD} is $\cO\left(\nicefrac{B L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right).$
% \textbf{\algname{Delayed SGD}.} In \algname{Delayed SGD} the update uses a gradient computed at a stale point. For a fixed delay $h \geq 0$, the update rule is given by
% $$
% x^{k+1} = x^k - \gamma \nabla f(x^{k-\delta}; \xi^{k})
% $$
% with $x^{-1} = x^{-2} = \dots = x^{-\delta} = x^{0}.$ This means that the gradient used for updating $x^{k+1}$ is computed at $x^{k-\delta}$. Figure~\ref{fig:delayed_sgd_graph} illustrates this idea.
% \subsection{Nonlinear graphs}
% \subsection{Tree graphs}
% In many cases, the computation tree $G = (V, E)$ has a tree structure. We now start with the celebrated \algname{Local SGD} method.
% \begin{figure}[t]
% \centering
% \begin{subfigure}[t]{0.48\textwidth}
% \centering
% \includegraphics[page=10,width=0.8\textwidth]{./graphs/graphs_new.pdf}
% \caption{\algname{Vanilla SGD}}
% \label{fig:sgd_graph}
% \end{subfigure}
% \hfill
% \begin{subfigure}[t]{0.48\textwidth}
% \centering
% \includegraphics[page=12,width=0.9\textwidth]{./graphs/graphs_new.pdf}
% \caption{\algname{Asynchronous SGD}/\algname{Ringmaster ASGD}}
% \label{fig:async_sgd_graph}
% \end{subfigure}
% \vspace{0.3cm}
% \begin{subfigure}[t]{0.7\textwidth}
% \centering
% \includegraphics[page=11,width=0.8\textwidth]{./graphs/graphs_new.pdf}
% \caption{\algname{Rennala SGD}}
% \label{fig:minibatch_sgd_graph}
% \end{subfigure}
% % \caption{Computation trees for different \algname{SGD} methods.}
% \end{figure}
% \subsection{Comparison and new algorithms}
% \label{sec:comparison}
Previously, we presented \algname{Rennala SGD} and \algname{Ringmaster ASGD} that can be analyzed using Theorem~\ref{thm:main}. This raises the question: Which method is most effective, and how should one choose the appropriate one? In the following sections, we discuss different factors one should consider when selecting a method, and present new algorithms. The discussion here is summarized in Table~\ref{tab:methods_summary_check}. Before we begin, it is important to note that the iteration complexity in Theorem~\ref{thm:main} does not reflect the true wall-clock performance.
% Instead, it serves as an intermediate result used to derive the time complexities presented below.
It serves as an intermediate result used to derive the time complexities presented below.
% \subsection{Computation complexity}
% \label{sec:opt_comp}
% Note that the derived iterations complexities \emph{do not} reflect the wall-clock performance of the methods.
\textbf{1.~Computational complexity.} One way to compare the methods is to analyze their time complexity under the \emph{$h_i$-fixed computation model} (see Sec.~\ref{sec:related_work}, \ref{sec:related_work_fixed}, and \ref{sec:proof_compt}). With a proper choice of the corresponding parameters, i.e., $B = \max\{1, \lceil \nicefrac{\sigma^2}{\varepsilon}\rceil\}$, both \algname{Rennala SGD} and \algname{Ringmaster ASGD} are optimal in terms of wall-clock time with the time complexity
% \eqref{eq:orig_rennala},
$\Theta(\min_{m \in [n]} [(\nicefrac{1}{m} \sum_{i=1}^{m} \nicefrac{1}{h_i})^{-1} \left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{m \varepsilon^2}\right)])$
provided that communication times are negligible. In the worst-case scenario, on the ``very bad function'' \citep{arjevani2022lower}, all these methods perform equally well. Next, we discuss the strengths and weaknesses of the methods that are not captured by the $h_i$-fixed computation model.
% For simplicity, consider the \emph{$h_i$-fixed computation model}, then the listed method require
% \begin{align*}
% \textstyle \Theta\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_{i}}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right).
% \end{align*}
% seconds to find an $\varepsilon$--stationary point.
% One might ask: if communication times are negligible, which method is the fastest?
% However, in practice, performance can vary significantly depending on the task. Intuitively, \algname{Ringmaster ASGD} is the fastest because it performs the highest number of updates, as illustrated in Section~\ref{}. For more details, please refer to our experimental section, where compare the methods numerically.
\textbf{2.~Number of model updates.} At the same time, comparing \eqref{eq:async_sgd} and \eqref{eq:BxYrFsyLoVzhayw} reveals that \algname{Rennala SGD} computes $B$ stochastic gradients at the same point, while \algname{Ringmaster ASGD} both computes and “explores” more by immediately updating the model as in \eqref{eq:async_sgd}. This feature makes \algname{Ringmaster ASGD} more practically appealing. In fact, this intuition can be formalized using a simple two-dimensional strongly convex quadratic function $f : \mathbb{R}^2 \to \mathbb{R}$ such that $f(x,y) = \mu x^2 / 2 + L y^2 / 2$ for all $x,y \in \mathbb{R}.$ For this function, we prove that \algname{Rennala SGD} requires $\widetilde{\Theta}\left({\color{red}\nicefrac{\sigma^2}{\varepsilon n}} \times h \times \nicefrac{L}{\mu}\right)$
seconds to achieve an $\varepsilon$--solution under the $h_i$-fixed computation model with $h_i = h$ for all $i \in [n]$, while \algname{Ringmaster ASGD} needs $\widetilde{\Theta}\left(h \times \nicefrac{L}{\mu}\right),$ which is $\nicefrac{\sigma^2}{\varepsilon n}$ seconds less (see formal result in Section~\ref{sec:quadratic}). \textbf{This is the first result showing that \algname{Ringmaster ASGD}/\algname{Asynchronous SGD} can be \emph{strongly} better than \algname{Rennala SGD}.}
% \subsection{Communication complexity}
% % \label{sec:simple_computation_model}
% \textbf{3.~Communication complexity.} In real-world scenarios, communication delays are critical and can become a system bottleneck. In distributed optimization, our objective is to develop efficient methods that minimize both communication and synchronization overhead. In this context, \algname{Ringmaster ASGD} is the least efficient, as it requires the most communication and is not \texttt{AllReduce} friendly due to its ``very asynchronous'' nature (see Alg.~\ref{alg:ringmasternew}). Formally, consider a simple computation model, where it takes $\tau$ seconds to send one stochastic gradient to a parameter-server in the centralized setting\footnote{A similar issue arises in the decentralized setting, as it would require \algname{Ringmaster ASGD} to frequently send stochastic gradients to all workers.}, and assume that all workers perform identically, i.e., $h_i = h$ for all $i \in [n].$ In this case, the total time complexity of \algname{Ringmaster ASGD} is given by
% $\Omega\left(\tau \left(\nicefrac{L \Delta}{\varepsilon} + {\color{red}\nicefrac{\sigma^2 L \Delta}{n \varepsilon^2}}\right) + h \left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right).$
% The total time complexity of \algname{Rennala SGD} is
% $\cO\left(\tau \times \nicefrac{L \Delta}{\varepsilon} + h \left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right),$
% which is better when $\tau$ and $\nicefrac{\sigma^2}{\varepsilon}$
% are large (see Section~\ref{sec:proof_communication}). Moreover, it is optimal in the centralized setting \citep{tyurin2024shadowheart,tyurin2024optimalgraph}.
\textbf{3.~Communication complexity.} Communication delays are a major bottleneck in real-world distributed systems. Thus, minimizing communication and synchronization overhead is crucial. \algname{Ringmaster ASGD} is the least efficient in this regard, requiring frequent communication and lacking \texttt{AllReduce} support due to its asynchronous design. In a simple model where sending one stochastic gradient takes $\tau$ seconds and all workers have identical speed $h_i = h$, the time complexity of \algname{Ringmaster ASGD} is
$\Omega\left(\tau \left(\nicefrac{L \Delta}{\varepsilon} + {\color{red}\nicefrac{\sigma^2 L \Delta}{n \varepsilon^2}}\right) + h \left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right).$
In contrast, \algname{Rennala SGD} achieves
$\cO\left(\tau \nicefrac{L \Delta}{\varepsilon} + h \left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)$ (see Sec.~\ref{sec:proof_communication}),
which is better when $\tau$ and $\nicefrac{\sigma^2}{\varepsilon}$ are large.
% .
% and is also optimal in the centralized setting~\citep{tyurin2024shadowheart,tyurin2024optimalgraph}.
This is the point where we asked ourselves if it is possible to design a method that has the optimal computational time complexity of \algname{Rennala SGD} and updates the iterates more frequently than it. It turns out this method is well-known and is called \algname{Local SGD}:
\algnamebf{Local SGD} (Section~\ref{sec:local_sgd}).
% (new)\footnote{while the method is well-known, this is the first paper that shows the time complexity optimality of this method. IMPROVE???}. One possible way to represent \algname{Local SGD} is to use the following description: \\
% \begin{itemize}
\begin{figure}[t]
\centering
% --- First line: three subfigures ---
\begin{subfigure}[c]{0.2\textwidth}
\centering
\includegraphics[page=18,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}
\begin{subfigure}[c]{0.35\textwidth}
\centering
\Large $\overset{\shortstack{\small \text{Sync. via server}\\\text{\small or \texttt{AllReduce}}}}{\Rightarrow}$
\end{subfigure}%
\begin{subfigure}[c]{0.4\textwidth}
\centering
\includegraphics[page=19,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}
\\
\vspace{0.3cm}
% --- Second line: last two subfigures ---
\begin{subfigure}[c]{0.25\textwidth}
\centering
\Large $\overset{\shortstack{\small \text{After two global steps}}}{\Rightarrow}$
\end{subfigure}%
\begin{subfigure}[c]{0.75\textwidth}
\centering
\includegraphics[page=13,width=\textwidth]{./graphs/graphs_new.pdf}
\end{subfigure}
% \caption{An example of a \algname{Local SGD} computation tree with two workers, each performing $M_1 = M_2 = 2$ local steps over two global steps. Note that the maximum distances $\textnormal{dist}(x^3,z_1^1)$ and $\textnormal{dist}(x^7,\bar{z}_2^1)$, when applying $\nabla f(z_1^1;\xi^1_1)$ to $x^3$ and $\nabla f(\bar{z}_2^1;\bar{\xi}^1_2)$ to $x^7$, are equal to $M_1 + M_2 - 1 = 3$.}
% \label{fig:local_sgd_graph}
\caption{An evolution of a \algname{Local SGD} computation tree with $B = 4$ and 2 workers, each performing local steps over 2 global steps. In the first round, they compute $M_1 = 2$ and $M_2 = 2$ local steps (first figure), after which they synchronize (second figure). In the second round, they compute $M_1 = 1$ and $M_2 = 3$ local steps and synchronize again (third figure). Note that the maximum distances $\textnormal{dist}(x^3,z_1^{0,1})$ and $\textnormal{dist}(x^7,z_2^{1,2})$, when applying $\nabla f(z_1^{0,1};\eta^{0,1}_1)$ to $x^3$ and $\nabla f(z_2^{1,2};\eta_2^{1,2})$ to $x^7$, are equal to $B - 1 = \sum_{i=1}^{n} M_i - 1 = 3$. Notice that each stochastic gradient is used $2$ times in the tree.}
\label{fig:local_sgd_graph}
\end{figure}
We consider the classical \algname{Local SGD} strategy, where each worker $i \in [n]$ performs $M_i$ local steps, after which the server aggregates the results (see Alg.~\ref{alg:local_sgd}). Unlike most previous approaches, however, the number of local steps ${M_i}$ may vary across workers. Moreover, the server waits for a specific condition before aggregating: $\sum_{i=1}^{n} M_i = B.$ This strategy is adaptive to fluctuations in the number of local steps performed by individual workers, as the server ensures the total number of steps across all workers reaches the target sum $\sum_{i=1}^{n} M_i = B,$ where $B$ is a hyperparameter. An example of the computation tree is shown in Figure~\ref{fig:local_sgd_graph}. In Section~\ref{sec:local_sgd}, we establish the \emph{iteration rate} of \algname{Local SGD} as $\cO\left(\nicefrac{B L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$. This result follows directly from Theorem~\ref{thm:main} via a simple geometric argument. In fact, looking at Figure~\ref{fig:local_sgd_graph} reveals that all the conditions of Theorem~\ref{thm:main} are satisfied. The only minor difficulty is to show that $R \eqdef \sup_{k \geq 0} \textnormal{dist}(x^k, z^k) \leq B - 1,$ which is guaranteed by the condition $\sum_{i=1}^{n} M_i = B.$
% Condition 2 of Theorem~\ref{thm:main} is satisfied because the stochastic gradients used to compute $z^{j}_i$ are also used to compute $x^{k},$ where $\nabla f(z^{j}_i; \xi^{j}_i)$ is applied to compute $x^{k+1}.$ For any such $z^{j}_i$ and $x^{k},$ $\textnormal{dist}(x^{k}, z^{j}_i) \leq \sum_{i=1}^{n} M_i - 1$ (see Figure~\ref{fig:local_sgd_graph}). Thus,
% the \emph{iteration rate} is $\cO\left(\nicefrac{(\sum_{i=1}^{n} M_i) L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right).$
What is novel in our version of \algname{Local SGD} is that it achieves better theoretical guarantees within the family of \algname{Local SGD} approaches (see Section~\ref{sec:compare}). Moreover, our stopping condition and the choice of $B$ together ensure its optimality under the $h_i$-fixed computation model (see Theorem~\ref{thm:local_computation}).
% What is novel in our version of \algname{Local SGD} is the stopping condition and the choice of $B$, which together ensure the optimality of it under the $h_i$-fixed computation model (see Thm.~\ref{thm:local_computation}). Moreover, our analysis leads to better theoretical guarantees in the family of \algname{Local SGD} approaches (see Section~\ref{sec:compare}).
% Moreover, it has the optimal communication complexity in the centralized setting (see Thm.~\ref{thm:local_sgd_comnunication}).
% The method is described as follows:
% For each worker $i \in [n]$, run local steps
% $$
% z^{j+1}_i = z^{j}_i - \gamma \nabla f(z^{j}_i; \xi^{j}_i)
% $$
% for $j = 0, \dots, M_i - 1$, with $z^{0}_i = w^0$ for all $i \in [n]$.
% Then, send the calculated stochastic gradients to the server and update the global model:
% \begin{align*}
% &w^{1} = w^{0} - \gamma \sum_{i=1}^{n} \sum_{j=0}^{M_i - 1} \nabla f(z^{j}_i; \xi^{j}_i),
% \end{align*}
% Repeat the previous two steps starting from $w^1,$ $w^{2},$ etc.
% % \end{itemize}
% The corresponding computation tree of \algname{Local SGD} can be constructed as
% \begin{alignat*}{2}
% x^{1} &= x^{0} - \gamma \nabla f(z^{0}_1; \xi^{0}_1),\, &\dots,\, & x^{M_1}
% = x^{M_1 - 1} - \gamma \nabla f(z^{M_1 - 1}_1; \xi^{M_1 - 1}_1),\\
% x^{M_1+1} &= x^{M_1} - \gamma \nabla f(z^{0}_2; \xi^{0}_2),\, &\dots,\, & x^{M_1 + M_2}
% = x^{M_1 + M_2 - 1} - \gamma \nabla f(z^{M_2 - 1}_2; \xi^{M_2 - 1}_2),\dots
% \end{alignat*}
% as illustrated in Figure~\ref{fig:local_sgd_graph}.
\algnamebf{Async-Local SGD} (Section \ref{sec:async_plus_local}). Another idea to leverage the practical benefits of \algname{Ringmaster ASGD}, while at the same time reducing the communication overhead, is to use \algname{Ringmaster ASGD} with local steps. The idea is to run $M$ local steps on each worker instead of immediately sending the computed stochastic gradients to the server in an asynchronous fashion (See Figure~\ref{fig:async_plus_local}). We formalize this algorithm and prove the iteration rate in Section~\ref{sec:async_plus_local}. Moreover, in Sections~\ref{sec:proof_compt} and~\ref{sec:proof_communication}, we suggest an optimal choice of parameters that leads to optimal computational complexity and reduced communication complexity, which is better than that of \algname{Ringmaster ASGD}. We get as similar result with a new method, \algname{Async-Batch SGD} (Section~\ref{sec:async_batch}).
\algnamebf{Dual-Process SGD} (Section~\ref{sec:bi_local_sgd}). We took a step further and developed a new local method inspired by \algname{Local SGD} and \algname{Async-Local SGD}. It is the first local method to achieve the optimal time complexity in the distributed setting, where workers have varying computation and communication times (see Section~\ref{sec:proof_communication_general}). However, unlike \algname{Local SGD}, it is not \texttt{AllReduce}-friendly.
\textbf{4.~Peak bandwidth.} Another critical factor is the peak bandwidth. The number of workers the parameter-server or the \texttt{AllReduce} operation can synchronize may be limited when the number of workers $n$ is huge. Notice that the worst-case peak bandwidth of \algname{Rennala SGD}, \algname{Ringmaster ASGD}, \algname{Local SGD}, and \algname{Async-Local SGD} is $\Theta\left(n\right).$
% To mitigate this issue, we propose a new method called \algname{Cycle SGD}: \\
\algnamebf{Cycle SGD} (Section~\ref{sec:circular_sgd}). To mitigate this issue, we propose a new method called \algname{Cycle SGD}. Similar to \algname{Local SGD}, each worker performs local steps. However, once the workers finish computing the initial stochastic gradients $\{\nabla f(z^{0}_i; \eta^{0}_i)\}$, only the first group of $s$ workers sends their gradients to the server, where $s$ is a hyperparameter. The server then aggregates these gradients and performs the update $w^{1} = w^0 - \gamma \sum_{i=1}^{s} \nabla f(z^{0}_i; \eta^{0}_i).$
Meanwhile, the first $s$ workers begin computing their local steps starting from $w^{1}$, while the remaining workers continue their current local computations. Next, the second group of $s$ workers sends their locally computed vectors, and this process continues in a circular manner. A computation tree presented in Figure~\ref{fig:circular_local_sgd_graph}.
% In Section~\ref{sec:circular_sgd}, we provide a formal descriptions and, using Theorem~\ref{thm:main}, prove that \algname{Cycle SGD} converges after $\cO\left(\nicefrac{n^2 L \Delta}{s \varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$ iterations.
The peak bandwidth of \algname{Cycle SGD} is $\cO\left(s\right)$ with $s = \min\left\{\max\{\lceil\nicefrac{n^2 \varepsilon}{\sigma^2}\rceil, 1\}, n\right\},$ which is smaller than $\Theta\left(n\right)$ when $\nicefrac{\sigma^2}{\varepsilon} \geq n.$
% \textbf{5.~Optimization with clusters.} Now consider a scenario with two or more clusters of workers, where each cluster benefits from low-latency, high-bandwidth communication (e.g., InfiniBand) while inter-cluster communication (e.g., via Ethernet) is relatively slow.
% \algnamebf{Local-Async SGD} (Section~\ref{sec:local_plus_async}) In this setting, we run \algname{Asynchronous SGD} within each cluster and synchronize the clusters after a fixed number of local iterations. Running \algname{Asynchronous SGD} is feasible if the communication inside the clusters is fast. However, the communication between the clusters is slower, which is why we can run \algname{Asynchronous SGD} only within clusters.
% In Section~\ref{sec:local_plus_async}, we formalize this approach, called \algname{Local-Async SGD}, and prove its iteration rate. Then, in Section~\ref{sec:proof_compt}, we show that the method achieves optimal computational time complexity. The key novelty here is in how the clusters are synchronized, which is described in Algorithm~\ref{alg:grouped_async_local_sgd}.
% \algnamebf{Nested Local-Async SGD} (Section~\ref{sec:hierarchy}) Our framework is broadly applicable. It even supports a two-level hierarchy, where each cluster consists of servers equipped with (4–8) GPUs. Within each cluster, the GPUs run \algname{Asynchronous SGD}, the servers synchronize locally, and then the clusters synchronize with each other.
% Analyzing such a setup using classical optimization tools would be highly challenging. In contrast, our framework enables a straightforward analysis through geometric graph reasoning.
% \textbf{5.~Optimization with clusters.} Consider a scenario with two or more clusters of workers, where intra-cluster communication (e.g., InfiniBand) is fast, but inter-cluster communication (e.g., Ethernet) is slow.
\textbf{5.~Optimization with clusters.} Consider a setup with many clusters of workers, where intra-cluster communication (e.g., InfiniBand) is fast and inter-cluster communication (e.g., Ethernet) is slow.
\begin{figure}[t]
\centering
\includegraphics[page=15,width=0.8\textwidth]{./graphs/graphs_new.pdf}
\caption{An example of the computation tree for \algname{Async-Local SGD} with $M = 2$. In this example, the first worker is significantly faster: before the second worker completes its first set of local steps, $x^0 \rightarrow z^{1,1}_2 \rightarrow z^{1,2}_2$, the first worker already completes two rounds of local updates and sends the corresponding stochastic gradients, $(\nabla f(x^0;\eta^{1,0}_{1}), \nabla f(z^{1,1}_{1};\eta^{1,1}_{1}))$ and $(\nabla f(x^{2};\eta^{2,0}_{1}), \nabla f(z^{2,1}_{1};\eta^{2,1}_{1}))$.}
\label{fig:async_plus_local}
\end{figure}
\algnamebf{Local-Async SGD} (Section~\ref{sec:local_plus_async})~We run \algname{Asynchronous SGD} within each cluster and synchronize clusters after a fixed number of local steps. This setup is feasible due to fast intra-cluster links, while slower inter-cluster links necessitate infrequent synchronization. In Section~\ref{sec:local_plus_async}, we formalize this method, \algname{Local-Async SGD}, and establish its iteration rate. Section~\ref{sec:proof_compt} proves it achieves optimal computational time complexity. A key novelty lies in the synchronization mechanism (see Alg.~\ref{alg:grouped_async_local_sgd}).
\algnamebf{Nested Local-Async SGD} (Section~\ref{sec:hierarchy})~Our framework extends to a two-level hierarchy: within each cluster, servers with 4–8 GPUs run \algname{Asynchronous SGD} locally, synchronize at the server level, and then synchronize across clusters. Analyzing such a setup using classical optimization tools would be highly challenging. In contrast, our framework enables a straightforward analysis through geometric graph reasoning.
% $\bullet$ . In this method, we run \algname{Asynchronous SGD} steps within each cluster, and several steps the clusters synchronize. We derive an optimal criterion when they should synchronize to get both the optimal computation and communication time complexities. See Section~\ref{}.
% \textbf{6.~Flexible synchronization.}
% Either they synchronize at the same time, or set a predefined number of local steps.
\textbf{6.~Flexible synchronization and} \algnamebf{Meta Local SGD} (Section~\ref{sec:adaptive_local_sgd}). We noticed that in all previous methods, the workers are synchronized in a predefined manner or rule. We want to add more flexibility to the synchronization process. Our idea is that the server (or the workers themselves, in a decentralized setup) can select any subset of workers based on any strategy (e.g., randomly or according to current communication speeds), gather their computed stochastic gradients, update the global model, and ask these workers to continue performing local steps from the new point.
% The formal description is provided in Section~\ref{sec:adaptive_local_sgd}.
However, such ``anarchic synchronization'' can result in a computation tree with a large $R$ if the selected strategy is not chosen carefully. To ensure that $R$ is bounded, in our meta-algorithm (Algorithm~\ref{alg:adaptive_local_sgd}), we track the current distances $\{d_i\}$ to the head of the main branch and the local steps $\{M_i\}$ performed by each worker. Then, by tracking the value $d_i + \sum_{i=1}^{n} M_i$ for all $i \in [n]$ and comparing it to a parameter $B$, we compulsorily synchronize (Hard Sync) all workers for which $d_i + \sum_{i=1}^{n} M_i = B$.
% This way, we can ensure that the maximum distance $R$ is bounded by $B$, and the iteration rate of this method is $\mathcal{O}\left(\nicefrac{B L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$. See details in Section~\ref{sec:adaptive_local_sgd}.
This way, we can ensure that $R$ is bounded by $B$, and the iteration rate of this method is $\mathcal{O}\left(\nicefrac{B L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$.
% See details in Section~\ref{sec:adaptive_local_sgd}.
% \section{Insights, Guidelines, and Conclusion}
\section{Insights and Guidelines}
\label{sec:conlusion}
All proposed methods share the same iteration rate of $\mathcal{O}\left(\nicefrac{(R + 1) L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right),$ where $R$ is controlled by a method-specific hyperparameter and, at the same time, $R$ is the largest tree distance between $x^k$ and $z^k$. For \algname{Rennala SGD}, $R = B - 1,$ where $B$ denotes the batch size; for \algname{Ringmaster ASGD}, $R = B - 1,$ where $B$ is the delay threshold; for \algname{Local SGD}, $R = B - 1,$ where $B$ corresponds to the number of local steps; for \algname{Cycle SGD}, $R = \nicefrac{n^2}{s},$ where $s$ is the group size, etc. In all these methods, $R$ can be controlled, and to achieve the best possible computational and communication guarantees, one should always choose $R = \Theta\left(\nicefrac{\sigma^2}{\varepsilon}\right)$ (see Sections~\ref{sec:proof_communication} and~\ref{sec:proof_compt}). We believe this is a fundamental principle underlying all parallel optimization methods, and it should be considered a guiding rule when developing new algorithms. This choice is also theoretically justified: by taking $R = \Theta\left(\nicefrac{\sigma^2}{\varepsilon}\right)$, the iteration rate does not change asymptotically: $\mathcal{O}\left(\nicefrac{(R + 1) L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right) = \mathcal{O}\left(\nicefrac{L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right)$. Larger values of $R$ allow the methods to be more ``parallel-friendly''. For instance, a large $R$ enables \algname{Ringmaster ASGD} to consider stochastic gradients with larger delays, while a large $R$ in \algname{Local SGD} allows the method to run more local steps. However, taking $R > \nicefrac{\sigma^2}{\varepsilon}$ results in a worse iteration rate, suggesting that the corresponding method operates in an overly ``anarchic'' asynchronous regime, which may lead to performance degradation. Geometrically, the theory suggests that, to achieve good performance, the tree distance between $x^k$ and $z^k$ in Figure~\ref{fig:fork} should not exceed $\nicefrac{\sigma^2}{\varepsilon}$.
Notice that there is no single ``best'' method in Table~\ref{tab:methods_summary_check}, which we believe is another fundamental law. Each method has its own strengths and weaknesses, and one should develop or choose the most appropriate method for the specific task. This process becomes easier with the help of our new framework, \algname{Birch SGD}, and insights.
While our primary focus is on training large-scale language and vision models, where the i.i.d. assumption is usually appropriate, we acknowledge that non-i.i.d. scenarios are also important and should be investigated in future work, where it may be necessary to add additional assumptions such as first-order and second-order similarity of the functions \citep{arjevani2015communication,mishchenko2022asynchronous}. Moreover, it would be interesting to extend our framework to methods with preconditioning \citep{kingma2014adam} and non-Euclidean geometry \citep{jordan6muon}.
We hope our observations will support the future development and analysis of asynchronous optimization methods. Building on these insights, we designed at least eight new methods using our proposed \algname{Birch SGD} framework and the main result, Theorem~\ref{thm:main}. By reducing the analysis and design of these methods to computation trees, our entire development becomes purely graph-geometrical, offering a new and simpler view on asynchronous optimization methods.
\section{Summary of experimental results}
In Section~\ref{sec:exp}, we provide a detailed comparison of methods on logistic regression, image classification with ResNet18 \citep{he2016deep}, and next-token prediction with GPT2 \citep{radford2019language}. When communication times are negligible (Fig.~\ref{fig:compute_heterogeneous_regime_nw16_main}), as expected from Table~\ref{tab:methods_summary_check} and the previous discussion, \algname{Ringmaster ASGD} and \algname{Async-Local SGD} converge faster on the logistic regression problem. However, when communication times are large (Fig.~\ref{fig:bandwidth_bound_regime_nw16_main}), \algname{Ringmaster ASGD} becomes less practical due to its frequent updates. \algname{Synchronized SGD} exhibits the worst performance across all setups. \algname{Rennala SGD} and \algname{Local SGD} are more stable, while \algname{Async-Local SGD} performs well due to its effective balance between frequent updates and local steps.
\begin{minipage}{\textwidth}
\begin{figure}[H] % requires \usepackage{float} for [H]
\centering
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=0.9\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/3103_basic_algos_comp_1_10_comm_0__3103_subset_ring_reduce_comp_1_10_comm_0__1105_ringmaster_sgd_compcomm_comp_1_10_comm_0_top1_nw16.pdf}
\caption{Computation times $h_i = 1$ or $10$ randomly, communication times $\tau_i = 0.$}
\label{fig:compute_heterogeneous_regime_nw16_main}
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=0.9\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/3103_basic_algos_comp_10_comm_100__3103_subset_ring_reduce_comp_10_comm_100__1105_ringmaster_sgd_compcomm_comp_10_comm_100_top1_nw16.pdf}
\caption{Computation times $h_i = 10$, communication times $\tau_i = 100.$}
\label{fig:bandwidth_bound_regime_nw16_main}
\end{minipage}
\end{figure}
\end{minipage}
% \begin{figure}[h]
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=0.7\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/3103_basic_algos_comp_1_10_comm_0__3103_subset_ring_reduce_comp_1_10_comm_0__1105_ringmaster_sgd_compcomm_comp_1_10_comm_0_top1_nw16.pdf}
% \caption{$h_i = 1$ or $10$ randomly, $\tau_i = 0$ $\forall i \in [n]$}
% \label{fig:compute_heterogeneous_regime_nw16_main}
% \end{subfigure}
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=0.7\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/3103_basic_algos_comp_10_comm_100__3103_subset_ring_reduce_comp_10_comm_100__1105_ringmaster_sgd_compcomm_comp_10_comm_100_top1_nw16.pdf}
% \caption{$h_i = 10$, $\tau_i = 100$ $\forall i \in [n]$}
% \label{fig:bandwidth_bound_regime_nw16_main}
% \end{subfigure}
% \caption{Experiments with logistic regression.}
% \end{figure}
\section*{Acknowledgements}
The work was supported by the grant for research centers in the field of AI provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 000000C313925P4F0002 and the agreement №139-10-2025-033.
\bibliography{iclr2026_conference}
\bibliographystyle{iclr2026_conference}
\appendix
\newpage
\tableofcontents
\newpage
\section{Additional Discussion}
\label{sec:related_work_fixed}
\subsection{Discussion of the computational time complexities}
In this section, we extend our discussion about the computational time complexities of the methods discussed in the main part of the paper.
To compare parallel and asynchronous methods, \citet{mishchenko2022asynchronous} proposed using the \emph{$h_i$-fixed computation model}. The idea is to assume that worker $i$ requires at most $h_i$ seconds to calculate one stochastic gradient for all $i \in [n] \eqdef \{1, \dots, n\}$ (without loss of generality, $h_1 \leq h_2 \leq \dots \leq h_n$). The authors considered \algname{Synchronized SGD}, an iterative process defined as $w^{k+1} = w^{k} - \frac{\gamma}{n} \sum_{i=1}^n \nabla f(w^k; \eta^k_i),$
where each worker calculates one stochastic gradient, synchronize, and a parameter server aggregates them to update the iterate\footnote{Alternatively, there is no physical parameter server, and all workers perform an Allreduce.}. Using the $h_i$-fixed computation model, it can be easily shown that \algname{Synchronized SGD} converges after
\begin{align}
\cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\label{eq:nsqmTfjy}
\end{align}
seconds, because the method waits for the slowest worker, whose time is $\max_{i \in [n]} h_i = h_n$.
\begin{algorithm}[H]
\caption{\algname{Asynchronous SGD}}
\label{alg:asgd}
\begin{algorithmic}
\STATE \textbf{Input:} point $w^0 \in \R^d$, stepsizes $\gamma_k \geq 0$
\STATE Workers start computing stochastic gradients at $w^0$
\FOR{$k = 0, 1, \dots$}
\STATE Gradient $\nabla f(w^{k-\delta^k}; \eta^{k-\delta^k}_{i})$ arrives from worker $i$
\STATE Update: $w^{k+1} = w^{k} - \gamma_k \nabla f(w^{k-\delta^k}; \eta^{k-\delta^k}_{i})$
\STATE Worker $i$ begins calculating at $w^{k+1}$
\ENDFOR
\end{algorithmic}
\end{algorithm}
\citet{mishchenko2022asynchronous,koloskova2022sharper} provided new analyses of \algname{Asynchronous SGD} (see Algorithm~\ref{alg:asgd}) and \citet{cohen2021asynchronous} developed \algname{Picky SGD} to show that this time complexity can be improved to
$$\textstyle \cO\left(\left(\frac{1}{n} \sum \limits_{i=1}^n \frac{1}{h_i}\right)^{-1}\left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right),$$
where the dependence on $\{h_i\}$ is harmonic instead of being based on the maximum. It turns out that this complexity can be further improved\footnote{Note that $\min\limits_{m \in [n]} g(m) \leq g(n)$ for any function $g \,:\, \N \to \R$} to
\begin{align}
\label{eq:orig_rennala}
\textstyle \Theta\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right),
\end{align}
which is achieved by the \algname{Rennala SGD} method \citep{tyurin2023optimal}. Moreover, \citet{tyurin2023optimal} proved a matching lower bound demonstrating that both this complexity and \algname{Rennala SGD} are optimal. Recently, \citet{maranjyan2025ringmaster} developed a new optimal \algname{Ringmaster ASGD} method, which is essentially \algname{Asynchronous SGD} with a key modification. Additionally, under the \emph{universal computation model}, \citet{tyurin2024tight,maranjyan2025ringmaster} showed that both \algname{Rennala SGD} and \algname{Ringmaster ASGD} remain optimal even when computation times are arbitrary, time-varying, and random.
\subsection{More related work}
Our focus is on the homogeneous setting, where all workers have access to the same data distribution or dataset. The heterogeneous data setting is equally important, especially in federated learning (FL) \citep{konevcny2016federated} due to privacy constraints. In this context, many other methods have been proposed, including \algname{Asynchronous SGD} \citep{mishchenko2022asynchronous,koloskova2022sharper}, \algname{Asgrad} \citep{islamov2024asgrad}, \algname{PIAG} \citep{wu2022delay}, and \algname{Malenia SGD} \citep{tyurin2023optimal}. Notably, \citet{tyurin2023optimal,tyurin2024tight} showed that \algname{Malenia SGD} is optimal under both the fixed and universal computation models, without requiring assumptions of bounded gradients or gradients dissimilarity.
In the homogeneous setting, numerous other works have studied asynchronous \algname{SGD} methods, including \citep{lian2015asynchronous,feyzmahdavian2016asynchronous,stich2020error,sra2016adadelay}. However, these methods typically require the assumption that the delays in the indices of stochastic gradients are bounded (on average in \citep{sra2016adadelay}). As a result, their theoretical guarantees in terms of computational time complexity are weaker than those in \citep{cohen2021asynchronous,koloskova2022sharper,mishchenko2022asynchronous,tyurin2023optimal,maranjyan2025ringmaster}, which do not rely on such assumptions.
\subsection{Relation to other frameworks}
There were several previous approaches to unify \algname{SGD} methods. \citet{gorbunov2021local} proposed using a parametric assumption to unify the analysis of local methods, \citet{wang2020tackling} unified \algname{FedAvg}-like methods, \citet{wang2018cooperative} proposed \algname{Cooperative SGD} to analyze different synchronization mechanisms through mixing matrices, \citet{huang2022tackling} analyzed a general sample-wise Push–Pull framework in the heterogeneous setting using two-level augmented graphs, and \citet{khaled2020unified,gorbunov2020unified} analyzed \algname{SGD} methods with variance-reduction and compression techniques. These approaches are related to, but not directly comparable with ours, and, to the best of our knowledge, our approach is new and orthogonal.
Another interesting work that analyzes SGD methods is \citep{even2024asynchronous}. Their work and ours both use graphs; however, we use graphs in completely different, orthogonal, and unrelated contexts. In their case, nodes represent computers (GPUs, CPUs, servers), and edges represent communication links. In our case, nodes represent points of an algorithm, (directed) edges indicate how one point is calculated from another, and the graphs evolve with every iteration. Similarly, \citet{huang2022tackling} also use a graph abstraction but with a different meaning for nodes and edges: in their case, nodes correspond to devices and iterates of data samples, edges represent both communication and computation links, and the number of nodes is fixed from the beginning since every node corresponds to one sample or worker. These are different and orthogonal approaches. Another important difference is that they compare methods using iteration complexities, whereas we use time complexities in Table~\ref{tab:methods_summary_check}, which is a more robust and suitable metric for asynchronous and parallel methods.
\section{Notations}
$\N \eqdef \{1, 2, \dots\};$ $\norm{x}$ is the output of the standard Euclidean norm for all $x \in \R^d$; $\inp{x}{y} = \sum_{i=1}^{d} x_i y_i$ is the standard dot product; $g = \cO(f):$ exist $C > 0$ such that $g(z) \leq C \times f(z)$ for all $z \in \mathcal{Z};$ $g = \Omega(f):$ exist $C > 0$ such that $g(z) \geq C \times f(z)$ for all $z \in \mathcal{Z};$ $g = \Theta(f):$ $g = \cO(f)$ and $g = \Omega(f);$ $g = \widetilde{\Theta}(f):$ the same as $g = \Omega(f)$ but up to logarithmic factors; $a \vee b \eqdef \max\{a, b\}.$
\section{Experiments}
\label{sec:exp}
\subsection{Setup}
The experiments were prepared in Python. The distributed environment was simulated with the Simpy Python library \citep{matloff2008introduction}.
There are two hardware setups:
$\bullet$ CPU Setup: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz 52 cores (for logistic regression experiments)
$\bullet$ GPU Setup: 2 $\times$ Nvidia A100 80 Gb, CPU: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz 128 cores (for ResNet18 and GPT2 experiments)
% However, influence of hardware is only on speed of simulation, not on convergence speed.
% Time metric in all experimnets is virtual time from simulator.
The distributed environment is simulated with the help of Simpy. To compare the methods, we consider different computation and communication scenarios by taking different computation times $\{h_i\}$ and communication times $\{\tau_i\}$ of the workers.
For each task, we perform a grid search to identify the best parameters and report the top results across all runs of each algorithm.
The individual grid search parameters are drawn from a set of values specified in Section~\ref{sec:exp:parameters}. We plot the convergence rates against the elapsed time.
% Accross all tasks setups, we evaluate convergence speed of all algorithms on same 4 regimes:
We evaluate the convergence speeds of all algorithms in four regimes:
% \begin{itemize}
$\bullet$ \emph{Classical}: $h_i = 10$ and $\tau_i = 0$ for all $i \in [n].$ All workers have the same computation times, and the communication times are ignored.
$\bullet$ \emph{Slow Communications}: $h_i = 10$ and $\tau_i = 100$ for all $i \in [n].$ The communication takes time.
$\bullet$ \emph{Heterogeneous Computations}: $h_i = $ random\_choice($\{1,10\}$) and $\tau_i = 0$ for all $i \in [n].$ All workers have the different computation times randomly sampled from the set $\{1, 10\}.$
$\bullet$ \emph{Heterogeneous Communications}: $h_i = 10$ and $\tau_i = $ random\_choice([1,100]) for all $i \in [n].$ All workers have the different communication times randomly sampled from the set $\{1, 10\}.$
% \end{itemize}
This setup allows us to observe how different algorithms perform across various regimes and to compare their convergence behaviors under differing computational and communication conditions.
% Proposed algorithms have different share of aggregation (how they group several steps or workers in one communication) and asynchrony (how worker individual gradient update is depends on others) properties.
% Experiments examine influence of these properties on convergence speed.
% Symmetric regime - all workers are identical, communication is fast. Synchronous SGD performes best. Others either match by behaving almost like it (Ringmaster and Async-Local SGDs), either stalls some updates or introduce additional inneficiences.
% \emph{Slow Communications} regime. When communication dominates, aggregating steps together and communicating only across subgroups becomes benefitial, so Cycle SGD start performing best. Rennala, Local and Async-Local SGDs comes next, as they aggregate local steps together. Synchronized and Ringmaster algorithms fails, as they communicate too often.
% At the same time, difference is decreases, when number of workers increases.
% Compute and communicate heterogeneous regimes. Low performing algorithms tie together slow and fast workers, causing fast workers to waste time or cause slow weights updates. Because of that, pure async methods like Ringmaster and Async-Local SGD performs best. And synchronized is slowest, as all workers performes as slowest ones.
% \newpage
\subsection{Experiments with logistic regression}
We begin our experiments with one the simplest ML problems---logistic regression on the MNIST dataset~\cite{lecun2010mnist}. In this setting, we evaluate three different numbers of workers: $n \in \{16, 64, 256\}$. We use the standard linear model with the logistic loss.
Starting with $n = 16$ workers, we perform a grid search over the parameters specified in Table~\ref{tab:mnist_nw16_config} across all four regimes. The corresponding results are shown in Figure~\ref{fig:all_regimes_nw16}. In the \emph{classical} setup (Figure~\ref{fig:symmetric_regime_nw16}), all algorithms perform similarly. However, \algname{Rennala SGD} and \algname{Local SGD} underperform slightly due to the inability to interrupt an already initiated local step, resulting in occasional update losses. In the \emph{slow communications} setup (Figure~\ref{fig:bandwidth_bound_regime_nw16}), \algname{Rennala SGD}, \algname{Local SGD}, and \algname{Async-Local SGD} perform better, as they aggregate local steps and reduce communication overhead. In contrast, \algname{Synchronized SGD} and \algname{Ringmaster ASGD} perform poorly due to excessive communication. In both the \emph{heterogeneous computations} (Figure~\ref{fig:compute_heterogeneous_regime_nw16}) and \emph{heterogeneous communications} (Figure~\ref{fig:communication_heterogeneous_regime_nw16}) regimes, \algname{Async-Local SGD} and \algname{Ringmaster ASGD} achieve the fastest performance. \algname{Synchronized SGD}, as expected, is the slowest because it is not robust to heterogeneous computations and communications.
For $n = 64$ (grid search parameters in Table~\ref{tab:mnist_nw64_config}, results shown in Figure~\ref{fig:all_regimes_nw64}) and $n = 256$ (grid search parameters in Table~\ref{tab:mnist_nw256_config}, results shown in Figure~\ref{fig:all_regimes_nw256}), we observe behavior similar to the $n = 16$ case across all four regimes.
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/0805_basic_algos_comp_10_comm_0__1105_ringmaster_sgd_compcomm_comp_10_comm_0_top1_nw16.pdf}
\caption{\emph{Classical} regime: $h_i = 10$, $\tau_i = 0$}
\label{fig:symmetric_regime_nw16}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/3103_basic_algos_comp_10_comm_100__3103_subset_ring_reduce_comp_10_comm_100__1105_ringmaster_sgd_compcomm_comp_10_comm_100_top1_nw16.pdf}
\caption{\emph{Slow Communications} regime: $h_i = 10$, $\tau_i = 100$ }
\label{fig:bandwidth_bound_regime_nw16}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/3103_basic_algos_comp_1_10_comm_0__3103_subset_ring_reduce_comp_1_10_comm_0__1105_ringmaster_sgd_compcomm_comp_1_10_comm_0_top1_nw16.pdf}
\caption{\emph{Heterogeneous Computations} regime: $h_i = $ random\_choice($\{1,10\}$), $\tau_i = 0$}
\label{fig:compute_heterogeneous_regime_nw16}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw16/3103_basic_algos_comp_10_comm_1_100__3103_subset_ring_reduce_comp_10_comm_1_100__1105_ringmaster_sgd_compcomm_comp_10_comm_1_100_top1_nw16.pdf}
\caption{\emph{Heterogeneous Communications} regime: $h_i = 10$, $\tau_i = $ random\_choice([1,100])}
\label{fig:communication_heterogeneous_regime_nw16}
\end{subfigure}
\caption{Comparison of different optimization algorithms across various distributed computing regimes with $n = 16$. Each plot shows the convergence behavior in terms of loss versus simulated wall-clock time.}
\label{fig:all_regimes_nw16}
\end{figure}
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw64/0805_basic_algos_comp_1_10_comm_0__1105_ringmaster_sgd_compcomm_comp_1_10_comm_0_top1_nw64.pdf}
\caption{\emph{Classical} regime: $h_i = 10$, $\tau_i = 0$}
\label{fig:symmetric_regime_nw64}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw64/2703_basic_comp_10_comm_100__2703_subset_ring_reduce_comp_10_comm_100__1105_ringmaster_sgd_compcomm_comp_10_comm_100_top1_nw64.pdf}
\caption{\emph{Slow Communications} regime: $h_i = 10$, $\tau_i = 100$ }
\label{fig:bandwidth_bound_regime_nw64}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw64/0805_basic_algos_comp_1_10_comm_0__1105_ringmaster_sgd_compcomm_comp_1_10_comm_0_top1_nw64.pdf}
\caption{\emph{Heterogeneous Computations} regime: $h_i = $ random\_choice($\{1,10\}$), $\tau_i = 0$}
\label{fig:compute_heterogeneous_regime_nw64}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw64/0805_basic_algos_comp_10_comm_1_100__1105_ringmaster_sgd_compcomm_comp_10_comm_1_100_top1_nw64.pdf}
\caption{\emph{Heterogeneous Communications} regime: $h_i = 10$, $\tau_i = $ random\_choice([1,100])}
\label{fig:communication_heterogeneous_regime_nw64}
\end{subfigure}
\caption{Comparison of different optimization algorithms across various distributed computing regimes with $n = 64$. Each plot shows the convergence behavior in terms of loss versus simulated wall-clock time.}
\label{fig:all_regimes_nw64}
\end{figure}
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw256/3003_basic_algos_comp_10_comm_0__3003_subset_ring_reduce_algos_comp_10_comm_0__1105_ringmaster_sgd_compcomm_comp_10_comm_0_top1_nw256.pdf}
\caption{\emph{Classical} regime: $h_i = 10$, $\tau_i = 0$}
\label{fig:symmetric_regime_nw256}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw256/3003_basic_algos_comp_10_comm_100__3003_subset_ring_reduce_algos_comp_10_comm_100__1105_ringmaster_sgd_compcomm_comp_10_comm_100_top1_nw256.pdf}
\caption{\emph{Slow Communications} regime: $h_i = 10$, $\tau_i = 100$ }
\label{fig:bandwidth_bound_regime_nw256}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw256/3003_basic_algos_comp_1_10_comm_0__3003_subset_ring_reduce_algos_comp_1_10_comm_0__1105_ringmaster_sgd_compcomm_comp_1_10_comm_0_top1_nw256.pdf}
\caption{\emph{Heterogeneous Computations} regime: $h_i = $ random\_choice($\{1,10\}$), $\tau_i = 0$}
\label{fig:compute_heterogeneous_regime_nw256}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_mnist_logreg_nw256/3003_basic_algos_comp_10_comm_1_100__3003_subset_ring_reduce_algos_comp_10_comm_1_100__1105_ringmaster_sgd_compcomm_comp_10_comm_1_100_top1_nw256.pdf}
\caption{\emph{Heterogeneous Communications} regime: $h_i = 10$, $\tau_i = $ random\_choice([1,100])}
\label{fig:communication_heterogeneous_regime_nw256}
\end{subfigure}
\caption{Comparison of different optimization algorithms across various distributed computing regimes with $n = 256$. Each plot shows the convergence behavior in terms of loss versus simulated wall-clock time.}
\label{fig:all_regimes_nw256}
\end{figure}
% However, in the \emph{slow communications} regime, the differences between algorithms become less pronounced.
% This is due to the increased number of workers, which enables larger step sizes and mitigates the impact of communication delays.
\clearpage
\newpage
\subsection{Experiments with ResNet18 and image classification}
We test the algorithms on the CIFAR10 \citep{krizhevsky2009learning} image recognition task with the ResNet18 \citep{he2016deep} deep neural network. For ResNet18, we similarly report the best convergence results from a grid search over the parameters listed in Table~\ref{tab:resnet18_nw8_config}, using a setup with $n = 8$ workers. Results for all algorithms across the four regimes are presented in Figure~\ref{fig:resnet18_all_regimes_nw8}.
The conclusions largely mirror those from the logistic regression experiments, with a few additional observations. In the \emph{classical} setup (Figure~\ref{fig:resnet18_symmetric_regime_nw8}), \algname{Ringmaster ASGD} outperforms all other methods. We believe that this is due to the frequent updates of the method. In the \emph{slow communications} regime (Figure~\ref{fig:resnet18_bandwidth_bound_regime_nw8}), the trends are consistent with those observed in the MNIST experiments: \algname{Ringmaster ASGD} becomes slower, while methods that are less communication-intensive achieve better performance. In the \emph{heterogeneous communications} (Figure~\ref{fig:resnet18_communication_heterogeneous_regime_nw8}) regime, \algname{Async-Local SGD} has the best performance due to the good balance of frequent model updates and local steps.
% and \emph{heterogeneous communications} (Figure~\ref{fig:resnet18_communication_heterogeneous_regime_nw8})
% regimes, the trends are similar to those observed with the MNIST experiments, although the convergence curves are even closer across algorithms.
\begin{figure}[h]
\centering
\hspace*{\fill}%
\begin{subfigure}[c]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_resnet18_nw8/1204_resnet18_basic_comp_10_comm_0__0305_ringmaster_compcomm_comp_10_comm_0_top1_nw8.pdf}
\caption{\emph{Classical} regime: $h_i = 10$, $\tau_i = 0$}
\label{fig:resnet18_symmetric_regime_nw8}
\end{subfigure}%
\hfill
\begin{subfigure}[c]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_resnet18_nw8/1204_resnet18_basic_comp_10_comm_100__0305_ringmaster_compcomm_comp_10_comm_100_top1_nw8.pdf}
\caption{\emph{Slow Communications} regime: $h_i = 10$, $\tau_i = 100$}
\label{fig:resnet18_bandwidth_bound_regime_nw8}
\end{subfigure}%
\hspace{0.5cm}
\begin{subfigure}[c]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_resnet18_nw8/1204_resnet18_basic_comp_1_10_comm_0__0205_resnet18_local_sgd_comp_1_10_comm_0__0305_ringmaster_compcomm_comp_1_10_comm_0_top1_nw8.pdf}
\caption{\emph{Heterogeneous Computations} regime: $h_i = $ random\_choice($\{1,10\}$), $\tau_i = 0$}
\label{fig:resnet18_compute_heterogeneous_regime_nw8}
\end{subfigure}
\hfill
\begin{subfigure}[c]{0.43\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_resnet18_nw8/1204_resnet18_basic_comp_10_comm_1_100__0305_ringmaster_compcomm_comp_10_comm_1_100_top1_nw8.pdf}
\caption{\emph{Heterogeneous Communications} regime: $h_i = 10$, $\tau_i = $ random\_choice($\{1,100\}$)}
\label{fig:resnet18_communication_heterogeneous_regime_nw8}
\end{subfigure}%
\caption{ResNet18 experiments with $n = 8$}
\label{fig:resnet18_all_regimes_nw8}
\end{figure}
\newpage
\subsection{Experiments with GPT2 and token prediction}
We also evaluate the algorithms on the Wikitext-2 \citep{merity2016pointer} next token prediction task with GPT2 \citep{radford2019language}. For GPT2, we evaluate all four regimes using a setup with $n = 8$ workers. To achieve faster and more robust convergence, we use the \algname{AdamW} normalization strategy\footnote{Instead of the \algname{SGD} step $w^{k+1} = w^{k} - \gamma g^k,$ where $g^k$ is a descent direction, we use the AdamW strategy with $g^k$.} only in these experiments with GPT2. The resulting convergence curves are shown in Figure~\ref{fig:gpt2_all_regimes_nw8}. Once again, the results are similar to those of the previous experiments. Due to hardware limitations, a narrower grid search range is used; therefore, it is possible that convergence could be further improved with a more extensive search.
\begin{figure}[H]
\centering
\hspace*{\fill}%
\begin{subfigure}[c]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_gpt2_nw8/1804_gpt2_basic_comp_10_comm_0__0705_gpt2_ringmaster_compcomm_10_comm_0__0805_gpt2_ringmaster_compcomm_10_comm_0_top1_nw8.pdf}
\caption{\emph{Classical} regime: $h_i = 10$, $\tau_i = 0$}
\label{fig:gpt2_symmetric_regime_nw8}
\end{subfigure}%
\hfill
\begin{subfigure}[c]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_gpt2_nw8/1804_gpt2_basic_comp_10_comm_100__0705_gpt2_ringmaster_compcomm_10_comm_100__0805_gpt2_ringmaster_compcomm_10_comm_100_top1_nw8.pdf}
\caption{\emph{Slow Communications} regime: $h_i = 10$, $\tau_i = 100$}
\label{fig:gpt2_bandwidth_bound_regime_nw8}
\end{subfigure}%
\begin{subfigure}[c]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_gpt2_nw8/1804_gpt2_basic_comp_1_10_comm_0__0705_gpt2_ringmaster_compcomm_1_10_comm_0_top1_nw8.pdf}
\caption{\emph{Heterogeneous Computations} regime: $h_i = $ random\_choice($\{1,10\}$), $\tau_i = 0$}
\label{fig:gpt2_compute_heterogeneous_regime_nw8}
\end{subfigure}
\hfill
\begin{subfigure}[c]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{paper_pdfs/draft_gpt2_nw8/1804_gpt2_basic_comp_10_comm_1_100__0705_gpt2_ringmaster_compcomm_10_comm_1_100__0805_gpt2_ringmaster_compcomm_10_comm_1_100_top1_nw8.pdf}
\caption{\emph{Heterogeneous Communications} regime: $h_i = 10$, $\tau_i = $ random\_choice($\{1,100\}$)}
\label{fig:gpt2_communication_heterogeneous_regime_nw8}
\end{subfigure}%%
\caption{GPT-2 experiments with $n = 8$}
\label{fig:gpt2_all_regimes_nw8}
\end{figure}
\subsection{Experiments with \algname{Cycle SGD} and peak bandwidth}
\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{paper_pdfs/cycle_vs_others_v2.pdf}
\caption{Comparison of methods in the setup, where the communication time depends on the number of synchronized workers. It takes $h_i = 10$ seconds to compute a stochastic gradient, and the communication time is $1.5 \times k$ seconds, where $k$ is the number of synchronized workers.}
\label{fig:cycle}
\end{figure}
In Figure~\ref{fig:cycle}, we present the comparison of \algname{Cycle SGD} with other methods to verify the advantageous theoretical property of \algname{Cycle SGD}. For this particular regime, when the computational times are the same, all methods except \algname{Cycle SGD} require all the $64$ workers to synchronize at the same time, leading to high peak bandwidth. At the same time, \algname{Cycle SGD} synchronizes only $8$ workers and reduces the communication time and the total time complexity, which we observe in Figure~\ref{fig:cycle} with logistic regression on the MNIST dataset.
\subsection{Sensitivity to the choice of $B$ in \algname{Rennala SGD}}
\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{paper_pdfs/rennala_comparison_v2.pdf}
\caption{\emph{Slow Communications} regime: $h_i = 10$, $\tau_i = 100$}
\label{fig:slowdonw}
\end{figure}
In this section, we aim to understand the sensitivity of the choice of the value $B$ in \algname{Rennala SGD}. From the proof of Theorem~\ref{thm:rennala}, we know that $R \eqdef \sup_{k \geq 0} \textnormal{dist}(x^k, z^k) = \Theta(B).$
By increasing $B = \Theta(R),$ we can examine whether the method starts to slow down. This slowdown is expected due to the discussion in Section~\ref{sec:conlusion}. In Figure~\ref{fig:slowdonw}, we present our results on logistic regression problem on the MNIST dataset and $n = 100$ workers, where we see that if we take $B$ too large, the method slows down accordingly.
\clearpage
\newpage
\subsection{Parameters of the experiments}
\label{sec:exp:parameters}
\begin{table}[!htbp]
\caption{Experimental configuration for logistic regression on MNIST with $n = 16$ workers.}
\label{tab:mnist_nw16_config}
\centering
\begin{tabular}{ll}
\toprule
\textbf{Parameter} & \textbf{Value} \\
\midrule
% Simulation time & 20000 \\
Batch size & 1 \\
Optimizer & SGD \\
Number of workers & 16 \\
\midrule
\multicolumn{2}{l}{\textbf{Algorithm-specific configurations:}} \\
\midrule
\textbf{Synchronized SGD} & $\gamma$ range: $[2^{-5}, 2^4]$ \\
\midrule
\textbf{Rennala SGD} & $\gamma$ range: $[2^{-15}, 2^{-3}]$ \\
& B set: \{128, 256, 512, 1024\} \\
\midrule
\textbf{Local SGD} & $\gamma$ range: $[2^{-15}, 2^{-3}]$ \\
& B set: \{128, 256, 512, 1024\} \\
\midrule
\textbf{Ringmaster ASGD} & $\gamma$ range: $[2^{-15}, 2^1]$ \\
& B set: \{128, 256, 512, 1024\} \\
\midrule
\textbf{Async-Local SGD} & $\gamma$ range: $[2^{-10}, 2^{1}]$ \\
& B set: \{64, 128, 256, 512, 1024\} \\
& M set: \{1, 2, 4, 8\} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!htbp]
\caption{Experimental configuration for logistic regression on MNIST with $n = 64$ workers.}
\label{tab:mnist_nw64_config}
\centering
\begin{tabular}{ll}
\toprule
\textbf{Parameter} & \textbf{Value} \\
\midrule
% Simulation time & 20000 \\
Batch size & 1 \\
Optimizer & SGD \\
Number of workers & 64 \\
\midrule
\multicolumn{2}{l}{\textbf{Algorithm-specific configurations:}} \\
\midrule
\textbf{Synchronized SGD} & $\gamma$ range: $[2^{-5}, 2^4]$ \\
\midrule
\textbf{Rennala SGD} & $\gamma$ range: $[2^{-15}, 2^{-3}]$ \\
& B set: \{1024, 2048, 4096\} \\
\midrule
\textbf{Local SGD} & $\gamma$ range: $[2^{-15}, 2^{-3}]$ \\
& B set: \{1024, 2048, 4096\} \\
\midrule
\textbf{Ringmaster ASGD} & $\gamma$ range: $[2^{-15}, 2^1]$ \\
& B set: \{512, 1024, 2048, 4096\} \\
\midrule
\textbf{Async-Local SGD} & $\gamma$ range: $[2^{-10}, 2^{1}]$ \\
& B set: \{64, 128, 256, 512, 1024, 4096\} \\
& M set: \{1, 2, 4, 8\} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!htbp]
\caption{Experimental configuration for logistic regression on MNIST with $n = 256$ workers.}
\label{tab:mnist_nw256_config}
\centering
\begin{tabular}{ll}
\toprule
\textbf{Parameter} & \textbf{Value} \\
\midrule
% Simulation time & 20000 \\
Batch size & 1 \\
Optimizer & SGD \\
Number of workers & 256 \\
\midrule
\multicolumn{2}{l}{\textbf{Algorithm-specific configurations:}} \\
\midrule
\textbf{Synchronized SGD} & $\gamma$ range: $[2^{-5}, 2^4]$ \\
\midrule
\textbf{Rennala SGD} & $\gamma$ range: $[2^{-15}, 2^{-3}]$ \\
& B set: \{1024, 2048, 4096, 8192, 16384\} \\
\midrule
\textbf{Local SGD} & $\gamma$ range: $[2^{-15}, 2^{-3}]$ \\
& B set: \{1024, 2048, 4096, 8192, 16384\} \\
\midrule
\textbf{Ringmaster ASGD} & $\gamma$ range: $[2^{-15}, 2^1]$ \\
& B set: \{512, 1024, 2048, 4096, 8192\} \\
\midrule
\textbf{Async-Local SGD} & $\gamma$ range: $[2^{-10}, 2^{1}]$ \\
& B set: \{128, 256, 512, 1024, 4096, 8192\} \\
& M set: \{1, 2, 4, 8\} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!htbp]
\begin{minipage}[t]{0.48\textwidth}
\caption{Experimental configuration for ResNet18 with $n = 8$ workers.}
\label{tab:resnet18_nw8_config}
\centering
\begin{tabular}{ll}
\toprule
\textbf{Parameter} & \textbf{Value} \\
\midrule
% Simulation time & 10000 \\
Batch size & 16 \\
Optimizer & SGD \\
Number of workers & 8 \\
\midrule
\multicolumn{2}{l}{\textbf{Algorithm-specific configurations:}} \\
\midrule
\textbf{Synchronized SGD} & $\gamma$ range: $[2^{-6}, 2^{-3}]$ \\
\midrule
\textbf{Rennala SGD} & $\gamma$ range: $[2^{-6}, 2^{-3}]$ \\
& B set: \{16, 32, 64\} \\
\midrule
\textbf{Local SGD} & $\gamma$ range: $[2^{-6}, 2^{-3}]$ \\
& B set: \{16, 32, 64\} \\
\midrule
\textbf{Ringmaster ASGD} & $\gamma$ range: $[2^{-6}, 2^{-3}]$ \\
& B set: \{16, 32, 64\} \\
\midrule
\textbf{Async-Local SGD} & $\gamma$ range: $[2^{-6}, 2^{-3}]$ \\
& B set: \{16, 32, 64\} \\
& M set: \{2, 4, 8\} \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\caption{Experimental configuration for GPT-2 with $n = 8$ workers.}
\label{tab:gpt2_nw8_config}
\centering
\begin{tabular}{ll}
\toprule
\textbf{Parameter} & \textbf{Value} \\
\midrule
% Simulation time & 30000, 100000 - depends on setup. See Figure~\ref{fig:gpt2_all_regimes_nw8} \\
Batch size & 32 \\
Optimizer & AdamW \\
Number of workers & 8 \\
\midrule
\multicolumn{2}{l}{\textbf{Algorithm-specific configurations:}} \\
\midrule
\textbf{Synchronized SGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
\midrule
\textbf{Rennala SGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
& B set: \{16, 32, 64\} \\
\midrule
\textbf{Local SGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
& B set: \{16, 32, 64\} \\
\midrule
\textbf{Ringmaster ASGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
& B set: \{16, 32, 64\} \\
\midrule
\textbf{Async-Local SGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
& B set: \{16, 32, 64\} \\
& M set: \{2, 4\} \\
\bottomrule
\end{tabular}
\end{minipage}
\end{table}
% \begin{table}[!htbp]
% \caption{Experimental configuration for GPT-2 with $n = 8$ workers.}
% \label{tab:gpt2_nw8_config}
% \centering
% \begin{tabular}{ll}
% \toprule
% \textbf{Parameter} & \textbf{Value} \\
% \midrule
% % Simulation time & 30000, 100000 - depends on setup. See Figure~\ref{fig:gpt2_all_regimes_nw8} \\
% Batch size & 32 \\
% Optimizer & AdamW \\
% Number of workers & 8 \\
% \midrule
% \multicolumn{2}{l}{\textbf{Algorithm-specific configurations:}} \\
% \midrule
% \textbf{Synchronized SGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
% \midrule
% \textbf{Rennala SGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
% & B set: \{16, 32, 64\} \\
% \midrule
% \textbf{Local SGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
% & B set: \{16, 32, 64\} \\
% \midrule
% \textbf{Ringmaster ASGD} & $\gamma$ range: $[2^{-11}, 2^{-10}]$ \\
% & B set: \{16, 32, 64\} \\
% \midrule
% \textbf{Async-Local SGD} & $\gamma$ range: $[2^{-11}, 2^{-7}]$ \\
% & B set: \{16, 32, 64\} \\
% & M set: \{2, 4\} \\
% \bottomrule
% \end{tabular}
% \end{table}
\clearpage
\newpage
\section{Proof of Theorem~\ref{thm:main}}
\subsection{Proof technique and reasons for choosing the conditions}
\label{sec:novel}
Before we state the main theorem and provide the proof, let us explain the intuition, the novelty, and how we identified the conditions of the theorem. The proof of the result is given in Section~\ref{sec:proof} and is relatively compact. We believe that the simplicity of our result, together with its ability to unify methods, constitutes an important contribution to the optimization community. While the initial part of the proof follows the same structure as in most related works, starting from \eqref{eq:tyfEHNd}, our treatment of the staleness term $\|x^k - z^k\|,$ which naturally arises from the step $x^{k+1} = x^k - \gamma \nabla f(z^k; \xi^k),$ is novel.
After many attempts to develop a universal theory, let us illustrate how we arrived at our conditions. Looking at Figure~\ref{fig:fork}, which provides all possible relations between $x^k$ and $z^k,$ one can easily get
% $$\|x^k - z^k\| = \gamma \|{\color{mydarkblue} \sum_{i=p}^{k - 1} \nabla f(z^i;\xi^i)} - {\color{mydarkred} \sum_{(w,\xi) \in S^k} \nabla f(w; \xi)}\|.$$
$$\norm{x^k - z^k} = \gamma \norm{{\color{mydarkblue} \sum_{i=p}^{k - 1} \nabla f(z^i;\xi^i)} - {\color{mydarkred} \sum_{(w,\xi) \in S^k} \nabla f(w; \xi)}}.$$ First, we noticed that any reasonable method should utilize ${\color{mydarkred} \sum_{(w,\xi) \in S^k} \nabla f(w; \xi)}$ in the computation of $z^k$ before applying $\nabla f(z^k; \xi^k)$ (see the previous discussion about Condition 2 in Section~\ref{sec:main}). This implies ${\color{mydarkred} \{(w; \xi)\}_{(w,\xi) \in S^k}} \subseteq {\color{mydarkblue} \{(z^i; \xi^i)\}_{i = p}^{k-1}},$ leading to the following \emph{identity}:
% $\|x^k - z^k\| = \gamma \|\sum_{j \in \bar{S}^k} \nabla f(z^j;\xi^j)\|$
$$\norm{x^k - z^k} = \gamma \norm{\sum_{j \in \bar{S}^k} \nabla f(z^j;\xi^j)}$$
for some set $\bar{S}^k \subseteq {\color{mydarkblue} \{p, \dots, k - 1\}}$ such that $\bar{S}^k \cup S^k = {\color{mydarkblue} \{p, \dots, k - 1\}}.$ The \emph{identity} says that the distance is roughly proportional to the number $\abs{\bar{S}^k}$ of stochastic gradients applied after $x^p$ and before $z^k,$ which is \emph{tightly} bounded by the tree distance from $x^k$ to the common ancestor $x^p,$ i.e., it is bounded by $\abs{{\color{mydarkblue} \{p, \dots, k - 1\}}}$ since $\bar{S}^k \subseteq {\color{mydarkblue} \{p, \dots, k - 1\}}.$
% or by the tree distance $\textnormal{dist}(x^k, z^k)$ (see Definition~\ref{def:dist}).
Under Condition 2, notice that $\abs{{\color{mydarkblue} \{p, \dots, k - 1\}}} = \max\{\abs{{\color{mydarkblue} \{p, \dots, k - 1\}}}, \abs{S^k}\} =: \textnormal{dist}(x^k, z^k),$ where we use $S^k \subseteq {\color{mydarkblue} \{p, \dots, k - 1\}}$ and Definition~\ref{def:dist}. Thus, to get a bound for $\|x^k - z^k\|,$ it is natural to introduce Condition 3, which allows us to conclude that $\abs{\bar{S}^k} \leq \textnormal{dist}(x^k, z^k) \leq R.$ It remains to use classical mathematical tools to obtain
% $\mathbb{E} \|x^k - z^k\|^2 \leq 2 \gamma^2 R \sum_{j=k - R}^{k - 1} \mathbb{E} \|\nabla f(z^j)\|^2 + 2 \gamma^2 R \sigma^2,$
$$\Exp{\norm{x^k - z^k}^2} \leq 2 \gamma^2 R \sum_{j=k - R}^{k - 1} \Exp{\norm{\nabla f(z^j)}^2} + 2 \gamma^2 R \sigma^2,$$
where the first term will be canceled by the corresponding term $- \frac{\gamma}{4} \Exp{\norm{\nabla f(z^k)}^2}$ from \eqref{eq:KexjcUoRoQTYKqIDO}.
\subsection{Full proof}
\label{sec:proof}
\MAINTHEOREM*
\begin{proof}
As the beginning, the analysis is standard. Using Assumption~\ref{ass:lipschitz_constant}, we have
\begin{align*}
f(x^{k+1}) \leq f(x^k) - \gamma \inp{\nabla f(x^k)}{\nabla f(z^k;\xi^k)} + \frac{L \gamma^2}{2} \norm{\nabla f(z^k;\xi^k)}^2
\end{align*}
for $x^{k+1} = x^{k} - \gamma \nabla f(z^k;\xi^k).$ Due to Condition 1 of the theorem and the variance decomposition equality,
\begin{align*}
&\ExpSub{k}{f(x^{k+1})}
\leq f(x^k) - \gamma \inp{\nabla f(x^k)}{\nabla f(z^k)} + \frac{L \gamma^2}{2} \ExpSub{k}{\norm{\nabla f(z^k;\xi^k)}^2} \\
&\quad= f(x^k) - \gamma \inp{\nabla f(x^k)}{\nabla f(z^k)} + \frac{L \gamma^2}{2} \norm{\nabla f(z^k)}^2 + \frac{L \gamma^2}{2} \ExpSub{k}{\norm{\nabla f(z^k;\xi^k) - \nabla f(z^k)}^2} \\
&\quad\leq f(x^k) - \gamma \inp{\nabla f(x^k)}{\nabla f(z^k)} + \frac{L \gamma^2}{2} \norm{\nabla f(z^k)}^2 + \frac{L \gamma^2 \sigma^2}{2},
\end{align*}
where $\ExpSub{k}{\cdot}$ is the expectation conditioned on $(x^k, z^k).$
In the last inequality, we use Assumption~\ref{ass:stochastic_variance_bounded}. Rewriting the dot product and using $\gamma \leq \frac{1}{2 L}$, we obtain
\begin{align}
&\ExpSub{k}{f(x^{k+1})} \nonumber \\
&\leq f(x^k) - \frac{\gamma}{2} \left(\norm{\nabla f(x^k)}^2 + \norm{\nabla f(z^k)}^2 - \norm{\nabla f(x^k) - \nabla f(z^k)}^2\right) + \frac{L \gamma^2}{2} \norm{\nabla f(z^k)}^2 + \frac{L \gamma^2 \sigma^2}{2} \nonumber \\
&\leq f(x^k) - \frac{\gamma}{2}\norm{\nabla f(x^k)}^2 - \frac{\gamma}{4} \norm{\nabla f(z^k)}^2 + \frac{\gamma}{2} \norm{\nabla f(x^k) - \nabla f(z^k)}^2 + \frac{L \gamma^2 \sigma^2}{2}. \label{eq:KexjcUoRoQTYKqIDO}
\end{align}
In the rest of the proof, we focus on $\norm{\nabla f(x^k) - \nabla f(z^k)}^2.$ Using Assumption~\ref{ass:lipschitz_constant}, we obtain
\begin{align}
\label{eq:tyfEHNd}
&\norm{\nabla f(x^k) - \nabla f(z^k)}^2 \leq L^2 \norm{x^k - z^k}^2.
\end{align}
Notice that there exist $p \in \{0, \dots, k\}$ and the closest common ancestor $x^{p}$ such that
\begin{align*}
x^k = x^{p} - \gamma \sum_{i=p}^{k - 1} \nabla f(z^i;\xi^i) = x^0 -\gamma \sum_{i=0}^{p - 1} \nabla f(z^i; \xi^i) - \gamma \sum_{i=p}^{k - 1} \nabla f(z^i;\xi^i)
\end{align*}
and
\begin{align*}
z^k = x^{p} - \gamma \sum_{(w,\xi) \in S^k} \nabla f(w; \xi) = x^0 -\gamma \sum_{i=0}^{p - 1} \nabla f(z^i; \xi^i) - \gamma \sum_{(w,\xi) \in S^k} \nabla f(w; \xi),
\end{align*}
where $S^k$ is the set of points and random variables used to compute $z^k$ starting from $x^{p}$ (see Figure~\ref{fig:fork}). Moreover, due to Condition 3, we have $\textnormal{dist}(x^{k}, z^{k}) \leq \max\{k - p, \abs{S^k}\} \leq R,$ meaning $p \geq k - R.$ In total,
\begin{align}
\label{eq:VIHiAoGKxHvVihzN}
k \geq p \geq k - R,
\end{align}
which we use later. Condition 2 assumes
\begin{align*}
&\textnormal{repr}(z^k) \eqdef \underbrace{\{(z^i; \xi^i)\}_{i = 0}^{p - 1}}_{A} \uplus \underbrace{\{(w; \xi)\}_{(w,\xi) \in S^k}}_{C} \\
&\subseteq \textnormal{repr}(x^k) \eqdef \underbrace{\{(z^i; \xi^i)\}_{i = 0}^{p - 1}}_{A} \uplus \underbrace{\{(z^i; \xi^i)\}_{i = p}^{k-1}}_{B},
\end{align*}
where $\uplus$ is the multiset union operation. Thus
\begin{align*}
\underbrace{\{(w; \xi)\}_{(w,\xi) \in S^k}}_{C} \subseteq \underbrace{\{(z^i; \xi^i)\}_{i = p}^{k-1}}_{B}
\end{align*}
and
\begin{align}
\label{eq:ITXwDTcpKnui}
x^k - z^k = - \gamma \left(\sum_{i=p}^{k - 1} \nabla f(z^i;\xi^i) - \sum_{(w,\xi) \in S^k} \nabla f(w; \xi)\right) = - \gamma \sum_{j \in \bar{S}^k} \nabla f(z^j;\xi^j),
\end{align}
where $\bar{S}^k$ is a set such that $\bar{S}^k \subseteq \{p, \dots, k - 1\}.$
% Moreover, due to Condition 3, we get $p \leq R.$
% \begin{align}
% \label{eq:kOUVqt}
% \abs{\bar{S}^k} \leq R
% \end{align}
% due to Condition 3. We will use \eqref{eq:kOUVqt} later.
Substituting \eqref{eq:ITXwDTcpKnui} to \eqref{eq:tyfEHNd},
\begin{align*}
&\norm{\nabla f(x^k) - \nabla f(z^k)}^2 \leq L^2 \gamma^2 \norm{\sum_{j \in \bar{S}^k} \nabla f(z^j;\xi^j)}^2.
\end{align*}
% Next, using the variance decomposition equality, we get
Next, using Young's inequality $\norm{x + y}^2 \leq 2 \norm{x}^2 + 2 \norm{y}^2$ for all $x, y \in \R^d$, we get
\begin{align*}
&\Exp{\norm{\nabla f(x^k) - \nabla f(z^k)}^2} \leq 2 L^2 \gamma^2 \Exp{\norm{\sum_{j \in \bar{S}^k} \nabla f(z^j)}^2} + 2 L^2 \gamma^2 \Exp{\norm{\sum_{j \in \bar{S}^k} (\nabla f(z^j;\xi^j) - \nabla f(z^j))}^2}.
\end{align*}
% Since $\{\xi^k\}_{k \geq 0}$ are mutually independent (Condition 1) and using Assumption~\ref{ass:stochastic_variance_bounded},
Since $\xi^j$ is statistically independent of
$\{(x^{i+1}, z^{i+1}, \xi^i)\}_{i=0}^{j-1}$
% $(z^j, \xi^0, \dots, \xi^{j - 1})$
for all $j \in \bar{S}^k$ (Condition 1) and using Assumption~\ref{ass:stochastic_variance_bounded},
% Note that $\bar{S}^k = \{i_{1}, \dots, i_{\ell}\}$ for some $\ell \leq k - p$ and $i_{1} < \dots < i_{\ell}.$ For all $m > j,$ consider
% \begin{align*}
% &\ExpSub{\bar{S}^k}{\inp{\nabla f(z^{i_m};\xi^{i_m}) - \nabla f(z^{i_m})}{\nabla f(z^{i_j};\xi^{i_j}) - \nabla f(z^{i_j})}} \\
% &=\ExpSub{\bar{S}^k}{\ExpSub{\xi^{i_m}}{\inp{\nabla f(z^{i_m};\xi^{i_m}) - \nabla f(z^{i_m})}{\nabla f(z^{i_j};\xi^{i_j}) - \nabla f(z^{i_j})}}}.
% \end{align*}
% We now ``freeze'' every random variance except $\xi^{i_m}.$ In this case, vectors $z^{i_m}$ and $z^{j_m}$ are deterministic due to Condition 1. Therefore
% Let us show that vector $z^{i_j}$ is also deterministic if we ``freeze'' $\xi^{i_m}.$ Using a contradiction argument, assume that $z^{i_j}$ is not deterministic. We know that $x^{i_m}$ depends on $z^{i_j}$ because $i_m > i_j.$ Thus $x^{i_m}$ is random and non-deterministic, which contradicts the argument from the previous paragraph. Therefore
% \begin{align*}
% &\ExpSub{\bar{S}^k}{\inp{\nabla f(z^{i_m};\xi^{i_m}) - \nabla f(z^{i_m})}{\nabla f(z^{i_j};\xi^{i_j}) - \nabla f(z^{i_j})}} = 0,
% \end{align*}
% and
\begin{align*}
\Exp{\norm{\nabla f(x^k) - \nabla f(z^k)}^2}
&\leq 2 L^2 \gamma^2 \Exp{\norm{\sum_{j \in \bar{S}^k} \nabla f(z^j)}^2} + 2 L^2 \gamma^2 \abs{\bar{S}^k} \sigma^2 \\
&\overset{\textnormal{Jensen's ineq.}}{\leq} 2 L^2 \gamma^2 \abs{\bar{S}^k} \sum_{j \in \bar{S}^k} \Exp{\norm{\nabla f(z^j)}^2} + 2 L^2 \gamma^2 \abs{\bar{S}^k} \sigma^2.
% &\leq L^2 \gamma^2 \abs{\bar{S}^k} \sum_{j \in \bar{S}^k} \Exp{\norm{\nabla f(z^j)}^2} + L^2 \gamma^2 \abs{\bar{S}^k} \sigma^2
\end{align*}
Due to $\bar{S}^k \subseteq \{p, \dots, k - 1\}$ and \eqref{eq:VIHiAoGKxHvVihzN}:
\begin{align*}
\Exp{\norm{\nabla f(x^k) - \nabla f(z^k)}^2}
&\leq 2 L^2 \gamma^2 R \sum_{j=k - R}^{k - 1} \Exp{\norm{\nabla f(z^j)}^2} + 2 L^2 \gamma^2 R \sigma^2.
\end{align*}
Substituting this inequality to \eqref{eq:KexjcUoRoQTYKqIDO} and taking the full expectation, we obtain
\begin{align}
\Exp{f(x^{k+1})}
&\leq \Exp{f(x^k)} - \frac{\gamma}{2}\Exp{\norm{\nabla f(x^k)}^2} - \frac{\gamma}{4} \Exp{\norm{\nabla f(z^k)}^2} + \frac{L \gamma^2 \sigma^2}{2} \nonumber \\
&\quad + \frac{\gamma}{2} \left(2 L^2 \gamma^2 R \sum_{j=k - R}^{k - 1} \Exp{\norm{\nabla f(z^j)}^2} + 2 L^2 \gamma^2 R \sigma^2\right) \nonumber \\
&\leq \Exp{f(x^k)} - \frac{\gamma}{2}\Exp{\norm{\nabla f(x^k)}^2} - \frac{\gamma}{4} \Exp{\norm{\nabla f(z^k)}^2} + L \gamma^2 \sigma^2 \nonumber \\
&\quad + L^2 \gamma^3 R \sum_{j=k - R}^{k - 1} \Exp{\norm{\nabla f(z^j)}^2} \label{eq:icrcqHMvMJ}
\end{align}
because $\gamma \leq \frac{1}{2 R L}.$ Note that $\sum_{k = 0}^{K - 1} \sum_{j=k - R}^{k - 1} \Exp{\norm{\nabla f(z^j)}^2} \leq R \sum_{k = 0}^{K - 1} \Exp{\norm{\nabla f(z^k)}^2}.$ Thus, summing \eqref{eq:icrcqHMvMJ} for $k = 0, \dots, K - 1$ and substituting $f^*,$
\begin{align*}
\Exp{f(x^{K}) - f^*}
&\leq f(x^0) - f^* - \frac{\gamma}{2} \sum_{k = 0}^{K - 1}\Exp{\norm{\nabla f(x^k)}^2} - \frac{\gamma}{4} \sum_{k = 0}^{K - 1} \Exp{\norm{\nabla f(z^k)}^2} + K L \gamma^2 \sigma^2 \nonumber \\
&\quad + L^2 \gamma^3 R^2 \sum_{k = 0}^{K - 1} \Exp{\norm{\nabla f(z^k)}^2} \\
&\leq f(x^0) - f^* - \frac{\gamma}{2} \sum_{k = 0}^{K - 1}\Exp{\norm{\nabla f(x^k)}^2} + K L \gamma^2 \sigma^2
\end{align*}
because $\gamma \leq \frac{1}{2 L R}.$ Finally, since $\Exp{f(x^{K}) - f^*} \geq 0,$
\begin{align*}
\frac{1}{K}\sum_{k = 0}^{K - 1}\Exp{\norm{\nabla f(x^k)}^2} \leq \frac{2 \Delta}{K \gamma} + 2 L \gamma \sigma^2.
\end{align*}
It is left to use that $\gamma = \min\{\frac{1}{2 L}, \frac{1}{2 R L}, \frac{\varepsilon}{4 \sigma^2 L}\}$ and the bound on $K$ from the theorem statement.
\end{proof}
\newpage
\section{Detailed Description of Algorithms and Iteration Rates}
\label{sec:algorithms}
In this section, we provide a detailed description together with theoretical analysis of the algorithms from the main part.
\subsection{\algname{Vanilla SGD}}
\label{sec:vanilla_sgd}
We start we the celebrated \algname{Vanilla SGD} algorithm, which formally can be implemented in the following way:
\begin{algorithm}[H]
\caption{\algname{Vanilla SGD}}
\label{alg:vanilla_sgd}
\begin{algorithmic}[1]
\STATE \textbf{Input:} starting point $w^0 \in \R^d$, step size $\gamma > 0$
\FOR{$k = 0, 1, 2, \dots$}
\STATE Sample $\eta^k \sim \mathcal{D}_{\xi}$ \hfill ($\{\eta^k\}$ are i.i.d.)
\STATE Compute stochastic gradient $\nabla f(w^k; \eta^k)$
\STATE Update $w^{k+1} = w^k - \gamma \nabla f(w^k; \eta^k)$
\ENDFOR
\end{algorithmic}
\end{algorithm}
The corresponding computation tree can defined by the recursion
\begin{align}
\label{eq:ksGyF}
w^{k+1} = w^k - \gamma \nabla f(w^k; \eta^k)
\end{align}
for all $k \geq 0.$
\begin{figure}[h]
\centering
\includegraphics[page=10,width=0.5\textwidth]{./graphs/graphs_new.pdf}
\caption{The computation tree of \algname{Vanilla SGD}}
\label{fig:sgd_graph}
\end{figure}
While the iteration rate of \algname{Vanilla SGD} is well-known \citep{lan2020first}, we prove its convergence using our new framework for clarity.
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree \eqref{eq:ksGyF} of \algname{Vanilla SGD}. Then, $\{x^k\}_{k \geq 0}$ is a main branch with $x^k = w^k$, $\{(z^k, \xi^k)\}_{k \geq 0}$ is the corresponding auxiliary sequence with $(z^k, \xi^k) = (w^k, \eta^k)$ (see Def.~\ref{def:branch}), and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{2 L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\end{theorem}
\begin{proof}
Indeed, $\{x^k\}_{k \geq 0}$ and $\{(z^k, \xi^k)\}_{k \geq 0}$ satisfy Def.~\ref{def:branch} (see Fig.~\ref{fig:sgd_graph}). Moreover, all conditions of Theorem~\ref{thm:main} are fulfilled: Condition 1 holds because the sequence $\{\eta^k\}$ is i.i.d., we have $\textnormal{repr}(z^k) = \textnormal{repr}(x^k)$ since $x^k = z^k$, and consequently, $R = \sup_{k \geq 0} \textnormal{dist}(x^k, z^k) = 0$.
\end{proof}
\newpage
\subsection{\algname{Rennala SGD}}
\label{sec:rennala_sgd}
We now apply Theorem~\ref{thm:main} to \algname{Rennala SGD}. The iteration rate of \algname{Rennala SGD} is also well-known \citep{tyurin2023optimal}, but we provide a proof for completness. \algname{Rennala SGD} can be formally described as follows:
\begin{algorithm}[H]
\caption{\algname{Rennala SGD} \citep{tyurin2023optimal}}
\label{alg:rennala}
\begin{algorithmic}[1]
\STATE \textbf{Input:} point $w^0 \in \R^d$, stepsize $\gamma > 0$, batch size $B\in \N$
\STATE Workers start computing stochastic gradients at $w^0$
\FOR{$k = 0, \dots, K - 1$}
\STATE $g^k_i = 0$ for all $i \in [n]$; $b=0$
\WHILE{$b < B$}
\STATE Wait for the moment when stochastic gradient is computed by worker
\STATE Gradient $\nabla f(w^{k - \delta}; \eta)$ is computed by worker $i,$ $\quad \eta \sim \mathcal{D}_{\xi}$ \\
\IF{$\delta = 0$}
\color{black}
\STATE Update $g^k_i = g^k_i + \nabla f(w^{k - \delta}; \eta)$ locally in worker $i$
\STATE $b=b+1$
\ELSE
\STATE Ignore $\nabla f(w^{k - \delta}; \eta)$
\ENDIF
\STATE Worker $i$ begins calculating gradient at $w^{k}$ \\
\ENDWHILE
\STATE Aggregate: $g^k = \sum_{i=1}^n g^k_i$ \hfill (e.g, via \texttt{AllReduce})
\STATE Update: $w^{k+1} = w^{k} - \gamma g^k$
\ENDFOR
\end{algorithmic}
\end{algorithm}
To use Theorem~\ref{thm:main}, we have to construct the computation tree of \algname{Rennala SGD}. It can be constructed in the following way:
\begin{alignat}{2}
x^{1} &= x^{0} - \gamma \nabla f(x^0;\xi^{0}), \quad &\dots,\quad & x^{B} = x^{B-1} - \gamma \nabla f(x^0;\xi^{B-1}), \label{eq:dOBxW}\\
x^{B+1} &= x^{B} - \gamma \nabla f(x^{B};\xi^{B}), \quad &\dots,\quad & x^{2 B} = x^{2 B-1} - \gamma \nabla f(x^B;\xi^{2 B - 1}), \quad \dots \nonumber,
\end{alignat}
where $\{\xi^i\}$ are i.i.d.~from $\mathcal{D}_{\xi}.$ See also a visualization in Figure~\ref{fig:minibatch_sgd_graph}. One can easily show that $w^{1} = x^{0}, w^{1} = x^{B}, w^{2} = x^{2 B},$ etc.
\begin{figure}[h]
\centering
\includegraphics[page=11,width=0.6\textwidth]{./graphs/graphs_new.pdf}
\caption{The computation tree of \algname{Rennala SGD}}
\label{fig:minibatch_sgd_graph}
\end{figure}
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree \eqref{eq:dOBxW} of \algname{Rennala SGD}, then $\{x^k\}_{k \geq 0}$ is a main branch, $\{(z^k, \xi^k)\}_{k \geq 0}$ with $(z^k, \xi^k) = (x^{B\flr{k / B}}, \xi^k)$ is the corresponding auxiliary sequence, and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{2 B L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\label{thm:rennala}
\end{theorem}
% The following proof is detailed to provide clearer initial intuition about the framework.
\begin{proof}
Clearly, $\{x^k\}_{k \geq 0}$ is a main branch and $\{(z^k, \xi^k)\}_{k \geq 0}$ is the corresponding sequence by the construction in \eqref{eq:dOBxW}. Moreover, $\xi^k$ is independent of $\{(x^{i+1}, z^{i+1}, \xi^i)\}_{i=0}^{k-1}$ in \eqref{eq:dOBxW} because $\{\xi^i\}$ are i.i.d. (Condition 1 is satisfied). Next, notice that
\begin{align*}
&\textnormal{repr}(z^0) = \textnormal{repr}(x^0) = \emptyset, \\
&\textnormal{repr}(z^1) = \textnormal{repr}(x^0) = \emptyset \subseteq \textnormal{repr}(x^1), \\
&\qquad\qquad \vdots \\
&\textnormal{repr}(z^{B-1}) = \textnormal{repr}(x^0) = \emptyset \subseteq \textnormal{repr}(x^{B-1})
\end{align*}
because $z^k = x^0$ for all $k < B.$ Next,
\begin{align*}
&\textnormal{repr}(z^B) = \textnormal{repr}(x^B), \\
&\textnormal{repr}(z^{B+1}) = \textnormal{repr}(x^B) \subseteq \textnormal{repr}(x^{B+1}), \\
&\quad\qquad\qquad \vdots \\
&\textnormal{repr}(z^{2 B-1}) = \textnormal{repr}(x^B) \subseteq \textnormal{repr}(x^{2 B-1}),
\end{align*}
because $z^k = x^B$ for all $B \leq k < 2 B,$ where $\textnormal{repr}(x^B) \subseteq \textnormal{repr}(x^{B+1}), \dots, \textnormal{repr}(x^B) \subseteq \textnormal{repr}(x^{2 B - 1})$ due to \eqref{eq:dOBxW}. We can continue and show that $\textnormal{repr}(z^k) \subseteq \textnormal{repr}(x^k)$ for all $k \geq 0$ (Condition 2 is satisfied). It is left to notice that
\begin{align*}
\sup_{k \geq 0} \textnormal{dist}(x^k, z^k) \leq B - 1,
\end{align*}
because
\begin{align*}
\textnormal{dist}(x^0, z^0) &= 0, \\
\textnormal{dist}(x^1, z^1) = \textnormal{dist}(x^1, x^0) &= 1, \\
&\vdots \\
\textnormal{dist}(x^{B-1}, z^{B-1}) = \textnormal{dist}(x^{B-1}, x^0) &= B - 1, \\
\textnormal{dist}(x^{B}, z^{B}) = \textnormal{dist}(x^{B}, x^{B}) &= 0, \\
\textnormal{dist}(x^{B + 1}, z^{B + 1}) = \textnormal{dist}(x^{B + 1}, x^{B}) &= 1, \\
&\vdots
\end{align*}
The maximum tree distance between $x^k$ and $z^k$ is $B - 1$. Thus, $R = B - 1$ in Condition 3.
\end{proof}
\newpage
\subsection{\algname{Local SGD}}
\label{sec:local_sgd}
The \algname{Local SGD} method is described in the following algorithm:
\begin{algorithm}[H]
\caption{\algname{Local SGD}}
\label{alg:local_sgd}
\begin{algorithmic}[1]
\REQUIRE Initial model $w^0$, step size $\gamma$, parameter $B$
\FOR{$k = 0, 1, 2, \dots$}
\STATE Broadcast $w^k$ to all workers
\FOR{each worker $i \in [n]$ \textbf{in parallel}}
\STATE Worker $i$ starts \texttt{LocalSGDWorker}($w^k, \gamma$) from Algorithm~\ref{alg:local_sgd_worker_function}
\ENDFOR
\STATE Wait for the moment when $\sum_{i=1}^{n} M_i = B$ \hfill ($\{M_i\}$ from \texttt{LocalSGDWorker}($w^k, \gamma$))
\STATE Ask workers to stop\footnotemark running \texttt{LocalSGDWorker}($w^k, \gamma$)
\STATE Aggregate $\gamma \sum_{i=1}^{n} \sum_{j=0}^{M_i - 1} \nabla f(z^{k,j}_i; \eta^{k,j}_i)$ from the workers (e.g, via \texttt{AllReduce})
\STATE Update $w^{k+1} = w^{k} - \gamma \sum_{i=1}^{n} \sum_{j=0}^{M_i - 1} \nabla f(z^{k,j}_i; \eta^{k,j}_i)$
\ENDFOR
\end{algorithmic}
\end{algorithm}
\newcommand{\footnotesametext}{Alternatively, allow the workers to finish computing their stochastic gradients without waiting for them (since \texttt{AllReduce} can be run in parallel), but discard these gradients in subsequent iterations, as they are no longer relevant. This approach may introduce a delay before the workers begin their next local steps. \\
Another option is to allow the workers to finish computing their stochastic gradients without waiting for them, and send these gradients in the next iteration. If some gradients are still not computed by then due to delays, simply discard them.}
% We leave it as future work.}
% One can show that this strategy does not degrade the theoretical convergence rate (up to constant factors). We leave it as future work for simplicity of presentation because it is a little bit technical without deep insights.}
\footnotetext{\footnotesametext \label{foot:local_sgd_stop}}
% \footnotetext{Alternatively, allow the workers to finish computing their stochastic gradients without waiting for them—since \texttt{AllReduce} can run in parallel—and send these gradients in the next iteration. If some gradients are still not computed by then due to delays, simply discard them. \label{foot:local_sgd_stop}}
\begin{algorithm}[H]
\caption{\texttt{LocalSGDWorker}($w, \gamma$) in worker $i$ at round $k$}
\label{alg:local_sgd_worker_function}
\begin{algorithmic}[1]
\STATE $z^{k,0}_i = w$
\STATE $M_i \gets 0$
\WHILE{True}
\STATE $z^{k,M_i + 1}_i = z^{k,M_i}_i - \gamma \nabla f(z^{k,M_i}_i; \eta^{k,M_i}_i), \quad \eta^{k,M_i}_i \sim \mathcal{D}_{\xi}$
\STATE $M_i = M_i + 1$
\ENDWHILE
\end{algorithmic}
\end{algorithm}
One key change compared to the previous work is that individual local steps $M_i$ are not predefined. Moreover, the server tracks the sum $\sum_{i=1}^{n} M_i$ and waits for the moment $\sum_{i=1}^{n} M_i = B$ before collecting the locally calculated gradients. With a proper choice of $B,$ we will prove the optimal computational time complexity of the method in Section~\ref{sec:proof_compt}.
The corresponding computation tree of \algname{Local SGD} can be constructed in the following way. Define $N_k \eqdef k \times B$ and take $k = 0.$ Then
\begin{eqnarray}
\begin{aligned}
&z^{k,1}_i = z^{k,0}_i - \gamma \nabla f(x^{N_k}; \eta^{k,0}_i), \\
&z^{k,2}_i = z^{k,1}_i - \gamma \nabla f(z^{k,1}_i; \eta^{k,1}_i), \\
&\qquad \vdots \\
&z^{k,M_i}_i = z^{k,M_i - 1}_i - \gamma \nabla f(z^{k,M_i - 1}_i; \eta^{k,M_i - 1}_i),
\label{eq:pbHgXapdjZGUQs}
\end{aligned}
\end{eqnarray}
for all $i \in [n],$ and
\begin{eqnarray}
\begin{aligned}
x^{N_k + 1} &= x^{N_k} - \gamma \nabla f(z^{k, 0}_1; \eta^{k, 0}_1), \\
&\vdots\\
x^{N_k + M_1} &= x^{N_k + M_1 - 1} - \gamma \nabla f(z^{k, M_1 - 1}_1; \eta^{k, M_1 - 1}_1), \\
x^{N_k + M_1+1} &= x^{N_k + M_1} - \gamma \nabla f(z^{k, 0}_2; \eta^{k, 0}_2), \\
&\vdots\\
x^{N_k + M_1 + M_2} &= x^{N_k + M_1 + M_2 - 1} - \gamma \nabla f(z^{k, M_2 - 1}_2; \eta^{k, M_2 - 1}_2), \\
&\vdots\\
x^{N_{k + 1}} &= x^{N_k + \sum_{i=1}^{n} M_i - 1} - \gamma \nabla f(z^{k, M_n - 1}_{n}; \eta^{k, M_n - 1}_{n}).
\label{eq:yapgNwHcHepGaY}
\end{aligned}
\end{eqnarray}
Repeat the previous steps with $k = k + 1$ starting at $x^{N_{k + 1}} = x^{N_k + B}.$ See illustration in Figure~\ref{fig:local_sgd_graph}. One can easily show that $w^{1} = x^{B}, w^{2} = x^{2 B}, \dots, w^{k} = x^{k B}, \dots, $ where $w^{k}$ is the sequence from Algorithm~\ref{alg:local_sgd}.
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree (\eqref{eq:pbHgXapdjZGUQs} and \eqref{eq:yapgNwHcHepGaY}) of \algname{Local SGD}, then $\{x^k\}_{k \geq 0}$ is a main branch
% , $\{(z^k, \xi^k)\}_{k \geq 0}$ with $$(z^k, \xi^k) = (x^{B\flr{k / B}}, \xi^k)$$ is the corresponding auxiliary sequence,
and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{2 B L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\label{thm:local_sgd}
\end{theorem}
Although the proof may seem technical due to the heavy notation in \eqref{eq:yapgNwHcHepGaY}, it is actually straightforward when you refer to Figure~\ref{fig:local_sgd_graph}. This figure clearly shows that all conditions of Theorem~\ref{thm:main} are satisfied with $R = B - 1$ because $\sum_{i=1}^n M_i = B$ in every global iteration. The condition $\sum_{i=1}^n M_i = B$ helps us to insure that the maximum tree distance $\sup_{k \geq 0} \textnormal{dist}(x^k, z^k) \leq B - 1.$
\begin{proof}
Clearly, $\{x^k\}_{k \geq 0}$ is a main branch by Definition~\ref{def:branch}. The corresponding auxiliary sequence can be inferred from \eqref{eq:yapgNwHcHepGaY}: $(z^0, \xi^0) = (z^{0, 0}_1, \eta^{0, 0}_1), \dots, (z^{M_1}, \xi^{M_1}) = (z^{0, M_1}_1, \eta^{0, M_1}_1),$ and etc. Condition 1 is satisfied because $\{\eta^{k, j}_{i}\}$ are i.i.d., and by the construction \eqref{eq:yapgNwHcHepGaY}. Condition 2 of Theorem~\ref{thm:main} holds because the same stochastic gradients used for computing $z^{k}$ are also used for $x^{k}$, as shown in Figure~\ref{fig:local_sgd_graph}. This can be formally verified using \eqref{eq:yapgNwHcHepGaY} and \eqref{eq:pbHgXapdjZGUQs}.
It is left to notice that
\begin{align*}
\sup_{k \geq 0} \textnormal{dist}(x^k, z^k) \leq B - 1
\end{align*}
because the maximum number of edges to the common closest ancestor can not exit $B - 1$.
\end{proof}
% Condition 2 of Theorem~\ref{thm:main} is satisfied because the stochastic gradients used to compute $z^{j}_i$ are also used to compute $x^{k},$ where $\nabla f(z^{j}_i; \xi^{j}_i)$ is applied to compute $x^{k+1}.$ For any such $z^{j}_i$ and $x^{k},$ $\textnormal{dist}(x^{k}, z^{j}_i) \leq \sum_{i=1}^{n} M_i - 1$ (see Figure~\ref{fig:local_sgd_graph}). Thus, the \emph{iteration rate} is $\cO\left(\nicefrac{(\sum_{i=1}^{n} M_i) L \Delta}{\varepsilon} + \nicefrac{\sigma^2 L \Delta}{\varepsilon^2}\right).$
\newpage
\subsection{\algname{Ringmaster ASGD}}
\label{sec:ringmaster_asgd}
\begin{algorithm}[H]
\caption{\algname{Ringmaster ASGD} \citep{maranjyan2025ringmaster}}
\label{alg:ringmasternew}
\begin{algorithmic}[1]
\STATE \textbf{Input:} point $w^0 \in \R^d$, stepsize $\gamma > 0,$ delay threshold $G \in \N$
\STATE Set $k = 0$
\STATE Workers start computing stochastic gradients at $w^0$
\WHILE{True}
\STATE Gradient $\nabla f(w^{k-\delta^k}; \eta^{k-\delta^k}_{i})$ arrives from worker $i$
\IF{$\delta^k < G$}
\STATE Update: $w^{k+1} = w^{k} - \gamma \nabla f(w^{k-\delta^k}; \eta^{k-\delta^k}_{i})$
\STATE Worker $i$ begins calculating at $w^{k+1}$ \hfill ($\{\eta^{k}_{i}\}$ are i.i.d.)
\STATE Update the iteration number $k = k + 1$
\ELSE
\STATE Ignore the outdated gradient $\nabla f(w^{k-\delta^k}; \eta^{k-\delta^k}_{i})$
\STATE Worker $i$ begins calculating at $w^{k}$
\ENDIF
\ENDWHILE
\end{algorithmic}
\end{algorithm}
In this method, a main branch can be defined as
\begin{align}
x^k = w^k
\label{eq:JYByrgqAiCTpxi}
\end{align}
and the auxiliary sequence is defined as $(z^k, \xi^k) = (x^{k - \delta^k}, \eta^{k - \delta^k}_{i})$ for all $k \geq 0.$
\begin{figure}[h]
\centering
\includegraphics[page=12,width=0.5\textwidth]{./graphs/graphs_new.pdf}
\caption{The computation tree of \algname{Ringmaster ASGD}}
\label{fig:async_sgd_graph}
\end{figure}
\begin{theorem}
\label{thm:ringmaster_asgd}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree of \algname{Ringmaster ASGD}, then $\{x^k\}_{k \geq 0},$ defined in \eqref{eq:JYByrgqAiCTpxi}, is a main branch
and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 G L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{2 G L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\end{theorem}
\begin{proof}
Condition 1 is satisfied because $\{\eta^{k-\delta^k}_{i}\}$ are i.i.d., $x^{k} = w^k$ and $z^k = x^{k-\delta^k}$ do not depend on $\xi^k = \eta^{k-\delta^k}_{i}.$ Condition 2 is satisfied because $\textnormal{repr}(z^{k}) = \textnormal{repr}(w^{k-\delta^k}) \subseteq \textnormal{repr}(w^{k}) = \textnormal{repr}(x^{k}).$ Condition 3 is satisfied with $R = G - 1$ because
\begin{align*}
\textnormal{dist}(x^k, z^k) = \textnormal{dist}(x^{k}, x^{k-\delta^k}) = \delta^k \leq G - 1,
\end{align*}
where the second equality due to the number of edges between $x^{k}$ and $x^{k-\delta^k}$ and the last inequality due to the fact that $\delta^k$ is bounded by $B$ in Algorithm~\ref{alg:ringmasternew}.
\end{proof}
\newpage
\subsection{\algname{Cycle SGD}}
\label{sec:circular_sgd}
We now present a new method, called \algname{Cycle SGD}:
\begin{algorithm}[H]
\caption{\algname{Cycle SGD}}
\label{alg:circular_local_sgd}
\begin{algorithmic}[1]
\REQUIRE Initial model $w^0$, step size $\gamma$, group size $s$
\STATE Partition workers into groups of size $s$:
$$G_1 = \{1, \dots, s\}, G_2 = \{s + 1, \dots, 2 s\}, \dots, G_{\lceil n/s \rceil} = \{(\lceil n/s \rceil - 1) s + 1, \dots, n\}$$
in a circular manner.
\STATE Broadcast $w^{0}$ to all workers and assign the local variables $z^0_i = w^{0}$ and $M_i = 0$ for all $i \in [n]$
\WHILE{True}
\FOR{group index $g = 1$ to $\lceil n/s \rceil$}
\FOR{each worker $i \in [n]$ \textbf{in parallel}}
\STATE $z_i^{M_i + 1} = z_i^{M_i} - \gamma \nabla f(z_i^{M_i}; \eta_i^{M_i}), \quad \eta_i^{M_i} \sim \mathcal{D}_{\xi}$
\STATE $M_i = M_i + 1$
\ENDFOR
\STATE Aggregate $\gamma \sum_{i \in G_g} \sum_{j=1}^{M_i} \nabla f(z^{j}_i; \eta^{j}_i)$ from the workers of group $G_g$ only
\STATE Server aggregates and updates the model:
$$
w^{r+1} = w^r - \gamma \sum_{i \in G_g} \sum_{j=0}^{M_i - 1} \nabla f(z^{j}_i; \eta^{j}_i)
$$
\STATE Broadcast $w^{r + 1}$ to all workers of group $g$ and assign the local variables $z^0_i = w^{r + 1}$ and $M_i = 0$ for all $i \in G_g$
\STATE $r = r + 1$
\ENDFOR
\ENDWHILE
\end{algorithmic}
\end{algorithm}
This method operates similarly to \algname{Local SGD}, with workers performing local steps. However, a key difference is that only $s$ workers synchronize at each step, rather than all $n$ workers. This strategy can be advantageous in scenarios where reducing peak bandwidth is desirable. A visualization of the corresponding computation tree is in Figure~\ref{fig:circular_local_sgd_graph}. For this algorithm, the first $\sum_{i=1}^{n} M_i$ nodes of the main branch can be defined as
\begin{equation}
\begin{aligned}
\label{eq:whoDleMteaoI}
x^{1} &= x^{0} - \gamma \nabla f(z^{0}_1; \eta^{0}_1), \\
&\vdots\\
x^{M_1} &= x^{M_1 - 1} - \gamma \nabla f(z^{M_1 - 1}_1; \eta^{M_1 - 1}_1), \\
&\vdots\\
x^{\sum_{i=1}^{s-1} M_i + 1} &= x^{\sum_{i=1}^{s-1} M_i} - \gamma \nabla f(z^{0}_{s}; \eta^{0}_{s}), \\
&\vdots \\
x^{\sum_{i=1}^{s} M_i} &= x^{\sum_{i=1}^{s} M_i - 1} - \gamma \nabla f(z^{M_s - 1}_s; \eta^{M_s - 1}_s), \\
&\vdots \\
\end{aligned}
\end{equation}
Notice that $x^{\sum_{i=1}^{s} M_i} \equiv w^{1},$ where we capture and unroll all stochastic gradients from the first group. The next nodes of the main branch can be defined in a similar way going through all groups circularly.
\begin{figure}[h]
\centering
\includegraphics[page=14,width=0.5\textwidth]{./graphs/graphs_new.pdf}
\caption{An example of \algname{Cycle SGD} computation tree.}
\label{fig:circular_local_sgd_graph}
\end{figure}
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree of \algname{Cycle SGD} (Alg.~\ref{alg:circular_local_sgd}), then $\{x^k\}_{k \geq 0},$ defined in \eqref{eq:whoDleMteaoI}, is a main branch
and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{8 n^2 L \Delta}{s \varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{s}{4 n^2 L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\label{thm:circular_local_sgd}
\end{theorem}
\begin{proof}
Once again, the proof is geometric. As an example, consider Figure~\ref{fig:circular_local_sgd_graph} together with Algorithm~\ref{alg:circular_local_sgd}. One can easily show that Conditions 1 and 2 are satisfied similarly to the proof of Theorem~\ref{alg:local_sgd}. However, the maximum tree distance is different since we synchronize the workers in a circular manner.
First, the number of local steps $M_i \leq \left\lceil\frac{n}{s}\right\rceil \leq \frac{2n}{s}$ because each worker computes one stochastic gradient in the inner loop and synchronizes every $\left\lceil\frac{n}{s}\right\rceil$ loops.
Next, the maximum tree distance between a point $x^{k}$ on the main branch and the corresponding point of the auxiliary sequence $z^{k}$ is at most $\frac{2n^2}{s}$. Let us explain this step. Consider any $x^{k}$ and $z^{k}$, and their closest common ancestor $w^k$ (in Figure~\ref{fig:circular_local_sgd_graph}, for instance, take $x^{5}$, $z^{2,2}_2$, and $x^{3}$ accordingly).
The number of edges from $z^{k}$ to $w^k$ never exceeds $\frac{2n}{s}$ due to the bound on the number of local steps. The number of edges from $x^{k}$ to $w^k$ never exceeds $\frac{2n^2}{s}$ because, while one worker performs local steps, other workers can grow the main branch by at most $\left\lceil\frac{n}{s}\right\rceil \times (n - 1) \leq \frac{2n (n - 1)}{s}$ points before the worker that computed $z^k$ is synchronized\footnote{For instance, see Figure~\ref{fig:circular_local_sgd_graph}, where, before the algorithm applies $\nabla f(z^{2,1}_{2};\xi^{2,1}_{2})$ from the second worker, the main branch grows by two edges, from $x^{3}$ to $x^{5}$, due to gradients computed by the first worker.}.
Thus, we can take $R = \frac{2n^2}{s}$ in Condition 3 of Theorem~\ref{thm:main}.
\end{proof}
% \STATE $z^0_i = w$
% \STATE $m = 0$
% \WHILE{True}
% \STATE $z^{m+1}_i = z^{m}_i - \gamma \nabla f(z^m_i; \eta), \quad \eta \sim \mathcal{D}_{\xi}$
% \STATE $m = m + 1$
% \ENDWHILE
% \begin{algorithm}[H]
% \caption{\algname{CircularLocalSGDWorker}($w, \gamma$) in worker $i$}
% \label{alg:circular_local_sgd_worker_function}
% \begin{algorithmic}[1]
% \STATE $z^0_i = w$
% \STATE $m = 0$
% \WHILE{True}
% \STATE $z^{m+1}_i = z^{m}_i - \gamma \nabla f(z^m_i; \eta), \quad \eta \sim \mathcal{D}_{\xi}$
% \STATE $m = m + 1$
% \ENDWHILE
% \end{algorithmic}
% \end{algorithm}
\newpage
\subsection{\algname{Async-Local SGD}}
\label{sec:async_plus_local}
The following algorithm is a mixture of \algname{Asynchronous SGD} and \algname{Local SGD}, which we formalize in the following way.
\begin{algorithm}[H]
\caption{\algname{Async-Local SGD}}
\label{alg:async_plus_local}
\begin{algorithmic}[1]
\STATE \textbf{Input:} point $x^0 \in \R^d$, stepsize $\gamma > 0,$ delay threshold $B \in \N,$ number of local steps $M$
\STATE Set $k = 0$
\STATE Workers start running local steps at $w^0$ with Alg.~\ref{alg:local_sgd_worker_function_mix} for $M$ steps
\WHILE{True}
\STATE Sum $\gamma \sum_{p = 0}^{M - 1} \nabla f(z^{p}_{i_k}; \eta^{p}_{i_k})$ arrives from some worker $i_k$
\STATE Find the tree distance $\delta^k = \textnormal{dist}(w^{k}, z^{0}_{i_k})$ \\ (delay $\delta^k$ of $w^{k - \delta^k},$ at which point worker $i_k$ started local steps)
\IF{$\delta^k < B$}
\STATE Update: $w^{k+1} = w^{k} - \gamma \sum_{p = 0}^{M - 1} \nabla f(z^{p}_{i_k}; \eta^{p}_{i_k})$
\STATE Worker $i$ starts running local steps at $w^{k+1}$ with Alg.~\ref{alg:local_sgd_worker_function_mix} for $M$ steps
\STATE Update the iteration number $k = k + 1$
\ELSE
\STATE Ignore the outdated sum $\gamma \sum_{p = 0}^{M - 1} \nabla f(z^{p}_{i_k}; \eta^{p}_{i_k})$
\STATE Worker $i$ starts running local steps at $w^{k}$ with Alg.~\ref{alg:local_sgd_worker_function_mix} for $M$ steps
\ENDIF
\ENDWHILE
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[H]
\caption{\texttt{LocalSGDWorker}($w, \gamma, M$) in worker $i$}
\label{alg:local_sgd_worker_function_mix}
\begin{algorithmic}[1]
\STATE $z^{0}_i = w$
\FOR{$p = 0, \dots M - 1$}
\STATE $z^{p + 1}_i = z^{p}_i - \gamma \nabla f(z^{p}_i; \eta^{p}_i), \quad \eta^{p}_i \sim \mathcal{D}_{\xi}$
\ENDFOR
\STATE Send to the server $\gamma \sum_{p = 0}^{M - 1} \nabla f(z^{p}_i; \eta^{p}_i)$
\end{algorithmic}
\end{algorithm}
If $M = 1,$ then this method reduces to \algname{Ringmaster ASGD} (Alg.~\ref{alg:ringmasternew}). Taking $M > 1,$ we can improve the time complexity of \algname{Ringmaster ASGD} by decreasing the number of times when workers synchronize with the server. For this method, it is natural to take a main branch as
\begin{equation}
\begin{aligned}
x^{1} &= x^{0} - \gamma \nabla f(z^{0}_{i_1}; \eta^{0}_{i_1}), \\
&\vdots \\
x^{M} &= x^{M - 1} - \gamma \nabla f(z^{M - 1}_{i_1}; \eta^{M - 1}_{i_1}),\\
&\vdots \\
x^{M (k - 1) + 1} &= x^{M (k - 1)} - \gamma \nabla f(z^{0}_{i_k}; \eta^{0}_{i_k}), \\
&\vdots, \\
x^{M k} &= x^{M k - 1} - \gamma \nabla f(z^{M - 1}_{i_k}; \eta^{M - 1}_{i_k}), \\
&\vdots
\label{eq:kfHzyNb}
\end{aligned}
\end{equation}
and so on. Notice that $x^0 \equiv w^0, x^{M} \equiv w^1,$ etc.
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree of \algname{Async-Local SGD} (Alg.~\ref{alg:async_plus_local}), then $\{x^k\}_{k \geq 0},$ defined in \eqref{eq:kfHzyNb}, is a main branch
and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 (B + M - 1) L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{4 (B + M - 1) L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\label{thm:async_plus_local}
\end{theorem}
\begin{proof}
Similar to the previous proofs, Condition 1 is satisfied for the main branch $\{x^k\}_{k \geq 0}$ because all random variables $\{\eta^{i}_{j}\}$ in \eqref{eq:kfHzyNb} are i.i.d., and $x^{0}$ and $z^{0}_{i_1}$ do not depend on $\eta^{0}_{i_1}.$ Points $x^{M - 1}$ and $z^{M - 1}_{i_1}$ do not depend on $\eta^{M - 1}_{i_1},$ and so on. Conditions 2 is satisfied because all stochastic gradients used to compute $z^{p}_{i_k}$ are also used to compute the corresponding point on the main branch for all $p \in \{0, \dots, M - 1\}$ and $k \geq 0$ (see Figure~\ref{fig:async_plus_local}). Condition 3 is satisfied with $R = B - 1 + M - 1 = B + M - 2$ due to the inequality $\delta^k = \textnormal{dist}(w^{k}, z^{0}_{i_k}) < B$ in Algorithm~\ref{alg:async_plus_local} and the fact every worker calculates $M$ stochastic gradients, which ensures that the tree distance between $z^{0}_{i_k}$ and the corresponding point from the main brain branch is at most $B - 1,$ the tree distance between $z^{1}_{i_k}$ and the corresponding point from the main brain branch is at most $B - 2,$ $\dots,$ the tree distance between $z^{M - 1}_{i_k}$ and the corresponding point from the main brain branch is at most $B + M - 2.$
\end{proof}
\subsection{\algname{Async-Batch SGD}}
\label{sec:async_batch}
This method does the same steps as \algname{Async-Local SGD} with the only difference that the workers calculate mini-batches instead of local steps:
\begin{algorithm}[H]
\caption{\texttt{BatchSGDWorker}($w, \gamma, M$) in worker $i$}
\label{alg:local_sgd_worker_function_mix_batch}
\begin{algorithmic}[1]
\STATE $z^{0}_i = w$
\FOR{$p = 0, \dots M - 1$}
\STATE Calculate $\nabla f(z^{p}_i; \eta^{p}_i), \quad \eta^{p}_i \sim \mathcal{D}_{\xi}$
\STATE $z^{p + 1}_i = z^{p}_i$
\ENDFOR
\STATE Send to the server $\gamma \sum_{p = 0}^{M - 1} \nabla f(z^{p}_i; \eta^{p}_i)$
\end{algorithmic}
\end{algorithm}
One can easily show that these methods share the same theoretical guarantees (Sections~\ref{sec:async_plus_local}, \ref{sec:proof_compt}, and \ref{sec:proof_communication}) as \algname{Async-Local SGD}.
\newpage
\subsection{\algname{Local-Async SGD}}
\label{sec:local_plus_async}
One way to interpret the following algorithm is that the workers are partitioned into groups, with each group running \algname{Asynchronous SGD}. Then, at certain points, all workers synchronize, and start running \algname{Asynchronous SGD} at a new point. One of the important novelties here is the condition $\sum_{g=1}^{s} m_g = B,$ which, with a proper $B,$ leads to the optimal computational time complexity (Section~\ref{sec:proof_compt}).
\begin{algorithm}[H]
\caption{\algname{Local-Async SGD}}
\label{alg:grouped_async_local_sgd}
\begin{algorithmic}[1]
\REQUIRE Initial model $w^0$, step size $\gamma$, parameter $B,$ group partitions $G_1, \dots, G_s$
\FOR{$k = 0, 1, 2, \dots$}
\STATE Broadcast $w^k$ to all groups
\FOR{each worker $g \in [s]$ \textbf{in parallel}}
\STATE Group $g$ starts AsynchronousSGDGroup($w^k, \gamma$) from Algorithm~\ref{alg:local_sgd_worker_function_mix_sgd}
\ENDFOR
\STATE Wait for the moment when $\sum_{g=1}^{s} m_g = B$ \hfill ($\{m_g\}$ from AsynchronousSGDGroup($w^k, \gamma$))
\STATE Ask the groups to stop\footnotemark running AsynchronousSGDGroup($w^k, \gamma$)
\STATE Aggregate $\gamma \sum_{g=1}^{s} \sum_{j=0}^{m_g - 1} \nabla f(v^{j-\delta^{j}}_g; \eta^{j}_{g})$ from the groups \hfill ($\{\eta^{j}_g\}$ are i.i.d.)
\STATE Update $w^{k+1} = w^{k} - \gamma \sum_{g=1}^{s} \sum_{j=0}^{m_g - 1} \nabla f(v^{j-\delta^{j}}_g; \eta^{j}_{g})$
\ENDFOR
\end{algorithmic}
\end{algorithm}
% \footnotetext{Alternatively, allow the workers to finish computing their stochastic gradients without waiting for them (\texttt{AllReduce} can be run in parallel), but discard these gradients in subsequent iterations, as they are no longer relevant. This approach may introduce a delay before the workers begin their next local steps.}
\footnotetext{\footnotesametext}
% \footnotetext{Alternatively, allow the workers to finish computing their stochastic gradients without waiting for them—since \texttt{AllReduce} can run in parallel—and send these gradients in the next iteration. If some gradients are still not computed by then due to delays, simply discard them.}
\begin{algorithm}[H]
\caption{AsynchronousSGDGroup($w, \gamma$) in group $g$}
\label{alg:local_sgd_worker_function_mix_sgd}
\begin{algorithmic}
\STATE \textbf{Input:} point $v^0_g \in \R^d$, stepsize $\gamma > 0$
\STATE Set $m_g = 0$
\STATE Workers from group $g$ start computing stochastic gradients at $v^0_g$
\WHILE{True}
\STATE Gradient $\nabla f(v^{m_g-\delta^{m_g}}_g; \eta^{m_g}_{g})$ arrives from worker $i$ with delay $\delta^{m_g}$
\STATE Update: $v^{m_g + 1}_g = v^{m_g}_g - \gamma \nabla f(v^{m_g-\delta^{m_g}}_g; \eta^{m_g}_{g})$
\STATE Worker $i$ begins calculating stochastic gradient at $v^{m_g + 1}_g$
\STATE Update the iteration number $m_g = m_g + 1$
\ENDWHILE
\end{algorithmic}
\end{algorithm}
For this method, it is natural to take a main branch of the computation tree as
\begin{equation}
\begin{aligned}
\label{eq:OjRVuNSYZjFplZrfxBaN}
x^{1} &= x^{0} - \gamma \nabla f(v^{0-\delta^{0}}_1; \eta^{0}_{1}), \\
&\vdots \\
x^{m_1} &= x^{m_1 - 1} - \gamma \nabla f(v^{m_1 - 1 -\delta^{m_1 - 1}}_1; \eta^{m_1 - 1}_{1}),\\
&\vdots \\
x^{\sum_{g=1}^{s - 1} m_i + 1} &= x^{\sum_{g=1}^{s - 1} m_i} - \gamma \nabla f(v^{0-\delta^{0}}_s; \eta^{0}_{s}), \\
&\vdots, \\
x^{\sum_{g=1}^{s} m_i} &= x^{\sum_{g=1}^{s} m_i - 1} - \gamma \nabla f(v^{m_s - 1 -\delta^{m_s - 1}}_s; \eta^{m_s - 1}_{s}) \\
&\vdots,
\end{aligned}
\end{equation}
where one can see that $x^{\sum_{g=1}^{s} m_i} \equiv x^{B} \equiv w^1,$ and $\{v^{j}_{g}\}$ is defined in Algorithm~\ref{alg:local_sgd_worker_function_mix_sgd}.
\begin{figure}[H]
\centering
\includegraphics[page=16,width=0.8\textwidth]{./graphs/graphs_new.pdf}
\caption{An example of a \algname{Local-Async SGD} computation tree with two groups and $B = 5$. One group performs $m_1 = 3$ steps of \algname{Asynchronous SGD}, while the other performs $m_2 = 2$ steps. Note that the maximum tree distance is $\textnormal{dist}(x^4, v^{1 - \delta^1}_1)$ when applying $\nabla f(v^{1 - \delta^1}_1; \eta^1_1)$ to $x^4$, and it equals $B - 1 = m_1 + m_2 - 1 = 4$. Then, the groups synchronize and continue from $x^5.$}
\label{fig:local_sgd_worker_function_mix_sgd}
\end{figure}
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree of \algname{Local-Async SGD} (Alg.~\ref{alg:grouped_async_local_sgd}), then $\{x^k\}_{k \geq 0},$ defined in \eqref{eq:OjRVuNSYZjFplZrfxBaN}, is a main branch
and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{4 B L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\label{thm:grouped_async_local_sgd}
\end{theorem}
\begin{proof}
The proof closely follows that of Theorem~\ref{thm:local_sgd}, with the only difference being that the \emph{auxiliary branches} in Algorithm~\ref{alg:local_sgd_worker_function_mix_sgd} are constructed using asynchronous steps rather than local steps (compare Figure~\ref{fig:local_sgd_graph} and Figure~\ref{fig:local_sgd_worker_function_mix_sgd}). As in Theorem~\ref{thm:local_sgd}, the condition $\sum_{g=1}^{s} m_g = B$ ensures that $\sup_{k \geq 0} \textnormal{dist}(x^k, z^k) \leq B - 1.$
\end{proof}
\newpage
\subsection{\algname{Nested Local-Async SGD}}
\label{sec:hierarchy}
In this section, we formalize a hierarchical version of Algorithm~\ref{alg:grouped_async_local_sgd}. Our framework, Theorem~\ref{thm:main}, is flexible enough to support such a two-level structure, where each cluster consists of servers equipped with (4–8) GPUs. The GPUs run \algname{Asynchronous SGD}, the servers synchronize within their clusters, and finally, the clusters synchronize with each other.
In the following algorithm, all workers are partitioned into $\{G_{ij}\}$ groups, where $i$ is the cluster index and $j$ is the server index within the cluster. The set $G_{ij}$ contains the indices of the workers (GPUs).
\begin{algorithm}[H]
\caption{\algname{Nested Local-Async SGD}}
\label{alg:grouped_async_local_sgd_2}
\begin{algorithmic}[1]
\REQUIRE Initial model $w^0$, step size $\gamma$, parameters $B_i,$ global parameter $B,$ group partitions $\{G_{ij}\}$
\FOR{$k = 0, 1, 2, \dots$}
\STATE Broadcast $w^k$ to all clusters
\FOR{each cluster $i$ \textbf{in parallel}}
\STATE Set $w^{0}_i = w^k$
\FOR{$p_i = 0, 1, 2, \dots$}
\STATE Broadcast $w^{p_i}_i$ to all local groups
\FOR{each server $j$ \textbf{in parallel}}
\STATE Group $G_{ij}$ starts AsynchronousSGDGroup($w^{p_i}_i, \gamma$) from Algorithm~\ref{alg:local_sgd_worker_function_mix_sgd}
\ENDFOR
\STATE Cluster $i$ waits for the moment when $\sum_{j} m_{ij} = B_i$
\STATE Ask the groups in cluster $i$ to stop running AsynchronousSGDGroup($w^{p_i}_i, \gamma$)
\STATE Update $w^{p_i+1}_i = w^{p_i}_i - \gamma \sum_{j} \sum_{\ell=0}^{m_{ijp_i} - 1} \nabla f(v^{{\ell}-\delta^{{\ell}}}_{ijp_i}; \eta^{{\ell}}_{ijp_i})$
\ENDFOR
\ENDFOR
\STATE Wait for the moment the total number of local steps in the clusters starting from the last broadcast is $B$
\STATE Ask all groups in all servers to stop running AsynchronousSGDGroup($w^k, \gamma$)
\STATE Update $w^{k+1} = w^{k} - \sum_{i} (w^{p_i}_i - w^{0}_i) = w^{k} - \gamma \sum_{i} \sum_{k=0}^{p_i - 1} \sum_{j} \sum_{\ell=0}^{m_{ijk} - 1} \nabla f(v^{{\ell}-\delta^{{\ell}}}_{ijk}; \eta^{{\ell}}_{ijk})$
\ENDFOR
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[H]
\caption{AsynchronousSGDGroup($w, \gamma$) in group $G_{ij}$}
\label{alg:local_sgd_worker_function_mix_sgd_2}
\begin{algorithmic}
\STATE \textbf{Input:} point $v^0_{ijp_i} \in \R^d$, stepsize $\gamma > 0$
\STATE Set $m_{ij} = 0$
\STATE Workers from group $G_{ij}$ start computing stochastic gradients at $v^0_{ijp_i}$
\WHILE{True}
\STATE Gradient $\nabla f(v^{m_{ijp_i}-\delta^{m_{ijp_i}}}_{ijp_i}; \eta^{m_{ijp_i}}_{{ijp_i}})$ arrives from worker $i$ with delay $\delta^{m_{ijp_i}}$
\STATE Update: $v^{m_{ijp_i} + 1}_{ijp_i} = v^{m_{ijp_i}}_{ijp_i} - \gamma \nabla f(v^{m_{ijp_i}-\delta^{m_{ijp_i}}}_{ijp_i}; \eta^{m_{ijp_i}}_{{ijp_i}})$
\STATE Worker $i$ begins calculating stochastic gradient at $v^{m_{ijp_i} + 1}_{ijp_i}$
\STATE Update the iteration number $m_{ijp_i} = m_{ijp_i} + 1$
\ENDWHILE
\end{algorithmic}
\end{algorithm}
We believe that analyzing this algorithm directly using classical optimization tools would be challenging due to heavy notations. However, using our framework and geometrical graph reasoning, we can easily prove the iteration rate of this algorithm. As in all previous cases, a main branch ${x^k}$ can be defined by taking each component of the sum $\sum_{i} \sum_{k=0}^{p_i - 1} \sum_{j} \sum_{\ell=0}^{m_{ijk} - 1} \nabla f(v^{{\ell}-\delta^{{\ell}}}_{ijk}; \eta^{{\ell}}_{ijk})$ and applying each stochastic gradient to $x^0, x^{1} = x^0 - \gamma \nabla f(v^{0}_{110}; \eta^{0}_{110}),$ and so on.
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree of \algname{Nested Local-Async SGD} (Alg.~\ref{alg:grouped_async_local_sgd_2}), then $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{4 B L}, \frac{\varepsilon}{4 \sigma^2 L}\}$ for the main branch $\{x^k\}$ (slightly informally) defined above.
\label{thm:grouped_async_local_sgd_2}
\end{theorem}
\begin{proof}
Similarly to the previous proofs, Conditions 1 and 2 are satisfied by the construction of the algorithm. Using geometric graph reasoning, Condition 3 is satisfied with $R \leq B$ due to the requirement that ``the total number of local steps in the clusters starting from the last broadcast is $B$.'' This ensures that the distance between the points of the main branch and the corresponding points of the auxiliary sequence defined by $v^{\cdot}_{\cdot}$ does not exceed $B$.
\end{proof}
\begin{remark}
\label{remark:gpu}
One can see that the converge rate does not depend on $\{B_i\}.$ Theoretically, it is sufficient to take $B_i = \infty.$ However, practically, it may be better to take $B_i < \infty$ to ensure that the GPUs synchronize more often and share information with others, but it can lead to communication overhead and less efficient utilization of the GPUs.
\end{remark}
\newpage
\subsection{\algname{Meta Local SGD}}
\label{sec:adaptive_local_sgd}
Compared to previous algorithms, \algname{Meta Local SGD} is an abstract or meta-method, as it includes one abstract step: ``Wait if needed and take any set of workers~$S$.'' This step is not explicitly defined, allowing users to apply any strategy they prefer. It may be random, where the algorithm chooses a uniformly random subset; it may follow a condition as in \algname{Local SGD}, where the algorithm waits until $\sum_{i=1}^{n} M_i = B$; or it may be based on the current communication speeds of the workers, where the algorithm selects the workers with the fastest communication speeds at the current optimization moment.
However, as we explain in the main part, this can lead to a computation tree with a large~$R$. That is why we check the condition $\max_{j \in [n]} d_j + \sum_{i=1}^{n} M_i < B$ in the algorithm, where $\{d_i\}$ are the current distances to the head of the main branch and $\{M_i\}$ are the local steps performed by each worker.
If this condition is satisfied, we can take any set of workers~$S$. Otherwise, we find a set of workers $S = \{j \in [n]\,|\,d_j + \sum_{i=1}^{n} M_i = B\}$ and ask them to send their calculated stochastic gradients. The latter case is required to synchronize workers with ``very old stochastic gradients''. Intuitively, if we do not synchronize them, their stochastic gradients may become too outdated and harmful to the optimization process.
\begin{algorithm}[h]
\caption{\algname{Meta Local SGD}}
\label{alg:adaptive_local_sgd}
\begin{algorithmic}[1]
\STATE \textbf{Input:} point $w^0 \in \R^d$, stepsize $\gamma > 0,$ parameter $B \in \N$
\STATE Set an auxiliary distance variable $d_i = 0$ for all $i \in [n]$
\STATE Workers start running local steps at $w^0$ with Alg.~\ref{alg:adaptive_local_sgd_worker}
\FOR{$k = 0, 1, \dots$}
\IF{$\max_{j \in [n]} d_j + \sum_{i=1}^{n} M_i < B$}
\STATE {\color{mydarkgreen} Wait if needed and take any set of workers $S$ \hfill (Soft Sync)} \\
(Here, we do not specify the selection method, it could be random or based on the current communication speeds. One can choose any strategy.)
\ELSE
\STATE {\color{mydarkred} Find a set of workers $S = \{j \in [n]\,|\,d_j + \sum_{i=1}^{n} M_i = B\}$ \hfill (Hard Sync)}
\ENDIF
\STATE Ask workers from $S$ to send the calculated stochastic gradients and stop the loops in Alg.~\ref{alg:adaptive_local_sgd_worker}
\STATE Receive $\gamma \sum_{i \in S} \sum_{p = 0}^{M_i - 1} \nabla f(z^{p}_{i}; \eta^{p}_{i})$
\STATE Update: $w^{k+1} = w^{k} - \gamma \sum_{p = 0}^{M_i - 1} \nabla f(z^{p}_{i}; \eta^{p}_{i})$
\STATE Worker from $S$ start running local steps at $w^{k+1}$ with Alg.~\ref{alg:adaptive_local_sgd_worker}
\STATE Set $d_i = 0$ for all $i \in S$
\STATE Update $d_i = d_i + \sum_{j \in S} M_j$ for all $i \not\in S$
% \ENDIF
\ENDFOR
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[h]
\caption{\texttt{LocalSGDWorker}($w, \gamma$) in worker $i$}
\label{alg:adaptive_local_sgd_worker}
\begin{algorithmic}[1]
\STATE $z^{0}_i = w$
\STATE $M_i = 0$
\WHILE{True}
\STATE Calculate $\nabla f(z^{M_i}_i; \eta^{M_i}_i), \quad \eta^{M_i} \sim \mathcal{D}_{\xi}$
\IF{$\max_{j \in [n]} d_j + \sum_{i=1}^{n} M_i < B$}
\STATE $z^{M_i + 1}_i = z^{M_i}_i - \gamma \nabla f(z^{M_i}_i; \eta^{M_i}_i)$
\STATE $M_i = M_i + 1$
\ENDIF
\ENDWHILE
\end{algorithmic}
\end{algorithm}
A main branch in the computation tree can be defined as follows. Assume that $S^k = \{i_1, \dots, i_{p_k}\}$ is the set of workers participating in iteration~$k$. Then, the computation tree with the main branch $\{x^k\}$ can be constructed as
\begin{equation}
\begin{aligned}
x^{1} &= x^{0} - \gamma \nabla f(z^{0}_{i_1}; \eta^{0}_{i_1}), \\
x^{2} &= x^{1} - \gamma \nabla f(z^{1}_{i_1}; \eta^{1}_{i_1}), \\
&\vdots \\
x^{M_{i_1}} &= x^{M_{i_1} - 1} - \gamma \nabla f(z^{M_{i_1} - 1}_{i_1}; \eta^{M_{i_1} - 1}_{i_1}), \\
&\vdots \\
x^{\sum_{j = 1}^{p_k - 1} M_{i_j} + 1} &= x^{\sum_{j = 1}^{p_k - 1} M_{i_j}} - \gamma \nabla f(z^{0}_{i_{p_k}}; \eta^{0}_{i_{p_k}}), \\
&\vdots \\
x^{\sum_{j = 1}^{p_k} M_{i_j}} &= x^{\sum_{j = 1}^{p_k} M_{i_j} - 1} - \gamma \nabla f(z^{M_{i_{p_k}} - 1}_{i_{p_k}}; \eta^{M_{i_{p_k}} - 1}_{i_{p_k}}), \\
&\vdots
\label{eq:LHrbSkGWBtmHHQL}
\end{aligned}
\end{equation}
Notice that the end of each iteration block can be written as
\begin{align*}
w^1 \equiv x^{\sum_{j = 1}^{p_0} M_{i_j}}, \quad w^2 \equiv x^{\sum_{j = 1}^{p_0} M_{i_j} + \sum_{j = 1}^{p_1} M_{i_j}}, \quad \text{and so on}.
\end{align*}
\begin{figure}[h]
\centering
\includegraphics[page=17,width=0.6\textwidth]{./graphs/graphs_new.pdf}
\caption{An example of the computation tree for \algname{Meta Local SGD} with two workers. In this example, the first worker completes its first set of local steps, $x^0 \rightarrow z^{1,1}_2 \rightarrow z^{1,2}_2$, and sends the stochastic gradients, which are used to calculate $x^1$ and $x^2$. A similar sequence of steps is repeated by the first worker to produce $x^2 \rightarrow z^{2,1}_2 \rightarrow z^{2,2}_2$, followed by $x^3$ and $x^4$. At the same time, the second worker has only completed $x^0 \rightarrow z^{1,1}_1 \rightarrow z^{1,2}_1$ and has not yet synchronized or sent the corresponding stochastic gradients. At this moment in time, the number of local steps is $M_2 = 2$ and $d_2 = 4$, because $d_2$ is the number of edges between the current main branch head $x^4$ and the point $x^0$, where the local branch of the second worker started. At the same time, $M_1 = 0$ and $d_1 = 0$, because the first worker has just started the third set of local steps at $x^4$ and has not yet calculated local stochastic gradients.}
\end{figure}
\begin{theorem}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree of \algname{Meta Local SGD} (Alg.~\ref{alg:adaptive_local_sgd}), then $\{x^k\}_{k \geq 0},$ defined in \eqref{eq:LHrbSkGWBtmHHQL}, is a main branch
and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{4 B L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\end{theorem}
\begin{proof}
Similarly to the previous proofs, it is clear that Conditions 1 and 2 from Theorem~\ref{thm:main} are satisfied for the main branch~\eqref{eq:LHrbSkGWBtmHHQL}.
It remains to show that $\textnormal{dist}(x^k, z^k) \le B$ for all $k \geq 0$. In the algorithm, we track two key sets of variables: $\{d_i\}$ and $\{M_i\}$. The variable $M_i$ denotes the current number of local steps performed by worker~$i$, while $d_i$ represents the number of edges between the current end of the main branch and the point where worker~$i$ began its local updates. When worker $i \notin S$, the distance $d_i$ increases as follows: $d_i = d_i + \sum_{j \in S} M_j,$ since the workers in $S$ extend the main branch with their accumulated local updates.
The algorithm is constructed so that the quantity $\max_{j \in [n]} d_j + \sum_{i=1}^{n} M_i$ remains bounded by $B$ throughout the entire optimization process, ensuring that Condition~3 is satisfied with $R = B$. To clarify, assume that $i \in S$ in Algorithm~\ref{alg:adaptive_local_sgd}. In the worst-case scenario, all other workers $j \in S$, with $j \ne i$, apply their local updates, increasing the tree distance from worker~$i$’s branch to the main branch by at most $\sum_{j \in S, j \ne i} M_j$. Thus, the updated tree distance becomes at most $d_i + \sum_{j \in [n], j \ne i} M_j.$
Since worker~$i$ has also performed $M_i$ local steps, the tree distance is bounded by $d_i + \sum_{j \in [n], j \ne i} M_j + M_i \le B.$
\end{proof}
\newpage
\subsection{\algname{Dual-Process SGD}}
\label{sec:bi_local_sgd}
We now present a new method, \algname{Dual-Process SGD}, which is very similar to \algname{Local SGD}. In fact, when communication is free, the two methods are equivalent. However, \algname{Local SGD} requires all workers to send the sum of stochastic gradients only at the end of each round. In contrast, in \algname{Dual-Process SGD}, workers do not wait until the end of the round; instead, they begin communicating sequentially as soon as possible.
Initially, each worker waits for the first stochastic gradients with index $0$ and immediately sends them once available. Then, while these are being transmitted, the workers continue their local computations. After the server receives the gradients with index $0$, the workers begin sending the next batch of stochastic gradients, starting from index $1$ up to the latest index they have computed at that moment. This process continues until the server has received a total of $B$ stochastic gradients, accumulated through the communicated sums. This logic is implemented in Algorithm~\ref{alg:bi_local_sgd_worker_function}.
\begin{algorithm}[H]
\caption{\algname{Dual-Process SGD}}
\label{alg:bi_local_sgd}
\begin{algorithmic}[1]
\REQUIRE Initial model $w^0$, step size $\gamma$, parameter $B$
\FOR{$k = 0, 1, 2, \dots$}
\STATE Broadcast $w^k$ to all workers
\FOR{each worker $i \in [n]$ \textbf{in parallel}}
\STATE Worker $i$ starts \texttt{DualProcessLocalSGDWorker}($w^k, \gamma$) from Algorithm~\ref{alg:bi_local_sgd_worker_function}
\ENDFOR
\STATE Start receiving the sum from the workers
\STATE Wait for the moment when the total \# of received gradients $\sum_{i=1}^{n} M_i = B$
\STATE Ask workers to stop running \texttt{DualProcessLocalSGDWorker}($w^k, \gamma$)
\STATE Update $w^{k+1} = w^{k} - \gamma \sum_{i=1}^{n} \sum_{j=0}^{M_i - 1} \nabla f(z^{k,j}_i; \eta^{k,j}_i)$
\ENDFOR
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[H]
\caption{\texttt{DualProcessLocalSGDWorker}($w, \gamma$) in worker $i$ at round $k$}
\label{alg:bi_local_sgd_worker_function}
\begin{algorithmic}[1]
\STATE $z^{k,0}_i = w$
\STATE $\widetilde{M}_i = \bar{M}_i = M_i = 0$
\STATE \textbf{Launch in parallel the following two processes:}
\STATE \textbf{Process 1:}
\WHILE{True}
\STATE Calculate $\nabla f(z^{k,\widetilde{M}_i}_i; \eta^{k,\widetilde{M}_i}_i), \quad \eta^{k,\widetilde{M}_i}_i \sim \mathcal{D}_{\xi}$
\STATE $z^{k,\widetilde{M}_i + 1}_i = z^{k,\widetilde{M}_i}_i - \gamma \nabla f(z^{k,\widetilde{M}_i}_i; \eta^{k,\widetilde{M}_i}_i)$
\STATE $\widetilde{M}_i = \widetilde{M}_i + 1$
\ENDWHILE
\STATE
\STATE \textbf{Process 2:}
\WHILE{True}
\STATE Wait until at least one new stochastic gradient is computed in Process 1.
\STATE Set temporary variable $\bar{M}_i = \widetilde{M}_i$
\STATE Send $\sum\limits_{j = M_i}^{\bar{M}_i - 1} \nabla f(z^{k,j}_i; \eta^{k,j}_i)$
\STATE Wait until the transmission is complete
\STATE Update $M_i = \bar{M}_i$
\ENDWHILE
\end{algorithmic}
\end{algorithm}
The computation tree of \algname{Dual-Process SGD} defined in \eqref{eq:pbHgXapdjZGUQs} and \eqref{eq:yapgNwHcHepGaY} is similar to \algname{Local SGD}.
\begin{theorem}
\label{thm:bi_process}
Let Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} hold. Consider the computation tree (\eqref{eq:pbHgXapdjZGUQs} and \eqref{eq:yapgNwHcHepGaY}) of \algname{Dual-Process SGD}, then $\{x^k\}_{k \geq 0}$ is a main branch
and $\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for all
$$
K \geq \frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}.
$$
with step size $\gamma = \min\{\frac{1}{2 B L}, \frac{\varepsilon}{4 \sigma^2 L}\}.$
\end{theorem}
\begin{proof}
The proof is exactly the same as in Theorem~\ref{thm:local_sgd} since the computation tree of \algname{Dual-Process SGD} is similar to \algname{Local SGD}.
\end{proof}
\newpage
% \section{Derivation of the results from Table~\ref{tab:methods_summary_check}}
% \section{Computational Complexities of Algorithms}
\section{Computational Time Complexities of Algorithms under $h_i$-Fixed Computation Model}
\label{sec:proof_compt}
To compare methods, we consider the \emph{$h_i$-fixed computation model} \citep{mishchenko2022asynchronous}.
In this model, it is assumed that
\begin{equation}
\begin{aligned}
\text{worker } i \text{ takes no more than } h_i \text{ seconds} \text{ to compute a single stochastic gradient}
\label{eq:worker-time}
\end{aligned}
\end{equation}
and
\begin{equation}
\label{eq:sorted_t_i}
0<h_1\le h_2 \le \cdots \le h_n,
\end{equation}
without loss of generality.
Note that it is possible to consider the \emph{universal computation model} \citep{tyurin2024tight} and capture virtually all possible computation behaviors of the workers. While the \emph{$h_i$-fixed computation model} may seem more restrictive, it turns out that all optimal methods \citep{maranjyan2025ringmaster} in the \emph{universal computation model} are also optimal in the \emph{$h_i$-fixed computation model}. Thus, for simplicity, we stick to the \emph{$h_i$-fixed computation model}.
\subsection{\algname{Rennala SGD}}
\begin{theorem}[\algname{Rennala SGD}]
Consider Theorem~\ref{thm:rennala} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Rennala SGD} (Alg.~\ref{alg:rennala}) is
\begin{align}
\cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right)
\label{eq:MFvzCXtfgWSefbzZ}
\end{align}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:rennala_computation}
\end{theorem}
We start with the following lemma.
\begin{lemma}
Let us define
\begin{align}
\label{eq:batch_time}
T_{\textnormal{R}}(B) \eqdef 2 \min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + m)\right]
\end{align}
Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_{\textnormal{R}}(B)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_{\textnormal{R}}(B)$ seconds, and so on.
\label{lemma:rennala}
\end{lemma}
\begin{proof}
The idea of \algname{Rennala SGD} (Alg.~\ref{alg:rennala}) is pretty simple. Notice that all workers calculate stochastic gradients at the same point in parallel until the server collects a batch of size $B$ (condition $\delta = 0$ ensures that). Since they work in parallel, under the fixed computational model, after $t$ seconds the workers will calculate
\begin{align}
\sum_{i = 1}^n \max\left\{\flr{\frac{t}{h_i}} - 1, 0\right\}
\label{eq:KmmgS}
\end{align}
stochastic gradients because $\flr{\frac{t}{h_i}}$ is the number of stochastic gradients computed by worker $i$ in $t$ seconds. We subtract $1$ because at most one stochastic gradient be can be ignored due to the condition $\delta = 0$ in Alg.~\ref{alg:rennala}.
Notice that
\begin{align*}
T_{\textnormal{R}}(B) = 2 \left(\sum_{i=1}^{m^*} \frac{1}{h_i}\right)^{-1} (B + m^*)
\end{align*}
for some $m^* \in [n].$ Substituting it to \eqref{eq:KmmgS}, we get
\begin{align*}
\sum_{i = 1}^n \max\left\{\flr{\frac{T_{\textnormal{R}}(B)}{h_i}} - 1, 0\right\}
&\geq \sum_{i = 1}^{m^*} \max\left\{\flr{\frac{T_{\textnormal{R}}(B)}{h_i}} - 1, 0\right\} \geq \sum_{i = 1}^{m^*} \flr{\frac{T_{\textnormal{R}}(B)}{h_i}} - m^* \\
&\geq \sum_{i = 1}^{m^*} \frac{T_{\textnormal{R}}(B)}{h_i} - 2 m^* = 2 (B + m^*) - 2 m^* \geq B.
\end{align*}
Thus, after $T_{\textnormal{R}}(B)$ seconds, the server collects $B$ stochastic gradients, which is equivalent to calculating $x^{1}, \dots, x^{B}$ of the main branch. The same argument can be applied to the next $B$ point of the main branch, and so on.
\end{proof}
\begin{proof}[Proof of Theorem~\ref{thm:rennala_computation}]
Due to Theorem~\ref{thm:rennala}, we know that
$\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for
$$
K = \left\lceil\frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}\right\rceil.
$$
From Lemma~\ref{lemma:rennala}, we know that the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_{\textnormal{R}}(B)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_{\textnormal{R}}(B)$ seconds, and so on. Thus, the total time to find an $\varepsilon$--stationary point is
\begin{align*}
\cO\left(T_{\textnormal{R}}(B) \times \frac{K}{B}\right) = \cO\left(T_{\textnormal{R}}(B) \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{B \varepsilon^2}\right)\right).
\end{align*}
Using the choice of $B,$
\begin{align*}
\cO\left(T_{\textnormal{R}}(B) \times \frac{K}{B}\right)
&= \cO\left(\min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + m)\right] \times \frac{L \Delta}{\varepsilon}\right) \\
&= \cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right).
\end{align*}
\end{proof}
\subsection{\algname{Ringmaster ASGD}}
\begin{theorem}[\algname{Ringmaster ASGD}]
Consider Theorem~\ref{thm:ringmaster_asgd} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Ringmaster ASGD} is
\begin{align*}
\cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:ringmaster_computation}
\end{theorem}
We use Lemma~4.1 from \citep{maranjyan2025ringmaster}.
\begin{lemma}(\citep{maranjyan2025ringmaster})
\label{lem:time_R}
Let the workers' computation times satisfy the \emph{$h_i$-fixed computation model} (\eqref{eq:worker-time} and \eqref{eq:sorted_t_i}).
Let $B$ be the delay threshold of Alg.~\ref{alg:ringmasternew}.
The time required to complete any $B$ consecutive iterate updates of Alg.~\ref{alg:ringmasternew} is at most
\begin{equation}
\label{eq:time_R}
T_A(B) \eqdef 2 \min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^m \frac{1}{\tau_i}\right)^{-1} \left( 1 + \frac{R}{m} \right)\right].
\end{equation}
\end{lemma}
\begin{corollary}
In view of Lemma~\ref{lem:time_R}, Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_A(B)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_A(B)$ seconds, and so on.
\label{cor:time_R}
\end{corollary}
\begin{proof}[Proof of Theorem~\ref{thm:ringmaster_computation}]
The proof of Theorem~\ref{thm:ringmaster_computation} is similar to the proof of Theorem~\ref{thm:rennala_computation}. From Corollary~\ref{cor:time_R}, we know that the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_{\textnormal{A}}(B)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_{\textnormal{A}}(B)$ seconds, and so on. Thus, the total time to find an $\varepsilon$--stationary point is
\begin{align*}
\cO\left(T_{\textnormal{A}}(B) \times \frac{K}{B}\right) = \cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right).
\end{align*}
\end{proof}
\subsection{\algname{Local SGD}}
\begin{theorem}[\algname{Local SGD}]
Consider Theorem~\ref{thm:local_sgd} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Local SGD} (Alg.~\ref{alg:local_sgd}) is
\begin{align*}
\cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:local_computation}
\end{theorem}
\begin{lemma}
Let us define
\begin{align}
T_{\textnormal{L}}(B) \eqdef 2 \min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + m)\right]
\end{align}
Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_{\textnormal{L}}(B)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_{\textnormal{L}}(B)$ seconds, and so on.
\label{lemma:local_sgd}
\end{lemma}
\begin{proof}
The idea is the same as in Lemma~\ref{lemma:rennala}. All workers calculate stochastic gradients in parallel, with the only difference being that the points at which they compute the stochastic gradients differ due to the local steps. If the server can stop the workers, then after $t$ seconds it is possible to collect
\begin{align} \sum_{i=1}^{n} \flr{\frac{t}{h_i}} \label{eq:xpSINXLbaoIj} \end{align}
stochastic gradients. If it is infeasible to stop the calculations (see footnote~\ref{foot:local_sgd_stop}), then after $t$ seconds it is possible to collect
\begin{align} \sum_{i = 1}^n \max\left\{\flr{\frac{t}{h_i}} - 1, 0\right\}, \label{eq:xpSINXLbaoIj2} \end{align}
where we subtract $1$ because at most one stochastic gradient can be ignored if it is nonrelevant. Similarly to Lemma~\ref{lemma:rennala}, substituting $T_{\textnormal{L}}(B)$ into \eqref{eq:xpSINXLbaoIj} and \eqref{eq:xpSINXLbaoIj2}, one can show that $T_{\textnormal{L}}(B)$ is sufficient to collect $B = \sum_{i=1}^{n} M_i$ stochastic gradients, or, in other words, to calculate $x^{1}, \dots, x^{B}$ of the main branch. The same argument can be applied to the next $B$ points of the main branch, and so on.
\end{proof}
\begin{proof}[Proof of Theorem~\ref{thm:local_computation}]
The proof essentially the same as the proof of Theorem~\ref{thm:rennala_computation}.
\end{proof}
\subsection{\algname{Local-Async SGD}}
\begin{theorem}[\algname{Local-Async SGD}]
Consider Theorem~\ref{thm:grouped_async_local_sgd} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Local-Async SGD} (Alg.~\ref{alg:grouped_async_local_sgd}) is
\begin{align*}
\cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:grouped_async_local_sgd_comp}
\end{theorem}
\begin{lemma}
Let us define
\begin{align}
T_{\textnormal{LA}}(B) \eqdef 2 \min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + m)\right]
\end{align}
Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_{\textnormal{LA}}(B)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_{\textnormal{LA}}(B)$ seconds, and so on.
\label{lemma:grouped_async_local_sgd}
\end{lemma}
\begin{proof}
The idea is the same as in Lemmas~\ref{lemma:rennala} and \ref{lemma:local_sgd}. All workers calculate stochastic gradients in parallel, with the only difference being that the points at which they compute the stochastic gradients differ due to the asynchronous steps in the groups. Similarly, one can show that $T_{\textnormal{LA}}(B)$ is sufficient time to calculate $B = \sum_{g=1}^{s} m_g$ stochastic gradients in Algorithm~\ref{alg:grouped_async_local_sgd}, or, equivalently, to calculate $x^{1}, \dots, x^{B}$ of the main branch. The same argument can be applied to the next $B$ point of the main branch.
\end{proof}
\begin{proof}[Proof of Theorem~\ref{thm:grouped_async_local_sgd_comp}]
The proof essentially the same as the proof of Theorem~\ref{thm:rennala_computation}.
\end{proof}
\subsection{\algname{Nested Local-Async SGD}}
\begin{theorem}[\algname{Nested Local-Async SGD}]
Consider Theorem~\ref{thm:grouped_async_local_sgd_2} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Nested Local-Async SGD} (Alg.~\ref{alg:grouped_async_local_sgd_2}) is
\begin{align*}
\cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}$ and $B_i = \infty$\footnote{It is possible to take $B_i < \infty$, but the computational time complexity may decrease due to less utilization of workers. For simplicity, in this theorem, we take $B_i = \infty$. See also Remark~\ref{remark:gpu}.} for all $i \in [n].$
\end{theorem}
\begin{proof}
The proof essentially the same as the proof of Theorem~\ref{thm:rennala_computation}.
\end{proof}
\subsection{\algname{Async-Local SGD}}
\begin{theorem}[\algname{Async-Local SGD}]
Consider Theorem~\ref{thm:async_plus_local} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Async-Local SGD} (Alg.~\ref{alg:async_plus_local}) is
\begin{align*}
\cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}$ and $M = \max\left\{\left\lceil\frac{\sigma^2}{n \varepsilon}\right\rceil, 1\right\}.$
\label{thm:async_plus_local_comp}
\end{theorem}
\begin{lemma}
Let us define
\begin{align}
T_{\textnormal{AL}}(B,M) \eqdef 2 \min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + M m)\right]
\end{align}
Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_{\textnormal{AL}}(B,M)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_{\textnormal{AL}}(B,M)$ seconds, and so on.
\label{lemma:async_plus_local_comp}
\end{lemma}
\begin{proof}
Let us fix $B$ and $M \geq 1.$ Note that
\begin{align}
T_{\textnormal{AL}}(B,M) \eqdef 2 \min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + M m)\right] = 2 \left(\sum_{i=1}^{m^*} \frac{1}{h_i}\right)^{-1} (B + M m^*)
\label{eq:jyojpUFuuTnWRxYhSd}
\end{align}
for some $m^* \in [n],$ which depends on $B$ and $M.$
For any $k \geq 1$, consider the sequence $x^{k}, \dots, x^{k + B}$ on the main branch. Using a proof by contradiction, assume that it requires more than $T_{\textnormal{AL}}(B,M)$ seconds to calculate $x^{k + 1}, \dots, x^{k + B}.$ Thus, the algorithm can progress up to $x^{k + B - 1}$ after $T_{\textnormal{AL}}(B,M)$ seconds.
In Algorithm~\ref{alg:async_plus_local}, each worker computes $M$ stochastic gradients and sends their sum to the server. The server then performs the update $w^{k+1} = w^{k} - \gamma \sum_{p = 0}^{M - 1} \nabla f(z^{p}_{i_k}; \eta^{p}_{i_k}),$ which is equivalent to extending the main branch by $M$ points. Therefore, after $t$ seconds, the main branch will have progressed by at least
\begin{align}
\sum_{i=1}^{n} \max\left\{M \left\lfloor\frac{t}{M h_i}\right\rfloor - M, 0\right\},
\label{eq:pjvGDHlntkmUMbtMmnp}
\end{align}
points (which is less than $B$ by assumption). This is because worker $i$ requires at most $M h_i$ seconds to compute $M$ stochastic gradients before sending them to the server. Note that during any $B - 1$ consecutive updates on the main branch, the server may ignore $M$ gradients from each worker at most once, because $\delta^k$ can be $\geq B$ at most once during $B - 1$ consecutive updates. This explains the subtraction of $M$ in the formula.
Substituting $T_{\textnormal{AL}}(B,M)$ to \eqref{eq:pjvGDHlntkmUMbtMmnp},
\begin{align*}
&\sum_{i=1}^{n} \max\left\{M \left\lfloor\frac{T_{\textnormal{AL}}(B,M)}{M h_i}\right\rfloor - M, 0\right\} \geq \sum_{i=1}^{m^*} \max\left\{M \left\lfloor\frac{T_{\textnormal{AL}}(B,M)}{M h_i}\right\rfloor - M, 0\right\} \\
&\geq \sum_{i=1}^{m^*} M \left\lfloor\frac{T_{\textnormal{AL}}(B,M)}{M h_i}\right\rfloor - M m^* \geq \sum_{i=1}^{m^*} \frac{T_{\textnormal{AL}}(B,M)}{h_i} - 2 M m^*
\end{align*}
because $\flr{x} \geq x - 1$ for all $x \in \R.$ Using \eqref{eq:jyojpUFuuTnWRxYhSd},
\begin{align*}
&\sum_{i=1}^{n} \max\left\{M \left\lfloor\frac{T_{\textnormal{AL}}(B,M)}{M h_i}\right\rfloor - M, 0\right\} \geq 2 (B + M m^*) - 2 M m^* \geq B.
\end{align*}
Thus, after $T_{\textnormal{AL}}(B,M)$ seconds, the server collects $B$ stochastic gradients. It is equivalent to calculating $x^{k + 1}, \dots, x^{k + B},$ which contradicts the assumption.
\end{proof}
\begin{proof}[Proof of Theorem~\ref{thm:async_plus_local_comp}]
Due to Theorem~\ref{thm:async_plus_local}, we know that
$\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for
$$
K = \left\lceil\frac{4 (B + M - 1) L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}\right\rceil.
$$
From Lemma~\ref{lemma:rennala}, we know that the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $T_{\textnormal{AL}}(B,M)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $T_{\textnormal{AL}}(B,M)$ seconds, and so on. Thus, the total time to find an $\varepsilon$--stationary point is
\begin{align*}
\cO\left(T_{\textnormal{AL}}(B,M) \times \frac{K}{B}\right) = \cO\left(T_{\textnormal{AL}}(B,M) \times \left(\frac{L \Delta (B + M)}{B \varepsilon} + \frac{\sigma^2 L \Delta}{B \varepsilon^2}\right)\right).
\end{align*}
Using the choice of $B$ and $M,$ we obtain $M \leq B$ and
\begin{align*}
\cO\left(T_{\textnormal{R}}(B) \times \frac{K}{B}\right)
&= \cO\left(T_{\textnormal{AL}}(B,M) \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{B \varepsilon^2}\right)\right) \\
&= \cO\left(\min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + M m)\right] \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{B \varepsilon^2}\right)\right) \\
&= \cO\left(\min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} (B + M m)\right] \times \frac{L \Delta}{\varepsilon}\right)
\end{align*}
because $B \geq \frac{\sigma^2}{\varepsilon}.$ Since $M \leq \frac{\sigma^2}{n \varepsilon} + 1$ and $B \leq \frac{\sigma^2}{\varepsilon} + 1,$
\begin{align*}
\cO\left(T_{\textnormal{R}}(B) \times \frac{K}{B}\right)
&= \cO\left(\min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} \left(1 + \frac{\sigma^2}{\varepsilon} + m + \frac{m \sigma^2}{n \varepsilon}\right)\right] \times \frac{L \Delta}{\varepsilon}\right) \\
&= \cO\left(\min_{m\in[n]} \left[\left(\sum_{i=1}^m \frac{1}{h_i}\right)^{-1} \left(\frac{\sigma^2}{\varepsilon} + m\right)\right] \times \frac{L \Delta}{\varepsilon}\right) \\
&= \cO\left(\min_{m\in[n]} \left[\left(\frac{1}{m} \sum_{i=1}^m \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2} \right)\right]\right),
\end{align*}
where we use that $m \leq n$ for all $m \in [n].$
\end{proof}
\subsection{\algname{Cycle SGD}}
\begin{theorem}[\algname{Cycle SGD}]
Consider Theorem~\ref{thm:circular_local_sgd} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Cycle SGD} (Alg.~\ref{alg:circular_local_sgd}) is
\begin{align*}
\cO\left(\max\limits_{i \in [n]} h_i \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right)
\end{align*}
with $s = \min\left\{\max\left\{\ceil{\frac{n^2\varepsilon}{\sigma^2}}, 1\right\}, n\right\}.$
\label{thm:circular_local_sgd_computation}
\end{theorem}
\begin{proof}
According to Theorem~\ref{thm:circular_local_sgd},
$\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for
$$
K = \left\lceil\frac{8 n^2 L \Delta}{s \varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}\right\rceil.
$$
In the beginning, the algorithm has ``warm-up'', where, in the first iteration of the inner loop, the server collects $s$ stochastic gradients from $s$ workers, which is equivalent to calculating $x^{1}, \dots, x^{s}$ of the main branch. Then, the server collects $2 s$ stochastic gradients from the next group of $s$ workers because they calculated $s$ stochastic in the previous iteration. Starting from the $\ceil{\frac{n}{s}}$\textsuperscript{th} iteration, each group of $s$ workers will return $s \times \ceil{\frac{n}{s}}$ stochastic gradients in every subsequent iteration. Every iterations takes at most $\max\limits_{i \in [n]} h_i$ seconds, because they work in parallel and calculate one stochastic gradient.
Thus, the total time to calculate $x^1, \dots, x^K$ and find an $\varepsilon$--stationary point is
\begin{align*}
&\cO\left(\underbrace{\max\limits_{i \in [n]} h_i \times \ceil{\frac{n}{s}}}_{\textnormal{``warm-up'' phase}} + \max\limits_{i \in [n]} h_i \times \frac{K}{\left(s \times \ceil{\frac{n}{s}}\right)}\right) \\
&=\cO\left(\max\limits_{i \in [n]} h_i \times \frac{n}{s} + \max\limits_{i \in [n]} h_i \times \left(\frac{n L \Delta}{s \varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right) \\
&=\cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{n L \Delta}{s \varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right).
\end{align*}
because $\frac{L \Delta}{\varepsilon} \geq \frac{1}{2}$ without loss of generality (if $\frac{L \Delta}{\varepsilon} < \frac{1}{2},$ then $x^0$ is an $\varepsilon$--stationary point). Finally,
\begin{align*}
&\cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{n L \Delta}{s \varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right) = \cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\end{align*}
due to the choice of $s.$
% In every global iteration of Algorithm~\ref{alg:circular_local_sgd}, the server collects $s$ stochastic gradients from $s$ workers, which equivalently to calculating $x^{1}, \dots, x^{s}$ of the main branch. The worst-case time to $s$ stochastic gradients from $s$ workers is $\max\limits_{i \in [n]} h_i$ because they work in parallel. Similarly, it takes $\max\limits_{i \in [n]} h_i$ seconds to calculate $x^{s + 1}, \dots, x^{2 s},$ ans so on. Thus, the total time to find an $\varepsilon$--stationary point is
% \begin{align*}
% \cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{8 n^2 L \Delta}{s \varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}\right)\right)
% \end{align*}
\end{proof}
\subsection{\algname{Dual-Process SGD}}
\begin{theorem}[\algname{Dual-Process SGD}]
Consider Theorem~\ref{thm:bi_process} and its conditions. Under the $h_i$-fixed computation model \eqref{eq:worker-time}, the computational time complexity of \algname{Dual-Process SGD} (Alg.~\ref{alg:bi_local_sgd}) is
\begin{align*}
\cO\left(\min\limits_{m \in [n]} \left[\left(\frac{1}{m} \sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{m \varepsilon^2}\right)\right]\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:bi_local_computation}
\end{theorem}
\begin{proof}
The proof is essentially the same as the proof of Theorem~\ref{thm:local_computation} since \algname{Dual-Process SGD} is equivalent to \algname{Local SGD} if the communication times are ignored.
\end{proof}
\newpage
\section{Total Time Complexities of Algorithms under $(h,\tau)$-Fixed Computation Model}
\label{sec:proof_communication}
To compare the communication complexities and the total time complexities of the methods, we now assume that it takes $\tau$ seconds to send a vector from a worker to a parameter server and $\tau$ seconds to send a vector from the server to the workers in the centralized setting. Alternatively, it takes $\tau$ seconds to send a vector to all other workers in the decentralized setting. Moreover, we assume that all workers have the same computational performance: worker $i$ takes $h$ seconds to compute a single stochastic gradient for all $i \in [n]$. We refer to this as the \emph{$(h,\tau)$-fixed computation model}.
Note that it is possible to assume that each worker has its own communication time bound $\tau_i$ and computation time bound $h_i$ and consider \emph{$(h_i,\tau_i)$-fixed computation model} \citep{tyurin2024shadowheart}. However, for simplicity, we assume $\tau_i = \tau$ and $h_i = h$ for all $i \in [n].$ See Section~\ref{sec:proof_communication_general} for a more general case \emph{$(h_i,\tau_i)$-fixed computation model}.
% This assumption is sufficient for comparing the methods.
% In all these theorems, we only consider the total time complexity metric, which is formally a term w.r.t. $\tau$ in the total time complexity required to find $\varepsilon$--stationary point under \emph{$(h,\tau)$-fixed computation model}.
\subsection{\algname{Rennala SGD}}
\begin{theorem}
Consider Theorem~\ref{thm:rennala} and its conditions. Under $(h,\tau)$-fixed computation model, the total time complexity of \algname{Rennala SGD} (Alg.~\ref{alg:rennala}) is
\begin{align*}
\cO\left(\tau \times \frac{L \Delta}{\varepsilon} + h \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:rennala_comnunication}
\end{theorem}
\begin{proof}
Note that the communication between of vectors happens every $B$ calculated stochastic gradients, which is equivalent to every $B$ updates of the main branch. Thus the total number of communications is
\begin{align*}
\cO\left(\frac{K}{B}\right),
\end{align*}
where $K = \Theta\left(\frac{B L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{\varepsilon^2}\right)$ due to Theorem~\ref{thm:rennala}. The total communication complexity is
\begin{align*}
\cO\left(\tau \times \frac{K}{B}\right) = \cO\left(\frac{\tau}{B} \times \left(\frac{B L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{\varepsilon^2}\right)\right) = \cO\left(\tau \times \frac{L \Delta}{\varepsilon}\right),
\end{align*}
where we use the choice of $B.$ It left to take into account the computation factor, which is the same as in Theorem~\ref{thm:rennala_computation}.
\begin{align*}
\eqref{eq:MFvzCXtfgWSefbzZ} = \cO\left(h \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\end{align*}
under the $(h,\tau)$-fixed computation model.
\end{proof}
\subsection{\algname{Local SGD}}
\label{sec:local_sgd_time}
\begin{theorem}
Consider Theorem~\ref{thm:local_sgd} and its conditions. Under $(h,\tau)$-fixed computation model, the total time complexity of \algname{Local SGD} (Alg.~\ref{alg:local_sgd}) is
\begin{align*}
\cO\left(\tau \times \frac{L \Delta}{\varepsilon} + h \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:local_sgd_comnunication}
\end{theorem}
\begin{proof}
The proof essentially the same as the proof of Theorem~\ref{thm:rennala_comnunication}.
\end{proof}
\subsection{\algname{Cycle SGD}}
\begin{theorem}
Consider Theorem~\ref{thm:circular_local_sgd} and its conditions. Under $(h,\tau)$-fixed computation model, the total time complexity of \algname{Cycle SGD} (Alg.~\ref{alg:circular_local_sgd}) is
\begin{align*}
\cO\left(\tau \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right) + h \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\end{align*}
with $s = \min\left\{\max\left\{\ceil{\frac{n^2\varepsilon}{\sigma^2}}, 1\right\}, n\right\}.$
\end{theorem}
\begin{proof}
Similarly to the proof of Theorem~\ref{thm:circular_local_sgd_computation}, one can show that the total time complexity is
\begin{align*}
&\cO\left(\underbrace{(\tau + h) \times \ceil{\frac{n}{s}}}_{\textnormal{``warm-up'' phase}} + (\tau + h) \times \frac{K}{\left(s \times \ceil{\frac{n}{s}}\right)}\right)
% &=\cO\left(\max\limits_{i \in [n]} h_i \times \frac{n}{s} + \max\limits_{i \in [n]} h_i \times \left(\frac{n L \Delta}{s \varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right) \\
% &=\cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{n L \Delta}{s \varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right).
\end{align*}
because every worker from group $s$ sends one vector $\sum_{j=1}^{M_i} \nabla f(z^{j}_i; \eta^{j}_i)$ to the server in the inner loop.
% because $\frac{L \Delta}{\varepsilon} \geq \frac{1}{2}$ without loss of generality (if $\frac{L \Delta}{\varepsilon} < \frac{1}{2},$ then $x^0$ is an $\varepsilon$--stationary point). Finally,
% \begin{align*}
% &\cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{n L \Delta}{s \varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right) = \cO\left(\max\limits_{i \in [n]} h_i \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
% \end{align*}
% due to the choice of $s.$
Substituting the choice of $s,$ one can get the final result.
\end{proof}
\subsection{\algname{Async-Local SGD}}
\begin{theorem}
% [\algname{Async-Local SGD}]
\label{thm:async_local_comm}
Consider Theorem~\ref{thm:async_plus_local} and its conditions. Under $(h,\tau)$-fixed computation model, the total time complexity of \algname{Async-Local SGD} (Alg.~\ref{alg:async_plus_local}) is
\begin{align*}
\cO\left(\tau \times \frac{L \Delta}{\varepsilon} + h \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}$ and $M = \max\left\{\left\lceil\frac{\sigma^2}{n \varepsilon}\right\rceil, 1\right\}.$
\end{theorem}
\begin{proof}
Under $(h,\tau)$-fixed computation model, all workers send the sums of $M$ stochastic gradients at the same time. According to Theorem~\ref{thm:async_plus_local}, the server should collect
\begin{align*}
\cO\left(\frac{(B + M - 1) L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{\varepsilon^2}\right) = \cO\left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{\varepsilon^2}\right)
\end{align*}
stochastic gradients, where the last equality due to the choice of $B$ and due to $B \geq M.$ Since the workers work in parallel and have the equal performance, only $\Theta\left(\min\left\{\frac{B}{M}, n\right\}\right) = \Theta\left(\min\left\{\max\left\{1, \frac{\sigma^2}{M \varepsilon}\right\}, n\right\}\right)$ workers will participate in optimization. Thus, every worker, which participates in optimization, has to send
\begin{align*}
\cO\left(\frac{L \Delta}{\min\left\{\frac{B}{M}, n\right\} \varepsilon} + \frac{\sigma^2 L \Delta}{\min\left\{\frac{B}{M}, n\right\} \varepsilon^2}\right) = \cO\left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2} + \frac{M L \Delta}{\varepsilon}\right)
\end{align*}
stochastic gradients. Such a worker calculates $M$ stochastic gradients and only then sends the sum; thus, the maximum number of communications by one worker is
\begin{align*}
\cO\left(\frac{L \Delta}{M \varepsilon} + \frac{\sigma^2 L \Delta}{M n \varepsilon^2} + \frac{L \Delta}{\varepsilon}\right).
\end{align*}
For every communication, the worker needs to send $M$ stochastic gradients, which takes $h$ seconds, and sends a sum, which takes $\tau$ seconds. Thus, the total time complexity is
\begin{align}
\label{eq:uYeRvdpZTBNBlUowOqx}\cO\left(\left(\tau + M h\right) \left(\frac{L \Delta}{M \varepsilon} + \frac{\sigma^2 L \Delta}{M n \varepsilon^2} + \frac{L \Delta}{\varepsilon}\right)\right).
\end{align}
Substituting the choice of $M,$ we get the final result.
\end{proof}
\subsection{\algname{Ringmaster ASGD}}
\begin{theorem}
Consider Theorem~\ref{thm:ringmaster_asgd} and its conditions. Under $(h,\tau)$-fixed computation model, the total time complexity of \algname{Ringmaster ASGD} is
\begin{align}
\cO\left(\tau \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right) + h \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right)
\label{eq:SXtBColyTD}
\end{align}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\end{theorem}
\begin{proof}
The proof repeats the proof of Theorem~\ref{thm:async_local_comm}. The only difference is that the workers send $M = 1$ stochastic gradients. Substituting $M = 1$ to \eqref{eq:uYeRvdpZTBNBlUowOqx}, we get the final result.
\end{proof}
\begin{remark}
While \eqref{eq:SXtBColyTD} is only an upper bound, using the same steps as in the proof of Theorem~\ref{thm:async_local_comm}, one can easily show that the total time complexity of \algname{Ringmaster ASGD} is lower bounded by
\begin{align}
\Omega\left(\tau \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right) + h \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right),
\end{align}
assuming that the iteration rate $\Theta\left(\frac{B L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{\varepsilon^2}\right)$ from Theorem~\ref{thm:ringmaster_asgd} is tight. As far as we know, this is the current state-of-the-art iteration rate of an \algname{Asynchronous SGD}-like method \citep{maranjyan2025ringmaster,mishchenko2022asynchronous,koloskova2022sharper,cohen2021asynchronous}.
\end{remark}
\section{Comparison Between Our \algname{Local SGD} and the Canonical \algname{Local SGD}}
\label{sec:compare}
In this section, we show that our version of \algname{Local SGD} (Algorithm~\ref{alg:local_sgd}) achieves a better time complexity than the classical \algname{Local SGD}. Although we focus in this section only on \algname{Local SGD}, we expect similar improvements to extend to other new methods from Table~\ref{tab:methods_summary_check}. The purpose of this section is to highlight the tightness of the \algname{Birch SGD} framework, using \algname{Local SGD} as a case study.
In Section~\ref{sec:local_sgd_time}, we prove that our version of \algname{Local SGD} yields the total time complexity
\begin{align}
\label{eq:iDJXGXS}
\Theta\left(\tau \frac{L \Delta}{\varepsilon} + h \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{n \varepsilon^2}\right)\right).
\end{align}
We now illustrate that this result is provably better than the best theoretical result for the canonical version of \algname{Local SGD} (Algorithm~\ref{alg:local_sgd_old}) known to us.
\begin{algorithm}[H]
\caption{\algname{Local SGD} (\algname{FedAvg}) \citep{mcmahan2017communication}}
\label{alg:local_sgd_old}
\begin{algorithmic}[1]
\REQUIRE initial model $x^0$, step size $\gamma$, \# of local steps $K$
\FOR{$k = 0, 1, 2, \ldots$}
\STATE Broadcast $x^k$ to all workers
\FOR{each worker $i \in \{1,\ldots,n\}$ \textbf{in parallel}}
\STATE $z^{k,0}_{i} = x^k$
\FOR{$j = 0, \ldots, K - 1$}
\STATE $z^{k,j + 1}_{i} = z^{k,j}_{i} - \gamma \nabla f(z^{k,j}_{i}; \eta^{k,j}_{i})$
\ENDFOR
\ENDFOR
\STATE $x^{k+1} = \frac{1}{n} \sum_{i=1}^{n} z^{k,K}_{i}$
\ENDFOR
\end{algorithmic}
\end{algorithm}
To the best of our knowledge, the state-of-the-art analysis of Algorithm~\ref{alg:local_sgd_old} in the nonconvex setting is provided by \citet{koloskova2020local,luo2025revisiting}. Under Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded}, with a proper $\gamma,$ they establish the state-of-the-art iteration complexity
\begin{align*}
\Theta\left(\frac{L \Delta}{\varepsilon} + \frac{L\sigma^2 \Delta}{n K \varepsilon^2} + \frac{L \sigma \Delta }{K^{1/2} \varepsilon^{3/2}}\right)
\end{align*}
for finding an $\varepsilon$--stationary point for all $K \geq 1$. Next, under \emph{$(h,\tau)$-fixed computation model}, this iteration complexity yields the time complexity
\begin{align*}
\bar{T} \eqdef \tau \left(\frac{L \Delta}{\varepsilon} + \frac{L\sigma^2 \Delta}{n K \varepsilon^2} + \frac{L \sigma \Delta }{K^{\frac{1}{2}} \varepsilon^{\frac{3}{2}}}\right) + h K \left(\frac{L \Delta}{\varepsilon} + \frac{L\sigma^2 \Delta}{n K \varepsilon^2} + \frac{L \sigma \Delta }{K^{\frac{1}{2}} \varepsilon^{\frac{3}{2}}}\right)
\end{align*}
(up to constant factors) because in each iteration the workers communicate, which takes $\tau$ seconds, and each worker (in parallel) computes $K$ stochastic gradients, which takes $h \times K$ seconds. Ignoring non-negative terms,
\begin{align*}
\bar{T} \geq \tau \left(\frac{L \Delta}{\varepsilon} + \frac{L \sigma \Delta }{K^{\frac{1}{2}} \varepsilon^{\frac{3}{2}}}\right) + h \left(\frac{K L \Delta}{\varepsilon} + \frac{L\sigma^2 \Delta}{n \varepsilon^2} + \frac{K^{\frac{1}{2}} L \sigma \Delta }{\varepsilon^{\frac{3}{2}}}\right)
\end{align*}
and $\bar{T}$ is lower bounded by
\begin{align}
\label{eq:xqoybRsGbImqW}
\Theta\left(\sqrt{\tau h \frac{L^2 \sigma^2 \Delta^2}{\varepsilon^{3}}} + \tau \frac{L \Delta}{\varepsilon} + h \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L\Delta}{n \varepsilon^2}\right)\right)
\end{align}
for all $K \geq 1$ due to the AM-GM inequality. Notice that $\eqref{eq:iDJXGXS} \leq \eqref{eq:xqoybRsGbImqW}.$ However, $\eqref{eq:xqoybRsGbImqW}$ can be arbitrarily larger due to the first term. Indeed, for sufficiently large $n$, we have
\begin{align*}
\eqref{eq:iDJXGXS} = \Theta\left(\tau \frac{L \Delta}{\varepsilon} + h \left(\frac{L \Delta}{\varepsilon}\right)\right),
\end{align*}
while
\begin{align*}
\eqref{eq:xqoybRsGbImqW} = \Theta\left(\sqrt{\tau h \frac{L^2 \sigma^2 \Delta^2}{\varepsilon^{3}}} + \tau \frac{L \Delta}{\varepsilon} + h \left(\frac{L \Delta}{\varepsilon}\right)\right).
\end{align*}
Note that the latter expression has a $\nicefrac{1}{\varepsilon^{3/2}}$ dependency, whereas our result has a $\nicefrac{1}{\varepsilon}$ dependency. Thus, our result is provably tighter.
Note that we obtain the time complexity \eqref{eq:iDJXGXS} for several other new methods, including \algname{Async-Local SGD}, \algname{Async-Batch SGD}, and \algname{Dual-Process SGD}.
\section{Total Time Complexities of Algorithms under $(h_i,\tau_i)$-Fixed Computation Model}
\label{sec:proof_communication_general}
We now assume that each worker has its own communication time bound $\tau_i$ and computation time bound $h_i$ and consider \emph{the $(h_i,\tau_i)$-fixed computation model} \citep{tyurin2024shadowheart}. It takes $\tau_i$ seconds to send a vector from worker $i$ to a parameter server and $\tau_i$ seconds to send a vector from the server to worker $i$ in the centralized setting. Alternatively, it takes $\tau_i$ seconds to send a vector to all other workers in the decentralized setting.
This setting reduces to $h_i$-fixed computation model when $\tau_i = 0$ for all $i \in [n],$ and reduces to $(h,\tau)$-fixed computation model when $h_i = h$ and $\tau_i = \tau$ for all $i \in [n].$ Without loss of generality, we assume that $\max\{h_1, \tau_1\} \leq \dots \leq \max\{h_n, \tau_n\}.$ Otherwise, the workers can be sorted according to these inequalities.
Notice that \algname{Rennala SGD}, \algname{Local SGD}, and \algname{Cycle SGD} wait for the slowest worker by the designs. If $\max_{i \in [n]} \tau_i \to \infty,$ then their total complexity tends to $\infty.$ Thus, they are suboptimal under the $(h_i,\tau_i)$-fixed computation model. \algname{Ringmaster ASGD} is suboptimal even under the $(h,\tau)$-fixed computation model. \algname{Async-Local SGD} and \algname{Async-Batch SGD} are optimal under the $(h,\tau)$-fixed computation model, but we conjecture that they are suboptimal under the $(h_i,\tau_i)$-fixed computation model.
We now prove that \algname{Dual-Process SGD} is optimal under the $(h_i,\tau_i)$-fixed computation model within the family of methods that communicate either with a server (centralized setting) or with each other (decentralized setting).
\subsection{\algname{Dual-Process SGD}}
\begin{theorem}[\algname{Dual-Process SGD}]
Consider Theorem~\ref{thm:bi_process} and its conditions. Under the $(h_i, \tau_i)$-fixed computation model, the total time complexity of \algname{Dual-Process SGD} (Alg.~\ref{alg:bi_local_sgd}) is
\begin{align*}
\cO\left(\min\limits_{m \in [n]} \left[\max\left\{\max\{h_m, \tau_m\}, \left(\sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \frac{\sigma^2}{\varepsilon}\right\}\right] \frac{L \Delta}{\varepsilon}\right)
\end{align*}
with $B = \max\left\{\left\lceil\frac{\sigma^2}{\varepsilon}\right\rceil, 1\right\}.$
\label{thm:bi_comm}
\end{theorem}
This complexity is optimal for distributed methods without compression communicating with a server (centralized setting) or with each other (decentralized setting) \citep{tyurin2024shadowheart, tyurin2024optimalgraph}. Notice that it is robust to slow communications. Indeed, if $\tau_n \to \infty,$ then this complexity will ignore worker $n$ due to the $\min\limits_{m \in [n]}$ operation.
\begin{lemma}
\label{lemma:bi_comp}
Let us define
\begin{align}
T(B) \eqdef 4 \min\limits_{m \in [n]} \left[\max\left\{\max\{h_m, \tau_m\}, \left(\sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} B \right\}\right].
\end{align}
Under the $(h_i, \tau_i)$-fixed computation model, the time required to calculate $x^{1}, \dots, x^{B}$ of the main branch is at most $3 T(B)$ seconds, the time required to calculate $x^{B + 1}, \dots, x^{2B}$ is at most $3 T(B)$ seconds, and so on.
\end{lemma}
\begin{proof}
Notice that
\begin{align*}
T(B) = 4 \max\left\{\max\{h_{m^*}, \tau_{m^*}\}, \left(\sum\limits_{i=1}^{{m^*}} \frac{1}{h_i}\right)^{-1} B\right\}.
\end{align*}
for some $m^* \in [n].$
The idea is similar as in Lemmas~\ref{lemma:rennala} and \ref{lemma:local_sgd}. All workers calculate stochastic gradients in parallel. For all $t \geq 0,$ after $t$ seconds the first $m^*$ workers can calculate at at least
\begin{align}
\sum_{i = 1}^{m^*} \max\left\{\flr{\frac{t}{h_i}} - 1, 0\right\},
\label{eq:GKmoVZLNzkCeR}
\end{align}
stochastic gradients, where we subtract $1$ because at most one stochastic gradient can be ignored.
Substituting $T(B)$ into \eqref{eq:GKmoVZLNzkCeR}, we have
\begin{align*}
\sum_{i = 1}^{m^*} \max\left\{\flr{\frac{T(B)}{h_i}} - 1, 0\right\} \geq \sum_{i = 1}^{m^*} \flr{\frac{T(B)}{h_i}} - m^* \geq \sum_{i = 1}^{m^*} \frac{T(B)}{h_i} - 2 m^*.
\end{align*}
Recall that $\max\{h_1, \tau_1\} \leq \dots \leq \max\{h_{m^*}, \tau_{m^*}\}.$ Thus,
\begin{align*}
T(B) \geq 2 \max\{h_{m^*}, \tau_{m^*}\} + 2 \left(\sum\limits_{i=1}^{{m^*}} \frac{1}{h_i}\right)^{-1} B \geq 2 h_{i} + 2 \left(\sum\limits_{i=1}^{{m^*}} \frac{1}{h_i}\right)^{-1} B
\end{align*}
for all $i \leq m^*,$ and
\begin{align*}
\sum_{i = 1}^{m^*} \max\left\{\flr{\frac{T(B)}{h_i}} - 1, 0\right\}
\geq \sum_{i = 1}^{m^*} \left(2 + \frac{2}{h_i}\left(\sum\limits_{i=1}^{{m^*}} \frac{1}{h_i}\right)^{-1} B\right) - 2 m^* \geq B.
\end{align*}
Thus, by the time $T(B),$ the first $m^*$ workers can calculate $B$ stochastic gradients.
Next, we need to estimate the communication time. It takes at most $\max_{i \in [m^*]}\tau_{i} \leq \max\{h_{m^*}, \tau_{m^*}\} \leq T(B)$ seconds to receive a vector from the server (in the decentralized setting, we do not account this time). Similarly, it takes at most $\max_{i \in [m^*]}\tau_{i} \leq T(B)$ seconds to send a vector to the server (in the decentralized setting, to send a vector to other workers). Thus, one round in Alg~\ref{alg:bi_local_sgd} takes at most $3 \times T(B)$ seconds, which is equivalent to calculating $x^{1}, \dots, x^{B}$ of the main branch. The same argument can be applied to the next $B$ point of the main branch, and so on.
\end{proof}
\begin{proof}[Proof of Theorem~\ref{thm:bi_comm}]
The proof is similar to the proof of Lemma~\ref{lemma:rennala}. Due to Theorem~\ref{thm:rennala}, we know that
$\frac{1}{K}\sum_{k=0}^{K-1}\Exp{\|\nabla f(x^k)\|^2} \le \varepsilon$ for
$$
K = \left\lceil\frac{4 B L \Delta}{\varepsilon} + \frac{8 \sigma^2 L \Delta}{\varepsilon^2}\right\rceil.
$$
Using Lemma~\ref{lemma:bi_comp}, the total time to find an $\varepsilon$--stationary point is
\begin{align*}
\cO\left(T(B) \times \frac{K}{B}\right) = \cO\left(T(B) \times \left(\frac{L \Delta}{\varepsilon} + \frac{\sigma^2 L \Delta}{B \varepsilon^2}\right)\right).
\end{align*}
Using the choice of $B,$
\begin{align*}
\cO\left(T(B) \times \frac{K}{B}\right)
&= \cO\left(\min\limits_{m \in [n]} \left[\max\left\{\max\{h_m, \tau_m\}, \left(\sum\limits_{i=1}^{m} \frac{1}{h_i}\right)^{-1} \frac{\sigma^2}{\varepsilon} \right\}\right] \times \frac{L \Delta}{\varepsilon}\right).
\end{align*}
\end{proof}
\newpage
\section{Performance of \algname{Rennala SGD} and \algname{Ringmaster ASGD} on a Quadratic Function}
\label{sec:quadratic}
In this section, we formally prove that the convergence of \algname{Ringmaster ASGD} can be provably faster than \algname{Rennala SGD} due to the frequent model updates.
\begin{theorem}
Consider \algname{Rennala SGD} (Alg.~\ref{alg:rennala}) and \algname{Ringmaster ASGD} (Alg.~\ref{alg:ringmasternew}) with the optimal parameters $B$ from Sec.~\ref{sec:proof_compt}.
Then, there exists a $\mu$-strongly convex function and corresponding stochastic gradients that satisfy Assumptions~\ref{ass:lipschitz_constant}, \ref{ass:lower_bound}, and \ref{ass:stochastic_variance_bounded} with $\nicefrac{\sigma^2}{\varepsilon} \geq n$, such that \algname{Rennala SGD}, with \textbf{any step size} $\gamma,$ requires
\begin{align*} \widetilde{\Theta}\left({\color{red} \frac{\sigma^2}{n \varepsilon}} \times h \times \frac{L}{\mu}\right) \end{align*}
seconds to find $\varepsilon$-stationary point under the $h_i$-fixed computation model \eqref{eq:worker-time} with $h_i = h$ for all $i \in [n].$ At the same time, there \textbf{exists a step size} for \algname{Ringmaster ASGD} such that it requires at most
\begin{align*} \widetilde{\cO}\left(h \times \frac{L}{\mu} \right) \end{align*}
seconds to find $\varepsilon$-stationary point.
\end{theorem}
\begin{proof}
In this construction, we take $ f : \mathbb{R}^2 \to \mathbb{R} $ such that
\begin{align}
f(w \equiv (x, y)) = \frac{\mu x^2}{2} + \frac{L y^2}{2}
\label{eq:wyFdEHmlCZV}
\end{align}
for all $ x, y \in \mathbb{R} $. Moreover, we assume that the stochastic gradients $ \nabla f(w; \xi) $ are equal to the true gradient $ \nabla f(w) $; thus, there is no randomness. Note that \emph{a priori}, both methods do not have this information and therefore must choose $ B = \Theta\left(\frac{\sigma^2}{\varepsilon}\right),$ even though the effective variance is zero.
By the design of \algname{Rennala SGD}, its algorithm is equivalent to the following steps:
\begin{align}
w^{t+1} = w^{t} - \gamma B \nabla f(w^{t})
\label{eq:SkGZWJehKCRTC}
\end{align}
because the workers calculate $B$ gradients in every global round. Each round takes
\begin{align*}
\Theta\left(h \times \frac{B}{n}\right) = \Theta\left(h \frac{\sigma^2}{n \varepsilon}\right)
\end{align*}
seconds, because the workers have the computation speed $h$ and $B = \Theta\left(\frac{\sigma^2}{\varepsilon}\right)$ in Theorem~\ref{thm:rennala_computation}.
It is well known that the sequence~\eqref{eq:SkGZWJehKCRTC} requires
\begin{align*}
\widetilde{\Theta}\left(\frac{L}{\mu}\right)
\end{align*}
iterations (up to logarithmic factors) to find an $\varepsilon$-solution or $\varepsilon$-stationary point with the function \eqref{eq:wyFdEHmlCZV}, even when the step size $\gamma$ can be tuned. Thus, the computational time complexity of \algname{Rennala SGD} is
\begin{align*}
\widetilde{\Theta}\left(\frac{\sigma^2}{n \varepsilon} \times h \times \frac{L}{\mu}\right)
\end{align*}
seconds.
Consider now the steps of \algname{Ringmaster ASGD} w.r.t. the first argument $x$. In this algorithm, we take $\gamma = \frac{1}{2 L n}.$ In the case when the computation time is equal for all workers, the first $n$ steps are
\begin{align*}
&x^1 = x^0 - \gamma \mu x^0 = (1 - \gamma \mu) x^0, \\
&x^2 = x^1 - \gamma \mu x^0 = (1 - 2 \gamma \mu) x^0, \\
&\vdots \\
&x^n = x^{n - 1} - \gamma \mu x^0 = (1 - n \gamma \mu) x^0,
\end{align*}
because the workers start calculating at the same point and return the gradients at the same time. Notice that $0 \leq x^n \leq \dots \leq x^2 \leq x^1.$ Then, the first worker starts calculating at $x^1,$ the seconds worker starts calculating at $x^2,$ and so on. Therefore, the next steps are
\begin{align*}
x^{n + 1} &= x^n - \gamma \mu x^{1} \\
&= (1 - n \gamma \mu) x^0 - \gamma \mu (1 - \gamma \mu) x^0 = (1 - (n + 1) \gamma \mu + \gamma^2 \mu^2) x^0, \\
x^{n + 2} &= x^{n+1} - \gamma \mu x^{2} \\
&= (1 - (n + 1) \gamma \mu + \gamma^2 \mu^2) x^0 - \gamma \mu (1 - 2 \gamma \mu) x^0 = (1 - (n + 2) \gamma \mu + 3 \gamma^2 \mu^2) x^0, \\
&\vdots \\
x^{2 n} &= x^{2 n - 1} - \gamma \mu x^{n} = \left(1 - 2 n \gamma \mu + \frac{n(n+1)}{2} \gamma^2 \mu^2\right) x^0 \leq \left(1 - n \gamma \mu\right)^2 x^0.
% &= \left((1 - n \gamma \mu)^2 - n^2 \gamma^2 \mu^2 + \frac{n(n+1)}{2} \gamma^2 \mu^2\right) x^0.
\end{align*}
For $\gamma = \nicefrac{1}{2 L n},$ we have $0 \leq x^{2n} \leq \dots \leq x^{n + 1} \leq x^n \leq \dots \leq x^2 \leq x^1.$
Using mathematical induction, assume that $0 \leq x^{k n} \leq \dots \leq x^1$ for some $k \geq 1$ and $x^{p n} \leq \left(1 - n \gamma \mu\right)^p x^0$ for all $p \leq k,$ which is true for $k = 2$ (base case). We now prove it for $k + 1.$ \algname{Ringmaster ASGD} calculates $x^{k n + 1}$ as follows:
\begin{align*}
x^{k n + 1} &= x^{k n} - \gamma \mu x^{(k - 1) n + 1},
\end{align*}
which ensures that $x^{k n + 1} \leq (1 - \gamma \mu) x^{k n} \leq x^{k n}$ and $x^{k n + 1} \geq x^{(k - 1) n + 1} - \gamma \mu x^{(k - 1) n + 1} \geq 0$ for $\gamma = \nicefrac{1}{2 L n}.$ We can continue:
\begin{align*}
x^{k n + 2} &= x^{k n + 1} - \gamma \mu x^{(k - 1) n + 2},
\end{align*}
which ensures that
\begin{align*}
x^{k n + 2} &\leq x^{k n + 1} - \gamma \mu x^{k n} \leq (1 - \gamma \mu) x^{k n} - \gamma \mu x^{k n} = (1 - 2 \gamma \mu) x^{k n}
\end{align*}
and $x^{k n + 2} \geq x^{(k - 1) n + 2} - \gamma \mu x^{(k - 1) n + 2}.$ Continuing, we have
\begin{align*}
x^{(k + 1) n} = x^{(k + 1) n - 1} - \gamma \mu x^{k n}.
\end{align*}
One can show that
\begin{align*}
x^{(k + 1) n} \leq (1 - (n - 1) \gamma \mu) x^{k n} - \gamma \mu x^{k n} \leq (1 - n \gamma \mu) x^{k n},
\end{align*}
and $x^{(k + 1) n} \geq x^{k n} - \gamma \mu x^{k n} \geq 0.$ We have proved the next case, $k + 1,$ of the mathematical induction.
Thus, the sequence $\{x^{p n}\}_{p \geq 2}$ monotonically decreases with the rate
\begin{align*}
x^{p n} \leq \left(1 - n \gamma \mu\right)^p x^0.
\end{align*}
Using the same reasoning, we one can show the similar result holds for the second argument $y$ of the function but with $L$ instead of $\mu.$
Recall that it takes $h$ seconds to calculate $x^{n}$ because $n$ workers work in parallel, it takes $h$ seconds to calculate $x^{2 n},$ and so on. Thus, the computational time complexity of \algname{Ringmaster ASGD} is
\begin{align*}
\widetilde{\cO}\left(h \times \frac{L}{\mu}\right)
\end{align*}
with step size $\gamma = \frac{1}{2 L n}.$
% In general, for all $1 \leq j \leq n,$ we have
% \begin{align*}
% x^{k n + j} &= x^{k n + j - 1} - \gamma \mu x^{(k - 1) n + j},
% \end{align*}
% \begin{align*}
% x^{p n + j} = \left(1 - (p n + j) \gamma \mu + \sum_{i=1}^{p} (-1)^{i+1}\frac{((i - 1) n + j)((i - 1) n + j +1)}{2} \gamma^{i + 1} \mu^{i + 1}\right) x^0
% \end{align*}
% for all $0 \leq p \leq k - 1$ and $1 \leq j \leq n.$ We now prove that it holds for $p = k$ and $1 \leq j \leq n.$ The base cases have been verified for $p = 1$ and $p = 2.$ Next, we have
% \begin{align*}
% x^{k n + 1} &= x^{k n + 1} - \gamma \mu x^{(k - 1) n + 1} \\
% &= \left(1 - (k n) \gamma \mu + \sum_{i=1}^{k} (-1)^{i+1} \frac{((i - 1) n)((i - 1) n +1)}{2} \gamma^{i + 1} \mu^{i + 1}\right) x^0 \\
% &\quad - \gamma \mu \left(1 - ((k - 1) n + 1) \gamma \mu + \sum_{i=1}^{k - 1} (-1)^{i+1} \frac{((i - 1) n + 1)((i - 1) n + 2)}{2} \gamma^{i + 1} \mu^{i + 1}\right) x^0 \\
% &= \left(1 - (k n + 1) \gamma \mu + \sum_{i=1}^{k} (-1)^{i+1} \frac{((i - 1) n)((i - 1) n +1)}{2} \gamma^{i + 1} \mu^{i + 1}\right) x^0 \\
% &\quad + \left(((k - 1) n + 1) \gamma^2 \mu^2 + \sum_{i=1}^{k - 1} (-1)^{i+2}\frac{((i - 1) n + 1)((i - 1) n + 2)}{2} \gamma^{i + 2} \mu^{i + 2}\right) x^0 \\
% \end{align*}
% \begin{align*}
% x^{2 k n + 1} &= 2 k n - \gamma \mu x^{1} \\
% \end{align*}
\end{proof}
\end{document}
\usepackage{graphicx}
\usepackage{apptools}
\usepackage[flushleft]{threeparttable}
\usepackage{array,booktabs,makecell}
\usepackage{multirow}
\usepackage{nicefrac}
\usepackage{amsmath,amsfonts,bm,amssymb}
\usepackage{wrapfig}
\usepackage{caption}
\usepackage{siunitx}
\usepackage{nccmath}
\usepackage{empheq}
\usepackage{bbm}
\usepackage{tabularx}
\usepackage{thm-restate}
\usepackage{algorithmic}
\usepackage{algorithm}
% \usepackage[capitalize,noabbrev]{cleveref}
% COLORS
\usepackage{color}
\usepackage{colortbl}
\definecolor{bgcolor}{rgb}{0.76,0.88,0.50}
\definecolor{bgcolor0}{rgb}{0.93,0.99,1}
\definecolor{bgcolor1}{rgb}{0.8,1,1}
\definecolor{bgcolor2}{rgb}{0.8,1,0.8}
\definecolor{bgcolor3}{rgb}{0.50,0.90,0.50}
\usepackage{tcolorbox}
\usepackage{pifont}
\definecolor{mydarkgreen}{RGB}{39,130,67}
\definecolor{mydarkorange}{RGB}{236,147,14}
\definecolor{mydarkred}{RGB}{192,47,25}
\definecolor{ruby}{RGB}{155,17,30}
\definecolor{chili}{RGB}{191,0,0}
\definecolor{sangria}{RGB}{146,0,10}
\definecolor{burgundy}{RGB}{128,0,32}
\definecolor{darkred}{RGB}{132,0,0}
\definecolor{cherry}{RGB}{192,0,0}
\definecolor{grey}{RGB}{80,80,80}
\definecolor{blue}{RGB}{0,0,255}
\newcommand{\green}{\color{mydarkgreen}}
\newcommand{\orange}{\color{mydarkorange}}
\newcommand{\blue}{\color{blue}}
\newcommand{\red}{\color{cherry}}
\newcommand{\markchanges}{}
\usepackage[textsize=tiny]{todonotes}
\usepackage{xspace}
\newcommand{\dr}[1]{{{\sf\red #1}}\xspace}
% \newcommand{\algname}[1]{{{\sf\footnotesize #1}}\xspace}
\usepackage[scaled=0.86]{helvet}
\newcommand{\algname}[1]{{\sf #1}}
\newcommand{\algnamebf}[1]{{\sffamily\bfseries #1}}
\newcommand{\dataname}[1]{{\tt\footnotesize\color{blue}#1}\xspace}
\newcommand{\ind}{\perp\!\!\!\!\perp}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\sqnorm}[1]{\left\| #1 \right\|^2}
\newcommand{\lin}[1]{\left\langle #1\right\rangle} % inner product
\newcommand{\inp}[2]{\left\langle#1,#2\right\rangle} % inner product
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\R}{\mathbb{R}} % reals
\newcommand{\N}{\mathbb{N}} % reals
\newcommand{\E}[1]{\mathbb{E}\left[#1\right]}
\newcommand{\Med}[1]{\mathrm{Med}\left[#1\right]}
\newcommand{\Var}[1]{\mathrm{Var}\left(#1\right)}
\newcommand{\Exp}[1]{{\mathbb{E}}\left[#1\right]}
\newcommand{\ExpSub}[2]{{\mathbb{E}}_{#1}\left[#2\right]}
\newcommand{\ExpCond}[2]{{\mathbb{E}}\left[\left.#1\right\vert#2\right]}
\newcommand{\Prob}[1]{\mathbb{P}\left(#1\right)} % probability
\newcommand{\ProbCond}[2]{\mathbb{P}\left(#1\middle\vert#2\right)}
% caligraphic
\newcommand{\cA}{\mathcal{A}}
\newcommand{\cB}{\mathcal{B}}
\newcommand{\cC}{\mathcal{C}}
\newcommand{\cD}{\mathcal{D}}
\newcommand{\cE}{\mathcal{E}}
\newcommand{\cF}{\mathcal{F}}
\newcommand{\cG}{\mathcal{G}}
\newcommand{\cH}{\mathcal{H}}
\newcommand{\cI}{\mathcal{I}}
\newcommand{\cJ}{\mathcal{J}}
\newcommand{\cK}{\mathcal{K}}
\newcommand{\cL}{\mathcal{L}}
\newcommand{\cM}{\mathcal{M}}
\newcommand{\cN}{\mathcal{N}}
\newcommand{\cO}{\mathcal{O}}
\newcommand{\cP}{\mathcal{P}}
\newcommand{\cQ}{\mathcal{Q}}
\newcommand{\cR}{\mathcal{R}}
\newcommand{\cS}{\mathcal{S}}
\newcommand{\cT}{\mathcal{T}}
\newcommand{\cU}{\mathcal{U}}
\newcommand{\cV}{\mathcal{V}}
\newcommand{\cW}{\mathcal{W}}
\newcommand{\cX}{\mathcal{X}}
\newcommand{\cY}{\mathcal{Y}}
\newcommand{\cZ}{\mathcal{Z}}
% bold matrices
\newcommand{\mA}{\mathbf{A}}
\newcommand{\mX}{\mathbf{X}}
\newcommand{\mM}{\mathbf{M}}
\newcommand{\mN}{\mathbf{N}}
\newcommand{\mI}{\mathbf{I}}
\newcommand{\mJ}{\mathbf{J}}
\newcommand{\mL}{\mathbf{L}}
\newcommand{\mD}{\mathbf{D}}
\newcommand{\mO}{\mathbf{O}}
\newcommand{\eqdef}{:=}
\newcommand{\Mod}[1]{\ (\mathrm{mod}\ #1)}
\makeatletter
\newcommand{\vast}{\bBigg@{4}}
\DeclareMathOperator*{\argmin}{arg\,min}
\def\<{\left\langle}
\def\>{\right\rangle}
\def\[{\left[}
\def\]{\right]}
\def\({\left(}
\def\){\right)}
\newcommand{\flr}[1]{\left\lfloor #1\right\rfloor}
\newcommand{\ceil}[1]{\left\lceil #1\right\rceil}
\usepackage{thmtools}
\usepackage{longtable}