Linear Convergent Decentralized Optimization with Compression
Extracted source text · arXiv 2007.00232
\documentclass{article} % For LaTeX2e
\usepackage{iclr2021_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\input{macros.tex}
\usepackage{etoc}
\usepackage{hyperref}
\usepackage{url}
%% My
\usepackage{amsmath}
\usepackage{algpseudocode} %not compatible with algorithmic
\usepackage{algorithm}
\usepackage{comment}
\usepackage{setspace}
\usepackage{empheq}
\usepackage{euscript}
% \usepackage{caption}
\usepackage{enumitem}
\usepackage{subfig}
\usepackage{tikz}
\usetikzlibrary{cd}
\title{Linear Convergent Decentralized Optimization with Compression}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{
Xiaorui Liu$^1$, Yao Li$^{2,3}$, Rongrong Wang$^{3,2}$, Jiliang Tang$^1$ \& Ming Yan$^{3,2}$ \\
~$^1$ Department of Computer Science and Engineering \\
~$^2$ Department of Mathematics \\
~$^3$ Department of Computational Mathematics, Science and Engineering \\
~Michigan State University, East Lansing, MI 48823, USA \\
~\texttt{\{xiaorui,liyao6,wongron6,tangjili,myan\}@msu.edu}
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
% \newcommand{\eqref}{(\ref{#1})}
\renewcommand{\eqref}[1]{(\ref{#1})}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\etocdepthtag.toc{mtchapter}
\etocsettagdepth{mtchapter}{subsection}
\etocsettagdepth{mtappendix}{none}
\begin{abstract}
Communication compression has become a key strategy to speed up distributed optimization. However, existing decentralized algorithms with compression mainly focus on compressing DGD-type algorithms. They are unsatisfactory in terms of convergence rate, stability, and the capability to handle heterogeneous data. Motivated by primal-dual algorithms, this paper proposes the first \underline{L}in\underline{EA}r convergent \underline{D}ecentralized algorithm with compression, LEAD. Our theory describes the coupled dynamics of the inexact primal and dual update as well as compression error, and we provide the first consensus error bound in such settings without assuming bounded gradients. Experiments on convex problems validate our theoretical analysis, and empirical study on deep neural nets shows that LEAD is applicable to non-convex problems.
\end{abstract}
\input{section/intro}
\input{section/relate}
\input{section/algorithm}
\input{section/theory}
\input{section/experiment}
\input{section/conclusion}
\section*{Acknowledgements}
Xiaorui Liu and Dr. Jiliang Tang are supported by the National Science Foundation (NSF) under grant numbers CNS-1815636, IIS-1928278, IIS-1714741, IIS-1845081, IIS-1907704, and IIS-1955285. Yao Li and Dr. Ming Yan are supported by NSF grant DMS-2012439 and Facebook Faculty Research Award (Systems for ML). Dr. Rongrong Wang is supported by NSF grant CCF-1909523.
\bibliography{section/ref}
\bibliographystyle{iclr2021_conference}
\newpage
\appendix
\input{section/appendix/supp-expe}
\input{section/appendix/supp-theory}
% \input{section/appendix/supp-theory-2}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
%% macros for commenting
\usepackage{mathtools}
%\usepackage[normalem]{ulem} % to use \sout
%% math packages
% \usepackage{amsmath} % not needed for Springer
%\usepackage{amsmath}
\usepackage{amssymb}
%\usepackage{mathrsfs}
\usepackage{amsthm}
\usepackage{color}
%\usepackage{subfigure}
%\usepackage{url}
%% a light-weight algorithm environment
\newtheorem{algo}{Algorithm}
\newcommand{\overbar}[1]{\mkern 1.5mu\overline{\mkern-1.5mu#1\mkern-1.5mu}\mkern 1.5mu}
%% highlighting and commenting
\newcommand{\outline}[1]{{\color{brown}#1}}
\newcommand{\rev}[1]{{\color{blue}#1}}
\newcommand{\remove}[1]{{\sout{#1}}}
\newcommand{\cut}[1]{{}}
\newcommand{\my}[1]{{\color{red}(#1 -- Ming)}} % comments
\newcommand{\commwy}[1]{{\color{red}(#1 -- Wotao)}} % comments
\newcommand{\commrw}[1]{{\color{red}(#1 -- Reviewer)}} % comments
% \newcommand{\xr}[1]{{\color{red}(#1 --- Xiaorui)}} % comments
\newcommand{\xr}[1]{{\color{red}(#1)}} % comments
\newcommand{\yl}[1]{{\color{blue}(#1 --- Yao)}} % comments
\newcommand{\wrr}[1]{{\color{blue}(#1 --- Rongrong)}} % comments
%% macros for letters
%
%\newcommand{\ta}{{\tilde{a}}}
%\newcommand{\tb}{{\tilde{b}}}
%\newcommand{\tc}{{\tilde{c}}}
%\newcommand{\td}{{\tilde{d}}}
%\newcommand{\te}{{\tilde{e}}}
%\newcommand{\tf}{{\tilde{f}}}
%\newcommand{\tg}{{\tilde{g}}}
%\newcommand{\th}{{\tilde{h}}}
%\newcommand{\ti}{{\tilde{i}}}
%\newcommand{\tj}{{\tilde{j}}}
%\newcommand{\tk}{{\tilde{k}}}
%\newcommand{\tl}{{\tilde{l}}}
%\newcommand{\tm}{{\tilde{m}}}
%\newcommand{\tn}{{\tilde{n}}}
%\newcommand{\to}{{\tilde{o}}}
%\newcommand{\tp}{{\tilde{p}}}
%\newcommand{\tq}{{\tilde{q}}}
%\newcommand{\tr}{{\tilde{r}}}
%\newcommand{\ts}{{\tilde{s}}}
%\newcommand{\tt}{{\tilde{t}}}
%\newcommand{\tu}{{\tilde{u}}}
%\newcommand{\tv}{{\tilde{v}}}
%\newcommand{\tw}{{\tilde{w}}}
%\newcommand{\tx}{{\tilde{x}}}
%\newcommand{\ty}{{\tilde{y}}}
%\newcommand{\tz}{{\tilde{z}}}
\newcommand{\vA}{{\mathbf{A}}}
\newcommand{\vB}{{\mathbf{B}}}
\newcommand{\vC}{{\mathbf{C}}}
\newcommand{\vD}{{\mathbf{D}}}
\newcommand{\vE}{{\mathbf{E}}}
\newcommand{\vF}{{\mathbf{F}}}
\newcommand{\vG}{{\mathbf{G}}}
\newcommand{\vH}{{\mathbf{H}}}
\newcommand{\vI}{{\mathbf{I}}}
\newcommand{\vJ}{{\mathbf{J}}}
\newcommand{\vK}{{\mathbf{K}}}
\newcommand{\vL}{{\mathbf{L}}}
\newcommand{\vM}{{\mathbf{M}}}
\newcommand{\vN}{{\mathbf{N}}}
\newcommand{\vO}{{\mathbf{O}}}
\newcommand{\vP}{{\mathbf{P}}}
\newcommand{\vQ}{{\mathbf{Q}}}
\newcommand{\vR}{{\mathbf{R}}}
\newcommand{\vS}{{\mathbf{S}}}
\newcommand{\vT}{{\mathbf{T}}}
\newcommand{\vU}{{\mathbf{U}}}
\newcommand{\vV}{{\mathbf{V}}}
\newcommand{\vW}{{\mathbf{W}}}
\newcommand{\vX}{{\mathbf{X}}}
\newcommand{\vY}{{\mathbf{Y}}}
\newcommand{\vZ}{{\mathbf{Z}}}
\newcommand{\cA}{{\mathcal{A}}}
\newcommand{\cB}{{\mathcal{B}}}
\newcommand{\cC}{{\mathcal{C}}}
\newcommand{\cD}{{\mathcal{D}}}
\newcommand{\cE}{{\mathcal{E}}}
\newcommand{\cF}{{\mathcal{F}}}
\newcommand{\cG}{{\mathcal{G}}}
\newcommand{\cH}{{\mathcal{H}}}
\newcommand{\cI}{{\mathcal{I}}}
\newcommand{\cJ}{{\mathcal{J}}}
\newcommand{\cK}{{\mathcal{K}}}
\newcommand{\cL}{{\mathcal{L}}}
\newcommand{\cM}{{\mathcal{M}}}
\newcommand{\cN}{{\mathcal{N}}}
\newcommand{\cO}{{\mathcal{O}}}
\newcommand{\cP}{{\mathcal{P}}}
\newcommand{\cQ}{{\mathcal{Q}}}
\newcommand{\cR}{{\mathcal{R}}}
\newcommand{\cS}{{\mathcal{S}}}
\newcommand{\cT}{{\mathcal{T}}}
\newcommand{\cU}{{\mathcal{U}}}
\newcommand{\cV}{{\mathcal{V}}}
\newcommand{\cW}{{\mathcal{W}}}
\newcommand{\cX}{{\mathcal{X}}}
\newcommand{\cY}{{\mathcal{Y}}}
\newcommand{\cZ}{{\mathcal{Z}}}
%\newcommand{\ri}{{\mathrm{i}}}
%\newcommand{\rr}{{\mathrm{r}}}
\newcommand{\EE}{{\mathbb{E}}}
%% macros for math notions and operators
\newcommand{\RR}{\mathbb{R}}
\newcommand{\CC}{\mathbb{C}}
\newcommand{\ZZ}{\mathbb{Z}}
\renewcommand{\SS}{{\mathbb{S}}}
\newcommand{\SSp}{\mathbb{S}_{+}}
\newcommand{\SSpp}{\mathbb{S}_{++}}
%\newcommand{\sign}{\mathrm{sign}}
%\newcommand{\vzero}{\mathbf{0}}
%\newcommand{\vone}{{\mathbf{1}}}
\newcommand{\st}{{\text{s.t.}}} % subject to
\newcommand{\St}{{\mathrm{subject~to}}} % subject to
\newcommand{\op}{{\mathrm{op}}} % subscript for operator norm
\newcommand{\opt}{{\mathrm{opt}}} % subscript for optimal solution
%\newcommand{\supp}{{\mathrm{supp}}} % support
\newcommand{\Prob}{{\mathrm{Prob}}} % probability
\newcommand{\Diag}{{\mathrm{Diag}}} % vector -> diagonal matrix
%\newcommand{\diag}{{\mathrm{diag}}} % matrix diagonal -> vector
\newcommand{\dom}{{\mathrm{dom}}} % domain
\newcommand{\range}{{\mathbf{range}}} % domain
%\newcommand{\grad}{{\nabla}} % gradient
\newcommand{\tr}{{\mathrm{tr}}} % trace
\newcommand{\TV}{{\mathrm{TV}}} % total variation
\newcommand{\Proj}{{\mathrm{Proj}}}
\newcommand{\prj}{{\mathrm{prj}}}
\newcommand{\prox}{\mathbf{prox}}
\newcommand{\refl}{\mathbf{refl}}
\newcommand{\reflh}{\refl^{\bH}}
\newcommand{\proxh}{\prox^{\bH}}
\newcommand{\minimize}{\text{minimize}}
\newcommand{\bgamma}{\boldsymbol{\gamma}}
\newcommand{\bsigma}{\boldsymbol{\sigma}}
\newcommand{\bomega}{\boldsymbol{\omega}}
\newcommand{\blambda}{\boldsymbol{\lambda}}
\newcommand{\bH}{\vH}
\newcommand{\bbH}{\mathbb{H}}
\newcommand{\bB}{\boldsymbol{\cB}}
\newcommand{\Tau}{\mathrm{T}}
\newcommand{\tnabla}{\widetilde{\nabla}}
\newcommand{\TDRS}{T_{\mathrm{DRS}}}
\newcommand{\TPRS}{T_{\mathrm{PRS}}}
\newcommand{\TFBS}{T_{\mathrm{FBS}}}
\newcommand{\best}{\mathrm{best}}
\newcommand{\kbest}{k_{\best}}
\newcommand{\diff}{\mathrm{diff}}
\newcommand{\barx}{\bar{x}}
\newcommand{\xgbar}{\bar{x}_g}
\newcommand{\xfbar}{\bar{x}_f}
\newcommand{\hatxi}{\hat{\xi}}
\newcommand{\xg}{x_g}
\newcommand{\xf}{x_f}
\newcommand{\du}{\mathrm{d}u}
\newcommand{\dy}{\mathrm{d}y}
\newcommand{\kconvergence}{\stackrel{k \rightarrow \infty}{\rightarrow }}
\newcommand{\Null}{\mathbf{Null}}
\newcommand{\Span}{\mathbf{span}}
\makeatletter
\let\@@span\span
\def\sp@n{\@@span\omit\advance\@multicnt\m@ne}
\makeatother
\DeclareMathOperator{\shrink}{shrink} % shrinkage
%\DeclareMathOperator*{\argmin}{arg\,min}
%\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\Min}{minimize}
\DeclareMathOperator*{\Max}{maximize}
\DeclareMathOperator*{\Fix}{Fix}
\DeclareMathOperator*{\zer}{zer}
\DeclareMathOperator*{\nablah}{\nabla^{\bH}}
\DeclareMathOperator*{\gra}{gra}
\DeclarePairedDelimiter{\dotpb}{\langle}{\rangle_{\bH}}
\DeclarePairedDelimiter{\dotpv}{\langle}{\rangle_{\vH}}
\DeclarePairedDelimiter{\dotp}{\langle}{\rangle}
%% macros for environments math equations
\newcommand{\MyFigure}[1]{../fig/#1}
\newcommand{\bc}{\begin{center}}
\newcommand{\ec}{\end{center}}
\newcommand{\bdm}{\begin{displaymath}}
\newcommand{\edm}{\end{displaymath}}
\newcommand{\beq}{\begin{equation}}
\newcommand{\eeq}{\end{equation}}
\newcommand{\bfl}{\begin{flushleft}}
\newcommand{\efl}{\end{flushleft}}
\newcommand{\bt}{\begin{tabbing}}
\newcommand{\et}{\end{tabbing}}
\newcommand{\beqn}{\begin{align}}
\newcommand{\eeqn}{\end{align}}
\newcommand{\beqs}{\begin{align*}} % no equation numbers
\newcommand{\eeqs}{\end{align*}} % no equation numbers
%% macros for theorem-like environments
\newtheorem{theorem}{Theorem}
\newtheorem{assumption}{Assumption}
\newtheorem{condition}{Condition}
\newtheorem{rul}{Rule}
\newtheorem{definition}{Definition}
\newtheorem{corollary}{Corollary}
\newtheorem*{corollary*}{Corollary}
\newtheorem{remark}{Remark}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem{example}{Example}
%\newtheorem*{proof}{Proof}
%\newtheorem{example}[remark]{Example}
\newcommand{\Ker}{\mathbf{Ker}}
\newcommand{\Range}{\mathbf{Range}}
\newlength\myindent
\setlength\myindent{2em}
\newcommand\bindent{%
\begingroup
\setlength{\itemindent}{\myindent}
\addtolength{\algorithmicindent}{\myindent}
}
\newcommand\eindent{\endgroup}
\section{Introduction}
Distributed optimization solves the following optimization problem
\begin{align}
\vx^* :=\argmin_{\vx\in\mathbb{R}^d} \Big [ f(\vx):=\frac{1}{n}\sum_{i=1}^n f_i(\vx) \Big ] \label{problem}
\end{align}
with $n$ computing agents and a communication network. Each $f_i(\vx): \mathbb{R}^d \rightarrow \mathbb{R}$ is a local objective function of agent $i$ and typically defined on the data $\mathcal{D}_i$ settled at that agent. The data distributions $\{\mathcal{D}_i\}$ can be heterogeneous depending on the applications such as in federated learning. The variable $\vx\in\mathbb{R}^d$ often represents model parameters in machine learning. A distributed optimization algorithm seeks an optimal solution that minimizes the overall objective function $f(\vx)$ collectively. According to the communication topology, existing algorithms can be conceptually categorized into centralized and decentralized ones. Specifically, centralized algorithms require global communication between agents (through central agents or parameter servers).
% such as in the parameter-server or all-reduce architectures.
While decentralized algorithms only require local communication between connected agents and are more widely applicable than centralized ones. In both paradigms, the computation can be relatively fast with powerful computing devices; efficient communication is the key to improve algorithm efficiency and system scalability, especially when the network bandwidth is limited.
In recent years, various communication compression techniques, such as quantization and sparsification, have been developed to reduce communication costs. Notably, extensive studies~\citep{1-bit-sgd, NIPS2017_qsgd, signSGD-ICML18, Stich:2018:SSM:3327345.3327357, KarimireddyRSJ19feedback, mishchenko2019distributed, tang2019doublesqueeze, liu2019double} have utilized gradient compression to significantly boost communication efficiency for centralized optimization. They enable efficient large-scale optimization while maintaining comparable convergence rates and practical performance with their non-compressed counterparts. This great success has suggested the potential and significance of communication compression in decentralized algorithms.
While extensive attention has been paid to centralized optimization, communication compression is relatively less studied in decentralized algorithms because the algorithm design and analysis are more challenging in order to cover general communication topologies. There are recent efforts trying to push this research direction.
For instance, DCD-SGD and ECD-SGD~\citep{tang_NIPS2018_7992} introduce difference compression and extrapolation compression to reduce model compression error.
\citep{reisizadeh2019exact, reisizadeh2019robust} introduce QDGD and QuanTimed-DSGD to achieve exact convergence with small stepsize. DeepSqueeze~\citep{dblpDeepSqueeze} directly compresses the local model and compensates the compression error in the next iteration.
CHOCO-SGD~\citep{ksj2019choco, koloskova*2020decentralized} presents a novel quantized gossip algorithm that reduces compression error by difference compression and preserves the model average. Nevertheless, most existing works focus on the compression of primal-only algorithms, i.e., reduce to DGD~\citep{nedic2009distributed, yuan2016convergence} or P-DSGD~\citep{lian2017can}. They are unsatisfying in terms of convergence rate, stability, and the capability to handle heterogeneous data. Part of the reason is that they inherit the drawback of DGD-type algorithms, whose convergence rate is slow in heterogeneous data scenarios where the data distributions are significantly different from agent to agent.
In the literature of decentralized optimization, it has been proved that primal-dual algorithms can achieve faster converge rates and better support heterogeneous data~\citep{ling2015dlm, extra_shi, li2019decentralized, yuan2020can}. However, it is unknown whether communication compression is feasible for primal-dual algorithms and how fast the convergence can be with compression. In this paper, we attempt to bridge this gap by investigating the communication compression for primal-dual decentralized algorithms.
Our major contributions can be summarized as:
\begin{itemize}[leftmargin=0.2in]
\item We delineate two key challenges in the algorithm design for communication compression in decentralized optimization, i.e., data heterogeneity and compression error, and motivated by primal-dual algorithms, we propose a novel decentralized algorithm with compression, LEAD.
\item
We prove that for LEAD, a constant stepsize in the range $(0, 2/(\mu+L)]$ is sufficient to ensure linear convergence for strongly convex and smooth objective functions. To the best of our knowledge, LEAD is the first linear convergent decentralized algorithm with compression. Moreover, LEAD provably works with unbiased compression of arbitrary precision.
\item We further prove that if the stochastic gradient is used, LEAD converges linearly to the $O(\sigma^2)$ neighborhood of the optimum with constant stepsize.
% , where $\sigma^2$ is the stochastic variance bound.
LEAD is also able to achieve exact convergence to the optimum with diminishing stepsize.
\item Extensive experiments on convex problems validate our theoretical analyses, and the empirical study on training deep neural nets shows that LEAD is applicable for nonconvex problems. LEAD achieves state-of-art computation and communication efficiency in all experiments and significantly outperforms the baselines on heterogeneous data.
Moreover, LEAD is robust to parameter settings and needs minor effort for parameter tuning.
\end{itemize}
\section{Related Works}
\label{sec:related_work}
% \paragraph{Decentralized optimization.}
Decentralized optimization can be traced back to the work by~\citet{tsitsiklis1986distributed}. DGD~\citep{nedic2009distributed} is the most classical decentralized algorithm. It is intuitive and simple but converges slowly due to the diminishing stepsize that is needed to obtain the optimal solution~\citep{yuan2016convergence}. Its stochastic version D-PSGD~\citep{lian2017can} has been shown effective for training nonconvex deep learning models.
Algorithms based on primal-dual formulations or gradient tracking are proposed to eliminate the convergence bias in DGD-type algorithms and improve the convergence rate, such as D-ADMM~\citep{mota2013d}, DLM~\citep{ling2015dlm}, EXTRA~\citep{extra_shi}, NIDS~\citep{li2019decentralized}, $D^2$~\citep{pmlr-v80-tang18a}, Exact Diffusion~\citep{yuan2018exact}, OPTRA\citep{xu2020accelerated}, DIGing~\citep{nedic2017achieving}, GSGT~\citep{pu2020distributed}, etc.
% \paragraph{Communication compression.}
Recently, communication compression is applied to decentralized settings by~\citet{tang_NIPS2018_7992}. It proposes two algorithms, i.e., DCD-SGD and ECD-SGD, which require compression of high accuracy and are not stable with aggressive compression. \citet{reisizadeh2019exact, reisizadeh2019robust} introduce QDGD and QuanTimed-DSGD to achieve exact convergence with small stepsize and the convergence is slow. DeepSqueeze~\citep{dblpDeepSqueeze} compensates the compression error to the compression in the next iteration.
Motivated by the quantized average consensus algorithms, such as~\citep{carli2010gossip}, the quantized gossip algorithm CHOCO-Gossip~\citep{ksj2019choco} converges linearly to the consensual solution. Combining CHOCO-Gossip and D-PSGD leads to a decentralized algorithm with compression, CHOCO-SGD, which converges sublinearly under the strong convexity and gradient boundedness assumptions. Its nonconvex variant is further analyzed in~\citep{koloskova*2020decentralized}. A new compression scheme using the modulo operation is introduced in~\citep{lu2020moniqua} for decentralized optimization. A general algorithmic framework aiming to maintain the linear convergence of distributed optimization under compressed communication is considered in~\citep{magnusson2020maintaining}. It requires a contractive property that is not satisfied by many decentralized algorithms including the algorithm in this paper.
\section{Algorithm}
We first introduce notations and definitions used in this work.
We use bold upper-case letters such as $\vX$ to define matrices and bold lower-case letters such as $\vx$ to define vectors. Let $\vone$ and $\vzero$ be vectors with all ones and zeros, respectively. Their dimensions will be provided when necessary. Given two matrices $\vX,~\vY\in\RR^{n\times d}$, we define their inner product as $\langle \vX, \vY \rangle=\text{tr}(\vX^\top\vY)$ and the norm as $\|\vX\|=\sqrt{\langle \vX, \vX \rangle}$. We further define $\langle \vX, \vY \rangle_{\vP}=\text{tr}(\vX^\top \vP\vY)$ and $\|\vX\|_{\vP}=\sqrt{\langle \vX, \vX \rangle}_{\vP}$ for any given symmetric positive semidefinite matrix $\vP \in \mathbb{R}^{n\times n}$.
For simplicity, we will majorly use the matrix notation in this work. For instance, each agent $i$ holds an individual estimate $\vx_i \in \mathbb{R}^d$ of the global variable $\vx \in \mathbb{R}^d$.
Let $\vX^k$ and $\nabla \vF(\vX^k)$ be the collections of $\{\vx_i^k\}_{i=1}^n$ and $\{\nabla f_i(\vx_i^k)\}_{i=1}^n$ which are defined below:
% \vspace{-0.1in}
\begin{align}
\vX^k = \left [ {\vx_1^k}, \dots, {\vx_n^k} \right ]^\top \in \mathbb{R}^{n\times d},
~~~\nabla \vF(\vX^k) = \left [ \nabla f_1(\vx_1^k),\dots,\nabla f_n(\vx_n^k) \right ]^\top \in \mathbb{R}^{n\times d}. %\nonumber
% \nabla \vF(\vX^k; \xi^k) &= \left [ \nabla f_1(\vx_1^k; \xi_1^k),\dots,\nabla f_n(\vx_n^k; \xi_n^k) \right ]^\top \in \mathbb{R}^{n\times d}. %\nonumber
\label{equ:notation}
\end{align}
We use $\nabla \vF(\vX^k; \xi^k)$ to denote the stochastic approximation of $\nabla \vF(\vX^k)$.
With these notations, the update $\vX^{k+1} = \vX^k - \eta \nabla \vF(\vX^k; \xi^k)$ means that $\vx_i^{k+1}=\vx_i^k-\eta\nabla f_i(\vx_i^k; \xi_i^k)$ for all $i$.
%We use $\overbar \vx$ as the average of all rows of $\vX$.
In this paper, we need the average of all rows in $\vX^k$ and $\nabla \vF(\vX^k)$, so we define $\overbar \vX^k = ({\vone^\top \vX^k})/{n}$ and $\overbar {\nabla} \vF(\vX^k) = ({\vone^\top \nabla \vF(\vX^k)})/{n}$. They are row vectors, and we will take a transpose if we need a column vector.
%We define $\hat \vY$ as the estimate of $\vY$ with compression error.
The pseudoinverse of a matrix $\vM$ is denoted as $\vM^{\dagger}$. The largest, $i$th-largest, and smallest nonzero eigenvalues of a symmetric matrix $\vM$ are $\lambda_{\max}(\vM)$, $\lambda_i(\vM)$, and $\lambda_{\min}(\vM)$.% , respectively.
\begin{assumption}[Mixing matrix]
\label{ass:mixing}
The connected network $\cG=\{\cV,\cE \}$ consists of a node set $\cV=\{1,2,\dots,n\}$ and an undirected edge set $\cE$. The primitive symmetric doubly-stochastic matrix $\vW=[w_{ij}]\in\RR^{n\times n}$ encodes the network structure such that $w_{ij}=0$ if nodes $i$ and $j$ are not connected and cannot exchange information.
\end{assumption}
Assumption~\ref{ass:mixing} implies that $-1<\lambda_n(\vW)\leq\lambda_{n-1}(\vW)\leq\cdots\lambda_2(\vW) < \lambda_1(\vW)=1$ and $\vW\vone = \vone$~\citep{xiao2004fast, extra_shi}. The matrix multiplication $\vX^{k+1}=\vW \vX^k$ describes that agent $i$ takes a weighted sum from its neighbors and itself, i.e., $\vx_i^{k+1} = \sum_{j\in\EuScript{N}_{i}\cup \{i\}}{w}_{ij}\vx^k_j$, where $\EuScript{N}_i$ denotes the neighbors of agent $i$.
\subsection{The Proposed Algorithm}
The proposed algorithm LEAD to solve problem~\eqref{problem} is showed in Alg.~\ref{algocomplete} with matrix notations for conciseness. We will refer to the line number in the analysis. A complete algorithm description from the agent's perspective can be found in Appendix~\ref{sec:efficient}.
% For the initialization, we require $\vD^1 = (\vI-\vW) \vZ$ for some $\vZ\in\RR^{n\times d}$. One simple way is to set $\vD^1=\vzero^{n\times d}$.
The motivation behind Alg.~\ref{algocomplete} is to achieve two goals: (a) consensus ($\vx_i^k -(\overbar\vX^k)^\top\rightarrow \vzero$) and (b) convergence ($(\overbar \vX^k)^\top \rightarrow \vx^*$). We first discuss how goal (a) leads to goal (b) and then explain how LEAD fulfills goal (a).
\begin{algorithm}[!ht]
\caption{LEAD}
\label{algocomplete}
\setstretch{1.3}
\textbf{Input:} Stepsize $\eta$, parameter ($\alpha,~\gamma$), $\vX^0,~\vH^1,~\vD^1 = (\vI-\vW)\vZ$ for any $\vZ$ \\
\noindent\textbf{Output: } $\vX^K$ or $1/n\sum_{i=1}^n \vX^K_i$
% \begin{algorithmic}
\begin{algorithmic}[1]
\hspace{-25pt}
\begin{minipage}[t]{0.58\textwidth}
\State $\vH_w^1 = \vW\vH^1$
\State $\vX^1 = \vX^0 - \eta \nabla \vF(\vX^0;\xi^0)$
\For{$k=1,2,\cdots, K-1$}
\State $\vY^k = \vX^k-\eta \nabla \vF(\vX^k;\xi^k)-\eta \vD^k$
\State $\hat \vY^k, \hat \vY^k_w, \vH^{k+1}, \vH_w^{k+1}$ = \Call{Comm} {$\vY^k, \vH^k, \vH_w^k$}
\State $\vD^{k+1} = \vD^k + \frac{\gamma}{2\eta} (\hat \vY^k-\hat \vY^k_w)$
\State $\vX^{k+1} = \vX^k - \eta \nabla \vF(\vX^k;\xi^k) - \eta \vD^{k+1}$
\EndFor
\end{minipage}
% }
% \begin{minipage}[t]{0.46\textwidth}
\begin{minipage}[t]{0.38\textwidth}
\Procedure{Comm}{$\vY, \vH, \vH_w$}
% \STATE $\vU = \vY - \vH$
% \STATE $\vQ =$ \Call{Compress}{$\vU$} \hfill $\vartriangleright$ Compression
% \State $\vQ =$ \Call{Compress}{$\vY - \vH$} \hfill $\vartriangleright$ Compression
\State $\vQ =$ \Call{Compress}{$\vY - \vH$} %$\vartriangleright$ Compression
\State $\hat \vY = \vH + \vQ$
\State $\hat \vY_w = \vH_w + \vW\vQ$ % \hfill $\vartriangleright$ Communication
\State $\vH = (1-\alpha)\vH + \alpha \hat \vY $
\State $\vH_w = (1-\alpha)\vH_w + \alpha \hat \vY_w $
\State \textbf{Return:} $\hat \vY, \hat \vY_w, \vH, \vH_w$
\EndProcedure
\end{minipage}
% }
%\State \textbf{Output:} $\vX^{K}$
\end{algorithmic}
% \normalsize
% \end{algorithm*}
\end{algorithm}
% \textbf{In essence, LEAD runs the approximate gradient descent globally and reduces to the exact gradient descent under consensus.}
In essence, LEAD runs the approximate SGD globally and reduces to the exact SGD under consensus.
One key property for LEAD is $\vone_{n\times 1}^\top \vD^{k}=\vzero$, regardless of the compression error in $\hat \vY^k$. It holds because that for the initialization, we require $\vD^1 = (\vI-\vW) \vZ$ for some $\vZ\in\RR^{n\times d}$, e.g., $\vD^1=\vzero^{n\times d}$, and that the update of $\vD^k$ ensures $\vD^k\in\Range(\vI-\vW)$ for all $k$ and $\vone_{n\times 1}^\top(\vI-\vW)=\vzero$ as we will explain later.
%the update $\vD^{k+1} = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW)\hat \vY^k$ (line 7) and initialization $\vD^1=(\vI-\vW)\vZ$ ensure $\vD^k \in \text{range}(\vI-\vW)$ and thus $\vone^\top \vD^{k}=\vzero$. Importantly, this property holds regardless of the compression error in $\hat \vY^k$.
% $\hat \vY^s, ~\forall s=\{1,2,\dots,k-1\}$.
Therefore, multiplying $(1/n){\vone_{n\times 1}^\top}$ on both sides of { Line} 7 leads to a global average view of Alg.~\ref{algocomplete}:
% \[\overbar \vX^{k+1}=\overbar \vX^k-\eta \overbar {\nabla}\vF(\vX^k).\]
\begin{align}
\label{equ:average_gd}
\overbar \vX^{k+1}=\overbar \vX^k-\eta \overbar {\nabla} \vF(\vX^k; \xi^k),
\end{align}
which doesn't contain the compression error.
Note that this is an approximate SGD step because, as shown in~\eqref{equ:notation}, the gradient $\nabla \vF(\vX^k; \xi^k)$ is not evaluated on a global synchronized model $\overbar\vX^k$.
However, if the solution converges to the consensus solution, i.e., $\vx_i^k-(\overbar\vX^k)^\top \rightarrow \vzero$,
% or equivalently $\vx^k_i \approx \overbar\vX^k, \forall i\in\{1,2,\dots,n\}$,
then
% $\EE_{\xi^k} (\overbar {\nabla} \vF(\vX^k; \xi^k))^\top- {\nabla}f((\overbar\vX^k; \xi^k)^\top)\rightarrow \vzero$
$\EE_{\xi^k} [\overbar {\nabla} \vF(\vX^k; \xi^k)- {\nabla}f(\overbar\vX^k; \xi^k)] \rightarrow \vzero$
and~\eqref{equ:average_gd} gradually reduces to exact SGD.
With the establishment of how consensus leads to convergence, the obstacle becomes how to achieve consensus under local communication and compression challenges. It requires addressing two issues, i.e., data heterogeneity and compression error. To deal with these issues, existing algorithms, such as DCD-SGD, ECD-SGD, QDGD, DeepSqueeze, Moniqua, and CHOCO-SGD, need a diminishing or constant but small stepsize depending on the total number of iterations.
However, these choices unavoidably cause slower convergence and bring in the difficulty of parameter tuning.
% In contrary, one distinctive advantage of LEAD is that a constant step size is sufficient to ensure the consensual solution and achieve linear convergence if full gradient is available as proved in Section~\ref{sec:theory}.
% This is achieved by a novel way of resolving both the data heterogeneity and compression error issues.
In contrast, LEAD takes a different way to solve these issues, as explained below.
\textbf{Data heterogeneity}.
It is common in distributed settings that there exists data heterogeneity among agents, especially in real-world applications where different agents collect data from different scenarios. In other words, we generally have $f_i(\vx) \neq f_j(\vx)$ for $i \neq j$. The optimality condition of problem~\eqref{problem} gives $\vone_{n\times 1}^\top \nabla \vF(\vX^*) = \vzero$, where $\vX^*=[\vx^*,\cdots,\vx^*]$ is a consensual and optimal solution. The data heterogeneity and optimality condition imply that there exist at least two agents $i$ and $j$ such that $\nabla f_i(\vx^*)\neq \vzero$ and $\nabla f_j(\vx^*)\neq \vzero$. As a result, a simple D-PSGD algorithm cannot converge to the consensual and optimal solution as $\vX^* \neq \vW \vX^* -\eta \EE_{\xi} \nabla \vF(\vX^*; \xi)$ even when the stochastic gradient variance is zero.
\textit{Gradient correction}.
Primal-dual algorithms or gradient tracking algorithms are able to convergence much faster than DGD-type algorithms by handling the data heterogeneity issue, as introduced in Section~\ref{sec:related_work}. Specifically, LEAD is motivated by the design of primal-dual algorithm NIDS~\citep{li2019decentralized} and the relation becomes clear if we consider the two-step reformulation of NIDS adopted in~\citep{li2019linear}:
\begin{align}
\vD^{k+1} &= \vD^k + \frac{\vI-\vW}{2\eta} (\vX^k - \eta \nabla \vF(\vX^k) - \eta \vD^k), \\
\vX^{k+1} &= \vX^k - \eta \nabla \vF(\vX^k) - \eta \vD^{k+1} \label{equ:two_step2},
\end{align}
where $\vX^k$ and $\vD^k$ represent the primal and dual variables respectively.
% \footnote{Precisely speaking, $\vD^k$ is a linear transformation of the dual variable in NIDS, but we call $\vD^k$ the dual variable with a little bit abuse of notation.}.
The dual variable $\vD^k$ plays the role of gradient correction. As $k \rightarrow \infty$, we expect $\vD^k \rightarrow -\nabla \vF(\vX^*)$ and $\vX^k$ will converge to $\vX^*$ via the update in~\eqref{equ:two_step2}
since $\vD^{k+1}$ corrects the nonzero gradient $\nabla \vF(\vX^k)$ asymptotically.
The key design of Alg.~\ref{algocomplete} is to provide compression for the auxiliary variable defined as $\vY^k = \vX^k - \eta \nabla \vF(\vX^k) - \eta \vD^k$.
Such design ensures that the dual variable $\vD^k$ lies in $\textbf{Range}(\vI-\vW)$, which is essential for convergence. Moreover, it achieves the implicit error compression as we will explain later. To stabilize the algorithm with inexact dual update, we introduce a parameter $\gamma$ to control the stepsize in the dual update. Therefore, if we ignore the details of the compression, Alg.~\ref{algocomplete} can be concisely written as
\begin{align}
\vY^k & = \vX^k - \eta \nabla \vF(\vX^k; \xi^k) - \eta \vD^k \\
\vD^{k+1} &= \vD^k + \frac{\gamma}{2\eta} (\vI-\vW) \hat \vY^k \\
\vX^{k+1} &= \vX^k - \eta \nabla \vF(\vX^k; \xi^k) - \eta \vD^{k+1}
\end{align}
where $\hat \vY^k$ represents the compression of $\vY^k$ and $\vF(\vX^k; \xi^k)$ denote the stochastic gradients.
Nevertheless, how to compress the communication and how fast the convergence we can attain with compression error are unknown.
In the following, we propose to carefully control the compression error by difference compression and error compensation such that the inexact dual update (Line 6) and primal update (Line 7) can still guarantee the convergence as proved in Section~\ref{sec:theory}.
\textbf{Compression error}.
Different from existing works, which typically compress the primal variable $\vX^k$ or its difference, LEAD first construct an intermediate variable $\vY^k$ and apply compression to obtain its coarse representation $\hat \vY^k$ as shown in the procedure $\Call{Comm}{\vY, \vH, \vH_w}$:
% \vspace{-0.1in}
\begin{itemize}[leftmargin=0.3in]
\setlength\itemsep{-0.2em}
\item Compress the difference between $\vY$ and the state variable $\vH$ as $\vQ$;
\item $\vQ$ is encoded into the low-bit representation, which enables the efficient local communication step $\hat \vY_w = \vH_w + \vW\vQ$. It is
% {\textbf{the only communication step}} in each iteration.
\textit{the only communication step} in each iteration.
\item Each agent recovers its estimate $\hat \vY$ by $\hat \vY = \vH + \vQ$ and we have $\hat \vY_w = \vW \hat \vY$.% provided that $\vH_w=\vW\vH$ as we will see next. In other words, the $i$-th row of $\hat \vY_w$ contains the weighted sum of its neighboring agents' estimate $\hat \vy_j, \forall{j\in \EuScript{N}_{i}}$ and $\hat \vy_i$ itself, i.e., $(\hat{\vy}_w)_i=\sum_{j\in\EuScript{N}_{i}\cup \{i\}} w_{ij}\hat \vy_j$.
\item States $\vH$ and $\vH_w$ are updated based on $\hat \vY$ and $\hat \vY_w$, respectively. We have $\vH_w = \vW \vH$.% given the initialization $\vH_w^1 = \vW\vH^1$.
\end{itemize}
By this procedure, we expect when both $\vY^k$ and $\vH^k$ converge to $\vX^*$, the compression error vanishes asymptotically due to the assumption we make for the compression operator in Assumption~\ref{ass:compression}.
\begin{remark}
Note that difference compression is also applied in DCD-PSGD~\citep{tang_NIPS2018_7992} and CHOCO-SGD~\citep{ksj2019choco}, but their state update is the simple integration of the compressed difference. We find this update is usually too aggressive and cause instability as showed in our experiments. Therefore, we adopt a momentum update $\vH = (1-\alpha)\vH + \alpha \hat \vY$ motivated from DIANA~\citep{mishchenko2019distributed}, which reduces the compression error for gradient compression in centralized optimization.
\end{remark}
\textit{Implicit error compensation.}
% \textit{Error compensation.}
On the other hand, even if the compression error exists, LEAD essentially compensates for the error in the inexact dual update (Line 6), making the algorithm more stable and robust. To illustrate how it works, let $\vE^k = \hat \vY^k- \vY^k$ denote the compression error and $\ve^k_i$ be its $i$-th row. The update of $\vD^k$ gives
\begin{align*}
% \vD^{k+1} &= \vD^k + \frac{\gamma}{2\eta} (\hat \vY^k- \hat \vY^k_w) = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW)\hat \vY^k = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW) \vY^k+ \frac{\gamma}{2\eta} (\vI-\vW) \vE^k
\vD^{k+1} &= \vD^k + \frac{\gamma}{2\eta} (\hat \vY^k- \hat \vY^k_w) = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW) \vY^k+ \frac{\gamma}{2\eta} (\vE^k -\vW \vE^k)
%\vD^{k+1} &= \vD^k + \frac{\gamma}{2\eta} (\hat \vY^k- \hat \vY^k_w) = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW)\hat \vY^k = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW) \vY^k+ \frac{\gamma}{2\eta} (\vE^k -\vW \vE^k)
\end{align*}
% $$\vD^{k+1} = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW)\hat \vY^k = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW)\hat \vY^k = (\vI-\vW) \vY^k + (\vI-\vW) \vE^k$$
where $-\vW \vE^k$ indicates that agent $i$ spreads total compression error $-\sum_{j\in \EuScript{N}_i\cup \{i\} } w_{ji}\ve^k_i = -\ve^k_i$ to all agents and $\vE^k$ indicates that each agent compensates this error locally by adding $\ve^k_i$ back.
This error compensation also explains why the global view in~\eqref{equ:average_gd} doesn't involve compression error.
\begin{remark}
Note that in LEAD, the compression error is compensated into the model $\vX^{k+1}$ through Line 6 and Line 7 such that the gradient computation in the next iteration is aware of the compression error. This has some subtle but important difference from the error compensation or error feedback in~\citep{1-bit-sgd, pmlr-v80-wu18d, Stich:2018:SSM:3327345.3327357, KarimireddyRSJ19feedback, tang2019doublesqueeze, liu2019double, dblpDeepSqueeze}, where the error is stored in the memory and only compensated after gradient computation and before the compression.
\end{remark}
\begin{remark}
The proposed algorithm, LEAD in Alg.~\ref{algocomplete}, recovers NIDS~\citep{li2019decentralized}, D$^2$~\citep{pmlr-v80-tang18a}, Exact Diffusion~\citep{yuan2018exact}. These connections are established in Appendix~\ref{append:connects}.
\end{remark}
% \section{Convergence analysis}
\section{Theoretical Analysis}
\label{sec:theory}
In this section, we show the convergence rate for the proposed algorithm LEAD. Before showing the main theorem, we make some assumptions, which are commonly used for the analysis of decentralized optimization algorithms. All proofs are provided in Appendix~\ref{sec:allproof}.
\begin{assumption}[Unbiased and $C$-contracted operator]
\label{ass:compression}
The compression operator $Q:\RR^{d}\rightarrow\RR^{d}$ is unbiased, i.e., $\EE Q(\vx)=\vx$, and there exists $C\geq 0$ such that $\EE \|\vx-Q(\vx)\|_2^{2}\leq C\|\vx\|^{2}_2$ for all $\vx\in\mathbb{R}^d$.
\end{assumption}
\begin{assumption}[Stochastic gradient]
\label{ass:stochatic}
The stochastic gradient $\nabla f_i(\vx;\xi)$ is unbiased, i.e., $\EE_\xi \nabla f_i(\vx; \xi)=\nabla f_i(\vx)$, and the stochastic gradient variance is bounded: $\EE_\xi \| \nabla f_i(\vx;\xi)- \nabla f_i(\vx)\|_2^{2}\leq \sigma_i^2$ for all $i \in [n]$. Denote $\sigma^2=\frac{1}{n}\sum_{i=1}^{n}\sigma_i^2.$
\end{assumption}
\begin{assumption}\label{ass:smooth}Each $f_{i}$ is $L$-smooth and $\mu$-strongly convex with $L\geq \mu> 0$, i.e., for $i=1,2,\dots, n$ and $\forall \vx,\vy\in\RR^{d}$, we have
\begin{align*}
f_{i}(\vy)+\dotp{\nabla f_{i}(\vy), \vx-\vy}+\frac{\mu}{2}\|\vx-\vy\|^{2}\leq f_{i}(\vx) &\leq f_{i}(\vy)+\dotp{\nabla f_{i}(\vy), \vx-\vy}+\frac{L}{2}\|\vx-\vy\|^{2}. \label{lipschitz} %\\
% \textstyle f_{i}(\vx) &\geq f_{i}(\vy)+\dotp{\nabla f_{i}(\vy), \vx-\vy}+\frac{\mu}{2}\|\vx-\vy\|^{2}. \label{strcov}
\end{align*}
\end{assumption}
% \begin{theorem}[Linear convergence]
\begin{theorem}[Constant stepsize]
\label{thm:linear_convergence}
Let $\{\vX^k, \vH^k, \vD^k\}$ be the sequence generated from Alg.~\ref{algocomplete} and $\vX^*$ is the optimal solution with $\vD^*=-\nabla\vF(\vX^*)$. Under Assumptions~\ref{ass:mixing}-\ref{ass:smooth}, for any constant stepsize $\eta \in (0, {2}/({\mu+L})]$, if the compression parameters $\alpha$ and $\gamma$ satisfy
\begin{align}
\gamma &\in \left(0,\min\Big\{\frac{2}{(3C+1)\beta}, \frac{2\mu\eta(2-\mu\eta) }{[2-\mu\eta(2-\mu\eta)]C\beta} \Big\}\right), \\
\alpha &\in \left[\frac{C\beta\gamma}{2(1+C)}, \frac{1}{a_1} \min\Big\{\frac{2-\beta\gamma}{4-\beta\gamma}, \mu\eta(2-\mu\eta) \Big\}\right],
\end{align}
with $\beta\coloneqq\lambda_{\max}(\vI-\vW)$. Then, in total expectation we have
\begin{align}
\frac{1}{n}\EE\mathcal{L}^{k+1} \leq \rho \frac{1}{n}\EE\mathcal{L}^k + \eta^2\sigma^2,
\end{align}
where
% \begin{align}
% \rho = \max \{1-\frac{\mu(2\eta-L\eta^2)-a_1\alpha}{1-a_1\alpha}, 1-\frac{1}{1+\lambda_{max}(M)}, 1-\alpha \}
% \end{align}
\begin{align*}
\mathcal{L}^k &\coloneqq (1-a_1\alpha)\|\vX^{k}-\vX^*\|^2 + (2\eta^2/\gamma) \EE\|\vD^{k}-\vD^*\|^2_{(\vI-\vW)^\dagger}+ a_1\|\vH^k-\vX^*\|^2, \label{eqn_Lk}\\
\rho& \coloneqq\max\left\{ \frac{1-\mu\eta(2-\mu\eta)}{1-a_1\alpha},
% {2\lambda_{\max}((\vI-\vW)^\dagger)-\gamma\over 2\lambda_{\max}((\vI-\vW)^\dagger)},
1-{\gamma\over 2\lambda_{\max}((\vI-\vW)^\dagger)},~~
1-\alpha\right\}<1, a_1\coloneqq\frac{4(1+C)}{C\beta\gamma+2}
% a_1\coloneqq\frac{4(1+C)}{C\beta\gamma+2}.
% \text{and}~~\vM \coloneqq 2(\vI-\vW)^{\dagger} - \gamma \vI.
%\tilde \lambda \coloneqq 1-\frac{\gamma}{\gamma+\lambda_{\max}(\vM)}
%&\yl{\tilde{\lambda}=\max_{i=1,\cdots, n-1}\Big\{\frac{\lambda_i(\vM)}{\gamma+\lambda_i(\vM)}\Big\}=1-\frac{\gamma}{\gamma+\lambda_{\max}(\vM)}}.
\end{align*}
% The result holds for $C=0$, if we let $\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta}={2\over \beta}$, which is the limit when $C\rightarrow0$.
The result holds for $C\rightarrow 0$.
\end{theorem}
\begin{corollary}[Complexity bounds]
\label{thm:complexity_bound}
Define the condition numbers of the objective function and communication graph as $\kappa_f=\frac{L}{\mu}$ and $\kappa_g
% =\lambda_{\max}(\vI-\vW) \cdot \lambda_{\max}((\vI-\vW)^{\dagger})
=\frac{\lambda_{\max}(\vI-\vW)}{\lambda_{\min}^{+}(\vI-\vW)}$, respectively.
Under the same setting in Theorem~\ref{thm:linear_convergence}, we can choose $\eta=\frac{1}{L}, \gamma= \min \{\frac{1}{C\beta\kappa_f}, \frac{1}{(1+3C)\beta}\}$, and $\alpha=\mathcal{O}(\frac{1}{(1+C)\kappa_f})$ such that
$$\rho = \max \left\{1-\mathcal{O}\Big(\frac{1}{(1+C)\kappa_f}\Big), 1-\mathcal{O}\Big(\frac{1}{(1+C)\kappa_g}\Big), 1-\mathcal{O}\Big(\frac{1}{C\kappa_f \kappa_g }\Big)\right\}.$$
% This recovers classical rates in the following specific cases:
% Some specific cases include:
With full-gradient (i.e., $\sigma=0$), we obtain the following complexity bounds:
\begin{itemize}
\item LEAD converges to the $\epsilon$-accurate solution with the iteration complexity
$$\mathcal{O}\Big ( \big ( (1+C)(\kappa_f + \kappa_g) + C\kappa_f \kappa_g \big ) \log \frac{1}{\epsilon} \Big ).$$
\item When $C=0$ (i.e., there is no compression),
% we can set $\eta=\mathcal{O}(\frac{1}{L}), \gamma=1, \alpha=1$ and $a_1=0$.
we obtain $\rho = \max \{ 1-\mathcal{O}(\frac{1}{\kappa_f}), 1-\mathcal{O}(\frac{1}{\kappa_g}) \},$ and
the iteration complexity $\mathcal{O}\Big ( (\kappa_f + \kappa_g) \log \frac{1}{\epsilon} \Big )$. This exactly recovers the convergence rate of NIDS~\citep{li2019decentralized}.
\item When $C\leq\frac{\kappa_f+\kappa_g}{\kappa_f\kappa_g+\kappa_f+\kappa_g},$ the asymptotical complexity is $\mathcal{O}\Big ( (\kappa_f + \kappa_g) \log \frac{1}{\epsilon} \Big )$, which also recovers that of NIDS~\citep{li2019decentralized} and indicates that the compression doesn't harm the convergence in this case.
\item With $C=0$ (or $C\leq\frac{\kappa_f+\kappa_g}{\kappa_f\kappa_g+\kappa_f+\kappa_g}$) and fully connected communication graph (i.e., $\vW=\frac{\vone\vone^{\top}}{n}$), we have $\beta=1$ and $\kappa_g=1$. Therefore, we obtain $\rho = 1-\mathcal{O}(\frac{1}{\kappa_f})$ and the complexity bound $\mathcal{O} (\kappa_f log \frac{1}{\epsilon} )$. This recovers the convergence rate of gradient descent~\citep{nesterov2013introductory}.
\end{itemize}
\end{corollary}
% \begin{corollary}
% \label{thm:kappa_dominate}
% Under the setting in Theorem~\ref{thm:linear_convergence}, when the condition number $\kappa=\frac{L}{\mu}$ dominates, we can choose $\eta=\frac{1}{L}, \gamma\propto \mathcal{O}(\frac{1}{C\beta\kappa})$ and $\alpha=\frac{C\beta\gamma}{2(1+C)}$ such that the convergence rate $\rho$ is in the order $1-\mathcal{O}(\frac{1}{(1+C)\kappa})$.
% \end{corollary}
\begin{remark}
Under the setting in Theorem~\ref{thm:linear_convergence}, LEAD converges linearly to the $\mathcal{O}(\sigma^2)$ neighborhood of the optimum and converges linearly exactly to the optimum if full gradient is used, e.g., $\sigma=0$. The linear convergence of LEAD holds when $\eta<2/L$,
but we omit the proof.
\end{remark}
\begin{remark}[Arbitrary compression precision]
Pick any $\eta\in(0,{2}/({\mu+L})],$ based on the compression-related constant $C$ and the network-related constant $\beta$, we can select $\gamma$ and $\alpha$ in certain ranges to achieve the convergence. It suggests that LEAD supports unbiased compression with arbitrary precision, i.e., any $C>0$.
\end{remark}
\begin{corollary}[Consensus error]
\label{thm:consensus}
Under the same setting in Theorem~\ref{thm:linear_convergence}
, let $\overbar{\vx}^k=\frac{1}{n}\sum_{i=1}^n\vx^k_i$ be the averaged model and $\vH^0=\vH^1$, then all agents achieve consensus at the rate
\begin{align}
\frac{1}{n}\sum_{i=1}^{n}\EE\left\|\vx_i^k-\overbar{\vx}^k\right\|^2 \leq \frac{2\mathcal{L}^{0}}{n}\rho^{k} + \frac{2\sigma^2}{1-\rho}\eta^2.
\end{align}
where $\rho$ is defined as in Corollary~\ref{thm:complexity_bound} with appropriate parameter settings.
\end{corollary}
\begin{theorem}[Diminishing stepsize]
\label{thm:diminish}
Let $\{\vX^k, \vH^k, \vD^k\}$ be the sequence generated from Alg.~\ref{algocomplete} and $\vX^*$ is the optimal solution with $\vD^*=-\nabla\vF(\vX^*)$.
Under Assumptions~\ref{ass:mixing}-\ref{ass:smooth}, if $\eta_k=\frac{2\theta_5}{\theta_3\theta_4\theta_5 k+2}$ and $\gamma_k=\theta_4\eta_k$, by taking $\alpha_k=\frac{C\beta\gamma_k}{2(1+C)}$, in total expectation we have
\begin{align}
% \frac{1}{n}\EE\mathcal{L}^k\leq\frac{2\max\left\{{ \frac{1}{n}\mathcal{L}^0},\frac{2\sigma^2\theta_5}{\theta_3\theta_4}\right\}}{\theta_3\theta_4 \theta_5 k+2}. \\
% \frac{1}{n}\EE\mathcal{L}^k \lesssim O(\frac{1}{k})
\frac{1}{n}\sum_{i=1}^{n}\EE\left\|\vx_i^k-\vx^*\right\|^2 \lesssim \mathcal{O}\left(\frac{1}{k}\right)
\end{align}
where
% \begin{align}
% \rho = \max \{1-\frac{\mu(2\eta-L\eta^2)-a_1\alpha}{1-a_1\alpha}, 1-\frac{1}{1+\lambda_{max}(M)}, 1-\alpha \}
% \end{align}
% \begin{align*}
% $\mathcal{L}^k &\coloneqq (1-a_1\alpha_k)\|\vX^{k}-\vX^*\|^2 + (2\eta_k^2/\gamma_k) \EE\|\vD^{k+1}-\vD^*\|^2_{(\vI-\vW)^\dagger}+ a_1\|\vH^k-\vX^*\|^2 \label{eqn_Lk}$
% \end{align*}
% and
$\theta_1, \theta_2, \theta_3, \theta_4$ and $\theta_5$ are constants defined in the proof. The complexity bound for arriving at the $\epsilon$-accurate solution is $\mathcal{O}(\frac{1}{\epsilon})$.
\end{theorem}
\begin{remark}
Compared with CHOCO-SGD, LEAD requires unbiased compression and the convergence under biased compression is not investigated yet.
% This is an acceptable assumption considering that lots of biased compression can be converted to an unbiased one by appropriate random dithering.
The analysis of CHOCO-SGD relies on the bounded gradient assumptions, i.e., $\|\nabla f_i(\vx)\|^2\leq G$, which is restrictive because it conflicts with the strong convexity while LEAD doesn't need this assumption.
% LEAD converges linearly, which is significantly faster than the sublinear convergence of CHOCO-SGD~\citep{ksj2019choco} for the full-gradient case. Moreover, LEAD only makes mild assumptions.
Moreover, in the theorem of CHOCO-SGD, it requires a specific point set of $\gamma$ while LEAD only requires $\gamma$ to be within a rather large range. This may explain the advantages of LEAD over CHOCO-SGD in terms of robustness to parameter setting.
\end{remark}
\section{Numerical Experiment}
\label{sec:experiment}
We consider three machine learning problems -- $\ell_2$-regularized linear regression, logistic regression, and deep neural network.
The proposed LEAD is compared with QDGD~\citep{reisizadeh2019exact}, DeepSqueeze~\citep{dblpDeepSqueeze}, CHOCO-SGD~\citep{ksj2019choco}, and two non-compressed algorithms DGD~\citep{yuan2016convergence} and NIDS~\citep{li2019decentralized}.
\textbf{Setup.}
We consider eight machines connected in a ring topology network. Each agent can only exchange information with its two 1-hop neighbors. The mixing weight is simply set as $1/3$. For compression, we use the unbiased $b$-bits quantization method with $\infty$-norm
\begin{align}
Q_{\infty}(\vx):=\left(\|\vx\|_{\infty} 2^{-(b-1)} \sign(\vx) \right) \cdot \left \lfloor{\frac{2^{(b-1)}|\vx|}{\|\vx\|_{\infty}}+\vu}\right \rfloor,
\end{align}
% \vspace{-0.1in}
% \begin{align}
% Q(\vx):= \|\vx\|_{\infty} \cdot \text{sign}(\vx) \cdot \frac{1}{2^{(b-1)}-1} \cdot \left \lfloor{\frac{(2^{(b-1)}-1)|\vx|}{\|\vx\|_{\infty}}+\vu}\right \rfloor \nonumber
% \end{align}
where $\cdot$ is the Hadamard product, $|\vx|$ is the elementwise absolute value of $\vx$, and $\vu$ is a random vector uniformly distributed in $[0, 1]^d$. Only sign$(\vx)$, norm $\|\vx\|_{\infty}$, and integers in the bracket need to be transmitted.
Note that this quantization method is similar to the quantization used in QSGD~\citep{NIPS2017_qsgd} and CHOCO-SGD~\citep{ksj2019choco}, but we use the $\infty$-norm scaling instead of the $2$-norm. This small change brings significant improvement on compression precision as justified both theoretically and empirically in Appendix~\ref{sec:quantization}.
In this section, we choose $2$-bit quantization and quantize the data blockwise (block size = $512$).
For all experiments, we tune the stepsize $\eta$ from $\{0.01, 0.05, 0.1, 0.5\}$. For QDGD, CHOCO-SGD and Deepsqueeze, $\gamma$ is tuned from $\{0.01, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0\}$. Note that different notations are used in their original papers. Here we uniformly denote the stepsize as $\eta$ and the additional parameter in these algorithms as $\gamma$ for simplicity.
For LEAD, we simply fix $\alpha=0.5$ and $\gamma=1.0$ for all experiments since we find LEAD is robust to parameter settings as we validate in the parameter sensitivity analysis in Appendix~\ref{sec:para_analysys}. This indicates the minor effort needed for tuning LEAD. Detailed parameter settings for all experiments are summarized in Appendix~\ref{apend:para_setting}.
\textbf{Linear regression.}
We consider the problem: $f(\vx)=\sum_{i=1}^n (\|\vA_i\vx-\vb_i\|^2+\lambda \|\vx\|^2)$. Data matrices $\vA_i \in \mathbb{R}^{200 \times 200}$ and the true solution $\vx'$ is randomly synthesized. The values $\vb_i$ are generated by adding Gaussian noise to $\vA_i\vx'$. We let $\lambda=0.1$ and the optimal solution of the linear regression problem be $\vx^*$. We use full-batch gradient to exclude the impact of gradient variance.
The performance is showed in Fig.~\ref{fig:linear_regression}. The distance to $\vx^*$ in Fig.~\ref{fig:ls_a} and the consensus error in Fig.~\ref{fig:ls_c} verify that LEAD converges exponentially to the optimal consensual solution. It significantly outperforms most baselines and matches NIDS well under the same number of iterations. Fig.~\ref{fig:ls_b} demonstrates the benefit of compression when considering the communication bits. Fig.~\ref{fig:ls_d} shows that the compression error vanishes for both LEAD and CHOCO-SGD while the compression error is pretty large for QDGD and DeepSqueeze because they directly compress the local models.
\begin{figure*}[!ht]
\vspace{-0.1in}
\centering
\subfloat[$\|\vX^k-\vX^*\|_F$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LinearReg/LR_bits_para2_cRate1_err_iter.pdf}
\vspace{-0.35in}
\label{fig:ls_a}
\end{minipage}
}
\subfloat[$\|\vX^k-\vX^*\|_F$]{
\begin{minipage}[t]{0.42\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LinearReg/LR_bits_para2_cRate1_err_cost.pdf}
\vspace{-0.35in}
\label{fig:ls_b}
\end{minipage}
}
% \vspace{-0.1in}
\\
\subfloat[$\|\vX^k-{\vone_{n\times 1}\overbar\vX^k}\|_F$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LinearReg/LR_bits_para2_cRate1_consensus_iter.pdf}
\vspace{-0.35in}
\label{fig:ls_c}
\end{minipage}
}
\subfloat[Compression error]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LinearReg/LR_bits_para2_cRate1_worker_residual_iter.pdf}
\vspace{-0.35in}
\label{fig:ls_d}
\end{minipage}
}
\caption{Linear regression problem.}
\label{fig:linear_regression}
% \vspace{-5pt}
\end{figure*}
\textbf{Logistic regression.}
We further consider a logistic regression problem on the MNIST dataset. The regularization parameter is $10^{-4}$. We consider both \textit{homogeneous} and \textit{heterogeneous} data settings. In the homogeneous setting, the data samples are randomly shuffled before being uniformly partitioned among all agents such that the data distribution from each agent is very similar. In the heterogeneous setting, the samples are first sorted by their labels and then partitioned among agents.
Due to the space limit, we mainly present the results in heterogeneous setting here and defer the homogeneous setting to Appendix~\ref{append:addition_experiment}. The results using full-batch gradient and mini-batch gradient (the mini-batch size is $512$ for each agent) are showed in Fig.~\ref{fig:logistic_heter_full} and Fig.~\ref{fig:logistic_heter_mini} respectively and both settings shows the faster convergence and higher precision of LEAD.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%% Logitstic regression: unshuffled case
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure*}[!ht]
\vspace{-0.2in}
\centering
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LogisticReg/lr_mnist_shuffleFalse_train_con_loss__epoch_bits2.pdf}
\vspace{-0.35in}
\label{fig:lgu_c}
\end{minipage}
}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LogisticReg/lr_mnist_shuffleFalse_train_con_loss__bits_bits2.pdf}
\vspace{-0.35in}
\label{fig:lgu_d}
\end{minipage}
}
% \vspace{-0.1in}
\caption{Logistic regression problem in the heterogeneous case (full-batch gradient).}
\label{fig:logistic_heter_full}
\end{figure*}
% \vspace{-1in}
\begin{figure*}[!ht]
\vspace{-0.2in}
\begin{center}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/Stochastic_LogisticReg/lr_mnist_stochastic_shuffleFalse_train_con_loss__epoch_bits2.pdf}
\vspace{-0.35in}
\end{minipage}
}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/Stochastic_LogisticReg/lr_mnist_stochastic_shuffleFalse_train_con_loss__bits_bits2.pdf}
\vspace{-0.35in}
\end{minipage}
}
\end{center}
\vspace{-0.1in}
\caption{Logistic regression in the heterogeneous case (mini-batch gradient).}
\label{fig:logistic_heter_mini}
\end{figure*}
\begin{figure*}[!ht]
\vspace{-0.2in}
\begin{center}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/deeplearn/alexnet_cifar_shuffleTrue_train_con_loss__bits_bits2.pdf}
\vspace{-0.25in}
\end{minipage}
}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/deeplearn/alexnet_cifar_shuffleFalse_train_con_loss__bits_bits2.pdf}
\vspace{-0.25in}
\end{minipage}
}
\end{center}
\vspace{-0.1in}
\caption{Stochastic optimization on deep neural network ($*$ means divergence).}
\label{fig:alexnet_heter}
\vspace{-0.1in}
\end{figure*}
\textbf{Neural network.} We empirically study the performance of LEAD in optimizing deep neural network by training AlexNet ($240$ MB) on CIFAR10 dataset. The mini-batch size is $64$ for each agents. Both the homogeneous and heterogeneous case are showed in Fig.~\ref{fig:alexnet_heter}. In the homogeneous case, CHOCO-SGD, DeepSqueeze and LEAD perform similarly and outperform the non-compressed variants in terms of communication efficiency, but CHOCO-SGD and DeepSqueeze need more efforts for parameter tuning because their convergence is sensitive to the setting of $\gamma$.
In the heterogeneous cases, LEAD achieves the fastest and most stable convergence. Note that in this setting, sufficient information exchange is more important for convergence because models from different agents are moving to significantly diverse directions. In such case, DGD only converges with smaller stepsize and its communication compressed variants, including QDGD, DeepSqueeze and CHOCO-SGD, diverge in all parameter settings we try.
In summary, our experiments verify our theoretical analysis and show that LEAD is able to handle data heterogeneity very well. Furthermore, the performance of LEAD is robust to parameter settings and needs less effort for parameter tuning, which is critical in real-world applications.
\section{Conclusion}
In this paper, we investigate the communication compression in decentralized optimization. Motivated by primal-dual algorithms, a novel decentralized algorithm with compression, LEAD, is proposed to achieve faster convergence rate and to better handle heterogeneous data while enjoying the benefit of efficient communication. The nontrivial analyses on the coupled dynamics of inexact primal and dual updates as well as compression error establish the linear convergence of LEAD when full gradient is used and the linear convergence to the $\mathcal{O}(\sigma^2)$ neighborhood of the optimum when stochastic gradient is used.
Extensive experiments validate the theoretical analysis and demonstrate the state-of-the-art efficiency and robustness of LEAD. LEAD is also applicable to non-convex problems as empirically verified in the neural network experiments but we leave the non-convex analysis as the future work.
\newpage
% {\Large{\textbf{Supplemental Materials}}}
%{\Large{\textbf{Appendix}}}
\def\tmptitle{Contents of Appendix}
\etocsettocstyle{\Large\textbf{\tmptitle}\par\normalsize
}{\bigskip}
\etocdepthtag.toc{mtappendix}
\etocsettagdepth{mtchapter}{none}
\etocsettagdepth{mtappendix}{subsection}
\tableofcontents
\newpage
\section{LEAD in agent's perspective}
\label{sec:efficient}
In the main paper, we described the algorithm with matrix notations for concision. Here we further provide a complete algorithm description from the agents' perspective.
\begin{algorithm}[H]
\caption{LEAD in Agent's Perspective}
\textbf{input:} stepsize $\eta$, compression parameters ($\alpha,~\gamma$), initial values $\vx_i^0$,~$\vh_i^1$,~$\vz_i,~\forall i\in\{ 1,2,\dots,n \}$\\ %, $\vD^1 = (\vI-\vW) \vZ$ for some $\vZ$\\
\noindent\textbf{output:} $\vx_i^K, ~\forall i\in\{1,2,\dots,n\}$ or $\frac{\sum_{i=1}^n \vx_i^K}{n}$
\begin{algorithmic}[1]
\setstretch{1.3}
% \begin{algorithmic}
\For{each agent $i\in\{1,2,\dots, n\}$}
\State $\vd_i^1 = \vz_i - \sum_{j\in \EuScript{N}_i\cup \{i\} } w_{ij}\vz_j$
\State $(\vh_w)_i^1 = \sum_{j\in \EuScript{N}_i\cup \{i\} } w_{ij}(\vh_w)_j^1$
\State $\vx_i^1 = \vx_i^0 - \eta \nabla f_i(\vx_i^0;\xi_i^0)$
\EndFor
\For{$k=1,2,\dots, K-1$} in parallel for all agents $i\in\{1,2,\dots,n\}$
\State compute $\nabla f_i(\vx_i^k; \xi_i^k)$ \hfill $\vartriangleright$ Gradient computation
\State $\vy_i^k = \vx_i^k- \eta \nabla f_i(\vx_i^k;\xi^k_i)-\eta \vd_i^k$
\State $\vq_i^k = {\text{Compress}} (\vy_i^k-\vh_i^k)$ \hfill $\vartriangleright$ Compression
\State $\hat \vy_i^k = \vh_i^k + \vq_i^k$
\For{neighbors $j\in\EuScript{N}_i$}
\State Send $\vq_i^k$ and receive $\vq_j^k$ \hfill $\vartriangleright$ Communication
\EndFor
\State ${(\hat \vy_w)}_i^{k} = {(\vh_w)}_i^{k} + \sum_{j\in \EuScript{N}_i\cup \{i\} } w_{ij}\vq_j^k $
\State $\vh_i^{k+1} = (1-\alpha)\vh_i^k + \alpha \hat \vy_i^{k} $
\State ${(\vh_w)}_i^{k+1} = (1-\alpha){(\vh_w)}_i^{k} + \alpha {(\hat\vy_w)}_i^{k} $
\State $\vd_i^{k+1} = \vd_i^k + \frac{\gamma}{2\eta} \big(\hat \vy_i^{k}-{(\hat \vy_w)}_i^{k}\big )$
\State $\vx_i^{k+1} = \vx_i^k - \eta \nabla f_i(\vx_i^k; \xi_i^k) - \eta \vd_i^{k+1}$ \hfill $\vartriangleright$ Model update
\EndFor
\end{algorithmic}
\label{alg:agent}
\end{algorithm}
\section{Connections with exiting works}
\label{append:connects}
The non-compressed variant of LEAD in Alg.~\ref{algocomplete} recovers NIDS~\citep{li2019decentralized}, $D^2$~\citep{pmlr-v80-tang18a} and Exact Diffusion~\citep{yuan2018exact} as shown in Proposition~\ref{thm:equivalence_nids}. In Corollary~\ref{thm:reduce_nids}, we show that the convergence rate of LEAD exactly recovers the rate of NIDS when $C=0, \gamma=1$ and $\sigma=0$.
\begin{proposition}[Connection to NIDS, $D^2$ and Exact Diffusion]
\label{thm:equivalence_nids}
When there is no communication compression (i.e., $\hat\vY^k=\vY^k$) and $\gamma=1$, Alg.~\ref{algocomplete} recovers $D^2$:
\begin{align}
\vX^{k+1} = \frac{\vI+\vW}{2}\Big(2\vX^{k} - \vX^{k-1} - \eta \nabla \vF(\vX^k;\xi^k) + \eta \nabla \vF(\vX^{k-1};\xi^{k-1}) \Big). \label{equ:nids}
\end{align}
Furthermore, if the stochastic estimator of the gradient $\nabla\vF(\vX^k; \xi^k)$ is replaced by the full gradient, it recovers NIDS and Exact Diffusion with specific settings. %with an adjusted mixing matrix $\widetilde \vW = \frac{\vI+\vW}{2}$
%\begin{align}
% \vX^{k+1} = \frac{\vI+\vW}{2} \Big (2\vX^{k} - \vX^{k-1} - \eta \nabla \vF(\vX^k; \xi^k) + \eta \nabla \vF(\vX^{k-1}; \xi^{k-1}) \Big ). \label{equ:d2}
%\end{align}
\end{proposition}
\begin{corollary}[Consistency with NIDS]
\label{thm:reduce_nids}
When $C=0$ (no communication compression), $\gamma=1$ and $\sigma=0$ (full gradient), LEAD has the convergence consistent with NIDS with $\eta\in(0,2/(\mu+L)]$:
\begin{align}
\mathcal{L}^{k+1} \leq \max \left\{1- \mu(2\eta-\mu\eta^2), 1-\frac{1}{2\lambda_{\max}((\vI-\vW)^{\dagger})}\right\} \mathcal{L}^k.
\end{align}
%\yl{when $C=0,$ i.e., there is no compression, $\alpha$ can take $0$ and $\gamma$ can be $1$, in this case $\rho=\max\{1-\mu(2\eta-L\eta^2), 1-\frac{1}{2\lambda_{\max}((\vI-\vW)^{\dagger})}\},$ which recovers NIDS.}
\end{corollary}
See the proof in~\ref{proof:consistent_nids}.
% \label{apdx:proof_equivalence_nids}
\begin{proof}[Proof of Proposition~\ref{thm:equivalence_nids}]
Let $\gamma=1$ and $\hat \vY^k=\vY^k$. Combing Lines 4 and 6 of Alg.~\ref{algocomplete} gives
\begin{align}
\vD^{k+1} &= \vD^k + \frac{\vI-\vW}{2\eta}(\vX^k-\eta \nabla \vF(\vX^k;\xi^k)-\eta \vD^k). \label{equ:nids_1}
\end{align}
Based on Line 7, we can represent $\eta \vD^{k}$ from the previous iteration as
\begin{align}
\eta \vD^{k} = \vX^{k-1} - \vX^{k} - \eta \nabla \vF(\vX^{k-1}; \xi^{k-1}). \label{equ:nids_2}
\end{align}
Eliminating both $\vD^k$ and $\vD^{k+1}$ by substituting~\eqref{equ:nids_1}-\eqref{equ:nids_2} into Line 7, we obtain
\begin{align}
\vX^{k+1} &= \vX^k - \eta \nabla \vF(\vX^k;\xi^k) - \Big (\eta \vD^{k}+\frac{\vI-\vW}{2}(\vX^k-\eta \nabla \vF(\vX^k;\xi^k)-\eta \vD^k) \Big) \quad(\text{from}~\eqref{equ:nids_1})\nonumber\\
&= \frac{\vI+\vW}{2}(\vX^k - \eta \nabla \vF(\vX^k;\xi^k)) - \frac{\vI+\vW}{2} \eta \vD^{k} \nonumber\\
&= \frac{\vI+\vW}{2} (\vX^k - \eta \nabla \vF(\vX^k;\xi^k) ) - \frac{\vI+\vW}{2} ( \vX^{k-1} - \vX^{k} - \eta \nabla \vF(\vX^{k-1};\xi^{k-1}) ) \quad(\text{from}~\eqref{equ:nids_2}) \nonumber\\
&= \frac{\vI+\vW}{2} (2\vX^k -\vX^{k-1} - \eta \nabla \vF(\vX^k;\xi^k) + \eta \nabla \vF(\vX^{k-1};\xi^{k-1}) ),
\end{align}
which is exactly $D^2$. It also recovers Exact Diffusion with $\vA=\frac{\vI+\vW}{2}$ and $\vM = \eta\vI$ in Eq. (97) of~\citep{yuan2018exact}.
\end{proof}
\section{Compression method}
\label{sec:quantization}
\subsection{p-norm b-bits quantization}
\begin{theorem}[p-norm b-bit quantization]
\label{thm:pnorm_quantization}
Let us define the quantization operator as
\begin{align}
\label{equ:pnorm_bits}
% Q(\vx):= \|\vx\|_{\infty} \cdot \text{sign}(\vx) \cdot \frac{1}{2^{(b-1)}-1} \cdot \left \lfloor{\frac{(2^{(b-1)}-1)|\vx|}{\|\vx\|_{\infty}}+\vu}\right \rfloor \nonumber
Q_p(\vx):= \left(\|\vx\|_{p} \sign(\vx) 2^{-(b-1)}\right) \cdot \left \lfloor{\frac{2^{b-1}|\vx|}{\|\vx\|_{p}}+\vu}\right \rfloor %\nonumber
\end{align}
where $\cdot$ is the Hadamard product, $|\vx|$ is the elementwise absolute value and $\vu$ is a random dither vector uniformly distributed in $[0, 1]^d$. $Q_p(\vx)$ is unbiased, i.e., $\EE Q_p(\vx)=\vx$, and the compression variance is upper bounded by
\begin{align}
\EE \|\vx-Q_p(\vx)\|^2 \leq \frac{1}{4} \|\sign(\vx) 2^{-(b-1)}\|^2 \|\vx\|_{p}^2,
\end{align}
which suggests that $\infty$-norm provides the smallest upper bound for the compression variance
due to $\|\vx\|_p \leq \|\vx\|_q, \forall \vx$ if $1\leq q \leq p \leq\infty$.
\end{theorem}
\begin{remark}
For the compressor defined in~\eqref{equ:pnorm_bits}, we have the following the compression constant
$$C=\sup_{\vx} \frac{\|\sign(\vx) 2^{-(b-1)}\|^2 \|\vx\|_{p}^2}{4\|\vx\|^2}.$$
\end{remark}
\begin{proof}
Let denote $\vv=\|\vx\|_{p} \sign(\vx) 2^{-(b-1)}$, $s=\frac{2^{b-1}|\vx|}{\|\vx\|_{p}}$, $s_1=\left\lfloor{\frac{2^{b-1}|\vx|}{\|\vx\|_{p}}}\right \rfloor$ and $s_2=\left\lceil{\frac{2^{b-1}|\vx|}{\|\vx\|_{p}}}\right \rceil$. We can rewrite $\vx$ as $\vx = s \cdot \vv$.
For any coordinate $i$ such that $s_i=(s_1)_i$, we have $Q_p(\vx_i)=(s_1)_i \vv_i$ with probability $1$. Hence $\EE Q_p(\vx)_i=s_i\vv_i=\vx_i$ and
%\begin{align}
% \EE Q(\vx)= \|\vx\|_{p} \cdot \text{sign}(\vx) \cdot 2^{-(b-1)} \cdot \frac{2^{b-1}|\vx|}{\|\vx\|_{p}} = \text{sign}(\vx)\cdot \|\vx\| = \vx,
%\EE Q(\vx)= s \cdot \vv \cdot 1 + (s+1)\cdot \vv \cdot 0 = s\cdot \vv = \vx,
%\end{align}
%and
\begin{align*}
\EE (\vx_i-Q_p(\vx)_i)^2=(\vx_i-s_i\vv_i)^2=0.
%\cdot 1 + \|\vx-(s+1) \cdot \vv\|^2\cdot 0 = 0.
\end{align*}
For any coordinate $i$ such that $s_i\not=(s_1)_i$, we have $(s_2)_i-(s_1)_i=1$ and $Q_p(\vx)_i$ satisfies
\begin{equation*}
Q_p(\vx)_i=\left\{\begin{aligned}
& (s_1)_i\vv_i,\ \ \text{w.p.}\ (s_2)_i-s_i,\\
& (s_2)_i \vv_i,\ \ \text{w.p.}\ s_i-(s_1)_i.\\
\end{aligned}\right.
\end{equation*}
Thus, we derive
\begin{align*}
\EE Q_p(\vx)_i &= \vv_i (s_1)_i(s_2-s)_i + \vv_i (s_2)_i(s-s_1)_i = \vv_i s_i(s_2-s_1)_i = \vv_i s_i = \vx_i,
\end{align*}
and
\begin{align*}
\EE [\vx_i-Q_p(\vx)_i]^2 &= (\vx_i- \vv_i (s_1)_i )^2 (s_2-s)_i + (\vx_i-\vv_i (s_2)_i)^2 (s-s_1)_i \\
&=(s_2-s_1)_i \vx_i^2 + \big ((s_1)_i(s_2)_i(s_1-s_2)_i+s_i((s_2)_i^2-(s_1)_i^2)\big) \vv_i^2 -2s_i(s_2-s_1)_i\vx_i\vv_i \\
&=\vx_i^2 + \big(-(s_1)_i(s_2)_i+s_i(s_2+s_1)_i\big) \vv_i^2 -2s_i \vx_i\vv_i \\
&=(\vx_i - s_i \vv_i)^2 + \big (-(s_1)_i(s_2)_i+s_i(s_2+s_1)_i -s_i^2\big) \vv_i^2 \\
&=(\vx_i - s_i \vv_i)^2 + (s_2-s)_i(s-s_1)_i \vv_i^2 \\
&=(s_2-s)_i(s-s_1)_i \vv_i^2 \\
&\leq \frac{1}{4} \vv_i^2.
\end{align*}
Considering both cases, we have $\EE Q(\vx)=\vx$ and
\begin{equation*}
\begin{aligned}
\EE\|\vx-Q_p(\vx)\|^2&=\sum_{\{s_i=(s_1)_i\}}\EE [\vx_i-Q_p(\vx)_i]^2+\sum_{\{s_i\not=(s_1)_i\}}\EE [\vx_i-Q_p(\vx)_i]^2\\
&\leq 0+\frac{1}{4}\sum_{\{s_i\not=(s_1)_i\}}\vv_i^2\\
&\leq \frac{1}{4}\|\vv\|^2\\
&=\frac{1}{4}\|\sign(\vx) 2^{-(b-1)}\|^2\|\vx\|_p^2.
\end{aligned}
\end{equation*}
% Because $\|\vx\|_p \leq \|\vx\|_q, \forall \vx$ if $1\leq q \leq p$, choosing $p=\infty$ provides smallest upper bound for compression variance, i.e., compression error in expectation.
\end{proof}
% \subsection{Compression error}
\subsection{Compression error}
\begin{figure*}[!ht]
% \begin{minipage}[t]{0.5\textwidth}
\centering
\includegraphics[width=0.5\textwidth]{iclr_fig/compara_bits_norm.pdf}
\caption{Relative compression error $\frac{\|\vx-Q(\vx)\|_2}{\|\vx\|_2}$ for p-norm b-bit quantization}
\label{fig:pnorm_quan}
% \end{minipage}
\end{figure*}
\begin{figure*}[!ht]
% \begin{minipage}[t]{0.5\textwidth}
\centering
\includegraphics[width=0.5\textwidth]{iclr_fig/compara_all_quan.pdf}
\caption{Comparison of compression error $\frac{\|\vx-Q(\vx)\|_2}{\|\vx\|_2}$ between different compression methods}
% \label{fig:pnorm_quan}
% \end{minipage}
\label{fig:compare_quan_all}
\end{figure*}
To verify Theorem~\ref{thm:pnorm_quantization}, we compare the compression error of the quantization method defined in~\eqref{equ:pnorm_bits} with different norms ($p=1, 2,3,\dots, 6, \infty$). Specifically, we uniformly generate 100 random vectors in $\RR^{10000}$ and compute the average compression error. The result shown in Figure~\ref{fig:pnorm_quan} verifies our proof in Theorem~\ref{thm:pnorm_quantization} that the compression error decreases when $p$ increases. This suggests that $\infty$-norm provides the best compression precision under the same bit constraint.
Under similar setting, we also compare the compression error with other popular compression methods, such as top-k and random-k sparsification. The x-axes represents the average bits needed to represent each element of the vector. The result is showed in Fig.~\ref{fig:compare_quan_all}. Note that intuitively top-k methods should perform better than random-k method, but the top-k method needs extra bits to transmitted the index while random-k method can avoid this by using the same random seed. Therefore, top-k method doesn't outperform random-k too much under the same communication budget. The result in Fig.~\ref{fig:compare_quan_all} suggests that $\infty$-norm b-bits quantization provides significantly better compression precision than others under the same bit constraint.
\section{Experiments}
\subsection{Parameter sensitivity}
\label{sec:para_analysys}
In the linear regression problem, the convergence of LEAD under different parameter settings of $\alpha$ and $\gamma$ are tested. The result showed in Figure~\ref{fig:para_linear_regression} indicates that LEAD performs well in most settings and is robust to the parameter setting. Therefore, in this paper, we simply set $\alpha=0.5$ and $\gamma=1.0$ for LEAD in all experiment, which indicates the minor effort needed for parameter tuning.
\begin{figure*}[!ht]
% \vspace{-0.2in}
\centering
\subfloat[$\gamma=0.4$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/parameter/LR_sentivity_gamma0_4.pdf}
\vspace{-0.35in}
% \label{fig:para_g0.4}
\end{minipage}
}
\subfloat[$\gamma=0.6$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/parameter/LR_sentivity_gamma0_6.pdf}
\vspace{-0.35in}
% \label{fig:para_g0.6}
\end{minipage}
}
% \vspace{-0.1in}
\\
\subfloat[$\gamma=0.8$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/parameter/LR_sentivity_gamma0_8.pdf}
\vspace{-0.35in}
% \label{fig:para_g0.6}
\end{minipage}
}
\subfloat[$\gamma=1.0$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/parameter/LR_sentivity_gamma1.pdf}
\vspace{-0.35in}
% \label{fig:para_g1.0}
\end{minipage}
}
\caption{Parameter analysis on linear regression problem.}
% \end{center}
\label{fig:para_linear_regression}
\vspace{-5pt}
\end{figure*}
\subsection{Experiments in homogeneous setting}
\label{append:addition_experiment}
The experiments on logistic regression problem in homogeneous case are showed in Fig.~\ref{fig:logistic_homo_full} and Fig.~\ref{fig:logistic_homo_mini}. It shows that DeepSqueeze, CHOCO-SGD and LEAD converges similarly while DeepSqueeze and CHOCO-SGD require to tune a smaller $\gamma$ for convergence as showed in the parameter setting in Section~\ref{apend:para_setting}. Generally, a smaller $\gamma$ decreases the model propagation between agents since $\gamma$ changes the effective mixing matrix and this may cause slower convergence. However, in the setting where data from different agents are very similar, the models move to close directions such that the convergence is not affected too much.
\begin{figure*}[!ht]
\begin{center}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LogisticReg/lr_mnist_shuffleTrue_train_con_loss__epoch_bits2.pdf}
\vspace{-0.35in}
\end{minipage}
}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/LogisticReg/lr_mnist_shuffleTrue_train_con_loss__bits_bits2.pdf}
\vspace{-0.35in}
\end{minipage}
}
\end{center}
\vspace{-0.1in}
\caption{Logistic regression in the homogeneous case (full-batch gradient)}
\label{fig:logistic_homo_full}
\end{figure*}
\begin{figure*}[!ht]
\begin{center}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/Stochastic_LogisticReg/lr_mnist_stochastic_shuffleTrue_train_con_loss__epoch_bits2.pdf}
\vspace{-0.35in}
\end{minipage}
}
\subfloat[Loss $f(\overbar \vX^k)$]{
\begin{minipage}[t]{0.4\textwidth}
\centering
\includegraphics[width=1.0\textwidth]{iclr_fig/Stochastic_LogisticReg/lr_mnist_stochastic_shuffleTrue_train_con_loss__bits_bits2.pdf}
\vspace{-0.35in}
\end{minipage}
}
\end{center}
\vspace{-0.1in}
\caption{Logistic regression in the homogeneous case (mini-batch gradient)}
\label{fig:logistic_homo_mini}
\end{figure*}
\subsection{Parameter settings}
\label{apend:para_setting}
The best parameter settings we search for all algorithms and experiments are summarized in Tables~\ref{ls_para_table}--~\ref{deepnet_para_table}. QDGD and DeepSqueeze are more sensitive to $\gamma$ and CHOCO-SGD is slight more robust. LEAD is most robust to parameter settings and it works well for the setting $\alpha=0.5$ and $\gamma=1.0$ in all experiments in this paper.
\begin{table*}[!ht]
\centering
\begin{tabular}{|c|c|c|c|}\hline
Algorithm & $\eta$ & $\gamma$ & $\alpha$ \\\hline
DGD & $0.1$ & - & - \\\hline
NIDS & $0.1$ & - & - \\\hline
QDGD & $0.1$ & $0.2$ & - \\\hline
DeepSqueeze & $0.1$ & $0.2$ & - \\\hline
CHOCO-SGD & $0.1$ & $0.8$ & - \\\hline
LEAD & $0.1$ & $1.0$ & $0.5$ \\\hline
\end{tabular}
\caption{Parameter settings for the linear regression problem.}
\label{ls_para_table}
\end{table*}
\begin{table*}[!ht]
\begin{center}
\begin{minipage}[t]{0.45\textwidth}
\centering
\begin{tabular}{|c|c|c|c|}\hline
Algorithm & $\eta$ & $\gamma$ & $\alpha$ \\\hline
DGD & $0.1$ & - & - \\\hline
NIDS & $0.1$ & - & - \\\hline
QDGD & $0.1$ & $0.4$ & - \\\hline
DeepSqueeze & $0.1$ & $0.4$ & - \\\hline
CHOCO-SGD & $0.1$ & $0.6$ & - \\\hline
LEAD & $0.1$ & $1.0$ & $0.5$ \\\hline
\end{tabular}
\caption*{Homogeneous case}
\end{minipage}
\begin{minipage}[t]{0.45\textwidth}
\centering
\begin{tabular}{|c|c|c|c|}\hline
Algorithm & $\eta$ & $\gamma$ & $\alpha$ \\\hline
DGD & $0.1$ & - & - \\\hline
NIDS & $0.1$ & - & - \\\hline
QDGD & $0.1$ & $0.2$ & - \\\hline
DeepSqueeze & $0.1$ & $0.6$ & - \\\hline
CHOCO-SGD & $0.1$ & $0.6$ & - \\\hline
LEAD & $0.1$ & $1.0$ & $0.5$ \\\hline
\end{tabular}
\caption*{Heterogeneous case}
\end{minipage}
\end{center}
\caption{Parameter settings for the logistic regression problem (full-batch gradient).}
\label{lr_full_para_table}
\end{table*}
\begin{table*}[!ht]
\begin{center}
\begin{minipage}[t]{0.45\textwidth}
\centering
\begin{tabular}{|c|c|c|c|}\hline
Algorithm & $\eta$ & $\gamma$ & $\alpha$ \\\hline
DGD & $0.1$ & - & - \\\hline
NIDS & $0.1$ & - & - \\\hline
QDGD & $0.05$ & $0.2$ & - \\\hline
DeepSqueeze & $0.1$ & $0.6$ & - \\\hline
CHOCO-SGD & $0.1$ & $0.6$ & - \\\hline
LEAD & $0.1$ & $1.0$ & $0.5$ \\\hline
\end{tabular}
\caption*{Homogeneous case}
\end{minipage}
\begin{minipage}[t]{0.45\textwidth}
\centering
\begin{tabular}{|c|c|c|c|}\hline
Algorithm & $\eta$ & $\gamma$ & $\alpha$ \\\hline
DGD & $0.1$ & - & - \\\hline
NIDS & $0.1$ & - & - \\\hline
QDGD & $0.05$ & $0.2$ & - \\\hline
DeepSqueeze & $0.1$ & $0.6$ & - \\\hline
CHOCO-SGD & $0.1$ & $0.6$ & - \\\hline
LEAD & $0.1$ & $1.0$ & $0.5$ \\\hline
\end{tabular}
\caption*{Heterogeneous case}
\end{minipage}
\end{center}
\caption{Parameter settings for the logistic regression problem (mini-batch gradient).}
\label{lr_mini_para_table}
\end{table*}
\begin{table*}[!ht]
\begin{center}
\begin{minipage}[t]{0.45\textwidth}
\centering
\begin{tabular}{|c|c|c|c|}\hline
Algorithm & $\eta$ & $\gamma$ & $\alpha$ \\\hline
DGD & $0.1$ & - & - \\\hline
NIDS & $0.1$ & - & - \\\hline
QDGD & $0.05$ & $0.1$ & - \\\hline
DeepSqueeze & $0.1$ & $0.2$ & - \\\hline
CHOCO-SGD & $0.1$ & $0.6$ & - \\\hline
LEAD & $0.1$ & $1.0$ & $0.5$ \\\hline
\end{tabular}
\caption*{Homogeneous case}
\end{minipage}
\begin{minipage}[t]{0.45\textwidth}
\centering
\begin{tabular}{|c|c|c|c|}\hline
Algorithm & $\eta$ & $\gamma$ & $\alpha$ \\\hline
DGD & $0.05$ & - & - \\\hline
NIDS & $0.1$ & - & - \\\hline
QDGD & * & * & - \\\hline
DeepSqueeze & * & * & - \\\hline
CHOCO-SGD & * & * & - \\\hline
LEAD & $0.1$ & $1.0$ & $0.5$ \\\hline
\end{tabular}
\caption*{Heterogeneous case}
\end{minipage}
\end{center}
\caption{Parameter settings for the deep neural network. (* means divergence for all options we try)}
\label{deepnet_para_table}
\end{table*}
% \section{Appendix}
% \section{Theory}
\section{Proofs of the theorems}
\label{sec:allproof}
\subsection{Illustrative flow}\label{apdx:flow}
% The following flow graph depicts the relation between parameters and clarifies the range of conditional expectation.
The following flow graph depicts the relation between iterative variables and clarifies the range of conditional expectation. $\{\mathcal{G}_k\}_{k=0}^{\infty}$ and $\{\mathcal{F}_k\}_{k=0}^{\infty}$ are two $\sigma-$algebras generated by the gradient sampling and the stochastic compression respectively. They satisfy
$$\mathcal{G}_0\subset\mathcal{F}_0\subset\mathcal{G}_1\subset\mathcal{F}_1\subset\cdots\subset\mathcal{G}_k\subset\mathcal{F}_k\subset\cdots$$
%where $\EE_{\mathcal{G}}$ stands for the conditional expectation on stochastic gradient and $\EE_{\mathcal{F}}$ is on stochastic quantization method.
%\yl{we can replace $\nabla \vF(\vx)$ by $\vG$ and add one more sequence of $\sigma-$algebra $\mathcal{g}_k$}
\begin{tikzcd}
{(\vX^1, \vD^1, \vH^1)} \arrow[rd, "{\tiny\nabla \vF(\vX^1;\xi^1)\in\mathcal{G}_0}" description] & {(\vX^2, \vD^2, \vH^2)} \arrow[rd, "{\tiny \nabla \vF(\vX^2;\xi^2)\in\mathcal{G}_1}" description] & { (\vX^3, \vD^3, \vH^3)} \arrow[rd, "\cdots" description] & {(\vX^k, \vD^k, \vH^k)} \arrow[rd, "{\tiny \nabla \vF(\vX^k;\xi^k)\in\mathcal{G}_{k-1}}" description] & \cdots \\
& \vY^1 \arrow[u, "\tiny \vE^1" description] \arrow[r, "1\text{st round}"', dashed] \arrow[d, Rightarrow] & \vY^2 \arrow[u, "\tiny \vE^2" description] \arrow[d, Rightarrow] \arrow[r, "\cdots" description, dashed] & \vY^{k-1} \arrow[u, "\tiny \vE^{k-1}" description] \arrow[d, Rightarrow] \arrow[r, "(k-1)\text{th round}"', dashed] & \vY^{k} \arrow[u] \arrow[d, Rightarrow] \\
& \mathcal{F}_0 \arrow[r, "\subset"] & \mathcal{F}_1 \arrow[r, "\cdots" description] & \mathcal{F}_{k-2} \arrow[r, "\subset"] & \mathcal{F}_{k-1}
\end{tikzcd}
The solid and dashed arrows in the top flow illustrate the dynamics of the algorithm, while in the bottom, the arrows stand for the relation between successive $\mathcal{F}$-$\sigma$-algebras. The downward arrows determine the range of $\mathcal{F}$-$\sigma$-algebras. E.g., up to $\vE^k,$ all random variables are in $\mathcal{F}_{k-1}$ and up to $\nabla\vF(\vX^k;\xi^k)$, all random variables are in $\mathcal{G}_{k-1}$ with $\mathcal{G}_{k-1}\subset\mathcal{F}_{k-1}.$
%The conditional expectation of $\vE^k$ in the $k$th iteration is $\EE_{\mathcal{F}}\vE^k$ while $\EE_{\mathcal{G}}\nabla\vF(\vX^k,\xi^k)$ stands for the conditional expectation on the gradient sampling in the $k$th iteration.
Throughout the appendix, without specification, $\EE$ is the expectation conditioned on the corresponding stochastic estimators given the context.
\iffalse
\begin{tikzcd}
{(\vX^1, \vD^1, \vH^1)} \arrow[rd, "\nabla\vF(\vX^1)\in\mathcal{G}_0" description] & {(\vX^2, \vD^2, \vH^2)} \arrow[rd, "\nabla\vF(\vX^2)\in\mathcal{G}_1" description] & { (\vX^3, \vD^3, \vH^3)} \arrow[rd, "\cdots" description] & {(\vX^k, \vD^k, \vH^k)} \arrow[rd, "\nabla\vF(\vX^k)\in\mathcal{G}_{k-1}" description] & \cdots & \\
& \vY^1 \arrow[u, "\vE^1" description] \arrow[r, "1\text{st round}"', dashed] \arrow[d, Rightarrow] & \vY^2 \arrow[u, "\vE^2" description] \arrow[d, Rightarrow] \arrow[r, "\cdots" description, dashed] & \vY^{k-1} \arrow[u, "\vE^{k-1}" description] \arrow[d, Rightarrow] \arrow[r, "(k-1)\text{th round}"', dashed] & \vY^{k} \arrow[u] \arrow[d, Rightarrow] & \\
& \mathcal{F}_0 \arrow[r, "\subset"] & \mathcal{F}_1 \arrow[r, "\cdots" description] & \mathcal{F}_{k-2} \arrow[r, "\subset"] & \mathcal{F}_{k-1} &
\end{tikzcd}
\fi
% \begin{figure}[H]
% % \vspace{-20pt}
% \centering
% \begin{minipage}{.9\linewidth}
% \begin{algorithm}[H]
% \caption{The Proposed Algorithm: LEAD}
% \label{algorithm_1}
% \textbf{Input:} stepsize $\eta$, parameters ($\alpha,~\gamma$),~$\vX^0$,~$\vH^1$,~$\vD^1 = (\vI-\vW) \vZ$ for any $\vZ$\\
% \noindent\textbf{Output:} $\vX^K$
% \begin{algorithmic}[1]
% % \begin{algorithmic}
% %\setstretch{1.3}
% %\hspace*{\algorithmicindent} \textbf{Input: stepsize $\eta$, compression parameters ($\alpha,~\gamma$),~$\vX^0$,~$\vH^0$,~$\vZ^0$,~$\vD^1 = (\vI-\vW) \vZ^0$}
% \State $\vX^1 = \vX^0 - \eta \nabla \vF(\vX^0;\xi^0)$
% \For{$k=1,2,\dots, K-1$}
% \State $\vY^k = \vX^k-\eta \nabla \vF(\vX^k;\xi^k)-\eta \vD^k$
% % \STATE $\hat \vY^k = \vH^k + $\Call{Compress}{$\vY^k-\vH^k$}
% \State $\vQ^k = $\Call{Compress}{$\vY^k-\vH^k$}
% \State $\hat \vY^k = \vH^k + \vQ^k$
% \State $\vH^{k+1} = (1-\alpha)\vH^k + \alpha \hat \vY^k $
% \State $\vD^{k+1} = \vD^k + \frac{\gamma}{2\eta} (\vI-\vW)\hat \vY^k$ %\footnote{aaa}
% \State $\vX^{k+1} = \vX^k - \eta \nabla \vF(\vX^k;\xi^k) - \eta \vD^{k+1}$
% \EndFor
% \end{algorithmic}
% \end{algorithm}
% \end{minipage}
% \end{figure}
% % \vspace{-0.2in}
\subsection{Two central Lemmas}
% \begin{lemma}[Fundamental Equality of the algorithmic framework]
\begin{lemma}[Fundamental equality]
\label{thm:equality}
Let $\vX^*$ be the optimal solution, $\vD^*\coloneqq-\nabla\vF(\vX^*)$ and $\vE^k$ denote the compression error in the $k$th iteration, that is $\vE^k = \vQ^k-(\vY^k-\vH^k) = \hat \vY^k - \vY^k$. From Alg.~\ref{algocomplete}, we have
\begin{align}
& \|\vX^{k+1}-\vX^*\|^2 + ({\eta^2}/{\gamma}) \|\vD^{k+1}-\vD^*\|^2_\vM \nonumber \\
=& \|\vX^{k}-\vX^*\|^2 + ({\eta^2}/{\gamma}) \|\vD^{k}-\vD^*\|^2_\vM - ({\eta^2}/{\gamma}) \|\vD^{k+1}-\vD^k\|^2_\vM - \eta^2 \|\vD^{k+1}-\vD^*\|^2 \nonumber \\
&- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k;\xi^k)-\nabla \vF(\vX^*) \rangle + \eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2 + 2\eta \langle \vE^k, \vD^{k+1} - \vD^* \rangle,\nonumber
\end{align}
where $\vM\coloneqq 2(\vI-\vW)^{\dagger} - \gamma \vI$ and $\gamma<{2}/{\lambda_{\max}(\vI-\vW)}$ ensures the positive definiteness of $\vM$ over $\text{range}(\vI-\vW)$.
\end{lemma}
% \begin{lemma}[State inequality of the compression procedure]
\begin{lemma}[State inequality]
\label{thm:inequality}
Let the same assumptions in Lemma~\ref{thm:equality} hold. From Alg.~\ref{algocomplete}, if we take the expectation over the compression operator conditioned on the $k$-th iteration, we have
\begin{align}
\mathbb{E} \|\vH^{k+1} &-\vX^*\|^2 \leq (1-\alpha) \|\vH^{k}-\vX^*\|^2 + \alpha \mathbb{E}\|\vX^{k+1}-\vX^*\|^2 + \alpha \eta^2 \EE\|\vD^{k+1}-\vD^k\|^2 \nonumber \\ &+ \frac{2\alpha \eta^2}{\gamma} \EE\| \vD^{k+1}-\vD^k \|^2_{\vM}
+ \alpha^2 \mathbb{E} \|\vE^k\|^2 - \alpha\gamma \mathbb{E} \| \vE^k\|^2_{\vI-\vW} - \alpha(1-\alpha) \| \vY^k-\vH^k \|^2.\nonumber
\end{align}
\end{lemma}
\subsection{Proof of Lemma~\ref{thm:equality}}
\label{apdx:proof_lemma_equality}
Before proving Lemma~\ref{thm:equality}, we let $\vE^k =\hat \vY^k - \vY^k$ and introduce the following three Lemmas.
\begin{lemma}
Let $\vX^*$ be the consensus solution. %denotes the compression error which implies $\hat \vY^k = \vY^k + \vE^k$.
Then, from Line 4-7 of Alg.~\ref{algocomplete}, %algorithmic framework~\eqref{equ:frame_1}$\sim$\eqref{equ:frame_4},
we obtain
\begin{align}
\frac{\vI-\vW}{2\eta} (\vX^{k+1} - \vX^*) = \left(\frac{I}{\gamma} - \frac{\vI-\vW}{2}\right) (\vD^{k+1}-\vD^k) - \frac{\vI-\vW}{2\eta} \vE^k. \label{equ:equlity_1}
\end{align}
\end{lemma}
\begin{proof}From the iterations in Alg.~\ref{algocomplete}, we have
\begin{align}
\vD^{k+1}&=\vD^k+\frac{\gamma}{2\eta} (\vI-\vW) \hat \vY^k \qquad{(\text{from} \mbox{ Line 6})}\nonumber\\
&=\vD^k+\frac{\gamma}{2\eta} (\vI-\vW) (\vY^k+\vE^k) \nonumber\\
&=\vD^k+\frac{\gamma}{2\eta} (\vI-\vW) (\vX^k-\eta\nabla \vF(\vX^k;\xi^k) - \eta \vD^k + \vE^k) \qquad{(\text{from} \mbox{ Line 4})}\nonumber\\
&=\vD^k+\frac{\gamma}{2\eta} (\vI-\vW) (\vX^k-\eta\nabla \vF(\vX^k;\xi^k) -\eta \vD^{k+1} -\vX^* + \eta (\vD^{k+1}-\vD^k) + \vE^k)\nonumber\\
&=\vD^k+\frac{\gamma}{2\eta} (\vI-\vW) (\vX^{k+1} -\vX^*) + \frac{\gamma}{2} (\vI-\vW) (\vD^{k+1}-\vD^k) + \frac{\gamma}{2\eta} (\vI-\vW) \vE^k,\nonumber
\end{align}
where the fourth equality holds due to $(\vI-\vW)\vX^*=\vzero$ and the last equality comes from Line 7 of Alg.~\ref{algocomplete}. Rewriting this equality, and we obtain~\eqref{equ:equlity_1}.
\end{proof}
\begin{lemma}
\label{thm:xde_equality}
Let $\vD^*=-\nabla\vF(\vX^*)\in\Span\{\vI-\vW\}$, we have
\begin{align}
\langle \vX^{k+1}-\vX^*, \vD^{k+1} - \vD^k \rangle =& \frac{\eta}{\gamma} \|\vD^{k+1}-\vD^k \|^2_{\vM} - \langle \vE^k, \vD^{k+1} - \vD^k \rangle, \label{lemma5_a}\\
\langle \vX^{k+1}-\vX^*, \vD^{k+1} - \vD^* \rangle =& \frac{\eta}{\gamma} \langle \vD^{k+1}-\vD^k, \vD^{k+1} - \vD^* \rangle_{\vM} - \langle \vE^k, \vD^{k+1} - \vD^* \rangle,\label{lemma5_b}
\end{align}
where $\vM=2(\vI-\vW)^{\dagger} - \gamma \vI$ and $\gamma<
%\frac{2}{\beta}=
{2}/{\lambda_{\max}(\vI-\vW)}$ ensures the positive definiteness of $\vM$ over $\textnormal{span}\{\vI-\vW\}$.
\end{lemma}
\begin{proof}
Since $\vD^{k+1}\in\Span\{\vI-\vW\}$ for any $k$, we have
\begin{align*}
& \langle \vX^{k+1}-\vX^*, \vD^{k+1} - \vD^k \rangle \nonumber\\
=& \langle (\vI-\vW)(\vX^{k+1}-\vX^*), (\vI-\vW)^{\dagger} (\vD^{k+1} - \vD^k) \rangle \nonumber\\
=& \left\langle {\eta\over\gamma} ({2}\vI- \gamma(\vI-\vW))(\vD^{k+1}-\vD^k) - (\vI-\vW)\vE^k, (\vI-\vW)^{\dagger} (\vD^{k+1} - \vD^k)\right\rangle \qquad{(\text{from}~\eqref{equ:equlity_1})}\nonumber \\
=& \left\langle {\eta\over\gamma} (2(\vI-\vW)^{\dagger}- \gamma\vI\Big)(\vD^{k+1}-\vD^k) - \vE^k, \vD^{k+1} - \vD^k \right\rangle \nonumber\\
=& \frac{\eta}{\gamma} \|\vD^{k+1}-\vD^k \|^2_{\vM} - \langle \vE^k, \vD^{k+1} - \vD^k \rangle.
\end{align*}
Similarly, we have
\begin{align*}
& \langle \vX^{k+1}-\vX^*, \vD^{k+1} - \vD^* \rangle \\
=& \langle (\vI-\vW)(\vX^{k+1}-\vX^*), (\vI-\vW)^{\dagger} (\vD^{k+1} - \vD^*) \rangle \\
=& \left\langle {\eta\over\gamma} (2\vI- \gamma(\vI-\vW))(\vD^{k+1}-\vD^k) - (\vI-\vW)\vE^k, (\vI-\vW)^{\dagger} (\vD^{k+1} - \vD^*) \right\rangle \\
=& \left\langle {\eta\over\gamma} (2(\vI-\vW)^{\dagger}- \vI)(\vD^{k+1}-\vD^k) - \vE^k, \vD^{k+1} - \vD^* \right\rangle \\
=& \frac{\eta}{\gamma} \langle \vD^{k+1}-\vD^k, \vD^{k+1} - \vD^* \rangle_{\vM} - \langle \vE^k, \vD^{k+1} - \vD^* \rangle.
\end{align*}
To make sure that $\vM$ is positive definite over $\Span\{\vI-\vW\}$, we need $\gamma<%\frac{2}{\beta}=
{2}/{\lambda_{\max}(\vI-\vW)}.$
\end{proof}
\begin{lemma}
\label{thm:expect_error}
Taking the expectation conditioned on the compression in the $k$th iteration,
%\yl{Just a remark, in $k$th iteration, we generate $\vY^k, \vE^k, \vH^{k+1},\vD^{k+1}$ and $\vX^{k+1}$, taking expectation conditioned on the $k$th compression means we treat all variables generated before $\vE^k$ deterministic such as $\vY^{k}$, $\vX^k, \vH^k$ and $\vD^{k}$}
we have
\begin{align}
2\eta \mathbb{E} \langle \vE^k, \vD^{k+1} - \vD^* \rangle & = 2\eta \mathbb{E} \left\langle \vE^k, \vD^{k}+\frac{\gamma}{2\eta}(\vI-\vW)\vY^k + \frac{\gamma}{2\eta}(\vI-\vW) \vE^k - \vD^* \right\rangle \nonumber\\
& = \gamma \mathbb{E} \langle \vE^k, (\vI-\vW) \vE^k \rangle = \gamma \mathbb{E} \|\vE^k\|^2_{\vI-\vW}, \nonumber\\
2\eta \mathbb{E} \langle \vE^k, \vD^{k+1} - \vD^k \rangle & = 2\eta \mathbb{E} \left\langle \vE^k, \frac{\gamma}{2\eta}(\vI-\vW) \vY^k + \frac{\gamma}{2\eta}(\vI-\vW) \vE^k \right\rangle \nonumber\\
& = \gamma \mathbb{E} \langle \vE^k, (\vI-\vW) \vE^k \rangle = \gamma \mathbb{E} \|\vE^k\|^2_{\vI-\vW}. \nonumber
\end{align}
\end{lemma}
\begin{proof}
The proof is straightforward and omitted here.
\end{proof}
%%%%%%%%%%%% Lemma 1 %%%%%%%%%
\begin{proof}[Proof of Lemma~\ref{thm:equality}] From Alg.~\ref{algocomplete}, we have
\begin{align}
& 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k;\xi^k)-\nabla \vF(\vX^*) \rangle \nonumber\\
=& 2 \langle \vX^k-\vX^*, \eta \nabla \vF(\vX^k;\xi^k)-\eta\nabla \vF(\vX^*) \rangle \nonumber\\
=& 2 \langle \vX^k-\vX^*, \vX^k - \vX^{k+1} - \eta (\vD^{k+1} - \vD^*) \rangle \qquad {(\text{from} \mbox{ Line 7})}\nonumber\\
=& 2 \langle \vX^k-\vX^*, \vX^k - \vX^{k+1} \rangle - 2\eta \langle \vX^k-\vX^*, \vD^{k+1} - \vD^*\rangle \nonumber\\
=& 2 \langle \vX^k-\vX^*, \vX^k - \vX^{k+1}\rangle - 2\eta \langle \vX^k-\vX^{k+1}, \vD^{k+1} - \vD^*\rangle - 2\eta \langle \vX^{k+1}-\vX^*, \vD^{k+1} - \vD^*\rangle \nonumber\\
=& 2 \langle \vX^k-\vX^* - \eta (\vD^{k+1} - \vD^*) , \vX^k - \vX^{k+1} \rangle - 2\eta \langle \vX^{k+1}-\vX^*, \vD^{k+1} - \vD^* \rangle \nonumber\\
=& 2 \langle \vX^{k+1}-\vX^* + \eta (\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)), \vX^k - \vX^{k+1} \rangle - 2\eta \langle \vX^{k+1}-\vX^*, \vD^{k+1}-\vD^*\rangle \qquad {(\text{from} \mbox{ Line 7})}\nonumber\\
=& 2 \langle \vX^{k+1}-\vX^*, \vX^k - \vX^{k+1} \rangle + 2\eta \langle \nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*), \vX^k - \vX^{k+1} \rangle \nonumber \\
&- 2\eta \langle \vX^{k+1}-\vX^*, \vD^{k+1}-\vD^* \rangle. \label{lemma2_proof_1}
\end{align}
Then we consider the terms on the right hand side of~\eqref{lemma2_proof_1} separately.
Using $2\langle \vA-\vB, \vB-\vC \rangle = \|\vA-\vC\|^2 - \|\vB-\vC\|^2 - \|\vA-\vB\|^2$, we have
\begin{align}
2 \langle \vX^{k+1}-\vX^*, \vX^k - \vX^{k+1} \rangle
= & 2 \langle \vX^*-\vX^{k+1}, \vX^{k+1} - \vX^k \rangle \nonumber \\
= & \|\vX^{k} - \vX^* \|^2 - \|\vX^{k+1} - \vX^k \|^2 - \|\vX^{k+1} - \vX^* \|^2. \label{lemma2_proof_2a}
\end{align}
Using $2 \langle \vA, \vB \rangle = \|\vA\|^2 + \|\vB\|^2 - \|\vA-\vB\|^2$, we have
\begin{align}
& 2\eta \langle \nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*), \vX^k - \vX^{k+1} \rangle \nonumber\\
=& \eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2 + \|\vX^k - \vX^{k+1}\|^2 - \|\vX^k - \vX^{k+1} - \eta (\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*))\|^2 \nonumber\\
=& \eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2 + \|\vX^k - \vX^{k+1}\|^2 - \eta^2 \|\vD^{k+1} - \vD^*\|^2. \qquad {(\text{from} \mbox{ Line 7})}\label{lemma2_proof_2b}
\end{align}
Combining~\eqref{lemma2_proof_1},~\eqref{lemma2_proof_2a},~\eqref{lemma2_proof_2b}, and~\eqref{lemma5_a}, we obtain
\begin{align}
& 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k;\xi^k)-\nabla \vF(\vX^*) \rangle \nonumber\\
=& \underbrace{\|\vX^{k} - \vX^* \|^2 - \|\vX^{k+1} - \vX^k \|^2 - \|\vX^{k+1} - {\vX^*} \|^2}_{ 2\langle \vX^{k+1}-\vX^*, \vX^k - \vX^{k+1}\rangle} \nonumber\\
&+ \underbrace{\eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2 + \|\vX^k - \vX^{k+1}\|^2 - \eta^2 \|\vD^{k+1} - \vD^*\|^2}_{2\eta \langle \nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*), \vX^k - \vX^{k+1} \rangle} \nonumber\\
&-\underbrace{\Big( \frac{2\eta^2}{\gamma} \langle \vD^{k+1}-\vD^k, \vD^{k+1} - \vD^* \rangle_{\vM} - 2\eta \langle \vE^k,\vD^{k+1} - \vD^* \rangle \Big) }_{2\eta \langle \vX^{k+1}-\vX^*, \vD^{k+1} - \vD^* \rangle} \nonumber\\
=& \|\vX^{k} - \vX^* \|^2 - \|\vX^{k+1} - \vX^k \|^2 - \|\vX^{k+1} - \vX^* \|^2 \nonumber\\
&+\eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2 + \|\vX^k - \vX^{k+1}\|^2 - \eta^2 \|\vD^{k+1} - \vD^*\|^2 \nonumber\\
&+ \frac{\eta^2}{\gamma} \underbrace{\Big( \|\vD^{k} - \vD^* \|_\vM^2 - \|\vD^{k+1} - \vD^* \|_\vM^2 - \|\vD^{k+1} - \vD^k \|_\vM^2 \Big)}_{-2\langle \vD^{k+1}-\vD^k, \vD^{k+1} - \vD^* \rangle_{\vM}}
+ 2\eta \langle \vE^k, \vD^{k+1} - \vD^* \rangle, \nonumber
\end{align}
where the last equality holds because
\begin{align*}
2 \langle \vD^k-\vD^{k+1}, \vD^{k+1} - \vD^* \rangle_{\vM}
=& \|\vD^{k} - \vD^* \|_\vM^2 - \|\vD^{k+1} - \vD^* \|_\vM^2 - \|\vD^{k+1} - \vD^k \|_\vM^2.
\end{align*}
Thus, we reformulate it as
\begin{align*}
& \|\vX^{k+1}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \|\vD^{k+1}-\vD^*\|^2_\vM \nonumber \\
=& \|\vX^{k}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM - \frac{\eta^2}{\gamma} \|\vD^{k+1}-\vD^k\|^2_\vM - \eta^2 \|\vD^{k+1}-\vD^*\|^2 \nonumber \\
&- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k;\xi^k)-\nabla \vF(\vX^*) \rangle + \eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2 + 2\eta \langle \vE^k, \vD^{k+1} - \vD^* \rangle,
\end{align*}
which completes the proof.
\end{proof}
%%%%%%%%%%%% Lemma 2 %%%%%%%%%
\subsection{Proof of Lemma~\ref{thm:inequality}}
\label{apdx:proof_lemma_inequality}
\begin{proof}[Proof of Lemma~\ref{thm:inequality}] From Alg.~\ref{algocomplete}, we take the expectation conditioned on $k$th compression and obtain
\begin{align}
&\mathbb{E} \|\vH^{k+1}-\vX^*\|^2 \nonumber \\
=& \mathbb{E} \|(1-\alpha) (\vH^{k}-\vX^*) + \alpha (\vY^k-\vX^*) + \alpha \vE^k\|^2 \qquad{(\text{from} \mbox{ Line 13})}\nonumber\\
=& \|(1-\alpha) (\vH^{k}-\vX^*) + \alpha (\vY^k-\vX^*)\|^2 + \alpha^2 \mathbb{E} \|\vE^k\|^2 \nonumber\\
=& (1-\alpha) \|\vH^{k}-\vX^*\|^2 + \alpha \|\vY^k-\vX^*\|^2 - \alpha(1-\alpha) \|\vH^k-\vY^k\|^2 + \alpha^2 \mathbb{E} \|\vE^k\|^2. \label{equ:h_equ1} %\nonumber\\
%=& (1-\alpha) \|\vH^{k}-\vX^*\|^2 + \alpha \|\vY^k-\vX^*\|^2 - \alpha(1-\alpha) \|\vY^k-\vY^k\|^2 + \alpha^2\|\vE^k\|^2
% &\leq (1-\alpha) \|h^{k}-\vX^*\|^2 + \alpha \|y^k-\vX^*\|^2 - \alpha(1-\alpha) \|u^k\|^2 + \alpha^2(1-C)\|u^k\|^2 \\
% &= (1-\alpha) \|h^{k}-\vX^*\|^2 + \alpha \|y^k-\vX^*\|^2 - \alpha(1-\alpha(2-C)) \|u^k\|^2.
\end{align}
In the second equality, we used the unbiasedness of the compression, i.e., $\mathbb{E}\vE^k =\vzero$. The last equality holds because of
$$\|(1-\alpha)\vA + \alpha\vB\|^2 = (1-\alpha)\|\vA\|^2+\alpha\|\vB\|^2 - \alpha(1-\alpha)\|\vA-\vB\|^2.$$
In addition, by taking the conditional expectation on the compression, we have
\begin{align}
\|\vY^k-\vX^*\|^2 =& \|\vX^k-\eta \nabla \vF(\vX^k;\xi^k)-\eta \vD^k - \vX^*\|^2 \qquad{(\text{from} \mbox{ Line 4})}\nonumber \\
=&\EE\|\vX^{k+1} + \eta \vD^{k+1}- \eta \vD^k - \vX^*\|^2 \qquad{(\text{from} \mbox{ Line 7})}\nonumber\\
=&\EE\|\vX^{k+1}-\vX^*\|^2 + \eta^2 \EE\|\vD^{k+1}-\vD^k\|^2 + 2\eta \EE\langle \vX^{k+1}-\vX^*, \vD^{k+1}-\vD^k \rangle \nonumber\\
=&\EE\|\vX^{k+1}-\vX^*\|^2 + \eta^2 \EE\|\vD^{k+1}-\vD^k\|^2 \nonumber\\
& + \frac{2\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^k\|^2_{\vM} - 2\eta \EE\langle \vE^k, \vD^{k+1} - \vD^k \rangle. \qquad{(\text{from}~\eqref{lemma5_a})} \nonumber\\
=&\EE\|\vX^{k+1}-\vX^*\|^2 + \eta^2 \EE\|\vD^{k+1}-\vD^k\|^2 \nonumber\\
& + \frac{2\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^k\|^2_{\vM} - \gamma\EE\| \vE^k\|^2_{\vI-\vW}. \qquad{(\text{from} \mbox{ Line 6})} \label{equ:h_equ2}
\end{align}
% \begin{align}
% \mathbb{E} \|h^{k+1}-\vX^*\|^2 &\leq (1-\alpha) \|h^{k}-\vX^*\|^2 + \alpha \|\vX^{k+1}-\vX^*\|^2 + \alpha \eta^2 \|d^{k+1}-d^k\|^2 \\
% & + 2\alpha \eta^2 \| d^{k+1}-d^k \|^2_{M} - \alpha \mathbb{E} \langle e^k, (I-W) e^k \rangle - \alpha(1-\alpha(2-C)) \|u^k\|^2.
% \end{align}
Combing the above two equations~\eqref{equ:h_equ1} and~\eqref{equ:h_equ2} together, we have
% A tighter one:
\begin{align}
&\mathbb{E} \|\vH^{k+1}-\vX^*\|^2 \nonumber \\
\leq& (1-\alpha) \|\vH^{k}-\vX^*\|^2 + \alpha \mathbb{E}\|\vX^{k+1}-\vX^*\|^2 + \alpha \eta^2 \EE\|\vD^{k+1}-\vD^k\|^2 + \frac{2\alpha \eta^2}{\gamma}\EE \| \vD^{k+1}-\vD^k \|^2_{\vM} \nonumber\\
& - \alpha\gamma \mathbb{E} \| \vE^k\|^2_{\vI-\vW} + \alpha^2 \mathbb{E} \|\vE^k\|^2 - \alpha(1-\alpha) \|\vY^k-\vH^k\|^2,
\end{align}
which completes the proof.
\iffalse{\color{red}Where does the inequality come from?}\yl{we omit $\alpha\|\vY^k-\vX^*\|^2-\|\vY^k-\vX^*\|^2$ here and I clarify all conditional expectation where I set the expectation conditioned on random variable $\vE^k$ in $k$th iteration. Actually the filtration follows $\mathcal{F}_0\subset\mathcal{F}_1\subset\mathcal{F}_2\subset\cdots$ and
$$\underbrace{\underbrace{\underbrace{\vX^1,\vD^1,\vH^1,\vY^1}_{\mathcal{F}_0}\xrightarrow[]{\vE^1}\vH^2,\vD^2,\vX^2,\vY^2}_{\mathcal{F}_1}\xrightarrow[]{\vE^2}\vH^3,\vD^3,\vX^3,\vY^3}_{\mathcal{F}_2}\xrightarrow[]{\vE^3}\vH^4,\vD^4,\vX^4\cdots$$
and iteration follows
$$\vX^1,\vD^1,\vH^1,\overbrace{\vY^1\xrightarrow[]{\vE^1}\vH^2,\vD^2,\vX^2}^{1\text{st}\ \ \text{Iteration}},\overbrace{\vY^2 \xrightarrow[]{\vE^2}\vH^3,\vD^3,\vX^3}^{2\text{nd}\ \ \text{Iteration}},\overbrace{\vY^3 \xrightarrow[]{\vE^3}\vH^4,\vD^4,\vX^4}^{3\text{rd}\ \ \text{Iteration}}\cdots$$
}\fi
\end{proof}
%%%%%%%%%%%% main theorem %%%%%%%%%
\subsection{Proof of Theorem~\ref{thm:linear_convergence}}
\label{apdx:proof_linear_convergence}
\begin{proof}[Proof of Theorem~\ref{thm:linear_convergence}]
%{\color{blue}Proof: to show $\vX^k \rightarrow \vX^*, \vD^k \rightarrow \vD^* = -\nabla \vF(\vX^*), \vH^k \rightarrow \vX^*, \vY^k \rightarrow \vX^*$.}
Combining Lemmas~\ref{thm:equality},~\ref{thm:inequality}, and ~\ref{thm:expect_error}, we have the expectation conditioned on the compression satisfying
\begin{align}
& \mathbb{E}\|\vX^{k+1}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^*\|^2_{\vM} + a_1 \mathbb{E}\|\vH^{k+1}-\vX^*\|^2 \nonumber\\
\leq& \|\vX^{k}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM - \frac{\eta^2}{\gamma} \mathbb{E}\|\vD^{k+1}-\vD^k\|^2_\vM - \eta^2 \EE\|\vD^{k+1}-\vD^*\|^2 \nonumber\\
&- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k;\xi^k)-\nabla \vF(\vX^*) \rangle + \eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2 + \gamma \mathbb{E} \|\vE^k\|^2_{\vI-\vW} \nonumber\\
&+ a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2 + a_1 \alpha\mathbb{E} \|\vX^{k+1}-\vX^*\|^2 + a_1 \alpha \eta^2 \mathbb{E} \|\vD^{k+1}-\vD^k\|^2 \nonumber\\
&+ \frac{2a_1 \alpha \eta^2}{\gamma} \mathbb{E}\| \vD^{k+1}-\vD^k \|^2_{\vM} + a_1 \alpha^2 \mathbb{E} \|\vE^k\|^2 - a_1 \alpha \gamma \mathbb{E} \| \vE^k\|^2_{\vI-\vW} - a_1 \alpha(1-\alpha) \|\vY^k-\vH^k\|^2\nonumber\\
=& \underbrace{\|\vX^{k}-\vX^*\|^2 - 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k;\xi^k)-\nabla \vF(\vX^*) \rangle + \eta^2 \|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2}_{\cA} \nonumber\\
& + a_1 \alpha\mathbb{E} \|\vX^{k+1}-\vX^*\|^2+ \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM - \eta^2 \EE\|\vD^{k+1}-\vD^*\|^2\nonumber\\
&+ a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2 \underbrace{- (1-2a_1\alpha)\frac{\eta^2}{\gamma} \mathbb{E}\|\vD^{k+1}-\vD^k\|^2_\vM+ a_1 \alpha \eta^2 \mathbb{E} \|\vD^{k+1}-\vD^k\|^2}_{\cB} \nonumber\\
& + \underbrace{a_1 \alpha^2 \mathbb{E} \|\vE^k\|^2 +(1- a_1 \alpha) \gamma \mathbb{E} \| \vE^k\|^2_{\vI-\vW} - a_1 \alpha(1-\alpha) \|\vY^k-\vH^k\|^2}_{\cC}, \label{thm1_pf_1}
\end{align}
where $a_1$ is a non-negative number to be determined.
Then we deal with the three terms on the right hand side separately. We want the terms $\cB$ and $\cC$ to be nonpositive.
First, we consider $\cB$. Note that $\vD^k\in\Range{(\vI-\vW)}$. If we want $\cB\leq 0$, then, we need $1-2a_1\alpha> 0$, i.e., $a_1\alpha< 1/2$. Therefore we have
\begin{align*}
\cB = & - (1-2a_1\alpha)\frac{\eta^2}{\gamma} \mathbb{E}\|\vD^{k+1}-\vD^k\|^2_\vM+ a_1 \alpha \eta^2 \mathbb{E} \|\vD^{k+1}-\vD^k\|^2\\
\leq & \left(a_1\alpha-{(1-2a_1\alpha)\lambda_{n-1}(\vM)\over\gamma}\right) \eta^2\mathbb{E} \|\vD^{k+1}-\vD^k\|^2,
\end{align*}
where $\lambda_{n-1}(\vM)>0$ is the second smallest eigenvalue of $\vM$.
It means that we also need
$$a_1\alpha + \frac{(2a_1\alpha -1)\lambda_{n-1}(\vM)}{\gamma} \leq 0,$$
which is equivalent to
\begin{align}
a_1\alpha \leq {\lambda_{n-1}(\vM)\over \gamma + 2\lambda_{n-1}(\vM)}<1/2.
\end{align}
%{\color{red}what is $\beta$? $\lambda_{\max}(\vI-\vW)$?}\yl{Yes, the def of $\beta$ has been added in Thm1}
Then we look at $\cC$. We have
\begin{align}
\cC = & a_1 \alpha^2 \mathbb{E} \|\vE^k\|^2 +(1 - a_1 \alpha) \gamma \mathbb{E} \| \vE^k\|^2_{\vI-\vW} - a_1 \alpha(1-\alpha) \|\vY^k-\vH^k\|^2 \nonumber\\
\leq & ((1-a_1\alpha) \beta\gamma + a_1 \alpha^2 )\mathbb{E} \|\vE^k\|^2 - a_1 \alpha(1-\alpha) \|\vY^k-\vH^k \|^2 \nonumber\\
\leq & C((1-a_1\alpha) \beta \gamma + a_1 \alpha^2 ) \|\vY^k-\vH^k\|^2 - a_1 \alpha(1-\alpha) \|\vY^k-\vH^k\|^2 \nonumber
\end{align}
Because we have $1-a_1\alpha > 1/2$, so we need
\begin{align}
C((1-a_1\alpha) \beta\gamma + a_1 \alpha^2 ) - a_1 \alpha(1-\alpha)= (1+C)a_1 \alpha^2 - a_1(C\beta\gamma+1)\alpha + C\beta\gamma \leq 0.
\end{align}
That is
\begin{align}
&\alpha \geq \frac{a_1(C\beta\gamma+1) - \sqrt{a_1^2(C\beta\gamma+1)^2 - 4(1+C)Ca_1\beta\gamma}}{2(1+C)a_1} \eqqcolon \alpha_0,\\
&\alpha \leq \frac{a_1(C\beta\gamma+1) + \sqrt{a_1^2(C\beta\gamma+1)^2 - 4(1+C)Ca_1\beta\gamma}}{2(1+C)a_1} \eqqcolon \alpha_1.
\end{align}
% With smoothness and restricted strongly convex properties from Assumptions~\ref{ass:smooth}, we have $\|\nabla \vF(\vX^k) - \nabla \vF(\vX^*) \|^2 \geq \mu\|\vX^k-\vX^*\|^2$ and $L \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle\geq\|\nabla \vF(\vX^k)-\nabla\vF(\vX^*)\|^2$.
% Therefore,
% \begin{align}
% &- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle \nonumber\\
% \leq & -\eta^2 \|\nabla \vF(\vX^k) - \nabla \vF(\vX^*) \|^2 - \mu(2\eta-L\eta^2) \|\vX^k-\vX^*\|^2.\label{eq:innerp}
% \end{align}
% {\color{red} If the stepsie $\eta\leq 2/(\mu+L)$, we can use
% $$ \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle\geq{\mu L\over \mu+L}\|\vX^k-\vx^*\|^2+ {1\over \mu+L}\|\nabla \vF(\vX^k)-\nabla\vF(\vX^*)\|^2$$
% Then we will have
% \begin{align}
% &- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle \nonumber\\
% \leq & -\eta^2 \|\nabla \vF(\vX^k) - \nabla \vF(\vX^*) \|^2 - \mu(2\eta-\mu\eta^2) \|\vX^k-\vX^*\|^2.
% \end{align}
% I will add this tomorrow. --Yan
% }
Next, we look at $\cA$. Firstly, by the bounded variance assumption, we have the expectation conditioned on the gradient sampling in $k$th iteration satisfying
\begin{equation*}
\begin{aligned}
&\EE\|\vX^{k}-\vX^*\|^2 - 2\eta \EE\langle \vX^k-\vX^*, \nabla \vF(\vX^k;\xi^k)-\nabla \vF(\vX^*) \rangle + \eta^2 \EE\|\nabla \vF(\vX^{k};\xi^k) - \nabla \vF(\vX^*)\|^2\\
\leq&\|\vX^{k}-\vX^*\|^2 - 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle + \eta^2 \|\nabla \vF(\vX^{k}) - \nabla \vF(\vX^*)\|^2+n\eta^2\sigma^2
\end{aligned}
\end{equation*}
Then with the smoothness and strong convexity from Assumptions~\ref{ass:smooth}, we have the co-coercivity of $\nabla g_i(\vx)$ with $g_i(\vx):=f_i(\vx)-\frac{u}{2}\|\vx\|_2^2$, which gives
\begin{align*}
\langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle\geq{\mu L\over \mu+L}\|\vX^k-\vX^*\|^2+ {1\over \mu+L}\|\nabla \vF(\vX^k)-\nabla\vF(\vX^*)\|^2.
\end{align*}
When $\eta \leq 2/(\mu+L)$, we have
\begin{align*}
& \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle \\
=&\left(1-{\eta(\mu+L)\over2}\right)\langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle + {\eta(\mu+L)\over2}\langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle \\
\geq &\left(\mu-{\eta\mu(\mu+L)\over2}+{\eta\mu L\over 2}\right)\|\vX^k-\vX^*\|^2+ {\eta\over 2}\|\nabla \vF(\vX^k)-\nabla\vF(\vX^*)\|^2\\
=& \mu\left(1-{\eta\mu\over2}\right)\|\vX^k-\vX^*\|^2+ {\eta\over 2}\|\nabla \vF(\vX^k)-\nabla\vF(\vX^*)\|^2.
\end{align*}
Therefore, we obtain
\begin{align}
&- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle \nonumber\\
\leq & -\eta^2 \|\nabla \vF(\vX^k) - \nabla \vF(\vX^*) \|^2 - \mu(2\eta-\mu\eta^2) \|\vX^k-\vX^*\|^2.
\end{align}
Conditioned on the $k$the iteration, (i.e., conditioned on the gradient sampling in $k$th iteration), the inequality~\eqref{thm1_pf_1} becomes
\begin{align}
\label{equ:decay_inequality}
& \mathbb{E}\|\vX^{k+1}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^*\|^2_{\vM} + a_1 \mathbb{E}\|\vH^{k+1}-\vX^*\|^2 \nonumber\\
%\leq& \left(1- \mu(2\eta-\mu\eta^2)\right)\|\vX^{k}-\vX^*\|^2
%+ \left(\eta^2 - {2\eta\over \mu+L}\right)\|\nabla \vF(\vX^k)-\nabla\vF(\vX^*)\|^2
%+ a_1 \alpha\mathbb{E} \|\vX^{k+1}-\vX^*\|^2 \nonumber\\
%& + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM - \eta^2 \EE\|\vD^{k+1}-\vD^*\|^2 + a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2 \nonumber \\
\leq& \left(1- \mu(2\eta-\mu\eta^2)\right)\|\vX^{k}-\vX^*\|^2
+ a_1 \alpha\mathbb{E} \|\vX^{k+1}-\vX^*\|^2 \nonumber\\
& + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM - \eta^2 \EE\|\vD^{k+1}-\vD^*\|^2 + a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2+n\eta^2\sigma^2,
\end{align}
if the step size satisfies $\eta \leq {2\over \mu+L}$. Rewriting~\eqref{equ:decay_inequality}, we have
\begin{align}
& (1-a_1\alpha)\mathbb{E}\|\vX^{k+1}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^*\|^2_{\vM} + \eta^2 \EE\|\vD^{k+1}-\vD^*\|^2 + a_1 \mathbb{E}\|\vH^{k+1}-\vX^*\|^2 \nonumber\\
\leq& \left(1- \mu(2\eta-\mu\eta^2)\right)\|\vX^{k}-\vX^*\|^2
+ \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM + a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2+n\eta^2\sigma^2,
\end{align}
and thus
\begin{align}
& (1-a_1\alpha)\mathbb{E}\|\vX^{k+1}-\vX^*\|^2 + {\eta^2\over\gamma} \EE\|\vD^{k+1}-\vD^*\|^2_{\vM+\gamma\vI} + a_1 \mathbb{E}\|\vH^{k+1}-\vX^*\|^2 \nonumber\\
\leq& \left(1- \mu(2\eta-\mu\eta^2)\right)\|\vX^{k}-\vX^*\|^2
+ \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM + a_1(1-\alpha)\|\vH^{k}-\vX^*\|^2+n\eta^2\sigma^2.
\end{align}
% So the inequality~\eqref{thm1_pf_1} becomes
% \begin{align}
% & \mathbb{E}\|\vX^{k+1}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^*\|^2_{\vM} + a_1 \mathbb{E}\|\vH^{k+1}-\vX^*\|^2 \nonumber\\
% \leq& (1-\mu(2\eta-L\eta^2))\|\vX^{k}-\vX^*\|^2 + a_1 \alpha\mathbb{E} \|\vX^{k+1}-\vX^*\|^2 \nonumber\\
% & + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM - \eta^2 \EE\|\vD^{k+1}-\vD^*\|^2 + a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2.
% \end{align}
With the definition of $\cL^k$ in~\eqref{eqn_Lk}, we have
\begin{align}\label{ineq:L}
\EE\mathcal{L}^{k+1} \leq \rho \mathcal{L}^k + n\eta^2\sigma^2,
\end{align}
with
$$\rho =\max\left\{ \frac{1- \mu(2\eta-\mu\eta^2)}{1-a_1\alpha}, {\lambda_{\max}(\vM)\over\gamma+\lambda_{\max}(\vM)},1-\alpha\right\}.$$
where
$$\lambda_{\max}(\vM) = 2\lambda_{\max}((\vI-\vW)^{\dagger}) - \gamma.$$
Recall all the conditions on the parameters $a_1,~\alpha$, and $\gamma$ to make sure that $\rho<1$:
\begin{align}
a_1\alpha &\leq \frac{\lambda_{n-1}(\vM)}{\gamma + 2\lambda_{n-1}(\vM)}, \label{equ:lambdaM} \\
a_1\alpha &\leq \mu(2\eta-\mu\eta^2), \label{equ:stepsize}\\
\alpha &\geq \frac{a_1(C\beta\gamma+1) - \sqrt{a_1^2(C\beta\gamma+1)^2 - 4(1+C)C a_1\beta\gamma}}{2(1+C)a_1} \eqqcolon \alpha_0, \label{equ:alpha0} \\
\alpha &\leq \frac{a_1(C\beta\gamma+1) + \sqrt{a_1^2(C\beta\gamma+1)^2 - 4(1+C)C a_1\beta\gamma}}{2(1+C)a_1} \eqqcolon \alpha_1. \label{equ:alpha1}
\end{align}
In the following, we show that there exist parameters that satisfy these conditions.
\iffalse
Therefore, we obtain
\begin{align}
& \EE\|\vX^{k+1}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^*\|^2_\vM + a_1 \EE\|\vH^{k+1}-\vX^*\|^2 \nonumber\\
\leq& \|\vX^{k}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM - \frac{\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^k\|^2_\vM - \eta^2 \EE\|\vD^{k+1}-\vD^*\|^2 \nonumber\\
&- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle + \eta^2 \|\nabla \vF(\vX^{k}) - \nabla \vF(\vX^*)\|^2 \nonumber\\
&+ a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2 + a_1 \alpha \EE\|\vX^{k+1}-\vX^*\|^2 + a_1 \alpha \eta^2 \EE\|\vD^{k+1}-\vD^k\|^2 + \frac{2a_1 \alpha \eta^2}{\gamma} \EE \| \vD^{k+1}-\vD^k \|^2_{\vM}
\end{align}
With smoothness and restricted strongly convex properties from Assumptions~\ref{ass:smooth}, we have $\|\nabla \vF(\vX^k) - \nabla \vF(\vX^*) \|^2 \geq \mu\|\vX^k-\vX^*\|^2$ and $L \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle\geq\|\nabla \vF(\vX^k)-\nabla\vF(\vX^*)\|^2$.
Therefore,
\begin{align}\label{eq:innerp}
- 2\eta \langle \vX^k-\vX^*, \nabla \vF(\vX^k)-\nabla \vF(\vX^*) \rangle \leq -\eta^2 \|\nabla \vF(\vX^k) - \nabla \vF(\vX^*) \|^2 - \mu(2\eta-L\eta^2) \|\vX^k-\vX^*\|^2.
\end{align}
Combine~\eqref{eq:innerp}, we have
\begin{align}
& (1-a_1\alpha)\EE\|\vX^{k+1}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \EE\|\vD^{k+1}-\vD^*\|^2_{\vM+\gamma\vI} + a_1 \EE\|\vH^{k+1}-\vX^*\|^2 \nonumber\\
=& (1-\mu(2\eta-L\eta^2))\|\vX^{k}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM + a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2 \nonumber\\
& + a_1 \alpha \eta^2 \EE \|\vD^{k+1}-\vD^k\|^2 + \frac{(2a_1 \alpha-1)}{\gamma}\eta^2 \EE \| \vD^{k+1}-\vD^k \|^2_{\vM} \nonumber\\
\leq & (1-\mu(2\eta-L\eta^2))\|\vX^{k}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_\vM + a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2 \nonumber\\
%(1)\leq & (1-\mu(2\eta-L\eta^2))\|\vX^{k}-\vX^*\|^2 + \frac{\lambda_{\max}(\vM)}{1+\lambda_{\max}(\vM)}\frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_{\vM+\vI} + a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2 \\
%&\text{Or a tighter one} \\
\leq & (1-\mu(2\eta-L\eta^2))\|\vX^{k}-\vX^*\|^2 + \Big(1-\frac{\gamma}{\gamma+\lambda_{\max}(\vM)}\Big)\frac{\eta^2}{\gamma} \|\vD^{k}-\vD^*\|^2_{\vM+\gamma\vI} \nonumber\\
&+ a_1 (1-\alpha) \|\vH^{k}-\vX^*\|^2
\end{align}
if $2a_1\alpha -1 \leq 0 ~~(i.e., a_1\alpha \leq \frac{1}{2})$ and $a_1\alpha + \frac{(2a_1\alpha -1)\lambda_{n-1}(\vM)}{\gamma} \leq 0$, where $\lambda_{n-1}(\vM)>0$ is the second smallest eigenvalue of $\vM$.
Since $a_1\alpha \leq \frac{\lambda_{n-1}(\vM)}{\gamma+2\lambda_{n-1}(\vM)} \leq \frac{1}{2}$, $a_1\alpha \leq \frac{\lambda_{n-1}(\vM)}{\gamma+2\lambda_{n-1}(\vM)}$ is sufficient to guarantee above two conditions. Let $\tilde \lambda = %\lambda_{\max}[(\vM+\vI)^{-{1\over2}}\vM(\vM+\vI)^{-{1\over2}}]=
1-\frac{\gamma}{\gamma+\lambda_{\max}(\vM)}$
%\xr{TODO: prove it is less than 1}\yl{done, this bound is over $\Range(\vI-\vW)$, so we don't need to care $\lambda_{\min}(\vM)=-\gamma$. Also, $\vM+\gamma\vI$ is positive semidefinite on the whole space}.
and $\mathcal{L}^k = (1-a_1\alpha)\|\vX^{k}-\vX^*\|^2 + \frac{\eta^2}{\gamma} \|\vD^k-\vD^*\|_{\vM+\gamma\vI} + a_1\|\vH^k-\vX^*\|^2$ for simplicity , we have
\begin{align}\label{ineq:L}
\EE\mathcal{L}^{k+1} \leq \rho \mathcal{L}^k
\end{align}
where
% \begin{align}
% \rho = \max \{1-\frac{\mu(2\eta-L\eta^2)-a_1\alpha}{1-a_1\alpha}, 1-\frac{1}{1+\lambda_{\max}(M)}, 1-\alpha \}
% \end{align}
\begin{align}
\rho = \max \left\{1-\frac{\mu(2\eta-L\eta^2)-a_1\alpha}{1-a_1\alpha}, \tilde \lambda, 1-\alpha \right\}
\end{align}
In summary, to ensure linear convergence $\rho <1$, we need
\begin{align}
a_1\alpha &\leq \frac{\lambda_{n-1}(\vM)}{\gamma + 2\lambda_{n-1}(\vM)}, \label{equ:lambdaM} \\
a_1\alpha &\leq \mu\eta(2-L\eta), \label{equ:stepsize}\\
\alpha &\geq \frac{a_1(C\beta\gamma+1) - \sqrt{a_1^2(C\beta\gamma+1)^2 - 4(1+C)C a_1\beta\gamma}}{2(1+C)a_1} \eqqcolon \alpha_0, \label{equ:alpha0} \\
\alpha &\leq \frac{a_1(C\beta\gamma+1) + \sqrt{a_1^2(C\beta\gamma+1)^2 - 4(1+C)C a_1\beta\gamma}}{2(1+C)a_1} \eqqcolon \alpha_1. \label{equ:alpha1}
\end{align}
\else
\fi
% Equ~\eqref{equ:stepsize} suggests that the stepsize $\eta$ will have a lower bound because to cancel the compression error, $a_1\alpha $ cannot be infinitesimal~\eqref{equ:alpha0}. We also need to ensure
% $\alpha_0 \leq \frac{\lambda_{min}(M)}{1+2\lambda_{min}(M)}$~\eqref{equ:lambdaM}\eqref{equ:alpha0} which will have restriction on $C$. It suggests that we cannot get linear convergence with arbitrary compression precision.
% If there is no compression (C=1), then $\alpha_0=0 \leq \alpha \leq \alpha_1 =1$. If $\gamma \rightarrow 0$, then $0<\alpha<\frac{1}{2-C}$.
Since we can choose any $a_1$, we let
\begin{align*}
a_1=\frac{4(1+C)}{C\beta\gamma+2},
\end{align*} such that
\begin{align*}
a_1^2(C\beta\gamma+1)^2-4(1+C)Ca_1\beta\gamma = a_1^2.
\end{align*}
Then we have
\begin{align*}
\alpha_0=&\frac{C\beta\gamma}{2(1+C)}\rightarrow{0},\qquad \ \ \mbox{ as }\gamma\rightarrow{0}, \\
\alpha_1=&\frac{C\beta\gamma +2}{2(1+C)}\rightarrow{\frac{1}{1+C}},\ \mbox{ as }\gamma\rightarrow{0}.
\end{align*}
%Note, if $C=0$, we have $a_1 =2$, which makes $\alpha_0=0$ and $ \alpha_1 =1.$\footnote{Actually, we can take any $a_1$, but choosing $a_1=2$ simplifies the formula.} So $\alpha$ can take any value in $[0,1].$ %\yl{If $C=0$, when $\alpha=0$, we do not update the reference $\vH$; when $\alpha=1,$ we set $\vH=\vY$.}
Conditions~\eqref{equ:alpha0} and~\eqref{equ:alpha1} show
\begin{align*}
a_1\alpha\in\left[\frac{2C\beta\gamma}{C\beta\gamma+2}, 2\right]\rightarrow{[0,2]},\ \text{if}\ \ C=0 \ \text{or}\ \gamma\rightarrow{0}.
\end{align*}
Hence in order to make~\eqref{equ:lambdaM} and~\eqref{equ:stepsize} satisfied, it's sufficient to make
\beq\label{bound1}\frac{2C\beta\gamma}{C\beta\gamma+2}\leq\min\left\{\frac{\lambda_{n-1}(\vM)}{\gamma+2\lambda_{n-1}(\vM)}, \mu(2\eta-\mu\eta^2)\right\}=\min\left\{\frac{\frac{2}{\beta}-\gamma}{\frac{4}{\beta}-\gamma}, \mu(2\eta-\mu\eta^2)\right\}.\eeq
where we use $\lambda_{n-1}(\vM) = \frac{2}{\lambda_{\max}(\vI-\vW)}-\gamma = \frac{2}{\beta}-\gamma.$
%and
%\beq\label{bound2}\frac{2C\beta\gamma}{C\beta\gamma+2}\leq \mu\eta(2-L\eta),\eeq
%for fixed $\eta\in(0,\frac{2}{L}).$
When $C>0$, the condition~\eqref{bound1} is equivalent to
%$$\gamma\leq\frac{2\lambda_{n-1}(M)}{(2+3\lambda_{n-1}(M))(1-C)\beta}=\frac{2(\frac{2}{\beta}-\gamma)}{2+3(\frac{2}{\beta}-\gamma)(1-C)\beta},$$
\begin{equation}
\gamma\leq\min\left\{\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta},\frac{2 \mu\eta(2-\mu\eta)}{[2- \mu\eta(2-\mu\eta)]C\beta}\right\}.\label{eqn:gamma_bdd}
\end{equation}
% {\color{red}It is better to separately consider $C=0$ with the case of $C>0$. Because we will have $0/0$ in the result of Theorem 1 when $C=0$. We can just briefly mention the steps in the proof of Corollary 2. }
The first term can be simplified using
$$\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta} \geq \frac{2}{(3C+1)\beta} $$
due to $\sqrt{1-x}\leq 1-\frac{x}{2}$ when $x\in(0,1).$
Therefore, for a given stepsize $\eta$, if we choose
% \begin{align*}
% \gamma \in \left(0,\min\Big\{\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta},\frac{2 \mu\eta(2-\mu\eta)}{[2- \mu\eta(2-\mu\eta)]C\beta}, \frac{2}{\beta}\Big\}\right)
% \end{align*}
% \begin{align*}
% \gamma \in \left(0,\min\Big\{\frac{2}{(3C+1)\beta}, \frac{2 \mu\eta(2-\mu\eta)}{[2- \mu\eta(2-\mu\eta)]C\beta}, \frac{2}{\beta}\Big\}\right)
% \end{align*}
\begin{align*}
\gamma \in \left(0,\min\Big\{\frac{2}{(3C+1)\beta}, \frac{2 \mu\eta(2-\mu\eta)}{[2- \mu\eta(2-\mu\eta)]C\beta} \Big\}\right)
\end{align*}
and
\begin{align*}\alpha \in \left[\frac{C\beta\gamma}{2(1+C)},\min\Big\{\frac{C\beta\gamma +2}{2(1+C)}, \frac{2-\beta\gamma}{4-\beta\gamma}\frac{C\beta\gamma+2}{4(1+C)}, \mu\eta(2-\mu\eta)\frac{C\beta\gamma+2}{4(1+C)}\Big\}\right],
\end{align*}
then, all conditions~\eqref{equ:lambdaM}-\eqref{equ:alpha1} hold.
Note that $\gamma < \frac{2}{(3C+1)\beta}$ implies $\gamma<\frac{2}{\beta}$, which ensures the positive definiteness of $\vM$ over $\Span\{\vI-\vW\}$ in Lemma~\ref{thm:xde_equality}.
% {2}/{\lambda_{\max}(\vI-\vW)}.$
Note that $\eta\leq\frac{2}{\mu+L}$ ensures
%\beq \frac{2-\beta\gamma}{4-\beta\gamma}\frac{C\beta\gamma+2}{4(1+C)}\leq\frac{1}{4}\frac{C\beta\gamma +2}{2(1+C)}\eeq
%and
\beq\mu\eta(2-\mu\eta)\frac{C\beta\gamma+2}{4(1+C)}\leq \frac{C\beta\gamma +2}{2(1+C)}.\eeq
So, we can simplify the bound for $\alpha$ as
\begin{align*}
\alpha \in \left[\frac{C\beta\gamma}{2(1+C)},\min\Big\{\frac{2-\beta\gamma}{4-\beta\gamma}\frac{C\beta\gamma+2}{4(1+C)}, \mu\eta(2-\mu\eta)\frac{C\beta\gamma+2}{4(1+C)}\Big\}\right].
\end{align*}
Lastly, taking the total expectation on both sides of~\eqref{ineq:L} and using tower property, we complete the proof for $C>0$.
% \xr{When $C=0$, the condition~\eqref{bound1} is satisfied when $\gamma<2/\beta$ and $\eta \leq 2/(\mu+L)$.
% So, the upper bound of $\gamma$ in~\eqref{eqn:gamma_bdd} holds if we let $\frac{(6C+2)-\sqrt{(6C+2)^2-16C}}{2C\beta}=2/\beta$, which is the limit when $C\rightarrow 0$.
% Because $\beta<2$, we can choose $\gamma=1$, and no tuning parameter is needed to control the communication.}
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \begin{proof}[Proof of Corollary~\ref{thm:kappa_dominate}]
% We consider the case when $\kappa$ dominates. We can choose the stepsize $\eta=\frac{1}{L}$ such that
% $$\frac{2 \mu\eta(2-\mu\eta)}{[2- \mu\eta(2-\mu\eta)]C\beta} = \frac{4\kappa -1}{(2\kappa^2-2\kappa+1)C\beta} \propto O(\frac{1}{ C\beta \kappa})$$
% We can choose $\gamma$ in the order $O(\frac{1}{ C\beta \kappa})$ and a conservative choice for $\alpha$ is the lower bound $\frac{C\beta\gamma}{2(1+C)} \propto O(\frac{1}{(1+C)\kappa})$.
% Then we look at the three terms in the linear rate $\rho$.
% Note that $a_1\alpha =\frac{2C\beta\gamma}{C\beta\gamma+2} \propto O(C\beta\gamma) \propto O(\frac{1}{\kappa})$, so the first term is
% $$\frac{1-\mu\eta(2-\mu\eta)}{1-a_1\alpha} \propto \frac{1-\frac{1}{\kappa}(2-\frac{1}{\kappa})}{1-\frac{1}{\kappa}} = 1-\frac{1-\frac{1}{\kappa}}{\kappa-1} \propto 1-O(\frac{1}{\kappa}).
% $$
% The second term, $1-{\gamma\over 2\lambda_{\max}((\vI-\vW)^\dagger)}$, and the third term, $1-\alpha$, clearly provide bounds in the order $1-O(\frac{1}{C\kappa})$ and $1-O(\frac{1}{(1+C)\kappa})$, respectively.
% Therefore, the maximum of the three terms give a bound for $\rho$ in the order $1-O(\frac{1}{(1+C)\kappa})$.
% \end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{proof}[Proof of Corollary~\ref{thm:complexity_bound}]
Let's first define $\kappa_f = \frac{L}{\mu}$ and $\kappa_g = \frac{\lambda_{\max}(\vI-\vW)}{\lambda_{\min}^{+}(\vI-\vW)}=\lambda_{\max}(\vI-\vW)\lambda_{\max}((\vI-\vW)^{\dagger})$.
We can choose the stepsize $\eta=\frac{1}{L}$ such that the upper bound of $\gamma$ is
$$\begin{aligned}
% \gamma_{\text{upper}}=&\min\Big\{\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta},\frac{\frac{2}{\kappa_f}\left(2-\frac{1}{\kappa_f}\right) }{\left[2-\frac{1}{\kappa_f}\left(2-\frac{1}{\kappa_f}\right)\right]C\beta}, \frac{2}{\beta}\Big\} \\
\gamma_{\text{upper}}=&\min\Big\{\frac{2}{(3C+1)\beta},\frac{\frac{2}{\kappa_f}\left(2-\frac{1}{\kappa_f}\right) }{\left[2-\frac{1}{\kappa_f}\left(2-\frac{1}{\kappa_f}\right)\right]C\beta}, \frac{2}{\beta}\Big\}
\geq \min\left\{\frac{2}{(3C+1)\beta},\frac{1}{\kappa_f C\beta}\right\},
\end{aligned}$$
due to
% $\sqrt{1-x}\leq 1-\frac{x}{2}$ and
$\frac{x(2-x)}{2-x(2-x)}\geq\frac{x}{2-x}\geq x$ when $x\in(0,1).$
% $$\frac{2 \mu\eta(2-\mu\eta)}{[2- \mu\eta(2-\mu\eta)]C\beta} = \frac{4\kappa -1}{(2\kappa^2-2\kappa+1)C\beta} \propto O(\frac{1}{ C\beta \kappa}). $$
% Let $\eta=\mathcal{O}\left(\frac{1}{L}\right)$ and we pick the upper bound of $\gamma$ $$\gamma=\min\left\{\mathcal{O}\left(\frac{1}{(1+C)\beta}\right),\mathcal{O}\left(\frac{1}{\kappa_f C\beta}\right)\right\}.$$
Hence we can take $\gamma=\min \{\frac{1}{(3C+1)\beta}, \frac{1}{\kappa_f C\beta}\}$.
The bound of $\alpha$ is
$$\alpha\in\left[\frac{C\beta\gamma}{2(1+C)},\min\left\{\frac{2-\beta\gamma}{4-\beta\gamma}\frac{C\beta\gamma+2}{4(1+C)}, \frac{1}{\kappa_f}(2-\frac{1}{\kappa_f}) \frac{C\beta\gamma+2}{4(1+C)} \right \}\right]$$
% When $\gamma$ is chosen as $\frac{1}{\kappa_f C\beta}$, pick $$\alpha = \frac{C\beta\gamma}{2(1+C)}= \frac{1}{2(1+C)\kappa_f}.$$
% When $\gamma$ is chosen as $\frac{2}{(3C+1)\beta}$, the upper bound of $\alpha$ is
% \begin{align*}
% \alpha_{\text{upper}} &=\min\left\{\frac{2-\beta\gamma}{4-\beta\gamma}\frac{C\beta\gamma+2}{4(1+C)}, \frac{1}{\kappa_f}(2-\frac{1}{\kappa_f}) \frac{C\beta\gamma+2}{4(1+C)} \right\} \\
% &=\min\left\{ \frac{3C}{(6C+1)}, \frac{1}{\kappa_f}(2-\frac{1}{\kappa_f}) \right\} \frac{4C+1}{2(C+1)(3C+1)} \\
% &\geq \min\left\{ \frac{3C}{(6C+1)}, \frac{1}{\kappa_f} \right\} \frac{4C+1}{2(C+1)(3C+1)}
% \end{align*}
% We pick $\alpha=\min\left\{ \frac{3C}{(6C+1)}, \frac{1}{\kappa_f} \right\} \frac{4C+1}{2(C+1)(3C+1)} =\mathcal{O}\left(\frac{1}{(1+C)\kappa_f}\right).$
When $\gamma$ is chosen as $\frac{1}{\kappa_f C\beta}$, pick
\begin{align}
\label{equ:alpha_pick1}
\alpha = \frac{C\beta\gamma}{2(1+C)}= \frac{1}{2(1+C)\kappa_f}.
\end{align}
When $\frac{1}{(3C+1)\beta} \leq \frac{1}{\kappa_f C\beta}$, %we can choose
%$\gamma = \frac{1}{(3C+1)\beta} < \frac{2}{(3C+1)\beta}$,
the upper bound of $\alpha$ is
\begin{align*}
\alpha_{\text{upper}} &=\min\left\{\frac{2-\beta\gamma}{4-\beta\gamma}\frac{C\beta\gamma+2}{4(1+C)}, \frac{1}{\kappa_f}(2-\frac{1}{\kappa_f}) \frac{C\beta\gamma+2}{4(1+C)} \right\} \\
&=\min\left\{ \frac{6C+1}{12C+3}, \frac{1}{\kappa_f} (2-\frac{1}{\kappa_f} ) \right\} \frac{7C+2}{4(C+1)(3C+1)} \\
&\geq \min\left\{ \frac{6C+1}{12C+3}, \frac{1}{\kappa_f} \right\} \frac{7C+2}{4(C+1)(3C+1)}.
\end{align*}
In this case, we pick
\begin{align}
\label{equ:alpha_pick2}
\alpha=\min\left\{ \frac{6C+1}{12C+3}, \frac{1}{\kappa_f} \right\} \frac{7C+2}{4(C+1)(3C+1)}.
\end{align}
Note $\alpha = \mathcal{O}\left(\frac{1}{(1+C)\kappa_f}\right)$ since $\frac{6C+1}{12C+3}$ is lower bounded by $1\over 3$.
Hence in both cases (Eq.~\eqref{equ:alpha_pick1} and Eq.~\eqref{equ:alpha_pick2}), $\alpha=\mathcal{O}\left(\frac{1}{(1+C)\kappa_f}\right)$, and the third term of $\rho$ is upper bounded by
% $$1-\alpha=1-\mathcal{O}\left(\frac{1}{(1+C)\kappa_f}\right).$$
%$1-\mathcal{O}\left(\frac{1}{(1+C)\kappa_f}\right)$.
$$\begin{aligned}
1-\alpha\leq\max\left\{1-\frac{1}{2(1+C)\kappa_f},1-\min\left\{\frac{6C+1}{12C+3},\frac{1}{\kappa_f}\right\}\frac{7C+2}{4(1+C)(3C+1)}\right\}\\
\end{aligned}
$$
% \xr{Here since we choose $\gamma=\frac{2}{(3C+1)\beta}$, $2-\beta\gamma$ will give a $C$ in the numerator such that when $C\rightarrow 0$, we can't choose $\frac{1}{\kappa_f}$ as $\alpha$. Maybe we can try a smaller $\gamma$ such as $\gamma=\frac{1}{(3C+1)\beta}$. In this case, we have
% $$
% \alpha=\min\left\{ \frac{6C+1}{(12C+3)}, \frac{1}{\kappa_f} \right\} \frac{7C+2}{4(C+1)(3C+1)} =\mathcal{O}\left(\frac{1}{(1+C)\kappa_f}\right).$$
% and
% $$a_1\alpha = \min\left\{ \frac{6C+1}{(12C+3)}, \frac{1}{\kappa_f} \right\} =\mathcal{O}\left(\frac{1}{\kappa_f}\right) $$
% since $\frac{6C+1}{(12C+3)} \geq 1/3.$}
In two cases of $\gamma$, the second term of $\rho$ becomes
\begin{align*}
1-\frac{\gamma}{2\lambda_{\max}((\vI-\vW)^\dagger)}&=\max\left\{1-\frac{1}{2C\kappa_f\kappa_g}, 1-\frac{1}{(1+3C)\kappa_g} \right\} \\
%&= \max\left\{1-\mathcal{O}\left(\frac{1}{C\kappa_f\kappa_g}\right),1-\mathcal{O}\left(\frac{1}{(1+C)\kappa_g}\right)\right\}.
\end{align*}
Before analysing the first term of $\rho$, we look at $a_1\alpha$ in two cases of $\gamma$. When $\gamma=\frac{1}{\kappa_f C\beta}$,
$$a_1\alpha = \frac{2C\beta\gamma}{C\beta\gamma+2} = \frac{2}{2\kappa_f+1} \leq \frac{1}{\kappa_f}.$$
When $\gamma=\frac{1}{(3C+1)\beta}$,
$$a_1\alpha = \min\left\{ \frac{6C+1}{(12C+3)}, \frac{1}{\kappa_f} \right\} \leq\frac{1}{\kappa_f}.$$
% since $\frac{6C+1}{(12C+3)} \geq 1/3.$
% $$a_1\alpha = \min\{ \frac{3C}{6C+1}, \frac{1}{\kappa_f} \}.$$
% In both cases, $a_1\alpha = \mathcal{O}\left ( {\frac{1}{\kappa_f}} \right )$.
In both cases, $a_1\alpha \leq \frac{1}{\kappa_f}$.
Therefore, the first term of $\rho$ becomes
$$\frac{1-\mu\eta(2-\mu\eta)}{1-a_1\alpha} \leq \frac{1-\frac{1}{\kappa_f}(2-\frac{1}{\kappa_f})}{1-\frac{1}{\kappa_f}} = 1-\frac{1-\frac{1}{\kappa_f}}{\kappa_f-1} = 1-\frac{1}{\kappa_f}.$$
% \xr{I am not sure whether the above approximation is correct.}
% $$a_1\alpha\in\left[\frac{2C\beta\gamma}{C\beta\gamma+2},\min\left\{\frac{2-\beta\gamma}{4-\beta\gamma}, \mu\eta(2-\mu\eta)\right\}\right].$$
% Then $a_1\alpha =\mathcal{O}\left(\frac{1}{1+\kappa_f}\right)$ or $\min\left\{\mathcal{O}\left(1\right),\mathcal{O}\left(\frac{1}{\kappa_f}\right)\right\}=\mathcal{O}\left(\frac{1}{\kappa_f}\right).$
% Hence the first term of $\rho$ is $1-\mathcal{O}\left(\frac{1}{\kappa_f}\right)$ and
To summarize, we have
$$\begin{aligned}
\rho &\leq 1- \min\left\{ \frac{1}{\kappa_f},\frac{1}{2C\kappa_f\kappa_g}, \frac{1}{(1+3C)\kappa_g},\frac{1}{2(1+C)\kappa_f},\min\left\{\frac{6C+1}{12C+3},\frac{1}{\kappa_f}\right\}\frac{7C+2}{4(1+C)(3C+1)}\right\}\\
%&\coloneqq 1-\tilde\rho
\end{aligned}
$$
and therefore
$$\rho = \max \left\{1-\mathcal{O}\Big(\frac{1}{(1+C)\kappa_f}\Big), 1-\mathcal{O}\Big(\frac{1}{(1+C)\kappa_g}\Big), 1-\mathcal{O}\Big(\frac{1}{C\kappa_f \kappa_g }\Big)\right\}.$$
%Therefore, if $k\geq\frac{1}{\log\frac{1}{1-\tilde{\rho}}}\log\frac{\mathcal{L}^0}{n\epsilon}\geq\left(\frac{1}{\tilde{\rho}}-1\right)\log\frac{\mathcal{L}^0}{n\epsilon}$, we get $\epsilon$-accuracy solution.
%Note that
%$$\frac{1}{\tilde\rho}$$
With full-gradient (i.e., $\sigma=0$), we get $\epsilon-$accuracy solution with the total number of iterations
$$k\geq\widetilde{\mathcal{O}}((1+C)(\kappa_f+\kappa_g)+C\kappa_f\kappa_g).$$
When $C=0$, i.e., there is no compression, the iteration complexity recovers that of NIDS, $\widetilde{\mathcal{O}}\left(\kappa_f+\kappa_g\right).$
When $C\leq\frac{\kappa_f+\kappa_g}{\kappa_f\kappa_g+\kappa_f+\kappa_g},$ the complexity is improved to that of NIDS, i.e., the compression doesn't harm the convergence in terms of the order of the coefficients.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{proof}[Proof of Corollary~\ref{thm:consensus}]
Note that $(\overbar{\vx}^k)^\top=\overbar{\vX}^k$ and $\vone_{n\times 1}\overbar{\vX}^*=\vX^*$, then
%\begin{equation}
\begin{align}
\sum_{i=1}^n\EE\|\vx^k_i-\overbar{\vx}^k\|^2&=\EE\left\|\vX^k-\vone_{n\times 1}\overbar{\vX}^k\right\|^2\nonumber\\
&=\EE\left\|\vX^k-\vX^*+\vX^*-\vone_{n\times 1}\overbar{\vX}^k\right\|^2\nonumber\\
&=\EE\left\|\vX^k-\vX^*-\frac{\vone_{n\times 1}\vone_{n\times 1}^\top}{n}\Big(\vX^k-\vX^*\Big)\right\|\nonumber\\
&\leq\EE\|\vX^k-\vX^*\|^2\nonumber\\
&\leq \frac{\rho\EE\mathcal{L}^{k-1}+ n \eta^2\sigma^2(1-\rho)^{-1}}{1-a_1\alpha}\nonumber\\
&\leq 2\rho^{k}\mathcal{L}^0+2\frac{ n\eta^2\sigma^2}{1-\rho}.
\end{align}
%\end{equation}
The last inequality holds because we have $a_1\alpha\leq{1}/{2}.$
\end{proof}
\begin{proof}[Proof of Corollary~\ref{thm:reduce_nids}]
\label{proof:consistent_nids}
From the proof of Theorem~\ref{thm:linear_convergence}, when $C=0,$ we can set $\gamma=1$, $\alpha=1$, and $a_1=0$. Plug those values into $\rho$, and we obtain the convergence rate for NIDS.
%implies that for any constant stepsize $\eta \in (0,\frac{2}{L})$, we can set such that
\end{proof}
\subsection{Proof of Theorem~\ref{thm:diminish}}
\begin{proof}[Proof of Theorem~\ref{thm:diminish}]
In order to get exact convergence, we pick diminishing step-size, set $\alpha=\frac{C\beta\gamma}{2(1+C)}$, $a_1\alpha=\frac{2C\beta\gamma_k}{C\beta\gamma_k+2}$, $\theta_1=\frac{1}{2\lambda_{\max}((\vI-\vW)^\dagger)}$ and $\theta_2=\frac{C\beta}{2(1+C)},$ then
$$\rho_k =\max\left\{ 1-\frac{\mu\eta_k(2-\mu\eta_k)-a_1\alpha}{1-a_1\alpha},
% {2\lambda_{\max}((\vI-\vW)^\dagger)-\gamma\over 2\lambda_{\max}((\vI-\vW)^\dagger)},
1-\theta_1\gamma_k,
1-\theta_2\gamma_k\right\}$$
If we further pick diminishing $\eta_k$ and $\gamma_k$ such that $\mu\eta_k(2-\mu\eta_k)-a_1\alpha\geq a_1\alpha,$ then
$$\frac{\mu\eta_k(2-\mu\eta_k)-a_1\alpha}{1-a_1\alpha}\geq\frac{a_1\alpha}{1-a_1\alpha}=\frac{2C\beta\gamma_k}{2-C\beta\gamma_k}\geq C\beta\gamma_k.$$
Notice that $C\beta\gamma\leq\frac{2}{3}$ since $(3C+1)-\sqrt{(3C+1)^2-4C}$ is increasing in $C>0$ with limit $\frac{2}{3}$ at $\infty$.
In this case we only need,
\begin{align}
\gamma_k &\in \left(0,\min\Big\{\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta},\frac{2\mu\eta_k(2-\mu\eta_k) }{[4-\mu\eta_k(2-\mu\eta_k)]C\beta}, \frac{2}{\beta}\Big\}\right).
\end{align}
And
$$\rho_k \leq\max\left\{ 1-C\beta\gamma_k,
% {2\lambda_{\max}((\vI-\vW)^\dagger)-\gamma\over 2\lambda_{\max}((\vI-\vW)^\dagger)},
1-\theta_1\gamma_k,
1-\theta_2\gamma_k\right\}\leq1-\theta_3 \gamma_k$$
if $\theta_3=\min\{ \theta_1,\theta_2\}$ and note that $\theta_2\leq C\beta.$
We define
\begin{align*}
\mathcal{L}^k &\coloneqq (1-a_1\alpha_k)\|\vX^{k}-\vX^*\|^2 + (2\eta_k^2/\gamma_k) \EE\|\vD^{k+1}-\vD^*\|^2_{(\vI-\vW)^\dagger}+ a_1\|\vH^k-\vX^*\|^2 \label{eqn_Lk}.
\end{align*}
Hence $$ \EE\mathcal{L}^{k+1} \leq (1-\theta_3\gamma_k) \EE\mathcal{L}^k+n\sigma^2\eta_k^2.$$
From $a_1\alpha\leq\frac{\mu\eta_k(2-\mu\eta_k)}{2},$ we get
$$\frac{4C\beta\gamma_k}{C\beta\gamma_k+2}\leq\mu\eta_k(2-\mu\eta_k).$$
If we pick $\gamma_k=\theta_4\eta_k$, then it's sufficient to let
$$2C\beta\theta_4\eta_k\leq\mu\eta_k(2-\mu\eta_k).$$
Hence if $\theta_4<\frac{\mu}{C\beta}$ and let $\eta_{*}=\frac{2(\mu-C\beta\theta_4)}{\mu^2},$ then $\eta_k=\frac{\gamma_k}{\theta_4}\in(0,\eta_*)$ guarantees the above discussion and
$$ \EE\mathcal{L}^{k+1} \leq (1-\theta_3\theta_4\eta_k) \EE\mathcal{L}^k+n\sigma^2\eta_k^2.$$
So far all restrictions for $\eta_k$ are
$$\eta_k\leq\min\left\{\frac{2}{\mu+L}, \eta_*\right\}$$
and
$$\eta_k\leq \frac{1}{\theta_4}\min\left\{\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta},\frac{2}{\beta}\right\}$$
Let $\theta_5=\min\left\{\frac{2}{\mu+L}, \eta_*,\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta\theta_4},\frac{2}{\beta\theta_4}\right\}$, $\eta_k=\frac{1}{Bk+A}$ and $D=\max\left\{A \mathcal{L}^0,\frac{2n\sigma^2}{\theta_3\theta_4}\right\},$ we claim that if we pick $B=\frac{\theta_3\theta_4}{2}$ and some $A$, by setting $\eta_k=\frac{2}{\theta_3\theta_4 k+2A}$, we get
$$\EE\mathcal{L}^k\leq\frac{D}{Bk+A}.$$
Induction:\\
When $k=0,$ it's obvious. Suppose previous $k$ inequalities hold. Then
$$\EE\mathcal{L}^{k+1}\leq\left(1-\frac{2\theta_3\theta_4}{\theta_3\theta_4 k+2A}\right)\frac{2D}{\theta_3\theta_4 k+2A}+\frac{4n\sigma^2}{(\theta_3\theta_4 k+2A)^2}.$$
Multiply $M\coloneqq (\theta_3\theta_4 k+\theta_3\theta_4+2A)(\theta_3\theta_4 k+2A)(2D)^{-1}$ on both sides, we get
\begin{equation*}
\begin{aligned}
M\EE\mathcal{L}^{k+1}\leq&\left(1-\frac{2\theta_3\theta_4}{\theta_3\theta_4 k+2A}\right)(\theta_3\theta_4 k+\theta_3\theta_4+2A)+\frac{4n\sigma^2(\theta_3\theta_4 k+\theta_3\theta_4+2A)}{2D(\theta_3\theta_4 k+2A)}\\
=&\frac{2D(\theta_3\theta_4 k+2A-2\theta_3\theta_4)(\theta_3\theta_4 k+\theta_3\theta_4+2A)+4n\sigma^2(\theta_3\theta_4 k+\theta_3\theta_4+2A)}{2D(\theta_3\theta_4 k+2A)}\\
=&\frac{2D(\theta_3\theta_4 k+2A)^2+4n\sigma^2(\theta_3\theta_4 k+2A)-4D\theta_3\theta_4(\theta_3\theta_4 k+2A)+2D\theta_3\theta_4(\theta_3\theta_4 k+2A)}{2D(\theta_3\theta_4 k+2A)}\\
&+\frac{-4D(\theta_3\theta_4)^2+4n\sigma^2\theta_3\theta_4}{2D(\theta_3\theta_4 k+2A)}\\
\leq&\theta_3\theta_4 k+2A.
\end{aligned}
\end{equation*}
Hence $$\EE \mathcal{L}^{k+1}\leq\frac{2D}{\theta_3\theta_4 (k+1)+2A}$$
This induction holds for any $A$ such that $\eta_k$ is feasible, i.e.
$$\eta_0=\frac{1}{A}\leq\theta_5.$$
Here we summarize the definition of constant numbers:
\begin{align}
\theta_1&=\frac{1}{2\lambda_{\max}((\vI-\vW)^\dagger)},\ \theta_2=\frac{C\beta}{2(1+C)},\\
\theta_3&=\min\{\theta_1,\theta_2\},\ \theta_4\in\left(0,\frac{\mu}{C\beta}\right),\ \eta_*=\frac{2(\mu-C\beta\theta_4)}{\mu^2},\\
\theta_5&=\min\left\{\frac{2}{\mu+L}, \eta_*,\frac{(3C+1)-\sqrt{(3C+1)^2-4C}}{C\beta\theta_4},\frac{2}{\beta\theta_4}\right\}.
\end{align}
Therefore, let $A=\frac{1}{\theta_5}$ and $\eta_k=\frac{2\theta_5}{\theta_3\theta_4\theta_5 k+2},$ we get
$$\frac{1}{n}\EE\mathcal{L}^k\leq\frac{2\max\left\{{ \frac{1}{n}\mathcal{L}^0},\frac{2\sigma^2\theta_5}{\theta_3\theta_4}\right\}}{\theta_3\theta_4 \theta_5 k+2}.$$
Since $1-a_1\alpha_k \geq 1/2$, we complete the proof.
\end{proof}
\begin{thebibliography}{35}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Alistarh et~al.(2017)Alistarh, Grubic, Li, Tomioka, and
Vojnovic]{NIPS2017_qsgd}
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic.
\newblock {QSGD}: Communication-efficient sgd via gradient quantization and
encoding.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
1709--1720. 2017.
\bibitem[Bernstein et~al.(2018)Bernstein, Wang, Azizzadenesheli, and
Anandkumar]{signSGD-ICML18}
Jeremy Bernstein, Yu{-}Xiang Wang, Kamyar Azizzadenesheli, and Animashree
Anandkumar.
\newblock {SIGNSGD:} compressed optimisation for non-convex problems.
\newblock In \emph{Proceedings of the 35th International Conference on Machine
Learning}, pp.\ 559--568, 2018.
\bibitem[Carli et~al.(2010)Carli, Fagnani, Frasca, and
Zampieri]{carli2010gossip}
Ruggero Carli, Fabio Fagnani, Paolo Frasca, and Sandro Zampieri.
\newblock Gossip consensus algorithms via quantized communication.
\newblock \emph{Automatica}, 46\penalty0 (1):\penalty0 70--80, 2010.
\bibitem[Karimireddy et~al.(2019)Karimireddy, Rebjock, Stich, and
Jaggi]{KarimireddyRSJ19feedback}
Sai~Praneeth Karimireddy, Quentin Rebjock, Sebastian~Urban Stich, and Martin
Jaggi.
\newblock Error feedback fixes {SignSGD} and other gradient compression
schemes.
\newblock In \emph{Proceedings of the 36th International Conference on Machine
Learning}, pp.\ 3252--3261. PMLR, 2019.
\bibitem[Koloskova et~al.(2019)Koloskova, Stich, and Jaggi]{ksj2019choco}
Anastasia Koloskova, Sebastian~U. Stich, and Martin Jaggi.
\newblock Decentralized stochastic optimization and gossip algorithms with
compressed communication.
\newblock In \emph{Proceedings of the 36th International Conference on Machine
Learning}, pp.\ 3479--3487. PMLR, 2019.
\bibitem[Koloskova et~al.(2020)Koloskova, Lin, Stich, and
Jaggi]{koloskova*2020decentralized}
Anastasia Koloskova, Tao Lin, Sebastian~U Stich, and Martin Jaggi.
\newblock Decentralized deep learning with arbitrary communication compression.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Li \& Yan(2019)Li and Yan]{li2019linear}
Yao Li and Ming Yan.
\newblock On linear convergence of two decentralized algorithms.
\newblock \emph{arXiv preprint arXiv:1906.07225}, 2019.
\bibitem[Li et~al.(2019)Li, Shi, and Yan]{li2019decentralized}
Zhi Li, Wei Shi, and Ming Yan.
\newblock A decentralized proximal-gradient method with network independent
step-sizes and separated convergence rates.
\newblock \emph{IEEE Transactions on Signal Processing}, 67\penalty0
(17):\penalty0 4494--4506, 2019.
\bibitem[Lian et~al.(2017)Lian, Zhang, Zhang, Hsieh, Zhang, and
Liu]{lian2017can}
Xiangru Lian, Ce~Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji~Liu.
\newblock Can decentralized algorithms outperform centralized algorithms? a
case study for decentralized parallel stochastic gradient descent.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
5330--5340, 2017.
\bibitem[Ling et~al.(2015)Ling, Shi, Wu, and Ribeiro]{ling2015dlm}
Qing Ling, Wei Shi, Gang Wu, and Alejandro Ribeiro.
\newblock {DLM}: Decentralized linearized alternating direction method of
multipliers.
\newblock \emph{IEEE Transactions on Signal Processing}, 63\penalty0
(15):\penalty0 4051--4064, 2015.
\bibitem[Liu et~al.(2020)Liu, Li, Tang, and Yan]{liu2019double}
Xiaorui Liu, Yao Li, Jiliang Tang, and Ming Yan.
\newblock A double residual compression algorithm for efficient distributed
learning.
\newblock \emph{The 23rd International Conference on Artificial Intelligence
and Statistics}, 2020.
\bibitem[Lu \& De~Sa(2020)Lu and De~Sa]{lu2020moniqua}
Yucheng Lu and Christopher De~Sa.
\newblock Moniqua: Modulo quantized communication in decentralized {SGD}.
\newblock In \emph{Proceedings of the 37th International Conference on Machine
Learning}, 2020.
\bibitem[Magn{\'u}sson et~al.(2020)Magn{\'u}sson, Shokri-Ghadikolaei, and
Li]{magnusson2020maintaining}
Sindri Magn{\'u}sson, Hossein Shokri-Ghadikolaei, and Na~Li.
\newblock On maintaining linear convergence of distributed learning and
optimization under limited communication.
\newblock \emph{IEEE Transactions on Signal Processing}, 68:\penalty0
6101--6116, 2020.
\bibitem[Mishchenko et~al.(2019)Mishchenko, Gorbunov, Tak{\'a}{\v{c}}, and
Richt{\'a}rik]{mishchenko2019distributed}
Konstantin Mishchenko, Eduard Gorbunov, Martin Tak{\'a}{\v{c}}, and Peter
Richt{\'a}rik.
\newblock Distributed learning with compressed gradient differences.
\newblock \emph{arXiv preprint arXiv:1901.09269}, 2019.
\bibitem[Mota et~al.(2013)Mota, Xavier, Aguiar, and P{\"u}schel]{mota2013d}
Joao~FC Mota, Joao~MF Xavier, Pedro~MQ Aguiar, and Markus P{\"u}schel.
\newblock {D-ADMM}: A communication-efficient distributed algorithm for
separable optimization.
\newblock \emph{IEEE Transactions on Signal Processing}, 61\penalty0
(10):\penalty0 2718--2723, 2013.
\bibitem[Nedic \& Ozdaglar(2009)Nedic and Ozdaglar]{nedic2009distributed}
Angelia Nedic and Asuman Ozdaglar.
\newblock Distributed subgradient methods for multi-agent optimization.
\newblock \emph{IEEE Transactions on Automatic Control}, 54\penalty0
(1):\penalty0 48--61, 2009.
\bibitem[Nedic et~al.(2017)Nedic, Olshevsky, and Shi]{nedic2017achieving}
Angelia Nedic, Alex Olshevsky, and Wei Shi.
\newblock Achieving geometric convergence for distributed optimization over
time-varying graphs.
\newblock \emph{SIAM Journal on Optimization}, 27\penalty0 (4):\penalty0
2597--2633, 2017.
\bibitem[Nesterov(2013)]{nesterov2013introductory}
Yurii Nesterov.
\newblock \emph{Introductory lectures on convex optimization: A basic course},
volume~87.
\newblock Springer Science \& Business Media, 2013.
\bibitem[Pu \& Nedi{\'c}(2020)Pu and Nedi{\'c}]{pu2020distributed}
Shi Pu and Angelia Nedi{\'c}.
\newblock Distributed stochastic gradient tracking methods.
\newblock \emph{Mathematical Programming}, pp.\ 1--49, 2020.
\bibitem[Reisizadeh et~al.(2019{\natexlab{a}})Reisizadeh, Mokhtari, Hassani,
and Pedarsani]{reisizadeh2019exact}
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, and Ramtin Pedarsani.
\newblock An exact quantized decentralized gradient descent algorithm.
\newblock \emph{IEEE Transactions on Signal Processing}, 67\penalty0
(19):\penalty0 4934--4947, 2019{\natexlab{a}}.
\bibitem[Reisizadeh et~al.(2019{\natexlab{b}})Reisizadeh, Taheri, Mokhtari,
Hassani, and Pedarsani]{reisizadeh2019robust}
Amirhossein Reisizadeh, Hossein Taheri, Aryan Mokhtari, Hamed Hassani, and
Ramtin Pedarsani.
\newblock Robust and communication-efficient collaborative learning.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
8388--8399, 2019{\natexlab{b}}.
\bibitem[Seide et~al.(2014)Seide, Fu, Droppo, Li, and Yu]{1-bit-sgd}
Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu.
\newblock 1-bit stochastic gradient descent and application to data-parallel
distributed training of speech {DNNs}.
\newblock In \emph{Interspeech 2014}, September 2014.
\bibitem[Shi et~al.(2015)Shi, Ling, Wu, and Yin]{extra_shi}
Wei Shi, Qing Ling, Gang Wu, and Wotao Yin.
\newblock {EXTRA}: An exact first-order algorithm for decentralized consensus
optimization.
\newblock \emph{SIAM Journal on Optimization}, 25\penalty0 (2):\penalty0
944--966, 2015.
\bibitem[Stich et~al.(2018)Stich, Cordonnier, and
Jaggi]{Stich:2018:SSM:3327345.3327357}
Sebastian~U. Stich, Jean-Baptiste Cordonnier, and Martin Jaggi.
\newblock Sparsified {SGD} with memory.
\newblock In \emph{Proceedings of the 32nd International Conference on Neural
Information Processing Systems}, pp.\ 4452--4463, 2018.
\bibitem[Tang et~al.(2018{\natexlab{a}})Tang, Gan, Zhang, Zhang, and
Liu]{tang_NIPS2018_7992}
Hanlin Tang, Shaoduo Gan, Ce~Zhang, Tong Zhang, and Ji~Liu.
\newblock Communication compression for decentralized training.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
7652--7662. 2018{\natexlab{a}}.
\bibitem[Tang et~al.(2018{\natexlab{b}})Tang, Lian, Yan, Zhang, and
Liu]{pmlr-v80-tang18a}
Hanlin Tang, Xiangru Lian, Ming Yan, Ce~Zhang, and Ji~Liu.
\newblock {$D^2$}: Decentralized training over decentralized data.
\newblock In \emph{Proceedings of the 35th International Conference on Machine
Learning}, pp.\ 4848--4856, 2018{\natexlab{b}}.
\bibitem[Tang et~al.(2019{\natexlab{a}})Tang, Lian, Qiu, Yuan, Zhang, Zhang,
and Liu]{dblpDeepSqueeze}
Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce~Zhang, Tong Zhang, and
Ji~Liu.
\newblock Deepsqueeze: Decentralization meets error-compensated compression.
\newblock \emph{CoRR}, abs/1907.07346, 2019{\natexlab{a}}.
\newblock URL \url{http://arxiv.org/abs/1907.07346}.
\bibitem[Tang et~al.(2019{\natexlab{b}})Tang, Yu, Lian, Zhang, and
Liu]{tang2019doublesqueeze}
Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang, and Ji~Liu.
\newblock {DoubleSqueeze}: Parallel stochastic gradient descent with
double-pass error-compensated compression.
\newblock In \emph{Proceedings of the 36th International Conference on Machine
Learning}, pp.\ 6155--6165, 2019{\natexlab{b}}.
\bibitem[Tsitsiklis et~al.(1986)Tsitsiklis, Bertsekas, and
Athans]{tsitsiklis1986distributed}
John Tsitsiklis, Dimitri Bertsekas, and Michael Athans.
\newblock Distributed asynchronous deterministic and stochastic gradient
optimization algorithms.
\newblock \emph{IEEE transactions on automatic control}, 31\penalty0
(9):\penalty0 803--812, 1986.
\bibitem[Wu et~al.(2018)Wu, Huang, Huang, and Zhang]{pmlr-v80-wu18d}
Jiaxiang Wu, Weidong Huang, Junzhou Huang, and Tong Zhang.
\newblock Error compensated quantized {SGD} and its applications to large-scale
distributed optimization.
\newblock In \emph{Proceedings of the 35th International Conference on Machine
Learning}, pp.\ 5325--5333, 2018.
\bibitem[Xiao \& Boyd(2004)Xiao and Boyd]{xiao2004fast}
Lin Xiao and Stephen Boyd.
\newblock Fast linear iterations for distributed averaging.
\newblock \emph{Systems \& Control Letters}, 53\penalty0 (1):\penalty0 65--78,
2004.
\bibitem[Xu et~al.(2020)Xu, Tian, Sun, and Scutari]{xu2020accelerated}
Jinming Xu, Ye~Tian, Ying Sun, and Gesualdo Scutari.
\newblock Accelerated primal-dual algorithms for distributed smooth convex
optimization over networks.
\newblock In \emph{International Conference on Artificial Intelligence and
Statistics}, pp.\ 2381--2391. PMLR, 2020.
\bibitem[Yuan et~al.(2016)Yuan, Ling, and Yin]{yuan2016convergence}
Kun Yuan, Qing Ling, and Wotao Yin.
\newblock On the convergence of decentralized gradient descent.
\newblock \emph{SIAM Journal on Optimization}, 26\penalty0 (3):\penalty0
1835--1854, 2016.
\bibitem[Yuan et~al.(2018)Yuan, Ying, Zhao, and Sayed]{yuan2018exact}
Kun Yuan, Bicheng Ying, Xiaochuan Zhao, and Ali~H Sayed.
\newblock Exact diffusion for distributed optimization and learning—part i:
Algorithm development.
\newblock \emph{IEEE Transactions on Signal Processing}, 67\penalty0
(3):\penalty0 708--723, 2018.
\bibitem[Yuan et~al.(2020)Yuan, Xu, and Ling]{yuan2020can}
Kun Yuan, Wei Xu, and Qing Ling.
\newblock Can primal methods outperform primal-dual methods in decentralized
dynamic optimization?
\newblock \emph{arXiv preprint arXiv:2003.00816}, 2020.
\end{thebibliography}