Bregman meets L\'evy: Stochastic mirror descent with heavy-tailed noise in continuous and discrete time
Extracted source text · arXiv 2606.03769
%**********************************************************************
%*** HEAVY NOISE MD
%**********************************************************************
%!TEX TS-program = pdflatex
%!TEX encoding = UTF-8 Unicode
\synctex=1
%**********************************************************************
%*** DRAFT MODE
%**********************************************************************
% Global switch that controls the appearance of helper elements
\newif
\ifdraftmode
\draftmodefalse
%\draftmodetrue
\typeout{**********************************************************************}
\ifdraftmode
\typeout{*** DRAFT MODE IS ON}
\else
\typeout{*** DRAFT MODE IS OFF: CHECK FINAL DETAILS}
\fi
\typeout{**********************************************************************}
%**********************************************************************
%*** 1. DOCUMENT CLASS
%**********************************************************************
\RequirePackage{fix-cm}
\documentclass[reqno]{amsart}
\usepackage[margin=1.5in,bottom=1.25in]{geometry} % for tighter margins (80 CPL)
%**********************************************************************
%*** 2. CORE PACKAGES
%**********************************************************************
%----------------------------------------------------------------------
%% Basic math tools
%----------------------------------------------------------------------
\usepackage{amssymb} % for AMS symbols
\usepackage{amsfonts} % for AMS fonts
\usepackage{amsthm} % for theorems
\usepackage[foot]{amsaddr} % for author footnotes
\usepackage{mathtools} % for advanced math
\mathtoolsset{%
centercolon=true,
%above-intertext-sep = -1ex, % for equation spacing
%below-intertext-sep = -2ex, % for equation spacing
%showonlyrefs, % to show only referenced equations
}
%----------------------------------------------------------------------
%% Math alphabets
%----------------------------------------------------------------------
\usepackage[% % for math font selection
cal=cm,
%scr=euler,
%frak=euler
]
{mathalfa}
%**********************************************************************
%*** 3. FONTS
%**********************************************************************
%----------------------------------------------------------------------
%% Blackboard bold
%----------------------------------------------------------------------
\usepackage{dsfont} % for blackboard bold font
%\let\mathbb=\mathds
%----------------------------------------------------------------------
%% Libertine
%----------------------------------------------------------------------
\usepackage[tabular,lining,sf,mono=false]{libertine}
%\usepackage[libertine,mono=false,scale,varbb,smallerops]{newtx}
%----------------------------------------------------------------------
%% Times
%----------------------------------------------------------------------
%\usepackage{newtxtext}
%\usepackage[smallerops,varg,varbb,cmbraces]{newtxmath}
%\usepackage[varg]{txfonts}
%\let\mathbb=\varmathbb
%----------------------------------------------------------------------
%% Sans serif
%----------------------------------------------------------------------
\usepackage[scaled=.95]{AlegreyaSans}
%\usepackage[scale=.89]{cabin}
%\usepackage[book,lf,scale=0.82]{FiraSans}
%\usepackage[type1,sfdefault,scale=1]{sourcesanspro}
%----------------------------------------------------------------------
%% Monospaced
%----------------------------------------------------------------------
%\usepackage{lmodern}
%\renewcommand*\ttdefault{lmvtt}
\renewcommand*\ttdefault{txtt}
%\usepackage[scale=.96]{inconsolata}
%\usepackage[ttdefault=true,scale=.95]{AnonymousPro}
%\usepackage[scale=.81]{GoMono}
%----------------------------------------------------------------------
%% Encoding (beware of conflicts, comes last)
%----------------------------------------------------------------------
%\usepackage[utf8]{inputenc} % for backward compatibility (if needed)
\usepackage[T1]{fontenc} % for font encoding
%**********************************************************************
%*** 4. ANCILLARY PACKAGES
%**********************************************************************
%*** DO NOT CHANGE ORDER OF PACKAGES AS THIS
%*** COULD HAVE UNINTENDED CONSEQUENCES
%**********************************************************************
%----------------------------------------------------------------------
%% Acronyms
%----------------------------------------------------------------------
\usepackage{acronym} % for acronyms
\renewcommand*{\aclabelfont}[1]{\acsfont{#1}} % for acronym label font
\newcommand{\acli}[1]{\emph{\acl{#1}}} % for italicized acro
\newcommand{\aclip}[1]{\emph{\aclp{#1}}} % for italicized acro (plural)
\newcommand{\acdef}[1]{\define{\acl{#1}} \textup{(\acs{#1})}\acused{#1}} % for acro def
\newcommand{\acdefp}[1]{\define{\aclp{#1}} \textup{(\acsp{#1})}\acused{#1}} % for acro def (plural)
\newcommand{\Acdef}[1]{\define{\Acl{#1}} \textup{(\acs{#1})}\acused{#1}} % for acro def (capitalized)
\newcommand{\Acdefp}[1]{\define{\Aclp{#1}} \textup{(\acs{#1})}\acused{#1}} % for acro def (capitalized plural)
%----------------------------------------------------------------------
%% Captions
%----------------------------------------------------------------------
\usepackage[labelfont={bf,small},font=small]{caption} % for caption control
\captionsetup[algorithm]{font=sf,labelfont=bf,labelsep=colon} % for algorithm captions
%\captionsetup[table]{labelfont=normalfont,labelsep=colon} % for tables
%\usepackage{subcaption} % for subfigures
%\subcaptionsetup{labelfont=bf,justification=centering} % for subcaptions
%\subcaptionsetup[figure]{labelfont={bf,scriptsize},justification=centering} % for subfigure captions
%----------------------------------------------------------------------
%% Colors
%----------------------------------------------------------------------
\usepackage[svgnames]{xcolor} % for color
%% Reds
\definecolor{Bonfire}{HTML}{9E162E}
\definecolor{CardinalRed}{HTML}{C41E3A}
\definecolor{TuckOrange}{HTML}{D94415}
%% Greens
\definecolor{CadmiumGreen}{HTML}{097969}
\definecolor{Dartmouth}{HTML}{00693E}
\definecolor{ForestGreen}{HTML}{12312B}
\definecolor{RichForestGreen}{HTML}{0D1E1C}
\definecolor{SeaGreen}{HTML}{2E8B57}
\definecolor{SpringGreen}{HTML}{EAFAF1}
\definecolor{Jade}{HTML}{009900}
%% Blues
\definecolor{CobaltBlue}{HTML}{0047AB}
\definecolor{NavyBlue}{HTML}{000080}
\definecolor{RiverBlue}{HTML}{267ABA}
\definecolor{RiverNavy}{HTML}{003C73}
\definecolor{KleinBlue}{HTML}{002FA7}
\definecolor{OxfordBlue}{HTML}{002147}
\definecolor{SapphireBlue}{HTML}{0F52BA}
\definecolor{Zaffre}{HTML}{0819A8}
%% Personal
\colorlet{MyRed}{CardinalRed}
\colorlet{MyGreen}{Dartmouth}
\colorlet{MyBlue}{DodgerBlue}
\colorlet{MyViolet}{DarkOrchid}
\colorlet{MyLightRed}{MyRed!25}
\colorlet{MyLightGreen}{MyGreen!25}
\colorlet{MyLightBlue}{MyBlue!25}
%% Themes
\colorlet{PrimalColor}{MidnightBlue!50!DodgerBlue}
\colorlet{PrimalFill}{DodgerBlue!25}
\colorlet{DualColor}{MyRed}
\colorlet{AlertColor}{MyRed} % for alerts
\colorlet{BadColor}{MyRed} % for good
\colorlet{GoodColor}{MyGreen} % for good
\colorlet{LinkColor}{MediumBlue} % for hyperlinks
\colorlet{RevColor}{MediumBlue} % for revisions
\ifdraftmode
\colorlet{DraftColor}{MyRed} % for macros
\else
\colorlet{DraftColor}{black} % for macros
\fi
%----------------------------------------------------------------------
%% Document layout
%----------------------------------------------------------------------
%\usepackage[page]{appendix}
\newcommand{\afterhead}{.} % for changing headings
\newcommand{\ackperiod}{} % for AMS bug in acknowledgments
\newcommand{\para}[1]{\smallskip\paragraph{\textbf{#1\afterhead}}} % for paragraph control
%\addtolength{\parsep}{1pt}
%----------------------------------------------------------------------
%% Extra symbols
%----------------------------------------------------------------------
\usepackage{latexsym} % for symbols
\usepackage{fontawesome} % for icons
%\usepackage{fourier-orns} % for ornaments
\usepackage{pifont} % for dingbats
%\usepackage{wasysym} % for symbols
\newcommand{\asterism}{\ding{70}}
\newcommand{\smiley}{{\smaller[0.5]\draft{\faSmileO}}\xspace} % for smiley
\newcommand{\frownie}{{\smaller[0.5]\draft{\faFrownO}}\xspace} % for frownie
\newcommand{\here}{\draft{\ding{43}}\xspace} % for pointouts
\newcommand{\cmark}{\ding{51}\xspace} % for checkmark
\newcommand{\xmark}{\ding{55}\xspace} % for x-cross
%----------------------------------------------------------------------
%% Figures and graphics (load only if needed)
%----------------------------------------------------------------------
\usepackage{tikz} % for figures
\usetikzlibrary{calc,patterns} % for basic tikz figures
%----------------------------------------------------------------------
%% Lists and tables
%----------------------------------------------------------------------
\usepackage{array} % for flexible tables
\usepackage{booktabs} % for better tables
\usepackage[inline,shortlabels]{enumitem} % for lists
\setlist[1]{topsep=\smallskipamount,itemsep=\smallskipamount,left=\parindent}
\setlist[2]{left=0pt}
%\setlist[enumerate,1]{label=\upshape(\itshape\alph*\hspace*{1pt}\upshape)}
\makeatletter
\AddToHook{env/enumerate*/begin}{\ifhmode\unskip\fi}
\makeatother
%----------------------------------------------------------------------
%% Microtypography
%----------------------------------------------------------------------
\usepackage[kerning=true]{microtype} % for microtypography
%----------------------------------------------------------------------
%% Various
%----------------------------------------------------------------------
\usepackage{tabto} % for spacing
\usepackage{xspace} % for flexible spaces
%**********************************************************************
%*** 5. REFERENCES
%**********************************************************************
%----------------------------------------------------------------------
%% Bibliographic citations
%----------------------------------------------------------------------
\usepackage[sort&compress]{natbib} % for citations
\def\bibfont{\footnotesize}
\def\bibhang{24pt}
\def\bibsep{\smallskipamount}
%\def\BIBand{and}
%\def\newblock{\ }
\bibpunct[, ]{[}{]}{,}{}{,}{,}
%\citestyle{plainnat}
\setcitestyle{numbers,square}
%----------------------------------------------------------------------
%% Keys
%----------------------------------------------------------------------
\ifdraftmode
\usepackage[notref,notcite,color]{showkeys} % for showing label keys
\colorlet{refkey}{DraftColor}
\colorlet{labelkey}{DraftColor}
% \usepackage[inline]{showlabels} % for showing label keys
% \renewcommand{\showlabelfont}{\small\ttfamily\color{AlertColor}}
\fi
%----------------------------------------------------------------------
%% Hyperlinks
%----------------------------------------------------------------------
\usepackage{hyperref}
\hypersetup{
final,
colorlinks=true,
linktocpage=true,
%pdfstartpage=1,
pdfstartview=FitH,
breaklinks=true,
pdfpagemode=UseNone,
pageanchor=true,
pdfpagemode=UseOutlines,
plainpages=false,
bookmarksnumbered,
bookmarksopen=false,
bookmarksopenlevel=1,
hypertexnames=false,
pdfhighlight=/O,
%nesting=true,
%frenchlinks,
urlcolor=LinkColor,linkcolor=LinkColor,citecolor=LinkColor, % for on-screen
%urlcolor=black, linkcolor=black, citecolor=black, %pagecolor=black, % for printing
%pagecolor=LinkColor,
pdftitle={},
pdfauthor={},
pdfsubject={},
pdfkeywords={},
pdfcreator={pdfLaTeX},
pdfproducer={LaTeX with hyperref}
}
\newcommand{\EMAIL}[1]{\email{\href{mailto:#1}{#1}}}
\newcommand{\URLADDR}[1]{\urladdr{\href{#1}{#1}}}
%----------------------------------------------------------------------
%% Theorem tools
%----------------------------------------------------------------------
\usepackage{thmtools} % for theorem tools
\usepackage{thm-restate} % for restating theorems
%----------------------------------------------------------------------
%% Cleverefs (must go after hyperref)
%----------------------------------------------------------------------
%\numberwithin{equation}{section} % for cleveref (when using cleveref and hyperref)
\usepackage[sort&compress,capitalize,nameinlink]{cleveref} % for cleveref formatting
%\crefname{algorithm}{Alg.}{Algs.}
%\crefname{example}{Ex.}{Exs.}
\crefname{algo}{Algorithm}{Algorithms}
\crefname{assumption}{Assumption}{Assumptions}
\newcommand{\crefrangeconjunction}{\textendash} % for cleveref conjunctions
%\crefrangeformat{equation}{\upshape(#3#1#4)\textendash(#5#2#6)}
%\creflabelformat{part}{(#2#1#3)}
% Fixes cleveref bug with acronyms
\makeatletter
\AtBeginDocument
{
\def\ltx@label#1{\cref@label{#1}} % add braces
\def\label@in@display@noarg#1{\cref@old@label@in@display{#1}} % remove braces
}%
\makeatother
%----------------------------------------------------------------------
%% Only referenced equations
%----------------------------------------------------------------------
%DO NOT TURN ON
%\usepackage{autonum} % for citing referenced only / must go last
%**********************************************************************
%*** 6. ENVIRONMENTS
%**********************************************************************
%----------------------------------------------------------------------
%% Algorithms
%----------------------------------------------------------------------
\usepackage{algorithm} % for algorithm environments
\usepackage{algpseudocode} % for algorithm macros
\renewcommand{\algorithmiccomment}[1]{\hfill$\triangleright$ #1} % for algorithm comments
%----------------------------------------------------------------------
%% Boxes
%----------------------------------------------------------------------
\usepackage[most]{tcolorbox}
\newenvironment{hilite}{%
\begin{tcolorbox}[
enhanced,
breakable,
colback=black!10,
leftrule=1mm,
toprule=0pt,
bottomrule=0pt,
rightrule=0pt,
arc=0pt,
before skip=1em plus 2pt,
after skip=1em plus 2pt,
]
}{\end{tcolorbox}}
%----------------------------------------------------------------------
%% Theorem-like
%----------------------------------------------------------------------
\theoremstyle{plain}
\newtheorem{theorem}{Theorem} % for theorems
\newtheorem{corollary}{Corollary} % for corollaries
\newtheorem{lemma}{Lemma} % for lemmas
\newtheorem{proposition}{Proposition} % for propositions
\newtheorem{conjecture}{Conjecture} % for conjectures
\newtheorem{claim}{Claim} % for claims
\newtheorem*{theorem*}{Theorem} % for theorems
\newtheorem*{corollary*}{Corollary} % for corollaries (unnumbered)
%----------------------------------------------------------------------
%% Definition-like
%----------------------------------------------------------------------
\theoremstyle{definition}
\newtheorem{definition}{Definition} % for definitions
\newtheorem{assumption}{Assumption} % for assumptions
\newtheorem{example}{Example} % for examples
\newtheorem*{definition*}{Definition} % for definitions (unnumbered)
\newtheorem*{assumption*}{Assumptions} % for assumptions (unnumbered)
\newtheorem*{example*}{Example} % for examples (unnumbered)
%----------------------------------------------------------------------
%% Remark-like
%----------------------------------------------------------------------
\theoremstyle{remark}
\newtheorem{remark}{Remark} % for remarks
\newtheorem{notation}{Notation} % for notation
\newtheorem*{remark*}{Remark} % for remarks (unnumbered)
\newtheorem*{notation*}{Notation} % for notation (unnumbered)
\newtheorem*{note}{Note} % for note (unnumbered)
%----------------------------------------------------------------------
%% Environment end
%----------------------------------------------------------------------
%\def\endenv{\hfill\ding{69}} % for ending environments
%\def\endenv{\hfill\ding{166}} % for ending environments
%\def\endenv{\hfill\textbf{\S}} % for ending environments
\def\endenv{\hfill\asterism} % for ending environments
%----------------------------------------------------------------------
%% Proofs
%----------------------------------------------------------------------
%\newenvironment{Proof}[1][Proof]{\begin{proof}[#1]}{\end{proof}} % for redefining proofs
\renewcommand{\qedsymbol}{{\small$\blacksquare$}} % for qed symbol
\newcounter{proofstep}
\newenvironment{proofstep}[1]
{\vspace{3pt}
\refstepcounter{proofstep}%
\par\textit{Step~\arabic{proofstep}:~#1}.\,}
{\smallskip}
%----------------------------------------------------------------------
%% Numbering
%----------------------------------------------------------------------
%\numberwithin{equation}{section} % for equation numbering (beware of cleveref)
%\numberwithin{theorem}{section} % for theorem numbering
%\numberwithin{definition}{section} % for definition numbering
%\numberwithin{remark}{section} % for remark numbering
\numberwithin{example}{section} % for example numbering
%**********************************************************************
%*** 7. EDITING
%**********************************************************************
\ifdraftmode
\usepackage[showdeletions]{color-edits} % for editing macros / use [suppress] for final
\setlength{\marginparwidth}{.6in}
\else
\usepackage[suppress]{color-edits} % for editing macros / use [suppress] for final
\fi
\newcommand{\attn}{\textbf{\color{AlertColor}[!!]}\xspace} % for attention
\newcommand{\needref}{\textbf{\color{AlertColor}[??]}\xspace} % for question
\ifdraftmode
\newcommand{\draft}[1]{{\color{DraftColor}#1}} % for macro coloring
\else
\newcommand{\draft}[1]{#1} % for removing macro coloring
\fi
\newcommand{\define}[1]{\emph{\draft{#1}}} % for revision markup
\newcommand{\revise}[1]{{\color{RevColor}#1}} % for revision markup
%\newcommand{\revise}[1]{#1} % for revision markup
\newcommand{\lookout}[1]{{\bfseries\color{AlertColor}[#1]}} % for attention
\newcommand{\good}[1]{{\color{GoodColor}#1}} % for revision markup
\newcommand{\bad}[1]{{\color{BadColor}#1}} % for revision markup
\newcommand{\explain}[1]{\tag*{\small\ensuremath{\commentsymbol}\;\text{#1}}}
%\newcommand{\explain}[1]{}
%\def\beginrev{} % for revision markup
\def\beginrev{\color{RevColor}} % for revision markup
\def\beginrmv{\color{gray}} % for removal markup
\def\beginok{\color{MyGreen}} % for ok markup
\def\endedit{} % for ending markup
\def\endedit{\color{black}} % for ending markup
%**********************************************************************
%*** 8. MACROS
%**********************************************************************
%----------------------------------------------------------------------
%% Meta definitions
%----------------------------------------------------------------------
\newcommand{\newmacro}[2]{\newcommand{#1}{\draft{#2}}} % for semantic definitions
\newcommand{\newop}[2]{\DeclareMathOperator{#1}{\draft{#2}}} % for semantic definitions
\newcommand{\newoplims}[2]{\DeclareMathOperator*{#1}{\draft{#2}}} % for semantic definitions
%**********************************************************************
%*** MACROS: UNIVERSAL
%**********************************************************************
%----------------------------------------------------------------------
%% Aliases
%----------------------------------------------------------------------
\newcommand{\eps}{\varepsilon} % for better epsilon
\newcommand{\pd}{\partial} % for derivatives
\newcommand{\wilde}{\widetilde} % for wide tildes
%----------------------------------------------------------------------
%% Delimiters
%----------------------------------------------------------------------
\DeclarePairedDelimiter{\braces}{\{}{\}} % for braces
\DeclarePairedDelimiter{\bracks}{[}{]} % for brackets
\DeclarePairedDelimiter{\parens}{(}{)} % for parentheses
\DeclarePairedDelimiter{\of}{(}{)} % alias
\DeclarePairedDelimiter{\abs}{\lvert}{\rvert} % for absolute value
\DeclarePairedDelimiter{\ceil}{\lceil}{\rceil} % for ceiling
\DeclarePairedDelimiter{\floor}{\lfloor}{\rfloor} % for floor
\DeclarePairedDelimiter{\clip}{[}{]} % for clipping
\DeclarePairedDelimiter{\negpart}{[}{]_{-}} % for negative part
\DeclarePairedDelimiter{\pospart}{[}{]_{+}} % for positive part
\DeclarePairedDelimiter{\setof}{\{}{\}} % for sets
\DeclarePairedDelimiterX{\setdef}[2]{\{}{\}}{#1:#2} % for set builder notation
\DeclarePairedDelimiterXPP{\exclude}[1]{\mathopen{}\setminus}{\{}{\}}{}{#1} % for exclusions
\DeclarePairedDelimiter{\bra}{\langle}{\rvert} % for bras
\DeclarePairedDelimiter{\ket}{\lvert}{\rangle} % for kets
%\DeclarePairedDelimiterX{\braket}[2]{\langle}{\rangle}{#1\mathopen{}\delimsize\vert\mathopen{}#2} % for brakets
\DeclarePairedDelimiterX{\braket}[2]{\langle}{\rangle}{#1,#2} % for dual pairing
\DeclarePairedDelimiterX{\inner}[2]{\langle}{\rangle}{#1,#2} % for inner product
\DeclarePairedDelimiter{\norm}{\lVert}{\rVert} % for norm
\DeclarePairedDelimiterXPP{\dnorm}[1]{}{\lVert}{\rVert}{_{\draft{\ast}}}{#1} % for dual norm
%\newcommand{\dnorm}[1]{\norm{#1}_{\ast}} % for dual norm
\DeclarePairedDelimiterXPP{\onenorm}[1]{}{\lVert}{\rVert}{_{\draft{1}}}{#1} % for L1 norm
\DeclarePairedDelimiterXPP{\twonorm}[1]{}{\lVert}{\rVert}{_{\draft{2}}}{#1} % for L2 norm
\DeclarePairedDelimiterXPP{\pnorm}[1]{}{\lVert}{\rVert}{_{\draft{p}}}{#1} % for Lp norm
\DeclarePairedDelimiterXPP{\qnorm}[1]{}{\lVert}{\rVert}{_{\draft{q}}}{#1} % for Lq norm
\DeclarePairedDelimiterXPP{\supnorm}[1]{}{\lVert}{\rVert}{_{\draft{\infty}}}{#1} % for sup norm
\DeclarePairedDelimiterXPP{\matnorm}[1]{}{\lVert}{\rVert}{_{\draft{F}}}{#1} % for sup norm
%----------------------------------------------------------------------
%% Relations and arrows
%----------------------------------------------------------------------
\newop{\defeq}{\coloneqq} % for direct definition
\newop{\eqdef}{\eqqcolon} % for reverse definition
\newmacro{\from}{\colon} % for function definition
\newop{\too}{\rightrightarrows} % for correspondences
\newop{\injects}{\hookrightarrow} % for injections
\newop{\surjects}{\twoheadrightarrow} % for surjections
\newcommand{\compl}[1]{#1^{\draft{\mathtt{c}}}} % for complement
%----------------------------------------------------------------------
%% Number systems
%----------------------------------------------------------------------
\newmacro{\F}{\mathbb{F}} % generic field
\newmacro{\N}{\mathbb{N}} % for naturals
\newmacro{\Z}{\mathbb{Z}} % for integers
\newmacro{\Q}{\mathbb{Q}} % for rationals
\newmacro{\R}{\mathbb{R}} % for reals
\newmacro{\C}{\mathbb{C}} % for complex numbers (may clash)
%----------------------------------------------------------------------
%% Operators
%----------------------------------------------------------------------
\newoplims{\argmax}{arg\,max} % for argmax
\newoplims{\argmin}{arg\,min} % for argmin
\newoplims{\intersect}{\bigcap} % for intersections
\newoplims{\union}{\bigcup} % for unions
\newop{\aff}{aff} % for affine hull
\newop{\bd}{bd} % for boundary
\newop{\bigoh}{\mathcal{O}} % for Landau O
\newop{\tildeoh}{\mathcal{\tilde O}} % for polylog Landau O
\newop{\baroh}{\mathcal{\bar O}} % for polylog Landau O
\newmacro{\littleoh}{o} % for Landau O
\newop{\card}{\#} % for cardinality
\newop{\cl}{cl} % for closure
\newop{\conv}{conv} % for convex hull (but see also \simplex)
\newop{\crit}{crit} % for critical
\newop{\curl}{curl} % for curl
\newop{\diag}{diag} % for diagonal matrices
\newop{\diam}{diam} % for diameter
\newop{\dist}{dist} % for distance
\newop{\diver}{div} % for divergece
\newop{\dom}{dom} % for domain
\newop{\eig}{eig} % for eigens
\newop{\ess}{ess} % for essential
\newop{\grad}{grad} % for gradient
\newop{\Hess}{Hess} % for Hessian
\newop{\ind}{ind} % for index
\newop{\im}{im} % for image
\newop{\intr}{int} % for interior
\newop{\Jac}{Jac} % for Jacobian
\newop{\one}{\mathds{1}} % for indicator
\newop{\proj}{proj} % for projection
\newop{\prox}{prox} % for prox
\newop{\rank}{rank} % for rank
\newop{\relint}{ri} % for relative interior
\newop{\sign}{sgn} % for sign
\newop{\supp}{supp} % for support
\newop{\Sym}{Sym} % for symmetric
\newop{\tr}{tr} % for trace
\newop{\unif}{unif} % for uniform
\newop{\vol}{vol} % for volume
%----------------------------------------------------------------------
%% Text and formatting
%----------------------------------------------------------------------
\newcommand{\cf}{cf.\xspace} % for consistency
\newcommand{\eg}{e.g.,\xspace} % for consistency
\newcommand{\ie}{i.e.,\xspace} % for consistency
\newcommand{\wrt}{w.r.t.\xspace} % for consistency
\newcommand{\vs}{vs.\xspace} % for consistency
\newcommand{\viz}{viz.\xspace} % for consistency
\newcommand{\textbrac}[1]{\textup[#1\textup]} % for upshape brackets
\newcommand{\textpar}[1]{\textup(#1\textup)} % for upshape parentheses
\newmacro{\commentsymbol}{\triangleright} % for comment symbol
\newcommand{\dis}{\displaystyle} % for forcing display style
\newcommand{\txs}{\textstyle} % for forcing inline style
\newcommand{\eqcomma}{\,,} % for comma after displays
\newcommand{\eqstop}{\,.} % for period after displays
%----------------------------------------------------------------------
%% Various
%----------------------------------------------------------------------
\newcommand{\alt}[1]{#1'} % for alternate version
\newcommand{\altalt}[1]{#1''} % for second alternate
\newcommand{\avg}[1]{\bar#1} % for averaging
\newcommand{\est}[1]{\hat #1} % for estimates
%\newmacro{\argdot}{\kern.5pt\cdot\kern.5pt} % for argument
\newmacro{\argdot}{\boldsymbol{\cdot}} % for argument
\newmacro{\dd}{\kern0pt\:d} % for integration
\newmacro{\ddt}{\frac{d}{dt}} % for Leibniz
\newmacro{\del}{\partial} % for derivatives
\newcommand{\insum}{\sum\nolimits} % for compact sums
\newcommand{\inprod}{\prod\nolimits} % for compact products
\newmacro{\const}{c} % for generic constant
\newmacro{\Const}{C} % for generic constant
\newmacro{\param}{\theta} % for parameter
\newmacro{\params}{\Theta} % for set of parameters
\newmacro{\coef}{\lambda} % for generic coefficient
\newmacro{\fn}{f} % for generic function
\newmacro{\pexp}{p} % for first exponent
\newmacro{\qexp}{q} % for second exponent
\newmacro{\rexp}{r} % for third exponent
%**********************************************************************
%*** MACROS: SEMANTIC
%**********************************************************************
%----------------------------------------------------------------------
%% Counters
%----------------------------------------------------------------------
\newmacro{\iCount}{i} % for generic counter
\newmacro{\jCount}{j} % for second counter
\newmacro{\kCount}{k} % for third counter
\newmacro{\nCounts}{n} % for number of counts
\newmacro{\counts}{\mathcal{I}} % for set of counts
%----------------------------------------------------------------------
%% Points and sets
%----------------------------------------------------------------------
\newmacro{\point}{x} % for generic point
\newmacro{\pointalt}{\alt\point} % for other point
\newmacro{\pointaux}{u} % for auxiliary point
\newmacro{\points}{\mathcal{X}} % for set of points
\newmacro{\intpoints}{\relint\points} %for interior points
\newmacro{\base}{q} % for reference point
\newmacro{\basealt}{q} % for other reference point
\newmacro{\auxpoint}{u} % for auxiliary point
\newmacro{\elem}{a} % for generic element
\newmacro{\elemalt}{b} % for other element
\newmacro{\iElem}{a} % alias
\newmacro{\jElem}{b} % alias
\newmacro{\kElem}{c} % alias
\newmacro{\set}{\mathcal{A}} % for generic set
\newmacro{\borel}{\mathcal{B}} % for Borel sets
\newmacro{\closed}{\mathcal{C}} % for closed sets
\newmacro{\cpt}{\mathcal{K}} % for compact sets
\newmacro{\nhd}{\mathcal{U}} % for neighborhoods
\newmacro{\nhdalt}{\mathcal{V}} % for neighborhoods
\newmacro{\open}{\mathcal{U}} % for open sets
\newmacro{\domain}{\mathcal{D}} % for domain
\newmacro{\region}{\mathcal{R}} % for region
\newmacro{\interval}{\mathcal{I}} % for intervals
\newmacro{\rectangle}{\mathcal{R}} % for rectangles
\newmacro{\cone}{\mathcal{K}} % for cones
%----------------------------------------------------------------------
%% Curves and flows
%----------------------------------------------------------------------
\newmacro{\tstart}{0} % for time start
\renewcommand{\time}{\draft{t}} % for generic time
\newmacro{\timealt}{s} % for second time
\newmacro{\timealtalt}{\tau} % for third time
\newmacro{\horizon}{T} % for horizon
\newmacro{\curve}{\gamma} % for generic curve
\DeclarePairedDelimiterXPP{\curveof}[1]{\curve}{(}{)}{}{#1} % for curve
\DeclarePairedDelimiterXPP{\curveofX}[2]{\curve_{#1}}{(}{)}{}{#2} % for indexed curve
\DeclarePairedDelimiterXPP{\velof}[1]{\dot\curve}{(}{)}{}{#1} % for velocity of curve
\DeclarePairedDelimiterXPP{\velofX}[2]{\dot\curve_{#1}}{(}{)}{}{#2} % for indexed velocity
\newmacro{\flowmap}{\Phi} % for (semi)flow map
\DeclarePairedDelimiterXPP{\flowof}[2]{\flowmap_{#1}}{(}{)}{}{#2} % for flow
\newmacro{\traj}{x} % for generic trajectory
\newmacro{\dtraj}{\dot\traj} % for velocity of trajectory
\DeclarePairedDelimiterXPP{\trajof}[1]{\traj}{(}{)}{}{#1} % for trajectory
\DeclarePairedDelimiterXPP{\trajofX}[2]{\traj_{#1}}{(}{)}{}{#2} % for indexed trajectory
\DeclarePairedDelimiterXPP{\dtrajof}[1]{\dtraj}{(}{)}{}{#1} % for velocity of trajectory
\DeclarePairedDelimiterXPP{\dtrajofX}[2]{\dtraj_{#1}}{(}{)}{}{#2} % for velocity of indexed trajectory
%----------------------------------------------------------------------
%% Linear algebra
%----------------------------------------------------------------------
\newmacro{\vdim}{n} % dimension
\newmacro{\realspace}{\R^{\vdim}} % for real space
\newmacro{\iCoord}{i} % for coordinate index
\newmacro{\jCoord}{j} % for second coordinate
\newmacro{\kCoord}{k} % for third coordinate
\newmacro{\nCoords}{\vdim} % for dimension
\newmacro{\mCoords}{m} % for other dimension
\newmacro{\unitvec}{u} % for unit vector
\newmacro{\bvec}{e} % for basis vector
\newmacro{\bvecs}{\mathcal{E}} % for basis
\newmacro{\vecspace}{\mathcal{V}} % for generic vector space
\newmacro{\subspace}{\mathcal{W}} % for subspace
\newcommand{\dual}[1][\vecspace]{#1^{\ast}} % for dual space (\vecspace by default)
\newmacro{\dspace}{\dual[\vecspace]} % for dual space
\newmacro{\hilbert}{\mathcal{H}} % for Hilbert space
\newmacro{\banach}{\mathcal{X}} % for Banach space
\newmacro{\mat}{M} % for generic matrix
\newmacro{\hmat}{H} % for Hessian matrix
\newmacro{\ones}{\mathbf{1}} % for matrix of ones
\newmacro{\eye}{I} % for identity matrix
\newmacro{\zer}{\mathbf{0}} % for zero matrix
\newcommand{\mg}{\succ} % for positive-definite
\newcommand{\mgeq}{\succcurlyeq} % for positive-semidefinite
\newcommand{\ml}{\prec} % for negative-definite
\newcommand{\mleq}{\preccurlyeq} % for negative-semidefinite
\newmacro{\eigval}{\lambda} % for eigenvalue
\newmacro{\eigvec}{u} % for eigenvector
%----------------------------------------------------------------------
%% Geometry
%----------------------------------------------------------------------
\newmacro{\ball}{\mathbb{B}} % for ball
\newmacro{\sphere}{\mathbb{S}} % for sphere
\newmacro{\radius}{r}
\newmacro{\Radius}{R}
\newmacro{\mfld}{\mathcal{M}} % for manifold
\newcommand{\tanspace}[2]{\draft{T}_{#1}#2} % for tangent space
\newmacro{\tanvec}{z} % for tangent vector
\newmacro{\form}{\omega} % for generic form
\newmacro{\gmat}{g} % for metric tensor
\newmacro{\gdist}{\dist_{\gmat}} % for metric distance
%----------------------------------------------------------------------
%% Graph theory
%----------------------------------------------------------------------
\newmacro{\vertex}{v} % for generic vertex
\newmacro{\vertexalt}{w} % for other vertex
\newmacro{\iVertex}{\vertex_{\iCount}} % for first vertex
\newmacro{\jVertex}{\vertex_{\jCount}} % for second vertex
\newmacro{\kVertex}{\vertex_{\kCount}} % for third vertex
\newmacro{\nVertices}{V} % for number of vertices
\newmacro{\vertices}{\mathcal{V}} % for set of vertices
\newmacro{\edge}{e} % for generic edge
\newmacro{\edgealt}{\alt\edge} % for other edge
\newmacro{\iEdge}{\edge_{\iCount}} % for first edge
\newmacro{\jEdge}{\edge_{\jCount}} % for second edge
\newmacro{\kEdge}{\edge_{\kCount}} % for third edge
\newmacro{\nEdges}{E} % for number of edges
\newmacro{\edges}{\mathcal{\nEdges}} % for set of edges
\newmacro{\graph}{\mathcal{G}} % for graph
\newmacro{\graphfull}{\graph(\vertices,\edges)} % for graph, full description
%----------------------------------------------------------------------
%% Optimization
%----------------------------------------------------------------------
\newop{\minimize}{minimize} % for minimization
\newop{\opt}{Opt} % for value of problem
\newop{\gap}{Gap} % for gap function
\newmacro{\cvx}{\mathcal{C}} % for generic convex set
\newmacro{\obj}{f} % for objective function
\newmacro{\sobj}{F} % for stochastic objective
\newmacro{\oper}{A} % for operator
\newmacro{\vecfield}{v} % for vector field (selection etc.)
\newmacro{\subd}{\partial} % for subdifferential
\newmacro{\subsel}{\nabla} % for subdifferential selection
\newmacro{\gvec}{g} % for gradient vector
\newmacro{\gbound}{G} % for gradient bound
\newmacro{\vbound}{V} % for vector bound
\newmacro{\lips}{L} % for Lipschitz modulus
\newmacro{\strong}{\mu} % for strong convexity modulus
\newmacro{\smooth}{\beta} % for strong smoothness modulus
\newop{\tcone}{TC} % for tangent cone
\newop{\dcone}{\tcone^{\ast}} % for dual cone
\newop{\ncone}{NC} % for normal cone
\newop{\pcone}{PC} % for polar cone
\newop{\hull}{\Delta} % for convex hull
\newcommand{\sol}[1][\point]{#1^{\ast}} % for solution point (\point by default)
\newcommand{\sols}{\sol[\points]} % for set of solutions
%----------------------------------------------------------------------
%% Probability
%----------------------------------------------------------------------
\newop{\ex}{\mathbb{E}} % for expectation
\newop{\prob}{\mathbb{P}} % for probability
\newop{\Var}{\mathbb{V}} % for variance
\newop{\cov}{cov} % for covariance
\newop{\simplex}{\Delta} % for simplices
\providecommand\given{} % empty command for conditionals
\DeclarePairedDelimiterXPP{\exof}[1]{\ex}{[}{]}{}{% % for conditional expectations
\renewcommand\given{\nonscript\,\delimsize\vert\nonscript\,\mathopen{}} #1}
\DeclarePairedDelimiterXPP{\exwrt}[2]{\ex_{#1}}{[}{]}{}{% % for conditional expectations
\renewcommand\given{\nonscript\:\delimsize\vert\nonscript\:\mathopen{}} #2}
\DeclarePairedDelimiterXPP{\probof}[1]{\prob}{(}{)}{}{% % for conditional probabilities
\renewcommand\given{\nonscript\:\delimsize\vert\nonscript\:\mathopen{}} #1}
\DeclarePairedDelimiterXPP{\probwrt}[2]{\prob_{#1}}{(}{)}{}{% % for conditional probabilities
\renewcommand\given{\nonscript\:\delimsize\vert\nonscript\:\mathopen{}} #2}
\DeclarePairedDelimiterXPP{\oneof}[1]{\one}{\{}{\}}{}{#1} % for indicators
\DeclarePairedDelimiterXPP{\varof}[1]{\var}{[}{]}{}{% % for variance
\renewcommand\given{\nonscript\,\delimsize\vert\nonscript\,\mathopen{}} #1}
\DeclarePairedDelimiterXPP{\covof}[1]{\cov}{(}{)}{}{% % for covariance
\renewcommand\given{\nonscript\,\delimsize\vert\nonscript\,\mathopen{}} #1}
\newmacro{\event}{\mathcal{E}} % for generic event
\newmacro{\eventalt}{H} % for second event
\newmacro{\sample}{\omega} % for sample
\newmacro{\samples}{\Omega} % for set of samples
\newmacro{\filter}{\mathcal{F}} % for filtration
\newmacro{\probspace}{(\samples,\filter,\prob)} % for probability space
\newmacro{\pdist}{P} % for seed law
\newmacro{\history}{\mathcal{H}} % for filtrations
\newcommand{\as}{\draft{\textpar{a.s.}}\xspace} % for almost surely
\newmacro{\mean}{\mu} % for mean
\newmacro{\sdev}{\sigma} % for standard deviation
\newmacro{\variance}{\sdev^{2}} % for variance
\newmacro{\covmat}{\Sigma} % for covariance matrix
%----------------------------------------------------------------------
%% Sequences and indexing
%----------------------------------------------------------------------
\newcommand{\new}[1][\point]{#1^{+}} % for new iterate (x by default)
\newmacro{\seq}{a} % for generic sequence
\newmacro{\seqalt}{b} % for other sequence
\newmacro{\beforestart}{0} % for before start index
\newmacro{\start}{1} % for start index
\newmacro{\afterstart}{2} % for second index
\newmacro{\running}{\start,\afterstart,\dotsc} % for running
\newmacro{\run}{t} % for generic index
\newmacro{\runalt}{s} % for second index
\newmacro{\runaltalt}{\tau} % for third index
\newmacro{\nRuns}{T} % for index size
\newmacro{\runs}{\mathcal{\nRuns}} % for index set
\newcommand{\beforeinit}[1][\point]{\draft{#1}_{\afterstart}} % for zeroth iterate (X by default)
\newcommand{\init}[1][\point]{\draft{#1}_{\start}} % for initial iterate (X by default)
\newcommand{\afterinit}[1][\point]{\draft{#1}_{\afterstart}} % for second iterate (X by default)
\newcommand{\beforeiter}[1][\point]{\draft{#1}_{\runalt-1}} % for before running iterate (X by default)
\newcommand{\iter}[1][\point]{\draft{#1}_{\runalt}} % for running iterate (X by default)
\newcommand{\afteriter}[1][\point]{\draft{#1}_{\runalt+1}} % for after running iterate (X by default)
\newcommand{\prev}[1][\point]{\draft{#1}_{\run-1}} % for previous iterate (X by default)
\newcommand{\curr}[1][\point]{\draft{#1}_{\run}} % for current iterate (X by default)
\renewcommand{\next}[1][\point]{\draft{#1}_{\run+1}} % for next iterate (X by default)
\newcommand{\beforelast}[1][\point]{\draft{#1}_{\nRuns-1}} % for before last iterate (X by default)
\newcommand{\last}[1][\point]{\draft{#1}_{\nRuns}} % for last iterate (X by default)
\newcommand{\afterlast}[1][\point]{\draft{#1}_{\nRuns+1}} % for after last iterate (X by default)
%**********************************************************************
%*** MACROS: PAPER-SPECIFIC
%**********************************************************************
%----------------------------------------------------------------------
%% Parenthetical indexing
%----------------------------------------------------------------------
% Document switch that controls the appearance of indexed orbits
\newif
\ifparenargs
\parenargstrue
%\parenargsfalse
\ifparenargs
\DeclarePairedDelimiterXPP{\eval}[2]{#1}{(}{)}{}{#2} % for timed version
\DeclarePairedDelimiterXPP{\timed}[1]{#1}{(}{)}{}{\time} % for timed version
\else
\makeatletter
\newcommand{\eval}[2]{\eval@aux#1_\eval@stop{#2}}
\def\eval@aux#1_#2\eval@stop#3{%
\if\relax\detokenize{#2}\relax % no subscript present
#1_{#3}%
\else % had a subscript; append ,<arg>
#1_{#2{},#3}%
\fi
}
\makeatother
\newcommand{\timed}[1]{\eval{#1}{\time}}
\fi
%\makeatletter
%\newcommand{\timed}[1]{\timed@aux#1_\timed@stop}
%\def\timed@aux#1_#2\timed@stop{%
% \if\relax\detokenize{#2}\relax
% % no subscript present
% #1_{\time}%
% \else
% % had a subscript; append ,t
% #1_{#2{},\time}%
% \fi
%}
%\makeatother
%----------------------------------------------------------------------
%% Mirror descent and friends
%----------------------------------------------------------------------
\newmacro{\hreg}{h} % for regularizer
\newmacro{\proxdom}{\points_{\hreg}} % for prox-domain
\newmacro{\breg}{D} % for Bregman divergence
\newmacro{\mprox}{P} % for prox-mapping
\newmacro{\hconj}{h^{\ast}} % for conjugate
\newmacro{\mirror}{Q} % for mirror map
\newmacro{\fench}{F} % for Fenchel coupling
\newmacro{\hker}{\theta} % for kernel
\newmacro{\hstr}{K} % for strong convexity modulus
\newmacro{\hrange}{H} % for regularizer depth
\newmacro{\learn}{\eta} % for learning rate
\newmacro{\weight}{\lambda} % for regularization weight
\DeclarePairedDelimiterXPP{\bregof}[2]{\breg}{(}{)}{}{#1,#2} % for Bregman divergence
\DeclarePairedDelimiterXPP{\bregofX}[3]{\breg_{#1}}{(}{)}{}{#2,#3} % for Bregman divergence, specified
\DeclarePairedDelimiterXPP{\fenchof}[2]{\fench}{(}{)}{}{#1,#2} % for Fenchel coupling
\DeclarePairedDelimiterXPP{\fenchofX}[3]{\fench_{#1}}{(}{)}{}{#2,#3} % for Fenchel coupling, specified
\DeclarePairedDelimiterXPP{\proxof}[2]{\mprox_{#1}}{(}{)}{}{#2} % for Bregman prox step
\newmacro{\choice}{\mirror} % for choice map
\newmacro{\zone}{\mathbb{D}} % for Bregman zone
\newcommand{\lazy}[1]{#1^{\textup{lazy}}} % for lazy iterate
\newcommand{\eager}[1]{#1^{\textup{eager}}} % for eager iterate
\newop{\Eucl}{\Pi} % for Euclidean projection
\newop{\logit}{\Lambda} % for exponential choice map
\newop{\dkl}{KL} % for Kullback Leibler
\newmacro{\dvec}{w} % for generic dual vector
\newmacro{\dpoint}{y} % for generic dual point
\newmacro{\dpointalt}{\alt\dpoint} % for second dual point
\newmacro{\dpoints}{\mathcal{Y}} % for set of dual points
\newmacro{\score}{y} % for generic score
\newmacro{\scorealt}{\alt\score} % for second score
\newmacro{\scoreaux}{z} % for auxiliary score
\newmacro{\scores}{\payspace} % for set of scores
%----------------------------------------------------------------------
%% Stochastic analysis
%----------------------------------------------------------------------
\newmacro{\momexp}{p} % for moment exponent
\newmacro{\state}{X} % for main state variable
\newmacro{\dstate}{Y} % for dual state variable
\newmacro{\drift}{b} % for drift
\newmacro{\diffmat}{\Sigma} % for diffusion matrix
%----------------------------------------------------------------------
%% Stochastic processes
%----------------------------------------------------------------------
\newmacro{\brown}{W} % for Wiener process
\newmacro{\ito}{M} % for Itô martingale process
\newmacro{\levy}{L} % for Lévy process
\newmacro{\cadshort}{S} % for short cadlag jumps
\newmacro{\cadlong}{U} % for unbounded cadlag jumps
\ifparenargs
\DeclarePairedDelimiterXPP{\brownof}[1]{\brown}{(}{)}{}{#1} % for Wiener process in time
\DeclarePairedDelimiterXPP{\brownofi}[1]{\brown_{\iCoord}}{(}{)}{}{#1} % for i-indexed Wiener process
\DeclarePairedDelimiterXPP{\brownofj}[1]{\brown_{\jCoord}}{(}{)}{}{#1} % for j-indexed Wiener process
\DeclarePairedDelimiterXPP{\brownofX}[2]{\brown_{#1}}{(}{)}{}{#2} % for indexed Wiener process
\DeclarePairedDelimiterXPP{\itoof}[1]{\ito}{(}{)}{}{#1} % for Itô process in time
\DeclarePairedDelimiterXPP{\itoofi}[1]{\ito_{\iCoord}}{(}{)}{}{#1} % for i-indexed Itô process
\DeclarePairedDelimiterXPP{\itoofj}[1]{\ito_{\jCoord}}{(}{)}{}{#1} % for j-indexed Itô process
\DeclarePairedDelimiterXPP{\itoofX}[2]{\ito_{#1}}{(}{)}{}{#2} % for indexed Itô process
\DeclarePairedDelimiterXPP{\levyof}[1]{\levy}{(}{)}{}{#1} % for Lévy process in time
\DeclarePairedDelimiterXPP{\levyofi}[1]{\levy_{\iCoord}}{(}{)}{}{#1} % for i-indexed Lévy process
\DeclarePairedDelimiterXPP{\levyofj}[1]{\levy_{\jCoord}}{(}{)}{}{#1} % for j-indexed Lévy process
\DeclarePairedDelimiterXPP{\levyofX}[2]{\levy_{#1}}{(}{)}{}{#2} % for indexed Lévy process
\DeclarePairedDelimiterXPP{\stateof}[1]{\state}{(}{)}{}{#1} % for main state
\DeclarePairedDelimiterXPP{\stateofi}[1]{\state_{\iCoord}}{(}{)}{}{#1} % for i-indexed main state
\DeclarePairedDelimiterXPP{\stateofj}[1]{\state_{\jCoord}}{(}{)}{}{#1} % for j-indexed main state
\DeclarePairedDelimiterXPP{\stateofX}[2]{\state_{#1}}{(}{)}{}{#2} % for indexed main state
\DeclarePairedDelimiterXPP{\dstateof}[1]{\dstate}{(}{)}{}{#1} % for dual state
\DeclarePairedDelimiterXPP{\dstateofi}[1]{\dstate_{\iCoord}}{(}{)}{}{#1} % for i-indexed dual state
\DeclarePairedDelimiterXPP{\dstateofj}[1]{\dstate_{\jCoord}}{(}{)}{}{#1} % for j-indexed dual state
\DeclarePairedDelimiterXPP{\dstateofX}[2]{\dstate_{#1}}{(}{)}{}{#2} % for indexed dual state
\DeclarePairedDelimiterXPP{\dotdstateof}[1]{\dot\dstate}{(}{)}{}{#1} % for dual state
% \DeclarePairedDelimiterXPP{\learnof}[1]{\learn}{(}{)}{}{#1} % for learning rate
% \DeclarePairedDelimiterXPP{\dotlearnof}[1]{\dot\learn}{(}{)}{}{#1} % for learning rate
% \DeclarePairedDelimiterXPP{\shortof}[1]{\cadshort}{(}{)}{}{#1} % for short jumps
% \DeclarePairedDelimiterXPP{\longof}[1]{\cadlong}{(}{)}{}{#1} % for unbounded jumps
\else
\newcommand{\brownof}[1]{\brown_{#1}} % for Wiener process in time
\newcommand{\brownofi}[1]{\brown_{\iCoord,#1}} % for i-indexed Wiener process
\newcommand{\brownofj}[1]{\brown_{\jCoord,#1}} % for j-indexed Wiener process
\newcommand{\brownofX}[2]{\brown_{#1,#2}} % for indexed Wiener process
\newcommand{\itoof}[1]{\ito_{#1}} % for Itô process in time
\newcommand{\itoofi}[1]{\ito_{\iCoord,#1}} % for i-indexed Itô process
\newcommand{\itoofj}[1]{\ito_{\jCoord,#1}} % for j-indexed Itô process
\newcommand{\itoofX}[2]{\ito_{#1,#2}} % for indexed Itô process
\newcommand{\levyof}[1]{\levy_{#1}} % for Lévy process in time
\newcommand{\levyofi}[1]{\levy_{\iCoord,#1}} % for i-indexed Lévy process
\newcommand{\levyofj}[1]{\levy_{\jCoord,#1}} % for j-indexed Lévy process
\newcommand{\levyofX}[2]{\levy_{#1,#2}} % for indexed Lévy process
\newcommand{\stateof}[1]{\state_{#1}} % for main state
\newcommand{\stateofi}[1]{\state_{\iCoord,#1}} % for i-indexed main state
\newcommand{\stateofj}[1]{\state_{\jCoord,#1}} % for j-indexed main state
\newcommand{\stateofX}[2]{\state_{#1,#2}} % for indexed main state
\newcommand{\dstateof}[1]{\dstate_{#1}} % for dual state
\newcommand{\dstateofi}[1]{\dstate_{\iCoord,#1}} % for i-indexed dual state
\newcommand{\dstateofj}[1]{\dstate_{\jCoord,#1}} % for j-indexed dual state
\newcommand{\dstateofX}[2]{\dstate_{#1,#2}} % for indexed dual state
\newcommand{\dotdstateof}[1]{\dot\dstate_{#1}} % for dual state
% \newcommand{\learnof}[1]{\learn_{#1}} % for learning rate
% \newcommand{\dotlearnof}[1]{\dot\learn_{#1}} % for learning rate
% \newcommand{\shortof}[1]{\cadshort_{#1}} % for short jumps
% \newcommand{\longof}[1]{\cadlong_{#1}} % for unbounded jumps
\fi
\newmacro{\pois}{N} % for Poisson measure
\newmacro{\cpois}{\tilde{N}} % for Compensated Poisson measure
\newmacro{\jump}{z} % jump variable
\DeclarePairedDelimiterXPP{\poisof}[2]{\pois}{(}{)}{}{#1, #2} % for Poisson measure in time and space
\DeclarePairedDelimiterXPP{\cpoisof}[2]{\cpois}{(}{)}{}{#1, #2} % for compensated Poisson measure in time and space
\newmacro{\jumpof}{\Delta} % for jump processes
\newmacro{\levymes}{\nu} % for Lévy measure
\newmacro{\levymeas}{\levymes} % for Lévy measure
\newmacro{\cjump}{c} % threshold between bounded and unbounded jumps
%\newmacro{\cjumpinf}{\norm{z} < c} % set of bounded jumps
\newmacro{\cjumpinf}{\dnorm{\jumpvar} < \cjump} % set of bounded jumps
%\newmacro{\cjumpsup}{\norm{z} \geq c} % set of unbounded jumps
\newmacro{\cjumpsup}{\dnorm{\jumpvar} \geq \cjump} % set of unbounded jumps
%----------------------------------------------------------------------
%% Stochastic approximation
%----------------------------------------------------------------------
\newmacro{\model}{\hat\vecfield} % for signal
\newmacro{\sgrad}{\gvec} % for stochastic gradients
\newmacro{\step}{\gamma} % for step-size
\newmacro{\runtime}{\tau} % for proper time
\newmacro{\stepexp}{\ell_{\step}} % for step-size exponent
\newmacro{\learnexp}{\ell_{\learn}} % for learning rate exponent
\newmacro{\apt}{X} % for generic trajectory
\DeclarePairedDelimiterXPP{\aptof}[1]{\apt}{(}{)}{}{#1} % for trajectory
\DeclarePairedDelimiterXPP{\aptofX}[2]{\apt_{#1}}{(}{)}{}{#2} % for indexed trajectory
\newop{\orcl}{\mathsf{G}} % for oracle
\newop{\err}{\mathsf{\noise}} % for error
\newmacro{\seed}{\omega} % for seed
\newmacro{\seeds}{\Omega} % for seed space
%\newmacro{\error}{Z} % for error
\newmacro{\noise}{U} % for noise
\newmacro{\bias}{b} % for bias
\newmacro{\bbound}{B} % for bias bound
%\newmacro{\nbound}{\sdev} % for noise bound
\newmacro{\totbound}{\paybound} % for magnitude bound
\newmacro{\mombound}{V} % for moment bound
\newmacro{\snoise}{\xi} % for scalar noise
\newmacro{\sbias}{\chi} % for scalar bias
\newmacro{\mix}{\delta} % for query radius
\newmacro{\perturb}{z} % for perturbation
\newmacro{\pivot}{\point} % for pivot point
%**********************************************************************
%*** MACROS: AUTHOR-SPECIFIC
%**********************************************************************
%----------------------------------------------------------------------
%% PM MACROS
%----------------------------------------------------------------------
\addauthor[Pan]{PM}{MediumBlue}
\newcommand{\PM}{\PMmargincomment}
%\newcommand{\highbar}[1]{\overline{\mkern-2mu #1 \mkern-2mu}}
%\newcommand{\lowbar}[1]{\underline{\mkern-5mu #1 \mkern-2mu}}
\newcommand{\negspace}{\!\!\!} % for short negative space
\newmacro{\energy}{E} % for energy
\newmacro{\meas}{\mu} % for measure
\newmacro{\hit}{\tau} % for stopping time
\newmacro{\toler}{\eps} % for tolerance
\newmacro{\jumpvar}{z} % for jump variable
\newcommand{\contbound}{\sdev_{0}} % for magnitude of continuous part
%\newcommand{\shortbound}{\sdev_{<\cjump}} % for magnitude of short jumps
%\newcommand{\longbound}{\sdev_{\geq\cjump}} % for magnitude of long jumps
\newcommand{\shortbound}{\sdev_{\textup{short}}} % for magnitude of short jumps
\newcommand{\longbound}{\sdev_{\textup{long}}} % for magnitude of long jumps
\newcommand{\tamebound}{\sdev_{\textup{tame}}} % for tame noise
\newcommand{\heavybound}{\sdev_{\textup{heavy}}} % for heavy noise
\newcommand{\noisebound}{\sdev_{\textup{tot}}} % for total noise
\newmacro{\firstmart}{\xi} % for first martingale
\newmacro{\secondmart}{\zeta} % for second martingale
\newmacro{\cvxrate}{R}
\newmacro{\occrate}{B}
\newmacro{\tamecoef}{V_{\textup{tame}}}
\newmacro{\heavycoef}{V_{\textup{heavy}}}
\newmacro{\noisecoef}{V_{\sdev}}
\newmacro{\tameradius}{\precdist_{\textup{tame}}}
\newmacro{\heavyradius}{\precdist_{\textup{heavy}}}
\newmacro{\learnradius}{\precdist_{\learn}}
\newmacro{\critradius}{\precdist_{c}}
%----------------------------------------------------------------------
%% PL MACROS
%----------------------------------------------------------------------
\addauthor{PL}{MediumOrange}
\newcommand{\PL}{\PLmargincomment}
\newmacro{\poisful}{\poisof{d\time}{d\jump}}
\newmacro{\cpoisful}{\cpoisof{d\time}{d\jump}}
\newmacro{\poisfulalt}{\poisof{d\timealt}{d\jump}}
\newmacro{\cpoisfulalt}{\cpoisof{d\timealt}{d\jump}}
\DeclarePairedDelimiterXPP{\H2of}[2]{\mathcal{H}^2}{(}{)}{}{#1; #2} % space of integrable processes wrt. BM
\DeclarePairedDelimiterXPP{\Hlevyof}[4]{\mathcal{H}_{\levymes}^{#1}}{(}{)}{}{#2, #3; #4} % space of integrable processes wrt. compensated Poisson measure
\newmacro{\contnoise}{\contbound} % variance of the continuous part
\newmacro{\smalljumps}{\shortbound} % variance of the discontinuous part with bounded jumps
\newmacro{\bigjumps}{\longbound} % ~ p-moment of the the discontinuous part with unbounded jumps
\newmacro{\hess}{\nabla^2} % hessian
\newmacro{\diamX}{\norm{\points}} % diameter of X
\newcommand{\diamh}{\hrange} % h_max - h_min
\newcommand{\hcst}{\hstr} % strong convexity constant of h
\newmacro{\stateinit}{\point_0}
\newmacro{\dstateinit}{\dpoint_0}
\DeclarePairedDelimiterXPP{\meyerof}[1]{}{\langle}{\rangle}{}{#1} % Meyer's angular bracket / predictable quadratic variation
\DeclarePairedDelimiterXPP{\qvarof}[1]{}{[}{]}{}{#1} % quadratic variation
\newmacro{\precval}{\eps} % precision for values & complexity rates
\newmacro{\precdist}{\delta} % precision for distances
%**********************************************************************
%*** FRONT MATTER AND METADATA
%**********************************************************************
%----------------------------------------------------------------------
%% TITLE & AUTHORS
%----------------------------------------------------------------------
\title
[Stochastic mirror descent with heavy-tailed noise]
{Bregman Meets Lévy: Stochastic Mirror Descent\\
with Heavy-Tailed Noise in Continuous and Discrete Time}
%-------------------------------------------------------------------
\author
[P.~L.~Cauvin]
{Pierre-Louis Cauvin$^{c,\ast}$}
\address{$^{c}$\,%
Corresponding author.}
\address{$^{\ast}$\,%
Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France.}
\email{pierre-louis.cauvin@univ-grenoble-alpes.fr}
%-------------------------------------------------------------------
\author
[P.~Mertikopoulos]
{Panayotis Mertikopoulos$^{\ast}$}
\email{panayotis.mertikopoulos@imag.fr}
%-------------------------------------------------------------------
%----------------------------------------------------------------------
%% KEYWORDS
%----------------------------------------------------------------------
\subjclass[2020]{%
Primary 90C25, 60H10;
secondary 60G51, 90C15, 90C30, 68Q25.}
\keywords{%
Heavy-tailed noise;
stochastic mirror descent;
Lévy mirror flow;
Lévy processes;
concentration;
first-passage times;
rates of convergence.}
%----------------------------------------------------------------------
%% THANKS
%----------------------------------------------------------------------
%\thanks{Thanks go here.}
%----------------------------------------------------------------------
%% ACRONYMS
%----------------------------------------------------------------------
\newacro{LHS}{left-hand side}
\newacro{RHS}{right-hand side}
\newacro{iid}[i.i.d.]{independent and identically distributed}
\newacro{lsc}[l.s.c.]{lower semi-continuous}
\newacro{usc}[u.s.c.]{upper semi-continuous}
\newacro{rv}[r.v.]{random variable}
\newacro{whp}{with high probability}
\newacro{wp1}[w.p.$1$]{with probability $1$}
\newacro{cadlag}[càdlàg]{right continuous with left limits\,/\,\emph{continue à droite, limite à gauche}}
\newacro{NE}{Nash equilibrium}
\newacroplural{NE}[NE]{Nash equilibria}
\newacro{DA}{dual averaging}
\newacro{MD}{mirror descent}
\newacro{LMD}{lazy mirror descent}
\newacro{MF}{mirror flow}
\newacro{SMF}{stochastic mirror flow}
\newacro{LMF}{Lévy mirror flow}
\newacro{SDE}{stochastic differential equation}
\newacro{SGD}{stochastic gradient descent}
\newacro{SDA}{stochastic dual averaging}
\newacro{SMD}{stochastic mirror descent}
\newacro{SG}{stochastic gradient}
\newacro{SGO}{stochastic gradient oracle}
\newacro{SFO}{stochastic first-order oracle}
%**********************************************************************
%*** MAIN DOCUMENT BEGINS HERE
%**********************************************************************
\begin{document}
%----------------------------------------------------------------------
%% ABSTRACT
%----------------------------------------------------------------------
\begin{abstract}
\input{Abstract.tex}
\end{abstract}
%**********************************************************************
%*** BODY TEXT
%**********************************************************************
\allowdisplaybreaks % for breaking long displays
\acresetall % for resetting acros
\maketitle
\acused{iid}
\acused{LHS}
\acused{RHS}
%----------------------------------------------------------------------
%% INTRODUCTION
%----------------------------------------------------------------------
\section{Introduction}
\label{sec:introduction}
\input{Body/Introduction}
%----------------------------------------------------------------------
%% PRELIMS
%----------------------------------------------------------------------
\section{Setup and preliminaries}
\label{sec:prelims}
\input{Body/Prelims}
%----------------------------------------------------------------------
%% CONTINUOUS
%----------------------------------------------------------------------
\section{Analysis and results in continuous time}
\label{sec:continuous}
\input{Body/Continuous}
%----------------------------------------------------------------------
%% DISCRETE
%----------------------------------------------------------------------
\section{Analysis and results in discrete time}
\label{sec:discrete}
\input{Body/Discrete}
%----------------------------------------------------------------------
%% DISCUSSION
%----------------------------------------------------------------------
\section{Concluding remarks}
\label{sec:discussion}
\input{Body/Discussion}
%**********************************************************************
%*** ACKNOWLEDGMENTS
%**********************************************************************
\section*{Acknowledgments}
\input{Acks}
%**********************************************************************
%*** APPENDICES
%**********************************************************************
\appendix
\crefalias{section}{appendix}
\crefalias{subsection}{appendix}
\numberwithin{equation}{section} % for numbering in the appendix
\numberwithin{lemma}{section} % for numbering in the appendix
\numberwithin{proposition}{section} % for numbering in the appendix
\numberwithin{theorem}{section} % for numbering in the appendix
\numberwithin{corollary}{section} % for numbering in the appendix
\numberwithin{remark}{section} % for numbering in the appendix
%----------------------------------------------------------------------
%% APP: RELATED WORKS
%----------------------------------------------------------------------
\section{Related works}
\label{app:related}
\input{Appendix/App-Related-Works}
%----------------------------------------------------------------------
%% APP: BREGMAN
%----------------------------------------------------------------------
\section{Basic properties of Bregman methods}
\label{app:mirror}
\input{Appendix/App-Mirror}
%----------------------------------------------------------------------
%% APP: LEVY
%----------------------------------------------------------------------
\section{A primer on Lévy processes}
\label{app:levy}
\input{Appendix/App-Levy}
%----------------------------------------------------------------------
%% APP: CONTINUOUS
%----------------------------------------------------------------------
\section{Results in continuous time: Omitted proofs from \cref{sec:continuous}}
\label{app:continuous}
\input{Appendix/App-Continuous}
%----------------------------------------------------------------------
%% APP: DISCRETE
%----------------------------------------------------------------------
\section{Results in discrete time: Omitted proofs from \cref{sec:discrete}}
\label{app:discrete}
\input{Appendix/App-Discrete}
%**********************************************************************
%*** BIBLIOGRAPHY
%**********************************************************************
\bibliographystyle{icml}
\bibliography{bibtex/IEEEabrv,bibtex/Bibliography-PM,bibtex/HeavyTailsMD}
\end{document}
%----------------------------------------------------------------------
%% ABSTRACT
%----------------------------------------------------------------------
% !TEX root = ./Main.tex
%
%
We study the robustness of \acf{SMD} under heavy-tailed noise, focusing on whether the method retains its convergence guarantees when run with infinite-variance stochastic gradient input.
To address this question in a principled manner, we begin by introducing a continuous-time model of \ac{SMD} as a \ac{SDE} driven by a centered Lévy noise process with finite $\momexp$-th order moments, $1 < \momexp \leq 2$.
This scheme\textemdash which we call the \acdef{LMF}\textemdash arises naturally as the scaling limit of \ac{SMD} in the presence of heavy-tailed noise.
In particular, when $\momexp < 2$\textemdash the \define{heavy noise} regime\textemdash the trajectories of \ac{LMF} generically exhibit jump discontinuities of arbitrary magnitude which, if frequent enough, lead to infinite variance.
Nonetheless, despite this highly singular behavior, we show that \ac{LMF} attains $\precval$-optimality within $\bigoh(\precval^{-\momexp/(\momexp-1)})$ time in the convex case,
and within $\tildeoh(\precval^{-1/(\momexp-1)})$ time for (relatively) strongly convex objectives.
These guarantees provide a transparent characterization of the impact of frequent long jumps on the convergence of the process, and percolate to a series of matching discrete-time guarantees for several variants of \ac{SMD} under heavy-tailed noise.
%----------------------------------------------------------------------
%% INTRODUCTION
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
%----------------------------------------------------------------------
%% Background
%----------------------------------------------------------------------
\para{Background and motivation}
\Acf{SMD} and its variants comprise one of the most widely studied classes of first-order methods in stochastic optimization.
Originally conceived by \citet{Nem79} and \citet{NY83} as a general scheme for solving constrained convex minimization problems, \ac{SMD} has since grown into a mature algorithmic template with applications in min-max problems and variational inequalities \cite{NJLS09,Nes07,Nes09}, online optimization and bandits \cite{AB10,ALT15,BCB12,LS20},
games and equilibrium problems \cite{MZ19,CBL06,JNT11},
sampling \cite{HKRC18,CLL+20},
optimal transport \cite{ZPFP20,PC19},
and many other fields that are central to machine learning.
At a high level, \acl{MD} methods proceed by
replacing Euclidean projections with a non-Euclidean ``prox-type'' step\textemdash or, dually to this, by aggregating gradient steps in an unconstrained, dual space, and performing a ``mirror step'' at each iteration to generate a new query point and obtain a new gradient sample.
This mechanism links the geometry of the underlying problem with the prox-step defining the method and which, if chosen appropriately, can lead to superior, almost-dimension-free convergence guarantees.
A major challenge in this setting occurs when the gradient samples that are fed into the algorithm have infinite variance, thus jeopardizing the method's convergence\textemdash which usually presumes tame, light-tailed noise.
This is a particularly pernicious issue as, even in $1$-dimensional quadratic problems, running ``vanilla'' \ac{SGD} with heavy-tailed noise could lead to divergence in cases where the method is otherwise convergent \cite{HFH24,FHL25,ZKV+20}.
Due to the massive\textemdash and massively complex\textemdash structure of deep neural nets, this ``heavy noise'' regime is ubiquitous in contemporary machine learning models and architectures, \cf \cite{HM21b,GSZ21,Sim17,LZZ23,ZKV+20,WGZ+21} and references therein.
Especially in the context of deep learning, the heavy-tailed behavior of gradient noise was first highlighted by \citet{SSG19}, who argued empirically that gradient minibatches in deep nets follow a heavy-tailed $\alpha$-stable noise distribution, even in relatively small datasets\textemdash such as MNIST and CIFAR10.
Even though these findings were later challenged by \citet{XSS20}, subsequent work\textemdash both empirical and theoretical\textemdash has reaffirmed the heavy-tailed profile of the noise in a wide range of training regimes, from adaptive, momentum-based methods, to recurrent networks \citep{ZKV+20}, deep convolutional architectures \citep{ZFM+20,BWL24}, large language models \citep{ACS+23}, and reinforcement learning \citep{GZP+21}.
Several theoretical works further attribute these phenomena to multiplicative noise effects leading to the convergence of the iterates toward a heavy-tailed stationary distribution \citep{GSZ21,HM21b,PDS23,DM24a}.
%----------------------------------------------------------------------
%% Contribs
%----------------------------------------------------------------------
\para{Our contributions in the context of related work}
Our aim in this paper is precisely this:
\emph{to examine the robustness of \acl{SMD} methods under heavy noise.}
Building on a long-standing trend in optimization, we begin with a continuous-time viewpoint and we introduce the \acdef{LMF}, a \acf{SDE} formulation of \ac{SMD} driven by a centered Lévy noise process.
In contrast to existing \ac{SDE} models of \acl{MD} driven by \emph{Brownian} noise \cite{RB13,KBB15,MerSta18,MS21,MerSta18b,MS17-CDC,HKRC18,ZPFP20,CLL+20,LTVW22}, the introduction of Lévy noise changes the landscape dramatically.
If the \ac{SDE} is driven by an Itô, Brownian-like martingale, the solutions of the \ac{SDE} are inherently ``diffusive'':
\begin{enumerate*}
[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
they exhibit tame, light-tailed increments;
\item
their accumulation leads to a finite-variance process;
and
\item
the resulting sample paths are continuous \as.
\end{enumerate*}
By contrast, under Lévy noise, \ac{LMF} trajectories are ``purely discontinuous'':
\begin{enumerate*}
[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
their increments are no longer Gaussian but heavy-tailed;
\item
they may\textemdash \emph{and do!}\textemdash have infinite variance;
and
\item
their sample paths may\textemdash \emph{and do!}\textemdash exhibit jump discontinuities of arbitrary magnitude.
\end{enumerate*}
This leads to a highly challenging optimization landscape, especially because Itô's formula\textemdash the workhorse of stochastic calculus\textemdash cannot be applied in this context.
A key technical contribution of our paper\textemdash which, to the best of our knowledge, is new in the stochastic analysis literature\textemdash is a \emph{weak Itô formula} for convex functions that are not $C^{2}$ (so the standard Itô formula does not apply).
Thanks to this new technical tool, we analyze several variants of the method, with both constant and variable learning rate, and we establish the following series of results:
\begin{enumerate}
%[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
In convex problems, time-averaged \ac{LMF} orbits attain an $\precval$-approximate solution within $\tildeoh\parens[\big]{\precval^{-\momexp/(\momexp-1)}}$ time.
\item
In strongly convex problems, \ac{LMF} orbits converge at a geometric rate to an ``uncertainty ball'' whose radius $\learnradius$ scales with the method's learning rate and the intensity of the noise, and the fraction of the time spent $\precdist$-far from the problem's global minimum scales as $\bigoh(\learnradius^{2}/\precdist^{2})$.
\item
If the objective function is strongly convex relative to the method's Bregman function, \ac{LMF} orbits get within $\precdist$ of the problem's minimum in $\tildeoh(\precdist^{-2\momexp/(\momexp-1)})$ time (in terms of both ``first-passage'' and ``last-iterate'' guarantees).
\end{enumerate}
Subsequently, we turn to the bona fide, discrete-time setting, and we study different variants of \ac{SMD}
%\textemdash the ``dual averaging'' and ``lazy'' variants\textemdash
under heavy-tailed noise.
To connect with existing results, we first establish an $\tildeoh\parens[\big]{\precval^{-\momexp/(\momexp-1)}}$ ergodic convergence guarantee for relatively smooth convex functions, echoing the bounds of \citet{NY83} and \citet{VYB+22} for \ac{SMD} under a ``uniform smoothness'' assumption.
%(the only references that we are aware of that provide comparable convergence rates).
We then specialize to functions that are also strongly convex relative to the method's generating Bregman function:
here, we establish the convergence of the method's trajectories in mean square, and as in the continuous-time setting, we show that
\begin{enumerate*}
[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
the algorithm's orbits converge at a geometric rate to an ``uncertainty ball'' whose size scales with the intensity of the noise;
and
\item
the algorithm attains an $\precval$-optimal point in $\tildeoh(\precval^{-1/(\momexp-1)})$ iterations.
\end{enumerate*}
This is similar to the convergence rate estimates obtained by \citet{Liu25a} for \acl{SGD} and \acl{DA} with unbounded variance (for a more detailed comparison, see \cref{app:related}), but we are not otherwise aware of any results in the literature comparable to the derived concentration and first-passage estimates.
Importantly, our discrete-time results mirror our continuous-time findings, both qualitatively and quantitatively.
In fact, in all cases, the discrete-time bounds decompose into a term stemming from the continuous-time analysis, plus a ``discretization'' term.
This indicates that the proposed Lévy model is a faithful proxy for studying \ac{SMD} in the heavy-noise regime, thus opening up an important direction for further research in the heavy-noise regime.
%----------------------------------------------------------------------
%% PRELIMINARIES
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
%----------------------------------------------------------------------
%% Notation and terminology
%----------------------------------------------------------------------
\subsection{Notation and terminology}
\label{sec:notation}
We begin by collecting some basic definitions from convex analysis and optimization.
This is intended only to set terminology and notation;
for a detailed treatment, see \cite{Ber15,Roc70,BV04,Bub15}.
Throughout our paper, $\vecspace$ will denote a real $\vdim$-dimensional normed space with norm $\norm{\argdot}$ (typically an $L^{p}$-norm).
The dual space of $\vecspace$ will be denoted by $\dpoints \equiv \dual$, and we will write
$\dnorm{\dpoint} \defeq \sup\setdef{\braket{\dpoint}{\point}}{\norm{\point} \leq 1}$ for the \define{dual norm} of $\dpoint$ in $\dpoints$,
where
$\braket{\dpoint}{\point}$ denotes the \define{dual pairing} between $\dpoint\in\dpoints$ and $\point\in\vecspace$.
Following standard conventions in the field \cite{Roc70,RW98}, if $\obj\from\vecspace\to\R\cup\{\infty\}$ is an extended-real-valued function on $\vecspace$, we will write
$\dom\obj \defeq \setdef{\point\in\vecspace}{\obj(\point) < \infty}$ for the \define{effective domain} of $\obj$,
and
$\subd\obj(\point) \defeq \setdef{\dpoint\in\dpoints}{\obj(\pointalt) \geq \obj(\point) + \braket{\dpoint}{\pointalt - \point} \; \text{for all $\pointalt\in\vecspace$}}$ for the \define{subdifferential} of $\obj$ at $\point$.
Any $\gvec\in\subd\obj(\point)$ will be called a \define{subgradient} of $\obj$ at $\point$, and we will write $\dom\subd\obj \defeq \setdef{\point\in\dom\obj}{\subd\obj(\point) \neq \varnothing}$ for the \define{domain of subdifferentiability} of $\obj$.
By standard results \citep[Theorem 26.1]{Roc70}, we have $\relint\dom\obj \subseteq \dom\subd\obj \subseteq \dom\obj$.
Finally, we will use the ``soft-oh'' notation $\tildeoh(g(n))$ as a shorthand for $\bigoh(g(n)\log^{k}n)$ for some $k\geq0$, and the notation $\baroh(g(n))$ as a shorthand for $\bigoh(g(n)^{1+\littleoh(1)})$.
%We will use all this freely in the sequel.
%----------------------------------------------------------------------
%% Problem setup
%----------------------------------------------------------------------
\subsection{Problem setup}
\label{sec:setup}
In the rest of our paper, we focus on convex minimization problems of the form
\begin{equation}
\label{eq:opt}
\tag{Opt}
\begin{aligned}
\textrm{minimize}
&\quad
\obj(\point)
\\
\textrm{subject to}
&\quad
\point\in\points
\end{aligned}
\end{equation}
where $\points$ is a compact convex subset of $\vecspace$ and $\obj\from\vecspace\to\R\cup\{\infty\}
$ is a convex function satisfying the following standing assumptions:
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item
\define{Continuity:}
$\obj$ is finite and continuous on $\points$.
\item
\define{Smoothness:}
The subdifferential $\subd\obj$ of $\obj$ admits a \define{continuous selection}, \ie a continuous mapping $\nabla\obj$ such that $\nabla\obj(\point) \in \subd\obj(\point)$ for all $\point\in\dom\subd\obj$.
\end{enumerate}
%----------------------------------------------------------------------
\smallskip
\begin{remark}
In a slight abuse of terminology, we will refer to $\nabla\obj(\point)$ as the \define{gradient} of $\obj$ at $\point$.
This \emph{does not} imply\textemdash or presume\textemdash that $\subd\obj(\point)$ is a singleton;
instead, this assumption is intended to capture cases where $\dom\obj$ lies in a lower-dimensional subspace of $\vecspace$\textemdash like the probability simplex\textemdash but $\obj$ is smooth along any curve in $\points$.
Formally, it means that the directional derivative $\obj'(\point;\tanvec) = \left.d/dt\right|_{t=0^{+}} \obj(\point + t\tanvec)$ of $\obj$ at $\point$ is equal to $\braket{\nabla\obj(\point)}{\tanvec}$ for all vectors of the form $\tanvec = \pointalt-\point$, $\point\in\dom\subd\obj$, $\pointalt\in\points$.
\endenv
\end{remark}
%----------------------------------------------------------------------
We will specialize these assumptions later as needed.
For now, we only stress that the domain of subdifferentiability of $\obj$ need not extend to the boundary of $\points$;
in particular, the derivatives of $\obj$ may be unbounded over $\points$, even though $\points$ is assumed compact.
In this regard, the formulation \eqref{eq:opt} is sufficiently flexible to account for a wide range of optimization objectives that arise often in machine learning, signal processing and data science, but which may fail to be differentiable on the boundary of their domain\textemdash from Poisson problems to quantum state tomography and beyond \cite{LFN18,BBT17,AntBig06,BBDV09,BSTV18,ABM20}.
%but, otherwise, we will not mention them explicitly.
%----------------------------------------------------------------------
%% Stochastic gradients
%----------------------------------------------------------------------
\subsection{Stochastic gradients}
\label{sec:oracle}
To solve \eqref{eq:opt}, we will assume that the optimizer has access to a black-box oracle that returns a stochastic estimate of the gradient of $\obj$ at the point of interest.
Formally, when queried at a $\point\in\dom\subd\obj$, this mechanism returns a \acli{SG} of the form
\begin{equation}
\label{eq:SG}
\tag{SG}
\orcl(\point;\seed)
= \nabla\obj(\point)
+ \err(\point;\seed)
\end{equation}
where
\begin{enumerate*}
[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
$\seed$ is a random seed drawn from some (complete) probability space $\probspace$;
and
\item
$\err(\point;\seed)$ denotes an umbrella error term that captures all sources of randomness and uncertainty in the oracle.
\end{enumerate*}
In practice, $\orcl$ is queried repeatedly at a sequence of points $\curr\in\points$, $\run=\running$, with a sequence of seeds $\curr[\seed]$ drawn \ac{iid} from $\seeds$ according to $\prob$.
In so doing, we obtain the sequence of stochastic gradients
\begin{equation}
\label{eq:sgrad}
\curr[\sgrad]
= \orcl(\curr;\curr[\seed])
= \nabla\obj(\curr)
+ \curr[\noise]
% \eqcomma
\end{equation}
where $\curr[\noise]$ is a martingale difference sequence measuring the
%noise in the gradient
gradient noise at step $\run$.
For book-keeping purposes, we
%write $\curr[\filter]$ for
denote by $\curr[\filter]$
the history (adapted filtration) of $\curr$;
%, meaning in particular that
in particular, $\curr$ is $\curr[\filter]$-measurable, while $\curr[\seed]$, $\curr[\sgrad]$, and $\curr[\noise]$ are not.
With all this in hand, our standing assumptions for the gradient noise term $\curr[\noise] = \err(\curr;\curr[\seed])$ will be as follows:
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item
\define{Unbiasedness:}
%$\exof{\orcl(\point;\seed)} = \nabla\obj(\point)$ or, equivalently,
$\exof{\err(\point;\seed)} = 0$ for all $\point\in\dom\subd\obj$.
%\begin{equation}
%\label{eq:unbiased}
%\exof{\err(\point;\seed)}
% = 0
% \quad
% \text{for all $\point\in\dom\subd\obj$}.
%\end{equation}
\item
\define{Bounded $\momexp$-th central moments:}
$\exof{\dnorm{\err(\point;\seed)}^{\momexp}} \leq \sdev^{\momexp}$ for some $\momexp\in(1,2]$ and all $\point\in\dom\subd\obj$.
\end{enumerate}
Importantly, for $\momexp<2$, this assumption allows for heavy-tailed gradient input with possibly infinite variance.
For concreteness, we will refer to the case $\momexp<2$ as the \define{heavy-noise} regime, and to the case $\momexp = 2$ as the \define{tame regime}.
As we noted in the introduction, the heavy-noise regime is particularly relevant for applications to deep learning \citep{BWL24}, attention models \citep{ACS+23}, and reinforcement learning \citep{GZP+21}, where empirical studies have shown that stochastic gradients are typically heavy-tailed.
The assumption we employ above is the standard surrogate for heavy-noise models in the literature, \cf \citep{NY83,ZKV+20} and references therein.
%----------------------------------------------------------------------
%% Mirror setup
%----------------------------------------------------------------------
\subsection{Mirror descent and its friends}
\label{sec:mirror}
In the rest of our paper, we will focus on the seminal \acdef{MD} class of algorithms introduced by \citet{Nem79} and \citet{NY83} as a general scheme for solving constrained optimization problems of the form \eqref{eq:opt}.
This classical family of first-order methods replaces gradient steps by a ``mirror step'', that is, a non-Euclidean projection step which is tailored to the geometry of the problem under study.
%This classical family of first-order methods combines a gradient step with a ``mirror step'', that is, a non-Euclidean projection step tailored to the geometry of the problem under study.
The main ingredients of this template are as follows:
\begin{definition}
\label{def:mirror}
Let $\hreg\from\vecspace\to\R\cup\{\infty\}$ be a convex function with $\dom\hreg=\points$, and let $\proxdom \equiv \dom\subd\hreg$.
%denote the domain of subdifferentiability of $\hreg$.
We say that $\hreg$ is a \define{Bregman regularizer} on $\points$ if:
%We say that $\hreg\from\vecspace\to\R\cup\{\infty\}$ is a \define{Bregman regularizer} on $\points$
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
%\item
%$\dom\hreg=\points$ and $\hreg$ is continuous on $\points$.
\item
The subdifferential $\subd\hreg$ of $\hreg$ admits a continuous selection $\subsel\hreg(\point) \in \subd\hreg(\point)$ for all $\point\in\proxdom$.
\item
$\hreg$ is \define{strongly convex} relative to $\norm{\argdot}$ on $\points$, \viz
\begin{equation}
\label{eq:hstr}
\hreg(\pointalt)
\geq \hreg(\point)
+ \braket{\subsel\hreg(\point)}{\pointalt - \point}
+ \frac{\hstr}{2} \norm{\pointalt - \point}^{2}
\end{equation}
for some $\hstr>0$ and all $\point\in\proxdom$, $\pointalt\in\points$.
\end{enumerate}
We also define the \define{Bregman divergence} of $\hreg$ as
\begin{equation}
\label{eq:Bregman}
\breg(\base,\point)
= \hreg(\base)
- \hreg(\point)
- \braket{\nabla\hreg(\point)}{\base - \point}
\end{equation}
%for all $\base\in\points$, $\point\in\proxdom$,
and the associated \define{prox-mapping} as
\begin{equation}
\label{eq:mprox}
\proxof{\point}{\dvec}
= \argmin\nolimits_{\pointaux\in\points}
\braces{\braket{\dvec}{\point - \auxpoint} + \breg(\auxpoint,\point)}
\end{equation}
for all $\base\in\points$, $\point\in\proxdom$, and all $\dvec\in\dpoints$.
Finally, we define the \define{mirror map} induced by $\hreg$ as
\begin{equation}
\label{eq:mirror}
\mirror(\dpoint)
= \argmax\nolimits_{\point\in\points}
\braces{\braket{\dpoint}{\point} - \hreg(\point)}
% \quad
% \text{for all $\dpoint\in\dpoints$}.
\end{equation}
for all $\dpoint\in\dpoints$.
\endenv
\end{definition}
With all this in hand, the \acdef{SMD} algorithm unfolds iteratively as
\begin{equation}
\label{eq:SMD}
\tag{SMD}
\next
= \proxof{\curr}{-\curr[\step]\curr[\sgrad]}
\end{equation}
where $\curr[\sgrad]$, $\run=\running$, is a sequence of \aclp{SG} of the form \eqref{eq:sgrad}, and $\curr[\step] > 0$ is a variable step-size sequence, typically of the form $\curr[\step] \propto 1/\run^{\stepexp}$ for some $\stepexp\in[0,1]$.
Despite the method's remarkable and well-documented success\textemdash see \eg \cite{Bub15,Bec17,Lan20} and references therein\textemdash \eqref{eq:SMD} exhibits the somewhat strange feature that ``new gradient information enters the algorithm with decreasing weights''.
This observation was first articulated in this way by \citet{Nes09}, who used it as the starting point for the variant \acdef{SDA} scheme
\begin{equation}
\label{eq:SDA}
\tag{SDA}
\begin{aligned}
\next[\dpoint]
&= \curr[\dpoint]
- \curr[\sgrad]
\\
\next[\point]
&= \mirror(\next[\learn] \next[\dpoint])
\end{aligned}
\end{equation}
where $\curr[\learn] > 0$ is a variable nonincreasing \define{learning rate} sequence
%usually of the form $\curr[\learn] \propto 1/\run^{\learnexp}$, $\learnexp\in[0,1]$,
that \emph{post-multiplies} the aggregation of gradient steps $\next[\dpoint] \gets \curr[\dpoint] - \curr[\sgrad]$ in the dual\textemdash hence the name, see \cite{Xia10}.
More generally, \citet{Nes09} considered the template
\begin{equation}
\label{eq:SDA-gen}
\tag{SDA$_{\step,\learn}$}
\begin{aligned}
\next[\dpoint]
&= \curr[\dpoint]
- \curr[\step]\curr[\sgrad]
\\
\next[\point]
&= \mirror(\next[\learn] \next[\dpoint])
\end{aligned}
\end{equation}
which includes both a \emph{pre-} and a \emph{post-}multiplier of gradient steps.
When run with constant learning rate $\curr[\learn] \gets 1$ instead of a constant step-size, this template specializes to the iterative method known as \define{\acl{SMD} with lazy updates}\textemdash or \acli{LMD} \cite{Zin03,SS11,Bub15}\textemdash namely
\begin{equation}
\label{eq:LMD}
\tag{LMD}
\begin{aligned}
\next[\dpoint]
&= \curr[\dpoint]
- \curr[\step]\curr[\sgrad]
\\
\next[\point]
&= \mirror(\next[\dpoint])
\end{aligned}
\end{equation}
Importantly, when $\hreg$ is \define{steep}\textemdash in the sense that $\proxdom = \relint\points$,
%\footnote{Up to small technical differences\textemdash the lack of a nonempty topological interior\textemdash ``steepness'' is basically equivalent to the notion of a Legendre function.}
%%The main difference is that the domain of the latter has nonempty topological interior, so $\subd\hreg(\point)$ would be a singleton on $\relint\points$.}
so $\nabla\hreg$ blows up at the boundary of $\points$\textemdash the methods \eqref{eq:SMD} and \eqref{eq:LMD} are equivalent;
for completeness, we state and prove this fact formally in \cref{app:mirror}.
%where we also present a series of representative examples of the above algorithms.
%----------------------------------------------------------------------
\begin{remark}
At a high level, \acl{DA} provides a more flexible algorithmic template than \acl{MD}, all the while avoiding the inverse recency bias.
Discounting more recent gradient feedback in favor of older, more stale input could prove catastrophic in the presence of heavy-tailed noise, so we will focus primarily on \eqref{eq:SDA} in the sequel.
\endenv
\end{remark}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
%% CONTINUOUS
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
In this section, we introduce and analyze a general class of stochastic dynamics modeling the heavy noise regime in continuous time.
Specifically, we consider a \acf{SDE} driven by \define{Lévy noise}, a stochastic process with independent stationary increments and typically discontinuous paths.
Beyond its intrinsic interest, this continuous-time formulation serves as a principled stepping stone for understanding the convergence of \eqref{eq:SDA} under heavy noise, as it allows us to focus on the interplay between infinite-variance perturbations and unbounded jumps.
To streamline our presentation, the proofs of all results in this section are collected in \cref{app:levy,app:continuous}.
%----------------------------------------------------------------------
%% LMF
%----------------------------------------------------------------------
\subsection{The \acl{LMF}}
\label{sec:LMF}
In the absence of noise, a natural continuous-time model for \acl{MD} is given by the \acdef{MF} of $\obj$, defined here via the dynamics
\begin{equation}
\label{eq:MF}
\tag{MF}
\dotdstateof{\time}
= -\nabla\obj(\stateof{\time})
\quad
\text{with}
\quad
\stateof{\time}
= \mirror(\timed{\learn} \dstateof{\time})
\end{equation}
%where $\timed{\learn} > 0$ is a variable learning rate, as per \eqref{eq:SDA}.
where $\learn > 0$ is a nonincreasing, differentiable learning rate function, as per \eqref{eq:SDA}.
Following \cite{RB13,KBB15,MerSta18},
a standard way to account for noisy gradient observations in \eqref{eq:MF} is to consider the associated \acli{SMF}
\begin{equation}
\label{eq:SMF}
\tag{SMF}
\begin{aligned}
d\dstateof{\time}
&= -\nabla\obj(\stateof{\time}) \dd\time
+ \dd\itoof{\time}
\notag\\
\stateof{\time}
&= \mirror(\timed{\learn} \dstateof{\time})
\end{aligned}
\end{equation}
where $\itoof{\time}\in\dpoints$, $\time\geq0$, denotes a continuous square-integrable martingale.
%with values in $\dpoints$.
%In this way,
As such,
\eqref{eq:SMF}
can be seen as
%provides
a rigorous formulation of the informal Langevin dynamics
\begin{equation}
\dotdstateof{\time}
= -\nabla\obj(\stateof{\time})
+ \textrm{``noise''}
\end{equation}
with $\itoof{\time}$ playing the role of a catch-all, ``white noise'' term intended to capture all sources of randomness and uncertainty in the gradient signal $\nabla\obj(\stateof{\time})$.
Owing to its simplicity and flexibility, \eqref{eq:SMF} has been studied extensively in the literature\textemdash see \eg \cite{RB13,KB17,MerSta18,KBB15} for applications to optimization, \cite{HKRC18,ZPFP20,CLL+20,LTVW22} for sampling, and \cite{MM09,MM10,MV16,MerSta18b,BM17,MS17-CDC,CLM25,KDB14,KKDB15,LMBB25,LMBB25a} for variational inequalities, game theory, and beyond.
At the same time however, the driving noise process $\itoof{\time}$ of \eqref{eq:SMF} is inherently ``tame'':
\begin{enumerate*}
[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
its increments are Gaussian, and hence, light-tailed;
\item
their accumulation leads to a finite-variance process ($\exof{\dnorm{\itoof{\time}}^{2}} < \infty$ for all $\time$);
and
\item
as a result, the sample paths of \eqref{eq:SMF} remain (uniformly) continuous.
\end{enumerate*}
To overcome these limitations of \eqref{eq:SMF}, we will instead consider the \acli{LMF}
\begin{equation}
\label{eq:LMF}
\tag{LMF}
\begin{aligned}
d\dstateof{\time}
&= -\nabla\obj(\stateof{\time}) \dd\time
+ \dd\levyof{\time}
\notag\\
\stateof{\time}
&= \mirror(\timed{\learn} \dstateof{\time})
\end{aligned}
\end{equation}
where, echoing our assumptions for \eqref{eq:SG}, the process $\levyof{\time}$ is a $\dpoints$-valued centered Lévy process with $\exof{\dnorm{\levyof{\time}}^{\momexp}} < \infty$ for some $\momexp\in(1,2]$.
For completeness, we provide all necessary background on Lévy processes in \cref{app:levy}.
For the moment, we only note that, for $\momexp<2$, \eqref{eq:LMF} should be seen as a formal version of the dynamics
\begin{equation}
\dotdstateof{\time}
= -\nabla\obj(\stateof{\time})
+ \textrm{``heavy noise''}
\end{equation}
in the sense that
\begin{enumerate*}
[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
the increments of $\levyof{\time}$ are no longer Gaussian but heavy-tailed;
\item
the variance of $\levyof{\time}$ may be infinite;
and
\item
the sample paths of \eqref{eq:SMF} may exhibit jump discontinuities of arbitrary magnitude.
\end{enumerate*}
This non-diffusive behavior is much closer to what one would intuitively expect from the ``heavy noise'' regime.
Formally, referring to \cref{app:levy} for a more detailed treatment, a centered Lévy process $\levyof{\time}$ is a stochastic process with $\levyof{\tstart} = 0$, independent stationary increments, and sample paths that are \ac{cadlag}.
In particular, by the Lévy-Itô decomposition theorem \citep[Theorem~2.4.16]{App09}, the Lévy noise process in \eqref{eq:LMF} can be decomposed as
\begin{equation}
\label{eq:Levy-Ito}
\levyof{\time}
= \itoof{\time}
+ \timed{\cadshort}
+ \timed{\cadlong}
\end{equation}
where:
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item
$\itoof{\time}$ is a continuous square-integrable martingale
of the form
%given by
\begin{equation}
\label{eq:Levy-brown}
\itoofi{\time}
= \sum_{\jCoord=1}^{\mCoords} \diffmat_{\iCoord\jCoord} \brownofj{\time}
\end{equation}
where $\brownof{\time}$ is a standard Brownian motion in $\R^{\mCoords}$ and $\diffmat\in\R^{\nCoords\times\mCoords}$ is a diffusion matrix capturing possible correlations in the components of $\itoofi{\time}$;
by this token, we refer to $\itoof{\time}$ as the \define{diffusion component} of $\levyof{\time}$.
\item
$\timed{\cadshort}$ is a purely discontinuous \ac{cadlag} square-integrable martingale with jumps bounded by some fixed\textemdash but otherwise \emph{arbitrary}\textemdash positive constant $\cjump>0$;
we refer to it as the \define{short jump component} of $\levyof{\time}$.
\item
$\timed{\cadlong}$ is a purely discontinuous \ac{cadlag} martingale with finite $\momexp$-th moments and \emph{unbounded} jumps;
we call it the \define{unbounded jump component} of $\levyof{\time}$.
\end{enumerate}
%Collectively, we will refer to $\timed{\cadshort} + \timed{\cadlong}$ as the ``discontinuous component'' of $\levyof{\time}$.
%All processes $\itoof{\time}$, $\timed{\cadshort}$ and $\timed{\cadlong}$ above are assumed to start at $0$.
Importantly, a Lévy process has infinite variance only if it exhibits unbounded jumps, so \eqref{eq:LMF} concurrently allows the analysis of two important phenomena that cannot occur in \eqref{eq:SMF}:
heavy tails and long, unbounded jumps.
The distribution of these jumps is encoded by the
%so-called
\define{Lévy measure} of $\levyof{\time}$, a
%$\sigma$-finite
measure $\levymes$ on $\realspace\exclude{0}$
%which
which counts how many jumps of a given magnitude occur in unit time, \viz
\begin{equation}
\label{eq:Levy-meas}
\levymeas(\ball)
= \exof{\card\setdef{0\leq\time\leq 1}{\jumpof\levyof{\time} \in \ball}}
\eqcomma
\end{equation}
where $\ball$ is a Borel subset of $\realspace\exclude{0}$, and
\begin{equation}
\label{eq:Levy-jump}
\jumpof\levyof{\time}
\defeq \levyof{\time} - \levyof{\time-}
% \equiv \levyof{\time} - \lim_{\timealt\to\time^{-}} \levyof{\timealt}
\end{equation}
is the jump process of $\levyof{\time}$.%
\footnote{Here and throughout, we write $u(\time-) \defeq \lim_{\timealt\to\time^{-}} u(\timealt)$ for the left limit of a \ac{cadlag} function $u(\time)$.}
With this in mind, the intensity of each component of $\levyof{\time}$ can be quantified as follows:
\begin{subequations}
\label{eq:bounds}
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item
\define{Diffusion component:}
\begin{align}
\contbound^{2}
&= \matnorm{\diffmat}^{2}
= \tr\bracks{\diffmat^{\top}\diffmat}
\eqstop
\shortintertext{%
\item
\define{Short jump intensity:}
} % end intertext
\shortbound^{2}
&= \int_{0<\dnorm{\jumpvar}<\cjump}
\hspace{-.5em}
\dnorm{\jumpvar}^{2} \,\levymeas(d\jumpvar)
\eqstop
\shortintertext{%
\item
\define{Long jump intensity:}
} % end intertext
\longbound^{\momexp}
% &= \int_{\mathrlap{\dnorm{\jumpvar}\geq\cjump}}\quad
&= \int_{\dnorm{\jumpvar}\geq\cjump}
\hspace{-.5em}
\dnorm{\jumpvar}^{\momexp} \,\levymeas(d\jumpvar)
\eqstop
\end{align}
\end{enumerate}
Finally, with a fair amount of hindsight, we also set
\begin{align}
\label{eq:tame-heavy}
\tamebound^{2}
= \contbound^{2} + \shortbound^{2}
\quad
\text{and}
\quad
\heavybound^{\momexp}
= \longbound^{\momexp}
\end{align}
for the tame and heavy part of the noise respectively.
%and
%\begin{equation}
%\label{eq:noisebound}
%\noisebound^{2}
% = \tamebound^{2}
% + 2\heavybound^{2}
%\end{equation}
%for the total intensity of the noise in \eqref{eq:LMF}.
\end{subequations}
Each of these quantities controls a specific part of the noise process $\levyof{\time}$.
%\textemdash the diffusion and the short \vs long jump components respectively.
Clearly, the diffusive regime of \eqref{eq:SMF} is recovered when $\shortbound = \longbound = 0$, \ie when $\levyof{\time}$ has no jump component.
We elaborate on all this in \cref{app:levy}, where we also discuss in detail the well-posedness of \eqref{eq:LMF}\textemdash that is, the existence and uniqueness of strong solutions from every initial condition $\dstateof{\tstart} \in \dpoints$.
%----------------------------------------------------------------------
%% Results
%----------------------------------------------------------------------
\subsection{Analysis and results}
\label{sec:results-LMF}
We now turn our focus to the convergence guarantees of \eqref{eq:LMF} in the context of \eqref{eq:opt}.
To lighten notation, we will assume in the rest of this section that \eqref{eq:LMF} is initialized at the \define{prox-center} $\point_{c} = \argmin\hreg$ of $\points$, \ie with $\dstateof{\time} \gets 0$.
To provide some background, we begin with the tame, diffusive regime of \eqref{eq:SMF}, where \cite{MerSta18} showed that the time-averaged process
$\timed{\avg\state} = (1/\time) \int_{\tstart}^{\time} \stateof{\timealt} \dd\timealt$
enjoys the %value
bound
\begin{equation}
\label{eq:cvx-SMF}
\exof{\obj(\timed{\avg\state}) - \min\obj}
= \bigoh(1/\sqrt{\time})
%\obj(\timed{\avg\state}) - \min\obj
% = \tildeoh(1/\sqrt{\time})
% = \bigoh(\sqrt{(\log\log\time) / \time})
\eqstop
\end{equation}
The analysis of \cite{MerSta18} hinges on an application of Itô's formula\textemdash the chain rule of stochastic calculus \citep{Oks13,Kuo06}\textemdash to a carefully designed energy function.
To define it, consider first the \emph{Fenchel coupling} \citep{MerSta18}
\begin{equation}
\label{eq:Fench}
\fench(\base,\dpoint)
= \hreg(\base)
+ \hconj(\dpoint)
- \braket{\dpoint}{\base}
\hspace{1.5em}
\text{for $\base\in\points$, $\dpoint\in\dpoints$},
% \eqcomma
\end{equation}
where $\hconj(\dpoint) = \max_{\auxpoint\in\points} \braces{\braket{\dpoint}{\auxpoint} - \hreg(\auxpoint)}$ denotes the convex conjugate of $\hreg$.
%In a certain sense\textemdash which we make precise in \cref{app:mirror}\textemdash
In a
%certain
sense made precise in \cref{app:mirror},
the Fenchel coupling provides a measure of ``primal-dual'' divergence between $\base\in\points$ and $\dpoint\in\dpoints$, which is compatible with the Bregman setup of \cref{def:mirror}.
Accordingly, to account for the scaling of $\dstateof{\time}$ by $\learn$ in \eqref{eq:SMF}, a
%reasonable
natural choice of energy function is the \emph{$\learn$-deflated} Fenchel coupling
\begin{equation}
\label{eq:energy}
\timed{\energy}
= \frac{1}{\timed{\learn}} \fench(\base,\timed{\learn}\dstateof{\time})
\eqstop
\end{equation}
%which, under \eqref{eq:SMF}, evolves as
%\begin{flalign}
%\label{eq:energy-Ito}
%\timed{\energy}
% \leq \eval{\energy}{\tstart}
% &+ \int_{\tstart}^{\time} \braket{\nabla\obj(\stateof{\timealt})}{\base - \stateof{\timealt}} \dd\timealt
% \notag\\
% &- \int_{\tstart}^{\time} \frac{\eval{\learn}{\timealt}}{\sqlearnof{\timealt}} \bracks{\hreg(\base) - \hreg(\stateof{\timealt})} \dd\timealt
%\end{flalign}
The first challenge that arises in studying the evolution of $\timed{\energy}$ under \eqref{eq:LMF} is that
%\begin{enumerate*}
%[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
%\item
the analogue of Itô's lemma for Lévy-type processes includes second-order jump terms corresponding to the discontinuities of $\levyof{\time}$.
Thus, given that the unbounded jump component $\timed{\cadlong}$ of $\levyof{\time}$ may have infinite variance when $\momexp<2$, we cannot use crude estimates based on smoothness arguments to control these terms as in the diffusive case.
A second major difficulty is that, in general, the Fenchel coupling is Lipschitz smooth \emph{but not} $C^{2}$-smooth, so it must first be mollified in order to derive a ``weak'' version of Itô's lemma that only holds as an inequality (as opposed to an equality).
Putting all this together is a delicate task whose end result is \cref{thm:weakIto}, a ``weak Itô formula'' for Lipschitz-smooth convex functions relative to general Lévy-type integrators.
To the best of our knowledge, this result is new in the stochastic analysis literature;
however, because the statement itself requires additional technical machinery,
%the Poisson random measure $\poisful$ of $\levyof{\time}$, its compensated counterpart $\cpoisful$, and their use as stochastic integrators\textemdash
we defer the formal presentation and proof to \cref{subsec:weakIto}.
Instead, we only present here the application of this weak Itô formula to $\timed{\energy}$.
To that end, if we set with some hindsight
\begin{equation}
\label{eq:noisecoefs}
\tamecoef^{2}
= \frac{\tamebound^{2}}{2\hstr}
\quad
\text{and}
\quad
\heavycoef^{\momexp}
= \frac{\diamX^{2-\momexp}\heavybound^{\momexp}}{\hstr^{\momexp-1}}
\eqcomma
\end{equation}
we obtain the following explicit bound:
%----------------------------------------------------------------------
\begin{restatable}{proposition}{ENERGY}
\label{prop:energy-Ito}
Under \eqref{eq:LMF}, $\timed{\energy}$ is bounded as
\begin{align}
\label{eq:energy-Ito}
\timed{\energy} - \eval{\energy}{\tstart}
&\leq \int_{\tstart}^{\time} \braket{\nabla\obj(\stateof{\timealt})}{\base - \stateof{\timealt}} \dd\timealt
+ \timed{\firstmart}
+ \timed{\secondmart}
\notag\\
&+ \tamecoef^{2}
\int_{\tstart}^{\time} \eval{\learn}{\timealt} \dd\timealt
% \notag\\
% &+ \frac{\diamX^{2-\momexp}\heavybound^{\momexp}}{\hstr^{\momexp-1}}
+ \heavycoef^{\momexp}
\int_{\tstart}^{\time} \eval{\learn}{\timealt}^{\momexp-1} \dd\timealt
% \notag\\
% &+ \timed{\firstmart}
% + \timed{\secondmart}
\notag\\
&- \int_{\tstart}^{\time} \frac{\eval{\dot\learn}{\timealt}}{\eval{\learn}{\timealt}^{2}} \bracks{\hreg(\base) - \hreg(\stateof{\timealt})} \dd\timealt
% &+ \frac{\tamebound^{2}}{2\hstr}
\eqcomma
\end{align}
where
$\diamX \defeq \max_{\point,\pointalt} \norm{\pointalt - \point}$ is the diameter of $\points$,
and
$\timed{\firstmart}$, $\timed{\secondmart}$ are martingales \textpar{the former square-integrable}.
%, the latter simply integrable}.
\end{restatable}
%----------------------------------------------------------------------
We prove \cref{prop:energy-Ito} in \cref{app:continuous}, where we also provide explicit expressions and bounds for
%the martingales
%$\timed{\firstmart}$ and $\timed{\secondmart}$
$\firstmart$ and $\secondmart$.
%that appear in \eqref{eq:energy-Ito}, \cf \cref{lem:firstmart-bound,lem:secondmart-bound}.
Thanks to these bounds, we get the following guarantee for \eqref{eq:LMF}:
%----------------------------------------------------------------------
\begin{restatable}{theorem}{CVXLMF}
\label{thm:cvx-LMF}
Under \eqref{eq:LMF}, the time-averaged process $\timed{\avg\state} = (1/\time) \int_{\tstart}^{\time} \stateof{\timealt} \dd\timealt$ enjoys the bounds
\begin{align}
\label{eq:cvx-LMF-mean}
\exof{\obj(\timed{\avg\state})}
&\leq \min\obj
+ \cvxrate_{0}(\time)
\shortintertext{and, \acl{wp1},}
\label{eq:cvx-LMF-as}
\obj(\timed{\avg\state})
\leq \min\obj
&+ \cvxrate_{0}(\time)
% + \bigoh\parens*{\time^{-\frac{\momexp-1}{\momexp} + \littleoh(1)}}
+ \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}}
% \notag\\
+ \bigoh\parens*{\tamebound \diamX \sqrt{\frac{\log\log \time}{\time}}}
%\end{align}
\shortintertext{where}
%\begin{align}
\label{eq:cvxrate}
\cvxrate_{0}(\time)
= \frac{\hrange}{\time\timed{\learn}}
% &+ \frac{\tamebound^{2}}{2\hstr\time}
&+ \frac{\tamecoef^{2}}{\time}
\int_{\tstart}^{\time} \eval{\learn}{\timealt} \dd\timealt
% \notag\\
% &+ \frac{\diamX^{2-\momexp}\heavybound^{\momexp}}{\hstr^{\momexp-1}\time}
+ \frac{\heavycoef^{\momexp}}{\time}
\int_{\tstart}^{\time} \eval{\learn}{\timealt}^{\momexp-1} \dd\timealt
\end{align}
and $\hrange \defeq \max\hreg - \min\hreg$.
In particular, if \eqref{eq:LMF} is run with learning rate $\timed{\learn} = 1/\time^{1/\momexp}$, we have
\begin{align}
\label{eq:cvxrate-spec}
\cvxrate_{0}(\time)
&= \frac{\hrange}{\time^{(\momexp-1)/\momexp}}
% + \frac{\momexp}{\momexp-1} \frac{\tamebound^{2}}{2\hstr\time^{1/\momexp}}
+ \frac{\momexp}{\momexp-1} \frac{\tamecoef^{2}}{\time^{1/\momexp}}
% + \frac{\momexp \diamX^{2-\momexp} \heavybound^{\momexp}}{\hstr^{\momexp-1}\time^{(\momexp-1)/\momexp}}
+ \momexp \frac{\heavycoef^{\momexp}}{\time^{(\momexp-1)/\momexp}}
% &= \frac{\momexp}{\momexp-1} \frac{\tamebound^{2}}{2\hstr\time^{1/\momexp}}
% + \frac{\hstr^{\momexp-1}\hrange + \momexp \diamX^{2-\momexp} \heavybound^{\momexp}}{\hstr^{\momexp-1}\time^{(\momexp-1)/\momexp}}
% \notag\\
= \bigoh(1/\time^{(\momexp-1)/\momexp})
\eqstop
\end{align}
\end{restatable}
%----------------------------------------------------------------------
\cref{thm:cvx-LMF} is our main result for convex functions, so some remarks are in order.
%----------------------------------------------------------------------
\smallskip
\begin{remark}
From the expression \eqref{eq:cvxrate-spec} for $\cvxrate_{0}(\time)$, we further observe that $\obj(\timed{\avg\state}) \to \min\obj$ whenever the learning rate satisfies $\time\timed{\learn} \to \infty$ and $\timed{\learn} \to 0$.
Among such choices, the standard schedule $\timed{\learn} \propto 1/\time^{1/\momexp}$ is the one that optimizes
%the dependence of
the resulting upper bound in $\time$.
To connect with existing results, this gives $\exof{\obj(\timed{\avg\state})} - \min\obj = \bigoh(1/\sqrt{\time})$ in the tame regime $\momexp=2$, so we recover the diffusive expression \eqref{eq:cvx-SMF} for \eqref{eq:SMF} under Brownian noise \cite{MerSta18}.
On the other hand, under heavy noise ($1 < \momexp < 2$), the convergence rate of \eqref{eq:LMF} deteriorates smoothly with $\momexp$, matching the corresponding lower bounds of \cite{NY83}.
This deterioration is due expressly to the long-jump term $\bigoh(\heavybound^{\momexp}/\time^{(\momexp-1)/\momexp})$ in \eqref{eq:cvxrate-spec}, and suggests that the Lévy-driven dynamics \eqref{eq:LMF} comprise a particularly expressive model for the study of \acl{SMD} under heavy-tailed noise.
\endenv
\end{remark}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\smallskip
\begin{remark}
\label{remark:complexity_rate_convex_continuous}
We also note that, for any error
%tolerance
$\precval > 0$,
%an application of
\cref{thm:cvx-LMF} shows that running \eqref{eq:LMF} with constant %learning rate
$\learn = \bigoh\parens[\big]{\cramped{\precval^{1/(\momexp-1)}}}$ guarantees $\exof*{\obj(\avg{\state}(\time)) - \min\obj} \leq \precval$ within time $\time = \bigoh\parens[\big]{\precval^{-\momexp/(\momexp-1)}}$.
This expression in terms of $\precval$ provides a convenient baseline for comparing convergence rates
%obtained with
under constant learning rates;
we revisit this point later.
\endenv
\end{remark}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
%% Results
%----------------------------------------------------------------------
\subsection{Finer results under (relative) strong convexity}
\label{sec:strong-LMF}
In the rest of this section, we refine and expand on the convergence guarantees of \cref{thm:cvx-LMF} for functions that are \define{strongly convex} relative to either $\norm{\cdot}$ or $\hreg$.
Formally:
\begin{enumerate}
[label=\upshape(\itshape\roman*\hspace*{.5pt}\upshape)]
\item
$\obj$ is \define{$\strong$-strongly convex} if
\begin{equation}
\label{eq:strong}
\obj(\pointalt)
\geq \obj(\point)
+ \braket{\subsel\obj(\point)}{\pointalt - \point}
+ \frac{\strong}{2} \norm{\pointalt - \point}^{2}
\end{equation}
for all $\point,\pointalt\in\points$.
\item
$\obj$ is \define{$\strong$-strongly convex relative to $\hreg$} if
\begin{equation}
\label{eq:strong-rel}
\obj(\pointalt)
\geq \obj(\point)
+ \braket{\subsel\obj(\point)}{\pointalt - \point}
+ \strong \breg(\pointalt,\point)
\end{equation}
for all $\point\in\proxdom$ and all $\pointalt\in\points$.
\end{enumerate}
The notion of relative strong convexity is due to \citet{LFN18}, and is a refinement of ``vanilla'' strong convexity:
if $\obj$ is $\strong$-strongly convex relative to $\hreg$, then it is also $(\strong\hstr)$-strongly convex relative to $\norm{\cdot}$.
The converse, however, does not hold, especially when $\hreg$ is \define{steep} (that is, $\proxdom = \relint\points$).
In this case, the Bregman divergence $\breg(\pointalt,\point)$ may grow unbounded as $\point$ gets closer to the boundary of $\points$;
in turn, this means that the gradient of $\obj$ blows up at the boundary of $\points$, so a strongly convex function like $\norm{\point}^{2}$ cannot be strongly convex relative to $\hreg$ in this case.
We discuss this issue in more detail later;
for now, it suffices to keep in mind that the notion of relative strong convexity is more exacting, but also more aligned with the geometry of $\hreg$ (so one may reasonably expect better guarantees in this setting).
With all this in hand, we will examine the following questions for \eqref{eq:LMF} under (relative) strong convexity:
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item
The fraction of time that $\stateof{\time}$ spends near $\argmin\obj$.
\item
The time it takes $\stateof{\time}$ to get close to $\argmin\obj$.
%for the first time.
\item
The convergence speed of $\stateof{\time}$ itself (as opposed to its time average).
\end{enumerate}
To quantify the above,
fix a distance tolerance $\precdist>0$,
let $\sol\in\argmin\obj$ denote a solution of \eqref{eq:opt},
and let
\begin{equation}
\label{eq:nhd}
\ball_{\precdist}
= \setdef{\point\in\points}{\norm{\point - \sol} \leq \precdist}
\end{equation}
be a ball of radius $\precdist$ centered at $\sol\in\points$.
Finally, with some hindsight, consider the ``uncertainty radius''
\begin{equation}
\label{eq:learnradius}
\cramped{\learnradius^{2}}
= \frac{2}{\strong}
\bracks*{\learn \tamecoef^{2} + \learn^{\momexp-1} \heavycoef^{\momexp}}
\eqstop
\end{equation}
We then have the following guarantees for \eqref{eq:LMF}.
%----------------------------------------------------------------------
\begin{restatable}[Concentration]{theorem}{OCCMEAS}
\label{thm:occmeas}
Assume $\obj$ is $\strong$-strongly convex, and let
\begin{equation}
\label{eq:occupation}
\meas_{\time}(\ball_{\precdist})
= \frac{1}{\time}
\int_{\tstart}^{\time} \oneof{\stateof{\timealt} \in \ball_{\precdist}} \dd\timealt
% \int_{\tstart}^{\time} \oneof{\norm{\stateof{\timealt}} \leq \precdist} \dd\timealt
\end{equation}
be the fraction of time that $\stateof{\time}$ spends in $\ball_{\precdist}$ up to time $\time$.
If \eqref{eq:LMF} is run with constant $\timed{\learn} \equiv \learn$, we have
\begin{subequations}
\label{eq:occmeas}
\begin{equation}
\label{eq:occmeas-mean}
\exof{\meas_{\time}(\ball_{\precdist})}
\geq 1
- \frac{\cramped{\learnradius^{2}}}{\precdist^{2}}
- \frac{2\hrange}{\learn\strong\precdist^{2}\time}
\end{equation}
and, \acl{wp1},
\begin{align}
\label{eq:occmeas-as}
\meas_{\time}(\ball_{\precdist})
&\geq 1
- \frac{\cramped{\learnradius^{2}}}{\precdist^{2}}
- \frac{2\hrange}{\learn\strong\precdist^{2}\time}
+ \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}}
% \notag\\
+ \bigoh\parens*{\tamebound \diamX \sqrt{\frac{\log\log \time}{\time}}}
\eqstop
\end{align}
\end{subequations}
\end{restatable}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\begin{restatable}[Hitting time]{theorem}{HITTING}
\label{thm:hitting}
Assume $\obj$ is $\strong$-strongly convex, and let
\begin{equation}
\label{eq:hitting}
\hit_{\precdist}
= \inf\setdef{\time\geq 0}{\norm{\stateof{\time} - \sol} \leq \precdist}
\end{equation}
denote the first time that $\stateof{\time}$ gets within $\precdist$ of $\sol$ for some small enough $\precdist > 0$.
%If \eqref{eq:LMF} is run with constant learning rate $\timed{\learn} \equiv \learn$,
We then have the following estimates:
\begin{enumerate}
[left=0em,label=\bfseries{Case} \arabic*.]
\item
Heavy noise:
If $\momexp<2$ and \eqref{eq:LMF} is run with $\timed{\learn} = \bracks[\big]{\strong\precdist^{2} / (8 \heavycoef^{\momexp})}^{1/(\momexp-1)}$, then
\begin{equation}
\label{eq:hitting-heavy}
\exof{\hit_{\precdist}}
\leq 4\hrange
\parens*{\frac{8^{1/\momexp} \heavycoef}{\strong\precdist^{2}}}^{\momexp/(\momexp-1)}
% \frac{\parens*{8 \diamX^{2-\momexp} \heavybound^{\momexp}}^{1/(\momexp-1)}}{\parens{\strong\precdist^{2}}^{\momexp/(\momexp-1)}}
\eqstop
\end{equation}
\item
Tame noise:
If $\momexp=2$ and \eqref{eq:LMF} is run with $\timed{\learn} = \strong\precdist^{2} \cdot \hstr/ (2\tamebound^{2} + 4\heavybound^{2})$, then
\begin{equation}
\label{eq:hitting-tame}
\exof{\hit_{\precdist}}
\leq \frac{8\hrange}{\hstr} \frac{\tamebound^{2} + 2\heavybound^{2}}{\strong^{2}\precdist^{4}}
\eqstop
\end{equation}
\end{enumerate}
\end{restatable}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\begin{restatable}[Convergence]{theorem}{STRLMF}
\label{thm:str-LMF}
Assume $\obj$ is $\strong$-strongly convex relative to $\hreg$
with $\hreg$ steep.
Then:
%\eqref{eq:LMF} is run with a constant learning rate $\timed{\learn} \equiv \learn$.
%If $\hreg$ is steep, then
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item
If \eqref{eq:LMF} is run with constant $\timed{\learn} \equiv \learn$, we have:
\begin{align}
\label{eq:str-LMF-const}
\exof{\norm{\stateof{\time} - \sol}^{2}}
&\leq \cramped{\learnradius^{2}}/\hstr
% &\leq \frac{\learn\tamebound^{2}}{\strong\hstr^{2}}
% + \frac{2\diamX^{2-\momexp} \learn^{\momexp-1}\heavybound^{\momexp}}{\strong\hstr^{\momexp}}
% \notag\\
+ (2\hrange/\hstr) e^{-\strong\learn\time}
\eqstop
\end{align}
\item
If \eqref{eq:LMF} is run with $\timed{\learn} = (1+\time)^{-1/\momexp}$, we have:
\begin{align}
\label{eq:str-LMF-var}
\exof{\norm{\stateof{\time} - \sol}^{2}}
&\leq \frac{2}{\strong\hstr}
\frac{\hrange}{\momexp\time^{(\momexp-1)/\momexp}}
% \notag\\
+ \frac{2}{\strong\hstr}
\frac{\tamecoef^{2}}{\time^{1/\momexp}}
+ \frac{2}{\strong\hstr}
\frac{\heavycoef^{\momexp}}{\momexp\time^{(\momexp-1)/\momexp}}
\notag\\
&+ \frac{2\hrange}{\hstr}
% \exp\parens*{-\frac{\strong\momexp}{\momexp - 1}\parens*{\time^{\frac{\momexp-1}{\momexp}} - 1}}
\exp\parens*{- \strong\momexp \frac{\time^{1-1/\momexp} - 1}{\momexp - 1}}
\eqstop
\end{align}
In particular, $\stateof{\time} \to \sol$ in mean square as $\time\to\infty$.
\end{enumerate}
\end{restatable}
\Cref{thm:occmeas,thm:hitting,thm:str-LMF} are our main results for \eqref{eq:LMF} in the strongly convex regime, so we close this section with some remarks.
%----------------------------------------------------------------------
\smallskip
\begin{remark}
Note that \cref{thm:str-LMF} is stated in the context of \emph{relative}\textemdash as opposed to ordinary\textemdash strong convexity.
This is due to the fact that Grönwall-type arguments for the iterates of \acl{MD} (as opposed to time averages) require a two-way compatibility between norms and divergences, and this is absent when $\obj$ is norm-strongly convex.
This disparity is well-documented in the literature\textemdash for both stochastic \cite{Lu19,AIMM21} and deterministic methods \cite{LFN18,AIMM24}\textemdash so it is natural that \eqref{eq:LMF} is also subject to it.
For completeness, we also record in \cref{app:continuous} an ergodic convergence result under ``vanilla'' strong convexity, \cf \cref{thm:strongly_convex_continuous}.
\endenv
\end{remark}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\smallskip
\begin{remark}
It is also worth noting that \cref{thm:occmeas,thm:hitting,thm:str-LMF} paint a complementary picture.
At a high level, \cref{thm:occmeas} shows that, for any fixed $\learn$, there is a
%certain
``critical radius'' $\critradius \equiv \critradius(\learn,\strong,\sdev)$ such that, after a transient $\bigoh(1/\run)$ regime, $\stateof{\time}$ is concentrated within $\critradius$ of $\sol$, spending only an $\bigoh(\critradius^{2}/\precdist^{2})$ fraction of time $\precdist$-away from $\sol$.
This
%concentration
is reflected in the guarantee \eqref{eq:str-LMF-const} which
%likewise
shows
%that $\stateof{\time}$ converges at a geometric rate
the geometric convergence of $\stateof{\time}$
to an $\bigoh(\critradius)$-neighborhood of $\sol$
before
%the intensity of
the noise overshadows the drift of the process.
In particular, letting $\time\to\infty$ in \cref{thm:str-LMF} yields $\limsup_{\time \to \infty} \exof{\norm{\stateof{\time} - \sol}^2} \leq \critradius^{2}/\hstr$,
matching the upper bound obtained from \cref{thm:strongly_convex_continuous}, but applies to the \emph{last iterate} rather than to ergodic averages (and
%, in addition, this bound
is reached at a geometric rate under relative strong convexity, instead of $1/\time$ for standard strong convexity).
\endenv
\end{remark}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\smallskip
\begin{remark}
As in the case of \cref{thm:cvx-LMF}, running \eqref{eq:LMF} with $\learn = \bigoh\parens*{\precval^{1/(\momexp - 1)}}$ for a fixed tolerance level $\precval > 0$ guarantees $\exof*{\norm{\stateof{\time} - \sol}^2} \leq \precval$ within time $\time = \bigoh\big(\precval^{-1/(\momexp-1)} \log(1/\precval)\big)$.
Since $\momexp>1$, this is always sharper than the (ergodic) rate $\bigoh(\precval^{-\momexp/(\momexp-1)})$ of \cref{thm:cvx-LMF,thm:strongly_convex_continuous};
in particular, in the tame regime $\momexp=2$, this rate matches the well-known $\bigoh(1/\precval)$ rate for stochastic strongly convex optimization up to a logarithmic factor.
However, for $\momexp < 2$, this bound does not match the $\Omega(\precval^{-\momexp/\bracks{2(\momexp-1)}})$ lower bound \citep[Theorem 5]{ZKV+20} for smooth strongly convex functions.
We discuss this in detail in the next section.
\endenv
\end{remark}
%----------------------------------------------------------------------
%By contrast, \cref{thm:hitting} shows that it is possible to get arbitrarily close to $\sol$ at a quartic rate if the learning rate of \eqref{eq:LMF} is tuned accordingly.
%----------------------------------------------------------------------
%% PRELIMINARIES
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
With the continuous-time analysis in place, we return in this section to the discrete-time algorithmic setup of \eqref{eq:SDA}, where we show that, mutatis mutandis, the same convergence and concentration patterns persist.
The only additional element that we will require over \cref{sec:continuous} is a measure of smoothness for $\obj$ that is compatible with the Bregman geometry of $\hreg$.
This is mandated for two reasons:
First, under relative strong convexity, gradients may grow unbounded near the boundary of $\points$, so standard Lipschitz smoothness does not suffice.
Second, the heavy noise in the input is filtered through the mirror map $\mirror$, so, in order to control it in the primal space $\points$, we need a notion of smoothness which, in a sense, commutes with $\mirror$.
To that end, we will employ the notion of \define{relative smoothness} due to \cite{BDX11,BBT17,LFN18}.
Formally, $\obj$ is said to be \define{$\lips$-smooth relative to $\hreg$} if $\lips\hreg-\obj$ is convex, or, equivalently, if
\begin{equation}
\label{eq:RS}
\tag{RS}
\obj(\pointalt)
\leq \obj(\point)
+ \braket{\nabla\obj(\point)}{\pointalt - \point}
+ \lips \breg(\pointalt,\point)
\end{equation}
for all $\point\in\proxdom$, $\pointalt\in\points$.
Dually to strong convexity, if a function is $\lips$-smooth relative to $\norm{\cdot}$\textemdash that is, \eqref{eq:RS} holds with $\norm{\pointalt - \point}^{2}/2$ in lieu of $\breg(\pointalt,\point)$\textemdash then it is also $(\lips/\hstr)$-smooth relative to $\hreg$.
The converse however does not hold, and there is a broad range of problems in machine learning and data science\textemdash from Poisson inverse problems to entropically regularized optimal transport\textemdash that are smooth relative to a suitable Bregman function, but not in the ordinary sense.
With all this in hand, we state below a series of representative results for different incarnations of \ac{SMD} under heavy noise.
For convenience, we write throughout
\begin{equation}
\noisecoef^{\momexp}
= \diamX^{2-\momexp} \, (2/\hstr)^{\momexp-1} \, \sdev^{\momexp}
\end{equation}
in direct analogy with $\heavycoef$ in \eqref{eq:noisecoefs}.
In addition, to streamline our presentation,
%Unless explicitly stated otherwise,
we assume that all algorithms are initialized at the prox-center $\point_{c} = \argmin\hreg$ of $\points$, \ie with $\init[\dpoint] \gets 0$,
and we defer all proofs to \cref{app:discrete}.
We start with two results in the spirit of \cref{thm:cvx-LMF}.
%----------------------------------------------------------------------
\begin{figure*}[tbp]
\centering
\includegraphics[height=30ex]{Figures/DistributionTame.pdf}
\quad
\includegraphics[height=30ex]{Figures/DistributionHeavy.pdf}
\\[\smallskipamount]
\includegraphics[height=30ex]{Figures/FunctionValues.pdf}
\caption{The behavior of \eqref{eq:SDA} under tame and heavy noise.
We considered a strongly convex function over $\points = [0,1]\times[0,1]$ and we ran $N=200$ instances of \eqref{eq:SDA} with entropic regularization for $\nRuns = 10^{4}$ iterations and gradients perturbed by a centered $\alpha$-stable distribution with scale parameter $c=1$ and no skewness.
The density plot on the left shows a sample trajectory and the long-run distribution of the iterates of \eqref{eq:SDA} for the tame regime ($\alpha=\momexp=2$);
the one on the right shows the same information for the heavy-noise regime ($\alpha=\momexp=3/2$).
Finally, the bottom plot shows the evolution of $\obj(\last[\avg\point])$ under tame and heavy noise (red and blue coloring respectively; markers~=~mean, shading~=~range, and the dashed line represents a sample trajectory);
as predicted, $\obj(\last[\avg\point])$ follows a power law in both cases.}
\label{fig:plots}
%\vspace{-\baselineskip}
\end{figure*}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\begin{restatable}[\ac{SDA}, convex]{theorem}{CVXSDA}
\label{thm:cvx-SDA}
Suppose that $\obj$ is $\lips$-smooth relative to $\hreg$ and \eqref{eq:SDA} is run with learning rate $\curr[\learn] = \beta/\run^{1/\momexp}$, $\beta\leq 1/(\momexp \lips)$.
Then the time-averaged process $\last[\avg\point] = (1/\nRuns) \sum_{\run=\start}^{\nRuns} \curr$ enjoys the bound
\begin{align}
\exof*{\obj(\last[\avg\point])}
&\leq \min\obj
+ \frac{\obj(\init) - \min\obj}{\nRuns}
+ \frac{\diamh \hcst^{\momexp-1} + \beta^{\momexp} \, \noisecoef^{\momexp}}{\beta\nRuns^{(\momexp-1)/\momexp}}
% \notag
% \notag\\
% &+ \bracks*{\diamh / \beta + 2^{\momexp-1} \beta^{\momexp-1} \noisecoef^{\momexp}}
% \nRuns^{-\frac{\momexp-1}{\momexp}}
\eqstop
\notag
\end{align}
In particular, for
$\beta = \bracks{\hrange \hcst^{\momexp-1}/(\momexp-1)}^{1/\momexp} / \noisecoef$
%$\beta= \bracks[\big]{(\hstr/2)^{\momexp-1} \hrange \diamX^{\momexp-2} / (\momexp-1)}^{1/\momexp}/\sdev$,
we have
\begin{equation}
\exof*{\obj(\last[\avg\point])}
\leq \min\obj
+ \frac{\obj(\init) - \min\obj}{\nRuns}
% \\
% + \frac{\momexp\sdev\bracks{(\momexp-1) \diamX^{2-\momexp} (2\hrange/\hstr)^{\momexp-1}}^{1/\momexp}}{(\momexp - 1) \nRuns^{(\momexp-1)/\momexp}}\\
% + \frac{\momexp\sdev\bracks{\diamX^{2-\momexp} (2\hrange/\hstr)^{\momexp-1}}^{1/\momexp}}{\bracks{(\momexp - 1) \nRuns}^{(\momexp-1)/\momexp}}
+ \frac{\momexp \bracks{\hrange\hcst^{\momexp-1}/(\momexp-1)}^{(\momexp-1)/\momexp} \, \noisecoef}{\nRuns^{(\momexp-1)/\momexp}}
\eqstop
\end{equation}
\end{restatable}
%----------------------------------------------------------------------
Our next result concerns the relatively strongly convex case\textemdash so $\argmin\obj = \{\sol\}$ for some $\sol\in\points$.
Then:
%----------------------------------------------------------------------
\begin{restatable}[\acs{SDA}, strongly convex]{theorem}{STRSDA}
\label{thm:str-SDA}
Suppose that $\obj$ is $\strong$-strongly convex and $\lips$-smooth, both relative to $\hreg$, with $\hreg$ steep.
If \eqref{eq:SDA} is run with constant $\curr[\learn] \equiv \learn \leq 1/ (\momexp\lips)$, then
\begin{equation}
\exof[\big]{\norm{\curr - \sol}^{2}}
\leq \frac
{\learn^{\momexp - 1} \noisecoef^{\momexp}}
{\momexp \strong \hstr}
+ \frac{2\hrange}{\hstr} (1 - \learn \strong)^{\run - 1}.
\end{equation}
\end{restatable}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\begin{corollary}
\label{cor:str-SDA}
Fix some target accuracy $\precval > 0$.
If \eqref{eq:SDA} is run with small enough $\learn = \bigoh\big((\precval/\sdev^\momexp)^{1/(\momexp-1)}\big)$, then $\exof{\norm{\curr - \sol}^{2}} \leq \precval$ for all $\time = \Omega\parens*{(\sdev^{\momexp}/\precval)^{1/(\momexp-1)} \log(1/\precval)}$.
\end{corollary}
%----------------------------------------------------------------------
Our next two results echo \cref{thm:occmeas,thm:hitting}, and they estimate the fraction of time that $\curr$ spends away from $\argmin\obj$, and the time that it takes to get close to it for the first time.
%----------------------------------------------------------------------
\begin{restatable}[Concentration]{theorem}{OCCMEASDISC}
\label{thm:occmeas-disc}
Assume $\obj$ is $\strong$-strongly convex and $\lips$-smooth relatively to $\hreg$,
let $\ball_{\precdist}$ be a $\precdist$-ball in $\points$ around the global minimum $\sol$ of $\obj$,
and let
\begin{equation}
\label{eq:occupation-disc}
\meas_{\nRuns}(\ball_{\precdist})
= \frac{1}{\nRuns}
\sum_{\run=\start}^{\nRuns} \oneof{\curr \in \ball_{\precdist}}
\end{equation}
be the fraction of time that $\curr$ spends in $\ball_{\precdist}$ up to time $\nRuns$.
If \eqref{eq:SDA} is run with constant learning rate $\curr[\learn] \equiv \learn$, we have
\begin{equation}
\exof*{\meas_{\nRuns}(\ball_\precdist)}
\geq 1
- \frac{\precdist_{\learn}^{2}}{\precdist^{2}}
- \frac{2\bracks{\obj(\point_1) - \min\obj}}{\strong \precdist^2 \nRuns}
- \frac{2 \diamh}{\learn \strong \precdist^2 \nRuns}
% \\
% - \frac{2 \learn^{\momexp-1} \noisecoef^{\momexp}}{\momexp\strong\precdist^{2}}
% - \frac{2^\momexp \learn^{\momexp-1} \diamX^{2-\momexp} \sdev^\momexp}{\strong \momexp \hcst^{\momexp-1} \precdist^2}
\end{equation}
where, in a similar vein to \eqref{eq:learnradius}, we let
\begin{equation}
\label{eq:learnradius}
\learnradius^{2}
= 2 / (\strong\momexp) \cdot \learn^{\momexp-1} \noisecoef^{\momexp}
\eqstop
\end{equation}
\end{restatable}
%----------------------------------------------------------------------
%----------------------------------------------------------------------
\begin{restatable}[Hitting time]{theorem}{HITTINGDISC}
\label{thm:hitting-disc}
Assume $\obj$ is $\strong$-strongly convex and $\lips$-smooth relatively to $\hreg$, and let
\begin{equation}
\label{eq:hitting}
\hit_{\precdist}
= \inf\setdef{\run=\running}{\norm{\curr - \sol} \leq \precdist}
\end{equation}
denote the first time that $\curr$ gets within $\precdist$ of $\sol$ for some small enough $\precdist > 0$.
If \eqref{eq:SDA} is run with constant learning rate
$\learn < \cramped{\bracks{\strong\momexp\precdist^{2}/(2\noisecoef^{\momexp})}^{1/(\momexp-1)}}$, then
\begin{equation}
\label{eq:discrete_hitting_time_1}
\exof{\hit_{\precdist}}
\leq \frac{2}{\strong}
\frac{\diamh/\learn + \obj(\init) - \min\obj + \strong\precdist^{2}/2}{\precdist^{2} - \cramped{\learnradius^{2}}}
\eqstop
\end{equation}
In particular, if \eqref{eq:SDA} is run with $\learn = \cramped{\bracks{\strong\precdist^{2}/(2\noisecoef^{\momexp})}^{1/(\momexp-1)}}$, we have
\begin{align}
\label{eq:discrete_hitting_time_2}
\exof{\hit_{\precdist}}
&\leq \frac{\momexp}{\momexp-1}
\bracks*{
1
+ \frac{2\bracks{\obj(\init) - \min\obj}}{\strong\precdist^{2}}
+ \diamh \parens*{\frac{2\noisecoef}{\strong \precdist^{2}}}^{\frac{\momexp}{\momexp-1}}
} % end bracks
% \notag\\
= \bigoh\parens*{\frac{1}{\precdist^{2\momexp/(\momexp-1)}}}
\eqstop
% &\leq \dfrac{\momexp}{\momexp-1} \braces*{ \dfrac{2\diamh}{\strong\hcst} \parens*{ \dfrac{2^\momexp \diamX^{2-\momexp} \sdev^\momexp}{\strong} }^{\frac{1}{\momexp-1}} \precdist^{-\frac{2\momexp}{\momexp-1}} + \dfrac{2\big( \obj(\point_1) - \min \obj \big)}{\strong} \precdist^{-2} + 1 }.
\end{align}
\end{restatable}
%----------------------------------------------------------------------
%%----------------------------------------------------------------------
%\smallskip
%\begin{remark}
%Second, there is a very distinctive pattern that arises when comparing the corresponding continuous- and discrete-time results (\cref{thm:occmeas} to \cref{thm:occmeas-disc}, \cref{thm:hitting} to \cref{thm:hitting-disc}, etc.).
%In all cases, the derived discrete-time bounds consist of a term that is essentially identical to the corresponding continuous-time guarantee, plus a term involving the value difference $\obj(\init) - \min\obj$, which has no analogue in continuous time, and which essentially stems from the ``discretization'' of \eqref{eq:LMF}.
%Among others, this suggests that the proposed Lévy noise model is a faithful representation of the heavy-noise regime in continuous time;
%formalizing this pattern lies well beyond the scope of this work, but the above discussion provides a very strong rule of thumb in this regard.
%\endenv
%\end{remark}
%%----------------------------------------------------------------------
Finally, to complete our range of results, we provide below the analogous result for \eqref{eq:LMD} with a variable step-size.
%----------------------------------------------------------------------
\begin{restatable}[\acs{LMD}, variable $\step$]{theorem}{STRLMD}
\label{thm:str-LMD}
Suppose that $\obj$ is $\strong$-strongly convex and $\lips$-smooth, both relative to $\hreg$, with $\hreg$ steep.
If \eqref{eq:LMD} is run with step-size $\curr[\step] = \beta/\run$, $\beta \leq 1/(\momexp\lips)$, then
\begin{equation}
\label{eq:str-LMD}
\exof[\big]{\norm{\curr - \sol}^2}
= \begin{cases*}
\bigoh\parens{1/\run^{\momexp-1}}
&\quad
if $\momexp < 1 + \beta \strong$,
\\
\bigoh\parens{\log \run / \run^{\momexp-1}}
&\quad
if $\momexp = 1 + \beta \strong$,
\\
\bigoh\parens{1/\run^{\beta\strong}}
&\quad
if $\momexp > 1 + \beta \strong$.
\end{cases*}
\end{equation}
\end{restatable}
%----------------------------------------------------------------------
There are several points worth discussing at this point.
First, the choice of results to present in this section has not been arbitrary;
our aim was to show the broadest range of phenomena that arise.
Specifically, by the equivalence of \eqref{eq:SDA} and \eqref{eq:LMD} for constant $\step=\learn$, \cref{thm:str-SDA} also covers \eqref{eq:LMD};
and by the equivalence of \eqref{eq:LMD} and \eqref{eq:SMD} in the steep case, the bounds of \cref{thm:str-LMD} also apply directly to \eqref{eq:SMD}.
In \cref{app:discrete}, we provide a range of supplementary results that complete the picture.
%\textemdash but, otherwise, do not provide any further insights.
%To connect with the continuous-time setting, \cref{thm:cvx-SDA,thm:str-SDA} for \eqref{eq:SDA} are the discrete-time analogues of \cref{thm:cvx-LMF,thm:str-LMF} for \eqref{eq:LMF} respectively.
%In this regard,
%%\textemdash and up to modeling constants\textemdash
%we see that the convergence guarantees of \eqref{eq:LMF} and \eqref{eq:SDA} essentially coincide, showing
%%in this manner
%that the former is a genuinely faithful proxy for the latter in the heavy-noise regime.
Second, a salient pattern that arises when connecting the corresponding continuous- and discrete-time results
%(\cref{thm:occmeas} to \cref{thm:occmeas-disc}, \cref{thm:hitting} to \cref{thm:hitting-disc}, etc.)
is that the derived discrete-time bounds consist of a term that mirrors the corresponding continuous-time guarantee, plus a term involving the value difference $\obj(\init) - \min\obj$,
%which has no analogue in continuous time, and
which essentially stems from the ``discretization'' of \eqref{eq:LMF}.
This suggests that \eqref{eq:LMF} is a particularly faithful surrogate for the heavy-noise regime in continuous time\textemdash so the study of its properties proves highly beneficial for the discrete-time regime as well.
%formalizing this pattern lies well beyond the scope of this work, but the above discussion provides a very strong rule of thumb in this regard.
In terms of rates, the guarantees of \cref{thm:str-LMD} are fairly involved.
First, to provide some context, \cref{thm:str-LMD-var} in \cref{app:discrete} shows that running \eqref{eq:LMD} with $\curr[\step] \propto 1/\run^{1/\momexp}$ achieves an $\bigoh(1/\run^{(\momexp-1)/\momexp})$ convergence rate.
On the surface, this appears weaker than that of \cref{thm:str-LMD}, but there are several intricacies at play:
First, since $\lips \geq \strong$, we have $\beta \strong \leq \strong/(\momexp \lips) \leq 1/\momexp < 1$, so the rates in \cref{thm:str-LMD} are better when $\momexp \leq 1/(1-\beta \strong)$, but not otherwise.
Concretely, for $\beta = 1/(\momexp \lips)$, this condition reduces to $\momexp \leq 1 + \strong/\lips$, which generally holds only when $\momexp$ is close enough to $1$, that is, for very heavy noise;
in the other cases, the step-size $\curr[\step] \propto 1/\run^{1/\momexp}$ seems to be preferable.
Finally, we see that \eqref{eq:SDA} attains the improved complexity rate $\tildeoh\parens*{(\sdev^{\momexp}/\precval)^{1/(\momexp-1)}}$ with a suitably tuned $\learn$ (the case of a variable learning rate is more subtle, \cf \cref{thm:discrete_relative_strong_variable_learning_rate}).
As we mentioned in \cref{sec:continuous}, this does not match the $\bigoh(\precval^{-\momexp/\parens{2\momexp-2}})$ of \citet{ZKV+20} for strongly convex functions.
A key difference between our setting and \cite{ZKV+20} is that we work here with functions that are strongly convex relative to a steep regularizer over a bounded domain,
%\textemdash like entropically regularized optimal transport models\textemdash
so $\nabla\obj$ grows unbounded at the boundary of $\points$;
by contrast, the lower bounds of \citep{ZKV+20} concern problems whose gradients are bounded over \emph{any} bounded domain, so there is no overlap.
Beyond this difference, we conjecture that our rates can be tightened further with a different choice of energy function;
%but this lies beyond the scope of our work, so
we leave this as an open question for future work.
%----------------------------------------------------------------------
%% DISCUSSION
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
An important take-away of our results is that \acl{SMD} remains remarkably robust in the presence of heavy noise, and maintains its convergence and concentration guarantees despite the very long jumps exhibited by the noise.
A second is that our discrete-time results closely mirror our continuous-time findings, both qualitatively and quantitatively;
this indicates that the proposed Lévy model is itself a particularly faithful proxy for studying \ac{SMD} in the heavy-noise regime, thus opening up several important directions for further research in the heavy-noise regime.
A first such question would be to sharpen \cref{thm:str-SDA} under relative strong convexity, where we conjecture that the lower bound of \citet{ZKV+20} is achievable (even with unbounded gradients);
a second would be to investigate the use of \eqref{eq:LMF} for sampling from constrained spaces in the spirit of \citep{HKRC18,ZPFP20}.
We leave these questions for the future.
%----------------------------------------------------------------------
%% ACKS
%----------------------------------------------------------------------
% !TEX root = ./Main.tex
%
%
This research was supported in part by the French National Research Agency (ANR) in the framework of the PEPR IA FOUNDRY project (ANR-23-PEIA-0003).
PM is also a member of the Archimedes Research Unit/Athena RC, and was partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program.
%----------------------------------------------------------------------
%% APP: RELATED WORKS
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
In this first appendix, we discuss in detail some literature threads which, while relevant, would have otherwise disrupted the flow of our discussion.
\para{Stochastic mirror flows and Mirrored Langevin dynamics}
\emph{Stochastic mirror flows} were introduced and studied by \citet{RB13} as a noisy variant of the continuous-time mirror flow of \citet{NY83}, with subsequent works refining their convergence properties and extending the framework to state-dependent diffusion terms and variational inequalities \citep{MerSta18,MerSta18b}. Accelerated variants were later proposed, notably by \citet{KB17} and \citet{XWG18a,XWG18}, the latter also establishing convergence rates under relative strong convexity. All these works suppose $\lips$-smoothness of the objective function. Closely related are the \emph{mirrored Langevin dynamics}, which extend Langevin diffusions to non-Euclidean geometries for sampling measures on constrained spaces. The initial formulation, due to \citet{HKRC18}, considered a constant diffusion coefficient, while later studies introduced a diffusion term adapted to the geometry via the Hessian of the regularizer \citep{ZPFP20,CLL+20,LTVW22}, thereby placing mirrored Langevin dynamics as a special case of Riemannian Langevin methods \citep{RS02}. Convergence guarantees for convex and relatively strongly convex objectives were also established for these dynamics by \citet{TRRB23a} under a constant stepsize schedule.
%\begin{paragraph}{Heavy-tailed noise in applications and machine learning}
%
%Heavy-tailed stochastic fluctuations are ubiquitous in applications, including climate science \citep{Dit99}, biological evolution \citep{JMW12}, audio processing \citep{LB15}, and enonomics and finance \citep{Man97,IIW15}. In machine learning, the heavy-tailed behavior of gradient noise was first explored by \citet{SSG19}, though initially challenged by \citet{XSS20} who argued that the experiment choices were not accurate in practice. Subsequent empirical and theoretical work has nevertheless confirmed heavy-tailed behavior across a wide range of models and training regimes, including adaptive methods and recurrent neural networks \citep{ZKV+20}, deep convolutional architectures \citep{ZFM+20,BWL24}, spectral properties of weight matrices \citep{MM21}, large language models \citep{ACS+23}, and reinforcement learning \citep{GZP+21}. Several theoretical works further attribute these phenomena to multiplicative noise effects leading to the convergence of the iterates toward an heavy-tailed stationary distribution \citep{GSZ21,HM21b,PDS23,DM24a}.
% In climate change: \citet{Dit99}
%
% In biological evolution: \citet{JMW12}
%
% In audio signals: \citet{LB15}
%
% In economy and finance: \citet{Man97}, \citet{IIW15} and references therein
%
% In ML: \citet{SSG19} (gradient noise is heavy tailed), initially refuted by \citet{XSS20} (refuted previous works by 1. arguing that the noise is not isotropic which was assumed in \cite{SSG19}; and 2. non-anisotropic noise leads to Gaussian-type gradient noise when the noise is high enough), but subsequent works have also established heavy-tailed behaviors of gradient noise in a variety of ML architecture and setups: \citet{ZKV+20} (gradient noise is heavy tailed for some adaptive models, in recurrent neural networks), \citet{ZFM+20} (ADAM and SGD on AlexNet is better characterized by stable noise), \citet{MM21} (eigenvalues of the weights matrices in different layers exhibit a heavy-tailed behavior), \citet{BWL24} (gradient noise is heavy-tailed even in anisotropic models)
%
% In large language models: \citet{ACS+23}
%
% In reinforcement learning: \citet{GZP+21}
%
% Origin of heavy-tails in ML (iterates converge to heavy-tailed stationary distribution): \citet{GSZ21}, \citet{HM21b} (multiplicative noise leads to iterates converging toward heavy-tailed invariant measure), \citet{PDS23}, \citet{DM24a}
%\end{paragraph}
\para{Lévy-driven SDEs in stochastic optimization}
Stochastic differential equations driven by Lévy processes have gained attention in machine learning as models for stochastic optimization under heavy-tailed noise.
A first line of work focuses on sampling algorithms based on Lévy-driven dynamics, most notably the \emph{fractional Langevin Monte Carlo} method introduced by \citet{Sim17}, and further studied by \citep{YZ18a,NSR19,SZTG20,HMW21,ZZ23}, where Brownian motion is replaced by a stable Lévy process.
More general \emph{Lévy Langevin Monte Carlo} methods allowing arbitrary Lévy noise were also recently studied in \citet{Oec23} and \citet{BS25b}.
Beyond sampling, Lévy-driven SDEs have been proposed as continuous-time approximations of stochastic gradient methods, notably to explain metastability and escape phenomena observed in deep learning \citep{SSG19,NSGR19,SGN+19,ZFM+20,BWL24}.
Finally, Lévy-driven models have also been used to study generalization properties of SGD, suggesting that heavier-tailed processes should achieve better generalization errors \citep{SSDE20,RBG+23,RZGS23,DS24,DBS+25}.
% Sampling via Lévy-driven SDEs, the Fractional Langevin Monte Carlo ($\alpha$-stable noise): \citet{Sim17}, \citet{YZ18a} \citet{NSR19}, \citet{SZTG20}, \citet{HMW21}, \citet{ZZ23}
%
% Lévy Langevin Monte Carlo (general Lévy noise): \citet{Oec23}, \citet{BS25b}
%
% Metastability analysis of SGD: \citet{SSG19} \citet{NSGR19} \citet{SGN+19} (extended version of first two articles), \citet{ZFM+20}, \citet{BWL24}
%
% Generalization properties of SGD through Lévy-driven SDEs: \citet{SSDE20}, \citet{RBG+23}, \citet{RZGS23}, \citet{DS24}, \citet{DBS+25}
\para{Heavy-tailed noise in stochastic algorithms}
For \emph{standard SGD}, \citet{ZKV+20} showed that iterates may fail to converge for any fixed stepsize under heavy-tailed noise, even for strongly convex objectives, indicating that additional structure is generally required.
Convergence can nonetheless be recovered under strong assumptions on the objective, such as a bounded and uniformly $\momexp$-positive definite Hessian \citep{WGZ+21}, or by modifying the algorithm.
In the context of \emph{mirror descent}, \citet{NY83} established convergence for convex objectives with uniformly continuous mirror maps and proved the optimal in-expectation lower bound $\Omega\parens*{\time^{-(\momexp-1)/\momexp}}$.
This result was later refined by \citet{VYB+22} for uniformly convex regularizers, with improved dimension dependence, and extended to zeroth-order methods in \citet{KGDD23}.
For \emph{projected SGD on bounded domains}, recent work by \citet{FHL25} and \citet{Liu25a} showed that the optimal convex rates can in fact be achieved without modifying the algorithm; the latter also establishing this property for Dual Averaging and AdaGrad in online convex optimization.
A large body of work instead focuses on \emph{gradient clipping} as a stabilization mechanism under heavy-tailed noise.
This line of research was initiated by \citet{ZKV+20}, who proved optimal lower bounds in expectation of order $\Omega\parens*{\time^{-2(\momexp-1)/\momexp}}$ for strongly convex objectives and $\Omega\parens*{\time^{-(\momexp-1)/(3\momexp-2)}}$ for nonconvex smooth ones, and showed that clipped SGD can match these bounds.
Subsequent works derived high-probability guarantees matching these rates \citep{SDG+23} and extended the analysis to broader settings, including nonconvex \citep{CM21,LZZ23,AYS+24}, composite \citep{GSD+24} and distributed optimization \citep{GSD+24,YJK25}, as well as to more general $(L_0, L_1)$-smooth objectives \citep{CBHG25}.
Further refinements include variance-reduced and stabilized variants that can achieve even faster rates under additional structure \citep{PGKG24,Liu25b}.
The recent work of \citet{WR25} complements this line of work by providing a large-deviations and metastability analysis of clipped SGD without relying on a Lévy-driven SDE approximation.
Alternatives to clipping have also been proposed to address heavy-tailed noise without truncation.
Notably, normalized gradient methods achieve optimal or near-optimal convergence rates for nonconvex optimization \citep{LZ24b,HFH24,SLY25,YJK25}.
\para{Further comparison with \citep{Liu25a}}
For discrete-time algorithms, the closest work to ours is \citep{Liu25a}, which derives several convergence guarantees for Online Gradient Descent and Dual Averaging on bounded domains under heavy-tailed noise.
Importantly, while the same general strategy is used to handle unbounded variance (\cf the chain of inequalities leading to \citep[Eq.~5]{Liu25a}), that work is restricted to standard Euclidean norms, whereas our results hold under general Bregman divergences.
This extension to non-Euclidean norms was in fact already alluded to by \citet[Remark~4]{Liu25a}, but carrying it out requires substantially more care than what one might initially expect, notably the need to distinguish between \emph{standard} and \emph{relative} strong convexity---the former being norm-agnostic while the latter accommodates the geometry induced by the regularizer.
In particular, when the regularizer is steep, relative strong convexity requires the objective function to have unbounded gradients, making it necessary to work under the refined \emph{relative smoothness} framework of \citet{BBT17}.
By contrast, gradients are assumed bounded throughout \citep{Liu25a}, except for the last-iterate convergence rate of \citep[Theorem~10]{Liu25a} and its corollaries, which rely instead on the condition
\begin{equation}
\frac{\strong}{2} \norm{\point - \pointalt}^{2}
\leq \obj(\point) - \obj(\pointalt)
- \braket{\nabla \obj(\pointalt)}{\point - \pointalt}
\leq 2 G \norm{\point - \pointalt}
+ \frac{H}{2} \norm{\point - \pointalt}^{2}
\eqcomma
% \quad
% \point, \pointalt \in \points,
\end{equation}
for some $\strong, G, H \geq 0$ such that $G + H > 0$, and for all $\point,\pointalt\in\points$.
One may in fact observe that, in the Euclidean and compact setting of \citet{Liu25a}, the above condition effectively reduces to a bounded-gradient assumption, despite being formulated as a nonsmoothness condition.
However, unlike the other results of \citep{Liu25a}, its extension to the relatively smooth / relatively strongly convex setting now becomes rather natural, as it effectively amounts to replacing Euclidean norms by Bregman divergences.
With that said, this condition was only introduced in a revised version of \citep{Liu25a}, posted online concurrently with the initial drafting of the present paper, and whose existence was brought to our attention only later by a reviewer.
%\begin{paragraph}{Heavy-tailed noise in stochastic algorithms}
% For standard SGD, \citet{ZKV+20} has shown that iterates fails to converge for any fixed stepsize, hence indicating that some additional structure in the algorithms are needed to obtain convergence.
%
% In SGD: \citet{WGZ+21} (requires the Hessian of $\obj$ to be bounded and to be uniformly $\momexp$-positive definite),
%\end{paragraph}
%
% In mirror descent: \citet{NY83} (for convex functions and uniformly continuous mirror maps; also establish optimal in-expectation lower bound $\Omega\parens*{\time^{-\momexp/(\momexp-1)}}$, \citet{VYB+22} (for convex functions and uniformly convex regularizers; also improves the dependence on the dimension of the lower bound of \citet{NY83}), \citet{KGDD23} (extension of the results of \citet{VYB+22} to zeroth order convex stochastic optimization).
%
% In projected SGD: \citet{FHL25} (projected SGD on convex bounded domain converges without requiring any modification on the algorithm. Moreover, it reaches the optimal convex and strongly convex lower bounds), \citet{Liu25a} (various standard online convex optimization algorithms, such as online gradient descent, dual averaging and AdaGrad admits optimal regret bound---and thus also optimal convergence rate---for convex losses under heavy-tailed noise, as long as the domain is bounded)
%
% In clipped SGD: \citet{ZKV+20} (for strongly convex and nonconvex problems, also establish optimal lower bounds in expectation: $\Omega\parens*{\time^{-2(\momexp-1)\momexp}}$ for strongly convex functions, which is optimal, and a lower bound $\Omega\parens*{\time^{-(\momexp-1)/(3\momexp-2)}}$ for nonconvex smooth functions); \citet{CM21} (nearly optimal high-probability bound for nonconvex smooth optimization, \citet{SDG+23} (obtain high-probability lower bounds matching the in-expectation ones of \citet{ZKV+20} for strongly convex functions and establish an array of high probability convergences rates under a variety of different assumptions on the objective); \citet{LZZ23} (nearly optimal high-probability for nonconvex smooth optimization but without a restrictive assumption used in \cite{CM21}, and also shows that one can achieve even faster rates than those dictated by lower bound when the objective admits more structure, using a variance-reduced accelerated algorithm), \citet{PGKG24} (consider strongly convex stochastic problems with structure heavy-tailed noise that may not even have a finite expectation, and shows that faster convergence rates than the optimal rate can then be obtained under a refined version of clipped SGD using a stabilization of the stochastic gradients); \citet{GSD+24} (introduce new methods for composite and distributed optimization based on gradient clipping and prove tight high-probability convergences results), \citet{AYS+24}, \citet{CBHG25} (convergence for convex $(L_0, L_1)$-smooth optimization); \citet{WR25} (large deviations and metastability analysis of clipped SGD without using an SDE approximation); \citet{Liu25b} (refined analysis of clipped SGD to obtain better convergence rates)
%
% Without clipping: \citet{LZ24b} (show that the Batched Normalized Stochastic Gradient Descent with Momentum algorithm also achieves the optimal convergence rate without clipping, for nonconvex stochastic optimization. Also establish a convergence rate when the tail index $\momexp$ is unknown in advance); \citet{HFH24} (also investigate the use of Normalized SGD to avoid the issues of clipping and establish high-probability convergence rates for nonconvex smooth problems); \citet{SLY25} (again normalized SGD, how it can be combined with clipping for further improves rates, and introduce an accelerated variant under second-order smoothness); \citet{YJK25} (normalized SGD for decentralized first-order nonconvex optimization); \citet{FSCR25}
%\end{paragraph}
%----------------------------------------------------------------------
%% APP: MIRROR
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
%----------------------------------------------------------------------
%%% APP: MIRROR
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
In this appendix, we collect some background material, properties and examples regarding the mirror setup of \cref{sec:prelims}.
The results presented below (or a version thereof) are known in the literature;
nevertheless, we provide detailed proofs for completeness and to resolve any conflicts or ambiguities with different conventions in the literature.
%\hrule
%
%List of results that are needed:
%\begin{enumerate}
% \item[\checkmark] $\point = \mirror(\dpoint)$ if and only if $\dpoint \in \subd \hreg(\point)$; in particular, $\im \mirror = \dom \subd \hreg$.
% \item[\checkmark] $\nabla \hconj = \mirror$; in particular, $\mirror$ is $(1/\hcst)$-Lipschitz continuous.
% \item[\checkmark] $\breg(\basealt, \mirror(\dpoint)) \leq \fench(\basealt, \dpoint)$ for every $\basealt \in \points$ and $\dpoint \in \dpoints$, with equality whenever $\mirror(\dpoint) \in \intpoints$.
% \item[\checkmark] $\fench(\basealt, \dpoint) \geq \frac{\hcst}{2} \norm{\mirror(\dpoint) - \basealt}^2$ for every $\basealt \in \points$ and $\dpoint \in \dpoints$.
%\end{enumerate}
%
%\hrule
%----------------------------------------------------------------------
%% Prelims
%----------------------------------------------------------------------
\subsection{Refresher}
With notation as in \cref{sec:prelims}, let $\vecspace$ be a $\vdim$-dimensional normed space,
%with norm $\norm{\cdot}$,
let $\dpoints \defeq \dspace$ denote the (algebraic) dual of $\vecspace$,
and
let $\braket{\dpoint}{\point}$ denote the canonical bilinear pairing between $\point\in\vecspace$ and $\dpoint\in\dspace$.
If $\norm{\cdot}$ is a norm on $\vecspace$ we will also write
\begin{equation}
\label{eq:dnorm}
\dnorm{\dpoint} = \max\setdef{\braket{\dpoint}{\point}}{\norm{\point} \leq 1}
\end{equation}
for the induced dual norm on $\dpoints$, so $\abs{\braket{\dpoint}{\point}} \leq \norm{\point} \dnorm{\dpoint}$ for all $\point\in\vecspace$ and all $\dpoint\in\dpoints$ by construction.
Moreover, given a closed convex subset $\points$ of $\vecspace$, we also define:
\begin{enumerate}
\item
The \emph{tangent cone} to $\points$ at $\base\in\points$:
\begin{flalign}
\label{eq:tcone}
\tcone(\base)
&= \cl\setdef{\tanvec\in\vecspace}{\base+t\tanvec\in\points \; \text{for some $t>0$}}
\intertext{%
\ie as the closure of the set of rays emanating from $\base$ and meeting $\points$ in at least one other point.
\item
The \emph{dual cone} to $\points$ at $\base\in\points$:}
\label{eq:dcone}
\dcone(\base)
&= \setdef{\dvec\in\dpoints}{\braket{\dvec}{\tanvec} \geq 0 \; \text{for all $\tanvec\in\tcone(\base)$}}
\intertext{%
\item
The \emph{normal\,/\,polar cone} to $\points$ at $\base\in\points$:}
\label{eq:pcone}
\pcone(\base)
&= \setdef{\dvec\in\dpoints}{\braket{\dvec}{\tanvec} \leq 0 \; \text{for all $\tanvec\in\tcone(\base)$}}
\end{flalign}
\end{enumerate}
Following standard conventions in the field \cite{Roc70}, convex functions will be allowed to take values in the extended real line $\R\cup\{\infty\}$, and we will denote the \define{effective domain} of a convex function $\obj\from\vecspace\to\R\cup\{\infty\}$ as
\begin{equation}
\label{eq:domfun}
\dom\obj
\defeq \setdef{\point\in\vecspace}{\obj(\point) < \infty}
\eqstop
\end{equation}
When there is no danger of confusion, we will identify a convex function $\obj\from\vecspace\to\R$ with its restriction on $\dom\obj$;
in other words, we will treat $\obj$ interchangeably
as a function on $\dom\obj$ with values in $\R$,
or
as a function on $\vecspace$ with values in $\R\cup\{\infty\}$ (and finite on $\dom\obj$).
Throughout the sequel, we will assume that all functions under study are \define{proper}, that is, $\dom\obj\neq\varnothing$.
Then, given a proper function $\obj\from\vecspace\to\R\cup\{\infty\}$, the \define{subdifferential} of $\obj$ at $\point\in\dom\obj$ is defined as
\begin{equation}
\label{eq:subdiff}
\subd\obj(\point)
\defeq \setdef{\dpoint\in\dpoints}{\obj(\pointalt) \geq \obj(\point) + \braket{\dpoint}{\pointalt - \point} \; \text{for all $\pointalt\in\vecspace$}}
\end{equation}
and we denote the \define{domain of subdifferentiability} of $\obj$ as
\begin{equation}
\label{eq:domdiff}
\dom\subd\obj
= \setdef{\point\in\vecspace}{\subd\obj(\point) \neq \varnothing}
\eqstop
\end{equation}
%\ie the function $\obj\vert_{\points}\from\points\to\R$ with $\obj\vert_{\points}(\point) = \obj(\point)$ for all $\point\in\points$.
With all this in hand, a \define{regularizer} on a closed convex subset $\points$ of $\vecspace$ is a continuous function $\hreg\from\points\to\R$ which is \define{strongly convex} relative to $\norm{\argdot}$, \ie there exists some $\hstr>0$ such that
\begin{equation}
\label{eq:hstr}
\hreg(\coef\point + (1-\coef)\pointalt)
\leq \coef \hreg(\point)
+ (1-\coef) \hreg(\pointalt)
- \frac{\hstr}{2} \coef(1-\coef) \norm{\pointalt - \point}^{2}
\end{equation}
for all $\point,\pointalt\in\points$ and for all $\coef\in[0,1]$.
By standard arguments \cite{RW98,Ber15}, this immediately implies that
\begin{equation}
\label{eq:hstr-diff}
\hreg(\pointalt)
\geq \hreg(\point)
+ \hreg'(\point;\pointalt - \point)
+ \frac{\hstr}{2} \norm{\pointalt - \point}^{2}
\quad
\text{for all $\point,\pointalt\in\points$},
\end{equation}
where
\begin{equation}
\hreg'(\point;\pointalt-\point)
= \lim_{\theta\to0^{+}} \bracks{\hreg(\point + \theta(\pointalt-\point)) - \hreg(\point)} / \theta
\end{equation}
denotes the one-sided directional derivative of $\hreg$ at $\point$ along the direction of $\pointalt-\point$.
As per \cref{def:mirror}, we will say that $\hreg$ is a \define{Bregman regularizer} if $\subd\hreg$ admits a continuous selection $\subsel\hreg(\point) \in \subd\hreg(\point)$ for all $\point\in\dom\subd\hreg$.
For such a regularizer, we recall the following definitions:
\begin{subequations}
\begin{enumerate}
\item
The \define{prox-domain} of $\hreg$:
\begin{equation}
\label{eq:proxdom}
\proxdom
\defeq \dom\subd\hreg
\end{equation}
\item
The \define{convex conjugate} $\hconj\from\dpoints\to\R$ of $\hreg$:%
\begin{equation}
\label{eq:hconj}
\hconj(\dpoint)
\defeq \max_{\point\in\points} \{ \braket{\dpoint}{\point} - \hreg(\point) \}
\quad
\text{for all $\dpoint\in\dpoints$}.
\end{equation}
\item
The \define{mirror map} $\mirror\from\dpoints\to\points$ induced by $\hreg$:%
\begin{equation}
\label{eq:mirror}
\mirror(\dpoint)
\defeq \argmax_{\point\in\points} \{ \braket{\dpoint}{\point} - \hreg(\point) \}
\quad
\text{for all $\dpoint\in\dpoints$}.
\end{equation}
\item
The \define{Fenchel coupling} $\fench\from\points\times\dpoints\to\R$ defined by $\hreg$:%
\begin{equation}
\label{eq:Fench}
\fench(\base,\dpoint)
= \hreg(\base)
+ \hconj(\dpoint)
- \braket{\dpoint}{\base}
\quad
\text{for all $\base\in\points$, $\dpoint\in\dpoints$}.
\end{equation}
\item
The \define{Bregman divergence} $\breg\from\points\times\proxdom\to\R$ associated to $\hreg$:%
\begin{equation}
\label{eq:Breg}
\breg(\base,\point)
= \hreg(\base)
- \hreg(\point)
- \braket{\subsel\hreg(\point)}{\base - \point}
\quad
\text{for all $\base\in\points$, $\point\in\proxdom$}.
\end{equation}
\item
The \define{prox-mapping} $\mprox\from\proxdom\times\dpoints\to\proxdom$ induced by $\hreg$:%
\begin{equation}
\label{eq:mprox}
\proxof{\point}{\dvec}
= \argmin\nolimits_{\pointaux\in\points}
\braces{\braket{\dvec}{\point - \auxpoint} + \breg(\auxpoint,\point)}
\quad
\text{for all $\point\in\proxdom$, $\dvec\in\dpoints$}.
\end{equation}
\end{enumerate}
\end{subequations}
\begin{remark}
Of the items above, only the last two require $\hreg$ to be a Bregman regularizer;
the rest apply to any regularizer.
\endenv
\end{remark}
The proposition below provides some basic properties linking the above:
\begin{proposition}
\label{prop:mirror}
Let $\hreg$ be a $\hstr$-strongly convex regularizer on $\points$.
Then:
\begin{enumerate}
[\upshape(\itshape a\hspace*{1pt}\upshape)]
\item
$\mirror$ is single-valued on $\dpoints$.
\item
For all $\point\in\proxdom$ and all $\dpoint\in\dpoints$, we have
\begin{equation}
\label{eq:hinv}
\point
= \mirror(\dpoint)
\quad
\text{if and only if}
\quad
\dpoint
\in \subd\hreg(\point)
\eqstop
\end{equation}
\item
The image $\im\mirror$ of $\mirror$ is equal to the prox-domain of $\hreg$, and we have
\begin{equation}
\label{eq:imQ}
\relint\points
\subseteq \im\mirror
= \proxdom
\subseteq \points
\eqstop
\end{equation}
\item
%Danskin's theorem:
The convex conjugate $\hconj\from\dpoints\to\R$ of $\hreg$ is differentiable and satisfies
\begin{equation}
\label{eq:Danskin}
\mirror(\dpoint)
= \nabla\hconj(\dpoint)
\quad
\text{for all $\dpoint\in\dpoints$}.
\end{equation}
\item
$\mirror$ is $(1/\hstr)$-Lipschitz continuous, that is,
\begin{equation}
\label{eq:QLips}
\norm{\mirror(\dpointalt) - \mirror(\dpoint)}
\leq (1/\hstr) \dnorm{\dpointalt - \dpoint}
\quad
\text{for all $\dpoint,\dpointalt\in\dpoints$}.
\end{equation}
\item
Fix some $\dpoint\in\dpoints$ and let $\point = \mirror(\dpoint)$.
Then, for all $\pointalt\in\points$ we have:
%For all $\base\in\points$ and all $\dpoint\in\dpoints$, $\point=\mirror(\dpoint)$, we have
\begin{equation}
\label{eq:hdir}
\hreg'(\point;\pointalt - \point)
\geq \braket{\dpoint}{\pointalt - \point}
\eqstop
\end{equation}
%In particular, if $\subd\hreg$ admits a continuous selection $\subsel\hreg\from\proxdom\to\dpoints$, we have
%\begin{equation}
%\label{eq:subsel-var}
%\braket{\subsel\hreg(\point)}{\pointalt - \point}
% \geq \braket{\dpoint}{\pointalt - \point}
% \quad
% \text{for all $\point\in\proxdom$ and all $\pointalt\in\points$},
%\end{equation}
%or, equivalently,
%\begin{equation}
%\label{eq:subsel-cone}
%\subd\hreg(\point)
% = \subsel\hreg(\point) + \pcone(\point)
% \quad
% \text{for all $\point\in\proxdom$}.
%\end{equation}
\item
Fix some $\dpoint\in\dpoints$, and let $\point = \mirror(\dpoint)$.
Then $\mirror(\dpoint+\dvec) = \point$ for all $\dvec\in\pcone(\point)$.
\end{enumerate}
\end{proposition}
\begin{proof}
For the most part, these properties are well known in the literature (except possibly the last one), so we only provide a pointer or a short sketch for most of them.
\begin{enumerate}
[\upshape(\itshape a\hspace*{.5pt}\upshape)]
\item
This follows from the fact that $\hreg$ is strongly convex, so the $\argmax$ in \eqref{eq:mirror} is attained and is unique for all $\dpoint\in\dpoints$.
\item
By Fermat's rule \citep[Chap.~26]{Roc70}, we readily see that $\point$ solves \eqref{eq:mirror} if and only if $\dpoint - \subd\hreg(\point) \ni 0$, that is, if and only if $\dpoint\in\subd\hreg(\point)$.
%Since this implies that $\subd\hreg$, our claim follows.
\item
By \eqref{eq:hinv}, we readily get $\im\mirror = \proxdom$.
As for the second part of our claim, it follows from basic properties of the subdifferential, \cf \citet[Chap.~26]{Roc70}.
\item
This is simply Danskin's theorem, see \eg \citet[Proposition 5.4.8, Appendix B]{Ber15}.
\item
This is a consequence of the fact that $\hconj$ is $(1/\hstr)$-Lipschitz smooth, \cf \citet[Theorem 12.60(b)]{RW98}.
\item
Since $\dpoint \in \subd\hreg(\point)$ by \eqref{eq:hinv}, we readily get that
\begin{equation}
\hreg(\point + \theta(\pointalt-\point))
\geq \hreg(\point) + \theta \braket{\dpoint}{\pointalt-\point}
\quad
\text{for all $\theta\in[0,1]$}
\eqstop
\end{equation}
Hence, by rearranging and taking the limit $\theta\to0^{+}$,
we conclude that
\begin{equation}
\label{eq:dir-subdiff}
\hreg'(\point;\pointalt-\point)
= \lim_{\theta\to0^{+}} \frac{\hreg(\point + \theta(\pointalt-\point)) - \hreg(\point)}{\theta}
\geq \braket{\dpoint}{\pointalt-\point}
\end{equation}
as claimed.%
\footnote{The existence of the limit is guaranteed by elementary convex analysis arguments, \cf \citet[App.~B]{Ber15}.}
%Finally, for our last assertion, let $\tanvec = \pointalt - \point$ and set
%\begin{equation}
%\label{eq:hreg-ray}
%\phi(\theta)
% = \hreg(\point + \theta\tanvec)
% - \bracks{\hreg(\point) + \braket{\dpoint}{\theta\tanvec}}
% \quad
% \text{for all $\theta\in[0,1]$}
%\end{equation}
%%By combining \cref{eq:hstr-diff,eq:dir-subdiff}, we readily get
%%\begin{equation}
%%\hreg(\point + \theta\tanvec)
%% \geq \hreg(\point)
%% + \braket{\dpoint}{\theta\tanvec}
%% + \frac{\hstr}{2} \norm{\theta\tanvec}^{2}
%%\end{equation}
%so $\phi(\theta) \geq \hstr \theta^{2}\norm{\tanvec}^{2}/2 \geq 0$ for all $\theta\in[0,1]$.
%By construction, it is straightforward to verify that the function $\psi(\theta) = \braket{\subsel\hreg(\point+\theta\tanvec) - \dpoint}{\tanvec}$ is a selection of subgradients of $\phi$, \ie
%\begin{equation}
%\phi(\alt\theta)
% \geq \phi(\theta) + \psi(\theta) (\alt\theta - \theta)
% \quad
% \text{for all $\theta,\alt\theta\in[0,1]$}.
%% \eqstop
%\end{equation}
%Since $\psi$ is in addition continuous (because $\subsel\hreg$ is), it follows that $\phi'(\theta) = \psi(\theta)$ for all $\theta\in[0,1]$ by a well-known characterization of the one-sided derivatives of convex functions, \cf \citet[Theorem~24.2]{Roc70}.
%Hence, with $\phi$ convex and $\phi(\theta) \geq \phi(0)$ for all $\theta\in[0,1]$, we conclude that $\braket{\subsel\hreg(\point) - \dpoint}{\tanvec} = \psi(0) = \phi'(0) \geq 0$, and our proof is complete.
\item
By \eqref{eq:hinv} it suffices to show that $\dpoint+\dvec\in\subd\hreg(\point)$ for all $\dvec\in\pcone(\point)$.
However, if $\dvec\in\pcone(\point)$, we also have $\braket{\dvec}{\pointalt - \point} \leq 0$ for all $\pointalt\in\points$, and hence, with $\dpoint\in\subd\hreg(\point)$, we readily get
\begin{flalign}
\hreg(\pointalt)
&\geq \hreg(\point)
+ \braket{\dpoint}{\pointalt - \point}
\notag\\
&\geq \hreg(\point)
+ \braket{\dpoint + \dvec}{\pointalt - \point}
\quad
\text{for all $\pointalt\in\points$}.
\end{flalign}
This shows that $\dpoint+\dvec \in \subd\hreg(\point)$ and completes our proof.
\qedhere
\end{enumerate}
\end{proof}
Following \cite{MS16,MZ19}, we also define the \define{Fenchel coupling} associated to $\hreg$ as
\begin{equation}
\label{eq:Fench}
\fench(\base,\dpoint)
= \hreg(\base)
+ \hconj(\dpoint)
- \braket{\dpoint}{\base}
% = \max_{\point\in\points} \setof{\braket{\dpoint}{\point} - \hreg(\point)}
% - \bracks{\braket{\dpoint}{\base} - \hreg(\base)}
\quad
\text{for all $\base\in\points$, $\dpoint\in\dpoints$}.
\end{equation}
The next proposition shows that the Fenchel coupling can be seen as a ``primal-dual'' measure of divergence between $\base\in\points$ and $\dpoint\in\dpoints$:
\begin{proposition}
\label{prop:Fench}
Let $\hreg$ be a Bregman regularizer on $\points$.
Then, for all $\base\in\points$ and all $\dpoint\in\dpoints$, $\point=\mirror(\dpoint)$, we have:
\begin{subequations}
%\setlength{\abovedisplayskip}{3pt}
%\setlength{\belowdisplayskip}{3pt}
\begin{flalign}
\label{eq:Fench-posdef}
\quad
(a)
&\;\;
\fench(\base,\dpoint)
\geq 0
\;\;
\text{with equality if and only if $\base = \point$}.
&
\\
\label{eq:Fench-Breg}
\quad
(b)
&\;\;
\fench(\base,\dpoint)
\geq \breg(\base,\point)
\;\;
\text{with equality whenever $\point\in\relint\points$}.
&
\\
\label{eq:Fench-norm}
\quad
(c)
&\;\;
\fench(\base,\dpoint)
\geq \tfrac{1}{2} \hstr \, \norm{\mirror(\dpoint) - \base}^{2}.
&
% \\
%\label{eq:Fench-bound}
%\quad
%(c)
% &\quad
% \fench(\base,\dpointalt)
% \leq \fench(\base,\dpoint) + \braket{\dpointalt - \dpoint}{\mirror(\dpoint) - \base} + \tfrac{1}{2\hstr} \dnorm{\dpointalt-\dpoint}^{2}.
\end{flalign}
\end{subequations}
%In particular, if $\hreg(0) = 0$, we have
%\begin{equation}
%(\hstr/2) \norm{\mirror(\dpoint)}^{2}
% \leq \hconj(\dpoint)
% \leq -\min\hreg
% + \braket{\dpoint}{\mirror(\dpoint)}
% + (2/\hstr) \dnorm{\dpoint}^{2}
% \quad
% \text{for all $\dpoint\in\dpoints$}
%\end{equation}
\end{proposition}
\begin{proof}
These properties are all fairly well known in the literature;
we only provide a quick proof for completeness.
\begin{enumerate}
[\upshape(\itshape a\hspace*{.5pt}\upshape)]
\item
By the Fenchel\textendash Young inequality, we have $\hreg(\base) + \hconj(\dpoint) \geq \braket{\dpoint}{\base}$ for all $\base\in\points$, $\dpoint\in\dpoints$, with equality if and only if $\dpoint\in\subd\hreg(\base)$.
Our claim then follows from \eqref{eq:hinv}.
\item
By definition, we have
\begin{align}
\fench(\base,\dpoint)
&= \hreg(\base)
+ \hconj(\dpoint)
- \braket{\dpoint}{\base}
\notag\\
&= \hreg(\base)
+ \braket{\dpoint}{\point}
- \hreg(\point)
- \braket{\dpoint}{\base}
= \hreg(\base)
- \hreg(\point)
- \braket{\dpoint}{\base - \point}
\eqstop
\end{align}
Since $\point\in\relint\points$ and $\dpoint\in\subd\hreg(\point)$, we readily get $\braket{\dpoint}{\base - \point} = \hreg'(\point;\base-\point) = \braket{\subsel\hreg(\point)}{\base - \point}$, showing that $\fench(\base,\dpoint) = \breg(\base,\point)$ in this case.
Otherwise, by \eqref{eq:hdir}, we get $\braket{\dpoint}{\base - \point} \leq \hreg'(\point;\base-\point) = \braket{\subsel\hreg(\point)}{\base - \point}$, and the desired inequality follows.
\item
Arguing as above, we have
\begin{align}
\fench(\base,\dpoint)
&= \hreg(\base) + \hconj(\dpoint) - \braket{\dpoint}{\base}
\notag\\
&= \hreg(\base) + \braket{\dpoint}{\point} - \hreg(\point) - \braket{\dpoint}{\base}
\explain{since $\dpoint \in \subd\hreg(\point)$}
\\
&\geq \hreg(\base) - \hreg(\point) - \hreg'(\point;\base-\point)
\explain{by \cref{prop:mirror}}
\\
&\geq \tfrac{1}{2} \hstr \norm{\point - \base}^{2}
\explain{by \eqref{eq:hstr}}
\end{align}
so our proof is complete.
\qedhere
\end{enumerate}
\end{proof}
%In view of \cref{prop:Fench}, $\fench(\base,\dpoint)$ can be seen a ``primal-dual'' measure of divergence between $\base\in\points$ and $\dpoint\in\dpoints$.
%This observation will play a major role in the sequel.
Our last result at this point is a useful differentiation formula for the Fenchel coupling:
\begin{lemma}
\label{lem:dFench}
For all $\base\in\points$ and all $\dpoint\in\dpoints$, we have:
\begin{equation}
\label{eq:dFench}
\nabla_{\dpoint} \fench(\base,\dpoint)
= \mirror(\dpoint) - \base
\eqstop
\end{equation}
\end{lemma}
\begin{proof}
The proof follows immediately from Danskin's theorem, \cf \cref{eq:Danskin} of \cref{prop:mirror}.
\end{proof}
%----------------------------------------------------------------------
%% Basic lemmas
%----------------------------------------------------------------------
\subsection{Update lemmas}
Moving forward, we note that the basic update step of \eqref{eq:LMD} can be written as
\begin{equation}
\label{eq:update}
\new[\dpoint]
= \dpoint + \dvec
\quad
\text{and}
\quad
\new
= \mirror(\new[\dpoint])
\end{equation}
for some $\dpoint,\dvec\in\dpoints$.
We begin with the equivalence of the primal and dual updates\textemdash of \eqref{eq:SMD} and \eqref{eq:LMD} respectively\textemdash when $\hreg$ is steep, that is, $\proxdom=\relint\points$.
The key element of this equivalence is the general proposition below.
\begin{proposition}
\label{prop:eager-lazy}
Fix some $\dpoint\in\dpoints$, $\point=\mirror(\dpoint)$, and $\dvec\in\dpoints$.
Suppose further that $\dpoint - \nabla\hreg(\point)$ annihilates all tangent vectors to $\points$ at $\point$, \ie
\begin{equation}
\braket{\dpoint - \nabla\hreg(\point)}{\pointalt - \point}
= 0
\quad
\text{for all $\pointalt\in\points$}.
\end{equation}
Then:
\begin{equation}
\label{eq:eager-lazy}
\proxof{\point}{\dvec}
= \mirror(\dpoint + \dvec)
\eqstop
\end{equation}
%%Let $\hreg$ be a \acl{DGF} on $\points$ with associated prox-mapping $\mprox$ and mirror map $\mirror$,
%%and
%Consider the eager and lazy update rules
%\begin{subequations}
%\begin{align}
%\label{eq:mirror-eager}
%\eager\point
% &= \prox{\point}{\dvec}
% \\
%%\shortintertext{and}
%\label{eq:mirror-lazy}
%\lazy\point
% &= \mirror(\dpoint+\dvec)
%\end{align}
%\end{subequations}
%where the impulse vector $\dvec\in\dspace$ is arbitrary and, for consistency, the initial points $\point\in\points$ and $\dpoint\in\dpoints$ satisfy $\point = \mirror(\dpoint)$.
%Then, $\eager\point = \lazy\point$.
\end{proposition}
\begin{corollary}
\label{cor:eager-lazy}
\Cref{prop:eager-lazy} holds whenever $\point = \mirror(\dpoint) \in\relint\points$;
in particular, if $\hreg$ is steep, \eqref{eq:eager-lazy} holds for all $\dpoint\in\dpoints$.
%If $\proxdom \equiv \dom\subd\hreg = \relint(\points)$, then $\eager\point = \lazy\point$ for all $\point\in\proxdom$.
\end{corollary}
\begin{corollary}
\label{cor:SMD-LMD}
Suppose that \eqref{eq:SMD} and \eqref{eq:LMD} are initialized respectively at $\init\in\proxdom$ and $\init[\dpoint]\in\dpoints$, with $\init = \mirror(\init[\dpoint])$.
If $\hreg$ is steep and the algorithms are run with the same input sequence $\curr[\sgrad]$, $\run=\running$, then they generate the same sequence of iterates.
\end{corollary}
\begin{proof}[Proof of \cref{prop:eager-lazy}]
By the definitions \eqref{eq:mprox} and \eqref{eq:mirror} of $\mprox$ and $\mirror$ respectively, we have:
\begin{align}
\proxof{\point}{\dvec}
&= \argmin_{\pointalt\in\points}
\{\braket{\dvec}{\point - \pointalt} + \breg(\pointalt,\point)\}
\notag\\
&= \argmin_{\pointalt\in\points}
\{\braket{\dvec}{\point - \pointalt} + \hreg(\pointalt) - \hreg(\point) - \braket{\nabla\hreg(\point)}{\pointalt - \point}\}
\notag\\
&= \argmin_{\pointalt\in\points}
\{\braket{\dvec}{\point - \pointalt} + \hreg(\pointalt) - \hreg(\point) - \braket{\dpoint}{\pointalt - \point}\}
\notag\\
&= \argmax_{\pointalt\in\points}
\{\braket{\dpoint+\dvec}{\pointalt} - \hreg(\pointalt)\}
= \mirror(\dpoint+\dvec)
\eqstop
\tag*{\qedhere}
\end{align}
\end{proof}
We now proceed to state a series of identities and estimates for the Fenchel coupling before and after an update of the form \eqref{eq:update}.
These results are not new, \cf \cite{JNT11,MZ19,MLZF+19} and references therein;
however, the assumptions used to derive them vary in the literature, so we provide detailed proofs for completeness.
%All of the results that follow below are stated for a $\hstr$-strongly convex regularizer on $\points$.
The first is a primal-dual version of the so-called ``three-point identity'' for Bregman functions \citep{CT93}:
\begin{lemma}
\label{lem:3point}
Fix some $\base\in\points$, $\dpoint\in\dpoints$, and let $\point = \mirror(\dpoint)$.
Then, for all $\new[\dpoint]\in\dpoints$, we have:
\begin{equation}
\label{eq:3point}
\fench(\base,\new[\dpoint])
= \fench(\base,\dpoint)
+ \fench(\point,\new[\dpoint])
+ \braket{\new[\dpoint] - \dpoint}{\point - \base}.
\end{equation}
\end{lemma}
\begin{proof}
By definition, we have:
\begin{subequations}
\begin{align}
\label{eq:Fench1}
\fench(\base,\new[\dpoint])
&= \hreg(\base)
+ \hconj(\new[\dpoint])
- \braket{\new[\dpoint]}{\base}
\\
\label{eq:Fench2}
\fench(\base,\dpoint)
&= \hreg(\base)
+ \hconj(\dpoint)
- \braket{\dpoint}{\base}
\\
\label{eq:Fench3}
\fench(\point,\new[\dpoint])
&= \hreg(\point)
+ \hconj(\new[\dpoint])
- \braket{\new[\dpoint]}{\point}
\end{align}
\end{subequations}
Thus, subtracting \eqref{eq:Fench2} and \eqref{eq:Fench3} from \eqref{eq:Fench1}, and rearranging, we get
\begin{equation}
\fench(\base,\new[\dpoint])
= \fench(\base,\dpoint)
+ \fench(\point,\new[\dpoint])
- \hreg(\point)
- \hconj(\dpoint)
+ \braket{\new[\dpoint]}{\point}
- \braket{\new[\dpoint] - \dpoint}{\base}
\eqstop
\end{equation}
Our assertion then follows by recalling that $\point = \mirror(\dpoint)$, so $\hreg(\point) + \hconj(\dpoint) = \braket{\dpoint}{\point}$.
\end{proof}
The next result we present concerns the Fenchel coupling before and after a direct update step;
similar results exist in the literature, but we again provide a proof for completeness.
\begin{lemma}
\label{lem:onestep}
Fix some $\base\in\points$ and $\dpoint,\dvec\in\dpoints$.
Then, letting $\point = \mirror(\dpoint)$, $\new[\dpoint] = \dpoint + \dvec$, and $\new = \mirror(\new[\dpoint])$ as per \eqref{eq:update}, we have:
\begin{subequations}
\label{eq:onestep}
\begin{align}
\fench(\base,\new[\dpoint])
&= \fench(\base,\dpoint)
+ \braket{\dvec}{\new - \base}
- \fench(\new,\dpoint)
\label{eq:onestep1}
\\
&\leq \fench(\base,\dpoint)
+ \braket{\dvec}{\point - \base}
+ \frac{1}{2\hstr} \dnorm{\dvec}^{2}
\eqstop
\label{eq:onestep2}
\end{align}
\end{subequations}
\label{lemma:Fenchel_update_tame_noise}
\end{lemma}
\begin{proof}
By the three-point identity \eqref{eq:3point}, we have
\begin{equation}
\fench(\base,\dpoint)
= \fench(\base,\new[\dpoint])
+ \fench(\new,\dpoint)
+ \braket{\dpoint - \new[\dpoint]}{\new - \base}
\end{equation}
so our first claim is immediate.
For our second claim, rearranging terms and employing the Fenchel\textendash Young inequality gives
\begin{align}
\fench(\base,\new[\dpoint]) - \fench(\base, \dpoint)
&= \braket{\dvec}{\new - \base}
- \fench(\new,\dpoint)
\notag\\
&= \braket{\dvec}{\point - \base}
+ \braket{\dvec}{\new-\point}
- \fench(\new,\dpoint)
\notag\\
&\leq \braket{\dvec}{\point - \base}
+ \frac{1}{2\hstr} \dnorm{\dvec}^{2}
+ \frac{\hstr}{2} \norm{\point-\new}^{2}
- \fench(\new,\dpoint)
\end{align}
so our claim follows from \cref{prop:Fench}.
\end{proof}
In particular, as a consequence of \cref{lemma:Fenchel_update_tame_noise} we obtain the Lipschitzness of the Fenchel coupling with respect to its second variable:
\begin{lemma}
Fix some $\base \in \points$ and suppose that the diameter $\diamX$ of $\points$ is finite. Then, for all $\dpoint, \new[\dpoint] \in \dpoints$,
\begin{equation}
\abs{\fench(\base, \new[\dpoint]) - \fench(\base, \dpoint)} \leq \diamX \dnorm{\new[\dpoint] - \dpoint}.
\end{equation}
\label{lemma:F_Lipschitz}
\end{lemma}
\begin{proof}
Applying the first claim of \cref{lemma:Fenchel_update_tame_noise} to both couples $(\dpoint, \new[\dpoint])$ and $(\new[\dpoint], \dpoint)$ and using that $\fench \geq 0$ (\cf \cref{prop:Fench}), we readily get
\begin{align}
- \diamX \dnorm{\dpoint - \new[\dpoint]}
&\leq - \norm{\mirror(\new[\dpoint]) - \base}\dnorm{\new[\dpoint] - \dpoint}
\notag\\
&\leq - \braket*{\new[\dpoint] - \dpoint}{\mirror(\new[\dpoint]) - \base}
\notag\\
&\leq \fench(\base, \dpoint) - \fench(\base, \new[\dpoint])
% \notag\\
\leq \braket*{\dpoint - \new[\dpoint]}{\mirror(\dpoint) - \base}
\notag\\
&\leq \norm{\mirror(\dpoint) - \base}\dnorm{\dpoint - \new[\dpoint]}
\leq \diamX \dnorm{\dpoint - \new[\dpoint]},
\end{align}
which proves the result.
\end{proof}
Finally, we present a refinement of \cref{lemma:Fenchel_update_tame_noise} that will be needed in the sequel to deal with heavy-tailed noise.
\begin{lemma}
Fix some $\base \in \points$ and $\dpoint, \dvec \in \dpoints$, and suppose that the diameter $\diamX$ of $\points$ is finite. Then, letting $\point = \mirror(\dpoint), \new[\dpoint] = \dpoint + \dvec$, and $\new[\point] = \mirror(\new[\dpoint])$, we have
\begin{align}
\fench(\base, \new[\dpoint]) \leq \fench(\base, \dpoint) + \braket*{\dvec}{\point - \base} + \dfrac{{\diamX}^{2-r}}{\hcst^{r-1}} \dnorm{\dvec}^r
\label{eq:diffF_2}
\end{align}
for every $r \in [1, 2]$.
\label{lemma:diffF_upperbound}
\end{lemma}
\begin{proof}
From the first claim of \cref{lemma:Fenchel_update_tame_noise} and the positiveness of the Fenchel coupling, we get
\begin{align}
\fench(\base, \new[\dpoint])
&= \fench(\base, \dpoint) + \braket*{\dvec}{\point - \base}
+ \braket*{\dvec}{\new[\point] - \point} - \fench(\new, \dpoint)
\notag\\
&\leq \fench(\base, \dpoint) + \braket*{\dvec}{\point - \base} + \dnorm{\dvec}\norm{\new[\point] - \point}.
\end{align}
To continue, note that
\begin{align}
\norm{\new[\point] - \point}\dnorm{\dvec} = \norm{\new[\point] - \point}^{2-r} \norm{\mirror(\new[\dpoint]) - \mirror(\dpoint)}^{r-1} \dnorm{\dvec} \leq \dfrac{\diamX^{2-r}}{\hcst^{r-1}} \dnorm{\dvec}^r,
\end{align}
where the inequality follows by the boundedness of $\points$ and the fact that the mirror map $\mirror$ is $(1/\hcst)$-Lipschitz (\cf \cref{prop:mirror}).
This proves our assertion and concludes our proof.
\end{proof}
%----------------------------------------------------------------------
%% APP: LEVY
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
In this section, we review the necessary background on Lévy processes and stochastic analysis, with an emphasis on the tools required for the convergence analysis developed in the main text. The material presented here is largely standard, and we refer the interested reader to the monographs of \citet{App09}, \citet{Ber96} and \citet{Sat99} for comprehensive introductions to the subject. Throughout this section, we assume a filtered probability space $(\samples, \filter, (\filter(\time))_{\time \geq 0}, \prob)$ satisfying the usual conditions\footnote{That is, (a) the probability space $(\samples, \filter, \prob)$ is complete, meaning that every subset of a $\prob$-null set is measurable; and (b) the filtration $(\filter(\time))_{\time \geq 0}$ is right-continuous, \ie $\filter(\time) = \bigcap_{u > \time} \filter(u)$.}.
\subsection{Lévy processes}
A Lévy process $\levy = (\levyof{\time})_{\time \geq 0}$ is a continuous-time stochastic process taking values in $\mathbb{R}^\vdim$ and satisfying:
\begin{enumerate}
%\setlength{\itemindent}{.3in}
\item \textbf{Initial condition:} $\levyof{0} = 0$ \as
\item \textbf{Independent increments:} For every $0 \leq \time_1 < \dots < \time_k$, the increments $\levyof{\time_{i+1}} - \levyof{\time_i}$ are independent.
\item \textbf{Stationary increments:} For every $\time, \timealt > 0$, the distribution of $\levyof*{\time + \timealt} - \levyof{\time}$ is the same as $\levyof{\timealt}$.
\item \textbf{Stochastic continuity:} For every $\eps > 0$ and $\time \geq 0$, $\lim_{\timealt \to 0^+} \probof*{ \norm{\levyof{\time + \timealt} - \levyof{\time}} > \eps } = 0$.
\end{enumerate}
Under the standard assumptions made on the filtered space, one may always assume that $\levy$ admits càdlàg (\ie right-continuous with left limits) paths.
Lévy processes encompass many important continuous-time stochastic model; among the most prominent examples are the following:
\begin{example}[Brownian motion]
A \emph{Brownian motion} $\brown = (\brownof{\time})_{\time \geq 0}$ is a Lévy process whose increments $\brownof{\time + \timealt} - \brownof{\time}$ are distributed as $\mathcal{N}(0, \timealt \eye_\vdim)$. In particular, Brownian motions are the only Lévy processes that admit continuous sample paths, and they admit finite moments of any order.
\end{example}
\begin{example}[Poisson process]
A \emph{Poisson process} $\pois = (\pois(\time))_{\time\geq 0}$ with intensity $\lambda > 0$ is a nondecreasing, integer-valued Lévy process whose increments $\pois(\time + \timealt) - \pois(\time)$ are distributed as $\mathrm{Poisson}(\lambda \timealt)$. In particular, it also admits moments of all orders.
\end{example}
\begin{example}[Compound Poisson process]
Let $Z = (Z_i)_{i \in \N}$ be a sequence of independent and identically distributed random variables, and let $\pois$ be an independent Poisson process with intensity $\lambda > 0$. Then the stochastic process $X(\time) = \sum_{i=1}^{\pois(\time)} Z_i$ is called a \emph{compound Poisson process}. In particular, $X$ is a Lévy process. Moreover, a compound Poisson process admits moments of order $\momexp$ if and only if the common distribution of the $Z_i$ admits moments of order $\momexp$.
\end{example}
\begin{example}[Stable processes]
A rotationally invariant stable process with stability index $\alpha \in (0, 2]$ is a Lévy process $\levy^{\alpha}$ such that
\begin{equation}
\exof*{ e^{i \inner{u}{\levy^{\alpha}(\time)} }} = e^{-\sigma^\alpha \norm{u}^\alpha \time} \quad \text{for all } u \in \R^\vdim, \time \geq 0,
\end{equation}
where $\sigma > 0$ is a scale factor. When $\alpha = 2$, this definition recovers the Brownian motion. By contrast, when $\alpha < 2$, an $\alpha$-stable process admits finite moments only of order $\momexp < \alpha$, making such processes particularly well suited for modeling infinite-variance phenomena. More general definitions of stable processes without rotational invariance are available in the literature, although their explicit characterization is substantially more involved. We refer the interested reader to the monograph of \citet{ST94} for further details.
\end{example}
\subsection{Jump processes and Poisson random measures}
\label{subsec:random_measures}
As illustrated by the previous examples, Lévy processes are typically discontinuous in nature, making it essential to characterize the behavior of their \emph{jumps} (\ie of their discontinuities) in order to understand their long-term behavior.
Let $\jumpof \levyof{\time} = \levyof{\time} - \levyof{\time-}$ denote the jump size at time $\time$. We define a (random) measure $\poisful$ on $\R^+ \times (\R^\vdim - \{0\})$ by counting the jumps:
\begin{equation}
\poisof{[\timealt, \time]}{B} = \#\setdef*{ \timealt < r \leq \time }{ \jumpof\levyof{r} \in B } = \sum_{\timealt < r \leq \time} \one_{B}(\jumpof\levyof{r})
\end{equation}
for any Borel set $B \subset \R^\vdim - \{0\}$. Thus, $\poisof{[\timealt, \time]}{B}$ counts the number of jumps of size in $B$ occurring in the time interval $[\timealt, \time]$. In particular, $(\poisof{\time}{B})_{\time \geq 0} \defeq (\poisof{[0, \time]}{B})_{\time \geq 0}$ is a Poisson process with intensity $\levymes(B) \time$, where $\levymes$ is the measure on $\R^\vdim - \{0 \}$ defined by
\begin{equation}
\levymes(B) = \exof*{ \poisof{1}{B} } = \exof*{\sum_{0 < \time \leq 1} \one_{B}(\jumpof\levyof{\time})}.
\end{equation}
This measure is known as the \emph{Lévy measure} associated with $\levy$, and it also satisfies the integrability condition
\begin{equation}
\int_{\R^\vdim - \{0\}} (\norm{\jump}^2 \wedge 1) \levymes(d\jump) < \infty.
\end{equation}
An important consequence of this condition is that $\levymes$ is finite on any Borel set that is bounded away from $0$, which implies that a Lévy process admits only finitely many large jumps on any finite time interval. In contrast, $\levymes$ may be infinite at $0$, allowing for infinitely many jumps of small size. Another useful relation links the moments of $\levy$ to those of $\levymes$ \citep[Theorem 25.3]{Sat99}:
\begin{equation}
\exof*{\norm{\levyof{\time}}^\momexp} < \infty \text{ for all } \time \geq 0 \quad \Longleftrightarrow \quad \int_{\norm{\jump} \geq \cjump} \norm{\jump}^\momexp \levymes(d\jump) < \infty \text{ for all } \cjump > 0.
\end{equation}
This result illustrates that the tail behavior of a Lévy process---namely, whether it is heavy-tailed or light-tailed---is entirely determined by the contribution of large jumps, and not by the small jumps nor by the continuous component.
\begin{example}[Brownian motion, cont'd]
A Brownian motion $\brown$ has continuous sample paths and therefore admits no jumps; consequently, its Lévy measure is identically zero.
\end{example}
\begin{example}[Poisson process, cont'd]
All jumps of a Poisson process with intensity $\lambda$ are of size $1$, and the expected number of jumps occurring over the unit time interval $[0, 1]$ is $\lambda$. Hence, its Lévy measure is given by $\levymes = \lambda \delta_1$, where $\delta_1$ denotes the Dirac measure at $1$.
\end{example}
\begin{example}[Compound Poisson process, cont'd]
If $X$ is a compound Poisson process generated by a sequence of independent random variables with common distribution $\mu_Z$, then each jump of $X$ is conditionally distributed according to $\mu_Z$, and the expected number of jumps up to time $1$ is $\lambda$. As a result, the associated Lévy measure is $\levymes = \lambda \mu_Z$.
\end{example}
\begin{example}[Stable processes, cont'd]
If $\levy^\alpha$ is a rotationally invariant $\alpha$-stable process, then its Lévy measure takes the form $\levymes(d\jump) = C/\norm{\jump}^{n+\alpha}$ for some scaling constant $C > 0$. In contrast to compound Poisson processes, this Lévy measure explodes at $0$, implying that stable processes exhibit infinitely many jumps of small magnitude.
\end{example}
The random measure $\poisful$ is commonly referred to as a \emph{Poisson random measure} on $\R^+ \times (\R^\vdim - \{0\})$ with intensity $\levymes(d\jump)d\time$\footnote{In this section we have introduced Poisson random measures only through the specific construction arising from Lévy processes. More generally, Poisson random measures can be defined independently of Lévy processes and in much greater generality. We refer to \citep{App09} and \citep{Kal17} for more comprehensive treatments of this topic.}. Associated to $\poisful$, we also introduce the \emph{compensated Poisson random measure}
\begin{equation}
\cpoisful = \poisful - \levymes(d\jump) d\time.
\end{equation}
In contrast to $\poisful$, the compensated measure enjoys the property that $(\cpoisof*{\time}{B})_{\time \geq 0}$ is a martingale for every bounded Borel set $B \subset \R^\vdim - \{0\}$. This property makes $\cpoisful$ particularly well suited as an integrator for stochastic integrals, as discussed in the next subsection.
\subsection{Stochastic integration for jump processes}
\label{subsec:stochastic_integration}
To define stochastic integration of a stochastic process $(h(\time, \jump))_{\time \geq 0, \jump \in B}$ with respect to the measures $\poisful$ and $\cpoisful$, it is standard to require that $h(\cdot, \jump)$ be \emph{predictable}, meaning that $h(\time, \jump)$ is determined by $\filter(\time-)$\footnote{Rigorously, a stochastic process $(h(\time))_{\time \geq 0}$ is said to be \emph{predictable} if it is measurable with respect to the smallest $\sigma$-field generated by all adapted, left-continuous stochastic processes $(\psi(\time))_{\time \geq 0}$}. In particular, this condition is satisfied whenever the mapping $\time \mapsto h(\time, \jump)$ is \emph{left-continuous}. We then introduce the following class of stochastic processes:
\begin{equation}
\begin{aligned}
\Hlevyof{p}{T}{B}{\R^\vdim} = \bigg\{h \from [0, T] \times B \times \samples \to \R^\vdim : \,\, & h(\cdot, \jump) \text{ is } \text{predictable for all } \jump \in B\\
&\text{and } \exof*{\int_0^{T} \int_B \norm{h(\timealt, \jump)}^p \levymes(d\jump) d\timealt} < \infty\bigg\}
\end{aligned}
\end{equation}
for $p \in [1, 2]$, $T \geq 0$, and any Borel set $B \subset \R^\vdim - \{0\}$.
\begin{enumerate}
\item \textbf{Integration with respect to $\poisful$:} For integration against the raw measure $\poisful$, the integral is a sum over the jumps of the process. Specifically, if $h(\cdot, z)$ is a predictable process, then
\begin{equation}
\int_0^\time \int_B h(\timealt, \jump) \poisfulalt = \sum_{0 < \timealt \leq \time} h(\timealt, \jumpof\levyof{\timealt}) \one_B(\jumpof\levyof{\timealt}) \quad \text{for every } \time \leq T.
\end{equation}
\item \textbf{Integration with respect to $\cpoisful$:} There are two standard approaches to defining stochastic integrals with respect to the compensated measure $\cpoisful$. First, if $h \in \Hlevyof{1}{T}{B}{\R^\vdim}$, one can leverage the integration with respect to $\poisful$ to get
\begin{align}
M(\time)
&\defeq \int_0^\time \int_B h(\timealt, \jump) \cpoisfulalt
\notag\\
&= \int_0^\time \int_B h(\timealt, \jump) \poisfulalt - \int_0^\time \int_B h(\timealt, \jump) \levymes(d\jump)d\timealt \quad \text{for every } \time \leq T.
\end{align}
This construction results to the process $M(\time)$ being a \emph{càdlàg martingale} \citep[Chapter II, p.62]{IW81}.
Alternatively, if $h \in \Hlevyof{2}{T}{B}{\R^\vdim}$, then the stochastic integral $M(\time)$ can be defined as the $L^2$-limit of simple functions using the isometry
\begin{equation}
\exof*{ \norm*{\int_0^\time \int_B h(\timealt, \jump) \cpoisfulalt}^2 } = \exof*{ \int_0^\time \int_B \norm{h(\timealt, \jump)}^2 \levymes(d\jump) d\timealt }.
\end{equation}
In this case, $M(\time)$ is a \emph{càdlàg square-integrable martingale} \citep[Theorem 4.2.3]{App09}.
\end{enumerate}
Another class of stochastic processes that will be useful in the remainder of the paper is
\begin{equation}
\txs
\H2of{T}{\R^\vdim} = \setdef*{g \from [0, T] \times \samples \to \R^\vdim}{\text{$g$ is predictable and } \exof*{\int_0^\time \norm{g(\timealt)}^2 \dd\timealt} < \infty}, \quad T \geq 0.
\end{equation}
In particular, any $g \in \H2of{T}{\R^\vdim}$ can be integrated against a $\vdim$-dimensional Brownian motion $\brown$ as the $L^2$-limit of simple functions, using the standard Itô isometry formula
\begin{equation}
\exof*{ \norm*{ \int_0^\time g(\timealt)^{\top} d\brownof{\timealt} }^2 } = \exof*{ \int_0^\time \norm{g(\timealt)}^2 d\timealt }.
\end{equation}
As in the case of integration with respect to $\cpoisful$, the resulting stochastic integral is a \emph{continuous square-integrable martingale}.
\subsection{The Lévy-Itô decomposition}
\label{subsec:LevyIto}
Any Lévy process $\levy$ can be uniquely represented as a sum of a continuous part and independent jump components \citep[Theorem 2.4.16]{App09}: for every constant $c > 0$,
\begin{align}
\levyof*{\time} = \underbrace{b_\cjump \time + \diffmat \brownof*{\time}}_{\text{Continuous part}} + \underbrace{\int_0^\time \int_{\norm{z} < \cjump} \jump \cpoisfulalt}_{\text{Compensated small jumps}} + \underbrace{\int_0^\time \int_{\norm{z} \geq \cjump} \jump \poisfulalt}_{\text{Large jumps}},
\end{align}
where $b_c \in \R^\vdim$ is a deterministic drift, $\brown$ is a $d$-dimensional Brownian motion independent of $\pois$, and $\diffmat \in \R^{\vdim \times d}$ is a deterministic diffusion matrix. The constant $\cjump$ acts as a cutoff separating between bounded and unbounded jumps, and may be chosen arbitrarily. For example, if the jumps of $L$ are uniformly bounded by some $M > 0$, one may take $\cjump = M + 1$, in which case the large-jump component vanishes.
It is important to note that the small jumps must be integrated against the compensated measure $\cpoisful$, rather than the raw measure $\poisful$, to ensure that the stochastic integral is well defined even when $\int_{\norm{z} < \cjump} \norm{\jump} \levymes(d\jump) = \infty$.
In particular, if $\levy$ is an integrable centered Lévy process, then it admits the decomposition
\begin{align}
\levyof{\time} &= - \parens*{\int_{\norm{\jump} \geq \cjump} \jump \levymes(d\jump)}\time + \diffmat \brownof*{\time} + \int_0^\time \int_{\norm{\jump} < \cjump} \jump \cpoisfulalt + \int_0^\time \int_{\norm{\jump} \geq \cjump} \jump \poisfulalt
\notag\\
&= \diffmat \brownof*{\time} + \int_0^\time \int_{\norm{\jump} < \cjump} \jump \cpoisfulalt + \int_0^\time \int_{\norm{\jump} \geq \cjump} \jump \cpoisfulalt
\notag\\
&= \itoof{\time} + \timed{\cadshort} + \timed{\cadlong}
\end{align}
where $\itoof{\time}$ is a continuous square-integrable martingale, $\timed{\cadshort}$ is a purely discontinuous square-integrable martingale with bounded jumps, and $\timed{\cadlong}$ is a purely discontinuous martingale with possibly unbounded jumps (see \cref{subsec:stochastic_integration}).
With this decomposition in mind, the intensity of each component of $\levy$ can then be quantified as follows:
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item \define{Diffusion intensity:}
\begin{equation}
\contbound^2 = \exof*{\norm{\itoof{1}}^2} = \norm{\diffmat}_F^2 = \tr[\diffmat^\top \diffmat].
\end{equation}
\item \define{Short jump intensity:}
\begin{equation}
\shortbound^2 = \exof*{\norm{\cadshort(1)}^2} = \int_{\norm{\jump}<\cjump} \norm{\jump}^2 \levymes(d\jump).
\end{equation}
\item \define{Long jump intensity:}
\begin{equation}
\longbound^\momexp = \int_{\norm{\jump} \geq \cjump} \norm{\jump}^\momexp \levymes(d\jump) \geq C_\momexp \exof*{\norm{\cadlong(1)}^\momexp}, \quad \momexp \geq 0.
\end{equation}
\end{enumerate}
Owing to the relation between $\levy$ and its Lévy measure, the long jump intensity $\longbound^\momexp$ is finite if and only if $\levy(1)$ admits finite moments of order $\momexp$. Moreover, in general $\longbound^\momexp$ does not coincide with $\exof*{\norm{\cadlong(1)}^\momexp}$ unless $\momexp = 2$. We refer the interested reader to \cite{Nov75a} for a detailed discussion of this discrepancy and for estimates on the constant $C_\momexp$ relating the two quantities.
\subsection{Asymptotic behavior of discontinuous martingales}
In this subsection, we present two classical results from stochastic analysis---accordingly the \emph{law of large numbers} and the \emph{law of iterated logarithm}---adapted to the setting of discontinuous martingales. Unlike the continuous case, these results are less widely known for processes with jumps.
We begin with a strong law of large numbers for stochastic integrals with respect to compensated Poisson random measures. Throughout, we let $\poisful$ denote the random Poisson measure associated with a Lévy process $\levy$ having Lévy measure $\levymes$ (see \cref{subsec:random_measures}).
\begin{lemma}[Strong law of large numbers]
Fix some index $p \in [1, 2]$ and let $M(\time) = \int_0^\time \int_B H(\timealt, \jump) \cpoisfulalt$ with $H \in \Hlevyof{p}{T}{B}{\R^\vdim}$ for all $T \geq 0$. If there is a predictable increasing process $A$ such that $\lim A(\time) = \infty$ \as and
\begin{equation}
\int_0^\infty \int_B \dfrac{ \norm{H(\timealt, \jump)}^2 \wedge \norm{H(\timealt, \jump)}^p }{\bracks*{1 + A(\timealt)}^p} \levymes(d\jump) d\timealt < \infty \quad \as,
\end{equation}
then $M(\time) / A(\time) \to 0$ \as.
\label{lemma:SLLN}
\end{lemma}
\begin{proof}
This is a consequence of \citep[Lemma 3]{LM89}. Indeed, $M$ is a centered purely discontinuous martingale since $H \in \Hlevyof{p}{T}{B}{\R^\vdim}$ and, following the notations from \citep{LM89}, we can introduce the auxiliary process $W_p$ given by
\begin{equation}
W_p(\time) = \sum_{0 < \timealt \leq \time} \norm{\jumpof M(\timealt)}^2 \wedge \norm{\jumpof M(\timealt)}^p = \int_0^\time \int_B \norm{H(\timealt, \jump)}^2 \wedge \norm{H(\timealt, \jump)}^p \poisfulalt.
\end{equation}
In particular, its integrand satisfies
\begin{equation}
\exof*{ \int_0^\time \int_B \norm{H(\timealt, \jump)}^2 \wedge \norm{H(\timealt, \jump)}^p \levymes(d\jump) d\timealt} \leq \exof*{ \int_0^\time \int_B \norm{H(\timealt, \jump)}^p \levymes(d\jump) d\timealt} < \infty
\end{equation}
since $H \in \Hlevyof{p}{T}{B}{\R^\vdim}$, and thus
\begin{equation}
\overline{W_p}(\time) = \int_0^\time \int_B \norm{H(\timealt, \jump)}^2 \wedge \norm{H(\timealt, \jump)}^p \levymes(d\jump) d\timealt
\end{equation}
is the compensator of $W_p$, \ie the unique predictable càdlàg process starting from $0$ with finite variation over bounded intervals and such that $W_p - \overline{W_p}$ is a (local) martingale. According to \citep[Lemma 3]{LM89}, the strong law of large number $M(\time) / A(\time) \to 0$ therefore holds true whenever
\begin{equation}
\infty > \int_0^\infty \dfrac{d\overline{W_p}(\timealt)}{\bracks*{1 + A(\timealt)}^p} = \int_0^\infty \int_B \dfrac{ \norm{H(\timealt, \jump)}^2 \wedge \norm{H(\timealt, \jump)}^p }{\bracks*{1 + A(\timealt)}^p} \levymes(d\jump) d\timealt,
\end{equation}
which is exactly the condition stated in the lemma.
\end{proof}
To state the law of iterated logarithm for discontinuous martingales, we first need to introduce another essential object from stochastic analysis: the \emph{predictable quadratic variation}. For any càdlàg square-integrable local martingale $M$, its \emph{predictable quadratic variation} (also sometimes known as \emph{Meyer's angle braket}) is the unique increasing predictable process $\meyerof{M}$ such that $M^2 - \meyerof{M}$ is a local martingale. In particular, when $M$ arises as a stochastic integral with respect to either a Brownian motion or a compensated Poisson measure, the following identities hold:
\begin{subequations}
\begin{align}
\meyerof*{ \int_0^\time g(\timealt)^{\top} d\brownof{\timealt} } &= \int_0^\time \norm{g(\timealt)}^2 d\timealt, \quad g \in \H2of{T} {\R^\vdim};\\
\meyerof*{ \int_0^\time \int_{B} h(\timealt, \jump) \cpoisof{d\timealt}{d\jump} } &= \int_0^\time \int_B \norm{h(\timealt, \jump)}^2 \levymes(d\jump)d\timealt, \quad h \in \Hlevyof{2}{T}{B}{\R^\vdim}.
\end{align}
\end{subequations}
We can now formulate the law of iterated logarithm, which characterizes the asymptotic fluctuations of a martingale in terms of its predictable quadratic variation:
\begin{lemma}[Law of iterated logarithm, {\citep[Theorem 3]{Lep78}}]
Let $M$ be a square-integrable local martingale and assume that there exists a constant $C \geq 0$ such that $\abs{\jumpof M(\time)} \leq C$ \as for every $\time \geq 0$. Then, the trajectories of $M$ satisfy
\begin{equation}
\limsup_{\time \to \infty} \dfrac{\abs{M(\time)}}{\sqrt{ 2\meyerof{M}(\time) \log\log \meyerof{M}(\time) }} = 1 \quad \as
\end{equation}
on the event $\braces*{\lim_{\time \to \infty} \meyerof{M}(\time) = \infty}$.
\label{lemma:LIL}
\end{lemma}
\subsection{An Itô's formula for smooth convex function}
\label{subsec:weakIto}
Analogous to the chain rule in classical differential calculus, Itô's formula allows one to describe the evolution of a composition $\psi \circ \dstate$, where $\dstate$ is some $\vdim$-dimensional stochastic process and $\psi \from \R^\vdim \to \R$ is a deterministic function. Because of the way stochasticity is handled in integration with respect to martingales, Itô's formula typically includes an additional second-order term---commonly referred to as \emph{Itô's correction}---which requires $\psi$ to be at least twice continuously differentiable for the formula to hold.
However, in many applications, such as the analysis of stochastic mirror flows, the function $\psi$ of interest does not satisfy this regularity condition. The goal of this subsection is therefore to introduce a weak version of Itô's formula that apply to a more general class of functions: those that are only convex and $\lips$-smooth.
Recall that a function $\psi \from \R^\vdim \to \R$ is said to be \emph{$\lips$-smooth} if it satisfies
\begin{equation}
\norm{\nabla \psi(\dpoint) - \nabla \psi(\dpoint')} \leq \lips \dnorm{\dpoint - \dpoint'} \quad \text{for all } \dpoint, \dpoint' \in \R^\vdim.
\end{equation}
In particular, $\lips$-smooth functions that are also convex enjoy the following property:
\begin{lemma}
If $\psi$ is an $\lips$-smooth convex function, then $\psi$ is almost everywhere twice differentiable with Hessian $\hess \psi$ and
\begin{equation}
0 \leq \hess \psi \leq L \eye \quad \text{Lebesgue-a.e.}
\end{equation}
\label{lemma:L_smooth_convex}
\end{lemma}
\begin{proof}
The proof of this result is standard, see \eg \citep[Lemma C.3]{MerSta18}.
\end{proof}
A weak Itô's formula for smooth convex functions was previously established by \citet{MerSta18} for Itô diffusions, that is, for processes of the form $\dstateof{\time} = \dstateof{0} + \int_0^\time f(\timealt) d\timealt + \int_0^\time g(\timealt) d\brownof{\timealt}$ where $\brown$ is a standard Brownian motion. In the following theorem, we extend their result to the broader class of \emph{Lévy-type processes}, which are characterized as follows:
Let $\levy$ be a $k$-dimensional Lévy process with Lévy measure $\levymes$, and let $\poisful$ be the Poisson random measure controlling its jumps. Then, a \emph{Lévy-type process} $\dstate$ is a $\vdim$-dimensional càdlàg stochastic process of the form
\begin{align}
\dstateof{\time}
= \dstateof{0}
&+ \underbrace{\int_0^\time f(\timealt) \dd\timealt + \int_0^\time g(\timealt) \dd\brownof{\timealt}}_{\text{Continuous part } \dstate^c(\time)}
\notag\\
&+ \underbrace{\int_0^\time \int_{\cjumpinf} H(\timealt, \jump) \cpoisfulalt + \int_0^\time \int_{\cjumpsup} G(\timealt, \jump) \poisfulalt}_{\text{Discontinuous part } \dstate^d(\time)}
\end{align}
for some constant $c > 0$, where $\dstateof{0} < \infty$ \as, $\brown$ is a standard $d$-dimensional Brownian motion independent of $\pois$, and, for every $T \geq 0$,
\begin{enumerate}
\setlength{\itemindent}{.3in}
\item $f \from \R^+ \times \samples \to \R^\vdim$ is such that $\abs{f_i}^{1/2} \in \H2of{T}{\R}$;
\item $g \from \R^+ \times \samples \to \R^{\vdim \times d}$ is such that $g_i \in \H2of{T}{\R^d}$;
\item $H \from \R^+ \times \braces{\cjumpinf} \times \samples \to \R^\vdim$ is such that $H \in \Hlevyof{2}{T}{\braces{\cjumpinf}}{\R^\vdim}$;
\item $G \from \R^+ \times \braces{\cjumpsup} \times \samples \to \R^\vdim$ is predictable.
\end{enumerate}
By virtue of the Lévy-Itô decomposition (\cf \cref{subsec:LevyIto}), Lévy-type processes therefore provide a general and flexible framework to construct stochastic integrals with respect to a Lévy process.
This general representation of Lévy-type processes sets the stage for extending Itô’s formula: we can now formulate a weak version of the formula that applies to convex $\lips$-smooth functions, encompassing both the continuous and discontinuous components of $\dstate$:
\begin{theorem}[Weak Itô's formula for smooth convex functions]
% Let $\dstate$ be a $\vdim$-dimensional stochastic process of the form
% \begin{equation}
% \dstateof{\time} = \dstateof{0} + \int_0^\time f(\timealt) \dd\timealt + \int_0^\time g(\timealt) \dd\brownof{\timealt} + \int_0^\time \int_{\cjumpinf} H(\timealt, \jump) \cpoisful + \int_0^\time \int_{\cjumpsup} G(\timealt, \jump) \poisful
% \end{equation}
% for $\time \leq T$, where $\dstateof{0} < \infty$ \as, $\brown$ is a standard $d$-dimensional Brownian motion, $\pois$ is a Poisson random measure on $\R^+ \times (\R^k - \braces{0})$ with intensity $d\time \levymes(d\jump)$, and
% \begin{enumerate}
% \item $f \from [0, T] \times \samples \to \R^\vdim$ is such that $\abs{f_i}^{1/2} \in \H2of{T}{\R}$;
% \item $g \from [0, T] \times \samples \to \R^{\vdim \times d}$ is such that $g_i \in \H2of{T}{\R^d}$;
% \item $H \from [0, T] \times \braces{\cjumpinf} \times \samples \to \R^\vdim$ is such that $H \in \Hlevyof{2}{T}{\braces{\cjumpinf}}{\R^\vdim}$;
% \item $G \from [0, T] \times \braces{\cjumpsup} \times \samples \to \R^\vdim$ is predictable.
% \end{enumerate}
Let $\dstate$ be a $\vdim$-dimensional Lévy-type process. If $\psi \from \R^\vdim \to \R$ is an $L$-smooth convex function, then
\begin{align}
\psi(\dstateof{\time}) - \psi(\dstateof{0}) &\leq \int_0^\time \braket*{\nabla \psi(\dstateof{\timealt-})}{d\dstate^c(\timealt)}
\notag\\
&+ \int_0^\time \int_{\cjumpinf} \bracks*{\psi(\dstateof{\timealt-} + H(\timealt, \jump)) - \psi(\dstateof{\timealt-})} \cpoisfulalt
\notag\\
&+ \int_0^\time \int_{\cjumpsup} \bracks*{\psi(\dstateof{\timealt-} + G(\timealt, \jump)) - \psi(\dstateof{\timealt-})} \poisfulalt
\notag\\
&+ \dfrac{L}{2} \parens*{ \int_0^\time \norm{g(\timealt)}^2 d\timealt + \int_0^\time \int_{\cjumpinf} \norm{H(\timealt, \jump)}^2 \levymes(d\jump) d\timealt },
\end{align}
where $\dstateof{\timealt-}$ denotes the left limit of $\dstate$ at $\timealt$ \footnote{As discussed in \cref{subsec:stochastic_integration}, stochastic integrals with respect to a Poisson random measure require the integrands to be \emph{predictable}. Since $\dstate$ is right-continuous, its left limit $\dstateof{\time-}$ defines a left-continuous process which is predictable by definition, and thus all stochastic integrals appearing in \cref{thm:weakIto} are well-defined in this setting.}.
%where $\dstate^c$ denotes the continuous part of $\dstate$.
\label{thm:weakIto}
\end{theorem}
We first prove this theorem under the assumption $G \equiv 0$, that is, when the jumps of $\dstate$ are of bounded magnitude. This condition is convenient because, in this case, the discontinuous component of $\dstate$ reduces to a square-integrable martingale, which significantly simplifies the setting.
\begin{proof}[Proof of \cref{thm:weakIto} (case $G \equiv 0$)]
Let $\rho \from \R^\vdim \to \R$ be the unit mollifier given by
\begin{equation}
\rho(u) = \begin{cases}
c \exp\parens*{- \dfrac{1}{1 - \norm{u}^2}} & \text{if } \norm{u} \leq 1,\\
0 & \text{otherwise};
\end{cases}
\end{equation}
where $c$ is a positive normalizing constant such that $\int \rho(u) du = 1$. For every $\eps > 0$, we also define
\begin{equation}
\rho_\eps(w) = \eps^{-\vdim} \rho(w/\eps)
\end{equation}
and
\begin{equation}
\psi_\eps(\dpoint) = (\psi \ast \rho_\eps)(\dpoint) = \int \psi(\dpoint - w)\rho_\eps(w) dw.
\end{equation}
From classical smoothing arguments, we note that $\psi_\eps$ is twice continuously differentiable, and so we can apply the classical Itô's formula for Lévy-type stochastic integrals (\cf \citep[Theorem 4.4.7]{App09}) to get
\begin{subequations}
\begin{align}
\psi_\eps(\dstateof{\time}) - \psi_\eps(\dstateof{0})
&= \int_0^\time \braket*{\nabla \psi_\eps(\dstate)}{d\dstate^c}\label{eq:weak_ito_d1}
\\
&+ \int_0^\time \int_{\cjumpinf} \bracks*{ \psi_\eps(\dstate + H) - \psi_\eps(\dstate) } \cpoisfulalt
\label{eq:weak_ito_d2}\\
&+ \dfrac{1}{2} \int_0^\time \tr\bracks*{ \hess\psi_\eps(\dstate) gg^{\top} } d\timealt
\label{eq:weak_ito_d3}\\
&+ \int_0^\time \int_{\cjumpinf} \bracks*{ \psi_\eps(\dstate + H) - \psi_\eps(\dstate) - \braket*{ \nabla \psi_\eps(\dstate) }{H} } \levymes(d\jump) d\timealt,
\label{eq:weak_ito_d4}
\end{align}
\end{subequations}
where we have suppressed the dependence on time for notational convenience.
To bound \cref{eq:weak_ito_d3}, note that the definition of $\psi_\eps$ and \cref{lemma:L_smooth_convex} can be used to obtain
\begin{equation}
\tr\bracks*{\hess \psi_\eps(y) zz^{\top}} = \int_{\R^\vdim} \tr\bracks*{ \hess\psi (w) zz^{\top}} \rho_\eps(y - w)dw \leq L \norm{z}^2
\end{equation}
for every $y, z \in \R^{\vdim}$, which leads to
\begin{equation}
\text{\eqref{eq:weak_ito_d3}} \leq \dfrac{L}{2} \int_0^\time \norm{g(\timealt)}^2d\timealt.
\end{equation}
Furthermore, by the classical descent lemma for $L$-smooth functions (see \eg \citep[Theorem 2.1.5]{Nes04}), we get
\begin{equation}
\psi(y+h) - \psi(y) - \braket*{\nabla \psi(y)}{h} \leq \dfrac{L}{2} \norm{h}^2
\end{equation}
for all $y, h \in \R^\vdim$, so that
\begin{equation}
\text{\eqref{eq:weak_ito_d4}} \leq \dfrac{L}{2} \int_0^\time \int_{\cjumpinf} \norm{H(\timealt, \jump)}^2 \levymes(d\jump) d\timealt,
\end{equation}
where the right-hand side is almost-surely finite by assumption on $H$. We therefore obtain that
\begin{align}
\psi_\eps(\dstateof{\time}) - \psi_\eps(\dstateof{0})
&\leq \int_0^\time \braket*{\nabla \psi_\eps(\dstate)}{d\dstate^c}
\notag\\
&+ \int_0^\time \int_{\cjumpinf} \bracks*{ \psi_\eps(\dstate + H) - \psi_\eps(\dstate) } \cpoisfulalt
\notag\\
&+\dfrac{L}{2} \parens*{ \int_0^\time \norm{g}^2d\timealt + \int_0^\time \int_{\cjumpinf} \norm{H}^2 \levymes(d\jump) d\timealt}.
\label{eq:weak_ito_dbis}
\end{align}
To prove the weak Itô's formula, it only remains to show that we can replace $\psi_\eps$ by $\psi$ when taking the almost-sure limit as $\eps \to 0^+$ on both sides of the inequality. For the sake of readibility, let us denote $L(\psi_\eps)$ the left-hand side of \cref{eq:weak_ito_dbis} and $R(\psi_\eps)$ its right-hand side. We begin by establishing some groundwork that will become quite useful in the remaining of the proof.
First, note that for every $y,y' \in \R^\vdim$ and every differentiable function $\phi \from \R^\vdim \to \R$, we can write $\phi(y) - \phi(y') = \int_0^1 \braket*{\nabla \phi(y_r)}{y - y'} dr$ where $y_r = y + r(y' - y)$. Applying this equality to $\psi$ and $\psi_\eps$, we get
\begin{align}
\abs*{ \psi(y) - \psi(y') - \psi_\eps(y) + \psi_\eps(y') }
&= \abs*{ \int_0^1 \braket*{ \nabla \psi(y_r) - \nabla \psi_\eps(y_r)}{y - y'}dr }
\notag\\
&\leq \norm{y-y'} \int_0^1 \norm{\nabla \psi(y_r) - \nabla \psi_\eps(y_r)}dr.
\label{eq:diff_of_psi_1}
\end{align}
Now, by using the definition of $\psi_\eps$ and the $L$-smoothness of $\psi$, we can further derive
\begin{align}
\norm{\nabla \psi(y_r) - \nabla \psi_\eps(y_r)}
&\leq \int_{\R^\vdim} \norm{\nabla \psi(y_r) - \nabla \psi(y_r - w)} \rho_\eps(w) dw
\notag\\
&\leq L \int_{\R^\vdim} \norm{w} \eps^{-\vdim}\rho(w/\eps) dw
\notag\\
&= L \eps \int_{\norm{u}<1} \norm{u} \rho(u)du
\notag\\
&\leq L \eps,
\label{eq:diff_of_psi_2}
\end{align}
where the second-to-last inequality comes from the substitution $u = w/\eps$ and the fact that $\rho(u)$ is zero outside of $\norm{u} < 1$. In particular, it means that $\nabla \psi_\eps$ converges uniformly to $\nabla \psi$ on all $\R^\vdim$ and not only on compact subsets. Coming back to \cref{eq:diff_of_psi_1}, the previous estimate therefore yields
\begin{equation}
\abs*{\psi(y) - \psi(y') - \psi_\eps(y) + \psi_\eps(y')} \leq L \eps \norm{y - y'}
\end{equation}
for every $y, y' \in \R^\vdim$. Applying this inequality to the left-hand side $L$, we then readily get
\begin{equation}
\abs*{L(\psi) - L(\psi_\eps)} \leq L \eps \norm{\dstateof{\time} - \dstateof{0}},
\end{equation}
which implies that $L(\psi_\eps) \to L(\psi)$ almost-surely as $\eps \to 0^+$ since $\dstateof{\time}$ and $\dstateof{0}$ are both finite by assumption. Now, to derive the limit of $R(\psi_\eps)$, we use the $L^1$-norm and expend $d\dstate^c$ to get
\begin{align}
&\exof*{\abs{R(\psi) - R(\psi_\eps)}} \notag\\
&\leq \int_0^\time \exof*{ \abs*{ \braket*{\nabla \psi(\dstate) - \nabla \psi_\eps(\dstate)}{f} } } d\timealt + \exof*{\abs*{ \int_0^\time \braket*{\nabla \psi(\dstate) - \nabla \psi_\eps(\dstate)}{g d\brown}} }
\notag\\
+ \,\,
& \exof*{ \abs*{ \int_0^\time \int_{\cjumpinf} \bracks*{ \psi(\dstate + H) - \psi(\dstate) - \psi_\eps(\dstate+H) + \psi_\eps(\dstate) } \cpoisfulalt } }
\notag\\
&\leq \int_0^\time \exof*{ \norm{\nabla \psi(\dstate) - \nabla \psi_\eps(\dstate)}\norm{f} } d\timealt
+ \sqrt{\!\int_0^\time \exof*{\norm{\nabla \psi(\dstate) - \nabla \psi_\eps(\dstate)}^2 \norm{g}^2} d\timealt}
\notag\\
+ \,\,
& \sqrt{\! \int_0^\time \int_{\cjumpinf} \exof*{\abs*{ \psi(\dstate + H) - \psi(\dstate) - \psi_\eps(\dstate+H) + \psi_\eps(\dstate) }^2} \levymes(d\jump) d\timealt }
\end{align}
where the second inequality comes from Jensen's inequality and Itô's isometry formula for stochastic integrals. Thanks to the estimates \cref{eq:diff_of_psi_1,eq:diff_of_psi_2} that we have previously established, it follows that
\begin{align}
&\exof*{\abs*{R(\psi) - R(\psi_\eps)}} \\
\leq \,\, & L \eps
\parens*{
\exof*{\int_0^\time \norm{f} d\timealt}
+ \sqrt{\!\int_0^\time \exof*{\norm{g}^2 } d\timealt}
+ \sqrt{\!\int_0^\time \int_{\cjumpinf} \exof*{\abs*{H}^2} \levymes(d\jump) d\timealt }
}
\end{align}
where each integral is finite by assumption on the coefficients. We therefore get $R(\psi_\eps) \to R(\psi)$ in the $L^1$-norm as $\eps \to 0^+$, and thus there exists a subsequence $\eps(i) \to 0$ such that $R(\psi_{\eps(i)}) \to R(\psi)$ \as. Along this subsequence we also get $L(\psi_{\eps(i)}) \to L(\psi)$ \as, which proves that $L(\psi) \leq R(\psi)$ \as, hence finishing the proof.
\end{proof}
Having established \cref{thm:weakIto} for Lévy-type processes with bounded jumps, we now extend the result to the general case, where jumps of unbounded magnitude may occur. A key property of Lévy processes---and their associated Lévy measure---is that only finitely many large jumps can happen over any finite time interval. Exploiting this fact, we can apply the previously proven case $G\equiv 0$ on each subinterval between large jumps, and then account for the large jumps separately.
\begin{proof}[Proof of \cref{thm:weakIto} (general case)]
Let $(T_i)_{i \geq 1}$ denote the arrival times of the Poisson process $\parens*{\poisof{\time}{\braces{\cjumpsup}}}_{0 \leq \time \leq T}$ controlling the size of large jumps of $\dstate$ up to time $T$. Since the intensity measure $\levymes$ is assumed to be a Lévy measure, it is in particular finite on $\braces{\cjumpsup}$ and so there is only a finite number of jumps occurring in the interval $[0, T]$. Let $N$ denotes this (random) number of jumps. We further define $T_0 = 0$ and $T_{N+1} = T$ for notational convenience. Fix $\time \leq T$ and let $i$ be the random integer such that $\time \in [T_i, T_{i+1})$. At the time $\time$, we can therefore write
\begin{equation}
\dstateof{\time} = \dstateof{T_i} + \int_{T_i}^\time f(\timealt) d\timealt + \int_{T_i}^\time g(\timealt) d\brownof{\timealt} + \int_{T_i}^\time \int_{\cjumpinf} H(\timealt, \jump) \cpoisfulalt,
\end{equation}
which falls into the setup of \cref{thm:weakIto} with $G \equiv 0$ since there are no large jumps during this time interval (except at the endpoints), $T_i \leq T < \infty$ by construction, and thanks to the independence of the Poisson process $\poisof{\cdot}{\braces{\cjumpsup}}$ to $\poisof{\cdot}{\braces{\cjumpinf}}$ and $\brown$. It then follows from the proof of case $G \equiv 0$ that
\begin{align}
\psi(\dstateof{\time})
\leq \psi(\dstateof{T_i})
&+ \int_{T_i}^\time \braket*{\nabla \psi(\dstate)}{d\dstate^c} + \int_{T_i}^\time \int_{\cjumpinf} \bracks*{ \psi(\dstate + H) - \psi(\dstate) } \cpoisfulalt
\notag\\
&+ \dfrac{L}{2} \parens*{ \int_{T_i}^\time \norm{g}^2 d\timealt + \int_{T_i}^\time \int_{\cjumpinf} \norm{H}^2 \levymes(d\jump)d\timealt }.
\end{align}
On the other hand, if we define the Lévy process $\levyof{\time} = \int_{\cjumpsup} \jump \poisof{\time}{d\jump}$, we also have
\begin{align}
\psi(\dstateof{T_i})
&= \psi(\dstateof{T_i-}) + \jumpof \psi(\dstateof{T_i})
\notag\\
&= \psi(\dstateof{T_i-}) + \bracks*{ \psi(\dstateof{T_i-} + G(T_i, \jumpof \levyof{T_i}) - \psi(\dstateof{T_i-}) }.
\end{align}
Since $\time \mapsto \psi(\dstateof{\time-})$ is left-continuous, applying the case $G \equiv 0$ on each time interval $[T_0, T_1), \dots, [T_{i-2}, T_{i-1})$ therefore yields
\begin{alignat}{3}
\psi(\dstateof{\time})
&&\leq \psi(\dstateof{T_0})
&+ \int_{T_0}^\time \braket*{\nabla \psi(\dstate)}{d\dstate^c}
+ \int_{T_0}^\time \int_{\cjumpinf} \bracks*{ \psi(\dstate + H) - \psi(\dstate) } \cpoisfulalt
\notag\\
&&
&+ \dfrac{L}{2} \parens*{ \int_{T_0}^\time \norm{g}^2 d\timealt + \int_{T_0}^\time \int_{\cjumpinf} \norm{H}^2 \levymes(d\jump)d\timealt }
\notag\\
&&
&+ \sum_{j=1}^i \bracks*{ \psi(\dstateof{T_j-} + G(T_j, \jumpof \levyof{T_j}) - \psi(\dstateof{T_j-}) }
\notag\\
&&= \psi(\dstateof{0})
&+ \int_{0}^\time \braket*{\nabla \psi(\dstate)}{d\dstate^c}
+ \int_{0}^\time \int_{\cjumpinf} \bracks*{ \psi(\dstate + H) - \psi(\dstate) } \cpoisfulalt
\notag\\
&&
&+ \dfrac{L}{2} \parens*{ \int_{0}^\time \norm{g}^2 d\timealt + \int_{0}^\time \int_{\cjumpinf} \norm{H}^2 \levymes(d\jump)d\timealt }
\notag\\
&&
&+ \int_{0}^\time \int_{\cjumpsup} \bracks*{ \psi(\dstate + G) - \psi(\dstate) } \poisfulalt
\end{alignat}
by definition of the stochastic integration with respect to a Poisson random measure (\cf \cref{app:levy}). This is exactly the upper bound stated in \cref{thm:weakIto}, thus concluding the proof.
\end{proof}
%\subsection{Summary of some useful quantities and notations}
%
%\begin{itemize}
%\setlength{\itemindent}{.3in}
%\item $\levyof{\time}$ : Lévy process.
%\item $\jumpof \levyof{\time} = \levyof{\time} - \levyof{\time-}$ : jump process of $\levy$.
%\item $\poisof{[\timealt, \time]}{B} = \#\setdef*{\timealt \leq r \leq \time}{\jumpof\levyof{r} \in B}$ for Borel subsets $B \in \borel(\R^\vdim - \{0\})$ : Poisson random measure associated to jumps of $\levy$. Usually written $\poisful$ to emphasize the fact that it spans a (random) measure on $\R^+ \times (\R^\vdim - \{0\})$.
%\item $\levymes(B) = \exof*{\poisof{1}{B}}$ : Lévy measure on $\R^\vdim - \{0\}$.
%\item $\cpoisful = \poisful - \levymes(d\jump) d\time$ : compensated Poisson measure. $(\cpoisof{\time}{B})_{\time \geq 0}$ is a martingale, square-integrable iif $\int_B \dnorm{\jump}^2 \levymes(d\jump) < \infty$, so it is possible to integrate with respect to it using same kind of construction as for Brownian motions.
%\item $c$ : arbitrary cutoff constant to distinguish between bounded and unbounded jumps.
%\item $\meyerof{M}$ unique increasing predictable process such that $M^2 - \meyerof{M}$ is a martingale (presuppose that $M$ is a càdlàg square-integrable martingale) : usually called predictable quadratic variation or Meyer's angle bracket (warning: for jump processes, this is not equal to the classical quadratic variation defined as the limit of local variations, but for this paper in particular we only need predictable quadratic variations so the distinction is not important to make).
%\end{itemize}
%
%\begin{align}
%\H2of{T}{\R^\vdim} &= \setdef*{g \from [0, T] \times \samples \to \R^\vdim}{g \text{ is predictable and } \exof*{\int_0^\time \norm{g(\timealt)}^2 \dd\timealt} < \infty}\\
%\Hlevyof{p}{T}{B}{\R^\vdim} &= \setdef*{h \from [0, T] \times B \times \samples \to \R^\vdim}{h(\cdot, \jump) \text{ is predictable for all } \jump \in B \text{ and } \exof*{\int_0^\time \int_B \norm{h(\timealt, \jump)}^p \levymes(d\jump) d\timealt} < \infty}
%\end{align}
%
%
%Since $\levy$ is assumed to be centered, it can be written through its Lévy-Itô decomposition as
%\begin{equation}
% \levyof{\time} = - \parens*{\int_{\cjumpsup} \jump \levymes(d\jump)} \time + \diffmat \brownof{\time} + \int_{\cjumpinf} \jump \cpoisof{\time}{d\jump} + \int_{\cjumpsup} \jump \poisof*{\time}{d\jump},
%\end{equation}
%where $\cjump \in (0, \infty)$ is an arbitrary cutoff constant. Under this decomposition, the different quantities introduced in the main text to quantify the size of noise are given by:
%\begin{itemize}
%\setlength{\itemindent}{.3in}
% \item $\displaystyle\contnoise^2 = \exof*{\dnorm*{\diffmat \brownof{1}}^2} = \norm{\diffmat}_F^2 = \tr(\diffmat^{\top} \diffmat)$;
% \item $\displaystyle\smalljumps^2 = \exof*{ \dnorm*{ \int_{\cjumpinf} \jump \cpoisof{1}{d\jump} }^2 } = \int_{\cjumpinf} \dnorm{\jump}^2 \levymes(d\jump)$;
% \item $\displaystyle\bigjumps^\momexp = \int_{\cjumpsup} \dnorm{\jump}^\momexp \levymes(d\jump)$.
%\end{itemize}
%----------------------------------------------------------------------
%% APP: CONTINUOUS
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
In this section, we study the convergence properties of the \emph{Lévy mirror flow} \eqref{eq:LMF}, defined as the system of stochastic differential equations given by
\begin{equation}
\tag{LMF}
\begin{aligned}
d\dstateof{\time}
&= -\nabla\obj(\stateof{\time}) \dd\time
+ \dd\levyof{\time}
\notag\\
\stateof{\time}
&= \mirror(\learn(\time)\dstateof{\time})
\end{aligned}
\end{equation}
where $\levy$ is a $\dpoints$-valued centered Lévy process with $\exof*{\dnorm{\levyof{\time}}^\momexp} < \infty$ for some $\momexp \in (1, 2]$, and $\learn \from [0, \infty) \to (0, \infty)$ is a nonincreasing differentiable learning rate function.
To simplify notations, throughout this section we identify the dual space $\dpoints$ with $\R^n$, which allows us to define Lévy processes, Brownian motions and Poisson random measures in the standard way presented in \cref{app:levy}. In particular, $\levy$ admits the Lévy-Itô decomposition
\begin{equation}
\levyof{\time} = - \parens*{\int_{\cjumpsup} \!\! \jump \levymes(d\jump)} \time + \diffmat \brownof{\time} + \int_0^\time \int_{\cjumpinf} \!\! \jump \cpoisfulalt + \int_0^\time \int_{\cjumpsup} \!\! \jump \poisfulalt
\end{equation}
where $\cjump \in (0, \infty)$ is an arbitrary cutoff constant (\cf \cref{subsec:LevyIto}). Under this decomposition, the quantities introduced in the main text to quantify the noise intensity are given by:
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item \define{Diffusion intensity:}
\begin{equation}
\contbound^2 = \norm{\diffmat}_F^2 = \tr[\diffmat^\top \diffmat].
\end{equation}
\item \define{Short jump intensity:}
\begin{equation}
\shortbound^2 = \int_{\norm{\jump}<\cjump} \norm{\jump}^2 \levymes(d\jump).
\end{equation}
\item \define{Long jump intensity:}
\begin{equation}
\longbound^\momexp = \int_{\norm{\jump} \geq \cjump} \norm{\jump}^\momexp \levymes(d\jump).
\end{equation}
\end{enumerate}
Moreover, to highlight that $\contbound^2$ and $\shortbound^2$ correspond to the components of $\levy$ that admits moments of all orders, whereas $\longbound^\momexp$ captures the intensity of the component that may exhibit infinite variance, we also set
\begin{equation}
\tamebound^2 = \contbound^2 + \shortbound^2 \quad \text{and} \quad \heavybound^\momexp = \longbound^\momexp
\end{equation}
for the tame and heavy part of the noise respectively.
\subsection{Existence and uniqueness of solutions}
We begin this section by addressing an important aspect that was largely left implicit in the main text, namely the well-posedness of the system \eqref{eq:LMF}. Specifically, under which conditions \eqref{eq:LMF} admits a unique global solution for every initial condition.
In order to maintain a sufficiently general framework---one under which the various assumptions considered in the main text can coexist---we study the existence and uniqueness of solutions to \eqref{eq:LMF} under the following conditions on the objective function $\obj$:
\begin{assumption}
$\nabla \obj \circ \mirror$ is locally Lipschitz on $\dpoints$: for every $r > 0$, there exist a nonnegative constant $L_r \geq 0$ such that
\begin{equation}
\dnorm{(\nabla \obj \circ \mirror)(\dpoint) - (\nabla \obj \circ \mirror)(\dpoint')} \leq L_r \dnorm{\dpoint - \dpoint'} \quad \text{for all } \dpoint, \dpoint' \in \ball_r(0).
\tag{$\mathcal{H}_1$}
\label{assumption:H1}
\end{equation}
\end{assumption}
\begin{assumption}
$\nabla \obj \circ \mirror$ satisfies a global growth condition on $\dpoints$: there exists $M > 0$ such that
\begin{equation}
\dnorm{(\nabla \obj \circ \mirror)(\dpoint)} \leq M(1 + \dnorm{\dpoint}) \quad \text{for all } \dpoint \in \dpoints.
\tag{$\mathcal{H}_2$}
\label{assumption:H2}
\end{equation}
\end{assumption}
Assumption \eqref{assumption:H1} guarantees local existence and uniqueness of solutions, while \eqref{assumption:H2} prevents finite-time explosion and thus ensures global well-posedness. Together, these conditions yield the following result:
\begin{proposition}[Existence and uniqueness of solutions]
Under assumptions \eqref{assumption:H1} and \eqref{assumption:H2}, the stochastic differential equation \eqref{eq:LMF} admits a unique càdlàg strong solution $\state$ staying in $\points$ at all times. Moreover, if $\hreg$ is steep, then $\stateof{\time} \in \intpoints$ \as whenever $\stateof{0} \in \intpoints$.
\end{proposition}
\begin{proof}
Replacing $\stateof{\time}$ by $\mirror(\learn(\time)\dstateof{\time})$, the system \eqref{eq:LMF} can be written as the stochastic differential equation
\begin{equation}
d\dstateof{\time} = - (\nabla \obj \circ \mirror)(\learn(\time) \dstateof{\time}) d\time + d\levyof{\time}
\end{equation}
evolving in $\dpoints$. Let $r > 0$ and $\dpoint, \dpoint' \in \ball_r(0)$. For every $\time \geq 0$, we have $\learn(\time) \dpoint, \learn(\time) \dpoint' \in \ball_{\learn(\time) r}(0) \subseteq \ball_{\learn(0) r}(0)$ thanks to the fact that $\learn$ is a nonincreasing positive function. In particular, the local Lipschitzness \eqref{assumption:H1} of $\nabla \obj \circ \mirror$ yields
\begin{equation}
\dnorm{(\nabla \obj \circ\mirror)(\learn(\time) \dpoint) - (\nabla \obj \circ \mirror)(\learn(\time) \dpoint')} \leq L_{\learn(0) r} \dnorm{\learn(\time)(\dpoint - \dpoint')} \leq \learn(0) L_{\learn(0) r} \dnorm{\dpoint - \dpoint'}
\label{eq:local_lipschitz_proof}
\end{equation}
for all $\dpoint, \dpoint' \in \ball_r(0)$ and every $r > 0$. Moreover, we also get
\begin{equation}
\dnorm{(\nabla \obj \circ \mirror)(\learn(\time) \dpoint)} \leq M(1 + \dnorm{\learn(\time) \dpoint}) \leq M(1+\learn(0) \dnorm{\dpoint}) \leq \max(1, \learn(0)) M (1+ \dnorm{\dpoint})
\label{eq:global_growth_proof}
\end{equation}
by the global growth condition \eqref{assumption:H2}. Since the upper bounds of both \cref{eq:local_lipschitz_proof,eq:global_growth_proof} are uniform in $\time$, we can therefore use a standard existence theorem for solutions to SDEs driven by semimartingales (see \eg \citep[Theorem 1.3.4]{Mao91}) to obtain that \eqref{eq:LMF} admits a unique strong solution $\dstate$ having càdlàg paths.\footnote{To be rigorous, the argument above only establishes existence and uniqueness for the SDE $dY'(\time) = - (\nabla \obj \circ \mirror)(\learn(\time-) Y'(\time-)) d\time + d\levyof{\time}$, where the left limit of the drift is taken. This is due to the fact that stochastic processes are required to be predictable---and therefore almost left-continuous---for being used as integrands, see \cref{subsec:stochastic_integration}. However, in our case, $Y'$ has càdlàg paths and thus it can only have at most countably many jumps. Consequently, $Y'(\time-) = Y'(\time)$ Lebesgue almost everywhere. Since $\learn$ and $\nabla \obj \circ \mirror$ are continuous, it follows that $\int_0^\time (\nabla \obj \circ \mirror)(\learn(\timealt-)Y'(\timealt-)) d\timealt = \int_0^\time (\nabla \obj \circ \mirror)(\learn(\timealt)Y'(\timealt)) d\timealt$, which implies that $Y'$ is in fact also the unique solution to \eqref{eq:LMF}.}
Since $\stateof{\time} = \mirror(\learn(\time) \dstateof{\time}) \in \points$ by definition of \eqref{eq:LMF} and of the mirror map $\mirror$, it follows that the primal trajectories are also càdlàg and remain in $\points$ at any times. Furthermore, if $\hreg$ is steep, then $\im(Q) = \intpoints$ (\cf \cref{prop:mirror}), which immediately yields the second claim.
\end{proof}
\begin{remark}
It is worth noting that, by standard arguments, \eqref{assumption:H2} follows directly from \eqref{assumption:H1} if the local Lipschitz condition on $\nabla \obj \circ \mirror$ is strengthened to a \emph{global} Lipschitz condition. This specific kind of global condition was already employed in the analysis of \emph{Mirror Langevin dynamics} for sampling problems, \cf \citet{ZPFP20}.
\end{remark}
In the next two lemmas, we discuss standard settings under which the objective function $\obj$ satisfies the above assumptions ensuring existence and uniqueness of solution.
\begin{lemma}
If $\nabla \obj$ is globally $L$-Lipschitz continuous, then both \eqref{assumption:H1} and \eqref{assumption:H2} hold true.
\label{lemma:globally_Lips}
\end{lemma}
\begin{proof}
The mirror map $\mirror$ is $(1/\hcst)$-Lipschitz continuous since $\hreg$ is $\hcst$-strongly convex (\cf \cref{prop:mirror}), and thus
\begin{equation}
\dnorm{(\nabla \obj \circ \mirror)(\dpoint) - (\nabla \obj \circ \mirror)(\dpoint')} \leq L \norm{\mirror(\dpoint) - \mirror(\dpoint')} \leq \frac{L}{\hcst} \dnorm{\dpoint - \dpoint'}.
\end{equation}
This leads to $\nabla \obj \circ \mirror$ being globally Lipschitz continuous on $\dpoints$, which directly implies \eqref{assumption:H1} and \eqref{assumption:H2}.
\end{proof}
\begin{remark}
The globally Lipschitz setting of \cref{lemma:globally_Lips} coincide with the framework commonly adopted in prior analyses of stochastic mirror flows---and, more specifically, of the stochastic mirror flow with diffusive noise \eqref{eq:SMF}; see for instance \citep{RB13,MerSta18,XWG18}. By contrast, relaxing this requirement to \eqref{assumption:H1} and \eqref{assumption:H2} allows the use of objective functions whose gradients may blow up at the boundary of the state $\points$, a feature that is essential when $\hreg$ is steep and $\obj$ is strongly convex relative to $\hreg$.
\end{remark}
\begin{lemma}
If $\hreg$ is steep and $\nabla \obj$ is locally Lipschitz on $\intpoints$, then $\obj$ satisfies \eqref{assumption:H1}.
\label{lemma:local_lips}
\end{lemma}
\begin{proof}
Fix $\dpoint_0 \in \dpoints$. Since $\hreg$ is steep, we know that $\point_0 \defeq \mirror(\dpoint_0)$ belongs to $\intpoints$. In particular, there exists a relatively compact subset $\cpt \subset \intpoints$ such that $\point_0 \in \cpt$. By the local Lipschitzness of $\nabla \obj$, it follows that it is Lipschitz continuous on $\cpt$ with some finite constant $L(\cpt) \geq 0$. Now, due to $\mirror$ being continuous on $\dpoints$, we can find a neighborhood $\nhd$ of $\dpoint_0$ such that $\mirror(\dpoint) \in \cpt$ whenever $\dpoint \in \nhd$. We then proceed to show that $\nabla \obj \circ \mirror$ is Lipschitz continuous on $\nhd$. To do so, let $\dpoint, \dpoint' \in \nhd$. By construction of $\nhd$, we have $\mirror(\dpoint), \mirror(\dpoint') \in \cpt$, and thus
\begin{equation}
\dnorm{(\nabla f \circ \mirror)(\dpoint) - (\nabla f \circ \mirror)(\dpoint')} \leq L(\cpt) \norm{\mirror(\dpoint) - \mirror(\dpoint')} \leq \frac{L(\cpt)}{\hcst} \dnorm{\dpoint - \dpoint'},
\end{equation}
where we have used that $\mirror$ is $(1/\hcst)$-Lipschitz continuous on $\dpoints$ (see \cref{prop:mirror}). It follows that, for every $\dpoint_0 \in \dpoints$, $\nabla \obj \circ \mirror$ is Lipschitz continuous on a neighborhood of $\dpoint_0$, and thus $\nabla \obj \circ \mirror$ is locally Lipschitz on $\dpoints$ as needed.
\end{proof}
\begin{remark}
If $\hreg$ is not steep, the conclusion of \cref{lemma:local_lips} remains valid provided that $\nabla \obj$ is locally Lipschitz on $\points_\hreg \equiv \dom \subd \hreg$.
\end{remark}
Beyond providing existence and uniqueness of solutions to \eqref{eq:LMF}, assumptions \eqref{assumption:H1} and \eqref{assumption:H2} also allows us to obtain the finiteness of the moments of $\dstateof{\time}$ up to order $\momexp$.
\begin{lemma}[Moments of the dual process]
\label{lemma:moments_dual}
Under assumptions \eqref{assumption:H1} and \eqref{assumption:H2}, the solution $\dstateof{\time}$ to \eqref{eq:LMF} satisfies $\exof{\sup_{\timealt \leq \time} \dnorm{\dstateof{\timealt}}^\momexp} < \infty$ for every $\time \geq 0$.
\end{lemma}
\begin{proof}
Fix an horizon $\horizon > 0$ and let $0 \leq \time \leq \horizon$. Applying the standard inequality $(a + b + c)^\momexp \leq 3^{\momexp-1} (a^\momexp + b^\momexp + c^\momexp)$ to the definition of \eqref{eq:LMF}, we obtain
\begin{align}
\dnorm{\dstateof{\time}}^\momexp
&\leq 3^{\momexp-1} \bracks*{ \dnorm{\dstateof{0}}^\momexp + \dnorm*{ \int_0^\time (\nabla \obj \circ \mirror)(\learn(\timealt) \dstateof{\timealt}) d\timealt}^\momexp + \dnorm{\levyof{\time}}^\momexp }\notag\\
&\leq 3^{\momexp-1} \bracks*{ \dnorm{\dstateof{0}}^\momexp + \time^{\momexp-1} \int_0^\time \dnorm{ (\nabla \obj \circ \mirror)(\learn(\timealt) \dstateof{\timealt}) }^\momexp d\timealt + \sup_{r \leq \horizon} \dnorm{\levyof{r}}^\momexp }\notag\\
&\leq 3^{\momexp-1} \bracks*{ \dnorm{\dstateof{0}}^\momexp + \sup_{r \leq \horizon} \dnorm{\levyof{r}}^\momexp + \time^{\momexp-1} \int_0^\time M^\momexp (1 + \dnorm{\learn(\timealt) \dstateof{\timealt)}})^\momexp d\timealt }\notag\\
&\leq 3^{\momexp-1} \bracks*{ \dnorm{\dstateof{0}}^\momexp + \sup_{r \leq \horizon} \dnorm{\levyof{r}}^\momexp +
\frac{\big( 2M\time \big)^{\momexp}}{2\time} + \frac{\big( 2M\learn(0)\time \big)^{\momexp}}{2\time} \int_0^\time \dnorm{\dstateof{\timealt}}^\momexp d\timealt}\notag\\
&\leq 3^{\momexp-1} \bracks*{ \dnorm{\dstateof{0}}^\momexp + \sup_{r \leq \horizon} \dnorm{\levyof{r}}^\momexp +
% 2^{\momexp-1} M^\momexp \horizon^{\momexp-1} + 2^{\momexp-1} M^\momexp \learn(0)^\momexp \horizon^{\momexp-1} \int_0^\time \dnorm{\dstateof{\timealt}}^\momexp d\timealt}
\frac{\big( 2M\horizon \big)^{\momexp}}{2\horizon} + \frac{\big( 2M\learn(0)\horizon \big)^{\momexp}}{2\horizon} \int_0^\time \dnorm{\dstateof{\timealt}}^\momexp d\timealt}
\end{align}
where the second line comes from Hölder's inequality and the third one from the global growth condition \eqref{assumption:H2}.
For notational convenience, let
\begin{equation}
Z(\horizon) = 3^{\momexp-1} \braces*{ \dnorm{\dstateof{0}}^\momexp + \sup_{r \leq \horizon} \dnorm{\levyof{r}}^\momexp + \frac{\big( 2M\horizon \big)^{\momexp}}{2\horizon} } \quad \text{and} \quad C(\horizon) = \frac{\big( 6M\learn(0)\horizon \big)^{\momexp}}{6\horizon}.
\end{equation}
Taking the supremum over $\timealt \leq \time$ therefore yields
\begin{equation}
\sup_{\timealt \leq \time} \dnorm{\dstateof{\timealt}}^\momexp \leq Z(\horizon) + C(\horizon) \int_0^\time \sup_{u \leq \timealt} \dnorm{\dstateof{u}}^\momexp d\timealt.
\label{eq:supremum_before_gronwall}
\end{equation}
Given the particular form of the above inequality, one would be tempted to take the expectation and proceed by applying Grönwall's lemma. However, at this stage we do not know if the left-hand side of \cref{eq:supremum_before_gronwall} admits a finite expectation since this is the result we aim to prove, hence rendering the argument slightly more intricate.
To proceed, we instead need to localize our bound using the stopping times $\hit_n = \inf\setdef{\time \geq 0}{\dnorm{\dstateof{\time}} \geq n}$.
We first show that $\exof*{\sup_{\timealt \leq \hit_n \wedge \time} \dnorm{\dstateof{\timealt}}^\momexp}$ is finite. Indeed, note that if $\timealt < \hit_n \wedge \time$ then $\dnorm{\dstateof{\timealt}} < n$ by definition. On the other hand, for $\timealt = \hit_n \wedge \time$,
\begin{align}
\dnorm{ \dstateof{\hit_n \wedge \time}}
&= \dnorm{ \dstateof{\hit_n \wedge \time -} + \jumpof \dstateof{\hit_n \wedge \time} }
= \dnorm{ \dstateof{\hit_n \wedge \time-} + \jumpof \levyof{\hit_n \wedge \time} }
\notag\\
&\leq \dnorm{ \dstateof{\hit_n \wedge \time-} } + \dnorm{\levyof{\hit_n \wedge \time}} + \dnorm{\levyof{\hit_n \wedge \time -}}
\notag\\
&\leq n + 2 \sup_{r \leq \horizon} \dnorm{\levyof{r}}.
\label{eq:norm_of_dstate_localized}
\end{align}
But $\levyof{r}$ is a martingale with finite $\momexp$-th moment, so Doob's $L^\momexp$-maximal inequality (see \eg \citet{KS98}) can be applied to the positive submartingale $\dnorm{\levyof{r}}$ to obtain
\begin{equation}
\exof{\sup_{r \leq \horizon} \dnorm{\levyof{r}}^\momexp} \leq \parens*{\dfrac{\momexp}{\momexp-1}}^\momexp \exof{\dnorm{\levyof{\horizon}}^\momexp} < \infty.
\end{equation}
Combined with \cref{eq:norm_of_dstate_localized}, this yields
\begin{equation}
\exof{\sup_{\timealt \leq \hit_n \wedge \time} \dnorm{\dstateof{\timealt}}^\momexp} \leq 2^{\momexp-1} \braces*{ n^\momexp + 2 \exof{ \sup_{r \leq \horizon} \dnorm{\levyof{r}}^\momexp } } < \infty,
\label{eq:sup_levy_p_moment}
\end{equation}
which proves that $g_n(\time) \defeq \exof{\sup_{\timealt \leq \hit_n \wedge \time} \dnorm{\dstateof{\timealt}}^\momexp}$ is a well-defined function on $[0, \horizon]$. We can therefore apply \cref{eq:supremum_before_gronwall} with $\time \leftarrow \hit_n \wedge \time$ and take the expectation on both sides, leading to
\begin{equation}
g_n(\time) \leq \exof{Z(\horizon)} + C(\horizon) \exof*{ \int_0^{\hit_n \wedge \time} \sup_{u \leq \timealt} \dnorm{\dstateof{u}}^\momexp d\timealt } \leq \exof{Z(\horizon)} + C(\horizon) \int_0^\time g_n(\timealt) d\timealt,
\end{equation}
where $\exof{Z(\horizon)}$ is finite thanks to \cref{eq:sup_levy_p_moment}. Invoking Grönwall's lemma then provides the estimate
\begin{equation}
g_n(\time) \leq \exof{Z(\horizon)} e^{C(\horizon) \time} \leq \exof{Z(\horizon)} e^{C(\horizon) \horizon}.
\end{equation}
Note that this upper bound does not depend on $n$ and that $\sup_{\timealt \leq \hit_n \wedge \time} \dnorm{\dstateof{\timealt}}^\momexp$ increases to $\sup_{\timealt \leq \time} \dnorm{\dstateof{\timealt}}^\momexp$ as $n \to \infty$, so the monotone convergence theorem yields
\begin{equation}
\exof{\sup_{\timealt \leq \time} \dnorm{\dstateof{\timealt}}^\momexp} \leq \exof{Z(\horizon)} e^{C(\horizon) \horizon} < \infty,
\end{equation}
hence concluding the proof.
\end{proof}
In particular, \cref{lemma:moments_dual} implies that the drift term of \eqref{eq:LMF} admits finite moments of order $\momexp$ even when the gradient of the objective function explodes at the boundary of $\points$.
\begin{corollary}
\label{cor:moments_drift}
Under assumptions \eqref{assumption:H1} and \eqref{assumption:H2},
\begin{equation}
\exof*{ \int_0^\time \dnorm*{\nabla \obj(\stateof{\timealt})}^\momexp d\timealt} < \infty \quad \text{for every } \time \geq 0.
\end{equation}
\end{corollary}
\begin{proof}
Thanks to the growth condition \eqref{assumption:H2}, we can write
\begin{align}
\exof*{ \int_0^\time \dnorm{\nabla \obj(\stateof{\timealt})}^\momexp d\timealt}
&= \exof*{ \int_0^\time \dnorm{ (\nabla \obj \circ \mirror)(\learn(\timealt) \dstateof{\timealt})}^\momexp d\timealt }
\notag\\
&\leq \exof*{ \int_0^\time M^\momexp(1 + \dnorm{\learn(\timealt) \dstateof{\timealt}})^\momexp d\timealt}
\notag\\
&\leq 2^{\momexp-1} M^\momexp \parens*{ 1 + \learn(0)^\momexp \exof{ \sup_{\timealt \leq \time} \dnorm{\dstateof{\timealt}}^\momexp } }
\end{align}
where the right-hand side is finite due to \cref{lemma:moments_dual}, therefore proving the claim.
\end{proof}
\subsection{Convex setting}
Inspired by the continuous-time analysis of \citet{MerSta18}, in this section we study the convergence of \eqref{eq:LMF} for convex functions by analyzing the behavior of the \emph{$\learn$-deflated} Fenchel coupling
\begin{equation}
\energy(\basealt, \dpoint; \time) = \dfrac{1}{\learn(\time)} \fench(\basealt, \learn(\time) \dpoint), \quad \basealt \in \points, \dpoint \in \dpoints
\end{equation}
along trajectories of $\eqref{eq:LMF}$.
This analysis naturally calls for Itô's formula, the stochastic counterpart of the standard chain rule of differential calculus. Unfortunately, the classical version of Itô's formula only applies to functions that are twice continuously differentiable, which is generally not the case for $\dpoint \mapsto \energy(\basealt, \dpoint; \time)$ unless $\hreg$ is steep.
Nevertheless, as we show in \cref{subsec:weakIto}, Itô's formula can be extended to functions that are merely $\lips$-smooth and convex. Since $\energy$ satisfies these conditions by \cref{lem:dFench}, we may apply \cref{thm:weakIto} to derive the following result:
\begin{lemma}[Stochastic descent lemma]
For every fixed $\basealt \in \points$, the process $\energy(\time) \equiv \energy(\basealt, \dstateof{\time}; \time)$ satisfies
\begin{align}
d\energy \leq - \dfrac{\dot\learn}{\learn^2}\parens*{\hreg(\basealt) - \hreg(\state)} d\time - \braket*{\state - \basealt}{\nabla \obj(\state)} d\time + \dfrac{\learn \tamebound^2}{2\hcst} d\time + dI + d\firstmart + d\secondmart,
\end{align}
where:
\begin{enumerate}
[\upshape(1)]
\item
$\displaystyle I(\time) = \!\int_0^\time \! \int_{\cjumpsup} \dfrac{1}{\learn} \bracks*{\fench(\basealt, \learn \dstate + \learn \jump) - \fench(\basealt, \learn \dstate)} \levymes(d\jump) d\timealt - \!\int_0^\time \! \int_{\cjumpsup} \! \braket*{\state - \basealt}{\jump} \levymes(d\jump) d\timealt$,
\item
$\displaystyle \timed{\firstmart} = \int_0^\time \braket*{\state - \basealt}{\diffmat d\brown} + \int_0^\time \int_{\cjumpinf} \dfrac{1}{\learn} \bracks*{ \fench(\basealt, \learn \dstate + \learn \jump) - \fench(\basealt, \learn \dstate) } \cpoisfulalt$,
\item $\displaystyle \timed{\secondmart} = \int_0^\time \int_{\cjumpsup} \dfrac{1}{\learn} \bracks*{\fench(\basealt, \learn \dstate + \learn \jump) - \fench(\basealt, \learn \dstate)} \cpoisfulalt$.
\end{enumerate}
\label{lemma:dV_1}
\end{lemma}
\begin{proof}
First, let us define the auxiliary stochastic process $Z(\time) = \learn(\time) \dstateof{\time}$. Due to the integration by parts theorem for semi-martingales and thanks to the fact that $\learn$ is of finite variation, we get (\cf \citep[Lemma 20.10]{Kal21})
\begin{align}
dZ(\time) &= \dstateof{\time-} d\learn(\time) + \learn(\time-) d\dstateof{\time}
\notag\\
&= \dot\learn(\time) \dstateof{\time-} d\time + \learn(\time) d\dstateof{\time}
\notag\\
&= \dot\learn \dstate d\time
- \learn \bracks*{ \nabla \obj(\state) + \int_{\cjumpsup} \jump \levymes(d\jump) } d\time
\notag\\
&+ \learn \diffmat d\brown
+ \learn \int_{\cjumpinf} \jump \cpoisful
+ \learn \int_{\cjumpsup} \jump \poisful,
\end{align}
where we have suppressed the dependence on time in the last equality to lighten the notations. Note that $Z(\time)$ is evidently a Lévy-type process thanks to \cref{lemma:moments_dual,cor:moments_drift}.
Next, we consider the function $\psi(z) = \fench(\basealt, z)$, which is obviously convex as the sum of convex functions, and which satisfies $\nabla \psi(z) = \nabla \hconj(z) - \basealt = \mirror(z) - \base$ and thus is also $(1/\hcst)$-smooth by \cref{prop:mirror}.
Consequently, we can again use the integration by parts formula in tandem with the weak Itô's formula of \cref{thm:weakIto} to obtain
\begin{align}
d\energy &= -\dfrac{\dot\learn}{\learn^2} \psi(Z) d\time + \dfrac{1}{\learn} d\psi(Z)
\notag\\
&\leq -\dfrac{\dot\learn}{\learn^2} \psi(Z) d\time + \frac{1}{\learn} \braket*{\nabla \psi(Z)}{ dZ^c} + \dfrac{\learn\tamebound^2}{2\hcst} d\time
\notag\\
+& \int_{\cjumpinf} \! \dfrac{1}{\learn} \bracks*{ \psi(Z + \learn \jump) - \psi(Z) } \cpoisful + \int_{\cjumpsup} \! \dfrac{1}{\learn} \bracks*{ \psi(Z + \learn \jump) - \psi(Z) } \poisful,
\label{eq:dV1}
\end{align}
where $dZ^c$ denotes the continuous part of $Z$. Note that by definition of \eqref{eq:LMF}, we have $\nabla \psi(Z) = \mirror(Z) - \basealt = \state - \base$. Moreover, by optimality of $\state$, we also have $\hconj(Z) = \braket{\state}{Z} - \hreg(\state)$ and thus
\begin{equation}
- \dfrac{\dot\learn}{\learn^2} \psi(Z) = - \dfrac{\dot\learn}{\learn^2} \fench(\basealt, Z) = - \dfrac{\dot\learn}{\learn^2} \parens*{ \hreg(\basealt) - \hreg(\state) + \braket*{\state - \basealt}{Z} }.
\end{equation}
This leads to
\begin{align}
- \dfrac{\dot\learn}{\learn^2} \psi(Z) d\time + \frac{1}{\learn} \braket*{\nabla \psi(Z)}{dZ^c}
&= \bracks*{ - \dfrac{\dot\learn}{\learn^2} \psi(Z) + \dfrac{\dot\learn}{\learn} \braket*{\state - \basealt}{\dstate} } d\time
\notag\\
- & \braket*{\state - \basealt}{\nabla \obj(\state)} d\time - \int_{\cjumpsup} \braket*{\state - \basealt}{\jump}\levymes(d\jump)d\time
\notag\\
+ & \braket*{\state - \basealt}{\diffmat d\brown}
\notag\\
&= -\dfrac{\dot\learn}{\learn^2}\parens*{ \hreg(\basealt) - \hreg(\state) } d\time
\notag\\
- & \braket*{\state - \basealt}{\nabla \obj(\state)} d\time - \int_{\cjumpsup} \braket*{\state - \basealt}{\jump}\levymes(d\jump)d\time
\notag\\
+ & \braket*{\state - \basealt}{\diffmat d\brown},
\end{align}
which, when combined with \cref{eq:dV1} and after rearranging the terms, yields exactly the upper bound on $d\energy$ stated in the lemma.
\end{proof}
\begin{remark}
When $\levy$ is continuous---that is, when it reduces to a Brownian motion---both $I(\time)$ and $\timed{\secondmart}$ are identically zero, and $\xi(\time)$ simplifies to $\int_0^\time \braket*{\state - \basealt}{\diffmat d\brown}$. In this case, the analysis is considerably simpler, since the noise process affecting $\energy$ is just the scalar product between $\stateof{\time} - \basealt$ (which is always bounded) and the Brownian motion $\diffmat \brownof{\time}$ (which has exponentially light tails). Moreover, it then also recovers the analogous result proved for \eqref{eq:SMF} by \citet{MerSta18} under diffusion-like random perturbations.
\end{remark}
In the next three lemmas, we analyze the long-run behavior of the stochastic processes $I$, $\firstmart$ and $\secondmart$. We begin by showing that $I$ can be upper bounded in terms of the heavy noise intensity $\heavybound^\momexp$ and a specific integral involving the learning rate.
\begin{lemma}[Estimation of $I$]
$\displaystyle I(\time) \leq \dfrac{\diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} \int_0^\time \learn(\timealt)^{\momexp-1} d\timealt$, where $\diamX \defeq \max_{\point,\pointalt} \norm{\pointalt - \point}$ is the diameter of $\points$.
\label{lemma:I_upper_bound}
\end{lemma}
\begin{proof}
To bound this quantity, we readily obtain from \cref{lemma:diffF_upperbound}
\begin{align}
\int_{\cjumpsup}
&\dfrac{1}{\learn} \bracks*{ \fench(\basealt, Z + \learn \jump) - \fench(\basealt, Z) } \levymes(d\jump)
\notag\\
&\leq \int_{\cjumpsup} \dfrac{1}{\learn} \bracks*{ \braket*{Z + \learn \jump - Z}{\mirror(Z) - \basealt} + \dfrac{\diamX^{2-\momexp}}{\hcst^{\momexp-1}} \dnorm{Z + \learn \jump - Z}^\momexp } \levymes(d\jump)
\notag\\
&=\int_{\cjumpsup} \braket*{\state - \basealt}{\jump} \levymes(d\jump) + \dfrac{\diamX^{2-\momexp}}{\hcst^{\momexp-1}} \learn^{\momexp-1} \int_{\cjumpsup} \dnorm{\jump}^\momexp \levymes(d\jump),
\end{align}
where the first term compensates the second term of $I$, thus finishing the proof.
\end{proof}
Next, we use the stochastic integration tools introduced in \cref{subsec:stochastic_integration} to show that $\firstmart$ is a square-integrable martingale, and to characterize its asymptotic long-run behavior via the law of iterated logarithm.
\begin{lemma}[Asymptotic growth of $\firstmart$]
\label{lem:firstmart-bound}
$\firstmart$ is a square-integrable martingale satisfying $\timed{\firstmart} = \bigoh\parens*{\tamebound\diamX \sqrt{\time \log\log \time}}$ with probability $1$.
\label{lemma:firstmart-bound}
\end{lemma}
\begin{proof}
First, note that
\begin{equation}
\norm{\diffmat^{\top} (\state - \basealt)}^2 \leq \norm{\state - \basealt)}^2 \norm{\diffmat}_F^2 \leq \diamX^2 \contnoise^2
\end{equation}
so $\int_0^\time \braket*{\state - \basealt}{\diffmat d\brown}$ is a square-integrable martingale.
On the other hand, using \cref{lemma:F_Lipschitz}, we also get
\begin{align}
&\int_0^\time \int_{\cjumpinf} \dfrac{1}{\learn^2} \abs*{ \fench(\basealt, Z + \learn\jump) - \fench(\basealt, Z) } \levymes(d\jump) d\timealt
\\
\leq \,\, & \diamX^2 \int_0^\time \int_{\cjumpinf} \dfrac{1}{\learn^2} \dnorm{Z + \learn\jump - Z}^2 \levymes(d\jump)d\timealt\\
= \,\, & \diamX^2 \smalljumps^2 \time,
\end{align}
where $\smalljumps^2$ is finite since $\levymes$ is a Lévy measure. As a result,
\begin{equation}
G(\time, \jump) \defeq \dfrac{1}{\learn} \bracks*{\fench(\basealt, Z + \learn\jump) - \fench(\basealt, Z)} \in \Hlevyof{2}{T}{\braces{\cjumpinf}}{\R} \quad \text{for every } T \geq 0,
\end{equation}
and thus $\int_0^\time \int_{\cjumpinf} G(\timealt, \jump) \cpoisfulalt$ is also a square-integrable martingale. It then immediately follows that $\firstmart$ is itself a square-integrable martingale.
To obtain the asymptotic growth of $\firstmart$, we use the law of iterated logarithm for square-integrable martingales (\cf \cref{lemma:LIL}). Using standard stochastic calculus rules and the previous computations, we obtain that the predictable quadratic variation of $\firstmart$ satisfies
\begin{align}
d\meyerof{\firstmart} &= \norm{\diffmat^{\top} (\state - \basealt)}^2 d\time + \int_{\cjumpinf} \dfrac{1}{\learn^2} \abs{ \fench(\basealt, Z + \learn \jump) - \fench(\basealt, Z) }^2 \levymes(d\jump) d\time\\
&\leq \tamebound^2 \diamX^2 d\time.
\label{eq:meyer_of_xi1}
\end{align}
Moreover, the jumps of $\firstmart$ are bounded as $\abs{\jumpof \timed{\firstmart}} \leq c\diamX$ \as thanks to \cref{lemma:F_Lipschitz} and the fact that $\dnorm{\jump} \leq c$.
Assume first that $\lim_{\time \to \infty} \meyerof{\xi}(\time) = \infty$. Then, the law of iterated logarithm of \cref{lemma:LIL} applies to $\firstmart$ and yields
\begin{align}
\limsup_{\time \to \infty} \dfrac{\abs{\firstmart}}{\tamebound\diamX \sqrt{\time\log\log\time}}
&= \limsup_{\time\to\infty} \dfrac{\abs{\firstmart}}{\sqrt{ 2\meyerof{\firstmart} \log\log \meyerof{\firstmart}}} \dfrac{\sqrt{2\meyerof{\firstmart} \log \log \meyerof{\firstmart}}}{\tamebound\diamX \sqrt{\time\log\log\time}}
\notag\\
&\leq \limsup_{\time\to\infty} \dfrac{\abs{\firstmart}}{\sqrt{ 2\meyerof{\firstmart} \log\log \meyerof{\firstmart}}} \dfrac{2\tamebound\diamX \sqrt{\time\log\log\time}}{\tamebound\diamX \sqrt{\time\log\log\time}}
\leq 2,
\end{align}
where the second-to-last inequality holds thanks to \cref{eq:meyer_of_xi1} and the fact that $\time$ can be taken as large as needed. It follows that $\timed{\firstmart} = \bigoh\parens*{ \tamebound\diamX \sqrt{\time\log\log\time} }$ on the event $\braces{\lim_{\time \to \infty} \meyerof{\firstmart}(\time) = \infty}$.
On the other hand, when $\lim_{\time \to \infty} \meyerof{\firstmart}(\time) < \infty$, \citep[Proposition 1]{Lep78} implies that $\timed{\firstmart}$ converges to an almost-surely finite random variable. For this reason, it holds that $\timed{\firstmart} = o\parens*{ \tamebound\diamX \sqrt{\time\log\log\time} }$ on the event $\braces{\lim_{\time \to \infty} \meyerof{\firstmart}(\time) < \infty}$, and thus $\timed{\firstmart} = \bigoh\parens*{ \tamebound\diamX \sqrt{\time\log\log\time} }$ \as, which concludes the proof.
\end{proof}
Finally, we establish that $\secondmart$ is also a martingale using standard stochastic integration arguments. However, $\secondmart$ is not square-integrable in general, and we therefore rely on a refined version of the law of large numbers for discontinuous martingales to control its growth rate.
\begin{lemma}[Asymptotic growth of $\secondmart$]
\label{lem:secondmart-bound}
$\secondmart$ is a martingale satisfying $\timed{\secondmart} = \baroh\parens*{\time^{1/\momexp}}$ with probability $1$.\footnote{Recall that $g(t) = \baroh(t^\alpha)$ means that $g(t) = \bigoh\parens{\time^{\alpha + o(1)}}$, that is, $g(t) = \bigoh(l(t) t^\alpha)$ where $l$ is a function of subpolynomial growth such that $l(t)/t^\delta \to 0$ for all $\delta > 0$.}
\label{lemma:secondmart-bound}
\end{lemma}
\begin{proof}
Similarly to the proof of \cref{lemma:firstmart-bound}, we have
\begin{equation}
\int_0^\time \int_{\cjumpsup} \dfrac{1}{\learn} \abs*{ \fench(\basealt, Z + \learn \jump) - \fench(\base, Z) } \levymes(d\jump) \leq \diamX \parens*{\int_{\cjumpsup} \dnorm{\jump} \levymes(d\jump)} \time,
\end{equation}
where $\int_{\cjumpsup} \dnorm{\jump} \levymes(d\jump)$ is finite since $\levy$ is an integrable process. This means that
\begin{equation}
H(\time, \jump) \defeq \frac{1}{\learn} \bracks*{ \fench(\basealt, Z + \learn \jump) - \fench(\basealt, Z)} \in \Hlevyof{1}{T}{\braces{\cjumpsup}}{\R} \quad \text{for all } T \geq 0,
\end{equation} which immediately leads to $\secondmart$ being a martingale. Letting $A(\time) = \time^{\frac{1}{\momexp} + \precval}$ for some $\precval > 0$, we also have that
\begin{align}
\phi(\time) \defeq \int_{\cjumpsup} \dfrac{\abs*{H(\time, \jump)}^2 \wedge \abs*{H(\time, \jump)}^\momexp}{(1 + A(\time))^\momexp} \levymes(d\jump) &\leq \dfrac{\diamX^\momexp}{\time^{1 + \precval \momexp}} \int_{\cjumpsup} \dnorm{\jump}^\momexp \levymes(d\jump)\\
&= \dfrac{\diamX^\momexp \heavybound^\momexp}{\time^{1 + \precval \momexp}}
\end{align}
for $\time$ big enough. It follows that $\phi$ is almost-surely integrable on $[0, \infty)$, and thus $\timed{\secondmart}/\time^{\frac{1}{\momexp} + \precval} \to 0$ \as for every $\precval > 0$ by the strong law of large number (\cf \cref{lemma:SLLN}). It follows that $\timed{\secondmart} = \baroh\parens*{\time^{1/\momexp}}$, hence finishing the proof.
\end{proof}
With the preparatory estimates on the drift term $I$ and the martingales $\firstmart$ and $\secondmart$ in hand, we are now in a position to prove \cref{prop:energy-Ito} from the main text (restated below for convenience):
\ENERGY*
\begin{proof}[Proof of \cref{prop:energy-Ito}]
The result follows directly from \cref{lemma:dV_1} by integrating over the interval $[0, \time]$ and invoking the bound on $I$ established in \cref{lemma:I_upper_bound}. The martingale properties of $\firstmart$ and $\secondmart$---the former being square-integrable and the latter integrable---is then a consequence of \cref{lemma:firstmart-bound,lemma:secondmart-bound}.
\end{proof}
On the other hand, the previous result also allows us to derive the following theorem, which will serve as a baseline for many of the convergence rates established in the sequel.
\begin{theorem}
For every fixed $\basealt \in \points$, \eqref{eq:LMF} enjoys the bounds
\begin{align}
\exof*{\int_0^\time \braket*{\stateof{\timealt} - \basealt}{\nabla \obj(\stateof{\timealt})} d\timealt} \leq \time \timed{\cvxrate_0}
\label{eq:L2_bound_convex}
\end{align}
and, with probability one,
\begin{align}
\int_0^\time \braket*{\stateof{\timealt} - \basealt}{\nabla \obj(\stateof{\timealt})} d\timealt \leq \time\timed{\cvxrate_0} + \baroh\parens*{\time^{1/\momexp}} + \bigoh\parens*{\tamebound\diamX \sqrt{\time \log\log \time}},
\label{eq:almost_sure_bound_convex}
\end{align}
where
\begin{align}
\timed{\cvxrate_{0}}
= \frac{\braket*{\stateinit - \basealt}{\dstateinit}}{\time}
+ \frac{\hrange}{\time\timed{\learn}}
+ \frac{\tamebound^{2}}{2\hstr\time}
\int_{\tstart}^{\time} \eval{\learn}{\timealt} \dd\timealt
+ \frac{\diamX^{2-\momexp}\heavybound^{\momexp}}{\hstr^{\momexp-1}\time}
\int_{\tstart}^{\time} \eval{\learn}{\timealt}^{\momexp-1} \dd\timealt
\end{align}
and $\hrange \defeq \max\hreg - \min\hreg$.
% For every fixed $\basealt \in \points$,
% \begin{align}
% \exof*{\int_0^\time \braket*{\stateof{\timealt} - \basealt}{\nabla \obj(\stateof{\timealt})} d\timealt} \leq \braket*{\stateinit - \basealt}{\dstateinit} + \dfrac{\diamh}{\learn(\time)} + \dfrac{\tamebound^2}{2\hcst} \int_0^\time \learn(\timealt) d\timealt + \dfrac{\diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} \int_0^\time \learn(\timealt)^{\momexp-1} d\timealt
% \label{eq:L2_bound_convex}
% \end{align}
% and, for any $\precval > 0$,
% \begin{align}
% \begin{aligned}
% \int_0^\time \braket*{\stateof{\timealt} - \basealt}{\nabla \obj(\stateof{\timealt})} d\timealt &\leq \braket*{\stateinit - \basealt}{\dstateinit} + \dfrac{\diamh}{\learn(\time)} + \dfrac{\tamebound^2}{2\hcst} \int_0^\time \learn(\timealt) d\timealt + \dfrac{\diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} \int_0^\time \learn(\timealt)^{\momexp-1} d\timealt\\
% &+\bigoh\parens*{\tamebound\diamX \sqrt{\time \log\log \time}} + \baroh\parens*{\time^{1/\momexp}} \quad \as.
% \end{aligned}
% \label{eq:almost_sure_bound_convex}
% \end{align}
\label{thm:convex_convergence_continuous}
\end{theorem}
\begin{proof}
First, note that $\braket*{\stateof{\timealt-} - \basealt}{\nabla \obj(\stateof{\timealt})} = \braket*{\stateof{\timealt} - \basealt}{\nabla \obj(\stateof{\timealt})}$ Lebesgue-a.e. since the jumps of $\levy$ are countable, and hence
\begin{equation}
\int_0^\time \braket*{\stateof{\timealt-} - \basealt}{\nabla \obj(\stateof{\timealt})} d\timealt = \int_0^\time \braket*{\stateof{\timealt} - \basealt}{\nabla \obj(\stateof{\timealt})} d\timealt \quad \as.
\end{equation}
Moreover, the monotony of $\learn$ and the compactness of $\points$ allows us to write
\begin{equation}
-\int_0^\time \dfrac{\dot\learn}{\learn^2} \parens*{ \hreg(\basealt) - \hreg(\state) } d\timealt \leq \diamh \int_0^\time - \dfrac{\dot\learn}{\learn^2} d\timealt = \dfrac{\diamh}{\learn(\time)} - \dfrac{\diamh}{\learn(0)},
\end{equation}
where we recall that $\diamh = \max \hreg - \min \hreg$. On the other hand, from the definition of $\energy$ and by optimality of $\stateof{0} = \stateinit$, we also get
\begin{align}
\energy(0)
= \dfrac{1}{\learn(0)} \fench(\basealt, \learn(0) \dstateinit)
&= \dfrac{1}{\learn(0)}\bracks*{ \hreg(\basealt) - \hreg(\stateinit) + \braket*{\stateinit - \basealt}{\learn(0)\dstateinit} }
\notag\\
&\leq \dfrac{\diamh}{\learn(0)} + \braket*{\stateinit - \basealt}{\dstateinit}.
\end{align}
Combining these estimates with those of \cref{lemma:dV_1,lemma:I_upper_bound,lemma:firstmart-bound,lemma:secondmart-bound}, rearranging the terms, and using that $\energy(\time) \geq 0$ finally lead to the upper bound \eqref{eq:almost_sure_bound_convex}. The in-expectation inequality \eqref{eq:L2_bound_convex} also immediately follows since $\exof{\timed{\firstmart}} = \exof{\timed{\secondmart}} = 0$ in virtue of the fact that they are martingales starting from zero.
\end{proof}
\begin{remark}
Note that, compared to the definition of $\timed{\cvxrate_0}$ given in the main text, the version presented here includes the additional term $\frac{1}{\time}\braket*{\stateinit - \basealt}{\dstateinit}$. This accounts for the fact that \eqref{eq:LMF} may be initialized at any point $\stateinit \in \points_\hreg$, rather than only at the prox-center $\point_c = \argmin \hreg$ of $\points$, which was assumed in the main text for simplicity.
\end{remark}
With \cref{thm:convex_convergence_continuous} in hand, we can therefore establish \cref{thm:cvx-LMF} as a direct application of the convexity of $\obj$. The following result presents a slightly more general version of \cref{thm:cvx-LMF}, covering the case where \eqref{eq:LMF} is not necessarily initialized at the prox-center of $\points$:
\begin{theorem}[Convergence rate for convex functions]
Under \eqref{eq:LMF}, the time-averaged process $\timed{\avg\state} = (1/\time) \int_{\tstart}^{\time} \stateof{\timealt} \dd\timealt$ enjoys the bounds
\begin{align}
\exof{\obj(\timed{\avg\state})}
\leq \min\obj
+ \cvxrate_{0}(\time)
\end{align}
and, \acl{wp1},
\begin{align}
\obj(\timed{\avg\state})
\leq \min\obj
+ \cvxrate_{0}(\time)
+ \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}}
+ \bigoh\parens*{\tamebound \diamX \sqrt{\frac{\log\log \time}{\time}}}
\end{align}
In particular, if \eqref{eq:LMF} is run with learning rate $\timed{\learn} = 1/\time^{1/\momexp}$, we have
\begin{align}
\cvxrate_{0}(\time)
&= \frac{\braket*{\stateinit - \basealt}{\dstateinit}}{\time}
+ \frac{\momexp}{\momexp-1} \frac{\tamebound^{2}}{2\hstr\time^{1/\momexp}}
+ \frac{\hstr^{\momexp-1}\hrange + \momexp \diamX^{2-\momexp} \heavybound^{\momexp}}{\hstr^{\momexp-1}\time^{(\momexp-1)/\momexp}}
= \bigoh(1/\time^{(\momexp-1)/\momexp})
\eqstop
\end{align}
\label{thm:convex_convergence_continuous_explicit}
\end{theorem}
%\begin{corollary}
% If $\obj$ is convex and $\sol \in \argmin \obj$, then iterates of \eqref{eq:LMF} satisfy
% \begin{align}
% \exof*{ \obj(\avg{\state}(\time)) - \min \obj } \leq \dfrac{\braket*{\stateinit - \sol}{\dstateinit}}{\time} + \dfrac{\diamh}{\learn(\time)\time} + \dfrac{\tamebound^2}{2\time\hcst} \int_0^\time \learn(\timealt) d\timealt + \dfrac{\diamX^{2-\momexp} \heavybound^\momexp}{\time\hcst^{\momexp-1}} \int_0^\time \learn(\timealt)^{\momexp-1} d\timealt
% \end{align}
% and, for any $\precval > 0$,
% \begin{align}
% \begin{aligned}
% \obj(\avg{\state}(\time)) - \min \obj &\leq \dfrac{\braket*{\stateinit - \sol}{\dstateinit}}{\time} + \dfrac{\diamh}{\learn(\time)\time} + \dfrac{\tamebound^2}{2\time\hcst} \int_0^\time \learn(\timealt) d\timealt + \dfrac{\diamX^{2-\momexp} \heavybound^\momexp}{\time\hcst^{\momexp-1}} \int_0^\time \learn(\timealt)^{\momexp-1} d\timealt\\
% &+\bigoh\parens*{\tamebound\diamX \sqrt{\dfrac{\log\log \time}{\time}}} + \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}} \quad \as.
% \end{aligned}
% \end{align}
% In particular, running \eqref{eq:LMF} with learning rates $\learn(\time) = \time^{-1/\momexp}$ achieves
% \begin{align}
% \exof*{ \obj(\avg{\state}(\time)) - \min \obj } \leq \dfrac{\braket*{\stateinit - \sol}{\dstateinit}}{\time} + \dfrac{\momexp}{\momexp-1}\dfrac{\tamebound^2}{2\hcst} \dfrac{1}{\time^{\frac{1}{\momexp}}} + \parens*{ \diamh + \dfrac{\momexp \diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} } \dfrac{1}{\time^{\frac{\momexp-1}{\momexp}}}
% \end{align}
% and
% \begin{align}
% \begin{aligned}
% \obj(\avg{\state}(\time)) - \min \obj &\leq \dfrac{\braket*{\stateinit - \sol}{\dstateinit}}{\time} + \dfrac{\momexp}{\momexp-1}\dfrac{\tamebound^2}{2\hcst} \dfrac{1}{\time^{\frac{1}{\momexp}}} + \parens*{ \diamh + \dfrac{\momexp \diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} } \dfrac{1}{\time^{\frac{\momexp-1}{\momexp}}}\\
% &+\bigoh\parens*{\tamebound\diamX \sqrt{\dfrac{\log\log \time}{\time}}} + \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}} \quad \as.
% \end{aligned}
% \end{align}
%\label{corollary:convex_convergence_continuous_explicit}
%\end{corollary}
\begin{proof}
Since $\obj$ is convex,
\begin{equation}
\int_0^\time \braket*{\stateof{\timealt} - \sol}{\nabla \obj(\stateof{\timealt})} d\timealt \geq \int_0^\time \bracks*{ \obj(\stateof{\timealt}) - \obj(\sol) } d\timealt \geq \time \bracks*{ \obj(\avg{\state}(\time)) - \min \obj },
\end{equation}
and the results therefore follow directly from \cref{thm:convex_convergence_continuous}.
\end{proof}
%\begin{remark}
% From the expression of $\timed{\cvxrate_0}$, we further observe that $\obj(\timed{\avg\state}) \to \min\obj$ whenever the learning rate satisfies $\time\timed{\learn} \to \infty$ and $\timed{\learn} \to 0$. Among all such choices, the standard schedule $\timed{\learn} \propto 1/\time^{1/\momexp}$ is the one that optimizes the dependence of the resulting upper bound on $\time$.
%\end{remark}
\begin{remark}
By convexity, we also get
\begin{align}
\int_0^\time \braket*{\stateof{\timealt} - \sol}{\nabla \obj(\stateof{\timealt})} d\timealt &\geq \int_0^\time \bracks*{ \obj(\stateof{\timealt}) - \obj(\sol) } d\timealt\\
&\geq \time \bracks*{ \min_{0 \leq \timealt \leq \time} \obj(\stateof{\timealt}) - \min \obj }.
\end{align}
Consequently, the upper bounds of \cref{thm:convex_convergence_continuous_explicit} remain valid when $\obj(\avg{\state}(\time))$ is replaced by $\min_{0 \leq \timealt \leq \time} \obj(\stateof{\timealt})$.
\end{remark}
%\begin{remark}
% Let us define $\bar{\sdev} = \tamebound^2 + 2 \diamX^{2-\momexp} \hcst^{2-\momexp} \heavybound^\momexp$ as a measure of the total noise magnitude in the system. Then, for every target accuracy $\precval > 0$, an application of \cref{thm:convex_convergence_continuous_explicit} shows that running \eqref{eq:LMF} with a constant learning rate $\learn = \bigoh\parens*{ (\precval/\bar{\sdev})^{\frac{1}{\momexp-1}} }$ guarantees $\exof*{\obj(\avg{\state}(\time)) - \min\obj} \leq \precval$ for all times $\time \geq \bigoh\parens*{\bar{\sdev}^{\frac{1}{\momexp-1}} \precval^{-\frac{\momexp}{\momexp-1}}}$. This complexity rate of the time horizon $\time$ with respect to the accuracy $\precval$ provides a convenient baseline for comparing convergence rates obtained with constant learning rates, as will be illustrated throughout the remainder of the paper.
%\label{remark:complexity_rate_convex_continuous}
%\end{remark}
\subsection{Strongly convex setting}
We now specialize our results to functions that are strongly convex, either with respect to the Euclidean norm $\norm{\cdot}$ or relative to the regularizer $\hreg$.
First, assume that $\obj$ is $\strong$-strongly convex in the usual sense, that is,
\begin{equation}
\obj(\pointalt)
\geq \obj(\point)
+ \braket{\subsel\obj(\point)}{\pointalt - \point}
+ \frac{\strong}{2} \norm{\pointalt - \point}^{2}
\end{equation}
for all $\point \in \proxdom$, $\pointalt\in\points$. Applying the convergence guarantees of \cref{thm:convex_convergence_continuous} under this strong convexity assumption then yields the following rate of convergence of the time-averages toward the minimum of $\obj$:
\begin{theorem}[Convergence rate for strongly convex functions]
If $\obj$ is $\strong$-strongly convex with minimum $\sol \in \points$, then the time-averaged process $\timed{\avg\state}$ of \eqref{eq:LMF} enjoys the convergence rates
\begin{align}
\exof{\norm{\timed{\avg\state} - \sol}^2}
\leq \dfrac{2}{\strong}\cvxrate_{0}(\time)
\end{align}
and, \acl{wp1},
\begin{align}
\norm{\timed{\avg\state} - \sol}^2
\leq \dfrac{2}{\strong}\cvxrate_{0}(\time)
+ \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}}
+ \bigoh\parens*{\tamebound \diamX \sqrt{\frac{\log\log \time}{\time}}}
\end{align}
In particular, if \eqref{eq:LMF} is run with learning rate $\timed{\learn} = 1/\time^{1/\momexp}$, we have
\begin{align}
\cvxrate_{0}(\time)
&= \frac{\braket*{\stateinit - \basealt}{\dstateinit}}{\time}
+ \frac{\momexp}{\momexp-1} \frac{\tamebound^{2}}{2\hstr\time^{1/\momexp}}
+ \frac{\hstr^{\momexp-1}\hrange + \momexp \diamX^{2-\momexp} \heavybound^{\momexp}}{\hstr^{\momexp-1}\time^{(\momexp-1)/\momexp}}
= \bigoh(1/\time^{(\momexp-1)/\momexp})
\eqcomma
\end{align}
and $\timed{\avg\state} \to \sol$ both in mean square and with probability $1$ as $\time \to \infty$.
\label{thm:strongly_convex_continuous}
\end{theorem}
%\begin{corollary}
% If $\obj$ is $\strong$-strongly convex with minimum $\sol \in \points$, then iterates of \eqref{eq:LMF} satisfy
% \begin{align}
% \exof*{ \norm{\avg{\state}(\time) - \sol }^2} \leq \dfrac{2\braket*{\stateinit - \sol}{\dstateinit}}{\strong\time} + \dfrac{2\diamh}{\strong\learn(\time)\time} + \dfrac{\tamebound^2}{\time\strong\hcst} \int_0^\time \learn(\timealt) d\timealt + \dfrac{2\diamX^{2-\momexp} \heavybound^\momexp}{\time\strong\hcst^{\momexp-1}} \int_0^\time \learn(\timealt)^{\momexp-1} d\timealt
% \end{align}
% and, for any $\precval > 0$,
% \begin{align}
% \begin{aligned}
% \norm{\avg{\state}(\time) - \sol}^2 &\leq \dfrac{2\braket*{\stateinit - \sol}{\dstateinit}}{\strong\time} + \dfrac{2\diamh}{\strong\learn(\time)\time} + \dfrac{\tamebound^2}{\time\strong\hcst} \int_0^\time \learn(\timealt) d\timealt + \dfrac{2\diamX^{2-\momexp} \heavybound^\momexp}{\strong\time\hcst^{\momexp-1}} \int_0^\time \learn(\timealt)^{\momexp-1} d\timealt\\
% &+\bigoh\parens*{\dfrac{\tamebound\diamX }{\strong} \sqrt{\dfrac{\log\log \time}{\time}}} + \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}} \quad \as.
% \end{aligned}
% \end{align}
% In particular, running \eqref{eq:LMF} with learning rates $\learn(\time) = \time^{-1/\momexp}$ achieves
% \begin{align}
% \exof*{ \norm{\avg{\state}(\time) - \sol }^2} \leq \dfrac{2\braket*{\stateinit - \sol}{\dstateinit}}{\strong\time} + \dfrac{\momexp}{\momexp-1}\dfrac{\tamebound^2}{\strong\hcst} \dfrac{1}{\time^{\frac{1}{\momexp}}} + \dfrac{2}{\strong}\parens*{ \diamh + \dfrac{\momexp \diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} } \dfrac{1}{\time^{\frac{\momexp-1}{\momexp}}}
% \end{align}
% and
% \begin{align}
% \begin{aligned}
% \norm{\avg{\state}(\time) - \sol }^2 &\leq \dfrac{2\braket*{\stateinit - \sol}{\dstateinit}}{\strong\time} + \dfrac{\momexp}{\momexp-1}\dfrac{\tamebound^2}{\strong\hcst} \dfrac{1}{\time^{\frac{1}{\momexp}}} + \dfrac{2}{\strong}\parens*{ \diamh + \dfrac{\momexp \diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} } \dfrac{1}{\time^{\frac{\momexp-1}{\momexp}}}\\
% &+\bigoh\parens*{\dfrac{\tamebound\diamX}{\strong} \sqrt{\dfrac{\log\log \time}{\time}}} + \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}} \quad \as.
% \end{aligned}
% \end{align}
%\label{cor:strongly_convex_continuous}
%\end{corollary}
\begin{proof}
By strong convexity of $\obj$,
\begin{align}
\int_0^\time \braket*{\stateof{\timealt} - \sol}{\nabla \obj(\stateof{\timealt})}
&\geq \int_0^\time \bracks*{\obj(\stateof{\timealt}) - \min \obj + \dfrac{\strong}{2} \norm{\stateof{\timealt} - \sol}^2} d\timealt
\notag\\
&\geq \dfrac{\strong}{2} \int_0^\time \norm{\stateof{\timealt} - \sol}^2 d\timealt
\geq \dfrac{\strong\time}{2} \norm{\avg{\state}(\time) - \sol}^2,
\end{align}
and \cref{thm:convex_convergence_continuous} therefore yields the desired upper bounds.
\end{proof}
\begin{remark}
Similarly to the convex case, \cref{thm:strongly_convex_continuous} also yields the following complexity estimate: for any precision $\precval > 0$, choosing a constant learning rate $\learn = \bigoh\parens*{(\precval/\bar{\sdev})^{\frac{1}{\momexp-1}}}$ ensures that $\exof*{\norm{\avg{\state}(\time) - \sol}^2} \leq \precval$ for every times $\time = \Omega\parens*{ \bar{\sdev}^{\frac{1}{\momexp-1}} \precval^{-\frac{\momexp}{\momexp-1}} }$.
\label{remark:complexity_rate_strongly_convex_continuous}
\end{remark}
As in the main text, we now estimate the amount of time that $\stateof*{\time}$ spends in a neighborhood of $\argmin\obj$. The following theorem serves as an extension of \cref{thm:occmeas} to the setting where \eqref{eq:LMF} is initialized at an arbitrary point of $\points$, rather than at its prox-center.
%\OCCMEAS*
\begin{theorem}[Concentration around global minimum]
Assume that $\obj$ is $\strong$-strongly convex with minimum $\sol$, and let
\begin{equation}
\meas_{\time}(\nhd_{\precdist})
= \frac{1}{\time}
\int_{\tstart}^{\time} \oneof{\stateof{\timealt} \in \nhd_{\precdist}} \dd\timealt
% \int_{\tstart}^{\time} \oneof{\norm{\stateof{\timealt}} \leq \precdist} \dd\timealt
\end{equation}
be the fraction of time that $\stateof{\time}$ spends in $\nhd_{\precdist} = \setdef{\point \in \points}{\norm{\point - \sol} \leq \precdist}$ up to time $\time$.
If \eqref{eq:LMF} is run with constant learning rate $\timed{\learn} \equiv \learn$, we have
\begin{subequations}
\begin{equation}
\exof{\meas_{\time}(\nhd_{\precdist})}
\geq 1 - \occrate_{\time}(\precdist)
\end{equation}
and, \acl{wp1},
\begin{align}
\meas_{\time}(\nhd_{\precdist})
&\geq 1 - \occrate_{\time}(\precdist)
+ \baroh\parens*{\time^{-\frac{\momexp-1}{\momexp}}}
+ \bigoh\parens*{\tamebound \diamX \sqrt{\frac{\log\log \time}{\time}}}
\end{align}
\end{subequations}
where
\begin{equation}
\occrate_{\time}(\precdist)
% = \frac{2}{\strong\precdist^{2}}
% \bracks*{
% \frac{\hrange}{\learn\time}
% + \learn (\tamebound^{2})
% + \frac{\learn^{\momexp-1}\norm{\points}^{2-\momexp} \heavybound^{\momexp}}
% {\hstr^{\momexp-1}}
% }
= \frac{2\braket*{\stateinit - \sol}{\dstateinit}}{\strong \precdist^{2} \time}
+ \frac{2\hrange}{\learn\strong\precdist^{2}\time}
+ \frac{\learn \tamebound^{2}}{\strong\hstr\precdist^{2}}
+ \frac{2\learn^{\momexp-1}\norm{\points}^{2-\momexp} \heavybound^{\momexp}}
{\strong\hstr^{\momexp-1}\precdist^{2}}
\eqstop
\end{equation}
\label{thm:occupation_measure}
\end{theorem}
%\begin{theorem}
% Assume that $\obj$ is $\strong$-strongly convex with minimum $\sol \in \points$ and that $\learn(\time) \equiv \learn$ is constant. Then, for every radius $\precdist > 0$, the occupation measure concentrates around $\nhd_\precdist = \setdef{\point \in \points}{\norm{\point - \sol} \leq \precdist}$ as
% \begin{align}
% \exof{\meas_\time(\points \setminus \nhd_\precdist)} \leq \dfrac{2}{\strong \precdist^2} \bracks*{ \tamebound^2 \learn + \dfrac{\diamX^{2 - \momexp} \heavybound^{\momexp}}{\hcst^{\momexp - 1}} \learn^{\momexp - 1} }
% + \dfrac{2}{\strong \precdist^2 \time} \bracks*{ \braket*{\stateinit - \sol}{\dstateinit} + \dfrac{\diamh}{\learn}},
% \end{align}
% and, for any $\precval' \in (0, 1)$, as
% \begin{align}
% \meas_\time(\points \setminus \nhd_\precdist) &\leq \dfrac{2}{\strong \precdist^2} \bracks*{ \tamebound^2 \learn + \dfrac{\diamX^{2 - \momexp} \heavybound^{\momexp}}{\hcst^{\momexp - 1}} \learn^{\momexp - 1} } + \dfrac{2}{\strong \precdist^2 \time} \bracks*{ \braket*{\stateinit - \sol}{\dstateinit} + \dfrac{\diamh}{\learn}}\\
% &+ \dfrac{2}{\strong \precdist^2} \bracks*{ \bigoh\parens*{\tamebound\diamX \sqrt{\dfrac{\log\log\time}{\time}}} + o\parens*{\time^{- \frac{\momexp - 1}{\momexp} + \precval'}}} \quad \as.
% \end{align}
%\label{thm:occupation_measure}
%\end{theorem}
\begin{proof}
Since $\one\braces{\norm{\stateof{\time} - \sol} > \precdist} \leq \precdist^{-2} \norm{\stateof{\time} - \sol}^2$, the occupation measure $\meas_\time$ satisfies
\begin{align}
\meas_\time(\points \setminus \nhd_\precdist)
&= \dfrac{1}{\time} \int_0^\time \one\braces{\norm{\stateof{\timealt}-\sol} > \precdist} d\timealt
\notag\\
&\leq \dfrac{1}{\precdist^2 \time} \int_0^\time \norm{\stateof{\timealt} - \sol}^2 d\timealt
\notag\\
&\leq \dfrac{2}{\strong \precdist^2 \time} \int_0^\time \braket*{\stateof{\timealt} - \sol}{\nabla \obj(\stateof{\timealt})} d\timealt,
\end{align}
and \cref{thm:occupation_measure} therefore follows from \cref{thm:convex_convergence_continuous}.
\end{proof}
\begin{remark}
If the stochastic process $\state$ is \emph{positive recurrent} on $\intpoints$---that is, if any relatively compact subset of $\intpoints$ is reached by $\state$ in finite expected time---then the occupation measure $\meas_\time$ converges to the so-called \emph{invariant measure} $\meas$ of $\state$. This means that $\meas$ is the unique probability measure such that $\stateof{\time} \sim \meas$ whenever $\stateof{0} \sim \meas$. Under ergodicity of the dynamics, $\meas$ is also the measure towards which the law of $\stateof{\time}$ converges from \emph{any} initial condition. In this regards, \cref{thm:occupation_measure} therefore provides a non-asymptotic estimate on the concentration of the law of $\stateof{\time}$ around neighborhoods of the objective's minimum $\sol$. For the stochastic mirror flow \eqref{eq:SMF} with constant learning rate, \citet{MerSta18} showed that the primal trajectories are indeed positive recurrent---and hence admit a unique probability invariant measure---under Brownian-like perturbations. Extending this result to \eqref{eq:LMF} is, however, substantially more challenging: Lévy-driven SDEs exhibit more intricate dynamical properties, and establishing positive recurrence in this setting thus requires additional care. We therefore leave this question open for future works.
\end{remark}
%\begin{corollary}
% Under the same assumptions as in \cref{thm:occupation_measure}, the occupation measure satisfies
% \begin{equation}
% \limsup_{\time \to \infty} \exof{ \meas_\time(\points \setminus \nhd_\precdist) } \leq \dfrac{2}{\strong \precdist^2} \bracks*{ \tamebound^2 \learn + \dfrac{\diamX^{2 - \momexp} \heavybound^{\momexp}}{\hcst^{\momexp - 1}} \learn^{\momexp - 1} }
% \end{equation}
% and
% \begin{equation}
% \limsup_{\time \to \infty} \meas_\time(\points \setminus \nhd_\precdist) \leq \dfrac{2}{\strong \precdist^2} \bracks*{ \tamebound^2 \learn + \dfrac{\diamX^{2 - \momexp} \heavybound^{\momexp}}{\hcst^{\momexp - 1}} \learn^{\momexp - 1} } \quad \as.
% \end{equation}
%\end{corollary}
%
%\begin{proof}
% Follows directly by letting $\time \to \infty$ in \cref{thm:occupation_measure} with $\precval'$ chosen small enough.
%\end{proof}
As illustrated in \cref{remark:complexity_rate_strongly_convex_continuous}, \cref{thm:strongly_convex_continuous} provides a lower bound on the time beyond which the ergodic averages remain within at most $\precval$ of the objective's minimum in the mean square norm. The following theorem complements this result by giving an estimate on the average time required for the \emph{last iterate} itself to reach a $\precdist$-neighborhood of the minimum.
\begin{theorem}[Hitting time estimate]
Assume that $\obj$ is $\strong$-strongly convex with minimum $\sol \in \points$, and let
\begin{equation}
\hit_\precdist = \inf\setdef{\time \geq 0}{\norm{\stateof{\time} - \sol} \leq \precdist}
\end{equation}
denote the first time that $\stateof*{\time}$ gets within $\precdist$ of $\sol$ for some precision $\delta > 0$. If \eqref{eq:LMF} is run with constant learning rate
\begin{equation}
\learn < \min\parens*{ \dfrac{\strong \hcst}{2\tamebound^2} \precdist^2, \, \parens*{ \dfrac{\strong \hcst}{4 \hcst^{2-\momexp} \diamX^{2-\momexp} \heavybound^{\momexp}} }^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}} },
\label{eq:learning_rate_cdt}
\end{equation}
then the averaged value of $\hit_\precdist$ is bounded as
\begin{equation}
\exof*{\hit_\precdist} \leq \dfrac{2 \hcst \fench(\sol, \learn \dstateinit)}{\strong \hcst \learn \precdist^2 - \learn^2 \tamebound^2 - 2 \learn^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \heavybound^\momexp }.
\label{eq:hitting_time_bound_1}
\end{equation}
\label{thm:hittingtime}
\end{theorem}
\begin{proof}
Since the learning $\learn(\time)$ is assumed to be constant, \cref{lemma:dV_1,lemma:I_upper_bound,lemma:firstmart-bound,lemma:secondmart-bound} yield
\begin{equation}
\energy(\time) \leq \energy(0) - \int_0^\time \braket*{\state - \sol}{\nabla \obj(\state)}d\timealt + \dfrac{\learn}{2\hcst}\tamebound^2 \time + \dfrac{\learn^{\momexp-1} \diamX^{2-\momexp}}{\hcst^{\momexp-1}} \heavybound^\momexp \time + \xi(\time)
\end{equation}
where $\xi(\time) \defeq \timed{\firstmart} + \timed{\secondmart}$ is a martingale. Moreover, $\obj$ is assumed to be $\strong$-strongly convex and so $\braket*{\state - \sol}{\nabla \obj(\state)} \geq \frac{\strong}{2}\norm{\state - \sol}^2$. In particular, for any fixed $\time \geq 0$, we get
\begin{align}
0
&\leq \exof*{\energy(\hit_\precdist \wedge \time)}
\notag\\
&\leq \energy(0)
- \dfrac{\strong}{2} \exof*{ \int_0^{\hit_\precdist \wedge \time} \norm{\stateof{\timealt-} - \sol}^2 d\timealt }
+ \bracks*{ \dfrac{\learn}{2\hcst}\tamebound^2 + \dfrac{\learn^{\momexp-1} \diamX^{2-\momexp}}{\hcst^{\momexp-1}} \heavybound^\momexp } \exof*{\hit_\precdist \wedge \time},
\end{align}
where we have used that $\xi$ is a martingale starting from $0$ and that $\hit_\precdist \wedge \time$ is a bounded stopping time to get $\exof*{\xi(\hit_\precdist \wedge \time)} = 0$.
Now, note that by definition of $\hit_\precdist$, we have $\norm{\stateof*{\timealt-} - \sol}^2 \geq \precdist^2$ for every $\timealt \leq \hit_\precdist \wedge \time$ due to $\state$ having càdlàg paths. It follows that
\begin{equation}
0 \leq \energy(0) - \parens*{ \dfrac{\strong}{2} \precdist^2 - \dfrac{\learn}{2\hcst}\tamebound^2 - \dfrac{\learn^{\momexp-1} \diamX^{2 - \momexp} \heavybound^{\momexp}}{\hcst^{\momexp-1}} } \exof*{\hit_\precdist \wedge \time}.
\end{equation}
Now, under the assumption \cref{eq:learning_rate_cdt} made on the size of $\learn$, it is clear that the factor in front of $\exof*{\hit_\precdist \wedge \time}$ is strictly positive, which leads after rearranging the terms to
\begin{equation}
\exof*{\hit_\precdist \wedge \time} \leq \dfrac{2 \hcst \energy(0)}{ \strong \hcst \precdist^2 - \learn\tamebound^2 - 2\learn^{\momexp-1} \hcst^{2 - \momexp} \diamX^{2-\momexp} \heavybound^\momexp }.
\label{eq:bound_hitting_proof}
\end{equation}
Since the function $\time \mapsto \exof*{\hit_\precdist \wedge \time}$ is nondecreasing and since the right-hand side of \cref{eq:bound_hitting_proof} is uniform on $\time$, the monotone convergence theorem can be then invoked to prove the upper bound claimed in the theorem.
\end{proof}
In particular, for a sufficiently small target radius $\precdist$, the hitting time estimate of \cref{thm:hittingtime} can be optimized by an appropriate choice of learning rate $\learn$, making its dependence on $\precdist$ explicit. This leads to \cref{thm:hitting}, whose statement is here split in two parts into order to provide precise conditions on the target $\precdist$ depending on the nature of the noise. We first consider the case $\momexp < 2$, for which the upper bound is determined solely by the heavy-tailed intensity of the noise.
\begin{corollary}[Hitting time estimate, case $\momexp<2$]
Under the same assumptions as in \cref{thm:hittingtime}, further assume that $\dstateinit = 0$ and $\momexp < 2$. Then, for every precision $\precdist > 0$ satisfying
\begin{equation}
\precdist^2 \leq \dfrac{2 \tamebound^2\diamX}{\strong} \parens*{ \dfrac{2\heavybound^\momexp}{\tamebound^2} }^{\frac{1}{2-p}},
\label{eq:delta_small_cdt}
\end{equation}
running \eqref{eq:LMF} with constant learning rate
\begin{equation}
\learn = \parens*{ \dfrac{\strong \hcst}{8 \hcst^{2 - \momexp} \diamX^{2-\momexp} \heavybound^\momexp} }^{\frac{1}{\momexp - 1}} \precdist^{\frac{2}{\momexp - 1}}
\end{equation}
yields the hitting-time bound
\begin{equation}
\exof{\hit_\precdist} \leq \dfrac{4 \diamh}{\strong \hcst} \parens*{ \dfrac{8 \diamX^{2-\momexp} \heavybound^\momexp }{\strong}}^{\frac{1}{\momexp - 1}} \precdist^{- \frac{2\momexp}{\momexp - 1}}.
\end{equation}
\label{cor:hitting_time_heavy}
\end{corollary}
\begin{proof}
For simplicity, let us define $a = \tamebound^2$, $b = 2\hcst^{2-\momexp} \diamX^{2-\momexp} \heavybound^\momexp$ and $c = \strong \hcst$. In particular, the expression of the learning rate $\learn$ can be written as
\begin{equation}
\learn = \parens*{\dfrac{c}{4b}}^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}},
\end{equation}
and the condition of \cref{eq:delta_small_cdt} on the size of $\precdist$ implies that
\begin{equation}
\parens*{\dfrac{c}{2b}}^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}} \leq \parens*{\dfrac{c}{2a}} \precdist^2.
\end{equation}
We therefore obtain the upper bound
\begin{equation}
\learn \leq \dfrac{1}{2^{1/(\momexp-1)}} \parens*{\dfrac{c}{2a}} \precdist^2 \leq \dfrac{c}{4a} \precdist^2,
\end{equation}
where we have used that $p \in (1, 2)$ to get $2^{1/(\momexp-1)} \geq 2$. As a result, the denominator of \cref{eq:hitting_time_bound_1} satisfies
\begin{equation}
\learn\parens*{c\precdist^2 - a\learn - b\learn^{\momexp-1}} \geq \parens*{\dfrac{c}{4b}}^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}} \parens*{c\precdist^2 - a \dfrac{c}{4a} \precdist^2 - b \dfrac{c}{4b} \precdist^2} = \dfrac{c}{2} \parens*{\dfrac{c}{4b}}^{\frac{1}{\momexp-1}} \precdist^{\frac{2\momexp}{\momexp-1}}.
\end{equation}
Moreover, we also get $\fench(\sol, \learn \dstateinit) = h(\sol) - h(\stateinit) \leq \diamh$ since $\dstateinit=0$. The hitting time estimate of \cref{thm:hittingtime} therefore yields
\begin{equation}
\exof*{\hit_\precdist} \leq \dfrac{4\hcst \diamh}{c} \parens*{ \dfrac{4b}{c} }^{\frac{1}{\momexp-1}} \precdist^{-\frac{2\momexp}{\momexp-1}},
\end{equation}
which recovers our claim when substituting $a$, $b$ and $c$ with their respective values.
\end{proof}
On the other hand, when the noise is \emph{tame}---that is, when $\momexp=2$---we obtain the following optimized upper bound:
\begin{corollary}[Hitting time estimate, case $\momexp = 2$]
Under the same assumptions as in \cref{thm:hittingtime}, further assume that $\dstateinit = 0$ and $\momexp = 2$. Then, for every precision $\precdist > 0$, running \eqref{eq:LMF} with constant learning rate
\begin{equation}
\learn = \dfrac{\strong \hcst}{2\noisebound^2} \precdist^2
\end{equation}
yields the hitting-time bound
\begin{equation}
\exof{\hit_\precdist} \leq \dfrac{8 \noisebound^2 \diamh}{\strong^2 \hcst} \precdist^{-4},
\end{equation}
where $\noisebound^2 = \tamebound^2 + 2\heavybound^2$ is the total intensity of the noise.
\label{cor:hitting_time_light}
\end{corollary}
\begin{proof}
The upper bound is obtained directly when replacing $\learn$ by its value in \cref{thm:hittingtime} and by bounding the numerator as in the proof of \cref{cor:hitting_time_heavy}.
\end{proof}
We now strengthen our assumptions by considering objectives $\obj$ that are \define{$\strong$-strongly convex relative to $\hreg$}, that is, functions satisfying
\begin{equation}
\obj(\pointalt)
\geq \obj(\point)
+ \braket{\subsel\obj(\point)}{\pointalt - \point}
+ \strong \breg(\pointalt,\point)
\end{equation}
for all $\point\in\proxdom$ and all $\pointalt\in\points$.
This sharper condition allows us to refine the convergence analysis of \eqref{eq:LMF} to obtain stronger bounds on the last iterate rather than just on ergodic averages. We begin by establishing the first part of \cref{thm:str-LMD}, which corresponds to the case where \eqref{eq:LMF} is run with a constant learning rate:
\begin{theorem}[Convergence rate for relatively strongly convex functions, constant learning rate]
Let $\hreg$ be a steep regularizer. If $\obj$ is $\strong$-strongly convex relative to $\hreg$ with minimum $\sol \in \points$ and if $\learn(\time) \equiv \learn > 0$ is constant, then
\begin{equation}
\exof{\norm{\stateof{\time} - \sol}^2} \leq \dfrac{\tamebound^2}{\strong \hcst^2} \learn + \dfrac{2 \diamX^{2 - \momexp} \heavybound^\momexp}{\strong \hcst^\momexp} \learn^{\momexp - 1} + \dfrac{2 \breg(\sol, \stateinit)}{\hcst} e^{-\strong \learn \time}.
\end{equation}
% In particular,
% \begin{equation}
% \limsup_{\time \to \infty} \exof{\norm{\stateof{\time} - \sol}^2} \leq \dfrac{\tamebound^2}{\strong \hcst^2} \learn + \dfrac{2 \diamX^{2 - \momexp} \heavybound^\momexp}{\strong \hcst^\momexp} \learn^{\momexp - 1}.
% \end{equation}
\label{thm:relatively_convex_continuous1}
\end{theorem}
Before proving \cref{thm:relatively_convex_continuous1}, we first establish a preliminary technical lemma that will allows us to rigorously differentiate the map $\exof*{ \timed{\energy} }$ with respect to $\time$.
\begin{lemma}
Let $F \from [0, \infty) \to \R$ be a function such that
\begin{equation}
F(\time) - F(\timealt) \leq \int_{\timealt}^{\time} g(r) dr \quad \text{for every } \time \geq \timealt \geq 0,
\label{eq:bounded_to_differentiable}
\end{equation}
where $g \from [0, \infty) \to \R$ is a locally integrable function. Then $F$ is almost everywhere differentiable with $F'(\time) \leq g(\time)$.
\label{lemma:bounded_to_differentiable}
\end{lemma}
\begin{proof}
Let $G(\time) = \int_0^\time g(r) dr$ and $H = F - G$. Then \cref{eq:bounded_to_differentiable} implies that $H(\time) \leq H(\timealt)$ for all $\time \geq \timealt \geq 0$, and thus $H$ is a nonincreasing function. In virtue of Lebesgue differentiation theorem, $H$ is therefore almost everywhere differentiable (on open intervals) and satisfies $H'(\time) \leq 0$. Since $F = G + H$ is the sum of two almost everwhere differentiable function, it implies in turn that $F$ is also as such and that $F'(\time) = G'(\time) + H'(\time) \leq g(\time)$, which concludes the proof.
\end{proof}
\begin{proof}[Proof of \cref{thm:relatively_convex_continuous1}]
Starting from \cref{lemma:dV_1} and the estimates of \cref{lemma:I_upper_bound,lemma:firstmart-bound,lemma:secondmart-bound}, and noting that $\breg(\sol, \stateof{\time}) = \fench(\sol, \learn \dstateof{\time})$ since $\stateof{\time} \in \intpoints$ for all times (consequence of $\hreg$ being steep, \cf \cref{prop:Fench}), we get
\begin{align}
\dfrac{1}{\learn}
&\exof*{\breg(\sol, \stateof{\time})}
= \exof*{\energy(\time)}
\notag\\
&\leq \exof*{\energy(\timealt)}
- \exof*{\int_{\timealt}^\time \braket*{\state - \sol}{\nabla \obj(\state)} dr} + \dfrac{\learn}{2\hcst}\tamebound^2 \int_{\timealt}^\time dr + \dfrac{\learn^{\momexp-1} \diamX^{2-\momexp}}{\hcst^{\momexp-1}} \heavybound^\momexp \int_{\timealt}^\time dr
\notag\\
&\leq \dfrac{1}{\learn} \exof*{\breg(\sol, \stateof{\timealt})}
\notag\\
&\quad
- \int_{\timealt}^\time \bracks*{ \exof*{ \obj(\state) - \obj(\sol) + \strong \breg(\sol, \state)} - \parens*{ \dfrac{\learn\tamebound^2}{2\hcst} + \dfrac{\learn^{\momexp-1} \diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} } }dr
\notag\\
&\leq \dfrac{1}{\learn} \exof*{\breg(\sol, \stateof{\timealt})} - \int_{\timealt}^\time \bracks*{ \strong \exof*{\breg(\sol, \state)} - \parens*{ \dfrac{\learn\tamebound^2}{2\hcst} + \dfrac{\learn^{\momexp-1} \diamX^{2-\momexp} \heavybound^\momexp}{\hcst^{\momexp-1}} } }dr,
\end{align}
where the second inequality comes from the relative strong convexity of $\obj$, and the last inequality is due to $\sol$ being the minimum of $\obj$. Since the above inequality holds true for every $\time \geq \timealt \geq 0$, it follows from \cref{lemma:bounded_to_differentiable} that the function $\varphi(\time) \defeq \exof*{\breg(\sol, \stateof{\time})}$ is almost-everywhere differentiable and that it satisfies
\begin{equation}
\varphi'(\time) \leq -\strong \learn \varphi(\time) + c_1 \learn^2 + c_2 \learn^\momexp,
\end{equation}
where we have set $c_1 = \frac{1}{2\hcst}(\tamebound^2)$ and $c_2 = \frac{\diamX^{2-\momexp}}{\hcst^{\momexp-1}} \heavybound^\momexp$.
In particular, we can therefore use Grönwall's lemma to obtain
\begin{align}
\exof*{\breg(\sol, \stateof{\time})}
= \varphi(\time)
&\leq \varphi(0) e^{-\strong \learn \time} + \dfrac{c_1 \learn^2 + c_2 \learn^{\momexp}}{\strong \learn} \parens*{ 1 - e^{-\strong \learn \time} }
\notag\\
&\leq \dfrac{\tamebound^2}{2\hcst \strong} \learn + \dfrac{\diamX^{2 - \momexp} \heavybound^\momexp}{\strong \hcst^{\momexp-1}} \learn^{\momexp-1} + \breg(\sol, \stateinit)e^{-\strong \learn \time},
\end{align}
and the result then follows thanks to the inequality $\norm{\stateof{\time} - \sol}^2 \leq \frac{2}{\hcst} \breg(\sol, \stateof{\time})$ (\cf \cref{prop:Fench}).
\end{proof}
\begin{remark}
A result analogous to \cref{thm:relatively_convex_continuous1} for Brownian-type stochastic perturbations was established by \citet{TRRB23a} for the \emph{mirror Langevin dynamics}, a special case of \eqref{eq:SMF} with constant learning rate in which the diffusion coefficient is proportional to the Hessian of the regularizer. Their result arises as a byproduct of an interpretation of continuous-time mirror descents methods through the lens of inverse optimal control.
\end{remark}
%\begin{remark}
% In particular, letting $\time \to \infty$ in \cref{thm:relatively_convex_continuous1} yields
% \begin{equation}
% \limsup_{\time \to \infty} \exof{\norm{\stateof{\time} - \sol}^2} \leq \dfrac{\tamebound^2}{\strong \hcst^2} \learn + \dfrac{2 \diamX^{2 - \momexp} \heavybound^\momexp}{\strong \hcst^\momexp} \learn^{\momexp - 1}.
% \end{equation}
% This coincides with the upper bound that can be obtained from \cref{thm:strongly_convex_continuous}, but now applies to the \emph{last iterate} rather than to ergodic averages. Moreover, under relative strong convexity, this bound is reached at exponentially fast rate instead of the $1/\time$ rate for standard strong convexity, resulting in a significantly sharper guarantee.
%\end{remark}
%\begin{remark}
% Applying \cref{thm:relatively_convex_continuous1}, we see that for any target accuracy $\precval > 0$, the learning rate $\learn = \bigoh\parens*{(\precval/\bar{\sdev})^{\frac{1}{\momexp-1}}}$ ensures $\exof*{\norm{\stateof{\time} - \sol}^2} \leq \precval$ for every times $\time \geq \bigoh\parens*{ \bar{\sdev}^{\frac{1}{\momexp-1}} \precval^{-\frac{1}{\momexp-1}} \log(1/\precval) }$. Compared to the bounds obtained for convex and standard strongly convex functions, \cf \cref{remark:complexity_rate_convex_continuous,remark:complexity_rate_strongly_convex_continuous}, this rate is \emph{sharper} since $\frac{1}{\momexp-1} \leq \frac{\momexp}{\momexp-1}$. In particular, when $\momexp=2$, this matches the well-known $\bigoh\parens*{\precval^{-1}}$ complexity of discrete-time stochastic gradient methods on strongly convex functions \needref, up to a logarithmic factor. For $\momexp < 2$, however, the bound no longer attains the optimal complexity for first-order methods on strongly convex functions, which is of order $\bigoh\parens*{ \precval^{- \frac{\momexp}{2(\momexp-1)}} }$, \cf \citep[Theorem 5]{ZKV+20}.
%\label{remark:complexity_rate_relatively_convex_continuous1}
%\end{remark}
Finally, we now establish the second part of \cref{thm:str-LMD}, concerning the convergence rates of the last iterate when the learning rate is decreasing.
\begin{theorem}[Convergence rate for relatively strongly convex functions, decreasing learning rate]
Let $\hreg$ be a steep regularizer. If $\obj$ is $\strong$-strongly convex relative to $\hreg$ with minimum $\sol \in \points$ and if $\eta(\time) = (1+\time)^{-1/\momexp}$, then
\begin{align}
\exof{\norm{\stateof{\time} - \sol}^2}
&\leq \dfrac{\tamebound^2}{\strong \hcst^2} \time^{- \frac{1}{\momexp}}
+ \dfrac{2}{\strong \hcst} \parens*{ \dfrac{\diamX^{2 - \momexp} \heavybound^\momexp}{\hcst^{\momexp - 1}} + \dfrac{\diamh}{\momexp} } \time^{-\frac{\momexp - 1}{\momexp}}
\notag\\
&+ \dfrac{2 \breg(\sol, \stateinit)}{\hcst} \exp\bracks*{ - \frac{\strong \momexp}{\momexp - 1} \parens*{ \time^{\frac{\momexp - 1}{\momexp}} - 1 } }.
\end{align}
In particular, $\stateof{\time} \to \sol$ in mean square as $\time \to \infty$.
\label{thm:relatively_convex_continuous2}
\end{theorem}
\begin{proof}\let\qed\relax
Following the same arguments as in the proof of \cref{thm:relatively_convex_continuous1} but with a non-constant learning rate, we obtain that the function $\psi(\time) \defeq \frac{1}{\learn(\time)} \exof*{\breg(\sol, \stateof{\time})}$ is almost-everywhere differentiable and that it satisfies
\begin{align}
\psi'(\time) &\leq - \strong \exof*{\breg(\sol, \state)} + c_1 \learn(\time) + c_2 \learn(\time)^{\momexp-1} - \dfrac{\dot\learn(\time)}{\learn^2(\time)} \exof*{ \hreg(\sol) - \hreg(\state) }
\notag\\
&\leq - \strong \learn(\time) \psi(\time) + c_1 \learn(\time) + c_2 \learn(\time)^{\momexp-1} - \dfrac{\dot\learn(\time)}{\learn^2(\time)} \diamh,
\end{align}
where we recall that $c_1 = \frac{\tamebound^2}{2\hcst}$ and $c_2 = \frac{\diamX^{2-\momexp}}{\hcst^{\momexp-1}} \heavybound^\momexp$. Letting $\learn(\time) = (1+\time)^{-1/\momexp}$, we therefore get
\begin{equation}
\psi'(\time) \leq - \dfrac{\strong}{(1+\time)^{\frac{1}{\momexp}}} \psi(\time) + \dfrac{c_1}{(1+\time)^{\frac{1}{\momexp}}} + \dfrac{c_2}{(1+\time)^{\frac{\momexp-1}{\momexp}}} + \dfrac{\diamh}{\momexp (1+\time)^{\frac{\momexp-1}{\momexp}}},
\end{equation}
and Grönwall's lemma can then be invoked to write
\begin{align}
\psi(\time)
&\leq \psi(0) \exp\parens*{ -\strong \int_0^\time \dfrac{1}{(1+\timealt)^{\frac{1}{\momexp}}} d\timealt}
\notag\\
&+ \int_0^\time \parens*{ \dfrac{c_1}{(1+\timealt)^{\frac{1}{\momexp}}} + \dfrac{c_3}{(1+\timealt)^{\frac{\momexp-1}{\momexp}}} } \exp\parens*{ - \strong \int_{\timealt}^\time \dfrac{1}{(1+r)^{\frac{1}{\momexp}}} dr } d\timealt,
\label{eq:psi_inequality_1}
\end{align}
where we have set $c_3 = c_2 + \frac{\diamh}{\momexp}$. To estimate the integral appearing in the right-hand side of \eqref{eq:psi_inequality_1}, we need the following intermediate lemma:
\end{proof}
\begin{lemma}
For all $a, b \geq 0$, $c > 0$, and all exponents $p \in (1, 2]$, we have:
\begin{equation}
I(\time) \defeq \int_0^\time \parens*{ \dfrac{a}{(1+\timealt)^{\frac{1}{\momexp}}} + \dfrac{b}{(1+\timealt)^{\frac{\momexp-1}{\momexp}}}} \exp\parens*{ - c \int_{\timealt}^\time \dfrac{1}{(1+r)^{\frac{1}{\momexp}}} dr } d\timealt \leq \dfrac{1}{c} \parens*{ a + b (1+\time)^{\frac{2 - \momexp}{\momexp}} }.
\end{equation}
\label{lemma:integral_estimate}
\end{lemma}
\begin{proof}
To begin with, note that
\begin{equation}
\int_{\timealt}^\time \dfrac{1}{(1+r)^{1/\momexp}} dr = \dfrac{\momexp}{\momexp-1}\bracks*{(1+\time)^{\frac{\momexp-1}{\momexp}} - (1+\timealt)^{\frac{\momexp-1}{\momexp}}},
\end{equation}
so $I(\time)$ can be written as
\begin{multline}
I(\time) = \exp\parens*{ - \dfrac{c \momexp}{\momexp-1} (1+\time)^{\frac{\momexp-1}{\momexp}} } \int_0^\time \parens*{ a (1+\timealt)^{-\frac{1}{\momexp}} + b(1+\timealt)^{-\frac{\momexp-1}{\momexp}} } \\ \times \exp\parens*{ \dfrac{c\momexp}{\momexp-1} (1+\timealt)^{\frac{\momexp-1}{\momexp}}} d\timealt.
\end{multline}
Using the substitution $r \leftrightarrow (1+\timealt)^{\frac{\momexp-1}{\momexp}}$, we further get
\begin{equation}
I(\time) = \exp\parens*{ - \dfrac{c \momexp}{\momexp-1} (1+\time)^{\frac{\momexp-1}{\momexp}} } \dfrac{\momexp}{\momexp-1} \int_1^{(1+\time)^{\frac{\momexp-1}{\momexp}}} \parens*{a + b r^{\frac{2-\momexp}{\momexp-1}}} \exp\parens*{\dfrac{c \momexp}{\momexp-1} r} dr.
\end{equation}
Now, since $\frac{2-\momexp}{\momexp-1} \geq 0$ and $0 \leq r \leq (1+\time)^{\frac{\momexp-1}{\momexp}}$, we also have $r^{\frac{2-\momexp}{\momexp-1}} \leq (1+\time)^{\frac{2-\momexp}{\momexp}}$ and thus
\begin{align}
I(\time) &\leq \exp\parens*{ - \dfrac{c \momexp}{\momexp-1} (1+\time)^{\frac{\momexp-1}{\momexp}} } \dfrac{\momexp}{\momexp-1} \parens*{a + b (1+\time)^{\frac{2-\momexp}{\momexp}}} \int_1^{(1+\time)^{\frac{\momexp-1}{\momexp}}} \dfrac{\momexp}{\momexp-1} \exp\parens*{\dfrac{c\momexp}{\momexp-1} r} dr
\notag\\
&=\dfrac{1}{c}\parens*{a + b (1+\time)^{\frac{2-\momexp}{\momexp}}} \bracks*{ 1 - \exp \parens*{ \dfrac{c\momexp}{\momexp-1} \parens*{ 1 - (1+\time)^{\frac{\momexp-1}{\momexp}}} } }
\notag\\
&\leq \dfrac{1}{c}\parens*{a + b (1+\time)^{\frac{2-\momexp}{\momexp}}},
\end{align}
which concludes the proof.
\end{proof}
\begin{proof}[Proof of \cref{thm:relatively_convex_continuous2} (cont'd)]
Using \cref{lemma:integral_estimate} in \cref{eq:psi_inequality_1} and replacing the function $\psi(\time)$ by its expression, we readily get
\begin{align}
\exof*{(1+\time)^{1/\momexp} \breg(\sol, \stateof{\time})}
&\leq \breg(\sol, \stateinit) \exp\parens*{ - \dfrac{\strong \momexp}{\momexp-1} \parens*{ (1+\time)^{\frac{\momexp-1}{\momexp}} - 1} }
\notag\\
&+ \dfrac{1}{\strong}\parens*{c_1 + c_3(1+\time)^{\frac{2-\momexp}{\momexp}}}
\end{align}
which, after dividing both sides by $(1+\time)^{1/\momexp}$ and using using the inequality $\norm{\stateof{\time}-\sol}^2 \leq \frac{2}{\hcst} \breg(\sol, \stateof{\time})$, yields
\begin{align}
\exof*{\norm{\stateof{\time} - \sol}^2}
&\leq \dfrac{2c_1}{\hcst\strong} (1+\time)^{-\frac{1}{\momexp}}
+ \dfrac{2c_3}{\hcst \strong} (1+\time)^{-\frac{\momexp-1}{\momexp}}
\notag\\
&+ \dfrac{2\breg(\sol, \stateinit)}{\hcst} \exp\parens*{ - \dfrac{\strong \momexp}{\momexp-1} \parens*{ (1+\time)^{\frac{\momexp-1}{\momexp}} - 1 } }.
\end{align}
The final result is then obtained when replacing $c_1$ and $c_3$ with their respective values.
\end{proof}
\begin{remark}
For a learning rate $\learn \propto \time^{-1/\momexp}$, \cref{thm:relatively_convex_continuous2} implies that for any $\precval > 0$, the accuracy $\exof*{ \norm{\stateof{\time} - \sol}^2 } < \precval$ is reached for times $\time \geq \bigoh\parens*{\precval^{-\frac{\momexp}{\momexp-1}}}$. Unlike the complexity bound for constant learning rates, this rate is of the same order as those for convex or strongly convex functions, and therefore does not even attain the optimal $\bigoh\parens*{1/\precval}$ complexity for $\momexp=2$. As we discuss further in the discrete-time section, this limitation arises because a positive term proportional to $-\dot{\learn} / \learn^2$ always appears in the computations. This is a consequence of the learning rate acting on the aggregated gradient steps rather than on individual steps, which prevents the use of a $1/\time$-type learning rate to achieve sharper convergence rates, as is standard in the analysis of stochastic gradient methods.
\end{remark}
%----------------------------------------------------------------------
%% APP: DISCRETE
%----------------------------------------------------------------------
% !TEX root = ../Main.tex
In this section, we draw inspiration from the continuous-time analysis of \eqref{eq:LMF} to study the convergence properties of the Stochastic Dual Averaging \eqref{eq:SDA} algorithm given by
\begin{equation}
\tag{SDA}
\begin{aligned}
\next[\dpoint]
&= \curr[\dpoint]
- \curr[\sgrad]
\\
\next[\point]
&= \mirror(\next[\learn] \next[\dpoint])
\end{aligned}
\end{equation}
where $\curr[\learn] > 0$ is a nonincreasing \define{learning rate} sequence and $\curr[\sgrad]$ is a \emph{stochastic gradient} of the form $\curr[\sgrad] = \nabla\obj(\curr[\point]) + \curr[\noise]$ with gradient noise term $\curr[\noise] = \err(\curr;\curr[\seed])$ satisfying the following assumptions:
\begin{enumerate}
[label=\upshape(\itshape\alph*\hspace*{.5pt}\upshape)]
\item
\define{Unbiasedness:}
$\exof{\err(\point;\seed)} = 0$ for all $\point\in\dom\subd\obj$.
\item
\define{Bounded $\momexp$-th central moments:}
$\exof{\dnorm{\err(\point;\seed)}^{\momexp}} \leq \sdev^{\momexp}$ for some $\momexp\in(1,2]$ and all $\point\in\dom\subd\obj$.
\end{enumerate}
Throughout this section, we further assume that $\obj$ is \emph{$\lips$-smooth relative to $\hreg$}, that is,
\begin{equation}
\obj(\pointalt)
\leq \obj(\point)
+ \braket{\nabla\obj(\point)}{\pointalt - \point}
+ \lips \breg(\pointalt,\point) \quad \text{for all } \point\in\proxdom, \pointalt\in\points\eqstop.
\end{equation}
\subsection{Convex setting}
We begin by exploring the case where $\obj$ is a convex function. As in the continuous-time analysis, our approach is based on tracking the evolution of the \emph{$\learn$-deflated Fenchel coupling}
\begin{equation}
\curr[\energy](\basealt, \dpoint) = \dfrac{1}{\curr[\learn]} \fench(\basealt, \curr[\learn] \dpoint) \ \quad \basealt \in \points, \dpoint \in \dpoints
\end{equation}
along the iterates of \eqref{eq:SDA}. In contrast to \eqref{eq:LMF}, however, the discrete-time nature of \eqref{eq:SDA} allows us to derive the evolution of $\curr[\energy]$ without needing a weak stochastic chain rule (Itô’s formula), which considerably simplifies the analysis.
This leads to the following result, which can be viewed as a discrete-time analogue of the classical \emph{three-point identity} (\cref{lem:3point}) for the $\learn$-deflated Fenchel coupling.
\begin{lemma}[Three-point identity for $\learn$-deflated Fenchel coupling]
For every $\basealt \in \points$, the process $\curr[\energy] \defeq \curr[\energy](\basealt, \curr[\dpoint])$ along trajectories of \eqref{eq:SDA} satisfies
\begin{align}
\next[\energy] - \curr[\energy]
&= \parens*{ \dfrac{1}{\next[\learn]} - \dfrac{1}{\curr[\learn]} } \parens*{ \hreg(\basealt) - \hreg(\next[\point]) }
\notag\\
&+ \braket*{\curr[\point] - \basealt}{\next[\dpoint] - \curr[\dpoint]}
+ \braket*{\next[\point] - \curr[\point]}{\next[\dpoint] - \curr[\dpoint]} - \dfrac{1}{\curr[\learn]} \fench(\next[\point], \curr[\learn] \curr[\dpoint]).
\end{align}
\label{lemma:difference_of_V}
\end{lemma}
\begin{proof}
First, note that
\begin{equation}
\fench(\next[\point], \curr[\learn]\curr[\dpoint]) - \fench(\basealt, \curr[\learn]\curr[\dpoint]) = \hreg(\next[\point]) - \hreg(\basealt) - \curr[\learn] \braket*{\next[\point] - \basealt}{\curr[\dpoint]}
\eqcomma
\end{equation}
and, by optimality of $\next[\point]$, we also have
\begin{align}
\fench(\basealt, \next[\learn]\next[\dpoint])
&= \hreg(\basealt) + \hconj(\next[\learn]\next[\dpoint]) - \next[\learn] \braket*{\basealt}{\next[\dpoint]}
\notag\\
&= \hreg(\basealt) + \next[\learn] \braket*{\next[\point]}{\next[\dpoint]} - \hreg(\next[\point]) - \next[\learn]\braket*{\basealt}{\next[\dpoint]}
\notag\\
&= \hreg(\basealt) - \hreg(\next[\point]) + \next[\learn] \braket*{\next[\point] - \basealt}{\next[\dpoint]}.
\end{align}
This leads to
\begin{align}
\next[\energy] - \curr[\energy] + \dfrac{1}{\curr[\learn]} \fench(\next[\point], \curr[\learn]\curr[\dpoint])
&= \dfrac{1}{\next[\learn]} \fench(\basealt, \next[\learn]\next[\dpoint]) + \dfrac{1}{\curr[\learn]} \parens*{ \fench(\next[\point], \curr[\learn]\curr[\dpoint]) - \fench(\basealt, \curr[\learn]\curr[\dpoint]) }\notag\\
&= \parens*{\dfrac{1}{\next[\learn]} - \dfrac{1}{\curr[\learn]}} \parens*{ \hreg(\basealt) - \hreg(\next[\point]) }
\notag\\
&\quad
+ \braket*{\next[\point] - \basealt}{\next[\dpoint] - \curr[\dpoint]}
\notag\\
&= \parens*{\dfrac{1}{\next[\learn]} - \dfrac{1}{\curr[\learn]}} \parens*{ \hreg(\basealt) - \hreg(\next[\point]) }
\notag\\
&\quad
+ \braket*{\curr[\point] - \basealt}{\next[\dpoint] - \curr[\dpoint]}
+ \braket*{\next[\point] - \curr[\point]}{\next[\dpoint] - \curr[\dpoint]}
\eqcomma
\end{align}
which concludes the proof after rearranging the terms.
\end{proof}
We now invoke the relative smoothness of the objective together with the bounded $\momexp$-th central moment assumption on the gradient noise to obtain the following estimate:
\begin{lemma}[Heavy-tailed noise estimate]
If $\obj$ is relatively $\lips$-smooth and $\curr[\learn] \leq \frac{1}{\momexp \lips}$, then the iterates of \eqref{eq:SDA} satisfy
\begin{equation}
\braket*{\next[\point] - \curr[\point]}{\next[\dpoint] - \curr[\dpoint]} - \dfrac{1}{\curr[\learn]} \fench(\next[\point], \curr[\learn] \curr[\dpoint]) \leq \obj(\curr[\point]) - \obj(\next[\point]) + \dfrac{2^{\momexp - 1} \diamX^{2 -\momexp}}{\momexp \hcst^{\momexp-1}} \curr[\learn]^{\momexp - 1} \dnorm{\curr[\noise]}^{\momexp}.
\label{eq:heavy_tails_descent_discrete}
\end{equation}
\label{lemma:heavy_tails_descent}
\end{lemma}
\begin{proof}
Since $\next[\dpoint] - \curr[\dpoint] = - \curr[\sgrad] = - (\nabla \obj(\curr[\point]) + \curr[\noise])$ and $\momexp \in (1, 2]$, we can decompose the left-hand side of \cref{eq:heavy_tails_descent_discrete} as
\begin{align}
\braket*{\next[\point] - \curr[\point]}{\next[\dpoint] - \curr[\dpoint]} - \dfrac{1}{\curr[\learn]}\fench(\next[\point], \curr[\learn] \curr[\dpoint])
&= \braket*{\nabla \obj(\curr[\point])}{\curr[\point] - \next[\point]} - \dfrac{1}{\momexp \curr[\learn]} \fench(\next[\point], \curr[\learn]\curr[\dpoint])
\label{eq:xi_1_discrete}\\
&+ \braket*{\curr[\noise]}{\curr[\point] - \next[\point]} - \dfrac{\momexp-1}{\momexp\curr[\learn]} \fench(\next[\point], \curr[\learn]\curr[\dpoint])\label{eq:xi_2_discrete}.
\end{align}
Focusing first on bounding \eqref{eq:xi_1_discrete}, the relative smoothness of $\obj$ yields
\begin{align}
\text{\eqref{eq:xi_1_discrete}}
&\leq \obj(\curr[\point]) - \obj(\next[\point]) + \lips \breg(\next[\point], \curr[\point]) - \dfrac{1}{\momexp \curr[\learn]} \fench(\next[\point], \curr[\learn]\curr[\dpoint])
\notag\\
&\leq \obj(\curr[\point]) - \obj(\next[\point]) + \lips \fench(\next[\point], \curr[\learn]\curr[\dpoint]) - \dfrac{1}{\momexp \curr[\learn]} \fench(\next[\point], \curr[\learn]\curr[\dpoint])
\notag\\
&= \obj(\curr[\point]) - \obj(\next[\point]) - \dfrac{\lips}{\curr[\learn]} \parens*{ \dfrac{1}{\momexp \lips} - \curr[\learn] } \fench(\next[\point], \curr[\learn]\curr[\dpoint])
\notag\\
&\leq \obj(\curr[\point]) - \obj(\next[\point]),
\end{align}
where the second inequality comes from $\breg(\point, \mirror(\dpoint')) \leq \fench(\point, \dpoint')$ (\cf \cref{prop:Fench}) and the last inequality from the assumption that $\curr[\learn] \leq 1/(\momexp \lips)$. On the other hand, for \eqref{eq:xi_2_discrete}, we use the Cauchy-Schwarz inequality and the fact that $\fench(\point, \dpoint) \geq \frac{\hcst}{2} \norm{\point - \mirror(\dpoint)}^2$ (see \cref{prop:Fench}) to get
\begin{equation}
\text{\eqref{eq:xi_2_discrete}} \leq \dnorm{\curr[\noise]} \norm{\next[\point] - \curr[\point]} - \dfrac{\momexp-1}{\momexp \curr[\learn]} \dfrac{\hcst}{2} \norm{\next[\point] - \curr[\point]}^2.
\label{eq:noisebound_discrete}
\end{equation}
Since the diameter $\diamX$ of $\points$ is assumed to be finite, we know that $\norm{\next[\point] - \curr[\point]} \leq \diamX$ and therefore
\begin{align}
\norm{\next[\point] - \curr[\point]}^2
&= \diamX^{- \frac{2-\momexp}{\momexp - 1}} \diamX^{\frac{2-\momexp}{\momexp - 1}} \norm{\next[\point] - \curr[\point]}^2
\notag\\
&\geq \diamX^{-\frac{2-\momexp}{\momexp - 1}} \norm{\next[\point] - \curr[\point]}^{\frac{2-\momexp}{\momexp - 1}} \norm{\next[\point] - \curr[\point]}^2 = \diamX^{-\frac{2-\momexp}{\momexp - 1}} \norm{\next[\point] - \curr[\point]}^{\frac{\momexp}{\momexp - 1}}.
\end{align}
Putting this estimate back into \cref{eq:noisebound_discrete} then yields
\begin{align}
\eqref{eq:xi_2_discrete}
&\leq \dnorm{\curr[\noise]} \norm{\next[\point] - \curr[\point]} - \dfrac{\momexp-1}{\momexp} \dfrac{\hcst \diamX^{-\frac{2-\momexp}{\momexp-1}}}{2\curr[\learn]} \norm{\next[\point] - \curr[\point]}^{\frac{\momexp}{\momexp-1}}
\notag\\
&\leq \max_{w \geq 0} \parens*{ \dnorm{\curr[\noise]} w - \dfrac{\momexp - 1}{\momexp} C w^{\frac{\momexp}{\momexp-1}} }
\eqcomma
\end{align}
where we have defined $C = \frac{\hcst}{2\curr[\learn]}\diamX^{-\frac{2-\momexp}{\momexp-1}}$ for conciseness. The maximization problem is attained at $w = (\dnorm{\curr[\noise]}/C)^{\momexp - 1}$ and thus
\begin{align}
\eqref{eq:xi_2_discrete}
&\leq \dnorm{\curr[\noise]} \parens*{ \dfrac{\dnorm{\curr[\noise]}}{C} }^{\momexp -1} - \dfrac{\momexp-1}{\momexp} C \parens*{ \dfrac{\dnorm{\curr[\noise]}}{C} }^{\frac{\momexp}{\momexp-1}(\momexp-1)}
\notag\\
&= \parens*{1 - \frac{\momexp-1}{\momexp}} C^{1-\momexp} \dnorm{\curr[\noise]}^{\momexp} = \dfrac{2^{\momexp-1} \diamX^{2-\momexp} }{\momexp \hcst^{\momexp-1}} \curr[\learn]^{\momexp-1} \dnorm{\curr[\noise]}^{\momexp}.
\end{align}
Combining the upper bounds for \eqref{eq:xi_1_discrete} and \eqref{eq:xi_2_discrete} therefore yields the desired result.
\end{proof}
We can now combine the first two lemmas to obtain the following theorem, which---similarly to its continuous-time analogue---serves as the baseline for most of our subsequent results.
\begin{theorem}
If $\obj$ is relatively $\lips$-smooth, then for every $\basealt \in \points$ and every nonincreasing learning rate sequence with $\learn_1 \leq \frac{1}{\momexp \lips}$, the iterates of \eqref{eq:SDA} satisfy
\begin{multline}
\exof*{\sum_{\run=1}^\horizon \braket*{ \curr[\point] - \basealt}{\nabla \obj(\curr[\point])} } \leq \braket*{\point_1 - \basealt}{\dpoint_1} + \obj(\point_1) - \min \obj \\ + \dfrac{\diamh}{\learn_{\horizon+1}} + \dfrac{2^{\momexp - 1} \diamX^{2-\momexp}}{\momexp \hcst^{\momexp-1}} \sdev^{\momexp} \sum_{\run=1}^\horizon \curr[\learn]^{\momexp-1}.
\end{multline}
\label{thm:discrete_relative_smooth_cv}
\end{theorem}
\begin{proof}
Summing the equality of \cref{lemma:difference_of_V} over $\run = 1, \dots, \horizon$ and using the upper bound of \cref{lemma:heavy_tails_descent}, we get
\begin{align}
0
\leq \exof{\energy_\horizon}
\leq \energy_1
&+ \sum_{\run=1}^\horizon \parens*{\dfrac{1}{\next[\learn]} - \dfrac{1}{\curr[\learn]}} \exof*{\hreg(\basealt) - \hreg(\next[\point]) }
\notag\\
&- \exof*{\sum_{\run=1}^\horizon \braket*{\curr[\point] - \basealt}{\nabla \obj(\curr[\point])}} + \obj(\point_1) - \exof*{ \obj(\point_{\horizon+1}) }
\notag\\
&- \sum_{\run=1}^\horizon \exof*{\braket*{\curr[\point] - \basealt}{\curr[\noise]}} + \dfrac{2^{\momexp-1} \diamX^{2-\momexp}}{\momexp \hcst^{\momexp-1}} \sum_{\run=1}^\horizon \curr[\learn]^{\momexp-1} \exof*{\dnorm{\curr[\noise]}^{\momexp}}.
\end{align}
Note that since $\curr[\learn]$ is a nonincreasing sequence and since $\hreg$ is continuous on the compact set $\points$, we have
\begin{equation}
\sum_{\run=1}^\horizon \parens*{\dfrac{1}{\next[\learn]} - \dfrac{1}{\curr[\learn]}} \exof*{\hreg(\basealt) - \hreg(\next[\point])} \leq \sum_{\run=1}^\horizon \parens*{\dfrac{1}{\next[\learn]} - \dfrac{1}{\curr[\learn]}} \diamh = \dfrac{\diamh}{\learn_{\horizon+1}} - \dfrac{\diamh}{\learn_1}.
\end{equation}
Moreover, by optimality of $\point_1$,
\begin{equation}
\energy_1 = \frac{1}{\learn_1} \fench(\basealt, \learn_1 \dpoint_1) = \dfrac{1}{\learn_1} \parens*{ \hreg(\basealt) - \hreg(\point_1) - \braket*{\learn_1\dpoint_1}{\basealt - \point_1} } \leq \dfrac{\diamh}{\learn_1} - \braket*{\basealt - \point_1}{\dpoint_1}.
\end{equation}
Now, by the blanket assumptions made on the noise term $\curr[\noise]$ and by definition of the filtration $\curr[\filter]$, we have
\begin{equation}
\exof*{\dnorm{\curr[\noise]}^{\momexp}} = \exof*{ \exof*{ \dnorm{\err(\curr[\point];\curr[\seed])}^{\momexp} \given \curr[\filter] } } \leq \sdev^{\momexp}
\end{equation}
and
\begin{equation}
\exof*{ \braket*{\curr[\point] - \basealt}{\curr[\noise]} } = \exof*{ \exof*{ \braket*{ \curr[\point] - \basealt }{ \err(\curr[\point];\curr[\seed]) } \given \curr[\filter] } } = \exof*{ \braket*{ \curr[\point] - \basealt }{ \exof*{ \err(\curr[\point];\curr[\seed]) \given \curr[\filter]} } } = 0.
\end{equation}
All of these estimates ultimately lead to the bound
\begin{multline}
\exof*{ \sum_{\run=1}^\horizon \braket*{\curr[\point] - \basealt}{\nabla \obj(\curr[\point])}} \leq \braket*{\point_1 - \basealt}{\dpoint_1} + \obj(\point_1) - \min \obj \\ + \dfrac{\diamh}{\learn_{\horizon+1}} + \dfrac{2^{\momexp-1} \diamX^{2-\momexp}}{\momexp \hcst^{\momexp-1}} \sdev^{\momexp} \sum_{\run=1}^\horizon \curr[\learn]^{\momexp-1}
\eqcomma
\end{multline}
which concludes our proof.
\end{proof}
Specializing \cref{thm:discrete_convex_convergence} to convex objectives yields the following result, which generalizes \cref{thm:cvx-SDA} from the main text to arbitrary learning rates and arbitrary initial conditions.
\begin{theorem}[Convergence rate of \eqref{eq:SDA} for convex functions]
If $\obj$ is convex and relatively $\lips$-smooth, then for every nonincreasing learning rate sequence with $\learn_1 \leq \frac{1}{\momexp \lips}$ and for every $\sol \in \argmin\obj$, the iterates of \eqref{eq:SDA} satisfy
\begin{multline}
\exof*{ \obj(\avg{\point}_\horizon) - \min\obj} \leq \dfrac{\braket*{\point_1 - \sol}{\dpoint_1} + \obj(\point_1) - \min \obj}{\horizon} \\ + \dfrac{\diamh}{\horizon \learn_{\horizon+1}} + \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} \dfrac{\sum_{\run=1}^\horizon \curr[\learn]^{\momexp-1} }{\horizon}
\eqcomma
\end{multline}
where $\avg{\point}_\horizon = \frac{1}{\horizon}\sum_{\run=1}^\horizon \curr[\point]$.
In particular, the choice $\curr[\learn] = \beta \run^{-1/\momexp}$ with $\beta \leq \frac{1}{\momexp \lips}$ yields
\begin{multline}
\exof*{ \obj(\avg{\point}_\horizon) - \min\obj } \leq \dfrac{\braket*{\point_1 - \sol}{\dpoint_1} + \obj(\point_1) - \min \obj}{\horizon} \\ + \parens*{\dfrac{\diamh}{\beta} + \dfrac{2^{\momexp-1} \beta^{\momexp-1}\diamX^{2-\momexp} \sdev^{\momexp}}{\hcst^{\momexp-1}}} \horizon^{-\frac{\momexp-1}{\momexp}}.
\end{multline}
\end{theorem}
\begin{proof}
Due to $\obj$ being convex,
\begin{equation}
\sum_{\run=1}^\horizon \braket*{\curr[\point] - \sol}{\nabla \obj(\curr[\point])} \geq \sum_{\run=1}^\horizon \bracks*{ \obj(\curr[\point]) - \obj(\sol) } \geq \horizon \bracks*{ \obj(\avg{\point}(\horizon)) - \min \obj }
\eqcomma
\end{equation}
and the convergence rate for general learning rates is therefore an immediate consequence of \cref{thm:discrete_relative_smooth_cv}. The specialized bound for $\curr[\learn] = \beta \run^{-1/\momexp}$ is then established by using the upper bound $\sum_{\run=1}^\horizon \run^{-\frac{\momexp-1}{\momexp}} \leq \momexp \horizon^{1/\momexp}$ obtained from a standard integral estimation.
\end{proof}
\begin{remark}
For any precision $\precval > 0$, running \eqref{eq:SDA} with learning rate $\learn = \bigoh\parens*{(\precval/\sdev^{\momexp})^{\frac{1}{\momexp-1}}}$ yields $\exof*{\obj(\avg{\point}_\horizon) - \min \obj} \leq \precval$ for every $\horizon \geq \bigoh\parens*{(\sdev/\precval)^{\frac{\momexp}{\momexp-1}}}$, thus matching the complexity obtained in the continuous-time setting, \cf \cref{remark:complexity_rate_convex_continuous}.
\label{remark:complexity_rate_convex_discrete}
\end{remark}
\subsection{Strongly convex setting}
We now turn to the convergence analysis of \eqref{eq:SDA} for strongly convex objectives. Leveraging the upper bound in \cref{thm:discrete_convex_convergence}, we first obtain a convergence guarantee for the ergodic averages that holds for any strongly convex objective, independently of the choice of regularizer.
\begin{theorem}[Convergence rate of \eqref{eq:SDA} for strongly convex functions]
If $\obj$ is $\strong$-strongly convex with minimum $\sol \in \points$ and relatively $\lips$-smooth, then for every nonincreasing learning rate sequence with $\step_1 \leq \frac{1}{\momexp \lips}$, the iterates of \eqref{eq:SDA} satisfy
\begin{multline}
\exof*{ \norm{\avg{\point}_\horizon - \sol }^2} \leq 2\dfrac{\braket*{\point_1 - \sol}{\dpoint_1} + \obj(\point_1) - \min \obj}{\strong\horizon} \\ + \dfrac{2\diamh}{\strong\horizon \learn_{\horizon+1}} + \dfrac{2^{\momexp} \diamX^{2-\momexp} \sdev^{\momexp}}{\strong\momexp \hcst^{\momexp-1}} \dfrac{\sum_{\run=1}^\horizon \curr[\learn]^{\momexp-1} }{\horizon}.
\end{multline}
In particular, the choice $\curr[\learn] = \beta \run^{-1/\momexp}$ with $\beta \leq \frac{1}{\momexp \lips}$ yields
\begin{multline}
\exof*{ \norm{\avg{\point}_\horizon - \sol }^2} \leq 2\dfrac{\braket*{\point_1 - \sol}{\dpoint_1} + \obj(\point_1) - \min \obj}{\strong\horizon} \\ + \dfrac{2}{\strong}\parens*{\dfrac{\diamh}{\beta} + \dfrac{2^{\momexp-1} \beta^{\momexp-1}\diamX^{2-\momexp} \sdev^{\momexp}}{\hcst^{\momexp-1}}} \horizon^{-\frac{\momexp-1}{\momexp}}.
\end{multline}
\end{theorem}
\begin{proof}
By strong convexity of $\obj$,
\begin{align}
\sum_{\run=1}^\horizon \braket*{\curr[\point] - \sol}{\nabla \obj(\curr[\point])} \geq \dfrac{\strong}{2} \sum_{\run=1}^\horizon \norm{\curr[\point] - \sol}^2 \geq \dfrac{\strong\horizon}{2} \norm{\avg{\point}_\horizon - \sol}^2,
\end{align}
which directly yields the results by \cref{thm:discrete_relative_smooth_cv}.
\end{proof}
\begin{remark}
Similarly to \cref{remark:complexity_rate_convex_discrete}, we get that for every $\precval > 0$, the learning rate $\learn = \bigoh\parens*{(\precval/\sdev^{\momexp})^{\frac{1}{\momexp-1}}}$ achieves the accuracy $\exof*{ \norm{\avg{\point}_\horizon - \sol }^2} \leq \precval$ for all horizon $\horizon \geq \bigoh\parens*{(\sdev/\precval)^{\frac{\momexp}{\momexp-1}}}$.
\label{remark:complexity_rate_strongly_convex_discrete}
\end{remark}
We now proceed by estimating the fraction of time that \eqref{eq:SDA} spends away from $\argmin \obj$, and the time that it takes to get close to it for the first time. These results specialize to \cref{thm:occmeas-disc,thm:hitting-disc} from the main text when the algorithms are initialized to the prox-center $\point_c = \argmin \hreg$ of $\points$.
\begin{theorem}[Concentration around global minimum]
Assume that $\obj$ is $\strong$-strongly convex with minimum $\sol$ and $\lips$-relatively smooth. Let
\begin{equation}
\meas_\horizon(\nhd_\precdist) = \dfrac{1}{\horizon} \sum_{\run = 1}^\horizon \one\braces{\curr[\point] \in \nhd_\precdist}
\end{equation}
be the fraction of time that $\curr[\point]$ spends in $\nhd_\precdist = \setdef{\point \in \points}{\norm{\point - \sol} \leq \precdist}$ up to time $\horizon$. If \eqref{eq:SDA} is run with constant learning rate $\curr[\learn] \equiv \learn$, we have
\begin{equation}
\exof*{\meas_\horizon(\nhd_\precdist)} \geq 1 - \dfrac{2 \braket*{\point_1 - \sol}{\dpoint_1}}{\strong \precdist^2 \horizon} - \dfrac{2\big( \obj(\point_1) - \min\obj\big)}{\strong \precdist^2 \horizon} - \dfrac{2 \diamh}{\learn \strong \precdist^2 \horizon} - \dfrac{2^\momexp \learn^{\momexp-1} \diamX^{2-\momexp} \sdev^\momexp}{\strong \momexp \hcst^{\momexp-1} \precdist^2}.
\end{equation}
\end{theorem}
\begin{proof}
Markov's inequality together with the strong convexity of $\obj$ yield
\begin{align}
\exof*{\meas_\horizon(\points \setminus \nhd_\precdist)}
&= \dfrac{1}{\horizon} \sum_{\run=1}^\horizon \probof{ \norm{\curr[\point] - \sol} > \precdist }
\notag\\
&\leq \dfrac{1}{\horizon} \sum_{\run=1}^\horizon \dfrac{\exof*{\norm{\curr[\point] - \sol}^2}}{\precdist^2}
\leq \dfrac{2}{\strong \precdist^2 \horizon} \exof*{\sum_{\run=1}^\horizon \braket*{\curr[\point] - \sol}{\nabla \obj(\curr[\point])}}
\eqcomma
\end{align}
and the concentration bound therefore follows directly from \cref{thm:discrete_relative_smooth_cv}.
\end{proof}
\begin{theorem}[Hitting time estimate]
Assume that $\obj$ is $\strong$-strongly convex with minimum $\sol \in \points$ and relatively $\lips$-smooth. Let
\begin{equation}
\hit_\precdist = \inf\setdef{\time \in \N}{\norm{\curr[\point] - \sol} \leq \precdist}
\end{equation}
denote the first time $\curr[\point]$ gets within $\precdist$ of $\sol$ for some precision $\precdist > 0$. If \eqref{eq:SDA} is run with constant learning rate
\begin{equation}
\learn < \parens*{ \dfrac{ \strong \momexp \hcst }{ 2^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \sdev^{\momexp} } }^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}}
\eqcomma
\end{equation}
then the average value of $\hit_\precdist$ is bounded as
\begin{equation}
\exof*{\hit_\precdist} \leq \dfrac{ 2\momexp\hcst \fench(\sol, \learn \dpoint_1) + 2\momexp\hcst\big( \obj(\point_1) - \min\obj + \frac{\strong \precdist^2}{2} \big) \learn }{ \strong \momexp \hcst \precdist^2 \learn - 2^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \sdev^\momexp \learn^\momexp }.
\label{eq:discrete_hitting_time_1}
\end{equation}
In particular, running \eqref{eq:SDA} with $\dpoint_1 = 0$ and
\begin{equation}
\learn = \parens*{ \dfrac{ \strong \hcst }{ 2^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \sdev^{\momexp} } }^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}}
\end{equation}
yields the hitting-time estimate
\begin{equation}
\exof*{\hit_\precdist} \leq \dfrac{\momexp}{\momexp-1} \braces*{ \dfrac{2\diamh}{\strong\hcst} \parens*{ \dfrac{2^\momexp \diamX^{2-\momexp} \sdev^\momexp}{\strong} }^{\frac{1}{\momexp-1}} \precdist^{-\frac{2\momexp}{\momexp-1}} + \dfrac{2\big( \obj(\point_1) - \min \obj \big)}{\strong} \precdist^{-2} + 1 }.
\label{eq:discrete_hitting_time_2}
\end{equation}
\end{theorem}
\begin{remark}
Since $\dpoint_1 = 0$, the quantity $\obj(\point_1) - \min \obj$ in \cref{eq:discrete_hitting_time_2} is in fact equal to $\obj(\argmin \hreg) - \obj(\sol)$, that is, to the value difference between the \emph{prox-center} $\point_c = \argmin \hreg$ and the {minimizer} $\sol$ of $\obj$.
\end{remark}
\begin{proof}
Since the learning rate $\learn$ is assumed constant, summing the equality of \cref{lemma:difference_of_V} over $\run=1, \dots, \horizon$ and invoking the upper bound of \cref{lemma:heavy_tails_descent} yields
\begin{align}
0
& \leq \energy_\horizon \notag \\
& \leq \energy_1 - \sum_{\run=1}^\horizon \braket*{\curr[\point] - \sol}{\nabla \obj(\curr[\point])} + \obj(\point_1) - \min\obj \\
- \,\, & \sum_{\run=1}^\horizon \braket*{\curr[\point] - \sol}{\curr[\noise]} + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp} \learn^{\momexp-1}}{\momexp \hcst^{\momexp-1}} \sum_{\run=1}^\horizon \dnorm{\curr[\noise]}^\momexp.
\end{align}
By analogy with the arguments used to prove the hitting time estimate for continuous \eqref{eq:LMF} dynamics (\cf \cref{thm:hittingtime}), we then apply the above inequality to the bounded stopping times $\horizon \leftarrow \hit_\precdist \wedge \horizon$ and take the expectation on both sides, which leads to
\begin{align}
0
\leq \energy_1
&+ \obj(\point_1) - \min\obj -
\exof*{ \sum_{\run=1}^{\hit_\precdist \wedge \horizon} \braket*{\curr[\point] - \sol}{\nabla \obj(\curr[\point])} }
\notag\\
&+ \dfrac{2^{\momexp-1} \diamX^{2 - \momexp} \learn^{\momexp-1}}{\momexp \hcst^{\momexp-1}} \exof*{ \sum_{\run=1}^{\hit_\precdist \wedge \horizon} \dnorm{\curr[\noise]}^\momexp} - \exof*{\sum_{\run=1}^{\hit_\precdist \wedge \horizon} \braket*{\curr[\point] - \sol}{\curr[\noise]}}.
\end{align}
Before proceeding further, recall that $\filter_\run$ is the filtration generated by $\point_\runalt$, $1 \leq \runalt \leq \run$, and that $\curr[\noise]$ is a martingale difference sequence. In particular, $\curr[\noise]$ is $\filter_{\run+1}$-measurable and $\exof{\curr[\noise] \given \curr[\filter]} = 0$.
By the tower property of conditional expectations, we can therefore write
\begin{align}
\exof*{ \sum_{\run=1}^{\hit_\precdist \wedge \horizon} \braket*{\curr[\point] - \sol}{\curr[\noise]} }
&= \sum_{\run=1}^\horizon \exof*{ \one\braces{\hit_\precdist \geq \run} \braket*{\curr[\point] - \sol}{\curr[\noise]} }
\notag\\
&= \sum_{\run=1}^{\horizon} \exof*{ \exof*{ \one\braces{\hit_\precdist \geq \run} \braket*{\curr[\point] - \sol}{\curr[\noise]} \given \curr[\filter] } }\notag\\
&= \sum_{\run=1}^\horizon \exof*{ \one\braces{\hit_\precdist \geq \run} \braket*{ \curr[\point] - \sol }{ \exof*{ \curr[\noise] \given \curr[\filter] } } }
= 0,
\end{align}
where we have used that both $\curr[\point]$ and $\one\braces{\hit_\precdist \geq \run}$ are $\curr[\filter]$-measurable. Indeed, to prove the latter, one notes that
\begin{equation}
\braces{\hit_\precdist \geq \run} = \intersect_{\runalt=1}^{\run-1} \braces{ \norm{\point_\runalt - \sol} > \precdist } \in \curr[\filter]
\end{equation}
since all events $\braces{\norm{\point_\runalt - \sol} > \precdist}$, $1 \leq \runalt \leq \run - 1$, belong to $\curr[\filter]$. Similarly, the bounded $\momexp$-th central moment condition satisfied by $\curr[\noise]$ yields
\begin{align}
\exof*{ \sum_{\run=1}^{\hit_\precdist \wedge \horizon } \dnorm{\curr[\noise]}^\momexp}
&= \sum_{\run=1}^\horizon \exof*{ \one\braces{\hit_\precdist \geq \run} \dnorm{\noise}^\momexp }
\notag\\
&= \sum_{\run=1}^\horizon \exof*{ \one\braces{\hit_\precdist \geq \run} \exof*{ \dnorm{\noise}^\momexp \given \curr[\filter] } }
\leq \sdev^\momexp \exof*{ \sum_{\run=1}^{\hit_\precdist \wedge \horizon } 1 }
= \sdev^\momexp \exof*{\hit_\precdist \wedge \horizon}.
\end{align}
Moreover, by strong convexity of $\obj$ and definition of $\hit_\precdist$, we also get
\begin{align}
\sum_{\run=1}^{\hit_\precdist \wedge \horizon} \braket*{ \curr[\point] - \sol }{\nabla \obj(\curr[\point])}
&\geq \dfrac{\strong}{2} \sum_{\run=1}^{\hit_\precdist \wedge \horizon} \norm{\curr[\point] - \sol}^2
\notag\\
&\geq \dfrac{\strong}{2} \sum_{\run=1}^{\hit_\precdist \wedge \horizon - 1} \norm{\curr[\point] - \sol}^2
% \notag\\
\geq \dfrac{\strong}{2} \sum_{\run=1}^{\hit_\precdist \wedge \horizon - 1} \precdist^2 = \dfrac{\strong \precdist^2}{2} (\hit_\precdist \wedge \horizon - 1).
\end{align}
The previous computations lead to
\begin{align}
0 &\leq \energy_1 + \obj(\point_1) - \min\obj - \dfrac{\strong \precdist^2}{2} ( \exof*{\hit_\precdist \wedge \horizon} - 1 ) + \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \learn^{\momexp-1} \sdev^\momexp}{\momexp \hcst^{\momexp-1}} \exof*{\hit_\precdist \wedge \horizon}
\notag\\
&= \energy_1 + \obj(\point_1) - \min\obj + \dfrac{\strong \precdist^2}{2} - \parens*{ \dfrac{\strong \precdist^2}{2} - \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp} \learn^{\momexp-1}}{\momexp \hcst^{\momexp-1}} } \exof*{\hit_\precdist \wedge \horizon}
\eqcomma
\end{align}
where the multiplicative coefficient in front of $\exof*{\tau_\precdist \wedge \horizon}$ is (strictly) positive whenever the learning rate $\learn$ satisfies
\begin{equation}
\learn < \parens*{ \dfrac{ \strong \momexp \hcst }{ 2^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \sdev^{\momexp} } }^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}}.
\end{equation}
Rearranging the terms therefore yields the upper bound
\begin{align}
\exof*{\hit_\precdist \wedge \horizon}
&\leq \dfrac{ \energy_1 + \obj(\point_1) - \min\obj + \frac{\strong\precdist^2}{2} }{ \frac{\strong\precdist^2}{2} - \frac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp} \learn^{\momexp-1}}{\momexp \hcst^{\momexp-1}} }
\notag\\
&= \dfrac{ 2\momexp\hcst \fench(\sol, \learn \dpoint_1) + 2\momexp\hcst\big( \obj(\point_1) - \min\obj + \frac{\strong \precdist^2}{2} \big) \learn }{ \strong \momexp \hcst \precdist^2 \learn - 2^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \sdev^\momexp \learn^\momexp }
\eqcomma
\end{align}
which proves \cref{eq:discrete_hitting_time_1} when taking the monotone limit as $\horizon \to \infty$.
Now let us assume $\dpoint_1 = 0$ so that $\fench(\sol, \learn \dpoint_1) = \hreg(\sol) - \hreg(\point_1) \leq \diamh$, and choose
\begin{equation}
\learn = \parens*{ \dfrac{ \strong \hcst }{ 2^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \sdev^{\momexp} } }^{\frac{1}{\momexp-1}} \precdist^{\frac{2}{\momexp-1}}
\eqcomma
\end{equation}
which is the learning rate maximizing the denominator of \cref{eq:discrete_hitting_time_1}.
For notational convenience, we also define
\begin{align}
c_1 = 2 \momexp \hcst \diamh, \,\, c_2 = 2\momexp \hcst \big( \obj(\point_1) - \min\obj + \frac{\strong\precdist^2}{2} \big), \,\, c_3 = \strong \momexp \hcst \precdist^2, \,\, c_4 = (2 \sdev)^\momexp (\hcst \diamX)^{2-\momexp}.
\notag
\end{align}
In particular $\learn = [ c_3 / (\momexp c_4) ]^{1/(\momexp-1)}$ and we therefore obtain from \cref{eq:discrete_hitting_time_1} that
\begin{align}
\exof*{\hit_\precdist}
&\leq \dfrac{c_1 + c_2 \learn}{c_3 \learn - c_4 \learn^\momexp} \\
&= \bracks*{ c_3 \parens*{ \dfrac{c_3}{\momexp c_4} }^{\frac{1}{\momexp-1}} - c_4 \parens*{ \dfrac{c_3}{\momexp c_4} }^{\frac{\momexp}{\momexp-1}} }^{-1} \bracks*{ c_1 + c_2 \parens*{ \dfrac{c_3}{\momexp c_4} }^{\frac{1}{\momexp-1}} }\notag\\
&= \bracks*{ \parens*{ \dfrac{c_3^\momexp}{\momexp c_4} }^{\frac{1}{\momexp-1}} \big( 1 - \frac{1}{\momexp} \big) }^{-1} \bracks*{ c_1 + c_2 \parens*{ \dfrac{c_3}{\momexp c_4} }^{\frac{1}{\momexp-1}} }\notag\\
&= \dfrac{\momexp}{\momexp-1} \parens*{ \dfrac{\momexp c_4}{c_3^\momexp} }^{\frac{1}{\momexp-1}} \bracks*{ c_1 + c_2 \parens*{ \dfrac{c_3}{\momexp c_4} }^{\frac{1}{\momexp-1}} }\notag\\
&= \dfrac{\momexp}{\momexp-1} \braces*{ c_1 \parens*{ \dfrac{\momexp c_4}{c_3^\momexp} }^{\frac{1}{\momexp-1}} + \dfrac{c_2}{c_3} }\notag\\
&= \dfrac{\momexp}{\momexp-1} \braces*{ 2 \momexp \hcst \diamh \parens*{ \dfrac{\momexp 2^\momexp \hcst^{2-\momexp} \diamX^{2-\momexp} \sdev^\momexp }{\strong^\momexp \momexp^\momexp \hcst^\momexp \precdist^{2\momexp}} }^{\frac{1}{\momexp-1}} + \dfrac{2 \momexp \hcst \big( \obj(\point_1) - \min\obj + \frac{\strong \precdist^2}{2} \big)}{\strong \momexp \hcst \precdist^2} }\notag\\
&= \dfrac{\momexp}{\momexp-1} \braces*{ \dfrac{2\diamh}{\strong\hcst} \parens*{ \dfrac{2^\momexp \diamX^{2-\momexp} \sdev^\momexp}{\strong} }^{\frac{1}{\momexp-1}} \precdist^{-\frac{2\momexp}{\momexp-1}} + \dfrac{2\big( \obj(\point_1) - \min \obj \big)}{\strong} \precdist^{-2} + 1 }
\eqcomma
\end{align}
which concludes our proof.
\end{proof}
We next consider objectives whose convexity is adapted to the geometry induced by the regularizer, namely objectives that are \emph{strongly convex relative to $\hreg$}. Specifically, we assume that
\begin{equation}
\obj(\pointalt)
\geq \obj(\point)
+ \braket{\subsel\obj(\point)}{\pointalt - \point}
+ \strong \breg(\pointalt,\point)
\end{equation}
for all $\point\in\proxdom$ and all $\pointalt\in\points$.
We begin by establishing a convergence guarantee for constant learning rates, yielding a discrete-time analogue of \cref{thm:relatively_convex_continuous1}. The result also slightly refines \cref{thm:str-SDA} from the main text, as it no longer requires \eqref{eq:SDA} to be initialized at the prox-center of $\points$.
\begin{theorem}[Convergence rate of \eqref{eq:SDA} for relatively strongly convex functions, constant learning rate]
Let $\hreg$ be a steep regularizer. If $\obj$ is relatively $\lips$-smooth and relatively $\strong$-strongly convex with minimum $\sol \in \points$, then running \eqref{eq:SDA} with a constant learning rate $\learn \leq \frac{1}{\momexp \lips}$ achieves
\begin{equation}
\exof*{\norm{\curr[\point] - \sol}^2} \leq \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \strong \hcst^{\momexp}} \sdev^{\momexp} \learn^{\momexp - 1} + \dfrac{2 \breg(\sol, \point_1)}{\hcst} (1 - \learn \strong)^{\run - 1}.
\end{equation}
\label{thm:cv_relative_strong_discrete}
\end{theorem}
\begin{proof}
Starting from \cref{lemma:difference_of_V,lemma:heavy_tails_descent} with $\basealt = \sol$ and then using the relative strong convexity of $\obj$, we readily get
\begin{align}
\exof*{\next[\energy]}
&\leq \exof*{\curr[\energy]}
+ \exof*{\braket*{\sol - \curr[\point]}{\nabla \obj(\curr[\point])}}
\notag\\
&\qquad
+ \exof*{ \obj(\curr[\point]) - \obj(\next[\point]) } + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \hcst^{\momexp-1}} \learn^{\momexp-1} \sdev^{\momexp}
\notag\\
&\leq \exof*{\curr[\energy]} + \exof*{ \obj(\sol) - \obj(\curr[\point]) } - \strong\exof*{ \breg(\sol, \curr[\point]) }
\notag\\
&\qquad
+ \exof*{ \obj(\curr[\point]) - \obj(\next[\point]) } + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \hcst^{\momexp-1}} \learn^{\momexp-1} \sdev^{\momexp}.
\end{align}
But since $\hreg$ is steep, we have $\learn \curr[\energy] = \fench(\sol, \learn \curr[\dpoint]) = \breg(\sol, \curr[\point])$ for all times $\run \geq 1$, and thus
\begin{align}
\exof*{\breg(\sol, \next[\point])}
&\leq (1 - \learn \strong) \exof*{\breg(\sol, \curr[\point])} + \learn \exof*{ \obj(\sol) - \obj(\next[\point]) } + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \hcst^{\momexp-1}} \learn^{\momexp} \sdev^{\momexp}
\notag\\
&\leq (1 - \learn \strong) \exof*{\breg(\sol, \curr[\point])} + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \hcst^{\momexp-1}} \learn^{\momexp} \sdev^{\momexp}
\eqcomma
\end{align}
where we have used in the second inequality that $\sol$ is the minimum of $\obj$. Note in particular that $\learn \strong \leq 1$ due to the assumption that $\learn \leq 1/(\momexp \lips)$ and the fact that $\strong \leq \lips$ (\cf \citep[Proposition~1.1]{LFN18} ). By recurrence, we therefore get
\begin{equation}
\exof*{\breg(\sol, \curr[\point])} \leq (1 - \learn \strong)^{\run - 1} \breg(\sol, \point_1) + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \strong \hcst^{\momexp-1}} \learn^{\momexp-1} \sdev^{\momexp}
\eqcomma
\end{equation}
and it only remains to use the lower bound $\norm{\point - \point'}^2 \leq \frac{2}{\hcst} \breg(\point, \point')$ of \cref{prop:Fench} to obtain the claimed inequality.
\end{proof}
\begin{remark}
Due to \cref{thm:cv_relative_strong_discrete}, we note that the learning rate $\learn = \bigoh\parens*{(\precval/\sdev^{\momexp-1})^{\frac{1}{\momexp}}}$ for any accuracy $\precval > 0$ achieves the last iterate estimate $\exof*{ \norm{\curr[\point] - \sol}^2 } \leq \precval$ for all times $\time \geq \Omega\parens*{\sdev^{\frac{\momexp}{\momexp-1}} \precval^{-\frac{1}{\momexp-1}} \log(1/\precval)}$. This complexity rate is better than the one obtained for convex or strongly convex functions, and, as mentioned in the continuous-time section, this also recovers the optimal rate known for first-order stochastic methods when $\momexp=2$.
\end{remark}
Finally, when the learning rate sequence is proportional to $\run^{-1/\momexp}$, we leverage relative strong convexity together with a discrete-time Grönwall inequality to obtain the following convergence guarantee:
\begin{theorem}[Convergence rate of \eqref{eq:SDA} for relatively strongly convex functions, decreasing learning rate]
Let $\hreg$ be a steep regularizer. If $f$ is relatively $\lips$-smooth and relative $\strong$-strongly convex with minimum $\sol \in \points$, then running \eqref{eq:SDA} with learning rate $\curr[\learn] = \beta \run^{-1/\momexp}$ where $\beta \leq \frac{1}{\momexp \lips}$ achieves
\begin{align}
\exof*{\norm{\curr[\point] - \sol}^2}
&\leq \dfrac{2}{\strong \hcst \momexp} \parens*{ \dfrac{\diamh}{\beta} + \dfrac{2^{\momexp - 1} \beta^{\momexp -1} \diamX^{2 - \momexp}}{\hcst^{\momexp-1}} \sdev^{\momexp} } \run^{-\frac{\momexp-1}{\momexp}}
\notag\\
&+ \dfrac{2 \breg(\sol, \point_1)}{\hcst} \run^{-\frac{1}{\momexp}} \exp\bracks*{ - \dfrac{\strong \beta \momexp}{\momexp - 1} \parens*{ \run^{\frac{\momexp - 1}{\momexp}} - 1 } }.
\end{align}
In particular, $\curr[\point] \to \sol$ in mean square as $\time \to \infty$.
\label{thm:discrete_relative_strong_variable_learning_rate}
\end{theorem}
\begin{proof}
Following the same idea as in the proof of \cref{thm:cv_relative_strong_discrete} but with learning rate $\curr[\learn] = \beta \run^{-1/\momexp}$, we obtain
\begin{align}
\exof*{\next[\energy]}
&\leq \exof*{\curr[\energy]} + \parens*{\dfrac{1}{\next[\learn]} - \dfrac{1}{\curr[\learn]}} \diamh - \strong \exof*{\breg(\sol, \curr[\point])} + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \hcst^{\momexp-1}} \learn^{\momexp-1} \sdev^{\momexp}
\notag\\
&= (1 - \strong \curr[\learn])\exof*{\curr[\energy]} + \parens*{(\run+1)^{1/\momexp} - \run^{1/\momexp}} \dfrac{\diamh}{\beta} + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \hcst^{\momexp-1}} \learn^{\momexp-1} \sdev^{\momexp}.
\end{align}
Let $\psi(x) = x^{1/\momexp}$. Then, by the mean value theorem, there exists $\xi \in (\run, \run+1)$ such that
\begin{equation}
(\run+1)^{1/\momexp} - \run^{1/\momexp} = \psi(\run+1) - \psi(\run) \leq \psi'(\xi) = \frac{1}{\momexp} \xi^{-\frac{\momexp-1}{\momexp}} \leq \frac{1}{\momexp} \run^{-\frac{\momexp-1}{\momexp}} = \frac{1}{\momexp \beta^{\momexp-1}} \curr[\learn]^{\momexp-1}.
\end{equation}
This yields
\begin{equation}
\exof*{\next[\energy]} \leq (1-\strong \curr[\learn]) \exof*{\curr[\energy]} + \dfrac{1}{\momexp} \parens*{ \dfrac{\diamh}{\beta^{\momexp}} + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp} \sdev^{\momexp}}{\hcst^{\momexp-1}} } \curr[\learn]^{\momexp-1}.
\end{equation}
Letting $C$ be the constant in front of $\curr[\learn]^{\momexp-1}$ and iterating the inequality for all $\run = 1, \dots, \horizon-1$ then leads to
\begin{equation}
\exof*{\energy_\horizon} \leq \energy_1 \prod_{\run=1}^{\horizon-1} (1 - \strong \curr[\learn]) + C \sum_{\run=1}^{\horizon-1} \curr[\learn]^{\momexp-1} \prod_{\runalt = \run + 1}^{\horizon-1} (1 - \strong \learn_\runalt) \eqdef \energy_1 S_{1, \horizon} + C S_{2, \horizon}.
\label{eq:upperbound_on_V_discrete}
\end{equation}
To estimate $S_{1, \horizon}$, we use the classic inequality $1 - x \leq e^{-x}$ and an integral bound to obtain
\begin{align}
S_{1, \horizon}
&\leq \prod_{\run=1}^{\horizon-1} e^{-\strong \curr[\learn]}
= \exp\parens*{- \strong \sum_{\run=1}^{\horizon-1} \curr[\learn]}
= \exp\parens*{ -\beta \strong \sum_{\run=1}^{\horizon-1} \run^{-1/\momexp} }
\notag\\
&\leq \exp\parens*{ - \beta \strong \int_1^\horizon \time^{-1/\momexp} d\time }
= \exp\parens*{ - \dfrac{\strong \beta \momexp}{\momexp-1} \parens*{ \horizon^{\frac{\momexp-1}{\momexp}} - 1 }}.
\end{align}
On the other hand, note that $\curr[\learn]^{\momexp-1} = \curr[\learn]^{\momexp-2} \curr[\learn] \leq \learn_{\horizon}^{\momexp-2} \curr[\learn]$ for every $\run=1, \dots, \horizon-1$ due to $\momexp \leq 2$ and the fact that $\curr[\learn]$ is a nonincreasing sequence. This allows us to write
\begin{align}
S_{2, \horizon}
\leq \learn_{\horizon}^{\momexp-2} \sum_{\run=1}^{\horizon-1} \curr[\learn] \prod_{\runalt = \run + 1}^{\horizon-1} (1 - \strong \learn_{\runalt})
&= \dfrac{\learn_{\horizon}^{\momexp-2}}{\strong} \sum_{\run=1}^{\horizon-1}(1 + \strong \curr[\learn] - 1) \prod_{\runalt = \run + 1}^{\horizon - 1} (1 - \strong \learn_\runalt)
\notag\\
&= \dfrac{\learn_{\horizon}^{\momexp-2}}{\strong} \sum_{\run = 1}^{\horizon-1} \parens*{ \prod_{\runalt=\run+1}^{\horizon-1} (1 - \strong \learn_{\runalt}) - \prod_{\runalt=\run}^{\horizon-1} (1 - \strong \learn_{\runalt}) }
\notag\\
&=\dfrac{\learn_{\horizon}^{\momexp-2}}{\strong} \parens*{ 1 - \prod_{\runalt=1}^{\horizon-1} (1 - \strong \learn_\runalt) }
\notag\\
&\leq \dfrac{\learn_{\horizon}^{\momexp-2}}{\strong}.
\end{align}
Injecting the bounds for $S_{1,\horizon}$ and $S_{2,\horizon}$ back into \cref{eq:upperbound_on_V_discrete} and replacing $\energy_{\horizon}$ by its expression, it follows that
\begin{align}
\exof*{\breg(\sol, \curr[\state])}
= \curr[\learn] \exof*{\curr[\energy]}
&\leq \dfrac{\breg(\sol, \state_0)}{\run^{1/\momexp}} \exp\parens*{ - \dfrac{\strong \beta \momexp}{\momexp-1} \parens*{ \run^{\frac{\momexp-1}{\momexp}} - 1 }}
\notag\\
&+ \dfrac{1}{\strong \momexp} \parens*{ \dfrac{\diamh}{\beta} + \dfrac{2^{\momexp - 1} \beta^{\momexp -1} \diamX^{2 - \momexp}}{\hcst^{\momexp-1}} \sdev^{\momexp} } \run^{-\frac{\momexp-1}{\momexp}}
\eqcomma
\end{align}
and the claim is therefore proved by using the inequality $\norm{\sol - \curr[\state]}^2 \leq \frac{2}{\hcst} \breg(\sol, \curr[\state])$.
\end{proof}
\begin{remark}
Analogous to its associated continuous-time result (\cref{thm:relatively_convex_continuous2}), \cref{thm:discrete_relative_strong_variable_learning_rate} only implies that \eqref{eq:SDA} achieves a complexity rate of order $\time = \Omega\parens*{\precval^{-\frac{\momexp}{\momexp-1}}}$ for learning rates $\learn(\time) \propto \time^{-1/\momexp}$, which thus does not reach the optimal lower bound even for the tame regime $\momexp=2$.
\end{remark}
\subsection{Extension: Lazy Mirror Descent \eqref{eq:LMD} under heavy-tailed noise}
In this section, we extend our previous results to the stochastic mirror descent with lazy updates, described by the system
\begin{equation}
\tag{LMD}
\begin{aligned}
\next[\dpoint]
&= \curr[\dpoint]
- \curr[\step]\curr[\sgrad]
\\
\next[\point]
&= \mirror(\next[\dpoint])
\end{aligned}
\end{equation}
Following the discussion in the main text, the key difference between \eqref{eq:LMD} and \eqref{eq:SDA} is that, in \eqref{eq:LMD}, gradient steps are \emph{pre-multiplied} by a stepsize $\curr[\step]$, whereas in \eqref{eq:SDA} they are multiplied by a learning rate $\curr[\learn]$ \emph{after} aggregation.
As will become clear in the proofs, the convergence results for \eqref{eq:LMD} under heavy-tailed noise rely on the same arguments as those used for \eqref{eq:SDA} with a constant learning rate. In particular, we no longer need to introduce the $\learn$-deflated Fenchel coupling and can instead directly analyze the evolution of the Fenchel coupling along the iterates of \eqref{eq:LMD}:
\begin{lemma}[Three-point identity]
For every $\basealt \in \points$, the iterates of \eqref{eq:LMD} satisfy
\begin{equation}
\fench(\basealt, \next[\dpoint]) - \fench(\basealt, \curr[\dpoint]) = \braket*{\curr[\point] - \basealt}{\next[\dpoint] - \curr[\dpoint]} + \braket*{ \next[\point] - \curr[\point] }{ \next[\dpoint] - \curr[\dpoint] } - \fench(\next[\point], \curr[\dpoint]).
\end{equation}
\label{lemma:LMD_diff_of_F}
\end{lemma}
\begin{proof}
On one hand, we have
\begin{equation}
\fench(\next[\point], \curr[\dpoint]) - \fench(\basealt, \curr[\dpoint]) = \hreg(\next[\point]) - \hreg(\basealt) - \braket*{\next[\point] - \basealt}{\curr[\point]}
\eqcomma
\end{equation}
and, on the other one, using the optimality of $\next[\point]$,
\begin{equation}
\fench(\basealt, \next[\dpoint]) = \hreg(\basealt) + \hconj(\next[\dpoint]) - \braket*{\basealt}{\next[\dpoint]} = \hreg(\basealt) - \hreg(\next[\point]) + \braket*{\next[\point] - \basealt}{\next[\dpoint]}.
\end{equation}
Combining both of these equalities therefore yields
\begin{align}
\fench(\basealt, \next[\dpoint]) + \fench(\next[\point], \curr[\dpoint]) - \fench(\basealt, \curr[\dpoint])
&= \braket*{\next[\point] - \basealt}{\next[\dpoint] - \curr[\dpoint]}
\notag\\
&= \braket*{\curr[\point] - \basealt}{\next[\dpoint] - \curr[\dpoint]} + \braket*{\next[\point] - \curr[\point]}{\next[\dpoint] - \curr[\dpoint]}
\eqcomma
\end{align}
hence proving the claim.
\end{proof}
Analogously to \eqref{eq:SDA}, leveraging the relative smoothness of the objective and the moment bounds on the gradient noise leads to the following essential estimate:
\begin{lemma}[Heavy-tailed noise estimate]
If $\obj$ is relatively $\lips$-smooth and $\curr[\step] \leq \frac{1}{\momexp \lips}$, then trajectories of \eqref{eq:LMD} satisfy
\begin{equation}
\braket*{\next[\point] - \curr[\point]}{\next[\dpoint] - \curr[\dpoint]} - \fench(\next[\point], \curr[\dpoint]) \leq \curr[\step] \bracks*{ \obj(\curr[\point]) - \obj(\next[\point]) } + \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \hcst^{\momexp-1}} \curr[\step]^{\momexp} \dnorm{\curr[\noise]}^{\momexp}.
\end{equation}
\label{lemma:LMD_heavy_descent}
\end{lemma}
\begin{proof}
Since $\next[\dpoint] - \curr[\dpoint] = - \curr[\step] \curr[\sgrad] = - \curr[\step](\nabla \obj(\curr[\point]) + \curr[\noise])$, we have
\begin{align}
\dfrac{1}{\curr[\step]} \bracks*{ \braket*{ \next[\point] - \curr[\point] }{\next[\dpoint] - \curr[\dpoint]} - \fench(\next[\point], \curr[\dpoint])} &= \braket*{\nabla \obj(\curr[\point])}{\curr[\point] - \next[\point]} - \dfrac{1}{\momexp \curr[\step]} \fench(\next[\point], \curr[\dpoint])
\notag\\
&+ \braket*{\curr[\noise]}{\curr[\point] - \next[\point]} - \dfrac{\momexp-1}{\momexp \curr[\step]} \fench(\next[\point], \curr[\dpoint]),
\end{align}
and the rest of the proof can therefore be carried on by invoking the same computations as in \cref{lemma:heavy_tails_descent}.
\end{proof}
From the two lemmas above, we obtain the following theorem, providing the analogue of \cref{thm:discrete_convex_convergence} for \eqref{eq:LMD}.
\begin{theorem}
If $\obj$ is relative $\lips$-smooth, then for every $\basealt \in \points$ and every nonincreasing stepsize sequence with $\step_1 \leq \frac{1}{\momexp \lips}$, the iterates of \eqref{eq:LMD} satisfy
\begin{equation}
\exof*{ \sum_{\run=1}^\horizon \curr[\step] \braket*{ \curr[\point] - \basealt }{ \nabla \obj(\curr[\point]) }} \leq \fench(\basealt, \dpoint_1) + \step_1 \bracks*{ \obj(\point_1) - \min\obj } + \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} \sum_{\run=1}^{\horizon} \curr[\step]^{\momexp}.
\end{equation}
\end{theorem}
\begin{proof}
After rearranging the terms, \cref{lemma:LMD_diff_of_F,lemma:LMD_heavy_descent} yield
\begin{equation}
\curr[\step] \braket*{\curr[\point] - \basealt}{\curr[\sgrad]} \leq \fench(\basealt, \curr[\dpoint]) - \fench(\basealt, \next[\dpoint]) + \curr[\step] \bracks*{ \obj(\curr[\point]) - \obj(\next[\point])} + \dfrac{2^{\momexp-1} \diamX^{2-\momexp}}{\momexp \hcst^{\momexp-1}} \curr[\step]^{\momexp} \dnorm{\curr[\noise]}^{\momexp}.
\end{equation}
Summing over $\run=1, \dots, \horizon$ and taking the overall expectation, we therefore get
\begin{align}
\exof*{ \sum_{\run=1}^\horizon \curr[\step] \braket*{\curr[\point] - \basealt}{\nabla \obj(\curr[\point])}}
\leq \fench(\basealt, \dpoint_1) - \fench(\basealt, \next[\dpoint])
&+ \exof*{ \sum_{\run=1}^\horizon \curr[\step] \bracks*{ \obj(\curr[\point]) - \obj(\next[\point]) } }
\notag\\
&+ \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} \sum_{\run=1}^\horizon \curr[\step]^{\momexp}
\eqcomma
\end{align}
where we have used the blanket assumptions on $\curr[\noise]$ as in the proof of \cref{thm:discrete_relative_smooth_cv}. Now, note that we can write
\begin{align}
\sum_{\run=1}^\horizon \curr[\step] \bracks*{ \obj(\curr[\point]) - \obj(\next[\point]) }
&= \sum_{\run=0}^{\horizon-1} \next[\step] \obj(\next[\point]) - \sum_{\run=1}^\horizon \curr[\step] \obj(\next[\point])
\notag\\
&= \step_1 \obj(\point_1) - \step_\horizon \obj(\point_{\horizon+1}) + \sum_{\run=1}^{\horizon-1} (\next[\step] - \curr[\step]) \obj(\next[\point]),
\end{align}
but $\next[\step] - \curr[\step] \leq 0$ by assumption and $\obj(\next[\point]) \geq \min \obj$, so
\begin{align}
\sum_{\run=1}^\horizon \curr[\step] \bracks*{ \obj(\curr[\point]) - \obj(\next[\point]) }
&\leq \step_1 \obj(\point_1) - \step_\horizon \obj(\point_{\horizon+1}) + \sum_{\run=1}^{\horizon-1} (\next[\step] - \curr[\step]) \min \obj
\notag\\
&= \step_1 \bracks*{ \obj(\point_1) - \min\obj } + \step_\horizon\bracks*{ \min \obj - \obj(\point_{\horizon+1}) }
\notag\\
&\leq \step_1 \bracks*{ \obj(\point_1) - \min \obj }.
\end{align}
Moreover, $\fench(\basealt, \next[\dpoint]) \geq 0$ by \cref{prop:Fench}, and thus
\begin{equation}
\exof*{ \sum_{\run=1}^\horizon \curr[\step] \braket*{\curr[\point] - \basealt}{\nabla \obj(\curr[\point]) }} \leq \fench(\basealt, \dpoint_1) + \step_1 \bracks*{ \obj(\point_1) - \min \obj } + \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} \sum_{\run=1}^\horizon \curr[\step]^{\momexp}
\eqcomma
\end{equation}
which concludes the proof.
\end{proof}
As a first consequence, we obtain a convergence guarantee of \eqref{eq:LMD} for convex objectives that are smooth relative to $\hreg$:
\begin{theorem}[Convergence rate of \eqref{eq:LMD} for convex functions]
If $\obj$ is convex and relative $\lips$-smooth, then for every nonincreasing stepsize sequence with $\step_1 \leq \frac{1}{\momexp \lips}$, the iterates of \eqref{eq:LMD} satisfy
\begin{align}
\exof*{ \obj(\tilde{\point}_\horizon) - \min\obj}
&\leq \dfrac{ \fench(\argmin\obj, \dpoint_1) + \step_1 \bracks*{\obj(\point_1) - \min\obj} }{ \sum_{\run=1}^\horizon \curr[\step]} \notag \\
&+ \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} \dfrac{\sum_{\run=1}^\horizon \curr[\step]^{\momexp} }{\sum_{\run=1}^\horizon \curr[\step]}
\eqcomma
\end{align}
where $\tilde{\point}_\horizon = \sum_{\run=1}^\horizon \curr[\step] \curr[\point]/\sum_{\run=1}^\horizon \curr[\step]$.
In particular, the choice $\curr[\step] = \beta \run^{-1/\momexp}$ with $\beta \leq \frac{1}{\momexp \lips}$ yields
\begin{align}
\exof*{ \obj(\tilde{\point}_\horizon) - \min\obj }
&\lesssim \parens*{ \frac{1}{\beta}\fench(\argmin\obj, \dpoint_1) + \obj(\point_1) - \min \obj } \horizon^{-\frac{\momexp-1}{\momexp}}
\notag\\
&+ \dfrac{2^{\momexp-1} \beta^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} (\log\horizon) \horizon^{-\frac{\momexp-1}{\momexp}}.
\end{align}
\label{thm:discrete_convex_convergence}
\end{theorem}
\begin{proof}
Let $\basealt = \sol \in \argmin\obj$ satisfying $\fench(\sol, \dpoint_1) = \fench(\argmin \obj, \dpoint_1)$. Then, invoking the convexity of $\obj$ yields
\begin{align}
\sum_{\run=1}^\horizon \curr[\step] \braket*{\curr[\point] - \sol}{\nabla \obj(\curr[\point])}
&\geq \sum_{\run=1}^\horizon \curr[\step]\bracks*{ \obj(\curr[\point]) - \obj(\sol) }
= \bracks*{ \dfrac{\sum_{\run=1}^\horizon \curr[\step] \obj(\curr[\point])}{\sum_{\run=1}^\horizon \curr[\step]} - \min\obj } \sum_{\run=1}^\horizon \curr[\step]
\notag\\
&\geq \bracks*{ \obj( \tilde{\point}_\horizon) - \min \obj} \sum_{\run=1}^\horizon \curr[\step],
\end{align}
and it only remains to apply \cref{thm:discrete_relative_smooth_cv} and divide both sides by $\insum_{\run=1}^\horizon \curr[\step]$ to obtain the result.
\end{proof}
\begin{remark}
\cref{thm:discrete_convex_convergence} also recovers exactly the convergence rates established by \citet[p.~190]{NY83} and \citet[Theorem 6]{VYB+22} for optimizing convex function with \eqref{eq:SMD}. Two points are worth emphasizing: (a) our result remains valid when $\obj$ is only \emph{relatively} smooth with respect to the regularizer, whereas both of the cited works assume smoothness relative to the standard Euclidean norm, thus making our setting more flexible---especially when $\hreg$ is steep; and (b) their convergence rates also hold for regularizers that are not strongly convex but merely \emph{$(1, \momexp/(\momexp-1))$-uniformly convex}, that is, if
\begin{equation}
\breg(\point', \point) = \hreg(\point') - \hreg(\point) - \braket*{\nabla \hreg(\point)}{\point' - \point} \geq \dfrac{\momexp-1}{\momexp} \norm{\point - \point'}^{\frac{\momexp}{\momexp-1}} \quad \text{for all } \point, \point' \in \points_\hreg.
\end{equation}
Importantly, \cref{lemma:heavy_tails_descent} and, by extension, \cref{lemma:LMD_heavy_descent}, can be easily extended to this setting, since we then directly obtain $\fench(\point, \dpoint') \geq \frac{\momexp-1}{\momexp} \norm{\point - \mirror(\dpoint')}^{\frac{\momexp}{\momexp-1}}$ by \cref{prop:Fench}. Hence, all of our results remain valid when strong convexity is relaxed to uniform convexity of the regularizer.
\end{remark}
Analogously to stochastic dual averaging, we can then derive convergence rates for the \emph{last iterate} of \eqref{eq:LMD} under relative strong convexity. For a constant stepsize schedule, this gives the following bound, which coincides with that of \eqref{eq:SDA}.
\begin{theorem}[Convergence rate of \eqref{eq:LMD} for relatively strongly convex functions, constant stepsize]
Let $\hreg$ be a steep regularizer. If $\obj$ is relatively $\lips$-smooth and relatively $\strong$-strongly convex with minimum $\sol \in \points$, then running \eqref{eq:LMD} with constant stepsizes $\step \leq \frac{1}{\momexp \lips}$ achieves
\begin{equation}
\exof*{\norm{\curr[\point] - \sol}^2} \leq \dfrac{2^{\momexp-1} \diamX^{2 - \momexp}}{\momexp \strong \hcst^{\momexp}} \sdev^{\momexp} \step^{\momexp - 1} + \dfrac{2 \breg(\sol, \point_1)}{\hcst} (1 - \step \strong)^{\run - 1}.
\end{equation}
\end{theorem}
\begin{proof}
Since the stepsizes are constant, this result follows from the exact same arguments and computations used to prove \cref{thm:cv_relative_strong_discrete} for \eqref{eq:SDA} with constant learning rate.
\end{proof}
On the other hand, for a variable stepsize of order $1/\run^{1/\momexp}$ we obtain the following convergence guarantee, which requires a slightly more refined non-asymptotic version of Chung's lemma in order to be derived.
\begin{theorem}[Convergence rate of \eqref{eq:LMD} for relatively strongly convex functions, decreasing stepsize]
\label{thm:str-LMD-var}
Let $\hreg$ be a steep regularizer. If $\obj$ is relatively $\lips$-smooth and relative $\strong$-strongly convex with minimum $\sol \in \points$, then running \eqref{eq:LMD} with stepsizes $\curr[\step] = \beta \run^{-1/\momexp}$ where $\beta \leq \frac{1}{\momexp \lips}$ achieves the convergence rate
\begin{equation}
\exof*{\norm{\curr[\point] - \sol}^2} \leq \dfrac{2^{\momexp+1} \beta^{\momexp-1}\diamX^{2-\momexp} \sdev^{\momexp}}{\strong \hcst^\momexp} \time^{-\frac{\momexp-1}{\momexp}} + \dfrac{2 \breg(\sol, \point_1)}{\hcst} \exp\parens*{ - \dfrac{\strong \beta}{2} \run^{\frac{\momexp-1}{\momexp} }}
\end{equation}
for all times $\run \geq \bracks*{ \frac{2(\momexp-1)}{\momexp \strong \beta}}^{\frac{\momexp}{\momexp-1}}$. In particular, $\curr[\point] \to \sol$ in mean square as $\time \to \infty$.
\end{theorem}
\begin{proof}
Following the same arguments as in the proof of \cref{thm:discrete_relative_strong_variable_learning_rate} but starting from \cref{lemma:LMD_diff_of_F,lemma:LMD_heavy_descent} instead, we get
\begin{align}
\exof*{\breg(\sol, \next[\point])}
&\leq \exof*{\breg(\sol, \curr[\point])} - \strong \curr[\step] \exof*{\breg(\sol, \curr[\point])} + \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} \curr[\step]^{\momexp}
\notag\\
&= (1- \strong \curr[\step]) \exof*{\breg(\sol, \curr[\point])} + \dfrac{2^{\momexp-1}\diamX^{2-\momexp} \sdev^{\momexp}}{\momexp \hcst^{\momexp-1}} \curr[\step]^{\momexp}.
\label{eq:LMD_bregman_recurrence}
\end{align}
In particular, letting $\curr[\step] = \beta \run^{-1/\momexp}$, we obtain
\begin{equation}
\exof*{\breg(\sol, \next[\point])} \leq \parens*{1 - \frac{\strong\beta}{\run^{1/\momexp}}} \exof*{\breg(\sol, \curr[\point])} + \dfrac{2^{\momexp-1} \diamX^{2-\momexp} \sdev^{\momexp} \beta^{\momexp}}{\momexp \hcst^{\momexp-1}} \frac{1}{\run}.
\end{equation}
Defining $c = \strong \beta$ and $d = 2^{\momexp-1}\hcst^{1-\momexp} \diamX^{2-\momexp} \sdev^{\momexp} \beta^{\momexp}/momexp$, we can then use a non-asymptotic version of Chung's lemma (\cf \citep[Theorem 24]{JLMQ24}) to get that, for every $\run \geq \run_0 \geq \bracks*{ \frac{2(\momexp-1)}{\momexp c} }^{\momexp/(\momexp-1)}$,
\begin{align}
\exof*{\breg(\sol, \curr[\point])} &\leq \dfrac{2d}{c} \parens*{\run + \run_0}^{- \frac{\momexp-1}{\momexp}} + \breg(\sol, \point_1) \exp\parens*{ - \dfrac{c \run}{(\run + \run_0)^{1/\momexp}} }
\notag\\
&\leq \dfrac{2d}{c} \run^{-\frac{\momexp-1}{\momexp}} + \breg(\sol, \point_1) \exp\parens*{ - \dfrac{c \run}{(2\run)^{1/\momexp}} }
\notag\\
&= \dfrac{2d}{c} \run^{-\frac{\momexp-1}{\momexp}} + \breg(\sol, \point_1) \exp\parens*{ - \frac{c}{2} \run^{\frac{\momexp-1}{\momexp} }}
\eqcomma
\end{align}
and it only remains to use the upper bound $\norm{\curr[\point] - \sol}^2 \leq \frac{2}{\hcst} \breg(\sol, \curr[\point])$ to prove the claimed convergence rate.
\end{proof}
We conclude this section by proving \cref{thm:str-LMD}, restated here for completeness, which provides refined convergence rates for \eqref{eq:LMD} under the standard stepsize $\curr[\step] \propto 1/\run$.
\STRLMD*
%\begin{theorem}
% Let $\hreg$ be a steep regularizer. If $\obj$ is relatively $\lips$-smooth and relatively $\strong$-strongly convex with minimum $\sol \in \points$, then running \eqref{eq:LMD} with stepsizes $\curr[\step] = \beta \run^{-1}$ where $\beta \leq \frac{1}{\momexp \lips}$ achieves the asymptotic convergence rate
% \begin{equation}
% \exof*{\norm{\curr[\point] - \sol}^2} = \begin{cases}
% \bigoh\parens*{ \frac{1}{\run^{\momexp-1}} } & \text{if } \momexp < 1 + \beta \strong,
% \\
% \bigoh\parens*{\frac{\log \run}{\run^{\momexp-1}}} & \text{if } \momexp = 1 + \beta \strong,
% \\
% \bigoh\parens*{\frac{1}{\run^{\beta \strong}}} & \text{if } \momexp > 1 + \beta \strong.
% \end{cases}
% \end{equation}
%\end{theorem}
\begin{proof}
Coming back to \cref{eq:LMD_bregman_recurrence} and putting $\curr[\step] = \beta \run^{-1}$, we obtain
\begin{equation}
\exof*{\breg(\sol, \next[\point])} \leq \parens*{1 - \frac{\beta \strong}{\run}}\exof*{\breg(\sol, \curr[\point])} + \frac{d}{\run^{\momexp}}
\eqcomma
\end{equation}
and the asymptotic convergence rates are therefore direct consequences of Chung's lemma (see \eg \citep[Lemma 4, p.45]{Pol87}).
\end{proof}
\begin{remark}
Non-asymptotic convergence rates could also be obtained through the use of non-asymptotic Chung-type estimates, such as those in \citep[Theorem 24]{JLMQ24}, but we restrict our presentation here to asymptotic bounds for simplicity.
\end{remark}
%\begin{remark}
% Since $\lips \geq \strong$, we have $\beta \strong \leq \frac{\strong}{\momexp \lips} \leq \frac{1}{\momexp} < 1$, so the condition $\momexp < 1 + \beta \strong$ cannot hold for $\momexp=2$. In fact, only the opposite condition, $\momexp > 1 + \beta\strong$, can hold for tame moments ($\momexp=2$).
%\end{remark}
%
%\begin{remark}
%Compared to the rates obtained with stepsizes $\run^{-1/\momexp}$, the rates in this theorem are better whenever $\momexp \leq 1/(1-\beta \strong)$. In particular, for the choice $\beta = 1/(\momexp \lips)$, this condition reduces to $\momexp \leq 1 + \frac{\strong}{\lips}$, which is generally satisfies only when $\momexp$ is close enough to $1$.
%\end{remark}
\begin{remark}
Analogous to the convergence rates, we obtain that for every target $\precval > 0$, the stepsize sequence $\curr[\step] = \beta/\time$ achieves the accuracy $\exof*{\norm{\curr[\point] - \sol}^2} \leq \precval$ for every times
\begin{equation}
\time = \begin{cases}
\Omega\parens*{\precval^{-\frac{1}{\momexp-1}}} & \text{if } \momexp < 1 + \beta\strong\\
\Omega\parens*{\bracks*{ \frac{1}{\precval} \log\parens*{\frac{1}{\precval}} }} & \text{if } \momexp = 1 + \beta\strong\\
\Omega\parens*{\precval^{-\frac{1}{\beta\strong}}} & \text{if } \momexp > 1 + \beta\strong.
\end{cases}
\end{equation}
In particular, the first two regimes---that is, when $\momexp \leq 1 + \beta\strong$---matches the complexity rate obtained with a optimally tuned constant stepsize. By contrast, when $\momexp > 1 + \beta \strong$, the resulting complexity is always worse than that of the constant stepsize rule, although it remains sharper than the convex case as long as $\momexp \leq 1 + \frac{\beta\strong}{1-\beta\strong}$. This gap with the optimal lower bounds stems from our use of relative smoothness. Indeed, if one could take $\beta = 1/\strong$, the complexity would become $\Omega(\precval^{-1/(\momexp-1)})$ for all $\momexp \in (1, 2]$, thereby recovering the optimal lower bound for $\momexp=2$ and matching the constant-stepsize complexity when $\momexp<2$. However, the argument used in \cref{lemma:LMD_heavy_descent} additionally requires the condition $\beta \leq 1/(\momexp \lips)$, preventing this optimal stepsize choice. That said, we believe this limitation is methodological rather than fundamental. For instance, \citet{Lu19} obtained the optimal complexity bound $\Omega(1/\precval)$ for \eqref{eq:SMD} with stepsize $\curr[\step] \propto 1/\run$, under the relative boundedness condition
\begin{equation}
\exof*{\dnorm{\orcl(\point;\seed)}^2} \leq M^2 \dfrac{\breg(\base, \point)}{\norm{\base - \point}^2} \quad \text{for every } \point,\base \in \points
\end{equation}
on the stochastic oracle, showing that optimal rates are achievable in stochastic mirror descent under refined assumptions. Extending this approach to heavy-tailed noise is, however, nontrivial. In particular, we conjecture that matching the optimal convergence rate $\bigoh\parens*{\time^{-2(\momexp-1)/\momexp}}$ requires working with the $\momexp/2$-th power of the Bregman divergence, as was done by \citet{FHL25} for projected SGD (where the Bregman divergence reduces to the Euclidean distance). We therefore leave this question open for future work.
\end{remark}