Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent
Extracted source text · arXiv 2502.06719
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
%\usepackage{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\input{header_colt.tex}
\input{def.tex}
\title{Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Marina Sheshukova$^{1}$ \quad Sergey Samsonov$^1$ \quad Denis Belomestny$^{2,1}$ \quad \'Eric Moulines$^{3,4}$ \\
\textbf{Qi-Man Shao}$^5$ \quad \textbf{Zhuo-Song Zhang}$^5$ \quad \textbf{Alexey Naumov}$^{1,6}$ \\
$^1$HSE University \quad
$^2$Duisburg-Essen University \quad
$^3$CMAP, CNRS, \'Ecole Polytechnique \\
$^4$Mohamed Bin Zayed University of AI \quad
$^5$Southern University of Science and Technology \\
$^6$Steklov Mathematical Institute of Russian Academy of Sciences \\
\texttt{\{msheshukova, svsamsonov, anaumov\}@hse.ru} \quad
\texttt{denis.belomestny@uni-due.de} \quad \\
\texttt{eric.moulines@polytechnique.edu} \quad
\texttt{\{shaoqm,zhangzs3\}@sustech.edu.cn}
}
\begin{document}
\maketitle
\begin{abstract}
In this paper, we establish the non-asymptotic validity of the multiplier bootstrap procedure for constructing the confidence sets using the Stochastic Gradient Descent (SGD) algorithm. Under appropriate regularity conditions, our approach avoids the need to approximate the limiting covariance of Polyak-Ruppert SGD iterates, which allows us to derive approximation rates in convex distance of order up to $1/\sqrt{n}$. Notably, this rate can be faster than the one that can be proven in the Polyak-Juditsky central limit theorem. To our knowledge, this provides the first fully non-asymptotic bound on the accuracy of bootstrap approximations in SGD algorithms. Our analysis builds on the Gaussian approximation results for nonlinear statistics of independent random variables.
\end{abstract}
\section{Introduction}
\label{sec:intro}
\input{introduction.tex}
\section{Main results}
\label{sec:bootstrap}
This section establishes the nonasymptotic validity of the multiplier bootstrap method proposed in \cite{JMLR:v19:17-370}. We focus on smooth and strongly convex minimization problems, following the framework established in \cite{moulines2011non}, \cite{pmlr-v99-anastasiou19a} and \cite{shao2022berry}. The underlying procedure is based on perturbing the trajectory \eqref{eq:sgd_recursion_main}. Let
\(\mathcal{W}^{n-1} = \{w_{\ell}\}_{1 \leq \ell \leq n-1}\)
be i.i.d.\ random variables with distribution $\PP_w$, each with mean \(\PE[w_1] = 1\) and variance \(\var[w_1] = 1\). Assume that \(\mathcal{W}^{n-1}\) is independent of
\(\Xi^{n-1} = \{\xi_{\ell}\}_{1 \leq \ell \leq n-1}\). We then use \(\mathcal{W}^{n-1}\) to construct randomly perturbed trajectories of the SGD dynamics \eqref{eq:sgd_recursion_main}:
\begin{equation}
\label{eq:sgd_bootstrap}
\begin{split}
\theta_{k}^{\boot}
&= \theta_{k-1}^{\boot} - \alpha_{k} w_k F(\theta_{k-1}^\boot,\xi_{k}) \eqsp,~~ k \geq 1 \eqsp, ~~ \theta_{0}^{\boot} = \theta_{0} \in \rset^{d} \eqsp, \\
\prtheta_{n}^{\boot}
&= n^{-1} \sum_{k=0}^{n-1} \theta_k^{\boot} \eqsp, ~~n \geq 1 \eqsp.
\end{split}
\end{equation}
Note that, when generating different weights $w_{k}$, we can draw samples from the conditional distribution of $\bar{\theta}_{n}^\boot$ given the data $\Xi^{n-1}$. We further denote
\begin{align}
\label{def: PPb}
\PPb = \PP(\cdot \mid \Xi^{n-1})\eqsp, \quad \PEb = \PE(\cdot \mid \Xi^{n-1})\eqsp.
\end{align}
The core principle behind the bootstrap procedure \eqref{eq:sgd_bootstrap} is that the "bootstrap world" probabilities $\PPb\big(\sqrt{n} (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B\big)$ are close to $\PP\big(\sqrt{n} (\bar{\theta}_n - \thetas) \in B\big)$ for $B \in \Conv(\rset^{d})$. Formally, we say that the procedure \eqref{eq:sgd_bootstrap} is asymptotically valid if
\begin{equation}
\label{eq:boot_validity_supremum}
\sup_{B \in \Conv(\rset^{d})} \Big| \PPb\big(\sqrt{n} (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B \big) - \PP\big(\sqrt{n} (\bar{\theta}_n - \thetas) \in B\big) \Big|
\end{equation}
converges to $0$ in $\PP$-probability as $n \to \infty$. This result was obtained in \cite{JMLR:v19:17-370} under assumptions close to the original paper \cite{polyak1992acceleration}.
While an analytical expression for \(\PPb(\sqrt{n} (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B)\) is unavailable, it can be approximated via Monte Carlo simulations by generating \(M\) perturbed trajectories according to \eqref{eq:sgd_bootstrap}. Standard arguments (see, e.g., \cite[Section~5.1]{shao2003mathematical}) suggest that the accuracy of this Monte Carlo approximation scales as \(\mathcal{O}(M^{-1/2})\) when generating $M$ parallel perturbed trajectories in \eqref{eq:sgd_bootstrap}.
\par
\paragraph{Assumptions.} We impose the following regularity conditions on the objective function $f$:
\begin{assum}
\label{ass:L-smooth}
The function $f$ is two times continuously differentiable and $L_{1}$-smooth on $\rset^{d}$, i.e., there is a constant $L_{1} > 0$, such that for any $\theta,\theta' \in \rset^{d}$,
\begin{equation}
\label{eq:L-smooth}
\norm{\nabla f(\theta) - \nabla f(\theta')} \leq L_{1} \norm{\theta - \theta'}\eqsp.
\end{equation}
Moreover, we assume that $f$ is $\mu$-strongly convex on $\rset^{d}$, that is, there exists a constant $\mu > 0$, such that for any $\theta,\theta' \in \rset^{d}$, it holds that
\begin{equation}
\label{eq:strong_convex}
(\mu/2) \norm{\theta-\theta'}^2 \leq f(\theta) - f(\theta') - \langle \nabla f(\theta'), \theta - \theta' \rangle \eqsp.
\end{equation}
\end{assum}
\Cref{ass:L-smooth} implies the following two-sided bound on the Hessian of $f$:
\[
\textstyle
\mu \Id_d \preceq \nabla^2 f(\theta) \preceq L_1 \Id_d
\]
for all $\theta \in \rset^{d}$. We now state the assumptions on $F(\theta, \xi)$. Namely, we rewrite $F(\theta, \xi)$ as
\begin{equation}
\label{eq: gradF decompositon}
\textstyle
F(\theta_{k-1}, \xi_k) = \nabla f(\theta_{k-1}) + \zeta_k\eqsp,\quad \text{ where } \zeta_k := \zeta(\theta_{k-1}, \xi_k)\eqsp,
\end{equation}
that is, $\{ \zeta_k\}_{k \in \nset}$ is a sequence of $d$-dimensional random vectors, which distribution can depend upon $\theta_{k-1}$. Then the SGD recursion \eqref{eq:sgd_bootstrap} takes form
\begin{equation}
\label{eq:sgd_recursion_main_new}
\textstyle
\theta_{k} = \theta_{k-1} - \alpha_{k} (\nabla f(\theta_{k-1}) + \zeta_k) \eqsp, \quad \theta_0 \in \rset^{d}\eqsp.
\end{equation}
We impose the following assumption on the noise sequence $\zeta_k$:
\begin{assum}
\label{ass:bound_noise}
For each $ k \geq 1$, $\zeta_k$ admits the decomposition $\zeta_k = \eta(\xi_k) + g(\theta_{k-1}, \xi_k)$, where
\begin{enumerate}[(i),noitemsep,nolistsep]
\item $\{\xi_k\}_{k=1}^{n-1}$ is a sequence of i.i.d. random variables on $(\Zset,\Zsigma)$ with distribution $\PP_{\xi}$, $\eta: \Zset \rightarrow \rset^d$ is a function such that $\PE[\eta(\xi_1)]= 0$ and $\PE[\eta(\xi_1)\eta(\xi_1)^\top]= \Sigma_{\xi}$. Moreover, $\lambda_{\min}(\Sigma_{\xi}) > 0$.
\item The function $g: \rset^d \times \Zset \rightarrow \rset^d$ satisfies $\PE[g(\theta,\xi_1)]=0$ for any $\theta \in \rset^{d}$. Moreover, there exists $L_2 > 0$ such that for any $\theta, \theta' \in \rset^{d}$, it holds that
\begin{equation}
\label{eq: g_bound_norm}
\textstyle
\norm{g(\theta, \xi)-g(\theta', \xi)}\leq L_2\norm{\theta - \theta'} \quad \text{and} \quad g(\thetas, z) = 0\eqsp \text{ for all } z \in \Zset\eqsp.
\end{equation}
\item There exist $C_{1,\xi}, C_{2,\xi} > 0$ such that $\PP_{\xi}$-almost surely
\[
\textstyle
\norm{\eta(\xi)} \leq C_{1,\xi} \text{ and } \sup_{\theta} \norm{g(\theta, \xi)}\leq C_{2,\xi}\eqsp.
\]
\end{enumerate}
\end{assum}
As an example of a sequence $\zeta_k$ satisfying conditions (i) and (ii) from \Cref{ass:bound_noise}, consider the case when the stochastic estimates $F(\theta, \xi)$ satisfies:
\begin{enumerate}[noitemsep, nolistsep]
\item $\PE[F(\theta, \xi)] = \nabla f(\theta)$ for all $\theta \in \rset^{d}$;
\item $\norm{F(\theta, \xi) - F(\theta', \xi)} \leq L \norm{\theta - \theta'}$ for all $\xi \in \Zset$, and $\sup_{\theta}|F(\theta, \xi) - F(\thetas, \xi)| \leq c_{\xi}$ for some $c_{\xi} > 0$.
\end{enumerate}
In this case, (i) and (ii) from \Cref{ass:bound_noise} holds with $\eta(\xi) = F(\thetas, \xi)$ and $g(\theta, \xi) = F(\theta, \xi) - F(\thetas, \xi) - \nabla f (\theta)$. Additionally, note that the identity \eqref{eq: g_bound_norm} can be relaxed when one considers only last iterate bounds, such as $\PE[\norm{\theta_k - \thetas}^2]$, see \cite{moulines2011non}. Item (ii) from \Cref{ass:bound_noise} is often imposed when considering averaged iterates, see \cite[Assumption H2']{moulines2011non}, and \cite{durmus2020biassgd,shao2022berry}.
The assumption (iii) from \Cref{ass:bound_noise} is crucial to prove high-order moment bounds
\begin{equation}
\label{eq:hpd_bound_preliminary}
\textstyle
\PE^{1/p}[\norm{\theta_k - \thetas}^{p}] \text{ and } \PE^{1/p}[\norm{\theta_k^{\boot} - \thetas}^{p}]\eqsp,
\end{equation}
see \Cref{lem: high_prob_last_iter} in Appendix. In our proof, we closely follow the argument presented in \cite[Theorem 4.1]{harvey2019tight}, which requires that the noise variables $\zeta_k$ are almost sure bounded. This setting can be generalized to the case where $\zeta_k$ is sub-Gaussian conditioned on $\F_{k-1}$ with variance proxy which is uniformly bounded by a constant factor, that is, there is a constant $M$, such that $\PE[\exp\{\norm{F(\theta,\xi_1)}^2/M^2\}] \leq 2$ for any $\theta \in \rset^{d}$. This assumption is widely considered in the literature; see \cite{nemirovski2009robust,hazan2014beyond}, and the remarks in \cite{harvey2019tight}. However, when $\zeta_k = g(\theta_{k-1},\xi_k) + \eta(\xi_k)$ and $g$ is only Lipschitz w.r.t. $\theta$, its moments naturally scale with $\norm{\theta_{k-1} - \thetas}$, thus the sub-Gaussian bound with $M$ not depending upon $\theta$ is unlikely to hold. Other authors who considered bounds of type \eqref{eq:hpd_bound_preliminary}, e.g. \cite{rakhlin2012making}, made stronger assumption that $\sup_{\theta \in \rset^{d}}\norm{F(\theta,\xi)}$ is a.s. bounded. Another popular direction is to consider updates which involve gradient clipping; see e.g. \cite{sadiev2023high}. We leave further studies of clipped gradient schemes for future work. We further impose condition on the Hessian matrix $\nabla^2f(\theta)$ at $\thetas$:
\begin{assum}
\label{ass:hessian_Lipschitz_ball}
There exist $L_3, \beta > 0$ such that for all $\theta$ with $\norm{\theta-\thetas} \leq \beta$, it holds
\begin{equation}
\textstyle
\norm{\nabla^2f(\theta) -\nabla^2f(\thetas)}\leq L_3\norm{\theta-\thetas}\eqsp.
\end{equation}
\end{assum}
\Cref{ass:hessian_Lipschitz_ball} ensures that the Hessian of \(f\) is Lipschitz continuous in a neighborhood of \(\theta^*\).
Similar assumptions have been previously considered in \cite{shao2022berry} and \cite{pmlr-v99-anastasiou19a}, as well as in other works on first-order optimization methods, see, e.g. \cite{li2022root}. Several studies on the non-asymptotic analysis of SGD impose stronger smoothness assumptions, such as bounded derivatives of \(f\) up to order four, see \cite{durmus2020biassgd}. We also impose an assumption on the bootstrap weights $W_i$ used in the algorithm:
\begin{assum}
\label{ass:bound_bootstap_weights}
There exist constants $0 < W_{\min} < W_{\max} < +\infty$, such that $W_{\min}\leq w_1 \leq W_{\max}$ a.s.
\end{assum}
The original paper \cite{JMLR:v19:17-370} also considered positive bootstrap weights $w_i$. We have to impose boundedness of $w_i$ due to our high-probability bound from \Cref{lem: high_prob_last_iter}. A particular example of a distribution satisfying \Cref{ass:bound_bootstap_weights} and the constraints $\PE[w_1] = 1$, $\PVar[][w_1] = 1$ is provided in \Cref{sec:example_distribution}. We also consider the following bound for step sizes $\alpha_k$ and sample size $n$:
\begin{assum}
\label{ass:step_size_new_boot} Let $\alpha_k = c_0 \{k_0 +k\}^{-\gamma}$, where $\gamma \in (1/2, 1)$, and $c_0$, $k_0$ satisfy
\[
\textstyle
3c_0 W_{\max}^2 (L_1^2+L_2^2) \leq 1\eqsp, \quad k_0 \geq\max((\frac{2\gamma}{\mu c_0 W_{\min}})^{1/(1-\gamma)},(\frac{1}{\mu W_{\min}})^{1/\gamma})\eqsp.
\]
\end{assum}
\begin{assum}
\label{ass:n_lower_bound}
Number of observations $n$ is large enough. Precise expressions for $n$ are given in \Cref{sec: bootstrap validity section} (see \Cref{ass:n_lower_bound_full}).
\end{assum}
The particular bound on $k_0$ in \Cref{ass:step_size_new_boot} appears due to the high-order moment bounds (see \Cref{lem: high_prob_last_iter} in appendix). We discuss the bound on the number of observations imposed in \Cref{ass:n_lower_bound} later in the proof of \Cref{th:bootstrap_validity}.
\subsection{Non-asymptotic multiplier bootstrap validity}
\begin{theorem}
\label{th:bootstrap_validity}
Assume \Cref{ass:L-smooth} - \Cref{ass:n_lower_bound}. Then with $\PP$ - probability at least $1 - 2/n$, it holds
\begin{equation}
\label{eq:gaussian_approximation_bootstrap}
\sup_{B \in \Conv(\rset^{d})} |\PPb(\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B ) - \PP(\sqrt n (\bar{\theta}_n - \thetas) \in B)| \le \frac{\ConstC_1 \sqrt{\log n}}{n^{1/2}} + \frac{\ConstC_2 \log n}{n^{\gamma - 1/2} } + \frac{\ConstC_3 (\log n)^{3/2}}{n^{\gamma/2}} \eqsp,
\end{equation}
where $\ConstC_1, \ConstC_2$ and $\ConstC_3$ are given in \Cref{app:proof_th_main_1}, equation \eqref{eq: bootsrap constants def}.
\end{theorem}
\begin{remark}
The constants $C_1, C_2, C_3$ in \Cref{th:bootstrap_validity} depend on the problem dimension $d$ as well as on $\sigma_2$, $\sigma_4$, $C_{1, \xi}$, $C_{2,\xi}$ and $\|\theta_0 - \thetas \|$.
To make the dependence on $d$ explicit, we assume the natural scaling $
\sigma_2, \ \sigma_4, \ C_{1, \xi}, \ C_{2, \xi},\ \|\theta_0 - \theta^\ast\| \ \lesssim\ \sqrt{d}.
$
Under this assumption, \Cref{th:bound_kolmogorov_dist_pr_sigma_n} yields
\begin{multline}
\sup_{B \in \Conv(\rset^{d})} |\PPb(\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B ) - \PP(\sqrt n (\bar{\theta}_n - \thetas) \in B)| \\ \lesssim \frac{(d^2 + d^{3/2}\log(2d)) \sqrt{\log n}}{n^{1/2}} + \frac{d^{3/2} \log n}{n^{\gamma - 1/2} } + \frac{d^{3/2} (\log n)^{3/2}}{n^{\gamma/2}} \eqsp.
\end{multline}
Here the notation $\lesssim$ means that the inequality holds up to constants that do not depend on $n$ or $d$.
\end{remark}
\begin{remark}
\label{rem:step_size_remark}
It is possible to prove the result of \Cref{th:bootstrap_validity} for the step size $\alpha_k = c_0/(k+k_0)$. The required Gaussian approximation result with the covariance matrix $\Sigma_n$ is proved in \cite{shao2022berry}, and we expect that the only difference with \Cref{th:bootstrap_validity} will occur in extra $\log{n}$ factors in the corresponding bound and slightly different conditions on $c_0$ and $k_0$ in \Cref{ass:step_size_new_boot}.
\end{remark}
\begin{proof}[Proof sketch of Theorem \ref{th:bootstrap_validity}]The proof of non-asymptotic bootstrap validity is based on the Gaussian approximation performed both in the "real" world and bootstrap world together with an appropriate Gaussian comparison inequality:
\begin{tikzcd}[column sep = 110pt]
\label{eq:diagram}
\text{Real world: \quad\quad} \sqrt n (\bar{\theta}_{n} - \thetas) \arrow[<->]{r}{\text{Gaussian approximation, Th. \ref{th:bound_kolmogorov_dist_pr_sigma_n}}} & \mathcal N(0, \Sigma) \arrow[<->]{d}{\text{Gaussian comparison} }
\\
\text{Bootstrap world: } \sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \arrow[<->]{r}{\text{Gaussian approximation, Th. \ref{GAR bootstrap_main}}} & \mathcal N(0, \Sigma^\boot )\eqsp. \\
\end{tikzcd}
Here $\Sigma$ and $\Sigma^\boot$ are some covariance matrices to be chosen later. In order to understand where the Gaussian approximation comes from, we consider the process of linearization of statistics $\sqrt n (\bar{\theta}_{n} - \thetas)$ and $\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n)$. We provide details for $\sqrt n (\bar{\theta}_{n} - \thetas)$, and give similar derivations for $\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n)$ in \Cref{GAR boot world sec}. Denote $G = \nabla^2f(\thetas)$. We expand $\sqrt{n}(\bar{\theta}_{n} - \thetas)$ into a weighted sum of independent random vectors, along with the remaining terms of smaller order. By the Newton-Leibniz formula, we obtain
\begin{align}
\label{eq:H_theta_def}
\textstyle
\nabla f(\theta) = G(\theta-\thetas) + H(\theta),
\end{align}
where $H(\theta) = \int_{0}^1 (\nabla^2f(\thetas + t(\theta-\thetas)) - G)(\theta-\thetas)\, \rmd t$.
Note that $H(\theta)$ is of the order $\|\theta - \thetas\|^2$ (see \Cref{lem:H_theta_bound}). The recursion for the SGD algorithm error \eqref{eq:sgd_recursion_main_new} can be expressed as
\begin{equation}
\label{eq: sgd_reccurence_with_remainder_main}
\textstyle
\theta_k - \thetas = (\Id_d - \alpha_k G) (\theta_{k-1} - \thetas) - \alpha_k ( \eta(\xi_k) + g(\theta_{k-1}, \xi_k) + H(\theta_{k-1}))\eqsp.
\end{equation}
For $i \in \{0,\ldots,n-1\}$, we define the matrices
\begin{equation}
\label{eq:Q_i_def}
\textstyle
Q_i = \alpha_i\sum_{j=i}^{n-1}\prod_{k=i+1}^{j}(\Id_d-\alpha_k G)\eqsp,
\end{equation}
where empty products are defined to be equal to $\Id_d$ by convention.
Then taking average of \eqref{eq: sgd_reccurence_with_remainder_main} and rearranging the terms, we obtain the following expansion:
\begin{equation}
\label{eq:linear and nonlinear terms}
\textstyle
\sqrt{n}(\bar\theta_n -\thetas) = W + D\eqsp, \quad W = - \frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i\eta(\xi_i), \eqsp \quad D = \sqrt{n}(\bar\theta_n -\thetas) - W\eqsp.
\end{equation}
Note that $W$ is a weighted sum of i.i.d. random vectors with mean zero and covariance matrix
\begin{equation}
\label{eq:sigma_n_def}
\textstyle
\Sigma_n = n^{-1} \sum_{k=1}^{n-1} Q_k \noisecov Q_k^\top \eqsp,
\end{equation}
and $D$ is the remainder term which is defined in Appendix \ref{sec:Gar_real_world_proof}, equation \eqref{eq:linear and nonlinear terms appendix}. Furthermore, in \Cref{sec: proof of difference between cov} we show that $Q_i$ may be approximated by $G^{-\top}$ and $\Sigma_n$ approximates
\begin{equation}
\label{eq:Sigma_infty_def}
\textstyle
\Sigma_\infty = G^{-1} \Sigma_{\xi} G^{-\top}\eqsp.
\end{equation}
We expect that the summand $D$ does not significantly distort the asymptotic distribution for the linear statistic $W$, which should be Gaussian by virtue of the central limit theorem. An important question is the choice of the approximating Gaussian distribution $\mathcal N(0, \Sigma)$ with $\Sigma = \Sigma_n$ or $\Sigma_\infty$ as well their bootstrap counterpart $\Sigma^\boot$. This choice is instrumental in the sense that it does not change the procedure \eqref{eq:sgd_bootstrap}, but only affects the rates in \eqref{eq:boot_validity_supremum}. The authors of \cite{JMLR:v19:17-370} choose the approximation with $\mathcal N(0, \Sigma_{\infty})$ for their asymptotic analysis. A similar approach was considered in \cite[Theorem 3]{samsonov2024gaussian} for the LSA algorithm setting. However, as it will be shown later in \Cref{cor:berry-esseen}, this choice implies that the rate of normal approximation in \eqref{eq:boot_validity_supremum} is not faster than $n^{-1/4}$. At the same time, \Cref{th:bound_kolmogorov_dist_pr_sigma_n} and \Cref{GAR bootstrap_main} below demonstrate that we can achieve approximation rates of up to $n^{-1/2}$ by selecting $\Sigma = \Sigma_n$ in the diagram \ref{eq:diagram}, and its bootstrap-world counterpart in the Gaussian approximation. To finish the proof, it remains to apply the Gaussian comparison inequality; see \Cref{app:proof_th_main_1}. Detailed proof of \Cref{th:bootstrap_validity} is provided in \Cref{sec: bootstrap validity section}.
\end{proof}
\paragraph{Discussion.} In \cite{samsonov2024gaussian}, a counterpart of \Cref{th:bootstrap_validity} was established with an approximation rate of order $n^{-1/4}$ up to logarithmic factors for the setting of the LSA algorithm. The obtained rate is suboptimal, since the authors chose $\mathcal{N}(0,\Sigma_{\infty})$ for Gaussian approximation when showing bootstrap validity. A recent paper \cite{wu2024statistical} improved this rate to $n^{-1/3}$ for the temporal difference (TD) procedure with linear function approximation. The algorithm they considered is based on the direct estimate of $\Sigma_{\infty}$, yielding a rate of order $n^{-1/3}$ when approximating the distribution of $\sqrt{n}(\bar{\theta}_n-\thetas)$ by $\mathcal{N}(0,\hat{\Sigma}_n)$ with a suitably constructed $\Sigma_n$; see \cite[Theorem 3.4 and 3.5]{wu2024statistical}. The authors of \cite{chen2020aos} constructed a plug-in estimator $\hat{\Sigma}_n$ of $\Sigma_{\infty}$ and showed that $\PE[\norm{\hat{\Sigma}_n - \Sigma_{\infty}}] \lesssim C n^{-\gamma/2}$, $\gamma \in (1/2,1)$, under weaker assumptions than those considered in the current section. However, this bound is not sufficient to prove the analogue of the Gaussian comparison result \Cref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty} for $\mathcal{N}(0,\hat{\Sigma}_n)$ and $\mathcal{N}(0,\Sigma_{\infty})$ on a set with large $\PP$-probability. The latter result would require high-probability bounds on $\norm{\hat{\Sigma}_n - \Sigma_{\infty}}$, which are more complicated and may require additional assumptions compared to those in \cite{chen2020aos}. Moreover, estimating the error between the distribution of $\sqrt{n}(\bar{\theta}_n - \thetas)$ and $\mathcal{N}(0,\hat{\Sigma}_n)$ also requires a Berry-Esseen type bound on the distance between $\sqrt{n}(\bar{\theta}_n - \thetas)$ and $\mathcal{N}(0,\Sigma_\infty)$. As we show in \Cref{cor:berry-esseen}, the approximation rate in the latter problem vanishes as $\gamma \to 1$, which introduces an additional trade-off in the potential analysis of plug-in procedures based on estimating $\Sigma_\infty$. This effect highlights the main difference between the multiplier bootstrap approach and the plug-in approach of \cite{chen2020aos}.
\subsection{Gaussian approximation in the real world}
\label{sec:gaus_approximation}
For results of this section, assumptions \Cref{ass:bound_noise} and \Cref{ass:step_size_new_boot} can be relaxed. We impose a family of assumptions, denoted as \Cref{ass:noise_decomposition}($p$) with $p \geq 2$, on the noise sequence $\zeta_k$, and \Cref{ass:step_size} on the step sizes $\alpha_k$:
\begin{assum}[p]
\label{ass:noise_decomposition}
Conditions (i) and (ii) from \Cref{ass:bound_noise} holds. Moreover, there exists $\sigma_p > 0$ such that $\PE^{1/p}[\|\eta(\xi_1)\|^p]\leq \sigma_p\eqsp.$
\end{assum}
\begin{assum}
\label{ass:step_size}
Suppose that $\alpha_k = c_0 / (k_0+k)^{\gamma}$, where $\gamma \in (1/2, 1)$, $k_0\geq 1$, and $c_0$ satisfies $2c_0L_1\leq 1$.
\end{assum}
Note that $\Cref{ass:step_size_new_boot}$ implies \Cref{ass:step_size}, as well as \Cref{ass:bound_noise} implies \Cref{ass:noise_decomposition}($p$) for any $p \geq 2$. In the main result of this section we provide the Gaussian approximation result for $\sqrt{n}(\bar \theta_n -\thetas)$ with $\mathcal{N}(0,\Sigma_n)$, which refines the bounds obtained in \cite[Theorem~3.4]{shao2022berry} and is instrumental for further studies of normal approximation with $\mathcal{N}(0,\Sigma_{\infty})$ in \Cref{sec:shao_self-normalized}.
\begin{theorem}
\label{th:bound_kolmogorov_dist_pr_sigma_n}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($4$), \Cref{ass:step_size}. Then, with $Y \sim \mathcal{N}(0,\Id_d)$, it holds that
\begin{equation}
\label{eq:kolm-dist}
\kolmogorov(\sqrt{n} \Sigma_{n}^{-1/2}(\bar\theta_n -\thetas), Y) \leq \frac{\ConstC_4}{\sqrt{n}} + \frac{\ConstC_5}{n^{\gamma - 1/2}}+ \frac{\ConstC_6}{n^{\gamma/2}}\eqsp,
\end{equation}
where $\ConstC_4, \ConstC_5, \ConstC_6$ are given in Appendix \ref{sec:Gar_real_world_proof}, equation \eqref{eq:def_Const_M_3_i}. Moreover, since $\Sigma_n$ is non-degenerate, and an image of a convex set under non-degenerate linear mapping is a convex set, we have
\begin{align}
\kolmogorov(\sqrt{n}\Sigma_n^{-1/2}(\bar\theta_n -\thetas), Y) = \kolmogorov(\sqrt{n}(\bar \theta_n -\thetas), \Sigma_n^{1/2} Y)\eqsp.
\end{align}
\end{theorem}
\begin{remark}
The constants $C_4, C_5, C_6$ in \Cref{th:bound_kolmogorov_dist_pr_sigma_n} depend on the problem dimension $d$ as well as on $\sigma_2$, $\sigma_4$, and $\|\theta_0 - \thetas \|$.
To make the dependence on $d$ explicit, we assume the natural scaling $
\sigma_2, \ \sigma_4, \ \|\theta_0 - \theta^\ast\| \ \lesssim\ \sqrt{d}.
$
Under this assumption, \Cref{th:bound_kolmogorov_dist_pr_sigma_n} yields
\begin{equation}
\kolmogorov(\sqrt{n} \Sigma_{n}^{-1/2}(\bar\theta_n -\thetas), Y) \lesssim \frac{d^2}{\sqrt{n}} + \frac{d^{3/2}}{n^{\gamma - 1/2}}+ \frac{d^{3/2}}{n^{\gamma/2}}\eqsp,
\end{equation}
Here the notation $\lesssim$ means that the inequality holds up to constants that do not depend on $n$ or $d$.
\end{remark}
\begin{remark} When \( \gamma \to 1 \), the correction terms above scale as \(\mathcal{O}(1/\sqrt{n}) \), yielding the overall approximation rate that approaches \( 1/\sqrt{n} \). Expressions for $\ConstC_4, \ConstC_5, \ConstC_6$ from \Cref{th:bound_kolmogorov_dist_pr_sigma_n} depend upon the problem dimension $d$, parameters specified in \Cref{ass:L-smooth} - \Cref{ass:noise_decomposition}($4$)- \Cref{ass:hessian_Lipschitz_ball}-\Cref{ass:step_size}. Moreover, $\ConstC_5$ depends upon $\norm{\theta_0-\thetas}$. When $\gamma \in (0,1)$, we have that $1/n^{\gamma/2} < 1/n^{\gamma-1/2}$, thus, the term $\ConstC_5/n^{\gamma-1/2}$ dominates. We prefer to keep both terms in \eqref{eq:kolm-dist}, since they are responsible for the moments of statistics $\frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_iH(\theta_{i-1})$ and $\frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_ig(\theta_{i-1}, \xi_i)$, respectively. The first of them has non-zero mean, since $H(\theta_{i-1})$ is quadratic in $\norm{\theta_i-\thetas}^2$. When using constant step size SGD, one can correct this term using the Richardson-Romberg technique \cite{durmus2020biassgd,SheshukovaICLR}, however, it is unclear if this type of ideas can be generalized for diminishing step size.
\end{remark}
\begin{proof}[Proof sketch of \Cref{th:bound_kolmogorov_dist_pr_sigma_n}] The decomposition \eqref{eq:linear and nonlinear terms} represents a particular instance of the general problem of Gaussian approximation for nonlinear statistics of the form $\sqrt{n}(\bar\theta_n -\thetas)$, where the estimator is expressed as the sum of linear and nonlinear components.
To establish the Gaussian approximation result, we adapt the arguments from \cite{shao2022berry}, which can be stated as follows. Let \( X_1, \ldots, X_n \) be independent random variables taking values in some space \( \mathcal{X} \), and let \( T = T(X_1, \ldots, X_n) \) be a general \( d \)-dimensional statistic that can be decomposed as
\begin{equation}
\label{eq:W-D-decomposition}
\textstyle
W := W(X_1,\ldots,X_n) = \sum_{\ell = 1}^n Z_\ell, \quad D := D(X_1, \ldots, X_n) = T - W\eqsp.
\end{equation}
Here, we define \( Z_\ell = r_\ell(X_\ell) \), where \( r_\ell: \mathcal{X} \to \rset^d \) is a Borel-measurable function. The term \( D \) represents the nonlinear component and is treated as an error term, assumed to be "small" relative to \( W \) in an appropriate sense.
Suppose that \( \PE[Z_\ell] = 0 \) and that the $Z_{\ell}$ is normalized in such a way that \( \sum_{\ell=1}^n \PE[Z_\ell Z_\ell^\top] = \Id_d \) holds. Let $\Upsilon_n = \sum_{\ell=1}^n \PE[\|Z_\ell\|^3]$. Then, for \( Y \sim \mathcal{N}(0,\Id_d) \), the following bound holds:
\begin{equation}
\label{eq:shao_zhang_bound_main}
\kolmogorov(T, Y) \leq 259 d^{1/2} \Upsilon_{n} + 2 \PE[\|W\| \|D\|] + 2 \sum_{\ell=1}^n \PE[\|Z_\ell\| \|D - D^{(\ell)}\|],
\end{equation}
where \( D^{(\ell)} = D(X_1, \ldots, X_{\ell-1}, X_{\ell}^{\prime}, X_{\ell+1}, \ldots, X_n) \) and \( X_\ell^{\prime} \) is an independent copy of \( X_\ell \). This result follows from \cite[Theorem~2.1]{shao2022berry}. Furthermore, this bound can be extended to the case where \( \sum_{\ell=1}^n \PE[Z_\ell Z_\ell^\top] = \Sigma \succ 0 \), as detailed in \cite[Corollary~2.3]{shao2022berry}. In order to apply \eqref{eq:shao_zhang_bound_main}, we let $X_i = \xi_i$, $Z_{\ell} = h(X_{\ell})$, $\xi_{i}^{\prime}$ be an i.i.d. copy of $\xi_i$. Then we need to upper bound $\PE^{1/2}[\norm{D(\xi_1,\ldots,\xi_{n-1})}^2]$ and $\PE^{1/2}[\norm{D - D_{i}^{\prime}}^2]$, respectively. Detailed proof is given in \Cref{sec:Gar_real_world_proof}.
\end{proof}
\subsection{Gaussian approximation in the bootstrap world}
\label{GAR boot world sec}
In the main result of this section, we study the Gaussian approximation result for $\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n)$ with appropriate normal distribution with respect to $\PPb$. Despite this result is similar in its nature with the one of \Cref{th:bound_kolmogorov_dist_pr_sigma_n}, it requires to handle some significant challenges that arises when working in the "bootstrap world". Our first steps are the same as in \eqref{eq: sgd_reccurence_with_remainder_main} and \eqref{eq:sgd_bootstrap}:
\begin{equation}
\label{eq:bootstrap_reccurence_with_remainder}
\begin{split}
\theta_{k}^{\boot}-\theta_{k} &= (I-\alpha_{k}G)(\theta_{k-1}^{\boot}-\theta_{k-1}) \\
&- \alpha_k\bigl(H(\theta_{k-1}^{\boot}) +g(\theta_{k-1}^{\boot}, \xi_k) - H(\theta_{k-1}) -g(\theta_{k-1}, \xi_k)\bigr)\\&- \alpha_k(w_k-1)\bigl( G(\theta_{k-1}^{\boot}-\thetas)+\eta(\xi_k) + g(\theta_{k-1}^{\boot}, \xi_k) + H(\theta_{k-1}^{\boot})\bigr)\eqsp.
\end{split}
\end{equation}
Taking an average of \eqref{eq:bootstrap_reccurence_with_remainder} and rearranging the terms, we obtain a counterpart of \eqref{eq:linear and nonlinear terms}:
\begin{equation}
\label{eq:D_boot-def}
\begin{split}
&\sqrt{n}(\prtheta_n^{\boot}-\prtheta_{n}) = W^\boot + D^\boot \eqsp, \\
&W^\boot = -\frac{1}{\sqrt{n}} \sum_{i=1}^{n-1}(w_i-1) Q_{i} \eta(\xi_i) \eqsp, \quad D^\boot = \sqrt{n}(\prtheta_n^{\boot}-\prtheta_{n}) - W^\boot \eqsp.
\end{split}
\end{equation}
Here $W^\boot$ is a weighted sum of i.i.d. random variables $\Xi^{n-1}$, such that $\PEb[W^\boot] = 0$ and
\begin{equation}
\label{Sigman boot def}
\textstyle
\PEb[W^\boot \{W^\boot\}^{\top}] := \Sigma_{n}^\boot = n^{-1} \sum_{i=1}^{n-1} Q_i \eta (\xi_i) \eta(\xi_i)^\top Q_i^\top\eqsp,
\end{equation}
and $D^\boot$ is a non-linear statistic of $\Xi^{n-1}$. The principal difficulty arises when considering the conditional distribution of $\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n)$ given the data $\Xi^{n-1}$. In fact, the approach of \cite{shao2022berry} would require to control the second moments of $D^\boot$ and $D^\boot - \{D^\boot\}^{(i)}$ with respect to a bootstrap measure $\PPb$, on the high-probability event with respect to a measure $\PP$. At the same time, we loose a martingale structure of the summands in $D^\boot$, unless we condition on the extended filtration
\begin{equation}
\label{eq: extended filtration}
\widetilde{\F}_i = \sigma(w_1,\ldots w_i, \xi_1,\ldots \xi_i) \eqsp, 1 \le i \le n-1 \eqsp.
\end{equation}
Therefore, it is not clear if we can directly apply the approach of \cite{shao2022berry} discussed in Section~\ref{sec:gaus_approximation}. Instead, we have to use the linearization approach based on the high-order moment bounds for the remainder term $D^{\boot}$ (see \Cref{nonlinearapprox} in \Cref{sec: bootstrap validity section}). This justifies the strong bounded noise assumption \Cref{ass:bound_noise}, that we had to impose. We state the main result of this section below:
\begin{theorem}
\label{GAR bootstrap_main}
Assume \Cref{ass:L-smooth} - \Cref{ass:n_lower_bound}. Then with $\PP$ - probability at least $1 - 2/n$, it holds
\begin{equation}
\sup_{B \in \Conv(\rset^d)}|\PPb( \sqrt n\{\Sigma_n^\boot\}^{-1/2} (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B) - \PPb(Y^\boot \in B)| \leq \frac{M_{3,1}^\boot}{n^{1/2}} + \frac{M_{3,2}^\boot \log n}{n^{\gamma-1/2}} + \frac{M_{3,3}^\boot \log^{3/2} n}{n^{\gamma/2}} \eqsp,
\end{equation}
where $\{M_{3,i}^\boot\}_{i=1}^3$ are defined in Appendix \ref{subsec:GAR in the bootstrap world}, equation \eqref{M_3i boot def}.
\end{theorem}
\begin{proof}[Proof sketch of \Cref{GAR bootstrap_main}] We apply the bound
\begin{equation}
\label{crude bound for nonlinear stat}
\begin{split}
&\sup_{B \in \Conv(\rset^d)}|\PPb(\{\Sigma_n^\boot\}^{-\tfrac{1}{2}} (W^\boot + D^\boot) \in B) - \PPb(Y^\boot \in B)| \\
&\quad \leq \sup_{B \in \Conv(\rset^d)}|\PPb(\{\Sigma_n^\boot\}^{-\tfrac{1}{2}} W^\boot \in B) - \PPb(Y^\boot \in B)| + 2 c_d (\PEb[\norm{\{\Sigma_n^\boot\}^{-\tfrac{1}{2}} D^\boot}^p])^{\tfrac{1}{1+p}}\eqsp,
\end{split}
\end{equation}
where $c_d \le 4 d^{1/4}$ is the isoperimetric constant of the class of convex sets. The proof of \eqref{crude bound for nonlinear stat} is provided in \Cref{nonlinearapprox} in \Cref{subsec:GAR in the bootstrap world}.
We can control $\bar{\PE}[\norm{D^\boot}^p]$ by Burkholder's inequality, where $\bar{\PE}$ denotes the expectation w.r.t. the product measure $\PP_\xi^{\otimes n} \otimes \PP_w^{\otimes n}$. Then we proceed with Markov's inequality to obtain $\PP$ -- high-probability bounds on the behavior of $\PEb[\norm{D^\boot}^p]$. This result requires us to provide bounds for
\begin{equation}
\label{eq:hpd_bounds_last_iterate}
\textstyle
\bar{\PE}^{1/p}[\norm{\theta_k - \thetas}^{p}] \quad \text{ and } \quad \bar{\PE}^{1/p}[\norm{\theta_k^{\boot} - \thetas}^{p}]\eqsp, \quad k \in \{1,\ldots,n-1\}\eqsp,
\end{equation}
with $p \simeq \log{n}$ and polynomial dependence on $p$. To control the second term in the r.h.s. of \eqref{crude bound for nonlinear stat} we note that the matrix $\Sigma_n^\boot$ concentrates around $\Sigma_n$ due to the matrix Bernstein inequality (see \Cref{lem:matrix_bernstein} for details). Hence, there is a set $\Omega_1$ such that $\PP(\Omega_1) \geq 1-1/n$ and $\lambda_{\min}(\Sigma_n^\boot) > 0$ on $\Omega_1$. Moreover, on this set we may use Berry-Essen-type bound for non-i.i.d. sums of random vectors. Detailed proof is given in \Cref{subsec:GAR in the bootstrap world}.
\end{proof}
\begin{remark}
The constants $M_{3,1}^\boot, M_{3,2}^\boot, M_{3,3}^\boot$ in \Cref{GAR bootstrap_main} depend on the problem dimension $d$ as well as on $C_{1,\xi}$, $C_{2,\xi}$, and $\|\theta_0 - \thetas \|$.
To make the dependence on $d$ explicit, we assume the natural scaling $
C_{1,\xi}, \ C_{2,\xi},\ \|\theta_0 - \theta^\ast\| \ \lesssim\ \sqrt{d}.
$
Under this assumption, \Cref{GAR bootstrap_main} yields
\begin{equation}
\sup_{B \in \Conv(\rset^d)}|\PPb( \sqrt n\{\Sigma_n^\boot\}^{-1/2} (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B) - \PPb(Y^\boot \in B)| \lesssim \frac{d^2}{n^{1/2}} + \frac{d^{5/4} \log n}{n^{\gamma-1/2}} + \frac{d^{3/4} \log^{3/2} n}{n^{\gamma/2}}
\end{equation}
Here the notation $\lesssim$ means that the inequality holds up to constants that do not depend on $n$ or $d$.
\end{remark}
\subsection{Rate of convergence in the Polyak--Juditsky central limit theorem}
\label{sec:shao_self-normalized}
In the final part of this section, we discuss the issue of transition from $\Sigma_n$ to $\Sigma_\infty$ and estimation of convergence rates in the Polyak--Juditsky result \eqref{eq:CLT_fort}. We utilize the result of \Cref{th:bound_kolmogorov_dist_pr_sigma_n} together with the following lemma.
\begin{lemma}
\label{lem:bound_kolmogorov_dist_sigma_n_sigma_infty}
Assume that \cref{ass:L-smooth} and \Cref{ass:step_size} hold. Let \( Y, Y^\prime \sim \mathcal{N}(0, \Id_d) \). Then, the Kolmogorov distance between the distributions of \( \Sigma_n^{1/2} Y \) and \( \Sigma_\infty^{1/2} Y^\prime \) is bounded by
\begin{equation}
\kolmogorov (\Sigma_n^{1/2} Y, \Sigma_\infty^{1/2} Y^\prime) \leq C_{\infty} n^{\gamma-1} \eqsp,
\end{equation}
where the constant \( C_{\infty} \) is defined in \eqref{eq:def_C_infty}.
\end{lemma}
\Cref{th:bound_kolmogorov_dist_pr_sigma_n}, \Cref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty}, and triangle inequality imply the following result on closeness to $\mathcal{N}(0,\Sigma_{\infty})$.
\begin{theorem}
\label{cor:berry-esseen}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($4$), \Cref{ass:step_size}. Then, with $Y \sim \mathcal{N}(0,\Id_d)$ it holds that
\begin{equation}
\label{eq:Berry-Esseen_Bound_Sigma_infty}
\kolmogorov(\sqrt{n}(\bar\theta_n -\thetas), \Sigma_\infty^{1/2} Y) \leq
\frac{\ConstC_4}{\sqrt{n}} + \frac{\ConstC_5}{n^{\gamma - 1/2}}+ \frac{\ConstC_{6}}{n^{\gamma/2}} + \frac{C_\infty}{ n^{1 - \gamma}} \eqsp,
\end{equation}
where $\ConstC_4, \ConstC_5$ and $\ConstC_6$ are given in Theorem \ref{th:bound_kolmogorov_dist_pr_sigma_n}.
\end{theorem}
\vspace{-2mm}
\paragraph{Discussion.} Theorem~\ref{th:bound_kolmogorov_dist_pr_sigma_n} reveals that the normal approximation through \(\mathcal{N}(0,\Sigma_n)\) improves when the step sizes \(\alpha_k\) are less aggressive, that is, as \(\gamma \to 1\). However, \Cref{cor:berry-esseen} shows that there is a trade-off, since the rate at which \(\Sigma_n\) converges to \(\Sigma_{\infty}\) also affects the overall quality of the approximation. Optimizing the bound in \eqref{eq:Berry-Esseen_Bound_Sigma_infty} for \(\gamma\) yields an optimal value of \(\gamma = 3/4\), leading to the following approximation rate:
\begin{align}
\label{eq:optimal_Berry-Esseen_Bound_Sigma_infty}
\kolmogorov(\sqrt{n}(\bar\theta_n -\thetas), \Sigma_\infty^{1/2} Y) \leq \frac{\ConstC_1^{\prime}}{n^{1/4}} + \frac{\ConstC_2^{\prime}}{\sqrt{n}}(\norm{\theta_0-\thetas}+\norm{\theta_0-\thetas}^2)\eqsp,
\end{align}
where $\ConstC_1^{\prime}$ and $\ConstC_2^{\prime}$ are instance-dependent quantities (but not depending on $\norm{\theta_0-\thetas}$), that can be inferred from \Cref{cor:berry-esseen}. Given the result of \Cref{cor:berry-esseen} one can proceed with a non-asymptotic evaluation of the methods for constructing confidence intervals based on direct estimation of $\Sigma_{\infty}$, such as \cite{chen2020aos,zhu2023online_cov_matr}.
\vspace{-2mm}
\paragraph{Lower bounds}
We provide a lower bound indicating that the bound \Cref{cor:berry-esseen} is tight at least in some regimes of step size decay power $\gamma \in (1/2,1)$. For this aim we consider minimization problem \eqref{eq:stoch_minimization} of the following form:
\[
f(\theta) = \theta^2/2 \to \min_{\theta \in \rset}\eqsp, \quad \theta_0 = 0\eqsp.
\]
In this case $\thetas = 0$. We consider an additive noise model, that is, the stochastic gradient oracles $ F(\theta,\xi)$ have a form $F(\theta,\xi) = \theta + \xi$, where $\xi \sim \mathcal{N}(0,1)$. Enrolling \eqref{eq:sgd_recursion_main}, we get
\begin{equation}
\label{eq:theta_k_recursion}
\theta_k = -\sum_{j=1}^{k}\alpha_{j}\prod_{\ell=j+1}^{k}(1- a \alpha_\ell) \xi_{j} \text{ and } \sqrt{n}\bar\theta_{n} = -\frac{1}{\sqrt{n}}\sum_{j=1}^{n-1}Q_{j} \xi_{j}\eqsp,
\end{equation}
where $Q_{j} = \alpha_{j}\sum_{k=j}^{n-1}\prod_{\ell=j+1}^{k}(1-\alpha_{\ell})$.
Note that
$\sqrt{n}(\bar\theta_{n} - \thetas)$ follows normal distribution $\mathcal{N}(0,\sigma^2_{n,\gamma})$ with $\sigma^2_{n,\gamma} = \frac{1}{n}\sum_{j=1}^{n-1}Q_{j}^2$.
Due to \Cref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty} (see also equation \eqref{eq:difference_sigma_n_infty_appendix} in the Appendix), we have $G = 1, \Sigma_{\infty} = 1$, and $\sigma^2_{n,\gamma} \to 1$ as $n \to \infty\eqsp$. Moreover, the following lower bound holds:
\begin{theorem}
\label{prop:lower_bounds}
Consider the sequence $\{\theta_k\}_{k \in \nset}$ defined by the recurrence \eqref{eq:theta_k_recursion} with $\alpha_{j} = c_0/(1+j)^{\gamma}$. Then it holds, for the number of observations $n$ sufficiently large, that
\begin{equation}
\label{eq:variance_lower_bound}
\textstyle
|\sigma^2_{n,\gamma} - 1| > \frac{C_1(\gamma,c_0)}{n^{1-\gamma}}\eqsp,
\end{equation}
where the constant $C_1(\gamma,c_0)$ depends only upon $c_0$ and $\gamma$. Moreover, for $n$ large enough
\begin{equation}
\label{eq:Kolmogorov_lower_bound}
\textstyle
\kolmogorov(\sqrt{n}(\bar\theta_{n} - \thetas), \mathcal{N}(0,1)) > \frac{C_2(\gamma,c_0)}{n^{1-\gamma}}\eqsp.
\end{equation}
\end{theorem}
\textbf{Discussion.} Proof of \Cref{prop:lower_bounds} is provided in \Cref{sec:proof_lower_bound}, together with some simple numerical simulations which indicate the tightness of the lower bound \eqref{eq:variance_lower_bound}. Note that the bound \eqref{eq:Kolmogorov_lower_bound} reveals that the distribution of $\sqrt{n}(\bar\theta_{n} - \thetas)$ can not be approximated by $\mathcal{N}(0,\Sigma_{\infty})$ with the rate faster than $1/n^{1-\gamma}$. Moreover, it shows that the rate of normal approximation in \Cref{cor:berry-esseen} can not be improved when $\gamma \in [3/4;1)$. This fact is extremely important when taking into account the bootstrap validity result of \Cref{th:bootstrap_validity} and normal approximation in \Cref{th:bound_kolmogorov_dist_pr_sigma_n}. Indeed, both results suggests that the rates of normal approximation of order up to $1/\sqrt{n}$ can be achieved when $\gamma \to 1$, but they require to consider another covariance matrix $\Sigma_n$, corresponding to the linearized recurrence in \eqref{eq:sigma_n_def}. At the same time, in the regime $\gamma \to 1$, the approximation by $\mathcal{N}(0,\Sigma_{\infty})$ can be too slow. It is an interesting and, to the best of our knowledge, open question to provide lower bounds analogous to \Cref{prop:lower_bounds} which show the tightness of other summands in \Cref{cor:berry-esseen} in the regime $1/2 < \gamma < 3/4$.
\section{Conclusion}
\input{sgd_conclusion}
\clearpage
\newpage
% Acknowledgments---Will not appear in anonymized version
\bibliographystyle{plain}
\bibliography{references}
\newpage
%Checklist
%\input{checklist.tex}
\newpage
% % \crefalias{section}{appendix} % uncomment if you are using cleveref
\appendix
\tableofcontents
\section{Proof of Theorem \ref{th:bootstrap_validity}}
\label{sec: bootstrap validity section}
\input{Bootstrap_validity}
\section{Proof of Theorem \ref{th:bound_kolmogorov_dist_pr_sigma_n}}
\label{sec:Gar_real_world_proof}
\input{GAR_proof}
\section{Proof of Theorem \ref{GAR bootstrap_main}}
\label{subsec:GAR in the bootstrap world}
\input{GAR_bootstrap_proof}
\section{Proof of Theorem \ref{cor:berry-esseen}}
\label{sec:Gar_real_world_sigma_infty}
\input{PYCLT_proof}
\section{Lower bounds}
\label{sec:proof_lower_bound}
\input{lower_bounds}
\section{Results for the last iterate}
\subsection{Last iterate bound}
\label{sec:last_iterate}
\input{last_iterate}
\subsection{High probability bounds on the last iterate}
\label{sec:high_prob_last_iterate}
\input{high-prob-bound}
\section{Technical bounds}
\label{sec:append_technical}
\input{technical_bounds.tex}
\end{document}
% \usepackage{amsfonts,amsmath,amssymb}
% \usepackage{xcolor}
% \usepackage{hyperref}
% \usepackage{srcltx}
% \usepackage{etoolbox}
% \usepackage{nameref}
% \usepackage{comment}
% \usepackage{mathrsfs}
% %\usepackage{bbm}
% \usepackage{enumerate}
% \usepackage[shortlabels]{enumitem}
% \usepackage{amsfonts} % blackboard math symbols
% \usepackage{nicefrac} % compact symbols for 1/2, etc.
% \usepackage{mathtools}
% \usepackage{dsfont,mathrsfs}
% \usepackage{cleveref}
% \usepackage{tikz-cd}
% %\usepackage{caption}
% %\usepackage{subcaption}
% \usepackage{multirow}
% \usepackage{colortbl}
% \definecolor{bgcolor}{rgb}{0.8,1,1}
% \definecolor{bgcolor2}{rgb}{0.8,1,0.8}
% \definecolor{niceblue}{rgb}{0.0,0.19,0.56}
% \usepackage{threeparttable}
% %for coloured ticks and crosses
% \newcommand{\ag}[1]{{\color{orange}#1}}
% \newcommand{\myred}[1]{{\color{red}#1}}
% \newcommand{\myblue}[1]{{\color{blue}#1}}
% \usepackage{tcolorbox}
% \usepackage{pifont}
% \definecolor{mydarkgreen}{RGB}{39,130,67}
% \definecolor{mydarkred}{RGB}{192,47,25}
% \newcommand{\green}{\color{mydarkgreen}}
% \newcommand{\red}{\color{mydarkred}}
% \newcommand{\cmark}{{\green\ding{51}}}
% \newcommand{\xmark}{{\red\ding{55}}}
% \usepackage{xcolor}
% %\PassOptionsToPackage{linktocpage}{hyperref}
% %\usepackage[colorlinks=true, allcolors=blue]{hyperref}
% \definecolor{dmorange500}{HTML}{FF5F19}
% \definecolor{dmblue300}{HTML}{2267EB}
% \definecolor{dmred300}{HTML}{FF617B}
% \hypersetup{
% colorlinks,
% citecolor=dmblue300
% }
% %\usepackage[textwidth=2.25cm,textsize=footnotesize]{todonotes}
% %\usepackage[disable]{todonotes}
% %\newcommand{\eric}[1]{\todo{{\bf E:} #1}}
% %\newcommand{\erici}[1]{\todo[color=red!20,inline]{{\bf EM:} #1}}
% %\newcommand{\alain}[1]{\todo{{\bf AD:} #1}}
% %\newcommand{\alaini}[1]{\todo[color=red!20,inline]{{\bf AD:} #1}}
% %\newcommand{\alex}[1]{{\todo[color=blue!20]{{\bf A:} #1}}}
% %\newcommand{\alexi}[1]{\todo[color=blue!20,inline]{{\bf A:} #1}}
% % \newcommand{\som}[1]{\todo[color=green!20]{{\bf SS:} #1}}
% %\newcommand{\somi}[1]{\todo[color=green!20,inline]{{\bf SS:} #1}}
% %\usepackage{bm}
% \usepackage{xargs}
% \usepackage{multirow}
% \usepackage{aliascnt}
% \usepackage{autonum}
% \usepackage{upgreek}
% \newtheorem{property}{Fact}
% \newtheorem{assum}{A\hspace{-2pt}}
% \newtheorem{assumb}{B\hspace{-2pt}}
% \newtheorem{assumID}{IND\hspace{-2pt}}
% \newtheorem{assumUGE}{UGE\hspace{-2pt}}
% \newtheorem{assumUE}{UE\hspace{-2pt}}
% \newtheorem{assumSUP}{M\hspace{-2pt}}
% %\newtheorem{theorem}{Theorem}
% \crefname{theorem}{theorem}{Theorems}
% \Crefname{theorem}{Theorem}{Theorems}
% % Define a new counter for lemmas
% \newcounter{lemma}
% \renewcommand{\thelemma}{\arabic{lemma}}
% % Define the lemma environment
% \renewenvironment{lemma}[1][]{
% \refstepcounter{lemma}
% \noindent\textbf{Lemma~\thelemma. #1}\itshape
% }{\par}
% % Set up cleveref aliases
% \crefname{lemma}{lemma}{lemmas}
% \Crefname{lemma}{Lemma}{Lemmas}
% %\newaliascnt{remark}{theorem}
% %\newtheorem{remark}[remark]{Remark}
% %\aliascntresetthe{remark}
% \crefname{remark}{remark}{remarks}
% \Crefname{remark}{Remark}{Remarks}
% % Define a new counter for corollaries
% \newcounter{corollary}
% \renewcommand{\thecorollary}{\arabic{corollary}}
% % Define the corollary environment
% \renewenvironment{corollary}[1][]{
% \refstepcounter{corollary}
% \noindent\textbf{Corollary~\thecorollary. #1}\itshape
% }{\par}
% \crefname{corollary}{corollary}{corollaries}
% \Crefname{corollary}{Corollary}{Corollaries}
% %\newaliascnt{proposition}{theorem}
% %\newtheorem{proposition}{Proposition}
% %\aliascntresetthe{proposition}
% \crefname{proposition}{proposition}{propositions}
% \Crefname{proposition}{Proposition}{Propositions}
% %\newaliascnt{definition}{theorem}
% %\newtheorem{definition}[definition]{Definition}
% %\aliascntresetthe{definition}
% \crefname{definition}{definition}{definitions}
% \Crefname{Definition}{Definition}{Definitions}
% %\newtheorem{example}[theorem]{Example}
% \crefname{example}{example}{examples}
% \Crefname{Example}{Example}{Examples}
% \crefname{figure}{figure}{figures}
% \Crefname{Figure}{Figure}{Figures}
% \crefname{table}{table}{tables}
% \Crefname{Table}{Table}{Tables}
% \crefname{algorithm}{algorithm}{algorithms}
% \Crefname{Algorithm}{Algorithm}{Algorithms}
% \crefname{assum}{A\hspace{-3pt}}{A\hspace{-3pt}}
% \crefname{assumb}{B\hspace{-2pt}}{B\hspace{-2pt}}
% \crefname{assumUGE}{UGE\hspace{-1pt}}{UGE\hspace{-1pt}}
% \crefname{assumID}{IND\hspace{-1pt}}{IND\hspace{-1pt}}
% \crefname{assumUE}{UE\hspace{-1pt}}{UE\hspace{-1pt}}
% \crefname{assumM}{M\hspace{-1pt}}{M\hspace{-1pt}}
% \newlist{renumerate}{enumerate}{3}
% \setlist[renumerate]{wide, labelwidth=!, labelindent=0pt,label=(\roman*)}
% \newlist{aenumerate}{enumerate}{3}
% \setlist[aenumerate]{wide, labelwidth=!, labelindent=0pt,label=(\arabic*)}
% \newlist{aaenumerate}{enumerate}{3}
% \setlist[aaenumerate]{wide, labelwidth=!, labelindent=0pt,label=(\alph*)}
% \newlist{aenumerateSpace}{enumerate}{3}
% \setlist[aenumerateSpace]{wide, labelwidth=!,label=(\arabic*)}
% \newlist{benumerate}{enumerate}{3}
% \setlist[benumerate]{wide, labelwidth=!, labelindent=0pt,label=$\bullet$}
\usepackage{amsfonts,amsmath,amssymb}
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{srcltx}
\usepackage{etoolbox}
\usepackage{nameref}
\usepackage{comment}
\usepackage{mathrsfs}
%\usepackage{bbm}
\usepackage{enumerate}
\usepackage[shortlabels]{enumitem}
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{mathtools}
\usepackage{dsfont,mathrsfs}
\usepackage{cleveref}
\usepackage{tikz-cd}
\usepackage{amsthm}
%\usepackage{caption}
%\usepackage{subcaption}
\usepackage{multirow}
\usepackage{colortbl}
\definecolor{bgcolor}{rgb}{0.8,1,1}
\definecolor{bgcolor2}{rgb}{0.8,1,0.8}
\definecolor{niceblue}{rgb}{0.0,0.19,0.56}
\usepackage{threeparttable}
%for coloured ticks and crosses
\newcommand{\ag}[1]{{\color{orange}#1}}
\newcommand{\myred}[1]{{\color{red}#1}}
\newcommand{\myblue}[1]{{\color{blue}#1}}
\usepackage{tcolorbox}
\usepackage{pifont}
\definecolor{mydarkgreen}{RGB}{39,130,67}
\definecolor{mydarkred}{RGB}{192,47,25}
\newcommand{\green}{\color{mydarkgreen}}
\newcommand{\red}{\color{mydarkred}}
\newcommand{\cmark}{{\green\ding{51}}}
\newcommand{\xmark}{{\red\ding{55}}}
\usepackage{xcolor}
%\PassOptionsToPackage{linktocpage}{hyperref}
%\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\definecolor{dmorange500}{HTML}{FF5F19}
\definecolor{dmblue300}{HTML}{2267EB}
\definecolor{dmred300}{HTML}{FF617B}
\hypersetup{
colorlinks,
citecolor=dmblue300
}
%\usepackage[textwidth=2.25cm,textsize=footnotesize]{todonotes}
%\usepackage[disable]{todonotes}
%\newcommand{\eric}[1]{\todo{{\bf E:} #1}}
%\newcommand{\erici}[1]{\todo[color=red!20,inline]{{\bf EM:} #1}}
%\newcommand{\alain}[1]{\todo{{\bf AD:} #1}}
%\newcommand{\alaini}[1]{\todo[color=red!20,inline]{{\bf AD:} #1}}
%\newcommand{\alex}[1]{{\todo[color=blue!20]{{\bf A:} #1}}}
%\newcommand{\alexi}[1]{\todo[color=blue!20,inline]{{\bf A:} #1}}
% \newcommand{\som}[1]{\todo[color=green!20]{{\bf SS:} #1}}
%\newcommand{\somi}[1]{\todo[color=green!20,inline]{{\bf SS:} #1}}
%\usepackage{bm}
\usepackage{xargs}
\usepackage{multirow}
\usepackage{aliascnt}
\usepackage{autonum}
\usepackage{upgreek}
\newtheorem{property}{Fact}
\newtheorem{assum}{A\hspace{-2pt}}
\newtheorem{assumb}{B\hspace{-2pt}}
\newtheorem{assumID}{IND\hspace{-2pt}}
\newtheorem{assumUGE}{UGE\hspace{-2pt}}
\newtheorem{assumUE}{UE\hspace{-2pt}}
\newtheorem{assumSUP}{M\hspace{-2pt}}
\newtheorem{theorem}{Theorem}
\crefname{theorem}{theorem}{Theorems}
\Crefname{theorem}{Theorem}{Theorems}
% Define a new counter for lemmas
\newcounter{lemma}
\renewcommand{\thelemma}{\arabic{lemma}}
% Define the lemma environment
\newenvironment{lemma}[1][]{
\refstepcounter{lemma}
\noindent\textbf{Lemma~\thelemma. #1}\itshape
}{\par}
% Set up cleveref aliases
\crefname{lemma}{lemma}{lemmas}
\Crefname{lemma}{Lemma}{Lemmas}
%\newaliascnt{remark}{theorem}
%\newtheorem{remark}[remark]{Remark}
%\aliascntresetthe{remark}
\newcounter{remark}
\renewcommand{\theremark}{\arabic{remark}}
% Define the corollary environment
\newenvironment{remark}[1][]{
\refstepcounter{remark}
\noindent\textbf{Remark~\theremark. #1}\itshape
}{\par}
\crefname{remark}{remark}{remarks}
\Crefname{remark}{Remark}{Remarks}
% Define a new counter for corollaries
\newcounter{corollary}
\renewcommand{\thecorollary}{\arabic{corollary}}
% Define the corollary environment
\newenvironment{corollary}[1][]{
\refstepcounter{corollary}
\noindent\textbf{Corollary~\thecorollary. #1}\itshape
}{\par}
\crefname{corollary}{corollary}{corollaries}
\Crefname{corollary}{Corollary}{Corollaries}
%\newaliascnt{proposition}{theorem}
%\newtheorem{proposition}{Proposition}
%\aliascntresetthe{proposition}
% Define a new counter for corollaries
\newcounter{proposition}
\renewcommand{\theproposition}{\arabic{proposition}}
% Define the corollary environment
\newenvironment{proposition}[1][]{
\refstepcounter{proposition}
\newline\textbf{Proposition~\theproposition. #1}\itshape
}{\par}
\crefname{proposition}{proposition}{propositions}
\Crefname{proposition}{Proposition}{Propositions}
%\newaliascnt{definition}{theorem}
%\newtheorem{definition}[definition]{Definition}
%\aliascntresetthe{definition}
\crefname{definition}{definition}{definitions}
\Crefname{Definition}{Definition}{Definitions}
%\newtheorem{example}[theorem]{Example}
\crefname{example}{example}{examples}
\Crefname{Example}{Example}{Examples}
\crefname{figure}{figure}{figures}
\Crefname{Figure}{Figure}{Figures}
\crefname{table}{table}{tables}
\Crefname{Table}{Table}{Tables}
\crefname{algorithm}{algorithm}{algorithms}
\Crefname{Algorithm}{Algorithm}{Algorithms}
\crefname{assum}{A\hspace{-3pt}}{A\hspace{-3pt}}
\crefname{assumb}{B\hspace{-2pt}}{B\hspace{-2pt}}
\crefname{assumUGE}{UGE\hspace{-1pt}}{UGE\hspace{-1pt}}
\crefname{assumID}{IND\hspace{-1pt}}{IND\hspace{-1pt}}
\crefname{assumUE}{UE\hspace{-1pt}}{UE\hspace{-1pt}}
\crefname{assumM}{M\hspace{-1pt}}{M\hspace{-1pt}}
\newlist{renumerate}{enumerate}{3}
\setlist[renumerate]{wide, labelwidth=!, labelindent=0pt,label=(\roman*)}
\newlist{aenumerate}{enumerate}{3}
\setlist[aenumerate]{wide, labelwidth=!, labelindent=0pt,label=(\arabic*)}
\newlist{aaenumerate}{enumerate}{3}
\setlist[aaenumerate]{wide, labelwidth=!, labelindent=0pt,label=(\alph*)}
\newlist{aenumerateSpace}{enumerate}{3}
\setlist[aenumerateSpace]{wide, labelwidth=!,label=(\arabic*)}
\newlist{benumerate}{enumerate}{3}
\setlist[benumerate]{wide, labelwidth=!, labelindent=0pt,label=$\bullet$}
\newtheorem{assumprime}{\textbf{A'}\hspace{-1pt}}
\Crefname{assumprime}{\textbf{A'}\hspace{-1pt}}{\textbf{A'}\hspace{-1pt}}
\crefname{assumprime}{\textbf{A'}}{\textbf{A'}}
\newcommand{\dto}{\overset{d}{\to}}
\def\tvdist{\mathsf{d}_{\operatorname{tv}}}
\def\supconsteps{\supnorm{\funnoisew}}
\def\lesssimd{\lesssim_{d}}
\def\Cg{\operatorname{C}_{g}}
\def\driftV{V_{p}}
\def\driftVm{W_{p}}
\def\driftVmq{W_{q}}
\def\driftVs{W_{2}}
\def\driftVbars{\bar{W}_{2}}
\def\cost{\metricc}
\def\costV{\cost_V}
\def\chain{\theta}
\def\lambdap{\lambda_{2p}}
\newcommand{\PE}{\mathbb{E}}
\newcommand{\var}{\operatorname{Var}}
\newcommand{\PP}{\mathbb{P}}
\newcommandx{\genericb}[1][1=]{b_{#1}}
\newcommandx{\Constros}[1][1=]{\operatorname{C}_{\operatorname{Ros},#1}}
\newcommandx{\Constburk}[1][1=]{\operatorname{C}_{\operatorname{Burk}}}
\newcommandx{\driftW}[1][1=]{W_{#1}}
\def\DConstros{\operatorname{D}_{\operatorname{Ros}}}
\newcommand{\DConst}[1]{\operatorname{D}_{{\operatorname{Ros}}, #1}}
\def\Auxconst{\operatorname{c}}
\def\linearfun{\ell}
\def\metricz{\mathsf{d}_{\Zset}}
\newcommandx{\metricd}[1][1=]{\mathsf{d}_{#1}}
\def\Couplset{\mathscr{C}}
\newcommandx\invmeasure[1][1=]{\Pi_{#1}}
\newcommandx{\PPjoint}[1][1=]{\PP^{\MKjoint[#1]}}
\newcommandx{\PEjoint}[1][1=]{\PE^{\MKjoint[#1]}}
\newcommandx{\PEMID}[1][1=\alpha]{\PE^{\MK[#1]}}
\newcommandx{\PPMID}[1][1=\alpha]{\PP^{\MK[#1]}}
\def\xnormbound{\operatorname{M}}
\newcommand{\supnorm}[1]{\norm{ #1 }[\infty]}
\newcommandx{\MKjoint}[1][1=]{\bar{\operatorname{P}}_{#1}}
\newcommandx\costw[1][1=]{\mathsf{c}_{#1}}
\def\costwt{\tilde{\mathsf{c}}}
\newcommandx\Intergrdist[1][1=]{\mathbb{M}_{1}(#1)}
\def\Wspace{\mathbb{S}}
\def\wasserdist{\mathbf{W}}
\newcommandx{\mmarkov}[1][1=0]{m^{(\Markov)}_{#1}}
\def\constlip{\varkappa}
\def\metricbound{d_0}
\def\AuxconstM{c^{(\operatorname{M})}}
\def\cantheta{\vartheta}
\def\remB{\operatorname{B}}
\def\initdistr{\Xi}
\def\Deltatr{\Delta^{\operatorname{(tr)}}}
\def\Deltafl{\Delta^{\operatorname{(fl)}}}
\def\MarDeltatr{R^{\operatorname{(tr)}}}
\def\MarDeltafl{R^{\operatorname{(fl)}}}
\def\Conv{\mathsf{C}}
\def\seta{\mathcal{A}}
\def\sets{\mathcal{S}}
\def\H{\mathcal{H}}
\def\F{\mathcal{F}}
\def\Xset{\mathsf{X}}
\def\Xsigma{\mathcal{X}}
\def\Yset{\mathsf{Y}}
\def\Ysigma{\mathcal{Y}}
\def\Zset{\mathsf{Z}}
\def\Zsigma{\mathcal{Z}}
\def\LL{\mathsf{K}}
\def\MM{\mathsf{M}}
\def\rset{\mathbb{R}}
\def\qset{\ensuremath{\mathbb{Q}}}
\def\nset{\ensuremath{\mathbb{N}}}
\def\nsets{\ensuremath{\mathbb{N}^*}}
\def\nsetm{\ensuremath{\mathbb{N}_-}}
\def\zset{\ensuremath{\mathbb{Z}}}
\def\cset{\ensuremath{\mathbb{C}}}
\newcommand{\round}[1]{\ensuremath{\lfloor#1\rfloor}}
\newcommand{\roundup}[1]{\ensuremath{\lceil#1\rceil}}
\newcommand{\tih}{{\tilde h}}
\newcommand{\bh}{{\bar h}}
\newcommand{\msi}{\mathsf{I}}
\newcommand{\msj}{\mathsf{J}}
\newcommand{\comp}{\text{c}}
\newcommand{\SG}{\operatorname{SG}}
\newcommand{\Mat}[1]{{\bf{#1}}}
\def\MatB{B}
\def\MatP{P}
\def\covfeat{\Sigma_\varphi}
\renewcommand{\S}{\mathcal{S}}
\newcommand{\A}{\mathcal{A}}
\def\diag{\operatorname{diag}}
\def\PMDP{\MKQ}
\newcommand{\mrl}{\mathrm{L}}
\newcommand{\PCov}[1][1=]{\ensuremath{\operatorname{Cov}_{#1}}}
\newcommand{\Const}[1]{\operatorname{C}_{{#1}}}
\newcommand{\stConst}[1]{\operatorname{C}_{{#1}}^{\mathsf{st}}}
\newcommand{\aConst}[1]{\operatorname{C}_{{#1}}^{\alpha}}
\newcommand{\bConst}[1]{\operatorname{C}_{{\bf #1}}}
\newcommand{\bConstM}[1]{\operatorname{C}^{(\Markov)}_{{\bf #1}}}
\newcommand{\noiseconst}[1]{\operatorname{C}_{{\bf #1}}}
\newcommand{\smallConst}[1]{\operatorname{c}_{{#1}}}
\newcommand{\smallConstm}[1]{\operatorname{c}_{{#1}}^{(\Markov)} }
\def\Cmax{\operatorname{C}_{\max}}
\def\sfA{\mathsf{A}}
\def\sfB{\mathsf{B}}
\def\sfC{\mathsf{C}}
\def\sfD{\mathsf{D}}
% copied from Rosenthal's paper
\def\consta{\mathrm{a}}
\def\Consta{\mathrm{A}}
\def\Constb{\mathrm{B}}
\newcommand{\msy}{\mathsf{Y}}
\def\beps{\bar \varepsilon}
\def\thetainit{\theta_0}
\newcommandx\sequence[4][2=,3=,4=]
{\ifthenelse{\equal{#3}{}}{\ensuremath{\{ #1_{#2 #4}\}}}{\ensuremath{\{ #1_{#2 #4} \}_{#2 \in #3}}}}
\newcommandx\sequenceD[2][2=]
{\ifthenelse{\equal{#2}{}}{\ensuremath{\{ #1\}}}{\ensuremath{\{ #1\!~:\!~#2 \}}}}
\newcommandx\sequenceDouble[4][3=,4=]
{\ifthenelse{\equal{#3}{}}{\ensuremath{\{ (#1_{#3},#2_{#3})\}}}{\ensuremath{\{ (#1_{#3},#2_{#3})\}_{#3 \in #4}}}}
\newcommandx{\sequencen}[2][2=n\in\nset]{\ensuremath{\{ #1, \eqsp #2 \}}}
\newcommandx\sequencens[2][2=n]
{\ensuremath{\{ #1_{#2} \!~:\!~#2\in\nsets\}}}
\newcommandx\sequencet[4]
{\ensuremath{\{ #1{#2_{#3}} \, : \, \eqsp #3 \in #4 \}}}
\def\PE{\mathbb{E}}
\def\P{\mathbb{P}}
\def\ProdB{\Gamma}
\def\tProdB{\tilde{\Gamma}}
\def\Ga{G^{(\alpha)}}
\def\BlockDet{\bar{\Y}}
\def\gPois{\hat{g}}
\def\funcPois{\hat{f}}
\def\func{f}
\def\BlockB{\overline{B}}
\def\Cstepblock{c_1}
\newcommandx{\PVar}[1][1=]{\ensuremath{\operatorname{Var}_{#1}}}
\newcommandx\conststab[1][1=p]{\varkappa_{#1}}
\newcommand{\term}[2]{\mathsf{R}_{#2}(#1)}
\def\tildeo{\widetilde{\mathcal{O}}}
\newcommand{\ConstPR}[1]{\mathsf{C}_{#1}}
%\def\Cov
\def\noisecov{\Sigma_\xi}
\def\noisecovmarkov{\Sigma^{(\Markov)}_\varepsilon}
\def\metrics{\mathsf{d}_{\S}}
\def\ProdBa{\ProdB^{(\alpha)}}
\def\ProdBdet{G^{(\alpha)}}
\newcommandx{\MK}[1][1=\alpha]{\mathrm{P}_{#1}}
\newcommandx\MKK[1][1=\alpha]{\mathrm{K}_{#1}}
\def\MKQ{\mathrm{P}}
\def\MKR{\mathrm{R}}
%\def\MKJ{\mathrm{J}}
\def\filtr{\mathcal{F}}
\def\PEhat{\widehat{\PE}}
\def\PPhat{\widehat{\PP}}
\newcommandx{\PEtilde}[1][1=]{\PE^{\mathrm{K}_{#1}}}
\newcommandx{\PPtilde}[1][1=]{\PP^{\mathrm{K}_{#1}}}
\newcommand{\PEext}{\tilde{\PE}}
\newcommand{\PPext}{\tilde{\PP}}
%\newcommandx{\PEtilde}[1][1=]{\PE^{\mathrm{K}_{#1}}}
%\newcommandx{\PPtilde}[1][1=]{\PP^{\mathrm{K}_{#1}}}
%\def\MKQ{\mathrm{Q}_{m}}
\def\metricc{\mathsf{c}}
\def\Sigmabf{\boldsymbol{\Sigma}}
\def\lineG{\Sigmabf}
\def\lineGa{\Sigmabf^{\alpha}}
\newcommand{\abs}[1]{\vert #1\vert}
\newcommand{\absLigne}[1]{\vert #1\vert}
\newcommand{\absD}[1]{\left\vert #1\right\vert}
\newcommandx{\norm}[2][2=]{\Vert#1 \Vert_{{#2}}}
\newcommandx{\normLigne}[2][2=]{\Vert#1 \Vert_{{#2}}}
\newcommandx{\normLine}[2][2=]{\Vert#1 \Vert_{{#2}}}
\newcommandx{\normop}[2][2=]{\Vert{#1}\Vert_{{#2}}}
\newcommandx{\normopLigne}[2][2=]{\Vert{#1}\Vert_{{#2}}}
\newcommandx{\normopLine}[2][2=]{\Vert{#1}\Vert_{{#2}}}
%\newcommandx{\normopa}[2][2=]{\left\|{#1}\right\|_{{#2}}}
\newcommandx{\osc}[2][1=]{\mathrm{osc}_{#1}(#2)}
\newcommandx{\normopadapt}[2][2=]{\left\Vert{#1}\right\Vert_{{#2}}}
%%%%% These commands were used in the paper but never defined %%%%
\newcommandx{\normlip}[2][2=\operatorname{Lip}]{\Vert#1 \Vert_{{#2}}}
\newcommand{\lip}{\operatorname{L}}
\newcommandx{\lipspace}[1]{\lip_{#1}}
\newcommandx{\CPP}[3][1=]
{\ifthenelse{\equal{#1}{}}{{\mathbb P}\left(\left. #2 \, \right| #3 \right)}{{\mathbb P}_{#1}\left(\left. #2 \, \right | #3 \right)}}
\newcommandx{\CPPtilde}[3][1=]
{\ifthenelse{\equal{#1}{}}{{\tilde{\mathbb P}}\left(\left. #2 \, \right| #3 \right)}{{\tilde{\mathbb P}}_{#1}\left(\left. #2 \, \right | #3 \right)}}
\newcommand{\chunk}[3]{{#1}_{#2}^{#3}}
\def\Bell{\operatorname{Bell}}
\newcommand{\ensemble}[2]{\left\{#1\,:\eqsp #2\right\}}
\newcommand{\setens}[2]{\ensemble{#1}{#2}}
\def\etal{et al.}
\def\wrt{with respect to}
\def\iff{if and only if}
%\newcommand{\wrt}{w.r.t.}
\def\rhs{right-hand side}
\def\lhs{left-hand side}
\def\iid{i.i.d.}
\renewcommand{\-}{\mbox{-}}
\newcommandx{\as}[1][1=\PP]{\ensuremath{#1\, -\mathrm{a.s.}}}
\newcommand{\mae}{\textit{a.e.}}
\newcommand{\aae}[1]{\ensuremath{#1-\text{a.e.}}}
\newcommand{\aall}[1]{\ensuremath{#1-\mathrm{almost\ all}}}
\newcommand{\ie}{i.e.}
\newcommand{\wlogg}{without loss of generality}
\newcommand{\io}{\ensuremath{\mathrm{i.o.}}}
\newcommand{\ifof}{if and only if}
\newcommand{\eg}{e.g.}
\newcommand{\cf}{cf.}
\newcommand{\pdf}{probability density function}
\newcommand{\eqsp}{\;}
\newcommand{\eqspp}{\ \ }
\newcommand{\eqsppeq}{\quad \quad }
\newcommand{\Id}{\mathrm{I}}
\def\prtheta{\bar{\theta}}
\def\ttheta{\tilde{\theta}}
\def\utheta{\tilde{\theta}^{\sf (tr)}}
\def\vtheta{\tilde{\theta}^{\sf (fl)}}
\def\noise{\varepsilon}
\newcommand{\ConstC}{\operatorname{C}}
\newcommand{\ConstD}{\mathsf{D}}
\newcommand{\ConstDM}{\ConstD^{(\operatorname{M})}}
\newcommand{\ConstM}{\operatorname{M}}
\newcommand{\ConstK}{\operatorname{K}}
\def\nzero{n_0}
%\newcommandx\lpnorm[2][1=p]{\left\Vert #2 \right\Vert_{#1}}
\def\minorwas{\varepsilon}
\def\minorconst{\epsilon}
\newcommandx{\boundmetric}[1][1=]{\kappa_{\MKK[#1]}}
\newcommand{\CKset}{\bar{\mathsf{C}}}
\newcommand{\Jnalpha}[2]{J_{#1}^{(#2)}}
\newcommand{\Hnalpha}[2]{H_{#1}^{(#2)}}
\newcommand{\Jnalphawn}[1]{J^{(\alpha,#1)}}
\newcommand{\Hnalphawn}[1]{H^{(\alpha,#1)}}
\newcommand{\coint}[1]{\left[#1\right)}
\newcommand{\cointLine}[1]{[#1)}
\newcommand{\ocint}[1]{\left(#1\right]}
\newcommand{\ocintLine}[1]{(#1]}
\newcommand{\ooint}[1]{\left(#1\right)}
\newcommand{\oointLine}[1]{(#1)}
\newcommand{\ccint}[1]{\left[#1\right]}
\newcommand{\ccintLine}[1]{[#1]}
\def\LVset{\operatorname{L}_\infty^V}
\def\bLVset{\operatorname{L}_\infty^{\bar V}}
%\newcommand{\Vnorm}[2]{\ensuremath{\left\Vert#1\right\Vert_{\mathrm{#2}}}}
\newcommand{\VnormD}[2]{\ensuremath{\left\Vert#1\right\Vert_{#2}}}
\newcommand{\Vnorm}[2][1=V]{\| #2 \|_{#1}}
\newcommandx{\Nnorm}[2][1=V]{[ #2]_{#1}}
\newcommandx{\lipnorm}[2][1=g]{[ #1]_{#2}}
\def\Lclass{\mathcal L}
\newcommand{\PEC}{\overline{\mathbb{E}}}
\newcommandx{\CPE}[3][1=]{{\mathbb E}^{#3}_{#1}\left[#2\right]}
\newcommandx{\CPEext}[3][1=]{\tilde{\mathbb E}^{#3}_{#1}\left[#2\right]}
\newcommandx{\CPEtilde}[3][1=]{{\tilde{\mathbb E}}^{#3}_{#1}\left[#2\right]}
\newcommandx{\CPEs}[3][1=]{{\mathbb E}^{#3}_{#1}[#2]}
\def\bk{{\boldsymbol k}}
\def\bnu{\boldsymbol{\nu}}
\newcommand{\momentGq}[1][1=q]{\operatorname{G}_{#1}}
\def\htheta{\Theta}
\def\thetalim{\theta^\star}
\def\lthetalim{\Theta^\star}
\newcommand{\psmat}[2]{\langle #1, #2 \rangle}
\def\trace{\operatorname{Tr}}
\newcommand{\tvnorm}[1]{\left\Vert #1 \right\Vert_{\mathrm{TV}}}
\newcommand{\rmi}{\mathrm{i}}
\newcommand{\rme}{\mathrm{e}}
\newcommand{\rmd}{\mathrm{d}}
\def\funcAw{\mathbf{A}}
\def\funcBw{\mathbf{B}}
\newcommand{\funcA}[1]{\funcAw(#1)}
\newcommand{\funcF}[2]{\mathbf{F}_{#1}(#2)}
\def\funcbw{\mathbf{b}}
\newcommand{\funcb}[1]{\funcbw(#1)}
\newcommandx{\zmfuncA}[2][1=]{\tilde{\funcAw}^{#1}(#2)}
\newcommandx{\zmfuncAw}[1][1=]{\tilde{\funcAw}_{#1}}
\newcommandx{\zmfuncb}[2][1=]{\tilde{\funcbw}^{#1}(#2)}
\def\funnoisew{\varepsilon}
\newcommand{\funcnoise}[1]{\funnoisew(#1)}
\def\hfunnoisew{\widehat{\varepsilon}}
\newcommand{\hfunnoise}[1]{\hfunnoisew(#1)}
\newcommand{\funcAt}[1]{\funcAtilde(#1)}
\newcommand{\funcbt}[1]{\funcbtilde(#1)}
\newcommandx{\funcct}[2][1=]{\funcctilde^{#1}(#2)}
\newcommand{\frobnorm}[1]{\left\Vert #1 \right\Vert_{\mathrm{F}}}
\newcommand{\frobnormLine}[1]{\Vert #1 \Vert_{\mathrm{F}}}
\def\Cave{C_{\operatorname{ave}}}
\def\CpV{C_{p,V}}
\def\qcond{\kappa_{Q}}
\def\tqcond{\kappa_{\tQ}}
\def\qcondtild{\kappa_{\mathsf{\tilde{Q}}}}
\def\myqcond{\kappa}
\def\State{Z}
\def\state{z}
\def\tState{\tilde{Z}}
\def\Stateset{{\sf \State}}
\def\Statesigma{\mathcal{\State}}
\def\Cset{\operatorname{C}}
\def\CnormA{C_A}
\def\eligibility{\varphi}
\def\taumix{t_{\operatorname{mix}}}
\newcommand{\pscal}[2]{\langle#1\,,\,#2\rangle}
\newcommand{\indiacc}[1]{\boldsymbol{1}_{\{#1\}}}
\newcommand{\1}{\boldsymbol{1}}
\newcommand{\indin}[1]{\boldsymbol{1}\left\{#1\right\}}
\newcommand{\indinD}[1]{\boldsymbol{1}\{#1\}}
\def\linearbellman{\operatorname{T}}
\def\expo{\operatorname{t}}
\def\bb{\operatorname{b}}
\def\policy{\pi}
\newcommand{\ps}[2]{#1^\top #2}
\def\boundTDlambda{\operatorname{\epsilon}_{\max}}
\def\gain{\mathrm{R}}
\newcommand{\tcr}[1]{\textcolor{red}{#1}}
\newcommand{\tcg}[1]{\textcolor{green}{#1}}
\newcommandx{\CovC}[1][1=u]{\operatorname{C}_{#1}}
\def\poisson{\operatorname{Pois}}
\usepackage[textwidth=2.25cm, textsize=footnotesize]{todonotes}
% \usepackage[disable]{todonotes}
\newcommand{\eric}[1]{\todo{\textbf{ E:} #1}}
\newcommand{\erici}[1]{\todo[color=red!20,inline]{{\bf EM:} #1}}
\newcommand{\alex}[1]{{\todo[color=blue!20]{\textbf{ A:} #1}}}
\newcommand{\alexi}[1]{\todo[color=blue!20,inline]{{\bf A:} #1}}
\newcommand{\som}[1]{\todo[color=green!20]{\textbf{SS:} #1}}
\newcommand{\somi}[1]{\todo[color=green!20,inline]{{\bf SS:} #1}}
\newcommand{\BB}[1]{\todo[color=green!20]{{\bf BB:} #1}}
\newcommand{\Marina}[1]{\todo[color=green!20]{\textbf{ Marina:} #1}}
\def\red{\color{red}}
\def\msa{\mathsf{A}}
\def\msb{\mathsf{B}}
\def\msz{\mathsf{Z}}
\def\mcz{\mathcal{Z}}
\newcommand\borel[1]{\mathcal{B}(#1)}
\def\msc{\mathsf{C}}
\def\mcc{\mathcal{C}}
\def\msd{\mathsf{D}}
\def\mcd{\mathcal{D}}
\def\plusinfty{+\infty}
\DeclareMathAlphabet{\mathpzc}{OT1}{pzc}{m}{it}
\def\lyapV{\mathpzc{V}}
\def\Vtwop{\mathpzc{V}_{2p}}
\def\lyapW{\mathpzc{W}}
\def\Cf{C_f}
\def\CW{C_{\lyapW}}
\newcommand{\txts}{\textstyle}
\def\biasw{\operatorname{bias}}
\newcommandx{\bias}[1][1=\alpha]{\operatorname{B}_{#1}}
\newcommand{\proba}[1]{\mathbb{P}\left( #1 \right)}
\newcommand{\probasq}[1]{\mathbb{P}^{1/2}\left( #1 \right)}
\newcommand{\probasqLigne}[1]{\mathbb{P}^{1/2}( #1 )}
\newcommand{\probaCond}[2]{\mathbb{P}\left( \left. #1 \middle\vert #2 \right.\right)}
\newcommand{\probaLigne}[1]{\mathbb{P}( #1 )}
\newcommandx\probaMarkovTilde[2][2=]
{\ifthenelse{\equal{#2}{}}{{\widetilde{\mathbb{P}}_{#1}}}{\widetilde{\mathbb{P}}_{#1}\left[ #2\right]}}
\newcommand{\probaMarkov}[2]{\mathbb{P}_{#1}\left[ #2\right]}
\newcommand{\expe}[1]{\PE \left[ #1 \right]}
\newcommand{\expeLigne}[1]{\PE [ #1 ]}
\newcommand{\expeLine}[1]{\PE [ #1 ]}
\newcommand{\expeMarkov}[2]{\PE_{#1} \left[ #2 \right]}
\newcommand{\expeMarkovD}[3]{\PE_{#1}^{#3} \left[ #2 \right]}
\newcommand{\expeMarkovDD}[1]{\PE_{#1}}
\newcommand{\expeMarkovLigne}[2]{\PE_{#1} [ #2 ]}
\newcommand{\expeMarkovExpo}[3]{\PE_{#1}^{#2} \left[ #3 \right]}
\newcommand{\probaMarkovTildeDeux}[2]{\widetilde{\mathbb{P}}_{#1} \left[ #2 \right]}
\newcommand{\expeMarkovTilde}[2]{\widetilde{\PE}_{#1} \left[ #2 \right]}
\newcommand{\Cros}[1]{C_{\mathrm{Ros},#1}}
\newcommand{\Dros}[1]{D_{\mathrm{Ros},#1}}
\def\hg{\gPois}
\def\mcf{\mathcal{F}}
\newcommand{\crochet}[1]{\left\langle#1 \right\rangle}
\newcommand{\parenthese}[1]{\left(#1 \right)}
\newcommand{\parentheseLigne}[1]{(#1 )}
\newcommand{\parentheseDeux}[1]{\left[ #1 \right]}
\newcommand{\parentheseDeuxLigne}[1]{[ #1 ]}
\newcommand{\defEns}[1]{\left\lbrace #1 \right\rbrace }
\newcommand{\defEnsLigne}[1]{\lbrace #1 \rbrace }
\newcommand{\defEnsPoint}[1]{\left\lbrace #1 \right. }
\newcommand{\defEnsPointDeux}[1]{\left. #1 \right \rbrace }
\newcommand{\defEnsL}[1]{\left\lbrace #1 \right. }
\newcommand{\defEnsR}[1]{\left. #1 \right \rbrace }
\def\MWp{M_{W,p}}
\def\MWdelta{M_{W,(2-\delta)p}}
\newcommand{\indi}[1]{\1_{#1}}
\def\cupgamma{c_{\upgamma}}
\def\Rupgamma{R_{\upgamma}}
\def\bupgamma{\bb_{\upgamma}}
\def\tRupgamma{\tilde{R}_{\upgamma}}
\newcommand\cupgammaD[1]{c_{#1}}
\newcommand\RupgammaD[1]{R_{#1}}
\newcommand\tRupgammaD[1]{\tilde{R}_{#1}}
\newcommand\bupgammaD[1]{\bb_{#1}}
\def\tupgamma{\tilde{\upgamma}}
\def\tuptau{\tilde{\uptau}}
\def\cuptau{c_{\uptau}}
\def\Ruptau{R_{\uptau}}
\def\buptau{\bb_{\uptau}}
\def\tRuptau{\tilde{R}_{\uptau}}
\def\ctuptau{c_{\tuptau}}
\def\Rtuptau{R_{\tuptau}}
\def\btuptau{\bb_{\tuptau}}
\def\tRtuptau{\tilde{R}_{\tuptau}}
\def\msx{\mathsf{X}}
\def\rmY{\mathrm{Y}}
\def\bvarepsilon{\bar{\varepsilon}}
\newcommand{\ceil}[1]{\left\lceil #1 \right\rceil}
\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor}
\def\half{\nicefrac{1}{2}}
\def\tS{\tildFe{S}}
\def\bA{\bar{\mathbf{A}}}
\def\tA{\tilde{A}}
\def\ta{\tilde{a}}
\def\X{{\bf X}}
\def\Y{{\bf Y}}
\def\balpha{\bar{\alpha}}
\def\bdelta{\bar{\delta}}
\def\thetas{\thetalim}
\def\sphere{\mathbb{S}}
\def\realpart{\mathrm{Re}}
\def\transpose{\top}
\def\tQ{\tilde{Q}}
\def\Am{{\bf A}}
\def\bm{{\bf b}}
\def\funcctilde{\tilde{c}_u}
\def\Remainder{\Delta}
\def\barb{\bar{\mathbf{b}}}
\newcommandx{\driftb}[1][1=p]{\bar{b}_{#1}}
\def\KLs{\mathrm{KL}}
\def\thetalima{\theta_{\infty}^{(\alpha)}}
\def\tthetalima{\tilde{\theta}_{\infty}^{(\alpha)}}
\newcommand{\thetaliman}[1]{\theta_{\minfty:#1}^{(\alpha)}}
\def\Jlima{J_{\infty}^{(\alpha)}}
\def\Hlima{H_{\infty}^{(\alpha)}}
\def\Pens{\mathcal{P}}
\def\mrl{\mathrm{L}}
\def\rml{\mathrm{L}}
\def\Cov{\mathrm{Cov}}
\def\transpose{\top}
\def\mcb{\mathcal{B}}
\def\minfty{-\infty}
\newcommand{\convproba}[1]{\overset{\PP}{\underset{#1}{\longrightarrow}}}
\def\barA{\bar{A}}
\newcommand{\flecheLimiteProba}{\overset{\PP}{\underset{n\to+\infty}{\longrightarrow}}}
\def\Zbf{\mathbf{Z}}
\def\Abf{\mathbf{A}}
\def\Bbf{\mathbf{B}}
\def\bfZ{\Zbf}
\def\bfA{\Abf}
\def\bfB{\Bbf}
\def\eps{\varepsilon}
\def\talpha{\tilde{\alpha}}
\def\tvarphi{\tilde{\varphi}}
\def\red{\color{red}}
\newcommandx{\boldb}[1][1={q}]{\mathsf{b}_{#1}}
\newcommandx{\ConstGW}[1][1={n,\lyapW}]{\operatorname{G}_{#1}}
\newcommandx{\ConstMW}[1][1={n,\lyapW}]{\operatorname{M}_{#1}}
\newtheorem{assumTD}{\textbf{TD}\hspace{-1pt}}
\Crefname{assumTD}{\textbf{TD}\hspace{-1pt}}{\textbf{TD}\hspace{-1pt}}
\crefname{assumTD}{\textbf{TD}}{\textbf{TD}}
\newtheorem{assumptionC}{\textbf{C}\hspace{-1pt}}
\Crefname{assumptionC}{\textbf{C}\hspace{-1pt}}{\textbf{C}\hspace{-1pt}}
\crefname{assumptionC}{\textbf{C}}{\textbf{C}}
\newtheorem{assumptionM}{\textbf{UGE}\hspace{-1pt}}
\Crefname{assumptionM}{\textbf{UGE}\hspace{-1pt}}{\textbf{UGE}\hspace{-1pt}}
\crefname{assumptionM}{\textbf{UGE}}{\textbf{UGE}}
\def\pcost{p_{\cost}}
\def\cost{\mathsf{c}}
\def\costmarkov{\mathsf{c}}
\def\distance{\mathsf{d}}
\def\bconst{\mathfrak{b}}
\def\bconstM{\mathfrak{b}^{(\Markov)x}}
\newcommand{\ConstB}{\operatorname{B}}
\def\powerq{6}
\def\Constwasspoly{\mathsf{C}_1}
\newcommandx{\vartconstwas}[1][1=V]{c_{#1}}
\def\rootwas{\zeta}
\newcommandx{\deltawas}[1][1=*]{\delta_{#1}}
\def\scrE{\mathcal{E}}
\newcommand{\ConstDd}{\operatorname{D}}
\newcommandx{\wasser}[4][1=\distance,4=]{\mathbf{W}_{#1}^{#4}\left(#2,#3\right)}
\newcommandx{\covcoeff}[2]{\rho_{#1}^{(#2)}}
\def\smallAconst{\smallConst{\funcAw}}
\def\smallAconstM{\smallConstm{\funcAw}}
\def\QQ{\mathbb{Q}}
\def\shift{\operatorname{S}}
\def\mcg{\mathcal{G}}
\newcommand{\restric}[2]{\left(#1\right)_{#2}}
\newcommand{\dobrush}{\mathsf{\Delta}}
\newcommandx{\dobru}[3][1=,3=]{\dobrush_{#1}^{#3}( #2)} %%% dobrushin coefficient
\def\Ros{\operatorname{Ros}}
\def\thetadelta{\theta^{(\delta)}}
\def\bdelta{\bar{\delta}}
\newcommand{\ball}[2]{\mathrm{B}(#1,#2)}
\def\invariantQ{\pi}
\def\invariantMK{\pi_{\MK[]}}
\def\tildeb{\tilde{b}}
\def\eqdef{:=}
\def\mtt{\mathtt{m}}
\def\teta{\tilde{\eta}}
\def\HP{\mathrm{HP}}
\def\Etr{E^{{\sf tr}}}
\def\Efl{E^{{\sf fl}}}
\def\Gammabf{\boldsymbol{\Gamma}}
\def\checkA{\check{\mathbf{A}}}
\def\Ws{W^{\star}}
\def\Zs{Z^{\star}}
\def\tZs{\tilde{Z}^{\star}}
\def\tZ{\tilde{Z}}
\def\tmszn{\tilde{\msz}_{\nset}}
\def\tmczn{\tilde{\mcz}_{\nset}}
\def\qexponent{q}
\def\ppexponent{p}
\def\Markov{\mathrm{M}}
\def\bigO{\mathcal{O}}
\def\btheta{\bar{\theta}}
\def\rhotw{\rho}
\newcommand{\PEcoupling}[2]{\tilde{\PE}_{#1,#2}}
\newcommand{\PPcoupling}[2]{\tilde{\PP}_{#1,#2}}
\newcommand{\tcb}[1]{\textcolor{blue}{#1}}
\def\gauss{\mathcal{N}}
\newcommandx{\dlim}[1]{\ensuremath{\stackrel{#1}{\Longrightarrow}}}
\def\boot{\mathsf{b}}
\newcommand{\PPb}{\mathbb{P}^\boot}
\newcommand{\PEb}{\mathbb{E}^\boot}
\def\kolmogorov{\mathsf{d}_{\Conv}}
\newcommand{\rapprox}{\mathbin{\rotatebox[origin=c]{90}{$\approx$}}}
\def\bLambda{\bar{\Lambda}}
\def\Cond{\mathrm{Cond}}
Stochastic Gradient Descent (SGD) is a widely used first-order optimization method that is well suited for large data sets and online learning. The algorithm has attracted significant attention; see e.g. \cite{polyak1992acceleration,nemirovski2009robust,moulines2011non}. SGD aims to solve the optimization problem:
\begin{equation}
\label{eq:stoch_minimization}
f(\theta) \to \min_{\theta \in \rset^{d}}\eqsp, \qquad \nabla f(\theta) = \PE_{\xi \sim \PP_{\xi}}[F(\theta,\xi)] \eqsp,
\end{equation}
where $\xi$ is a random variable defined on a measurable space $(\Zset,\Zsigma)$. Instead of the exact gradient $\nabla f(\theta)$, the algorithm accesses only unbiased stochastic estimates $F(\theta,\xi)$.
\par
Throughout this work, we focus on the case of strongly convex objective functions and denote by $\thetas$ the unique minimizer of \eqref{eq:stoch_minimization}. The iterates $\theta_k$, $k \in \nset$, generated by SGD follow the recursive update:
\begin{equation}
\label{eq:sgd_recursion_main}
\theta_{k+1} = \theta_{k} - \alpha_{k+1} F(\theta_k,\xi_{k+1})\eqsp, \quad \theta_0 \in \rset^{d}\eqsp,
\end{equation}
where $\{\alpha_k\}_{k \in \nset}$ is a sequence of step sizes (or learning rates), which may be either diminishing or constant, and $\{\xi_k\}_{k \in \nset}$ is an \iid\, sequence sampled from $\PP_{\xi}$. Theoretical properties of SGD, particularly in the convex and strongly convex settings, have been extensively studied; see, e.g., \cite{nesterov:2004, moulines2011non, bubeck2015convex, lan_2020}. Many optimization algorithms build upon the recurrence \eqref{eq:sgd_recursion_main} to accelerate the convergence of the sequence $\theta_k$ to $\thetas$. Notable examples include momentum acceleration \cite{qian1999momentum}, variance reduction techniques \cite{defazio2014saga, schmidt2017minimizing}, and averaging methods. In this work, we focus on Polyak-Ruppert averaging, originally proposed in \cite{ruppert1988efficient} and \cite{polyak1992acceleration}, which improves convergence by averaging the SGD iterates \eqref{eq:sgd_recursion_main}. Specifically, the estimator is defined as
\begin{equation}
\label{eq:PR_estimate}
\bar \theta_n = \frac{1}{n}\sum_{i=0}^{n-1}\theta_i\eqsp, \quad n \in \nset\eqsp.
\end{equation}
It has been established (see \cite[Theorem 3]{polyak1992acceleration}) that under appropriate conditions on the objective function $f$, the noisy gradient estimates $F$, and the step sizes $\alpha_k$, the sequence of averaged iterates $\{\bar{\theta}_{n}\}_{n \in \nset}$ is asymptotically normal:
\begin{equation}
\label{eq:CLT_fort}
\textstyle
\sqrt{n}(\bar{\theta}_{n} - \thetas) \dto \gauss(0,\Sigma_\infty)\eqsp,
\end{equation}
where $\dto$ denotes convergence in distribution, and $\gauss(0,\Sigma_\infty)$ is a zero-mean Gaussian distribution with covariance matrix $\Sigma_\infty$, defined later in \Cref{sec:gaus_approximation}. This result raises two key questions:
\begin{enumerate}[(i),noitemsep,nolistsep]
\item what is the rate of convergence in \eqref{eq:CLT_fort}?
\item how can \eqref{eq:CLT_fort} be leveraged to construct confidence sets for $\thetas$, given that $\Sigma_\infty$ is unknown in practice?
\end{enumerate}
In our paper we aim to answer both questions. To quantify convergence rates in \eqref{eq:CLT_fort}, we employ convex distance, which is defined for random vectors $X, Y \in \rset^d$ as
\begin{equation}
\label{eq:convex-distance-definiton}
\textstyle
\kolmogorov(X, Y) = \sup_{B \in \Conv(\rset^{d})}\left|\P\bigl(X \in B\bigr) - \P(Y \in B)\right|\eqsp,
\end{equation}
where $\Conv(\rset^{d})$ denotes the collection of convex subsets of $\rset^{d}$. The authors of \cite{shao2022berry} derive Berry-Esseen-type bounds for $\kolmogorov(\sqrt{n}\Sigma_{n}^{-1/2}(\bar{\theta}_{n} - \thetas), \gauss(0,\Id_d))$, where
$\Sigma_n$ is the covariance matrix of the linearized counterpart of \eqref{eq:sgd_recursion_main}, see precise definitions below in \eqref{eq:sigma_n_def} . We complement this result with the rates of convergence in \eqref{eq:CLT_fort}. Interestingly, we also provide the lower bound on the convex distance $\kolmogorov(\sqrt{n}(\bar{\theta}_{n} - \thetas),\gauss(0,\Sigma_{\infty}))$, which indicates, that for some choice of step sizes $\alpha_k$ in \eqref{eq:sgd_recursion_main}, the normal approximation by $\gauss(0,\Sigma_{\infty})$ is less accurate, compared to normal approximation with other covariance matrix, in particular, with $\Sigma_n$. This effect has been previously observed in the bootstrap method for \iid\ observations without the context of gradients methods, see \cite[Theorem 3.11]{shao1995jackknife}.
One of the popular approaches for solving (ii) is based on the \emph{plug-in} methods \cite{chen2020aos,chen2021statistical}, which aim to construct an estimator $\hat{\Sigma}_n$ of $\Sigma_{\infty}$ directly. Theoretical guarantees for these methods typically focus on non-asymptotic bounds on how close $\hat{\Sigma}_n$ is to $\Sigma_{\infty}$, often in terms of $\PE[\norm{\hat{\Sigma}_n - \Sigma_{\infty}}]$. At the same time, the analysis of these methods bypasses item (i) and the issues related to the rate of convergence in \eqref{eq:CLT_fort}. In our paper we suggest, to the best of our knowledge, the first fully non-asymptotic analysis of procedure for constructing the confidence intervals, based on the bootstrap approach \cite{efron1992bootstrap,JMLR:v19:17-370}, which avoids the direct approximation of $\Sigma_{\infty}$. Moreover, theoretical analysis of the underlying procedure together with the results on normal approximation with $\mathcal{N}(0,\Sigma_{\infty})$ from (i) shows that the same approximation rate can not be achieved by the plug-in methods, at least for a certain range of step sizes $\alpha_k$ in \eqref{eq:sgd_recursion_main}. Our key contributions are as follows:
\begin{itemize}[noitemsep, nolistsep]
\item We establish the non-asymptotic validity of the multiplier bootstrap procedure introduced in \cite{JMLR:v19:17-370}. Under appropriate regularity conditions, our bounds imply that the distribution of $\sqrt{n}(\bar{\theta}_{n} - \thetas)$ can be approximated, up to logarithmic factors, at a rate of $1/n^{\gamma-1/2}$ for step sizes of the form $\alpha_k = c_0/(k+k_0)^{\gamma}, \gamma \in (1/2,1)$. To our knowledge, this provides the first fully non-asymptotic bound on the accuracy of bootstrap approximations in SGD algorithms. Notably, this rate can be faster than the one that we can prove in \eqref{eq:CLT_fort}. Our rates improve upon recent works \cite{samsonov2024gaussian, wu2024statistical}, which addressed the convergence rate in similar procedures for the LSA algorithm and TD learning, respectively.
\item Our analysis of the multiplier bootstrap procedure reveals an interesting property: unlike plug-in estimators, the validity of the bootstrap method does not directly depend on approximating $\sqrt{n}(\bar{\theta}_{n} - \thetas)$ by $\gauss(0,\Sigma_\infty)$. Instead, it requires approximating $\gauss(0,\Sigma_n)$ for some matrix $\Sigma_n$. The structure of $\Sigma_n$ and its associated convergence rates play a central role in our present analysis, both for the rate in \eqref{eq:CLT_fort} and for non-asymptotic bootstrap validity. Precise definitions are provided in \Cref{sec:gaus_approximation}.
\item We analyze the Polyak-Ruppert averaged SGD iterates \eqref{eq:PR_estimate} for strongly convex minimization problems and establish Gaussian approximation rates in \eqref{eq:CLT_fort} in terms of the convex distance. Specifically, we show that the approximation rate $\kolmogorov(\sqrt{n}(\bar{\theta}_{n} - \thetas), \gauss(0,\Sigma_\infty))$ is of order $n^{-1/4}$ when using the step size $\alpha_k = c_0/(k+k_0)^{3/4}$ with a suitably chosen $c_0$ and $k_0$. Our result is based on the techniques suggested in \cite{shao2022berry} and \cite{wu2024statistical}. We also provide the lower bound, which indicate that our rate of normal approximation with $\mathcal{N}(0,\Sigma_{\infty})$ is tight in the regime $\alpha_k = c_0/(k+k_0)^{\gamma}$ with $\gamma \geq 3/4$.
\end{itemize}
\vspace{-3mm}
\paragraph{Notations.} Throughout this paper, we use the following notations. For a matrix $A \in \rset^{d \times d}$ and a vector $x \in \rset^{d}$, we denote by $\norm{A}$ and $\norm{x}$ their spectral norm and Euclidean norm, respectively. We also write $\frobnorm{A}$ for the Frobenius norm of matrix $A$. Given a function $f: \rset^{d} \to \rset$, we write $\nabla f(\theta)$ and $\nabla^2 f(\theta)$ for its gradient and Hessian at a point $\theta$. Additionally, we use the standard abbreviations "i.i.d." for "independent and identically distributed" and "w.r.t." for "with respect to".
\paragraph{Literature review.} Asymptotic properties of the SGD algorithm, including the asymptotic normality of the estimator $\bar{\theta}_n$ and its almost sure convergence, have been extensively studied for smooth and strongly convex minimization problems \cite{polyak1992acceleration, kushner2003stochastic, benveniste2012adaptive}. Optimal mean-squared error (MSE) bounds for $\theta_n - \thetas$ and $\bar{\theta}_n - \thetas$ were established in \cite{nemirovski2009robust} for smooth and strongly convex objectives, and later refined in \cite{moulines2011non}. The case of constant-step size SGD for strongly convex problems has been analyzed in depth in \cite{durmus2020biassgd}. High-probability bounds for SGD iterates were obtained in \cite{rakhlin2012making} and later extended in \cite{harvey2019tight}. Both works address non-smooth and strongly convex minimization problems.
\par
It is important to note that the results discussed above do not directly imply convergence rates for $\sqrt{n}(\bar{\theta}_{n} - \thetas)$ to $\gauss(0,\Sigma_\infty)$ in terms of $\kolmogorov(\cdot, \cdot)$ or the Wasserstein distance. Among the relevant contributions in this direction, we highlight recent works \cite{srikant2024rates, samsonov2024gaussian, wu2024statistical}, which provide quantitative bounds on the convergence rate in \eqref{eq:CLT_fort} for iterates of the temporal difference learning algorithm and general linear stochastic approximation (LSA) schemes. However, these algorithms do not necessarily correspond to SGD with a quadratic objective $f$, as the system matrix in LSA is not necessarily symmetric. Convergence rates of order up to $1/\sqrt{n}$ were established in \cite{pmlr-v99-anastasiou19a} for a class of smooth
test functions. Recent paper \cite{agrawalla2023high} provide Berry-Essen bounds for last iterate of SGD for high-dimensional linear regression of order up to $n^{-1/4}$.
\par
Bootstrap methods for i.i.d. observations were first introduced in \cite{efron1992bootstrap}. In the context of SGD methods, \cite{JMLR:v19:17-370} proposed the multiplier bootstrap approach for constructing confidence intervals for $\thetas$ and established its asymptotic validity. The same algorithm was analyzed in \cite{samsonov2024gaussian} for the LSA algorithm, obtaining rate $n^{-1/4}$ for the accuracy of approximating the distribution of $\sqrt{n}(\bar{\theta}_n - \thetas)$ with the distribution of its bootstrap-world counterpart in terms of convex distance.
\par
Popular group of methods for constructing confidence sets for $\thetas$ is based on estimating the asymptotic covariance matrix $\Sigma_{\infty}$. Plug-in and batch-mean estimators for $\Sigma_{\infty}$ attracted lot of attention, see \cite{chen2020aos,chen2021statistical,chen2022online}, especially in the setting when the stochastic estimates of Hessian are available. The latter two papers focused on learning with contextual bandits. Estimates for $\Sigma_{\infty}$ based on batch-mean method and its online modification were considered in \cite{chen2020aos} and \cite{zhu2023online_cov_matr}. The authors in \cite{pmlr-v178-li22b} considered the asymptotic validity of the plug-in estimator for $\Sigma_{\infty}$ in the local SGD setting. \cite{zhong2023online} refined the validity guarantees for both the multiplier bootstrap and batch-mean estimates of $\Sigma_{\infty}$ for nonconvex problems. However, these papers typically provide recovery rates $\Sigma_{\infty}$, but only show asymptotic validity of the proposed confidence intervals. A notable exception is the recent paper \cite{wu2024statistical}, where the temporal difference (TD) learning algorithm was studied. The authors of \cite{wu2024statistical} provided purely non-asymptotic analysis of their procedure, obtaining the approximation rate $n^{-1/3}$ for the distribution of $\sqrt{n}(\bar{\theta}_n - \thetas)$ in terms of the convex distance.
In our paper, we performed the fully non-asymptotic analysis of the multiplier bootstrap procedure for SGD applied to strongly convex minimization problems. We showed that the algorithm can achieve approximation rates in convex distances of order up to $1/\sqrt{n}$. We highlight the fact that the validity of the multiplier bootstrap procedure does not require one to consider Berry-Esseen bounds with the asymptotic covariance matrix $\Sigma_{\infty}$, which is in sharp contrast to the methods that require direct estimation of $\Sigma_{\infty}$.
\setcounter{assumprime}{5}
\begin{assumprime}
\label{ass:n_lower_bound_full}
Number of observations $n$ satisfies $n \geq \rme^3$ and $\frac{n}{\log(2dn)}\geq \max(1, \frac{(20 C_{Q, \xi}C_{\Sigma}^2)^2}{9})$, where the constants $C_{Q, \xi}$ and $C_\Sigma$ are defined in \eqref{eq:const_C_Q_xi_def} and \eqref{eq:def_C_Sigma}, respectively.
\end{assumprime}
\subsection{Proof of Theorem \ref{th:bootstrap_validity}}
\label{app:proof_th_main_1}
Recall that our proof of non-asymptotic bootstrap validity is based on the Gaussian approximation performed in both the “real” world and the bootstrap world, combined with the Gaussian comparison inequality, that is:
\begin{tikzcd}[column sep = 110pt]
\label{eq:diagram_appendix}
\text{Real world: \quad\quad} \sqrt n (\bar{\theta}_{n} - \thetas) \arrow[<->]{r}{\text{Gaussian approximation, Th. \ref{th:bound_kolmogorov_dist_pr_sigma_n}}} & \mathcal N(0, \Sigma_n) \arrow[<->]{d}{\text{Gaussian comparison} }
\\
\text{Bootstrap world: } \sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \arrow[<->]{r}{\text{Gaussian approximation, Th. \ref{GAR bootstrap_main}}} & \mathcal N(0, \Sigma_n^\boot )\eqsp, \\
\end{tikzcd}
\\
where $\Sigma_n$ and $\Sigma_n^\boot$ are defined in \ref{eq:sigma_n_def} and \ref{Sigman boot def} respectively.
Recall that $Y \sim \mathcal{N}(0, I)$ under $\PP$ and $Y^b \sim \mathcal{N}(0, I)$ under $\PPb$ and $\PPb$ is defined in \eqref{def: PPb}.
To formalize the scheme described above, we consider the following decomposition
\begin{equation}
\sup_{B \in \Conv(\rset^{d})} |\PPb(\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B ) - \PP(\sqrt n (\bar{\theta}_n - \thetas) \in B)| \leq T_1 + T_2 + T_3 \eqsp,
\end{equation}
where
\begin{align}
&T_1 = \sup_{B \in \Conv(\rset^{d})} |\PP(\sqrt n (\bar{\theta}_n - \thetas)\in B)- \PP(\Sigma_n^{1/2}Y\in B)|\eqsp,\\
&T_2 = \sup_{B \in \Conv(\rset^d)}|\PPb( \sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B) - \PPb(\{\Sigma_n^\boot\}^{1/2}Y^\boot \in B)|\eqsp,\\
& T_3 = \sup_{B \in \Conv(\rset^{d})} |\PPb(\{\Sigma_n^\boot\}^{1/2}Y^\boot \in B)-\PP(\Sigma_n^{1/2}Y\in B)|\eqsp.
\end{align}
Hence, from \Cref{th:bound_kolmogorov_dist_pr_sigma_n} (see the proof in \ref{sec:Gar_real_world_proof})
it follows
\begin{equation}
T_1 \leq \frac{\ConstC_4}{\sqrt{n}} + \frac{\ConstC_5}{n^{\gamma - 1/2}}+ \frac{\ConstC_6}{n^{\gamma/2}}\eqsp,
\end{equation}
where $\ConstC_4, \ConstC_5, \ConstC_6$ are given in equation \eqref{eq:def_Const_M_3_i}.
Recall that
\begin{equation}
\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) = W^\boot + D^\boot\eqsp,
\end{equation}
where $W^\boot$ is a linear statistic of $\Xi^{n-1}$ with $\PEb[W^\boot\{W^\boot\}^{\top}] = \Sigma_n^b$ and $D^\boot$ is a non-linear statistic of $\Xi^{n-1}$. To control the terms $T_2$ and $T_3$, we require high-probability bounds. To this end, we introduce the following sets:
\begin{equation}
\label{def:Omega_0}
\Omega_0 =\{\{\PEb[\norm{D^\boot}^{p}]\}^{1/p} \leq M_{1,1}^{\boot}\rme^{1/p}p^{3/2}n^{2/p-\gamma/2} + M_{2,1}^{\boot}\rme^{2/p}p n^{1/2+3/p-\gamma}\}\eqsp,
\end{equation}
\begin{equation}
\label{def:Omega_1}
\Omega_1 = \{ \norm{\Sigma_n^\boot - \Sigma_n} \leq \frac{10C_{Q,\xi}\sqrt{\log(2d n)}}{3\sqrt{n}}\}\eqsp,
\end{equation}
where $M_{1,1}^{\boot},M_{2,1}^{\boot}$ and $C_{Q,\xi}$ are defined in \eqref{def:M_11_boot}, \eqref{def:M_21_boot}, \eqref{eq:const_C_Q_xi_def} respectively.
The first set ensures that $\Sigma_n^\boot$ concentrates around $\Sigma_n$, while the second set guarantees that the remainder term $D^\boot$ is small.
Therefore applying \Cref{GAR bootstrap_main} (see the proof in \ref{subsec:GAR in the bootstrap world}) and \Cref{lambda min boot} we get that on the set $\Omega_0\cap \Omega_1$ with $\PP(\Omega_0\cap \Omega_1)\geq 1-2/n$, it holds
\begin{equation}
\sup_{B \in \Conv(\rset^d)}|\PPb( \sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B) - \PPb(\{\Sigma_n^\boot\}^{1/2}Y^\boot \in B)| \leq \frac{M_{3,1}^\boot}{\sqrt{n}} + \frac{M_{3,2}^\boot \log n}{n^{\gamma-1/2}} + \frac{M_{3,3}^\boot \log^{3/2} n}{n^{\gamma/2}} \eqsp,
\end{equation}
where $\{M_{3,i}^\boot\}_{i=1}^3$ are defined in equation \eqref{M_3i boot def}.
To complete the proof, we need to bound the convex distance between the two Gaussians.
By \Cref{lem:bound_Q_i_and_Sigma_n}, we have $\|\Sigma_n^{-1/2}\| \le C_{\Sigma}$. Hence, due to Lemma \ref{lem:matrix_bernstein}, on the set $\Omega_1$ with $\PP(\Omega_1)\geq 1-1/n$ we have
\begin{equation}
\trace \{ (\Sigma_n^{-1/2} \Sigma_n^\boot \Sigma_n^{-1/2} - I_p)^2 \}\leq d \norm{(\Sigma_n^{-1/2} \Sigma_n^\boot \Sigma_n^{-1/2} - I_p)^2}^2 \leq dC_\Sigma^2\norm{\Sigma_n^\boot-\Sigma_n}^2 \leq \delta^2 \eqsp.
\end{equation}
where we have set
$$
\delta = \frac{10C_{Q,\xi} C_\Sigma^2 \sqrt{d \log(2d n)}}{3\sqrt{n}}
$$
Applying Lemma \ref{Pinsker} with probability at least $1-1/n$ we get
$$
T_3 \le \frac{5C_{Q,\xi} C_\Sigma^2 \sqrt{d \log(2d n)}}{\sqrt{n}} \eqsp.
$$
Collecting previous bounds and using triangle inequality we get that on the set $\Omega_0\cap \Omega_1$ with $\PP(\Omega_0\cap \Omega_1)\geq 1-2/n$, it holds:
\begin{multline}
\sup_{B \in \Conv(\rset^{d})} |\PPb(\sqrt n (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B ) - \PP(\sqrt n (\bar{\theta}_n - \thetas) \in B)| \le \frac{\ConstC_1 \sqrt{\log n}}{n^{1/2}} + \frac{\ConstC_2 \log n}{n^{\gamma - 1/2} } + \frac{\ConstC_3 \log^{3/2} n}{n^{\gamma/2}} \eqsp,
\end{multline}
where
\begin{align}
\label{eq: bootsrap constants def}
\ConstC_{1} = \ConstC_4 + M_{3,1}^\boot + 5C_{Q,\xi} C_\Sigma^2 \sqrt{d \log(2d)}, \quad \ConstC_{2} = \ConstC_5 + M_{3,2}^\boot, \quad \ConstC_{3} = \ConstC_{6} + M_{3,3}^\boot.
\end{align}
\subsection{Matrix Bernstein inequality for $
\Sigma_n^\boot$ and Gaussian comparison}
\begin{lemma}
\label{lem:matrix_bernstein}
Under assumptions \Cref{ass:L-smooth}, \Cref{ass:bound_noise}, \Cref{ass:step_size_new_boot}, \Cref{ass:n_lower_bound}, there is a set $\Omega_1 \in \F_{n-1}$, such that $\PP(\Omega_1) \geq 1 - 1/n$ and on $\Omega_1$ it holds that
\begin{equation}
\norm{\Sigma_n^\boot - \Sigma_n} \leq \frac{10C_{Q,\xi}\sqrt{\log(2d n)}}{3\sqrt{n}}
% \frac{4C_{Q,\xi}\log(2d n)}{3n} + \frac{2C_{Q,\xi}\sqrt{\log(2d n)}}{\sqrt{n}} \eqsp,
\end{equation}
where the constant $C_{Q,\xi}$ is given by
\begin{equation}
\label{eq:const_C_Q_xi_def}
C_{Q,\xi} := C_{Q}^2(C_{1,\xi}^2 + \lambda_{\max}(\Sigma_{\xi}))\eqsp,
\end{equation}
and $C_{1,\xi}$, $C_{Q}$ are defined in \Cref{ass:bound_noise} and Lemma \ref{lem:bound_Q_i_and_Sigma_n}, respectively.
\end{lemma}
\begin{proof}
Note that
\[
\Sigma_n^\boot - \Sigma_n = \frac{1}{n} \sum_{i=1}^{n-1} Q_i(\eta(\xi_i)\eta(\xi_i)^\top-\Sigma_{\xi})Q_i^\top\eqsp.
\]
For simplicity we denote $A_i = Q_i(\eta(\xi_i)\eta(\xi_i)^\top-\Sigma_{\xi})Q_i^\top$. Note that for any $i \in \{1, \ldots n-1\}$ it holds that
\begin{equation}
\PE[A_i] = 0 \eqsp, \quad \norm{A_i} \leq C_{Q,\xi}\eqsp, \quad \norm{\sum_{i=1}^{n-1}\PE[A_iA_i^\top]}\leq nC_{Q,\xi}^2\eqsp.
\end{equation}
Then, using matrix Bernstein inequality \cite[Chapter 6]{tropp2015introduction}, we obtain
\begin{equation}
\PP\left(\frac{1}{n}\norm{\sum_{i=1}^{n-1}A_i}\geq t\right)\leq 2d\exp\biggl\{\frac{-t^2n^2/2}{nC_{Q,\xi}^2 + nC_{Q,\xi}t/3}\biggr\} \eqsp.
\end{equation}
Taking $t_\delta = \frac{4C_{Q,\xi}\log(2d/\delta)}{3n} + \frac{2C_{Q,\xi}\sqrt{\log(2d/\delta)}}{\sqrt{n}}$, we obtain that with probability at least $1-\delta$, it holds
\begin{equation}
\frac{1}{n}\norm{\sum_{i=1}^{n-1}A_i}\leq t_\delta\eqsp.
\end{equation}
Setting $\delta = 1/n$ and applying \Cref{ass:n_lower_bound} completes the proof.
\end{proof}
\begin{corollary}
\label{lambda min boot}
Under assumptions \Cref{ass:L-smooth}, \Cref{ass:bound_noise}, \Cref{ass:step_size_new_boot}, \Cref{ass:n_lower_bound}, on $\Omega_1$ it holds that
\begin{equation}
\lambda_{\min}(\Sigma_n^\boot) \geq \frac{1}{2 C_\Sigma^2} \eqsp.
\end{equation}
\end{corollary}
\begin{proof}
Using eigenvalue stability (Lidski's) inequality, we obtain
\begin{equation}
\lambda_{\min}(\Sigma_n^\boot) \geq \lambda_{\min}(\Sigma_n)-\norm{\Sigma_n-\Sigma_n^\boot}\eqsp.
\end{equation}
Note that on $\Omega_1$, we have
\begin{equation}
\norm{\Sigma_n-\Sigma_n^\boot} \leq \frac{10C_{Q,\xi}\sqrt{\log(2d n)}}{3\sqrt{n}} \leq \frac{1}{2C_\Sigma^2}\eqsp,
\end{equation}
where in the last inequality we use \Cref{ass:n_lower_bound}.
\end{proof}
% \begin{lemma}
% \label{lem: gaussion comparison boot and real} Under assumptions \Cref{ass:L-smooth}, \Cref{ass:bound_noise}, \Cref{ass:step_size_new_boot}, \Cref{ass:n_lower_bound}. Then on $\Omega_1$, it holds that
% $$
% \kolmogorov(\{\Sigma_n^\boot\}^{1/2} \eta^\boot, \Sigma_n^{1/2}\eta) \le \frac{5C_{Q,\xi} C_\Sigma^2 \sqrt{d \log(2d n)}}{\sqrt{n}} \eqsp.
% $$
% \end{lemma}
% \begin{proof}
% By Lemma \ref{lem:bound_Q_i_and_Sigma_n}, $\|\Sigma_n^{-1/2}\| \le C_{\Sigma}$. Hence, due to Lemma \ref{lem:matrix_bernstein}, we have
% \begin{equation}
% \trace \{ (\Sigma_n^{-1/2} \Sigma_n^\boot \Sigma_n^{-1/2} - I_p)^2 \}\leq d \norm{(\Sigma_n^{-1/2} \Sigma_n^\boot \Sigma_n^{-1/2} - I_p)^2}^2 \leq dC_\Sigma^2\norm{\Sigma_n^\boot-\Sigma_n}^2 \leq \delta^2 \eqsp.
% \end{equation}
% where we have set
% $$
% \delta = \frac{10C_{Q,\xi} C_\Sigma^2 \sqrt{d \log(2d n)}}{3\sqrt{n}}
% $$
% We finish the proof applying Lemma \ref{Pinsker}.
% \end{proof}
\subsection{Example of distribution satisfying \Cref{ass:bound_bootstap_weights}}
\label{sec:example_distribution}
To construct examples of distributions satisfying the above assumption, one can use the beta distribution, which is defined on \([0, 1]\), and then shift and scale it. Set \( W = a + bX \) where \( X \sim \text{Beta}(\alpha, \beta) \) and \(a,b>0.\) We have \( \mathbb{E}[X] = \frac{\alpha}{\alpha + \beta}, \) \( \text{Var}(X) = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)} \) and \(a\leq W\leq a+b \) a.s. By solving (for \( a \) and \( b \)) the equations \( \mathbb{E}[W] = a + b\mathbb{E}[X] = 1 \) and \( \text{Var}(W) = b^2\text{Var}(X) = 1, \) we derive \(b=1/\sqrt{\text{Var}(X)}\) and \(a=1-\mathbb{E}[X]/\sqrt{\text{Var}(X)}.\) Note that $a>0$ provided $\alpha+\beta+1<\beta/\alpha$.
We first provide details of the expansion \eqref{eq:linear and nonlinear terms}. Recall that the error of SGD approximation may be rewritten as follows
\begin{equation}
\label{eq: sgd_reccurence_with_remainder}
\theta_k-\thetas = (\Id - \alpha_k G) (\theta_{k-1}-\thetas) - \alpha_k ( H(\theta_{k-1}) + \eta(\xi_k) + g(\theta_{k-1}, \xi_k))\eqsp.
\end{equation}
Iteratively spinning this expression out we get
\begin{equation}
\label{eq: sgd_reccurence_with_remainder_2}
\theta_k-\thetas = \prod_{j=1}^k(\Id - \alpha_j G) (\theta_{0}-\thetas) - \sum_{j=1}^k \alpha_j \prod_{i = j+1}^k (\Id - \alpha_i G) ( H(\theta_{j-1}) + \eta(\xi_j) + g(\theta_{j-1}, \xi_j))\eqsp.
\end{equation}
Taking average of \eqref{eq: sgd_reccurence_with_remainder} and changing the order of summation, we obtain
\begin{equation}
\sqrt{n}(\bar\theta_n -\thetas) = \frac{1} {\sqrt{n}\alpha_0}Q_0(\theta_0-\thetas) -\frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i(H(\theta_{i-1}) + \eta(\xi_i) + g(\theta_{i-1}, \xi_i)),
\end{equation}
where $Q_i$ is defined in \eqref{eq:Q_i_def}. Finally, we obtain
\begin{align}
\label{eq:linear and nonlinear terms appendix}
&\sqrt{n}(\bar\theta_n -\thetas) = W + D,\\ &D = \frac{1}{\sqrt{n}\alpha_0}Q_0(\theta_0-\thetas)-\frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_ig(\theta_{i-1}, \xi_i) - \frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_iH(\theta_{i-1})\eqsp,\\
&W = - \frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i\eta(\xi_i)\eqsp.
\end{align}
Under suitable assumptions on the step size and the Hessian, we can show that the matrices $Q_i$, defined in \eqref{eq:Q_i_def}, are uniformly bounded for all $i$. This fact will be used in the proof of \Cref{th:bound_kolmogorov_dist_pr_sigma_n}. Moreover, we also establish that $\lambda_{\min}(Q_i)$ is bounded away from zero, which in turn implies a lower bound on $\lambda_{\min}(\Sigma_n)$.
\begin{lemma}
\label{lem:bound_Q_i_and_Sigma_n}
Assume \Cref{ass:L-smooth} and \Cref{ass:step_size}. Then for any $i \in \{0, \ldots, n-1\}$ it holds that
\begin{equation}
\lambda_{\max}(Q_i) \leq C_Q\eqsp,
\end{equation}
where the constant $C_Q$ is defined in \eqref{eq:const_C_Q}
Moreover,
\begin{equation}
\lambda_{\min}(Q_i) \geq C_Q^{\min}\eqsp, \text{ and } \norm{\Sigma_n^{-1/2}} \leq C_{\Sigma}\eqsp,
\end{equation}
where the matrix $\Sigma_n$ is defined in \eqref{eq:sigma_n_def}, and $ C_Q^{\min}, C_{\Sigma}$ are defined in \eqref{def:C_Q_min} and \eqref{eq:def_C_Sigma} respectively.
\end{lemma}
The version of this lemma with proof and with explicit constants is given in \ref{subsec:proof_bound_Q_i_Sigma_n}.
\begin{proof}[Proof of Theorem \ref{th:bound_kolmogorov_dist_pr_sigma_n}]
We normalize the both parts of \eqref{eq:linear and nonlinear terms} by $\Sigma_n^{1/2}$ and obtain
\begin{equation}
\label{eq: norm_PR_decomposition}
\sqrt{n}\Sigma_n^{-\frac{1}{2}}(\bar\theta_n -\thetas) = \sum_{i=1}^{n-1}\underbrace{\frac{\Sigma_n^{-\frac{1}{2}}}{\sqrt{n}}Q_i\eta(\xi_i)}_{w_i} + D_{n,1} + D_{n,2} + D_{n,3}\eqsp,
\end{equation}
where we have set
\begin{equation}
\label{eq:D_n_1_3_def}
\begin{split}
D_{n,1} &= \frac{\Sigma_n^{-\frac{1}{2}}}{\sqrt{n}\alpha_0}Q_0(\theta_0-\thetas)\eqsp, \\
D_{n,2} &= -\frac{\Sigma_n^{-\frac{1}{2}}}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i H(\theta_{i-1})\eqsp, \\
D_{n,3} &= -\frac{\Sigma_n^{-\frac{1}{2}}}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i g(\theta_{i-1}, \xi_i)) \eqsp.
\end{split}
\end{equation}
Also, for any $1 \leq i \leq n-1$ we construct
\begin{align}
&D_{n,1}^{(i)} = \frac{\Sigma_n^{-1/2}}{\sqrt{n}\alpha_0}Q_0(\theta_0^{(i)}-\thetas) \eqsp,\\&
D_{n,2}^{(i)} = -\frac{\Sigma_n^{-1/2}}{\sqrt{n}}\sum_{j=1}^{n-1}Q_jH(\theta_{j-1}^{(i)})\eqsp,\\&
D_{n,3}^{(i)} = -\frac{\Sigma_n^{-1/2}}{\sqrt{n}}\sum_{j=1}^{n-1}Q_jg(\theta_{j-1}^{(i)}, \widetilde{\xi_j}^{(i)})),
\end{align}
where we set
\begin{equation}
\widetilde{\xi_j}^{(i)} =
\begin{cases}
\xi_j \eqsp, &\text{if } j \neq i\\
\xi_j' \eqsp, &\text{if } j = i \eqsp.
\end{cases}
\end{equation}
Define $D_n = D_{n,1} + D_{n,2} + D_{n,3}$, $D_n^{(i)}= D_{n,1}^{(i)} + D_{n,2}^{(i)} + D_{n,3}^{(i)}$, $W_n=\sum_{i=1}^{n-1}w_i$ and $\Upsilon_{n} = \sum_{i=1}^n\PE[\norm{\omega_i}^3]$(we keep the same notations as in the unnormalized setting for simplicity).
Let $Y \sim \mathcal{N}(0, I_d)$. Then, using \cite[Theorem 2.1]{shao2022berry}, we have
\begin{equation}
\kolmogorov (\sqrt{n}\Sigma_n^{-1/2}(\bar\theta_n-\thetas), Y) \leq 259d^{1/2}\Upsilon_n + 2\underbrace{\PE\{\norm{W_n}\norm{D_n}\}}_{R_1} + 2\underbrace{\sum_{i=1}^{n-1}\PE[\norm{\omega_i}\norm{D_n-D_n^{(i)}}]}_{R_2}\eqsp.
\end{equation}
Applying H\"{o}lder's inequality, we get
\begin{align}
&R_1 \leq \PE^{1/2}[\norm{W_n}^2]\PE^{1/2}[\norm{D_n}^2]\eqsp,\\
&R_2 \leq \sum_{i=1}^{n-1}\PE^{1/2}[\norm{\omega_i}^2]\PE^{1/2}[\norm{D_n-D_n^{(i)}}^2]\eqsp.
\end{align}
Note that $\PE^{1/2}[\norm{W_n}^2]=\sqrt{d}$. Applying \Cref{lem:bound_Q_i_and_Sigma_n}, we get $\PE^{1/2}\norm{w_i}^2\leq \frac{1}{\sqrt{n}}C_{\Sigma}C_Q\sigma_2$ and
$$
\Upsilon_n \leq \frac{1}{\sqrt{n}}(C_{\Sigma}C_Q\sigma_4)^3 \eqsp.
$$
To complete the proof, it remains to bound $\PE^{1/2}[\|D_n\|^2]$ and $\sum_{i=1}^{n-1}\PE^{1/2}[\|D_n - D_n^{(i)}\|^2]$.
The first term can be bounded using \Cref{lem:bound_Dn}, while the second is controlled via \Cref{lem:bound_sum_Dn-Dni}.
Combining these results, we obtain the following bound:
\begin{equation}
\kolmogorov(\sqrt{n} \Sigma_{n}^{-1/2}(\bar\theta_n -\thetas), Y) \leq \frac{\sqrt{d}M_{3,1}}{\sqrt{n}} + \frac{M_{3,2}}{\sqrt{n}}(\norm{\theta_0-\thetas}+\norm{\theta_0-\thetas}^2+\sigma_2+ \sigma_4^2) + M_{3,3}n^{1/2-\gamma}+M_{3,4}n^{-\gamma/2}\eqsp,
\end{equation}
where
\begin{equation}
\begin{split}
&M_{3,1} =259(C_{\Sigma}C_Q\sigma_4)^3\eqsp,\\
&M_{3,2} = 2\sqrt{d}M_{1,1} + C_{\Sigma}C_Q\sigma_2M_{2,1}\eqsp,\\
&M_{3,3} = 2\sqrt{d}M_{1,2}\sigma_4^2\eqsp,\\
&M_{3, 4} = (2\sqrt{d}M_{1,3}+M_{2,3}C_{\Sigma}C_Q\sigma_2)\sigma_2 + C_{\Sigma}C_QM_{2,2}\sigma_4^2\sigma_2\eqsp.
\end{split}
\end{equation}
Constants $M_{1,1}, M_{1,2}, M_{1,3}$ are defined in \eqref{eq:def_const_M_1} and $M_{2,1}, M_{2,2}, M_{3,3}$ are defined in \eqref{eq:def_const_M_2}.
We simplify the last inequality and get the statement of the theorem with
\begin{equation}
\label{eq:def_Const_M_3_i}
\begin{split}
&\ConstC_{4} = \sqrt{d} M_{3,1} + M_{3,2} \norm{\theta_0-\thetas}+\norm{\theta_0-\thetas}^2+\sigma_2+ \sigma_4^2 \eqsp,\\
&\ConstC_{5} = M_{3,3}\eqsp,\\
&\ConstC_{6} = M_{3,4}\eqsp.
\end{split}
\end{equation}
\end{proof}
\subsection{Bounds for $D_n$}
For simplicity of notations we define
\begin{equation}
\label{def:const_T_1_T_2}
\begin{split}
&T_1(A) = 1 + \frac{1}{A^{1/(1-\gamma)}(1-\gamma)}\Gamma(\frac{1}{1-\gamma})\eqsp,\\
&T_2(A) = 1 + \max\biggl(\exp\biggl\{\frac{1}{1-\gamma}\biggr\}\frac{1}{A^{1/(1-\gamma)}(1-\gamma)}\Gamma(\frac{1}{1-\gamma}), \frac{1}{A(1-\gamma)^2}\biggr)\eqsp.
\end{split}
\end{equation}
\begin{lemma}
\label{lem:bound_Dn}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($4$) and \Cref{ass:step_size}. Then it holds that
\begin{equation}
\PE^{1/2}[\norm{D_n}^2]\leq \frac{M_{1,1}}{\sqrt{n}}(\norm{\theta_0-\thetas}+\norm{\theta_0-\thetas}^2+\sigma_2+ \sigma_4^2)+M_{1,2}\sigma_4^2n^{1/2-\gamma}+M_{1,3}\sigma_2n^{-\gamma/2},
\end{equation}
where
\begin{equation}
\label{eq:def_const_M_1}
\begin{split} &M_{1,1}=C_{\Sigma}C_Q\biggl(T_1(\frac{\mu c_0}{4})(L_2+L_H)\max(\sqrt{C_{4,1}},\sqrt{C_1}) + k_0^\gamma/c_0\biggr)\\&M_{1,2}=C_{\Sigma}C_QL_H\sqrt{C_{4,2}}c_0\frac{1}{1-\gamma}\\&M_{1,3}=C_{\Sigma}C_QL_2\sqrt{C_{2}}\sqrt{c_0}\sqrt{\frac{1}{1-\gamma}}\eqsp,
\end{split}
\end{equation}
where $C_{4,1}$ and $C_{4,2}$ are defined in \Cref{cor:fourth_moment_bound_last_iterate}, $C_1$ and $C_2$ are defined in \Cref{lem:bound_last_iter_second_moment} and $T_1(\cdot)$ is defined in \cref{def:const_T_1_T_2}.
\end{lemma}
\begin{proof}
Using Minkowski's inequality and the definition of $D_n$, we obtain
\begin{equation}
\PE^{1/2}[\norm{D_n}^2] \leq
\PE^{1/2}[\norm{D_{n,1}}^2] +\PE^{1/2}[\norm{D_{n,2}}^2] + \PE^{1/2}[\norm{D_{n,3}}^2]\eqsp,
\end{equation}
and consider each of the terms $D_{n,1}, D_{n,2}, D_{n,3}$ separately. Applying \Cref{lem:bound_Q_i_and_Sigma_n}, we get
\begin{equation}
\PE^{1/2}[\norm{D_{n,1}}^2] \leq \frac{C_{\Sigma}C_{Q}k_0^\gamma}{\sqrt{n}c_0}\norm{\theta_0-\thetas}\eqsp.
\end{equation}
Now we consider the term $D_{n,2}$. Applying Minkowski's inequality, \Cref{lem:bound_Q_i_and_Sigma_n} and \Cref{lem:H_theta_bound}, we have
\begin{equation}
\PE^{1/2}[\norm{D_{n,2}}^2] \leq \frac{C_{\Sigma}C_{Q}}{\sqrt{n}}\sum_{i=1}^{n-1}\PE^{1/2}[\norm{H(\theta_{i-1})}^2]\leq \frac{C_{\Sigma}C_{Q}L_H}{\sqrt{n}}\sum_{i=1}^{n-1}\PE^{1/2}[\norm{\theta_{i-1}-\thetas}^4]\eqsp.
\end{equation}
For $D_{n,3}$ we note that $\{g(\theta_{i-1},\xi_i)\}_{i=1}^{n-1}$ is a martingale difference with respect to $\F_i$. Hence, using \Cref{lem:bound_Q_i_and_Sigma_n} and \Cref{ass:noise_decomposition}, we get
\begin{equation}
\PE^{1/2}[\norm{D_{n,3}}^2] \leq \frac{C_{\Sigma}C_{Q}}{\sqrt{n}}\left(\sum_{i=1}^{n-1}\PE[\norm{g(\theta_{i-1},\xi_i)}^2]\right)^{1/2}
\leq \frac{C_{\Sigma}C_{Q}L_2}{\sqrt{n}}\left(\PE[\sum_{i=1}^{n-1}\norm{\theta_{i-1}-\thetas}^2]\right)^{1/2}\eqsp.
\end{equation}
Hence, it is enough to upper bound $\PE[\norm{\theta_{i}-\thetas}^{2p}]$ for $p = 1$ and $p = 2$ and $i \in \{0,\ldots,n-2\}$. Using \Cref{lem:bound_last_iter_second_moment} and \Cref{lem:bound_sum_exponent}, we obtain
\begin{align}
&\left(\sum_{i=0}^{n-2}\PE[\norm{\theta_{i}-\thetas}^2]\right)^{1/2} \leq \left(\sum_{i=0}^{n-2} C_1\exp\biggl\{ -\frac{\mu c_0}{4}(i+k_0)^{1-\gamma}\biggr\}[\norm{\theta_0-\thetas}^2 + \sigma_2^2] + C_2\sigma_2^2\alpha_i\right)^{1/2}\\
&\qquad \leq \sqrt{C_1}\sqrt{T_1\biggl(\frac{\mu c_0}{4}\biggr)}[\norm{\theta_0-\thetas} + \sigma_2] + \sqrt{C_2}\sigma_2\sqrt{c_0}\left(\frac{(n-2+k_0)^{1-\gamma} -(k_0-1)^{1-\gamma}}{1-\gamma}\right)^{1/2},
\end{align}
where $T_1(\cdot)$ is defined in \eqref{def:const_T_1_T_2}. Using \Cref{cor:fourth_moment_bound_last_iterate} and \Cref{lem:bound_sum_exponent}, we get
\begin{align}
&\sum_{i=0}^{n-2}\PE^{1/2}[\norm{\theta_{i}-\thetas}^4] \leq \sum_{i=0}^{n-2} \sqrt{C_{4,1}}\exp\biggl\{ -\frac{\mu c_0}{4}i^{1-\gamma}\biggr\}[\norm{\theta_0-\thetas}^2 + \sigma_4^2] + \sqrt{C_{4,2}}\sigma_4^2\alpha_i\\& \leq \sqrt{C_{4,1}}T_1\biggl(\frac{\mu c_0}{4}\biggr)[\norm{\theta_0-\thetas}^2 + \sigma_4^2] + \sqrt{C_{4,2}}\sigma_4^2 c_0\left(\frac{(n-2+k_0)^{1-\gamma}- (k_0-1)^{1-\gamma}}{1-\gamma}\right)\eqsp.
\end{align}
We finish the proof, using simple inequality $(n-2+k_0)^{1-\gamma}- (k_0-1)^{1-\gamma} \leq n^{1-\gamma}$
\end{proof}
\subsection{Bounds for $D_n-D_n^{(i)}$}
\begin{lemma}
\label{lem:bound_sum_Dn-Dni}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($4$) and \Cref{ass:step_size}. Then it holds that
\begin{equation}
\sum_{i=1}^{n-1}\PE^{1/2}[\norm{D_n-D_n^{(i)}}^2] \leq \frac{M_{2,1}}{\sqrt{n}}(\norm{\theta_0-\thetas}+\norm{\theta_0-\thetas}^2+\sigma_2+ \sigma_4^2) + M_{2,2}\sigma_4^2n^{1/2-\gamma}+ M_{2,3}\sigma_2n^{1/2-\gamma/2},
\end{equation}
where
\begin{equation}
\label{eq:def_const_M_2}
\begin{split} &M_{2,1}=C_{\Sigma}C_QT_1(\frac{\mu c_0}{8})T_2(\frac{\mu c_0}{1-\gamma})(L_2+L_H)\max(\sqrt{2(C_1+c_0^2k_0^{-\gamma}R_1R_2)},c^2k_0^{-\gamma}\sqrt{R_{4,1}R_{4,2}})\\&M_{2,2}=C_{\Sigma}C_QL_Hc_0\sqrt{R_{4,1}R_{4,3}}T_2(\frac{\mu c_0}{1-\gamma})\frac{1}{1-\gamma}\\&M_{2,3}=\sqrt{2}C_{\Sigma}C_QL_2\sqrt{C_{2}+R_1R_3c_0T_2(\frac{\mu c_0}{1-\gamma})}\frac{1}{1-\gamma/2} \eqsp.
\end{split}
\end{equation}
Constants $R_1, R_2, R_3$ are defined in \eqref{eq:def_R_1_R_2_R_3} and constants $R_{4,1}, R_{4,2}, R_{4,3}$ are defined \eqref{eq:def_R_41_R_42_R_43}.
\end{lemma}
\begin{proof}
Using Minkowski's inequality and the definition of $D_n$ and $D_n^{(i)}$, we obtain
\begin{equation}
\sum_{i=1}^{n-1}\PE^{1/2}[\norm{D_n-D_n^{(i)}}^2] \leq \sum_{i=1}^{n-1}\PE^{1/2}[\norm{D_{n,2}-D_{n,2}^{(i)}}^2] +\sum_{i=1}^{n-1}\PE^{1/2}[\norm{D_{n,3}-D_{n,3}^{(i)}}^2]
\end{equation}
Define $\F_j^{(i)} = \F_j$ if $j\leq i$ and $\F_j^{(i)} = \sigma(\F_j \vee \sigma(\xi_i'))$ otherwise. Then $\{g(\theta_{j-1},\xi_j)-g(\theta_{j-1}^{(i)},\widetilde{\xi}_j)\}_{j=1}^{n-1}$ is a martingale difference with respect to $\F_j^{(i)}$. Hence, we have, using \Cref{lem:bound_Q_i_and_Sigma_n} and the fact that $\theta_{j-1} = \theta_{j-1}^{(i)}$ for $j \leq i$, we obtain that
\begin{align}
&\PE[\norm{D_{n,3}-D_{n,3}^{(i)}}^2] = \PE\norm{\frac{\Sigma_n^{-1/2}}{\sqrt{n}}\sum_{j=1}^{n-1}Q_j(g(\theta_{j-1},\xi_j)-g(\theta_{j-1}^{(i)},\widetilde{\xi}_j))}^2 \\
&\qquad \leq \frac{C_{\Sigma}^2C_Q^2}{n}\PE[\norm{g(\theta_{i-1},\xi_i)-g(\theta_{i-1},\xi_i')}^2] +\frac{C_{\Sigma}^2C_Q^2}{n}\sum_{j=i+1}^{n-1}\PE[\norm{g(\theta_{j-1},\xi_j)-g(\theta_{j-1}^{(i)},\xi_j)}^2]\eqsp.
\end{align}
Using \Cref{ass:noise_decomposition} and \Cref{lem:bound_Q_i_and_Sigma_n}, we get
\begin{equation}
\PE[\norm{D_{n,3}-D_{n,3}^{(i)}}^2] \leq \frac{2C_{\Sigma}^2C_Q^2L_2^2}{n}\PE[\norm{\theta_{i-1}-\thetas}^2] +\frac{C_{\Sigma}^2C_Q^2L_2^2}{n}\sum_{j=i+1}^{n-1}\PE[\norm{\theta_{j-1}-\theta_{j-1}^{(i)}}^2]\eqsp.
\end{equation}
Using \Cref{lem:bound_second_moment_difference} and \Cref{lem:bound_sum_exponent}, we obtain
\begin{align}
&\sum_{j=i+1}^{n-1}\PE[\norm{\theta_{j-1}-\theta_{j-1}^{(i)}}^2] \leq R_1R_2\exp\biggl\{-\frac{\mu c_0}{4}(i+k_0-1)^{1-\gamma}\biggr\}\alpha_i^2(\norm{\theta_{0}-\thetas}^2 + \sigma_2^2)T_2\biggl(\frac{\mu c_0}{1-\gamma}\biggr)(i+k_0)^{\gamma} \\
& \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad + R_1R_3\sigma_2^2\alpha_i^2T_2\biggl(\frac{\mu c_0}{1-\gamma}\biggr)(i+k_0)^{\gamma} \\
& \qquad \leq R_1R_3\sigma_2^2c_0T_2\biggl(\frac{\mu c_0}{1-\gamma}\biggr)\alpha_i + R_1R_2c_0^2k_0^{-\gamma}T_2\biggl(\frac{\mu c_0}{1-\gamma}\biggr)\exp\biggl\{-\frac{\mu c_0}{4}(i+k_0-1)^{1-\gamma}\biggr\}(\norm{\theta_{0}-\thetas}^2+\sigma_2^2)\eqsp.
\end{align}
Combining inequalities above, we get
\begin{align}
\sum_{i=1}^{n-1}\PE^{1/2}[\norm{D_{n,3}-D_{n,3}^{(i)}}^2] &\leq \frac{\sqrt{2}C_{\Sigma}C_QL_2}{\sqrt{n}}\sqrt{C_1+ c_0^2k_0^{-\gamma}R_1R_2T_2\biggl(\frac{\mu c_0}{1-\gamma}\biggr)}T_1\biggl(\frac{\mu c_0}{8}\biggr)(\norm{\theta_0-\thetas}+\sigma_2)\\ &+\frac{\sqrt{2}C_{\Sigma}C_QL_2}{\sqrt{n}}\sqrt{C_2+ R_1R_3c_0T_2\biggl(\frac{\mu c_0}{1-\gamma}\biggr)}\sigma_2\left(\frac{(n+k_0-2)^{1-\gamma/2}-(k_0-1)^{1-\gamma/2}}{1-\gamma/2}\right)\eqsp.
\end{align}
We now proceed with $\sum_{i=1}^{n-1}\PE^{1/2}[\norm{D_{n,2}-D_{n,2}^{(i)}}^2]$.
Using Minkowski's inequality together with \Cref{lem:bound_Q_i_and_Sigma_n} and \Cref{lem:H_theta_bound}, we get
\begin{equation}
\PE^{1/2}[\norm{D_{n,2}-D_{n,2}^{(i)}}^2] \leq \frac{C_{\Sigma} C_Q L_H}{\sqrt{n}}\sum_{j=i+1}^{n-1}\PE^{1/2}[\norm{\theta_{j-1}-\theta_{j-1}^{(i)}}^4]\eqsp.
\end{equation}
Applying \Cref{lem:bound_fourth_moment_difference} and \Cref{lem:bound_sum_exponent}, we get using that $\alpha_i^2 (i+k_0)^{\gamma} \leq \alpha_0^2k_0^{-\gamma}$ that
\begin{align}
\sum_{j=i+1}^{n-1}\PE^{1/2}[\norm{\theta_{j-1}-\theta_{j-1}^{(i)}}^4] &\leq c_0^2k_0^{-\gamma}\sqrt{R_{4,1}R_{4,2}}T_2(\frac{\mu c_0}{1-\gamma})\exp\{-\frac{\mu c_0}{4}(i+k_0-1)^{1-\gamma}\}(\norm{\theta_{0}-\thetas}^2 + \sigma_4^2) \\
& \qquad \qquad \qquad \qquad + \alpha_ic_0\sqrt{R_{4,1}R_{4,3}}T_2(\frac{\mu c_0}{1-\gamma})\sigma_4^2\eqsp.
\end{align}
Finally, applying \Cref{lem:bound_sum_exponent}, we get
\begin{align}
\sum_{i=1}^{n-1}\PE^{1/2}[\norm{D_{n,2}-D_{n,2}^{(i)}}^2] &\leq \frac{C_{\Sigma} C_Q L_H}{\sqrt{n}}c_0^2k_0^{-\gamma}\sqrt{R_{4,1}R_{4,2}}T_2(\frac{\mu c_0}{1-\gamma})T_1(\frac{\mu c_0}{4})(\norm{\theta_{0}-\thetas}^2 + \sigma_4^2) \\&+ \frac{C_{\Sigma} C_Q L_H}{\sqrt{n}}c_0\sqrt{R_{4,1}R_{4,3}}T_2(\frac{\mu c_0}{1-\gamma})\sigma_4^2(\frac{(n+k_0-2)^{1-\gamma}-(k_0-1)^{1-\gamma}}{1-\gamma})\eqsp.
\end{align}
We finish the proof, using that $(n-2+k_0)^{\beta}- (k_0-1)^{\beta} \leq n^{\beta}$ for $\beta \in (0,1)$.
\end{proof}
\subsection{Bounds for $\theta_k^{(i)}-\theta_k$}
Let $(\xi_1',\ldots, \xi_{n-1}')$ be an independent copy of $(\xi_1,\ldots, \xi_{n-1})$. For each $1 \leq i \leq n-1$, we construct the sequence $\theta_k^{(i)}$, $1 \leq k \leq n-1$, as follows:
\begin{equation}
\label{eq:independent_replace_construct}
\theta_k^{(i)} =
\begin{cases}
\theta_k\eqsp, \qquad &\text{if }k<i\\
\theta_{k-1}^{(i)} - \alpha_k(\nabla f(\theta_{k-1}^{(i)}) + g(\theta_{k-1}^{(i)}, \xi_k')+ \eta(\xi_k')) \eqsp,\qquad&\text{if }k=i\\
\theta_{k-1}^{(i)} - \alpha_k(\nabla f(\theta_{k-1}^{(i)}) + g(\theta_{k-1}^{(i)}, \xi_k)+\eta(\xi_k))\eqsp,\qquad&\text{if }k>i \eqsp.
\end{cases}
\end{equation}
\begin{lemma}
\label{lem:bound_second_moment_difference}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($2$) and \Cref{ass:step_size}. Then for any $k \in \nset$ and $1 \leq i \leq n-1$ it holds
\begin{equation}
\PE[\norm{\theta_k^{(i)}-\theta_k}^2] \leq \alpha_i^2R_1\exp\biggl\{-2\mu\sum_{j=i+1}^{k}\alpha_j\biggr\}\biggl(R_2\exp\biggl\{-\frac{\mu c_0}{4}(i+k_0-1)^{1-\gamma}\biggr\}(\norm{\theta_{0}-\thetas}^2 + \sigma_2^2) + R_3\sigma_2^2
\biggr),
\end{equation}
where we have set
\begin{equation}
\label{eq:def_R_1_R_2_R_3}
R_1 = 4 \exp\biggl\{\frac{2 c_0^2(L_1+L_2)^2}{2\gamma-1}\biggr\}\eqsp, \qquad R_2 = L_2^2C_1, \qquad R_3 =(1+C_2L_2) \eqsp.
\end{equation}
And constant $C_1$ and $C_2$ are defined in \Cref{lem:bound_last_iter_second_moment}.
\end{lemma}
\begin{proof}
By construction \eqref{eq:independent_replace_construct}, we have
\begin{equation}
\theta_k^{(i)}-\theta_k =
\begin{cases}
0\eqsp, \quad &\text{if }k<i\\
- \alpha_k\bigl(g(\theta_{k-1}, \xi_k')+\eta(\xi_k')-g(\theta_{k-1}, \xi_k)-\eta(\xi_k)\bigr)\eqsp,\quad&\text{if }k=i\\
\theta_{k-1}^{(i)}-\theta_{k-1}-\alpha_k\bigl(\nabla f(\theta_{k-1}^{(i)}) - \nabla f (\theta_{k-1}) +g(\theta_{k-1}^{(i)}, \xi_k)-g(\theta_{k-1}, \xi_k)\bigr) \eqsp,\quad&\text{if }k>i\\
\end{cases}
\end{equation}
Since $\xi_i'$ is independent copy of $\xi_i$, we obtain
\begin{align}
\PE[\norm{\theta_i^{(i)}-\theta_i}^2] &\overset{(a)}{\leq} 4\alpha_i^2(L_2^2\PE[\norm{\theta_{i-1}-\thetas}^2] + \sigma_2^2) \\&\overset{(b)}{\leq} 4\alpha_i^2\biggl(L_2^2C_1\exp\biggl\{-\frac{\mu c_0}{4}(i+k_0-1)^{1-\gamma}\biggr\}(\norm{\theta_{0}-\thetas}^2 + \sigma_2^2) + (1+C_2L_2)\sigma_2^2
\biggr),
\end{align}
where in (a) we used \Cref{ass:noise_decomposition}, and in (b) we used \Cref{lem:bound_last_iter_second_moment} and $\alpha_{k-1}L_2 \leq 1$.
For $k>i$, applying \cref{ass:noise_decomposition} and \cref{ass:L-smooth}, we have
\begin{align}
\PE [\norm{\theta_k^{(i)}-\theta_k}^2|\F_{k-1}] &\leq \norm{\theta_{k-1}^{(i)}-\theta_{k-1}}^2 -2\alpha_k\langle\theta_{k-1}^{(i)}-\theta_{k-1}, \nabla f(\theta_{k-1}^{(i)}) - \nabla f (\theta_{k-1}) \rangle \\&\qquad + 2\alpha_k^2(L_1+L_2)^2\norm{\theta_{k-1}^{(i)}-\theta_{k-1}}^2\eqsp.
\end{align}
Taking expectation from both sides and applying \Cref{ass:L-smooth} with \Cref{lem:bounds_on_sum_step_sizes}\ref{eq:sum_alpha_k_p}, we obtain
\begin{align}
\PE [\norm{\theta_k^{(i)}-\theta_k}^2] &\leq (1-2\alpha_k\mu + 2\alpha_k^2(L_1+L_2)^2)\PE[\norm{\theta_{k-1}^{(i)}-\theta_{k-1}}]^2 \\&\leq \exp\biggl\{\frac{2c_0^2(L_1+L_2)^2}{2\gamma-1}\biggr\}\exp\biggl\{-2\mu\sum_{j=i+1}^{k}\alpha_j\biggr\}\PE[ \norm{\theta_i^{(i)}-\theta_i}^2]\eqsp.
\end{align}
Combining the above inequalities completes the proof.
\end{proof}
\begin{lemma}
\label{lem:bound_fourth_moment_difference}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($4$) and \Cref{ass:step_size}. Then for any $k \in \nset$ and $1 \leq i \leq n-1$ it holds
\begin{equation}
\PE[\norm{\theta_k^{(i)}-\theta_k}^4] \leq\alpha_i^4R_{4,1}\exp\biggl\{ -4\mu\sum_{j=i+1}^k\alpha_j\biggr\}\biggl(R_{4,2}\exp\{-\frac{2\mu c_0}{4}(i+k_0-1)^{1-\gamma}\}(\norm{\theta_{0}-\thetas}^4 + \sigma_4^4) + R_{4,3}\sigma_4^4\biggr)
\end{equation}
where we have set
\begin{equation}
\label{eq:def_R_41_R_42_R_43}
R_{4,1} = 64\exp\biggl\{ \frac{4(L_1+L_2)^2(1+3c_0(L_1+L_2))^2)}{2\gamma-1}\biggr\}\eqsp, \quad R_{4,2} = L_2^4C_{4,1}\eqsp, \quad R_{4,3} =1+ L_2^2C_{4,2}\eqsp.
\end{equation}
And constant $C_{4,1}$, $C_{4,2}$ are defined in \Cref{cor:fourth_moment_bound_last_iterate}.
\end{lemma}
\begin{proof}
Repeating the proof of the \Cref{lem:bound_second_moment_difference} for $k = i$, we get
\begin{align}
\PE[\norm{\theta_i^{(i)}-\theta_i}^4] &\leq64\alpha_i^4(L_2^4\PE[\norm{\theta_{i-1}-\thetas}^4] + \sigma_4^4) \\& \leq 64\alpha_i^4\biggl(L_2^4C_{4,1}\exp\biggl\{-\frac{2\mu c_0}{4}(i+k_0-1)^{1-\gamma}\biggr\}(\norm{\theta_{0}-\thetas}^4 + \sigma_4^4) + (1+L_2^2C_{4,2})\sigma_4^4\biggr)\eqsp.
\end{align}
For $k>i$ we denote $\delta_k^{(i)} = \norm{\theta_k^{(i)}-\theta_k}$, similar to \eqref{eq:coupling_recurence}, we obtain
\begin{align}
E[\{\delta_k^{(i)}\}^4|\F_{k-1}] \leq (1 -4\mu\alpha_k + 4\alpha_k^2(L_1+L_2)^2(1+3c_0(L_1+L_2))^2)\{\delta_{k-1}^{(i)}\}^4\eqsp.
\end{align}
Using
\Cref{lem:bounds_on_sum_step_sizes}\ref{eq:sum_alpha_k_p}, we obtain
\begin{equation}
E[\{\delta_k^{(i)}\}^4]\leq \exp\biggl\{ \frac{4(L_1+L_2)^2(1+3c_0(L_1+L_2))^2)}{2\gamma-1}\biggr\}\exp\biggl\{ -4\mu\sum_{j=i+1}^k\alpha_j\biggr\} \PE[\norm{\theta_i^{(i)}-\theta_i}^4]\eqsp.
\end{equation}
Combining the above inequalities completes the proof.
\end{proof}
% \begin{theorem}
% \label{GAR bootstrap}
% Assume \Cref{ass:L-smooth} - \Cref{ass:n_lower_bound}. Then with $\PP$ - probability at least $1 - 2/n$, it holds
% \begin{equation}
% \begin{split}
% &\sup_{B \in \Conv(\rset^d)}|\PPb( \sqrt n\{\Sigma_n^\boot\}^{-1/2} (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B) - \PPb(Y^\boot \in B)| \\ &\qquad\qquad\le \frac{M_{3,1}^\boot}{n^{1/2}} + \frac{M_{3,2}^\boot \log n}{n^{\gamma-1/2}} + \frac{M_{3,3}^\boot \log^{3/2} n}{n^{\gamma/2}} \eqsp,
% \end{split}
% \end{equation}
% where
% \begin{equation}
% \label{M_3i boot def}
% \begin{split}
% &M_{3,1}^\boot = 259 (\sqrt{2} C_\Sigma C_Q C_{1, \xi})^3W_{\max} \sqrt{d} \eqsp,\\
% &M_{3,2}^\boot =2^{3/2}c_dC_{\Sigma}M_{2,1}^{\boot} e^{3/2+\gamma}\eqsp, \\
% &M_{3,2}^\boot = 2^{3/2}c_d C_{\Sigma}M_{1,1}^{\boot} e^{3/2+\gamma/2}\eqsp,
% \end{split}
% \end{equation}
% and $M_{1,1}^{\boot}, M_{2,1}^{\boot}$ are defined in \eqref{eq:def_M_11_M_21_boot}.
% \end{theorem}
Since the matrix $\Sigma_n^\boot$ concentrates around $\Sigma_n$ due to \Cref{lem:matrix_bernstein} , there is a set $\Omega_1$ such that $\PP(\Omega_1) \geq 1-1/n$ and $\lambda_{\min}(\Sigma_n^\boot) > 0$ on $\Omega_1$. Moreover, on this set
Applying Lemma \ref{nonlinearapprox} with
$$
X = \{\Sigma_n^\boot\}^{-1/2} W^\boot, \quad Y = \{\Sigma_n^\boot\}^{-1/2} D^\boot,
$$
we get
\begin{equation}
\begin{split}
&\sup_{B \in \Conv(\rset^d)}|\PPb( \sqrt n \{\Sigma_n^\boot\}^{-1/2} (\bar{\theta}_{n}^\boot - \bar{\theta}_n) \in B) - \PPb(Y^\boot \in B)| \\
& \qquad\qquad\le \sup_{B \in \Conv(\rset^d)}|\PPb(\{\Sigma_n^\boot\}^{-1/2} W^\boot \in B) - \PPb(Y^\boot \in B)| + 2 c_d (\PEb[\|\{\Sigma_n^\boot\}^{-1/2} D^\boot\|^p])^{1/(1+p)} \eqsp.
\end{split}
\end{equation}
By \citep{shao2022berry} (with $D = 0$) we may estimate
\begin{equation}
\label{CLT boot lin stat proof}
\begin{split}
&\sup_{B \in \Conv(\rset^d)}|\PPb(\{\Sigma_n^\boot\}^{-1/2} W^\boot \in B) - \PPb(Y^\boot \in B)| \\
&\qquad\qquad\qquad\qquad\le \frac{259 d^{1/2}}{n^{3/2}} \sum_{i = 1}^n \PEb[|w_i-1|^3] \| (\{\Sigma_n^\boot\}^{-1/2} Q_i
\eta(\xi_i)\|^3 \eqsp.
\end{split}
\end{equation}
Applying Lemma \ref{lem:bound_Q_i_and_Sigma_n} and Corollary \ref{lambda min boot} we get
\begin{equation}
\label{CLT boot lin stat proof 2}
\sup_{B \in \Conv(\rset^d)}|\PPb(\{\Sigma_n^\boot\}^{-1/2} W^\boot \in B) - \PPb(Y^\boot \in B)| \le \frac{259 d^{1/2} (\sqrt{2} C_\Sigma C_Q C_{1, \xi})^3 W_{\max}}{n^{1/2}} \eqsp.
\end{equation}
From Proposition \ref{prop:prob-D-boot-bound} and Corollary \ref{lambda min boot} it follows that on the set we $\Omega_0 \cap \Omega_1$ the following bound is satisfied
\begin{equation}
(\PEb[\|\{\Sigma_n^\boot\}^{-1/2} D^\boot\|^p])^{1/(p+1)} \leq \sqrt{2}C_{\Sigma}(M_{1,1}^{\boot}\rme^{1/p}p^{3/2}n^{2/p-\gamma/2} + M_{2,1}^{\boot}\rme^{2/p}p n^{1/2+3/p-\gamma})^{p/(p+1)} \eqsp.
\end{equation}
Since $p\geq 2, M_{1,1}^{\boot}, M_{2,1}^{\boot} \geq 1$, we obtain
\begin{equation}
(\PEb[\|\{\Sigma_n^\boot\}^{-1/2} D^\boot\|^p])^{1/(p+1)} \leq
\sqrt{2} C_{\Sigma}(\rme^{1/2} M_{1,1}^{\boot}p^{3/2} n^{\frac{2}{p+1}}n^{-\gamma/2}n^{\frac{\gamma/2}{(p+1)}} + \rme M_{2,1}^{\boot} p n^{\frac{3}{p+1}}n^{1/2-\gamma}n^{-\frac{1/2-\gamma}{p+1}}) \eqsp.
\end{equation}
Setting $p = \log n - 1$, we get
\begin{equation}
(\PEb[\|\{\Sigma_n^\boot\}^{-1/2} D^\boot\|^p])^{1/(p+1)} \leq
\sqrt{2}C_{\Sigma}(M_{1,1}^{\boot}(\log n)^{3/2} e^{3/2+\gamma/2}n^{-\gamma/2}+ M_{2,1}^{\boot} (\log n) e^{3/2+\gamma}n^{1/2-\gamma})\eqsp.
\end{equation}
Setting
\begin{equation}
\label{M_3i boot def}
\begin{split}
&M_{3,1}^\boot = 259 (\sqrt{2} C_\Sigma C_Q C_{1, \xi})^3W_{\max} \sqrt{d} \eqsp,\\
&M_{3,2}^\boot =2^{3/2}c_dC_{\Sigma}M_{2,1}^{\boot} e^{3/2+\gamma}\eqsp, \\
&M_{3,2}^\boot = 2^{3/2}c_d C_{\Sigma}M_{1,1}^{\boot} e^{3/2+\gamma/2}\eqsp,
\end{split}
\end{equation}
and $M_{1,1}^{\boot}, M_{2,1}^{\boot}$ are defined in \eqref{eq:def_M_11_M_21_boot} and
combining the above inequalities, we complete the proof.
\begin{remark}
We use \citep{shao2022berry} with $D = 0$ to prove \eqref{CLT boot lin stat proof} since we are not aware of Berry-Esseen results for non i.i.d. random vectors in dimension $d$ with precise constants and dependence on $d$. The result \cite{bentkus2004} may be applied for i.i.d. vectors only.
\end{remark}
\subsection{From non-linear to linear statistics}
\label{sec: from nonlinear to linear boot}
In this section we prove \eqref{crude bound for nonlinear stat}. We start from the definition of an isoperimetric constant.
Define
\[
A^{\varepsilon} = \{x \in \rset^d : \rho_A(x) \leq \varepsilon\}
\quad \text{and} \quad
A^{-\varepsilon} = \{x \in A : B_{\varepsilon}(x) \subset A\},
\]
where $\rho_A(x) = \inf\limits_{y \in A} \|x - y\|$ is the distance between $A \subset \rset^d$ and $x \in \rset^d$, and
\[
B_{\varepsilon}(x) = \{y \in \rset^d : \|x - y\| \leq \varepsilon\}.
\]
For some class $\mathcal A$ of subsets of $\rset^d$ we define its isoperimetric constant $a_d(\mathscr{A})$ (depending only on $d$ and $\mathscr{A}$) as follows: for all $A \in \mathscr{A}$ and $\varepsilon > 0$,
\[
\mathbb{P} \{ Y \in A^{\varepsilon} \setminus A \} \leq a_d \varepsilon, \quad
\mathbb{P} \{ Y \in A \setminus A^{-\varepsilon} \} \leq a_d \varepsilon
\]
where $Y$ follows the standard Gaussian distribution on $\rset^d$. \citep{ball_reverse_1993} has proved that
\begin{equation}
\label{ball}
e^{-1} \sqrt{\ln d} \leq \sup_{A \in \mathscr{C}} \int_{\partial A} p(x) \, \mathrm{d}s \leq 4 d^{1/4},
\end{equation}
where $p(x)$ is the standard normal $d$-dimensional density and $ds$ is the surfac emeasure on the boundary $\partial A$ of $A$. Using \eqref{ball} one can show that for the class of convex sets
\begin{equation}
\label{eq: isoperimetric const convex}
e^{-1} \sqrt{\ln d} \leq a_d(\Conv(\rset^d)) \le 4 d^{1/4} \eqsp.
\end{equation}
We denote $c_d = a_d(\Conv(\rset^d))$.
\begin{proposition}
\label{nonlinearapprox}
Let $\nu$ be a standard Gaussian measure in $\rset^d$. Then for any random vectors $X, Y$ taking values in $\rset^d$, and any $p \geq 1$,
\begin{equation}
\label{eq:shao_zhang_bound}
\sup_{B \in \Conv(\rset^d)}|\PP(X + Y \in B) - \nu(B)| \le \sup_{B \in \Conv(\rset^d)}|\PP(X
\in B) - \nu(B)| + 2 c_d^{p/(p+1)} \PE^{1/(p+1)}[\|Y\|^p]\eqsp,
\end{equation}
where $c_d$ is the isoperimetric constant of class $\Conv(\rset^d)$.
\end{proposition}
\begin{proof}
Let $\varepsilon \geq 0$. Define
$\rho(B) = \PP(X + Y \in B) - \nu(B)$.
Let $B$ be such that $\rho(B) \geq 0$. By Markov's inequality
\begin{multline}
\rho(B) \le \PP(X + Y \in B, |Y| \le \varepsilon) + \frac{1}{\varepsilon^{p}} \PE [\|Y\|^p] - \nu(B) \\
\le \sup_{A}|\PP(X \in A) - \nu(A)| + \PP(Y \in B^\varepsilon \setminus B) + \frac{1}{\varepsilon^{p}} \PE [\|Y\|^p].
\end{multline}
Choosing
\begin{equation}
\label{eps choice}
\varepsilon = \frac{1}{c_d^{1/(p+1)}}\PE^{1/(p+1)}[\|Y\|^p]
\end{equation}
we obtain
\begin{equation}
\sup_{B}|\PP(X + Y \in B) - \nu(B)| \le \sup_{B }|\PP(X \in B) - \nu(B)| + 2 c_d^{p/(p+1)} \PE^{1/(p+1)}[\|Y\|^p] \eqsp.
\end{equation}
Assume now that $\rho(B) < 0$. We distinguish between $B^{-\varepsilon} = \emptyset$ or $B^{-\varepsilon} \neq \emptyset$. In the first case, $\PP(Y \in B^{-\varepsilon}) = 0$ and
$$
-\rho(B) \le \gamma(B) = \PP(Y \in B) - \PP(Y \in B^{-\varepsilon}) = \PP(Y \in B \setminus B^{-\varepsilon}) \leq c_d \varepsilon.
$$
Finally, in the case $B^{-\varepsilon} \neq \emptyset$,
$$
-\rho(B) \le \sup_{A}|\PP(X \in A) - \nu(A)| + \PP(Y \in B \setminus B^{-\varepsilon}) + \frac{1}{\varepsilon^{p}} \PE [\|Y\|^p]\eqsp.
$$
Taking $\varepsilon$ as in \eqref{eps choice}
we conclude the proof.
\end{proof}
% \begin{lemma}
% Assum ...
% \begin{equation}
% \norm{\theta_k^{\boot} - \theta_k}^2 \lesssim \alpha_k \eqsp.
% \end{equation}
% \end{lemma}
% \begin{proof}
% Note that
% \begin{multline}
% \theta_k^{\boot} - \theta_k = \theta_{k-1}^{\boot} -\theta_{k-1}- \alpha_{k} w_k\{\nabla f(\theta_{k-1}^{\boot})+g(\theta_{k-1}^{\boot}, \xi_k) + \eta(\xi_k)\} + \alpha_{k}\{\nabla f(\theta_{k-1})+g(\theta_{k-1}, \xi_k) + \eta(\xi_k)\}\\=
% \theta_{k-1}^{\boot} -\theta_{k-1}- \alpha_{k}(w_k-1)\{\nabla f(\theta_{k-1}^{\boot})+g(\theta_{k-1}^{\boot}, \xi_k) + \eta(\xi_k)\}\\- \alpha_{k}\{\nabla f(\theta_{k-1}^{\boot})+g(\theta_{k-1}^{\boot}, \xi_k)-\nabla f(\theta_{k-1})-g(\theta_{k-1}, \xi_k) \}\eqsp.
% \end{multline}
% Using \cref{ass:L-co-coercivity-assum}, \cref{ass:L-smooth}, \Cref{ass:noise_decomposition} and \cref{ass:bound_noise}, we get
% \begin{align}
% &\PE_{w_k}[\norm{\theta_k^{\boot} - \theta_k}^2] \leq \norm{\theta_{k-1}^{\boot} -\theta_{k-1}}^2 + \alpha_{k}^2\norm{\nabla f(\theta_{k-1}^{\boot})+g(\theta_{k-1}^{\boot}, \xi_k) + \eta(\xi_k)}^2
% \\& \qquad \qquad-(2\alpha_k-\alpha_k^2L_3)\langle\nabla f(\theta_{k-1}^{\boot})
% +g(\theta_{k-1}^{\boot}, \xi_k)-\nabla f(\theta_{k-1})-g(\theta_{k-1}, \xi_k), \theta_{k-1}^{\boot} -\theta_{k-1}\rangle \\
% &\qquad \leq
% (1-(2\alpha_k-\alpha_k^2L_3)\mu)\norm{\theta_{k-1}^{\boot} -\theta_{k-1}}^2 + 2\alpha_k^2(L_1+L_2)^2\norm{\theta_{k-1}^{\boot}-\thetas}^2 + 2\alpha_k^2\norm{\eta(\xi_k)}^2
% \\& \qquad \qquad -(2\alpha_k-\alpha_k^2L_3)\langle
% g(\theta_{k-1}^{\boot}, \xi_k)-g(\theta_{k-1}, \xi_k), \theta_{k-1}^{\boot} -\theta_{k-1}\rangle\\
% &\qquad \leq ?
% % (1-\alpha_k(2/L_3 - \alpha_k)(L_1+L_2)^2 + 4\alpha_k^2(L_1+L_2)^2)\norm{\theta_{k-1}^{\boot} -\theta_{k-1}}^2 \\
% % &\qquad \qquad \qquad \qquad \qquad + 4\alpha_k^2(L_1+L_2)^2\norm{\theta_{k-1}-\thetas}^2 + 2\alpha_k^2 \norm{\eta(\xi_k)}^2\eqsp.
% \end{align}
% \end{proof}
\subsection{Bounds for $D^\boot$}
Recall that the term $D^\boot$ defined in \eqref{eq:D_boot-def}, has a form:
\begin{equation}
\label{eq:D_boot-def-appendix}
\begin{split}
D^\boot &= -\frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}(w_i-1) Q_{i}\biggl( G(\theta_{i-1}^{\boot}-\thetas) + g(\theta_{i-1}^{\boot}, \xi_i) + H(\theta_{i-1}^{\boot})\biggr) \\&- \frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i\biggl(H(\theta_{i-1}^{\boot}) +g(\theta_{i-1}^{\boot}, \xi_i) - H(\theta_{i-1}) -g(\theta_{i-1}, \xi_i)\biggr)\eqsp.
\end{split}
\end{equation}
To prove \Cref{GAR bootstrap_main}, we need to obtain a high-probability bound for the non-linear statistic $D^\boot$.
To this end, we first derive a bound on $\PE^{1/p}[\|D^\boot\|^p]$, where the expectation is taken with respect to the joint distribution of the bootstrap weights and the data $\Xi^{n-1}$.
We then apply Markov's inequality to convert this moment bound into a high-probability bound.
\begin{proposition} \label{prop:prob-D-boot-bound}
Assume \Cref{ass:L-smooth}- \Cref{ass:step_size_new_boot}. Then it holds for any $p \geq 2$ that
\begin{equation}
\label{eq:D-boot-joint-bound}
\PE^{1/p}[\norm{D^\boot}^p] \leq M_{1,1}^{\boot}\rme^{1/p}p^{3/2}n^{1/p-\gamma/2} + M_{2,1}^{\boot}\rme^{2/p}p n^{1/2+2/p-\gamma}\eqsp,
\end{equation}
where the constants are given by
\begin{equation}
\label{eq:def_M_11_M_21_boot}
\begin{split}
&M_{1,1}^{\boot} =4C_Q\max(L_1,L_2)\frac{\max(\sqrt{K_2},\sqrt{K_1})\sqrt{c_0k_0^{1-\gamma}}(W_{\max}+1)}{\sqrt{2}(1-\gamma)}\eqsp,\\
& M_{2,1}^{\boot}= 3C_QL_H\frac{c_0k_0^{1-\gamma}\max(K_2,K_1)(W_{\max}+1)}{2(1-\gamma)} \eqsp,
\end{split}
\end{equation}
and $K_1, K_2$ are defined in \eqref{eq:def_K_1}, \eqref{eq:def_K_2}, respectively.
Moreover, there is a set $\Omega_0 \in \F_{n-1} = \sigma(\xi_1,\ldots,\xi_{n-1})$, such that $\PP(\Omega_0) \geq 1-1/n$, and on $\Omega_0$ it holds that
\begin{equation}
\label{eq:D-boot-high-prob-bound}
\{\PEb[\norm{D^\boot}^{p}]\}^{1/p} \leq M_{1,1}^{\boot}\rme^{1/p}p^{3/2}n^{2/p-\gamma/2} + M_{2,1}^{\boot}\rme^{2/p}p n^{1/2+3/p-\gamma}\eqsp.
\end{equation}
\end{proposition}
\begin{proof}
We first show \eqref{eq:D-boot-joint-bound}. We split
\[
D^\boot = D^\boot_{1} + D^\boot_{2}\eqsp,
\]
where
\begin{equation}
\begin{split}
D^\boot_{1} &= -\frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}(w_i-1) Q_{i}\bigl( G(\theta_{i-1}^{\boot}-\thetas) + g(\theta_{i-1}^{\boot}, \xi_i)\bigr) - \frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i\bigl(g(\theta_{i-1}^{\boot}, \xi_i) -g(\theta_{i-1}, \xi_i)\bigr)\eqsp, \\
D^\boot_{2} &= -\frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}(w_i-1) Q_{i} H(\theta_{i-1}^{\boot}) - \frac{1}{\sqrt{n}}\sum_{i=1}^{n-1}Q_i\bigl(H(\theta_{i-1}^{\boot})- H(\theta_{i-1})\bigr)\eqsp.
\end{split}
\end{equation}
Applying Minkowski's inequality together with \Cref{lem:bound_p_moment_martingale_components} and \Cref{lem:bound_p_moment_for_H_component} we get \eqref{eq:D-boot-joint-bound}.
To proof \eqref{eq:D-boot-high-prob-bound} we consider
\begin{equation}
\Omega_0 =\{\{\PEb[\norm{D^\boot}^{p}]\}^{1/p} \leq M_{1,1}^{\boot}\rme^{1/p}p^{3/2}n^{2/p-\gamma/2} + M_{2,1}^{\boot}\rme^{2/p}p n^{1/2+3/p-\gamma}\}\eqsp.
\end{equation}
Note that by Markov's inequality
\begin{align}
&\PP(\Omega_0^c) \leq \frac{\PE[\{\PEb[\norm{D^\boot}^{p}]\}]}{n(M_{1,1}^{\boot}\rme^{2/p}p^{3/2}n^{2/p-\gamma/2} + M_{2,1}^{\boot}\rme^{1/p}p n^{1/2+1/p-\gamma})^p}\\& =
\frac{\PE[\norm{D^\boot}^{p}]}{n(M_{1,1}^{\boot}\rme^{1/p}p^{3/2}n^{1/p-\gamma/2} + M_{2,1}^{\boot}\rme^{2/p}p n^{1/2+2/p-\gamma})^p} \leq \frac{1}{n}\eqsp.
\end{align}
\end{proof}
\label{sec: Dboot bounds}
\begin{lemma}
\label{lem:bound_p_moment_martingale_components}
Assume \Cref{ass:L-smooth}-\Cref{ass:step_size_new_boot}. Then for any $p\geq 2$ it holds
\begin{equation}
\PE^{1/p}[\norm{D^\boot_{1}}^p]\leq M_{1,1}^{\boot}\rme^{1/p}p^{3/2}n^{1/p-\gamma/2} \eqsp,
\end{equation}
where
\begin{equation}
\label{def:M_11_boot}
M_{1,1}^{\boot} = 4C_Q\max(L_1,L_2)\frac{\max(\sqrt{K_2},\sqrt{K_1})\sqrt{c_0}(W_{\max}+1)}{\sqrt{2}(1-\gamma)}\eqsp,
\end{equation}
and $K_1, K_2$ are defined in \eqref{eq:def_K_1}, \eqref{eq:def_K_2}, respectively.
% \begin{enumerate}[(a)]
% \item \label{eq: bound_p_moment_sum_Q_g}\begin{align}
% &\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}Q_i\biggl(g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i) \biggr)}^{p}\biggr] \leq M_{1,1}^{\boot}\rme^{2/p}p^{3/2}n^{2/p-\gamma/2}\eqsp,\\
% &M_{1,1}^{\boot} = \frac{C_QL_2\sqrt{8\alpha_0}(C_{1,\xi}+2C_{2,\xi})k_0^{1/2-\gamma/2}}{\sqrt{\mu(1-\gamma)}}
% \end{align}
% \item \label{eq: bound_p_moment_sum_Q_g_boot}\begin{align}
% &\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}Q_i\biggl(g(\theta_{i-1}^{\boot},\xi_i)-g(\thetas,\xi_i) \biggr)}^{p}\biggr] \leq M_{1,2}^{\boot}\rme^{2/p}p^{3/2}n^{2/p-\gamma/2}\eqsp, \\
% &M_{1,2}^{\boot}=\frac{C_QL_2\sqrt{8\alpha_0}(C_{1,\xi}+2C_{2,\xi})W_{\max}k_0^{1/2-\gamma/2}}{\sqrt{\mu(1-\gamma)W_{\min}}}
% \end{align}
% \item \label{eq: bound_p_moment_sum_Q_boot_g_boot} \begin{align}
% &\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}(w_i-1)Q_i\biggl(g(\theta_{i-1}^{\boot},\xi_i)-g(\thetas,\xi_i) \biggr)}^{p}\biggr] \leq M_{1,3}^{\boot}\rme^{2/p}p^{3/2}n^{2/p-\gamma/2}\eqsp,\\
% &M_{1,3}^{\boot}=\frac{C_QL_2\sqrt{8\alpha_0}(C_{1,\xi}+2C_{2,\xi})(W_{\max}+1)W_{\max}k_0^{1/2-\gamma/2}}{\sqrt{\mu(1-\gamma)W_{\min}}}
% \end{align}
% \item \label{eq: bound_p_moment_sum_Q_boot_theta_boot} \begin{align}
% &\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}(w_i-1)Q_iG(\theta_{i-1}^{\boot}-\thetas)}^{p}\biggr] \leq M_{1,4}^{\boot}\rme^{2/p}p^{3/2}n^{2/p-\gamma/2}\eqsp,\\
% &M_{1,4}^{\boot}=\frac{C_QL_1\sqrt{8\alpha_0}(C_{1,\xi}+2C_{2,\xi})(W_{\max}+1)W_{\max}k_0^{1/2-\gamma/2}}{\sqrt{\mu(1-\gamma)W_{\min}}}
% \end{align}
% \end{enumerate}
\end{lemma}
\begin{proof}
We split \( D^\boot_{1} \) into four parts, where each part is a sum of martingale differences.
Note that $\{Q_i(g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i))\}_{i=1}^{n}$ is a martingale difference with respect to $\F_{i-1}$.
Then applying Burholder's inequality \cite[Theorem 8.6]{oskekowski2012sharp} together with Minkowski's inequality and \Cref{lem:bound_Q_i_and_Sigma_n}, we obtain that
\begin{align}
&\PE^{1/p}\bigl[\norm{\sum_{i=1}^{n-1}Q_i\bigl(g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i) \bigr)}^{p}\bigr] \\
&\qquad \qquad \leq p\bigl(\PE^{2/p}\bigl[\bigl( \sum_{i=1}^{n-1}\norm{Q_i\bigl(g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i) \bigr)}^2\bigr)^{p/2}\bigr]\bigr)^{1/2}
\\
&\qquad \qquad \leq C_Qp\bigl(\PE^{2/p}\bigl[\bigl( \sum_{i=1}^{n-1}\norm{g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i)}^2\bigr)^{p/2}\bigr]\bigr)^{1/2}\\
&\qquad \qquad \leq C_Qp\bigl(\sum_{i=1}^{n-1}\PE^{2/p}[\norm{g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i) }^{p}]\bigr)^{1/2}\eqsp.
\end{align}
Finally, using \Cref{ass:noise_decomposition} and \Cref{lem: high_prob_last_iter}, we obtain
\begin{align}
&\PE^{1/p}\bigl[\norm{\sum_{i=1}^{n-1}Q_i\bigl(g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i) \bigr)}^{p}\bigr] \leq pC_QL_2\bigl(\sum_{i=1}^{n-1}\PE^{2/p}[\norm{\theta_{i-1} -\thetas}^{p}]\bigr)^{1/2}\\
& \qquad \qquad \leq C_QL_2(\rme n)^{1/p}p^{3/2}\frac{\sqrt{K_2}}{\sqrt{2}}\bigl(\sum_{i=0}^{n-2}\alpha_i\bigr)^{1/2} \\
& \qquad \qquad \leq C_QL_2(\rme n)^{1/p}p^{3/2}\frac{\sqrt{K_2}}{\sqrt{2}}\bigl(c_0\frac{(k_0+n-2)^{1-\gamma}-(k_0-1)^{1-\gamma}}{1-\gamma}\bigr)^{1/2}\eqsp.
\end{align}
Since $k_0 \geq 1$ and $(k_0+n-2)^{1-\gamma}-(k_0-1)^{1-\gamma}\leq n^{1-\gamma}$ we complete the proof for $$
\PE^{1/p}\bigl[\norm{\sum_{i=1}^{n-1}Q_i\bigl(g(\theta_{i-1},\xi_i)-g(\thetas,\xi_i) \bigr)}^{p}\bigr] \eqsp.
$$
The proof for other three terms is analogous, since each of the terms
\begin{align}
\{Q_i\bigl(g(\theta_{i-1}^{\boot},\xi_i)-g(\thetas,\xi_i) \bigr)\}_{i=1}^{n-1}\eqsp,\{(w_i-1)Q_i\bigl(g(\theta_{i-1}^{\boot},\xi_i)-g(\thetas,\xi_i) \bigr)\}_{i=1}^{n-1}\eqsp,
\{(w_i-1)Q_iG(\theta_{i-1}^{\boot}-\thetas)\}_{i=1}^{n-1} \eqsp,
\end{align}
are martingale differences with respect to $\widetilde{\F}_{i-1}$ (see definition in \eqref{eq: extended filtration}).
We finish the proof applying Minkowski's inequality.
\end{proof}
\begin{lemma}
\label{lem:bound_p_moment_for_H_component}
Assume \Cref{ass:L-smooth}- \Cref{ass:step_size_new_boot}. Then for any $p\geq 2$ it holds
\begin{align}
&\PE^{1/p}[\norm{D_2^{\boot}}^p]\leq M_{2,1}^{\boot}\rme^{2/p}p n^{1/2+2/p-\gamma}\eqsp,\\
\end{align}
\begin{equation}
\label{def:M_21_boot}
M_{2,1}^{\boot}= 3C_QL_H\frac{c_0\max(K_2,K_1)(W_{\max}+1)}{2(1-\gamma)}\eqsp,
\end{equation}
and $K_1, K_2$ are defined in \eqref{eq:def_K_1}, \eqref{eq:def_K_2}, respectively.
% \begin{enumerate}[(a)]
% \item \label{eq: bound_p_moment_Q_i_H}\begin{align}
% &\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}Q_iH(\theta_k)}^{p}\biggr] \leq M_{2,1}^{\boot}\rme^{1/p}p n^{1/2+1/p-\gamma}\eqsp,\\
% & M_{2,1}^{\boot}= \frac{8C_QL_H(C_{1,\xi}+2C_{2,\xi})^{2}\alpha_0 k_0^{1-\gamma}}{\mu(1-\gamma)}
% \end{align}
% \item \label{eq: bound_p_moment_Q_i_H_boot}\begin{align}
% &\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}Q_iH(\theta_k^{\boot})}^{p}\biggr] \leq M_{2,2}^{\boot}\rme^{1/p}p n^{1/2+1/p-\gamma}\eqsp,\\
% & M_{2,2}^{\boot}= \frac{8C_QL_H(C_{1,\xi}+2C_{2,\xi})^{2}W_{\max}^2\alpha_0 k_0^{1-\gamma}}{\mu(1-\gamma)W_{\min}}
% \end{align}
% \item \label{eq: bound_p_moment_Q_i_boot_H_boot}\begin{align}
% &\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}(w_i-1)Q_iH(\theta_k^{\boot})}^{p}\biggr] \leq M_{2,3}^{\boot}\rme^{1/p}p n^{1/2+1/p-\gamma}\eqsp,\\
% & M_{2,3}^{\boot}= \frac{8C_QL_H(C_{1,\xi}+2C_{2,\xi})^{2}W_{\max}^2(W_{\max}+1)\alpha_0 k_0^{1-\gamma}}{\mu(1-\gamma)W_{\min}}
% \end{align}
% \end{enumerate}
\end{lemma}
\begin{proof}
Using Minkowski's inequality, we get
\begin{equation}
\label{eq: D_2 boot bound}
\begin{split}
\PE^{1/p}[\norm{D_2^{\boot}}^p] &\leq \frac{1}{\sqrt{n}}\PE^{1/p}[\norm{\sum_{i=1}^{n-1}Q_i H(\theta_{i-1})}^p]\\& +\frac{1}{\sqrt{n}}\PE^{1/p}[\norm{\sum_{i=1}^{n-1}(w_i-1) Q_{i}\biggl(H(\theta_{i-1}^{\boot})\biggr)}^p]\\
&+ \frac{1}{\sqrt{n}}\PE^{1/p}[\norm{\sum_{i=1}^{n-1}Q_iH(\theta_{i-1}^{\boot})}^p]\eqsp.
\end{split}
\end{equation}
We will now consider each term separately.
Using Minkowski's inequality together with \Cref{lem:H_theta_bound}, we obtain
\begin{equation}
\begin{split}
&\frac{1}{\sqrt{n}}\PE^{1/p}\biggl[\norm{\sum_{i=1}^{n-1}Q_iH(\theta_{i-1})}^{p}\biggr] \leq \frac{C_QL_H}{\sqrt{n}}\sum_{i=0}^{n-2}\PE^{1/p}\biggl[\norm{\theta_{i}-\thetas}^{2p}\biggr] \\&\qquad \qquad\leq \frac{C_QL_Hp}{\sqrt{n}}(\rme n)^{2/p}(K_2/2)\sum_{i=0}^{n-1}\alpha_i \\&\qquad \qquad \leq \frac{C_QL_Hp}{\sqrt{n}}(\rme n)^{2/p}(K_2/2)\biggl(c_0\frac{(k_0+n-2)^{1-\gamma}-(k_0-1)^{1-\gamma}}{1-\gamma} \biggr)\eqsp.
\end{split}
\end{equation}
Since $k_0 \geq 1$ and $(k_0+n-2)^{1-\gamma}-(k_0-1)^{1-\gamma}\leq n^{1-\gamma}$ we complete the proof for the first term in the r.h.s. of \eqref{eq: D_2 boot bound}. The proof for other two terms is analogous.
\end{proof}
The result follows immediately from the triangle inequality together with \Cref{th:bound_kolmogorov_dist_pr_sigma_n} and \Cref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty}. Therefore, it remains to prove \Cref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty} to complete the proof.
\subsection{Proof of Lemma \ref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty}}
\label{sec: proof of difference between cov}
By definition of $\Sigma_n$ and $\Sigma_\infty$ we may write
\begin{align}
\label{repr:Lstar_minus_Ln}
\Sigma_n - \Sigma_{\infty} = \underbrace{\frac{1}{n}\sum_{t=1}^{n-1} (Q_t - G^{-1})\noisecov G^{-\top} + \frac{1}{n} \sum_{t=1}^{n-1} G^{-1} \noisecov (Q_t - G^{-1})^\top}_{D_1} + \\ + \underbrace{\frac{1}{n} \sum_{t=1}^{n-1} (Q_t - G^{-1}) \noisecov (Q_t - G^{-1})^{\top}}_{D_2} - \frac{1}{n} \Sigma_{\infty} \eqsp.
\end{align}
The following lemma is an analogue of \citep[pp. 26-30]{wu2024statistical}.
\begin{lemma} The following identities hold
\begin{equation}
\label{repr:qtminusa}
Q_i - G^{-1} = S_i - G^{-1} G_{i:n-1}^{(\alpha)} ,~ S_i = \sum_{j=i+1}^{n-1} (\alpha_i - \alpha_j) G_{i+1:j-1}^{(\alpha)} \eqsp,
\end{equation}
and
\begin{equation}
\label{repr:sumqtminusa}
\sum_{i=1}^{n-1} (Q_i - G^{-1}) = -G^{-1} \sum_{j=1}^{n-1} G_{1:j}^{(\alpha)} \eqsp,
\end{equation}
where
\begin{equation}
G_{i:j}^{(\alpha)} = \prod_{k=i}^{j}(I-\alpha_kG)
\end{equation}
\end{lemma}
\begin{proof}
To prove \eqref{repr:sumqtminusa} we first change the order of summation and then use the properties of the telescopic sums we get
\begin{align}
\sum_{i=1}^{n-1} Q_i &= \sum_{i=1}^{n-1} \alpha_i \sum_{j=i}^{n-1}\prod_{k=i+1}^{j}(\Id-\alpha_k G) = \sum_{j=1}^{n-1} \sum_{i = 1}^j \alpha_i \prod_{k=i+1}^{j}(\Id-\alpha_k G) \\
& = \sum_{j=1}^{n-1} \sum_{i = 1}^j G^{-1} (\prod_{k = i+1}^{j} - \prod_{k = i}^{j}) (\Id - \alpha_k G) = G^{-1} \sum_{j=1}^{n-1} (\Id - \prod_{k = 1}^j (\Id - \alpha_k G)) \eqsp.
\end{align}
The proof of \eqref{repr:qtminusa} could be obtained by the following arguments. Note that
\begin{align}
\alpha_i G Q_i & = Q_i - (\Id - \alpha_i G) Q_i = \\
& = \alpha_i \Id + \alpha_i \sum_{j=i+1}^{n-1}\prod_{k=i+1}^{j}(\Id-\alpha_k G) - \alpha_i \sum_{j=i+1}^{n-1} \prod_{k=i}^{j-1}(\Id - \alpha_k G) - \alpha_i \prod_{k=i}^{n-1} (\Id - \alpha_k G)\eqsp.
\end{align}
It remains to note that
$$
\prod_{k=i+1}^{j}(\Id-\alpha_k G) - \prod_{k=i}^{j-1}(\Id - \alpha_k G) = (\alpha_i - \alpha_j) G \prod_{k=i+1}^{j-1} (\Id - \alpha_k G) \eqsp.
$$
The last two equations imply \eqref{repr:qtminusa}.
\end{proof}
\begin{lemma}
\label{bound_G_S}
It holds that
\begin{enumerate}[(a)]
\item
\begin{equation}
\norm{S_i}\leq C_{S}(i + k_0)^{\gamma-1}\eqsp,
\end{equation}
where
$$
C_{S} = 2c_0\exp\biggl\{\frac{\mu c_0}{k_0^{\gamma}}\biggr\}\biggl(2^{\gamma/(1-\gamma)}\frac{1}{\mu c_0} + (\frac{1}{\mu c_0})^{1/(1-\gamma)}\Gamma(\frac{1}{1-\gamma})\biggr)\eqsp.
$$
\item
\begin{equation}
\sum_{i=1}^{n-1}\norm{G_{i:n-1}^{(\alpha)}}^2\leq \frac{1}{1 - (1 - c_0\mu(n+k_0-2)^{-\gamma})^2}
\end{equation}
\item
\begin{equation}
\norm{\sum_{i=1}^{n-1}G_{i:n-1}^{(\alpha)}}\leq \frac{k_0^\gamma n^\gamma}{c_0\mu}
\end{equation}
\end{enumerate}
\end{lemma}
\begin{proof}
For simplicity we define $m_i^j = \sum_{k=i}^j (k+k_0)^{-\gamma}$.
Note that
\begin{equation}
\norm{\sum_{j=i+1}^{n-1} (\alpha_i - \alpha_j) G_{i+1:j-1}^{(\alpha)}} \leq \sum_{j=i}^{n-2} \frac{c_0}{(j+k_0+1)^\gamma}\biggl(\biggl(\frac{j+k_0+1}{i+k_0}\biggr)^\gamma - 1\biggr) \exp\{-\mu c_0m_{i+1}^{j}\}
\end{equation}
Following the proof of \cite[Lemma A.5]{wu2024statistical}, we have
\begin{equation}
\biggl(\frac{j+k_0+1}{i+k_0}\biggr)^\gamma - 1 \leq (i+k_0)^{\gamma-1}\biggl(1+ (1-\gamma)m_{i}^{j}\biggr)^{\gamma/(1-\gamma)}
\end{equation}
Hence, we obtain
\begin{align}
\norm{S_i} &\leq c_0(i+k_0)^{\gamma-1}\sum_{j=i}^{n-2} \frac{1}{(j+k_0+1)^\gamma}\biggl(1+ (1-\gamma)m_{i}^{j}\biggr)^{\gamma/(1-\gamma)} \exp\{-\mu c_0m_{i+1}^{j}\}\\ &\leq
c_0(i+k_0)^{\gamma-1}\sum_{j=i}^{n-2} \frac{1}{(j+k_0)^\gamma}\biggl(1+ (1-\gamma)m_{i}^{j}\biggr)^{\gamma/(1-\gamma)}\exp\{\mu c_0(k_0+i)^{-\gamma}\} \exp\{-\mu c_0m_{i}^{j}\}\\&\leq
c_0\exp\{\frac{\mu c_0}{k_0^{\gamma}}\}(i+k_0)^{\gamma-1}\sum_{j=i}^{n-2} (m_i^j-m_i^{j-1})\biggl(1+ (1-\gamma)m_{i}^{j}\biggr)^{\gamma/(1-\gamma)} \exp\{-\mu c_0m_{i}^{j}\}\\ &\leq
2c_0\exp\{\frac{\mu c_0}{k_0^{\gamma}}\}(i+k_0)^{\gamma-1}\int_{0}^{+\infty}\biggl(1+ (1-\gamma)m\biggr)^{\gamma/(1-\gamma)} \exp\{-\mu c_0m\}\rmd m \\ &\leq
2c_0\exp\{\frac{\mu c_0}{k_0^{\gamma}}\}(i+k_0)^{\gamma-1}\biggl(2^{\gamma/(1-\gamma)}\frac{1}{\mu c_0} + (\frac{1}{\mu c_0})^{1/(1-\gamma)}\Gamma(\frac{1}{1-\gamma})\biggr)\eqsp.
\end{align}
Note that
\begin{align}
\norm{\sum_{i=1}^{n-1}G_{i:n-1}^{(\alpha)}}&\leq \sum_{i=1}^{n-1}\prod_{k=i}^{n-1}(1-\alpha_k\mu) = \sum_{i=1}^{n-1}\prod_{k=i}^{n-1}\alpha_{i-1}^{-1}\alpha_{i-1}(1-\alpha_k\mu) \\&\leq\frac{(k_0+n-2)^\gamma}{c_0\mu}\sum_{i=1}^{n-1}\biggl(\prod_{k=i}^{n-1}(1-\alpha_k\mu)-\prod_{k=i-1}^{n-1}(1-\alpha_k\mu)\biggr)\leq \frac{k_0^{\gamma}n^{\gamma}}{\mu c_0}\eqsp,
\end{align}
where in the last inequality we use that $(k_0+n-2)^\gamma \leq (k_0n)^{\gamma}$.
Bound for $\sum_{i=1}^{n-1}\norm{G_{i:n-1}^{(\alpha)}}^2$ is obtained similarly to $ \norm{\sum_{i=1}^{n-1}G_{i:n-1}^{(\alpha)}}$.
\end{proof}
To finish the proof of Lemma \ref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty} we need to bound $D_1, D_2$. By \eqref{repr:sumqtminusa} we obtain
\begin{align}
\normop{\frac{1}{n}\sum_{i=1}^{n-1} (Q_i - G^{-1})\noisecov G^{-\top}} &= \normop{-\frac{1}{n} G^{-1} \sum_{j=1}^{n-1} G_{1:j}^{(\alpha)}\noisecov G^{-\top}} \\ &= \normop{n^{-1} \Sigma_{\infty} \sum_{j=1}^{n-1} G_{1:j}^{(\alpha)}} \leq n^{-1} \normop{\Sigma_{\infty}} \cdot \normop{\sum_{j=1}^{n-1} G_{1:j}^{(\alpha)}} \eqsp.
\end{align}
It remains to apply \Cref{lem:bound_Q_i_and_Sigma_n} which gives
\begin{align}
\normop{\frac{1}{n}\sum_{i=1}^{n-1} (Q_i - G^{-1})\noisecov G^{-\top}} \leq \normopadapt{\Sigma_{\infty}}C_Q \frac{k_0^\gamma n^{\gamma-1}}{c_0}
\end{align}
Hence,
\begin{align}
\label{lemma:D1_bound}
\normop{D_1} \leq 2 \normopadapt{\Sigma_{\infty}}C_Q \frac{k_0^\gamma n^{\gamma-1}}{c_0}
\end{align}
To bound $D_2$ we use \eqref{repr:qtminusa} which gives
\begin{align}
\label{lemma:D2_expanded_repr}
&n^{-1}\sum_{i=1}^{n-1}(Q_i - G^{-1}) \noisecov (Q_i - G^{-1})^\top \\
&\qquad = n^{-1}\sum_{i=1}^{n-1}\bigl(S_i - G^{-1} \prod_{k=i}^{n-1} (\Id - \alpha_k G)\bigr) \noisecov \bigl(S_i - G^{-1} \prod_{k=i}^{n-1} (\Id - \alpha_k G)\bigr)^\top \\
&\qquad = \underbrace{n^{-1} \sum_{i=1}^{n-1} S_i \noisecov S_i^\top}_{D_{21}} + \underbrace{n^{-1}\sum_{i=1}^{n-1}G^{-1} \prod_{k=i}^{n-1} (\Id - \alpha_k G) \noisecov G^{-\top} \prod_{k=i}^{n-1} (\Id - \alpha_k G)^\top}_{D_{22}} \\
&\qquad\qquad \qquad -\underbrace{n^{-1} \sum_{i=1}^{n-1} G^{-1} \prod_{k=i}^{n-1} (\Id - \alpha_k G) \cdot \noisecov S_i^\top}_{D_{23}} - \underbrace{n^{-1}\sum_{i=1}^{n-1} S_i \noisecov G^{-\top} \prod_{k=i}^{n-1} (\Id - \alpha_k G)^{\top}}_{D_{24}}\eqsp.
\end{align}
To bound $D_{21}$ we use \Cref{bound_G_S}, and obtain
\begin{align}
\label{lemma:D21_bound}
\normop{D_{21}} = \normop{n^{-1} \sum_{i=1}^{n-1} S_i \noisecov S_i^\top} &\leq n^{-1} \sum_{i=1}^{n-1} \normop{\noisecov} \normop{S_i}^2 \\ &\leq
n^{-1}\normop{\noisecov} C_{S}^2\sum_{i=1}^{n-1} (i + k_0)^{2(\gamma-1)}\\ &\leq n^{-1}\normop{\noisecov} C_{S}^2 \frac{(n + k_0-1)^{2\gamma-1}-k_0^{2\gamma-1}}{2\gamma-1}
\\ &\leq \normop{\noisecov} C_{S}^2 \frac{n^{2(\gamma-1)}}{2\gamma-1}
\end{align}
The bound for $D_{22}$ follows from \Cref{bound_G_S}
\begin{align}
\label{lemma:D22_bound}
\normop{D_{22}} &= \normop{n^{-1}\sum_{i=1}^{n-1} \prod_{k=i}^{n-1} (\Id - \alpha_k G) G^{-1} \noisecov G^{-\top} \prod_{k=i}^{n-1} (\Id - \alpha_k G)^\top} \leq n^{-1} \normop{\Sigma_\infty} \sum_{i=1}^{n-1} \normop{G_{i:n-1}^{(\alpha)}}^2 \\
&\leq
n^{-1} \frac{\normop{\Sigma_\infty}}{ 2c_0\mu(n+k_0-2)^{-\gamma} - c_0^2\mu^2(n+k_0-2)^{-2\gamma}} \leq
\normop{\Sigma_\infty} k_0^{\gamma}\frac{n^{\gamma-1}}{c_0\mu}\eqsp.
\end{align}
Since $D_{23} = D_{24}^\top$, we concentrate on $\normop{D_{24}}$. \Cref{bound_G_S} immediately imply
\begin{align}
\normop{D_{24}} & \leq n^{-1} \normop{\noisecov G^{-\top}} \sum_{i=1}^{n-1} \normop{S_i} \normop{\prod_{k=i}^{n-1} (\Id - \alpha_k G)^{\top}} \\ &\leq n^{-1} \normop{\noisecov}\frac{1}{\mu}C_{S}\sum_{i=1}^{n-1}(i+k_0)^{\gamma-1}\prod_{k=i}^{n-1}(1-\mu\frac{c_0}{(k+k_0)^\gamma}) \\ &\leq n^{-1} \normop{\noisecov}\frac{1}{\mu}C_{S}\sum_{i=1}^{n-1}(i+k_0)^{2\gamma-1}(i+k_0)^{-\gamma}\prod_{k=i+1}^{n-1}(1-\mu\frac{c_0}{(k+k_0)^\gamma})\\ &\leq \normop{\noisecov}C_{S}k_0^{2\gamma-1}\frac{n^{2(\gamma-1)}}{\mu^2c_0}
\end{align}
Combining all inequalities above, we obtain
\begin{equation}
\label{eq:difference_sigma_n_infty_appendix}
\norm{\Sigma_n-\Sigma_{\infty}} \leq C_{\infty}'n^{\gamma-1}\eqsp,
\end{equation}
where
\begin{equation}
\label{eq:def_C_infty_prime}
C_{\infty}' = (\frac{k_0^{\gamma}}{c_0\mu} +2C_{Q} \frac{k_0^\gamma}{c_0} +1)\normopadapt{\Sigma_{\infty}} + (C_{S}^2 \frac{1}{2\gamma-1}+
C_{S}\frac{k_0^{2\gamma-1}}{\mu^2c_0})\normop{\noisecov}\eqsp.
\end{equation}
To finish the proof it remains to apply Lemma \ref{Pinsker}, since
\begin{equation}
\label{eq:def_C_infty}
3/2\|\Sigma_n^{-1/2}\Sigma_{\infty}\Sigma_n^{-1/2}-I\|_{\mathsf{F}}\leq C_{\infty}n^{\gamma-1}\eqsp, \text{ where } C_{\infty}=3/2 \sqrt{d} C_\Sigma^2 C_{\infty}'\eqsp.
\end{equation}
In the following computations we provide a lower bound on the quantity $\bigl| \frac{1}{n}\sum_{j=1}^{n-1}Q_{j}^2 - 1 \bigr|$, provided that the number of observations $n$ is large enough. For simplicity in this bound we consider $k_0 = 1$. We first note that
\begin{align}
\frac{1}{n}\sum_{j=1}^{n-1}Q_{j}^2 - 1 = \frac{1}{n}\sum_{j=1}^{n-1}(Q_j - 1)(Q_j + 1) - \frac{1}{n} = \frac{T_1}{n} + \frac{T_2}{n}\eqsp,
\end{align}
where
\[
T_1 = \sum_{j=1}^{n-1}(Q_j - 1)^2\eqsp, \quad T_2 = - 2\sum_{j=1}^{n-1}(Q_j - 1) - 1\eqsp,
\]
and treat the terms $T_1$ and $T_2$ separately. Using the identity \eqref{repr:sumqtminusa}, we get, since $G = 1$, that
\[
\sum_{j=1}^{n-1}(Q_j - 1) = -\sum_{j=1}^{n-1}\prod_{\ell=1}^{j}(1 - \alpha_{\ell})\eqsp.
\]
Hence, with \Cref{lem:bound_sum_exponent},
\[
\bigl| \sum_{i=1}^{n-1}(Q_i - 1) \bigr| \leq \frac{C_{Q}}{c_0}\eqsp.
\]
Hence, we can conclude that
\[
|T_2| \leq \left(\frac{2C_{Q}}{c_0} + 1\right)\eqsp,
\]
and proceed with $T_1$. Here we notice that, applying \eqref{repr:qtminusa},
\[
Q_i - 1 = S_i - \prod_{\ell=i}^{n-1}(1-\alpha_{\ell})\eqsp, \quad S_i = \sum_{j=i+1}^{n-1} (\alpha_i - \alpha_j) \prod_{\ell=i+1}^{j-1}(1 - \alpha_{\ell})\eqsp.
\]
Thus, the term $T_1$ can be represented as
\begin{equation}
\label{eq:lower_bound_decomposition}
T_1 = \sum_{j=1}^{n-1}S_j^2 - 2 \sum_{j=1}^{n-1} S_j \prod_{\ell=j}^{n-1}(1-\alpha_{\ell}) + \sum_{j=1}^{n-1} \prod_{\ell=j}^{n-1} (1-\alpha_{\ell})^2\eqsp.
\end{equation}
Due to item (a) from \Cref{bound_G_S}, it holds that $|S_j| \leq C_{S} / (j+1)^{1-\gamma}$. Hence, similarly to the proof of \Cref{lem:bound_kolmogorov_dist_sigma_n_sigma_infty} we can show that
\[
\frac{1}{n} |\sum_{j=1}^{n-1}S_j^2|\leq C_s^2n^{2(\gamma-1)}/(2\gamma-1)\eqsp,
\]
and
\[
\frac{1}{n} |\sum_{j=1}^{n-1} S_j \prod_{\ell=j}^{n-1}(1-\alpha_{\ell})| \leq C_S n^{2(\gamma-1)}/c_0\eqsp.
\]
Now, we proceed with the last term in \eqref{eq:lower_bound_decomposition}, and provide a lower bound on the last remaining component of $T_{1}$ in \eqref{eq:lower_bound_decomposition}, that is,
\[
T_{3} = \sum_{j=1}^{n-1} \prod_{\ell=j}^{n-1} (1-\alpha_{\ell})^2\eqsp.
\]
Since $\alpha_j = \frac{c_0}{(1+j)^{\gamma}}$, we get, using an elementary inequality $1 - x \geq \exp\{-2x\}$, valid for $0 \leq x \leq 1/2$, we get that
\begin{align}
\sum_{j=1}^{n-1} \prod_{\ell=j}^{n-1} (1-\alpha_{\ell})^2 &\geq \sum_{j=1}^{n-1} \exp\biggl\{ - \sum_{\ell=j}^{n-1}\frac{4c_0}{(1 + \ell)^{\gamma}} \biggr\} \\
&\geq \sum_{j=1}^{n-1} \exp\biggl\{ - \frac{4c_0}{1-\gamma}(n^{1-\gamma} - j^{1-\gamma}) \biggr\} \\
&=\exp\biggl\{ - \frac{4c_0}{1-\gamma}n^{1-\gamma}\biggr\}\sum_{j=1}^{n-1} \exp\biggl\{ \frac{4c_0}{1-\gamma} j^{1-\gamma} \biggr\}
\end{align}
Now we get that
\begin{align}
\sum_{j=1}^{n-1} \exp\biggl\{ \frac{4c_0}{1-\gamma} j^{1-\gamma} \biggr\}
&\geq \int_{0}^{n - 1} \exp\biggl\{ \frac{4c_0}{1 - \gamma} y^{1 - \gamma} \biggr\} \, \rmd y \\
&= (n-1) \int_{0}^{1}\exp\biggl\{ \frac{4c_0}{1 - \gamma} ((n-1)z)^{1 - \gamma} \biggr\} \,\rmd z\eqsp.
\end{align}
Now we proceed with Laplace approximation (see e.g. \cite{fedoryuk1977metod} or \cite{olver1997asymptotics}) for the inner integral:
\begin{align}
\int_{0}^{1}\exp\biggl\{ \frac{4c_0}{1 - \gamma} ((n-1)z)^{1 - \gamma} \biggr\} \,\rmd z = \exp\biggl\{ \frac{4c_0}{1 - \gamma} (n-1)^{1 - \gamma} \biggr\}\frac{(n-1)^{\gamma-1}}{4c_0}\bigl[1 + \mathcal{O}(n^{\gamma-1})\bigr]
\end{align}
Since $n^{1-\gamma}-(n-1)^{1-\gamma} \leq 1$ and $\frac{n-1}{n}\geq 1/2$ for $n \geq 2$, we get
\begin{align}
\frac{1}{n}\sum_{j=1}^{n-1} \prod_{\ell=j}^{n-1} (1-\alpha_{\ell})^2 \geq \frac{1}{4c_0} \exp\biggl\{-\frac{8c_0}{1-\gamma}\biggr\}\frac{1}{(n-1)^{1-\gamma}} + \mathcal{O}(n^{2(\gamma-1)})\eqsp.
\end{align}
Hence, we conclude that for $n$ large enough,
\[
|\sigma^2_{n,\gamma} - 1| > \frac{C_1(\gamma,c_0)}{n^{1-\gamma}}\eqsp,
\]
and the statement follows. To prove the second part, it remains to apply the lower bound on the total variation distance between Gaussian random vectors given in \cite[Theorem~1.1]{Devroye2018}.
\subsection{Numerical demonstration}
\label{sec:numerical_demonstartion}
In order to illustrate numerically the tightness of bounds provided in \Cref{prop:lower_bounds}, we consider the following simple experiment. We consider the statistics
\[
|\sigma^2_{n,\gamma}-1| \cdot n^{1-\gamma}\eqsp, \quad n \in \{2^{10}, \ldots, 2^{27}\}\eqsp.
\]
We illustrate numerically the tightness of our bound in the Figure \ref{fig:results-expe} below by calculating
\[
n^{1-\gamma} \cdot |\sigma^2_{n,\gamma}-1|
\]
for different values of $\gamma \in \{0.5,\ldots,0.9\}$ and $n$. Here we fix the values of parameter $k_0 = 1$ and $c_0 = 1$. Code to reproduce the plot is provided in \url{https://anonymous.4open.science/r/gaussian_approximation_sgd-5C8F}.
\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{kolmogorov_dist_rescaled.png}
\caption{Numerical verification of the lower bound given in \Cref{prop:lower_bounds}}
\label{fig:results-expe}
\end{figure}
To prove Theorem \ref{th:bound_kolmogorov_dist_pr_sigma_n}, we need a bound on the $2p$-th moment for $p\geq 1$ of the last iterate. Our approach is based on induction: first, we establish the result for $p = 2$, and then we show how to proceed from $2(p - 1)$ to $2p$.
\begin{lemma}
\label{lem:bound_last_iter_second_moment}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($2$), and \Cref{ass:step_size}. Then for any $k \in \nset$ it holds that
\begin{equation}
\label{eq:bound_last_iter_second_moment}
\PE[\norm{\theta_k - \thetas}^{2}] \leq C_{1} \exp\biggl\{-\frac{\mu c_0}{4} (k+k_0)^{1-\gamma}\biggr\} \bigl[ \norm{\theta_0-\thetas}^2 + \sigma_2^2\bigr] + C_{2} \sigma_2^2 \alpha_{k} \eqsp,
\end{equation}
where $\sigma_2^2$ is defined in \Cref{ass:noise_decomposition}($2$), and the constants $C_1$ and $C_2$ are given by
\begin{equation}
\label{eq:const_C_1_C_2_def}
\begin{split}
C_{1} &= \exp\biggl\{\frac{3\mu c_0}{4(1-\gamma)}k_0^{1-\gamma}\biggr\}\biggl(\left(1 + L_2^{-2}\right) \exp\biggl\{\frac{6c_0^2L_2^2}{2\gamma-1}\biggr\} + \frac{2c_{0}^2}{2\gamma-1}\biggr) \eqsp,\\
C_{2} &= \frac{2^{1+\gamma}}{\mu}\eqsp.
\end{split}
\end{equation}
\end{lemma}
\begin{proof}
From \eqref{eq:sgd_recursion_main} and \cref{ass:noise_decomposition} it follows that
\begin{equation}
\norm{\theta_k-\thetas}^2 = \norm{\theta_{k-1}-\thetas}^2-2\alpha_k\langle \theta_{k-1}-\thetas , \nabla f(\theta_{k-1})+\zeta_k\rangle + \alpha_k^2\norm{\nabla f(\theta_{k-1})+\zeta_k}^2.
\end{equation}
Using \Cref{ass:L-smooth} and \Cref{ass:noise_decomposition}($2$), we obtain
\begin{align}
2\alpha_k\PE[\langle \theta_{k-1}-\thetas , \nabla f(\theta_{k-1})+\zeta_k\rangle|\mathcal{F}_{k-1}]
&= 2\alpha_k\langle \theta_{k-1}-\thetas , \nabla f(\theta_{k-1})-\nabla f(\thetas) \rangle.
\end{align}
Using \Cref{ass:noise_decomposition}($2$) and \Cref{ass:L-smooth}, we get
\begin{align}
\PE[\norm{\nabla f(\theta_{k-1})+\zeta_k}^2|\mathcal{F}_{k-1}]
&= \norm{\nabla f(\theta_{k-1})-\nabla f(\thetas)}^2 + \PE[\norm{\eta(\xi_k) + g(\theta_{k-1}, \xi_k)}^2|\mathcal{F}_{k-1}] \\ &\leq L_1 \pscal{\nabla f(\theta_{k-1}) - \nabla f(\thetas)}{\theta_{k-1}- \thetas} + 2L_2^2 \norm{\theta_{k-1}-\thetas}^2 + 2\sigma_2^2\eqsp.
\end{align}
Combining the above inequalities, we obtain
\begin{equation}
\label{eq:recursion_second_moment}
\PE [\norm{\theta_k-\thetas}^2]\leq (1-\mu \alpha_k (2 - \alpha_k L_1) +2 \alpha_k^2 L_2^2))\PE[\norm{\theta_{k-1}-\thetas}^2]+2\alpha_k^2\sigma_2^2.
\end{equation}
By applying the recurrence \eqref{eq:recursion_second_moment}, we obtain that
\begin{align}
\PE[\norm{\theta_k-\thetas}^2]
\leq A_{1,k} \norm{\theta_0-\thetas}^2 + 2 \sigma_2^2 A_{2,k}\eqsp,
\end{align}
where we have set
\begin{equation}
\label{eq:A_1_2_k_def}
\begin{split}
A_{1,k} &= \prod_{i=1}^k(1- (3/2)\alpha_i\mu+2 \alpha_i^2 L_2^2)\eqsp, \\
A_{2,k} &= \sum_{i=1}^k\prod_{j=i+1}^k(1-(3/2) \alpha_j\mu+2\alpha_j^2 L_2^2)\alpha_i^2\eqsp.
\end{split}
\end{equation}
Using the elementary bound $1+t \leq \rme^{t}$ for any $t\in \rset$, we get
\begin{equation}
A_{1,k} \leq \exp\biggl\{-(3/2) \mu\sum_{i=1}^k\alpha_i \biggr\}\exp\biggl\{2L_2^2\sum_{i=1}^k\alpha_i^2\biggr\}\eqsp.
\end{equation}
Using \Cref{lem:bounds_on_sum_step_sizes}, we obtain
\begin{equation}
\label{eq:A_1_k_bound}
A_{1,k} \leq c_{1} \exp\biggl\{-\frac{3\mu c_0}{4(1-\gamma)} (k+k_0)^{1-\gamma}\biggr\}\eqsp,
\end{equation}
where we have set
\begin{equation}
\label{eq:c_1_small}
c_1 = \exp\biggl\{\frac{2c_0^2L_2^2}{2\gamma-1} + \frac{3\mu c_0}{4(1-\gamma)}k_0^{1-\gamma}\biggr\}\eqsp.
\end{equation}
Now we estimate $A_{2,k}$. Let $k_1$ be the largest index $k$ such that $4 \alpha_k^2 L_2^2 \geq \alpha_k\mu$. Then, for $i > k_1$, we have that
\[
1- (3/2) \alpha_i \mu+2 \alpha_i^2 L_2^2 \leq 1 - \alpha_i \mu \eqsp.
\]
Thus, using the definition of $A_{2,k}$ in \eqref{eq:A_1_2_k_def}, we obtain that
\begin{align}
A_{2,k} \leq \sum_{i=1}^k \alpha_i^2 \prod_{j=i+1}^k(1-\alpha_j\mu) + \sum_{i=1}^{k_1}\alpha_i^2 \biggl\{ \prod_{j=i+1}^{k_1}(1+2\alpha_j^2 L_2) \biggr\} \biggl\{ \prod_{j=k_1+1}^{k}(1-\alpha_j\mu) \biggr\}\eqsp.
\end{align}
Note that
\begin{align}
\sum_{i=1}^{k_1}\alpha_i^2 \prod_{j=i+1}^{k_1}(1+2 \alpha_j^2L_2^2)
&=\frac{1}{ 2L_2^2}\sum_{i=1}^{k_1}\biggl(\prod_{j=i}^{k_1}(1+2 \alpha_j^2 L_2^2)-\prod_{j=i+1}^{k_1}(1+2 \alpha_j^2L_2^2)\biggr) \\
&\leq \frac{1}{2L_2^2}\prod_{j=1}^{k_1}(1+2\alpha_j^2 L_2^2) \leq \frac{1}{2L_2^2}\exp\biggl\{2L_2^2\sum_{j=1}^{k_1}\alpha_j^2\biggr\}.
\end{align}
Note, that for $k\leq k_1$, $\alpha_k \geq \mu/(4L_2^2)$, hence, we have
\begin{equation}
\prod_{j=k_1+1}^k(1-\alpha_j\mu) \leq \exp\biggl\{-\mu\sum_{i=1}^k\alpha_i\biggr\}\exp\biggl\{\mu\sum_{i=1}^{k_1}\alpha_i\biggr\} \leq \exp\biggl\{-\mu\sum_{i=1}^k\alpha_i\biggr\}\exp\biggl\{4L_2^2 \sum_{i=1}^{k_1}\alpha_i^2\biggr\}\eqsp.
\end{equation}
Moreover, for any $m\in\{1,\ldots ,k\}$, we obtain
\begin{align}
&\sum_{i=1}^k \alpha_i^2 \prod_{j=i+1}^k(1-\alpha_j\mu) = \sum_{i=1}^m\prod_{j=i+1}^k(1-\alpha_j\mu)\alpha_i^2 + \sum_{i=m+1}^k\prod_{j=i+1}^k(1-\alpha_j\mu)\alpha_i^2 \\
&\qquad \qquad \leq \prod_{j=m+1}^k(1-\alpha_j\mu)\sum_{i=1}^{m}\alpha_i^2 +\alpha_m\sum_{i=m+1}^k\prod_{j=i+1}^k(1-\alpha_j\mu)\alpha_i \\
&\qquad \qquad \leq \exp\biggl\{-\mu\sum_{j=m+1}^k\alpha_j\biggr\}\sum_{i=1}^{m}\alpha_i^2 + \frac{\alpha_m}{\mu}\sum_{i=m+1}^k\biggl(\prod_{j=i+1}^k(1-\alpha_j\mu)-\prod_{j=i}^k(1-\alpha_j\mu)\biggr) \\
&\qquad \qquad \leq \exp\biggl\{-\mu\sum_{j=m+1}^k\alpha_j\biggr\}\sum_{i=1}^{m}\alpha_i^2 + \frac{\alpha_m}{\mu}\biggl(1-\prod_{j=m+1}^k(1-\alpha_j\mu)\biggr) \\
&\qquad \qquad \leq \exp\biggl\{-\mu\sum_{j=m+1}^k\alpha_j\biggr\}\sum_{i=1}^{m}\alpha_i^2 + \frac{\alpha_m}{\mu}\eqsp.
\end{align}
Thus, setting $m= \lfloor k/2 \rfloor $, and using the definition of $A_{2,k}$ in \eqref{eq:A_1_2_k_def}, we obtain that
\begin{align}
A_{2,k} &\leq \exp\biggl\{\frac{-\mu c_0}{2(1-\gamma)}((k+k_0)^{1-\gamma}-(\lfloor k/2\rfloor+k_0)^{1-\gamma}))\biggr\} \frac{c_0^2}{2\gamma-1} + \frac{c_0}{\mu(k_0+\lfloor k/2 \rfloor)^{\gamma}} \\& \qquad \qquad \qquad \qquad+ c_2 \exp\biggl\{-\frac{\mu c_0}{2(1-\gamma)}(k+k_0)^{1-\gamma}\biggr\}\eqsp,
\end{align}
where we have set
\begin{equation}
\label{eq:c_2_small}
c_2 = \frac{1}{2L_2^2} \exp\biggl\{\frac{6c_0^2 L_2^2}{2\gamma-1}+ \frac{\mu c_0}{2(1-\gamma)}k_0^{1-\gamma}\biggr\}\eqsp.
\end{equation}
Using that $\lfloor k/2 \rfloor \leq k/2$ together with the elementary inequality
\[
\frac{x^{\beta}}{\beta} - \frac{(x/2)^{\beta}}{\beta} \geq \frac{x^{\beta}}{2}\eqsp,
\]
which is valid for $\beta \in (0,1]$, and $\frac{c_0}{\mu(k_0+\lfloor k/2 \rfloor)^{\gamma}} \leq \frac{2^{\gamma} c_0}{\mu (k+k_0)^{\gamma}}$, we obtain that
\begin{align}
A_{2,k} &\leq \exp\biggl\{-\frac{\mu c_0}{4} (k+k_0)^{1-\gamma}\biggr\}\exp\biggl\{\frac{\mu c_0}{2(1-\gamma)}k_0^{1-\gamma}\biggr\}\frac{c_0^2}{2\gamma-1} + \frac{2^{\gamma} c_0}{\mu (k+k_0)^{\gamma}} \\& \qquad \qquad \qquad \qquad + c_2 \exp\biggl\{\frac{-\mu c_0}{2(1-\gamma)}(k+k_0)^{1-\gamma}\biggr\}\eqsp.
\end{align}
Combining the bounds for $A_{1,k}$ and $A_{2,k}$, we obtain that
\begin{align}
\PE[\norm{\theta_k-\thetas}^2]
&\leq c_{1} \exp\biggl\{-\frac{\mu c_0}{(1-\gamma)} (k+k_0)^{1-\gamma}\biggr\} \norm{\theta_0 - \thetas}^2 \\
&+ \exp\biggl\{-\frac{\mu c_0}{4} (k+k_0)^{1-\gamma}\biggr\} \frac{2c_0^2 \sigma_2^2}{2\gamma-1}\exp\biggl\{\frac{\mu c_0}{2(1-\gamma)}k_0^{1-\gamma}\biggr\}+ \frac{2^{1+\gamma} c_0 \sigma_2^2}{\mu (k+k_0)^{\gamma}} \\
&+ 2 c_2 \sigma_2^2 \exp\biggl\{\frac{-\mu c_0}{2(1-\gamma)}(k+k_0)^{1-\gamma}\biggr\} \\
&\leq C_{1} \exp\biggl\{-\frac{\mu c_0}{4} (k+k_0)^{1-\gamma}\biggr\} \bigl[ \norm{\theta_0-\thetas}^2 + \sigma_2^2\bigr] + C_{2} \alpha_{k} \eqsp,
\end{align}
where we have set constants $C_1$ and $C_2$ using the definitions of $c_1$ and $c_2$ from \eqref{eq:c_1_small} and \eqref{eq:c_2_small}.
\end{proof}
The first term in the bound from \Cref{lem:bound_last_iter_second_moment} decays exponentially with $k$, which implies that $\mathbb{E}^{1/2}[\|\theta_k - \theta^*\|^2] \lesssim \alpha_k$. Below, we state this result with an explicit constant, as this specific form of the bound for the last iteration is needed for the induction step.
\begin{corollary}
\label{cor:second_moment_bound}
Under the assumptions of \Cref{lem:bound_last_iter_second_moment}, it holds that
\begin{equation}
\PE[\norm{\theta_k-\thetas}^2]\leq D_1 (\norm{\theta_0-\thetas}^2 + \sigma_2^2) \alpha_k\eqsp,
\end{equation}
where
\[
D_1 = C_1 (1/c_0 + C_{2}) \biggl(\frac{4\gamma}{(1-\gamma)\mu c_0 \rme}\biggr)^{\gamma/(1-\gamma)}\eqsp.
\]
\end{corollary}
\begin{proof}
Define $C_3 = (\frac{4\gamma}{(1-\gamma)\mu c_0 \rme})^{\gamma/(1-\gamma)} > 1$, then $\exp\{-\mu c_0(k+k_0)^{1-\gamma} / 4\}\leq C_3 (k+k_0)^{-\gamma}$, and the statement follows.
\end{proof}
Now we provide bound for p-moment of last iterate.
\begin{proposition} \label{prop:2p-moment-bound}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($2p$), and \Cref{ass:step_size}. Then for any $k \in \nset$ it holds that
\begin{equation}
\PE[\norm{\theta_k - \thetas}^{2p}] \leq C_{2p,1}\exp\biggl\{-\frac{p\mu c_0}{4}(k+k_0)^{1-\gamma}\biggr\}(\norm{\theta_0-\thetas}^{2p} + \sigma_{2p}^{2p}) + C_{2p,2}\sigma_{2p}^{2p}\alpha_k^p\eqsp,
\end{equation}
where
\begin{equation}
C_{2p,1} = 2^{2p-1}(D_{2(p-1)}C_4^pc_0^p + 1)c_4\eqsp,
\end{equation}
\begin{equation}
C_{2p,2} =2^{2p-1}D_{2(p-1)}C_4^p\frac{2^{1+\gamma p}}{\mu pc_0} \eqsp,
\end{equation}
constants $D_{2(p-1)}$ are defined in \eqref{eq:const_D_2p_def}, and
\begin{equation}
\begin{split}
&C_4^p = (4c_0^{1/2}2^{\gamma/2}+2^{\gamma}+ 4c_0)^p\\
&c_4 = \biggl(\exp\biggl\{\exp\biggl\{5pc_0(L_1+L_2)\biggr\}\frac{4p^2(L_1+L_2)^2}{2\gamma-1}\biggr\} + 1\biggr)\exp\biggl\{\frac{p\mu c_0}{1-\gamma}k_0^{1-\gamma}\biggr\}\frac{1}{\gamma(p+1)-1}
\end{split}
\end{equation}
\end{proposition}
\begin{proof}
We prove the statement by induction in $p$. We first assume that $\theta_0 = \thetas$ and then provide a result for arbitrary initial condition. The result for $p = 1$ is provided in \Cref{cor:second_moment_bound} . Assume that for any $t\leq p-1$ and all $k \in \nset$ we proved that
\begin{equation}
\label{eq:induction_assumption_p_moment}
\PE[\norm{\theta_k - \thetas}^{2t}] \leq D_{2t} \sigma_{2t}^{2t} \alpha_{k}^{t}\eqsp,
\end{equation}
and the sequence of constants $\{D_{2t}\}$ is non-decreasing in $t$. Inequality \eqref{eq:induction_assumption_p_moment} implies that, since $\sigma_{2t} \leq \sigma_{2p}$ for $t \leq p-1$,
\begin{equation}
\PE[\norm{\theta_k - \thetas}^{2t}] \leq D_{2t} \sigma_{2p}^{2t} \alpha_{k}^{t}\eqsp.
\end{equation}
For any $k\in \nset$ we denote $\delta_k= \norm{\theta_k - \thetas}$. Using \eqref{eq:sgd_recursion_main}, we get
\begin{align}
\delta_k^{2p}
&= \left(\delta_{k-1}^2 - 2\alpha_k\langle\theta_{k-1}-\thetas, \nabla f(\theta_{k-1}) + \zeta_k \rangle + \alpha_k^2\norm{\nabla f(\theta_{k-1}) + \zeta_k}^2\right)^p \\
&= \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}}}}{\sum}\frac{p!}{i!j!l!}\delta_{k-1}^{2i}(-2\alpha_k\langle\theta_{k-1}-\thetas, \nabla f(\theta_{k-1})+\zeta_k\rangle)^j\alpha_k^{2l}\norm{\nabla f(\theta_{k-1}) + \zeta_k}^{2l}.
\end{align}
Now we bound each term in the sum above.
\begin{enumerate}
\item First, for $i=p, j=0, l=0$, the corresponding term in the sum equals $\delta_{k-1}^{2p}$.
\item Second, for $i=p-1, j=1, l=0$, we obtain, applying \Cref{ass:L-smooth}, that
\begin{align}
2p\alpha_k \PE [\langle\theta_{k-1}-\thetas, \nabla f(\theta_{k-1}) +\zeta_k \rangle\delta_{k-1}^{2(p-1)}|\F_{k-1}]
&= 2p\alpha_k\langle\theta_{k-1}-\thetas, \nabla f(\theta_{k-1})- \nabla f(\thetas) \rangle\delta_{k-1}^{2(p-1)} \\
&\geq 2 p \mu \alpha_k \delta_{k-1}^{2 p}\eqsp.
\end{align}
\item Third, for $l \geq 1$ or $j \geq 2$ (that is, $2l+j \geq 2$), we use Cauchy-Schwartz inequality
\begin{equation}
|\langle\theta_{k-1}-\thetas, \nabla f(\theta_{k-1})+\zeta_k\rangle)^j| \leq \norm{\theta_{k-1}-\thetas}^j\norm{\nabla f(\theta_{k-1})+\zeta_k}^j\eqsp,
\end{equation}
moreover, applying \Cref{ass:L-smooth} and \Cref{ass:noise_decomposition}($2p$) together with the Lyapunov inequality, we get
\begin{align}
\PE[\norm{\nabla f(\theta_{k-1})+\zeta_k}^{2l+j}|\F_{k-1}]
&= \PE[\norm{\nabla f(\theta_{k-1})+g(\theta_{k-1}, \xi_k) + \eta(\xi_k)}^{2l+j}|\F_{k-1}] \\
&\leq 2^{2l+j-1}((L_1+L_2)^{2l+j}\delta_{k-1}^{2l+j} + \sigma_{2p}^{2l+j})\eqsp.
\end{align}
\end{enumerate}
Combining inequalities above, we get
\begin{multline}
\label{eq:delta_k_2p_decomposition}
\PE[\delta_k^{2p}|\F_{k-1}]\leq \biggl(1 -2p\mu\alpha_k + \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}: \\j+2l\geq 2 }}} {\sum}\frac{p!}{i!j!l!}\alpha_k^{j+2l}2^{2l+2j-1}(L_1+L_2)^{2l+j}\biggr) \delta_{k-1}^{2p} \\ + \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}: \\j+2l\geq 2 }}} {\sum}\frac{p!}{i!j!l!}\delta_{k-1}^{2i+j}\alpha_k^{j+2l}2^{2l+2j-1}\sigma_{2p}^{2l+j}\eqsp.
\end{multline}
Consider the first term above, and note that
\begin{align}
\label{eq:bound_sum_j_2l_p_moment}
&\qquad \qquad\qquad \qquad \qquad \qquad\underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}: \\j+2l\geq 2 }}} {\sum}\frac{p!}{i!j!l!}\alpha_k^{j+2l}2^{2l+2j-1}(L_1+L_2)^{2l+j}
\\
&\leq 2 \alpha_k^2 (L_1+L_2)^2 \biggl(\underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}: \\l\geq 1 }}} {\sum}\frac{p!}{i!j!l!}(4\alpha_k(L_1+L_2))^j(4\alpha_k^2(L_1+L_2)^2)^{l-1} + \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\};\\l=0;j\geq2 }}} {\sum}\frac{p!}{i!j!}(4\alpha_k(L_1+L_2))^{j-2}\biggr) \\& \qquad \qquad\qquad \qquad \qquad \leq 2 p^2\alpha_k^2 (L_1+L_2)^2(1+5\alpha_k(L_1+L_2))^{p} \eqsp.
\end{align}
% Consider now the second term in \eqref{eq:delta_k_2p_decomposition}. Note that $2 (\alpha_k\sigma_{2p}\delta_{k-1})^j\leq (\alpha_k\sigma_{2p})^{2j} + \delta_{k-1}^{2j}$.
Hence,
\begin{align}
\PE[\delta_k^{2p}] \leq \left(1 -2p\mu\alpha_k + 2 p^2\alpha_k^2 (L_1+L_2)^2(1+5\alpha_k(L_1+L_2))^{p} \right)\delta_{k-1}^{2p} + T_1\eqsp,
\end{align}
where we have defined
\begin{align}
T_1 &= \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}; \\j+2l\geq 2;\\i+j=p }}} {\sum}\frac{p!}{i!j!l!} \PE[\delta_{k-1}^{2i+j}] \alpha_k^{j+2l}2^{2l+2j-1}\sigma_{2p}^{2l+j}\eqsp.
% T_2 &= \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}; \\j+2l\geq 2;\\ i+j\leq p-1 }}} {\sum}\frac{p!}{i!j!l!}\PE[\delta_{k-1}^{2i}] \alpha_k^{2l+2j}2^{2l+2j-2}\sigma_{2p}^{2l+2j} \eqsp, \\
% T_3 &= \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}; \\j+2l\geq 2;\\ i+j \leq p-1 }}} {\sum}\frac{p!}{i!j!l!} \PE[\delta_{k-1}^{2i+2j}] \alpha_k^{2l}2^{2l+2j-2}\sigma_{2p}^{2l}\eqsp.
\end{align}
For the last term we apply H\"older's inequality together with induction assumption \eqref{eq:induction_assumption_p_moment} and $(k+k_0-1)^{-\gamma}\leq 2^{\gamma}(k+k_0)^{-\gamma}$ and obtain
\begin{align}
T_1 &\leq \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}; \\j+2l\geq 2}}} {\sum}\frac{p!}{i!j!l!}\PE^{1/2}[\delta_{k-1}^{2(i+\lfloor j/2 \rfloor)}]\PE^{1/2}[\delta_{k-1}^{2(i+\lceil j/2 \rceil)}]\alpha_k^{j+2l}2^{2l+2j-1}\sigma_{2p}^{2l+j} \\
& \leq D_{2(p-1)}\frac{(4c_0^{1/2}2^{\gamma/2}+2^{\gamma}+ 4c_0)^p}{2}c_0^p\sigma_{2p}^{2p}(k+k_0)^{-\gamma (p+1)}\eqsp.
\end{align}
% Similarly, for $T_2$ and $T_3$ we obtain
% \begin{align}
% &T_2 \leq D_{2(p-1)}\frac{(8\alpha_0+2^{\gamma})^p}{4}\alpha_0^p\sigma_{2p}^{2p}k^{-\gamma (p+1)}\eqsp, \\
% &T_3 \leq D_{2(p-1)}\frac{(4\alpha_0+5\cdot 2^{\gamma})^p}{4}\alpha_0^p\sigma_{2p}^{2p}k^{-\gamma (p+1)}\eqsp.
% \end{align}
Hence, combining the above bounds, we obtain that
\begin{equation}
\label{eq:p_moment_last_iterate_requrence}
\PE[\delta_k^{2p}]\leq (1 - 2p\mu\alpha_k + 16 \alpha_k^2 (L_1+L_2)^2 3^{p})\PE[\delta_{k-1}^{2p}] + D_{2(p-1)} C_{4}^{p} c_0^p\sigma_{2p}^{2p}k^{-\gamma (p+1)}\eqsp,
\end{equation}
where we have defined
\begin{equation}
\label{eq:C_4_const_def}
C_4^p = (4c_0^{1/2}2^{\gamma/2}+2^{\gamma}+ 4c_0)^p\eqsp.
\end{equation}
Note that
\begin{equation}
1 - 2p\mu\alpha_k + 2 p^2\alpha_k^2 (L_1+L_2)^2(1+5\alpha_k(L_1+L_2))^{p} > 1-2p\mu\alpha_k + \alpha_k^2\mu^2p^2 \geq 0\eqsp.
\end{equation}
Unrolling the recurrence \eqref{eq:p_moment_last_iterate_requrence}, we get
\begin{equation}
\PE[\delta_k^{2p}] \leq A_{2,k}'D_{2(p-1)}C_4^pc_0^p\sigma_{2p}^{2p},
\end{equation}
where we have set
\begin{equation}
\label{eq:A_2_k'_def}
A_{2,k}' = \sum_{t=1}^k\prod_{i=t+1}^k(1 - 2p\mu\alpha_i + 2 p^2\alpha_i^2 (L_1+L_2)^2(1+5\alpha_i(L_1+L_2))^{p})(t+k_0)^{-\gamma(p+1)}\eqsp.
\end{equation}
For simplicity, we define $C_{5}=2p^2(L_1+L_2)^2$. Let $k_1$ is the largest $k$ such that $\alpha_k^2C_5(1+5\alpha_i(L_1+L_2))^{p} \geq p\mu\alpha_k$. Then, for $i > k_1$, we have
\begin{equation}
1 - 2p\mu\alpha_i + C_5 \alpha_i^2 (1+5\alpha_i(L_1+L_2))^{p}\leq 1-p\mu\alpha_i \eqsp.
\end{equation}
Hence, using the definition of $A_{2,k}'$ in \eqref{eq:A_2_k'_def}, we get
\begin{align}
\begin{aligned}
A'_{2,k}&=\sum_{t=k_1+1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(t+k_0)^{-\gamma(p+1)} \\ &\qquad\qquad+ \prod_{t=k_1+1}^{k}\exp\biggl\{-p\mu\alpha_t\biggr\}\sum_{t=1}^{k_1}\prod_{i=t+1}^{k_1}\exp\biggl\{C_{5}\alpha_i^2(1+5\alpha_i(L_1+L_2))^{p}\biggr\}(t+k_0)^{-\gamma(p+1)}\\ &\leq
\sum_{t=1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(t+k_0)^{-\gamma(p+1)} \\&\qquad\qquad + \prod_{t=1}^{k}\exp\biggl\{-p\mu\alpha_t\biggr\}\prod_{t=1}^{k_1}\exp\{p\mu\alpha_t\}\prod_{i=1}^{k_1}\exp\biggl\{C_{5}\alpha_i^2(1+5\alpha_i(L_1+L_2))^{p}\biggr\}\sum_{t=1}^{k_1}(t+k_0)^{-\gamma(p+1)} \\ &\leq
\sum_{t=1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(t+k_0)^{-\gamma(p+1)} \\&\qquad\qquad+ \prod_{i=1}^{k_1}\exp\biggl\{2C_{5}\alpha_i^2(1+5\alpha_i(L_1+L_2))^{p}\biggr\}\prod_{t=1}^{k}\exp\biggl\{-p\mu\alpha_t\biggr\}\sum_{t=1}^{k_1}(t+k_0)^{-\gamma(p+1)}
\end{aligned}
\end{align}
For any $m\in\{1, \ldots k\}$ we have
\begin{align}
\begin{aligned}
&\sum_{t=1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(t+k_0)^{-\gamma(p+1)} \\&= \sum_{t=1}^{m}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(t+k_0)^{-\gamma(p+1)} + \sum_{t=m+1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(t+k_0)^{-\gamma(p+1)} \\&\leq
\prod_{i=m+1}^k\exp\{-p\mu\alpha_i\}\sum_{t=1}^{m}(t+k_0)^{-\gamma(p+1)} + \sum_{t=m+1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(m+k_0)^{-\gamma p}(t+k_0)^{-\gamma} \\&\leq
\prod_{i=m+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}\sum_{t=1}^{k}(t+k_0)^{-\gamma(p+1)} + (m+k_0)^{-\gamma p}\sum_{t=1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}(t+k_0)^{-\gamma}
\end{aligned}
\end{align}
Applying \Cref{lem:bounds_on_sum_step_sizes}\ref{eq:simple_bound_sum_alpha_k}, we have
\begin{align}
\begin{aligned}
&\sum_{t=1}^{k}\prod_{i=t+1}^k\exp\biggl\{-p\mu\alpha_i\biggr\}t^{-\gamma}\leq \sum_{t=1}^k\exp\biggl\{\frac{-p\mu c_0}{2(1-\gamma)}((k+k_0)^{1-\gamma}-(t+k_0)^{1-\gamma})\biggr\}(t+k_0)^{-\gamma}\\&\leq \exp\biggl\{\frac{-p\mu c_0}{2(1-\gamma)}(k+k_0)^{1-\gamma}\biggr\}\frac{2}{p\mu c_0}\int_{0}^{\frac{p\mu c_0}{2(1-\gamma)}(k+k_0)^{1-\gamma}}e^udu \leq \frac{2}{p\mu c_0}\eqsp.
\end{aligned}
\end{align}
Applying \Cref{lem:bounds_on_sum_step_sizes}\ref{eq:sum_alpha_k_p}, we get
\[
\sum_{i=1}^{k}(i+k_0)^{-\gamma(p+1)} \leq \frac{1}{(p+1)\gamma-1}\eqsp,
\]
and
\[
\sum_{i=1}^k2C_{5}\alpha_i^2(1+5\alpha_i(L_1+L_2))^{p} \leq 2C_5(1+5c_0(L_1+L_2))^p\sum_{i=1}^{+\infty}\alpha_k^2\leq \exp\biggl\{5pc_0(L_1+L_2)\biggr\}\frac{2C_5}{2\gamma-1}
\]
Substituting $m=\lfloor k/2\rfloor$ and applying \eqref{eq:simple_bound_sum_alpha_k}, we get
\begin{align}
&A_{2,k}' \leq \exp\biggl\{-\frac{p\mu c_0}{2(1-\gamma)}((k+k_0)^{1-\gamma}-(\lfloor k/2\rfloor+k_0)^{1-\gamma})\biggr\}\frac{1}{\gamma(p+1)-1} + \frac{2(\lfloor k/2\rfloor +k_0)^{-\gamma p}}{p\mu c_0}\\& + c_3\exp\biggl\{-\frac{p\mu c_0}{2(1-\gamma)}(k+k_0)^{1-\gamma}\biggr\},
\end{align}
where we have set
\begin{equation}
c_3 = \exp\biggl\{\exp\biggl\{5pc_0(L_1+L_2)\biggr\}\frac{2C_5}{2\gamma-1} +\frac{p\mu c_0}{2(1-\gamma)}k_0^{1-\gamma}\biggr \}\frac{1}{\gamma(p+1)-1}.
\end{equation}
Using that $\lfloor k/2 \rfloor \leq k/2$ together with the elementary inequality
\[
\frac{x^{\beta}}{\beta} - \frac{(x/2)^{\beta}}{\beta} \geq \frac{x^{\beta}}{2}\eqsp,
\]
which is valid for $\beta \in (0,1]$, and $\frac{2}{\mu pc_0 (\lfloor k/2 \rfloor + k_0)^{\gamma p}} \leq \frac{2^{1+\gamma p}}{\mu pc_0(k+k_0)^{\gamma p}}$, we obtain that
\begin{align}
A_{2,k}' &\leq \exp\biggl\{-\frac{p\mu c_0}{4}(k+k_0)^{1-\gamma}\biggr\}\exp\biggl\{\frac{p\mu c_0}{2(1-\gamma)}k_0^{1-\gamma}\biggr\}\frac{1}{\gamma(p+1)-1} \\& + \frac{2^{1+\gamma p}}{\mu pc_0 (k+k_0)^{\gamma p}}+ c_3\exp\biggl\{-\frac{p\mu c_0}{2(1-\gamma)}(k+k_0)^{1-\gamma}\biggr\} \\& \leq c_4\exp\biggl\{-\frac{p\mu c_0}{4}(k+k_0)^{1-\gamma}\biggr\} + c_5(k+k_0)^{-\gamma p},
\end{align}
where we have set
\begin{align}
&c_4 = \biggl(\exp\biggl\{\exp\biggl\{5pc_0(L_1+L_2)\biggr\}\frac{4p^2(L_1+L_2)^2}{2\gamma-1}\biggr\} + 1\biggr)\exp\biggl\{\frac{p\mu c_0}{1-\gamma}k_0^{1-\gamma}\biggr\}\frac{1}{\gamma(p+1)-1} \\&
c_5 = \frac{2^{1+\gamma p}}{\mu pc_0}
\end{align}
Finally, we get
\begin{equation}
\PE[\delta_k^{2p}]\leq C_{2p, 1}'\exp\biggl\{-\frac{p\mu c_0}{4}(k+k_0)^{1-\gamma}\biggr\}\sigma_{2p}^{2p} + C_{2p, 2}'\sigma_{2p}^{2p}\alpha_k^p,
\end{equation}
where
\begin{align}
&C_{2p, 1}' = D_{2(p-1)}C_4^pc_0^pc_4\\&
C_{2p, 2}' = D_{2(p-1)}C_4^pc_5.
\end{align}
To provide the result for arbitrary start point $\theta_0=\theta$ we consider the synchronous coupling construction defined by the recursions
\begin{align}
\label{eq:def_coupling_recursions}
&\theta_k = \theta_{k-1} - \alpha_k(\nabla f(\theta_{k-1})+g(\theta_{k-1}, \xi_k) + \eta(\xi_k)), \qquad \theta_0=\theta\\&
\theta_k' = \theta_{k-1}' - \alpha_k(\nabla f(\theta_{k-1}')+g(\theta_{k-1}', \xi_k) + \eta(\xi_k)), \qquad \theta_0'=\thetas
\end{align}
For any $k\in \nset$ we denote $\delta_{k}' = \norm{\theta_k - \theta_k'}$. Using \eqref{eq:def_coupling_recursions} together with \Cref{ass:L-smooth} and \Cref{ass:noise_decomposition}($2p$), we get
\begin{align}
&\delta_k'^{2p} = (\delta_{k-1}'^2 - 2\alpha_k\langle \theta_{k-1} - \theta_{k-1}', \nabla f(\theta_{k-1}) - \nabla f(\theta_{k-1}') + g(\theta_{k-1}, \xi_k) - g(\theta_{k-1}', \xi_k) \rangle + \alpha_k^2(L_1+L_2)^2\delta_{k-1}'^2)^p\\&\leq \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\}}}}{\sum}\frac{p!}{i!j!l!}\delta_{k-1}'^{2i}(- 2\alpha_k\langle \theta_{k-1} - \theta_{k-1}', \nabla f(\theta_{k-1}) - \nabla f(\theta_{k-1}') + g(\theta_{k-1}, \xi_k) - g(\theta_{k-1}', \xi_k) \rangle)^{j}(\alpha_k(L_1+L_2)\delta_{k-1}')^{2l}
\end{align}
Now we bound each term in the sum above.
\begin{enumerate}
\item First, for $i=p, j=0, l=0$, the corresponding term in the sum equals $\delta_{k-1}'^{2p}$.
\item Second, for $i=p-1, j=1, l=0$, we obtain, applying \Cref{ass:L-smooth}, that
\begin{align}
&2p\alpha_k \PE [\langle\theta_{k-1}-\theta_{k-1}', \nabla f(\theta_{k-1}) -\nabla f(\theta_{k-1}') +g(\theta_{k-1}, \xi_k) - g(\theta_{k-1}', \xi_k) \rangle\delta_{k-1}'^{2(p-1)}|\F_{k-1}] \\ &= 2p\alpha_k\langle\theta_{k-1}-\theta_{k-1}', \nabla f(\theta_{k-1})- \nabla f(\theta_{k-1}') \rangle\delta_{k-1}'^{2(p-1)}
\geq 2 p \mu \alpha_k \delta_{k-1}'^{2 p}\eqsp.
\end{align}
\item Third, for $l \geq 1$ or $j \geq 2$ (that is, $2l+j \geq 2$), we use Cauchy-Schwartz inequality together with \Cref{ass:noise_decomposition} and \Cref{ass:L-smooth}
\begin{equation}
|\langle\theta_{k-1}-\theta_{k-1}', \nabla f(\theta_{k-1}) -\nabla f(\theta_{k-1}') +g(\theta_{k-1}, \xi_k) - g(\theta_{k-1}', \xi_k) \rangle^j| \leq \norm{\theta_{k-1}-\theta_{k-1}'}^{2j}(L_1+L_2)^j\eqsp,
\end{equation}
\end{enumerate}
Combining inequalities above, we obtain
\begin{equation}
\label{eq:coupling_recurence}
\PE[\delta_{k}'^{2p}|\F_{k-1}] \leq (1- 2p\mu\alpha_k+ \underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\};\\j+2l\geq 2}}}{\sum}\frac{p!}{i!j!l!}2^j\alpha_k^{j+2l}(L_1+L_2)^{j+2l})\delta_{k-1}'^{2p}
\end{equation}
Similar to \eqref{eq:bound_sum_j_2l_p_moment}, we have
\begin{equation}
\underset{\mathclap{\substack{i+j+l=p;\\ i,j,l \in \{0,\ldots p\};\\j+2l\geq 2}}}{\sum}\frac{p!}{i!j!l!}2^j\alpha_k^{j+2l}(L_1+L_2)^{j+2l})\delta_{k-1}'^{2p} \leq \alpha_{k}^2p^2(L_1+L_2)^2(1+3\alpha_k(L_1+L_2))^{p}
\end{equation}
Enrolling recurrence \eqref{eq:coupling_recurence}, we get
\begin{align}
\PE[\delta_{k}'^{2p}] &\leq \exp\biggl\{-2p\mu\sum_{i=1}^k\alpha_i\biggr\}\exp\biggl\{ p^2(L_1+L_2)^2\sum_{i=1}^k\alpha_i^{2}(1+3\alpha_i(L_1+L_2))^p\biggr\}\norm{\theta_0-\thetas}^{2p}\\ &\leq c_6\exp\biggl\{-\frac{p\mu c_0}{1-\gamma}(k+k_0)^{1-\gamma}\biggr\}\norm{\theta_0-\thetas}^{2p}\eqsp,
\end{align}
where we have set
\begin{equation}
c_6 = \exp\biggl\{\exp\biggl\{ 3pc_0(L_1+L_2)\biggr\}\frac{p^2(L_1+L_2)^2)}{2\gamma-1} + \frac{p\mu c_0}{1-\gamma}k_0^{1-\gamma}\biggr\}\eqsp.
\end{equation}
It remains to note that
\begin{align}
\PE[\norm{\theta_k-\thetas}^{2p}]&\leq 2^{2p-1} \PE[\norm{\theta'_k-\thetas}^{2p}] + 2^{2p-1}\PE[\norm{\theta_k-\theta_k'}^{2p}] \\&\leq C_{2p,1} \exp\biggl\{-\frac{p\mu c_0}{4}(k+k_0)^{1-\gamma}\biggr\}(\norm{\theta_0-\thetas}^{2p} + \sigma_{2p}^{2p}) + C_{2p,2}\sigma_{2p}^{2p}\alpha_k^p\eqsp.
\end{align}
\end{proof}
For validity of induction in \Cref{prop:2p-moment-bound}, we need the following corollary.
\begin{corollary}
\label{cor:p_moment_bound}
Under the assumptions of \Cref{prop:2p-moment-bound}, it holds that
\begin{equation}
\PE[\norm{\theta_k-\thetas}^{2p}]\leq D_{2p} (\norm{\theta_0-\thetas}^{2p} + \sigma_2^{2p}) \alpha_k^p\eqsp,
\end{equation}
where
\begin{equation}
\label{eq:const_D_2p_def}
D_{2p} = C_{2p,1} (1/c_0^p + C_{2p, 2}) \biggl(\frac{4\gamma}{(1-\gamma)\mu p c_0 \rme}\biggr)^{\gamma p/(1-\gamma)}\eqsp.
\end{equation}
\end{corollary}
\begin{proof}
Define $C_5 = (\frac{4\gamma}{(1-\gamma)p\mu c_0 \rme})^{\gamma p/(1-\gamma)} > 1$, then $\exp\{-\mu pc_0(k+k_0)^{1-\gamma} / 4\}\leq C_5 (k+k_0)^{-p\gamma}$, and the statement follows.
\end{proof}
\begin{corollary}
\label{cor:fourth_moment_bound_last_iterate}
Assume \Cref{ass:L-smooth}, \Cref{ass:hessian_Lipschitz_ball}, \Cref{ass:noise_decomposition}($4$) and \Cref{ass:step_size}. Then for any $k \in \nset$ it holds that
\begin{equation}
\PE[\norm{\theta_k - \thetas}^{4}] \leq C_{4,1}\exp\biggl\{-\frac{2\mu c_0}{4}k^{1-\gamma}\biggr\}(\norm{\theta_0-\thetas}^{4} + \sigma_{4}^{4}) + C_{4,2}\sigma_{4}^{4}\alpha_k^2\eqsp,
\end{equation}
with
\begin{equation}
C_{4,1} = 2^{3}\biggl(C_1(1/c_0+C_2)\biggl(\frac{4\gamma}{(1-\gamma)\mu c_0 \rme}\biggr)^{\gamma/(1-\gamma)}\bigl(4c_0^{1/2}2^{\gamma/2}+2^{\gamma}+ 4c_0\bigr)^2c_0^2+1\biggr)c_{2,4}
\end{equation}
and
\begin{equation}
C_{4,2} =2^{3}C_1(1/c_0+C_2)\biggl(\frac{4\gamma}{(1-\gamma)\mu c_0 \rme}\biggr)^{\gamma/(1-\gamma)}\bigl(4c_0^{1/2}2^{\gamma/2}+2^{\gamma}+ 4c_0\bigr)^2c_{2,5}.
\end{equation}
Here $C_1$ and $C_2$ are defined in \Cref{lem:bound_last_iter_second_moment} and
\begin{align}
&c_{2,4} = \biggl(\exp\biggl\{\exp\biggl\{10c_0(L_1+L_2)\biggr\}\frac{16(L_1+L_2)^2}{2\gamma-1}\biggr\} + 1\biggr)\exp\biggl\{\frac{2\mu c_0}{1-\gamma}k_0^{1-\gamma}\biggr\}\frac{1}{3\gamma-1}\eqsp, \\&
c_{2,5} = \frac{2^{1+2\gamma}}{2\mu c_0}\eqsp.
\end{align}
\end{corollary}
\begin{proof}
The proof follows directly from \Cref{prop:2p-moment-bound} and \Cref{cor:second_moment_bound}.
\end{proof}
% \subsection{High probability bound of last iterate SGD}
% \input{high-prob-last_iter_sgd}
In this section, we establish a high-probability bound for the last iterate, which is instrumental in controlling the non-linear statistic $D^{\boot}$. Our analysis adapts the approach of \cite[Theorem 4.1]{harvey2019tight}, which relies on the assumption that the noise variables $\zeta_k$ are bounded almost surely.
% \begin{lemma}
% \label{lem: high_prob_last_iter_boot}
% Assume \Cref{ass:L-smooth}, \Cref{ass:bound_noise}, \Cref{ass:L-co-coercivity-assum},\Cref{ass:bound_bootstap_weights} \Cref{ass:step_size_new_boot}. Then for any $\delta \in (0,1)$ with probability at least $1-\delta$ for any $k\in\{1,\ldots n\}$ it holds
% \begin{equation}
% \norm{\theta_k^b - \thetas}^2 \leq \alpha_k K_1\log\biggl(\frac{\rme n}{\delta}\biggr)\eqsp,
% \end{equation}
% where
% \begin{equation}
% \label{eq:def_K_1}
% K_1 = \max\biggl(\frac{8W_{\max}^2(C_{1,\xi}+2C_{2,\xi})^2}{W_{\min}\mu},\frac{k_0^\gamma\norm{\theta_0-\thetas}^2}{c_0} \biggr)
% \end{equation}
% \end{lemma}
% \begin{proof}
% Using \eqref{eq:sgd_bootstrap}, we have
% \begin{align}
% \norm{\theta_k^b-\thetas}^2 &= \norm{\theta_{k-1}^b-\thetas}^2 - 2\alpha_kw_k\langle F(\theta_{k-1}^b, \xi_k), \theta_{k-1}^b-\thetas \rangle + \alpha_k^2w_k^2\norm{\nabla F(\theta_{k-1}^b, \xi_k)}^2 \\& \leq
% \norm{\theta_{k-1}^b-\thetas}^2 - 2\alpha_kw_k\langle F(\theta_{k-1}^b, \xi_k)- F(\thetas, \xi_k), \theta_{k-1}^b-\thetas \rangle \\& \qquad \qquad- 2\alpha_kw_k\langle \eta(\xi_k), \theta_{k-1}^b-\thetas \rangle+ 2\alpha_k^2w_k^2\norm{ F(\theta_{k-1}^b, \xi_k)- F(\thetas, \xi_k)}^2 + 2\alpha_k^2w_k^2\norm{\eta(\xi_k)}^2 \eqsp.
% \end{align}
% Using \Cref{ass:L-co-coercivity-assum} and \Cref{ass:step_size_new_boot}, we obtain
% \begin{align}
% \norm{\theta_k^b-\thetas}^2 &\leq \norm{\theta_{k-1}^b-\thetas}^2 - 2\alpha_kw_k(1-\alpha_kw_kL_4)\langle F(\theta_{k-1}^b, \xi_k)- F(\thetas, \xi_k), \theta_{k-1}^b-\thetas \rangle \\& \qquad \qquad \qquad - 2\alpha_kw_k\langle \eta(\xi_k), \theta_{k-1}^b-\thetas \rangle + 2\alpha_k^2w_k^2\norm{\eta(\xi_k)}^2 \\&\leq \norm{\theta_{k-1}^b-\thetas}^2 - \alpha_kw_k\langle F(\theta_{k-1}^b, \xi_k)- F(\thetas, \xi_k), \theta_{k-1}^b-\thetas \rangle \\& \qquad \qquad \qquad - 2\alpha_kw_k\langle \eta(\xi_k), \theta_{k-1}^b-\thetas \rangle + 2\alpha_k^2w_k^2\norm{\eta(\xi_k)}^2
% \end{align}
% Using \Cref{ass:L-smooth}, we have
% \begin{equation}
% \norm{\theta_k^b-\thetas}^2 \leq
% (1-\mu\alpha_kw_k)\norm{\theta_{k-1}^b-\thetas}^2 - \alpha_kw_k\langle g(\theta_{k-1}^b,\xi_k)+2\eta(\xi_k), \theta_{k-1}^b-\thetas \rangle + 2\alpha_k^2w_k^2\norm{\eta(\xi_k)}^2 \eqsp.
% \end{equation}
% Using \Cref{ass:bound_bootstap_weights}, we get
% \begin{equation}
% \label{eq:last_iter_reccurrence_high_prob}
% \begin{split}
% \norm{\theta_k^b-\thetas}^2 &\leq
% (1-\mu\alpha_kW_{\min})\norm{\theta_{k-1}^b-\thetas}^2 - \alpha_kw_k\langle g(\theta_{k-1}^b,\xi_k)+2\eta(\xi_k), \theta_{k-1}^b-\thetas \rangle \\
% & + 2\alpha_k^2W_{\max}^2\norm{\eta(\xi_k)}^2 \eqsp.
% \end{split}
% \end{equation}
% Define $Y_k = \alpha_k^{-1}\norm{\theta_k^b-\thetas}^2$, and $\hat X_{k-1} = \frac{w_k\langle g(\theta_{k-1}^b, \xi_k)+2\eta(\xi_k), \theta_{k-1}^b-\thetas \rangle}{W_{\max}(C_{2,\xi} + 2C_{1,\xi})\norm{\theta_{k-1}^b-\thetas}} $, then using \eqref{eq:last_iter_reccurrence_high_prob}, we obtain
% \begin{equation}
% Y_k \leq\alpha_k^{-1} \alpha_{k-1}(1-\mu W_{\min}\alpha_k)Y_{k-1} - \sqrt{\alpha_{k-1}}W_{\max}(C_{2,\xi} + 2C_{1,\xi})\hat X_{k-1}\sqrt{Y_{k-1}}+ 2W_{\max}^2\alpha_kC_{1,\xi}^2\eqsp.
% \end{equation}
% Note that
% \begin{align}
% \label{eq:bound_contracting_multiplier_last_iter}
% \frac{\alpha_{k-1}}{\alpha_k}(1-\mu W_{\min}\alpha_k) &= \biggl(\frac{k_0+k}{k_0+k-1}\biggr)^{\gamma} -\frac{\mu W_{\min}c_0}{(k_0+k-1)^{\gamma}}\\&\leq 1 + \frac{c_0(\gamma/c_0)}{k_0+k-1}-\frac{\mu W_{\min}c_0}{(k_0+k-1)^{\gamma}}\\& = 1 - \alpha_{k-1}\biggr( \mu W_{\min} -\frac{(\gamma/c_0)}{(k_0+k-1)^{1-\gamma}}\biggl)\eqsp.
% \end{align}
% Since $k_0\geq \biggl(\frac{2\gamma}{c_0\mu W_{\min}}\biggr)^{1/(1-\gamma)}$, we have
% \begin{equation}
% Y_k \leq(1-\frac{\mu W_{\min}}{2}\alpha_{k-1})Y_{k-1} - \sqrt{\alpha_{k-1}}W_{\max}(C_{2,\xi} + 2C_{1,\xi})\hat X_{k-1}\sqrt{Y_{k-1}}+ 2W_{\max}^2\alpha_kC_{1,\xi}^2\eqsp.
% \end{equation}
% Note that using \Cref{ass:noise_decomposition} and \Cref{ass:bound_noise}, we have
% \begin{equation}
% \begin{split}
% &\PE[\hat X_{k-1}|\widetilde{\F}_{k-1}] = 0\\
% &\norm{\hat X_{k-1}} \leq \frac{\norm{w_i}(\norm{g(\theta_{k-1}^b, \xi_k)}+2\norm{\eta(\xi_k)})\norm{\theta_{k-1}^b-\thetas \rangle}}{W_{\max}(C_{2,\xi} + 2C_{1,\xi})\norm{\theta_{k-1}^b-\thetas}} \leq 1 \eqsp,
% \end{split}
% \end{equation}
% where $\widetilde{\F}_{k-1}$ is defined in \eqref{eq: extended filtration}. Then using \cite[Theorem 4.1]{harvey2019tight}, with probability at least $1-\delta$ for $\forall k\in\{1,\ldots n\}$
% \begin{equation}
% Y_k \leq K_1 \log\biggl(\frac{\rme n}{\delta}\biggr),
% \end{equation}
% where $K_1$ is given in \eqref{eq:def_K_1}, and the statement follows.
% \end{proof}
\begin{lemma}
\label{lem: high_prob_last_iter_boot}
Assume \Cref{ass:L-smooth}, \Cref{ass:bound_noise},\Cref{ass:bound_bootstap_weights}, \Cref{ass:step_size_new_boot}. Then for any $\delta \in (0,1)$ with probability at least $1-\delta$ for any $k\in\{1,\ldots n\}$ it holds
\begin{equation}
\norm{\theta_k^b - \thetas}^2 \leq \alpha_k K_1\log\biggl(\frac{\rme n}{\delta}\biggr)\eqsp,
\end{equation}
where
\begin{equation}
\label{eq:def_K_1}
K_1 = \max\biggl(\frac{12W_{\max}^2(C_{1,\xi}+C_{2,\xi})^2}{W_{\min}\mu},\frac{k_0^\gamma\norm{\theta_0-\thetas}^2}{c_0} \biggr)
\end{equation}
\end{lemma}
\begin{proof}
Using \eqref{eq:sgd_bootstrap}, we have
\begin{align}
\norm{\theta_k^b-\thetas}^2 &= \norm{\theta_{k-1}^b-\thetas}^2 - 2\alpha_kw_k\langle F(\theta_{k-1}^b, \xi_k), \theta_{k-1}^b-\thetas \rangle + \alpha_k^2w_k^2\norm{\nabla F(\theta_{k-1}^b, \xi_k)}^2 \\& \leq
\norm{\theta_{k-1}^b-\thetas}^2 - 2\alpha_kw_k\langle \nabla f(\theta_{k-1}^b), \theta_{k-1}^b-\thetas \rangle - 2\alpha_kw_k\langle g(\theta_{k-1}^b, \xi_k) + \eta(\xi_k), \theta_{k-1}^b -\thetas \rangle \\& \qquad \qquad + 2\alpha_k^2w_k^2\norm{ f(\theta_{k-1}^b) + g(\theta_{k-1}^b, \xi_k) + \eta(\xi_k)}^2 \eqsp.
\end{align}
Using \Cref{ass:L-smooth} and \Cref{ass:bound_noise} together with \Cref{ass:bound_bootstap_weights} we obtain
\begin{align}
\norm{\theta_k^b-\thetas}^2 &\leq (1-2\alpha_k\mu W_{\min} + 3\alpha_k^2(L_1^2+L_2^2)W_{\max}^2)\norm{\theta_{k-1}^b-\thetas}^2 \\& \qquad \qquad- 2\alpha_kw_k\langle g(\theta_{k-1}^b, \xi_k)+ \eta(\xi_k), \theta_{k-1}^b-\thetas \rangle + 3\alpha_k^2w_k^2\norm{\eta(\xi_k)}^2\\&\leq (1-2\alpha_k\mu W_{\min} + 3\alpha_k^2(L_1^2+L_2^2)W_{\max}^2)\norm{\theta_{k-1}^b-\thetas}^2 \\& \qquad \qquad- 2\alpha_kw_k\langle g(\theta_{k-1}^b, \xi_k)+ \eta(\xi_k), \theta_{k-1}^b-\thetas \rangle + 3\alpha_k^2W_{\max}^2C_{1,\xi}^2
\end{align}
Using \Cref{ass:step_size_new_boot}, we have
\begin{equation}
\label{eq:last_iter_reccurrence_high_prob}
\norm{\theta_k^b-\thetas}^2 \leq
(1-\mu\alpha_kW_{\min})\norm{\theta_{k-1}^b-\thetas}^2 - 2\alpha_kw_k\langle g(\theta_{k-1}^b,\xi_k)+\eta(\xi_k), \theta_{k-1}^b-\thetas \rangle + 3\alpha_k^2W_{\max}^2C_{1,\xi}^2\eqsp.
\end{equation}
Define $Y_k = \alpha_k^{-1}\norm{\theta_k^b-\thetas}^2$, and $\hat X_{k-1} = \frac{2w_k\langle g(\theta_{k-1}^b, \xi_k)+\eta(\xi_k), \theta_{k-1}^b-\thetas \rangle}{2W_{\max}(C_{2,\xi} + C_{1,\xi})\norm{\theta_{k-1}^b-\thetas}} $, then using \eqref{eq:last_iter_reccurrence_high_prob}, we obtain
\begin{equation}
Y_k \leq\alpha_k^{-1} \alpha_{k-1}(1-\mu W_{\min}\alpha_k)Y_{k-1} - \sqrt{\alpha_{k-1}}2W_{\max}(C_{2,\xi} + 2C_{1,\xi})\hat X_{k-1}\sqrt{Y_{k-1}}+ 3W_{\max}^2\alpha_kC_{1,\xi}^2\eqsp.
\end{equation}
Note that
\begin{align}
\label{eq:bound_contracting_multiplier_last_iter}
\frac{\alpha_{k-1}}{\alpha_k}(1-\mu W_{\min}\alpha_k) &= \biggl(\frac{k_0+k}{k_0+k-1}\biggr)^{\gamma} -\frac{\mu W_{\min}c_0}{(k_0+k-1)^{\gamma}}\\&\leq 1 + \frac{c_0(\gamma/c_0)}{k_0+k-1}-\frac{\mu W_{\min}c_0}{(k_0+k-1)^{\gamma}}\\& = 1 - \alpha_{k-1}\biggr( \mu W_{\min} -\frac{(\gamma/c_0)}{(k_0+k-1)^{1-\gamma}}\biggl)\eqsp.
\end{align}
Since $k_0\geq \biggl(\frac{2\gamma}{c_0\mu W_{\min}}\biggr)^{1/(1-\gamma)}$, we have
\begin{equation}
Y_k \leq(1-\frac{\mu W_{\min}}{2}\alpha_{k-1})Y_{k-1} - \sqrt{\alpha_{k-1}}2W_{\max}(C_{2,\xi} + C_{1,\xi})\hat X_{k-1}\sqrt{Y_{k-1}}+ 3W_{\max}^2\alpha_kC_{1,\xi}^2\eqsp.
\end{equation}
Note that using and \Cref{ass:bound_noise} and \Cref{{ass:bound_bootstap_weights}}, we have
\begin{equation}
\begin{split}
&\PE[\hat X_{k-1}|\widetilde{\F}_{k-1}] = 0\\
&\norm{\hat X_{k-1}} \leq \frac{2\norm{w_k}(\norm{g(\theta_{k-1}^b, \xi_k)}+\norm{\eta(\xi_k)})\norm{\theta_{k-1}^b-\thetas \rangle}}{2W_{\max}(C_{2,\xi} + C_{1,\xi})\norm{\theta_{k-1}^b-\thetas}} \leq 1 \eqsp,
\end{split}
\end{equation}
where $\widetilde{\F}_{k-1}$ is defined in \eqref{eq: extended filtration}. Then using \cite[Theorem 4.1]{harvey2019tight}, with probability at least $1-\delta$ for $\forall k\in\{1,\ldots n\}$
\begin{equation}
Y_k \leq K_1 \log\biggl(\frac{\rme n}{\delta}\biggr),
\end{equation}
where $K_1$ is given in \eqref{eq:def_K_1}, and the statement follows.
\end{proof}
\begin{corollary}
\label{cor: last_iter_boot_p_moment}
Under the assumptions of \Cref{lem: high_prob_last_iter_boot}
for any $ k\in\{1,\ldots n\}$ and any $p\geq 2$ it holds
\begin{equation}
\PE^{2/p}[\norm{\theta_k^b - \thetas}^{p}] \leq p \alpha_k(\rme n)^{2/p} K_1/2 \eqsp,
\end{equation}
where $K_1$ is defined in \eqref{eq:def_K_1}.
\end{corollary}
\begin{proof}
Note that from \Cref{lem: high_prob_last_iter_boot} for $\forall k\in\{1,\ldots n\}$ and for any $t\geq 0$ it holds
\begin{equation}
\PP\bigl[\norm{\theta_k^b - \thetas}^2 \geq t\bigr]\leq f(t)\eqsp,
\end{equation}
where
\[
f(t) = \rme n\exp\biggl\{-\frac{t}{K_1\alpha_k}\biggr\}\eqsp.
\]
Then, we have
\begin{align}
\PE[\norm{\theta_k^b - \thetas}^{p}] &= \int_{0}^{+\infty}\PP[\norm{\theta_k^b - \thetas}^{p} > u]\rmd u \leq\int_{0}^{+\infty}\rme n\exp\biggl\{-\frac{u^{2/p}}{K_1\alpha_k}\biggr\}\rmd u \\&=\rme n(p/2)\biggl(K_1\alpha_k\biggr)^{p/2}\int_{0}^{+\infty}\rme^{-x}x^{p/2-1}\rmd x \leq\rme n\biggl((p/2)K_1\alpha_k\biggr)^{p/2}\eqsp,
\end{align}
where in the last inequality we use that $\Gamma(p/2) \leq (p/2)^{p/2 -1}$ (see \cite[Theorem 1.5]{AndersonGammaFunc}).
% For arbitrary $\theta_0^\boot = \theta_0$ we consider synchronous coupling contraction. Applying the similar arguments as in \Cref{prop:2p-moment-bound}, we obtain
% \begin{equation}
% \PE[\norm{\theta_k^\boot-\theta_k'^\boot}^p] \leq \PE^{1/2}[\norm{\theta_k^\boot-\theta_k'^\boot}^{2p}] \leq (C_{p}^\boot)^p\exp\biggl\{-\frac{p\mu c_0W_{\min}}{1-\gamma}\biggr\}\norm{\theta_0-\thetas}^p\eqsp,
% \end{equation}
% where
% \begin{equation}
% C_{p}^\boot = \exp\biggl\{\exp\biggl\{ 3pc_0W_{\max}(L_1+L_2)\biggr\}\frac{2p^2W_{\max}^2(L_1+L_2)^2)}{2\gamma-1} + \frac{p\mu c_0}{1-\gamma}k_0^{1-\gamma}\biggr\}\eqsp.
% \end{equation}
\end{proof}
\begin{lemma}
\label{lem: high_prob_last_iter}
Assume \Cref{ass:L-smooth}, \Cref{ass:bound_noise}, \Cref{ass:step_size_new_boot}. Then for any $\delta \in (0,1)$ with probability at least $1-\delta$ for any $k\in\{1,\ldots n\}$ it holds
\begin{equation}
\norm{\theta_k - \thetas}^2 \leq \alpha_k K_2 log\biggl(\frac{\rme n}{\delta}\biggr)\eqsp,
\end{equation}
where
\begin{equation}
\label{eq:def_K_2}
K_2 = \max\biggl(\frac{12(C_{1,\xi}+C_{2,\xi})^2}{\mu}, \frac{k_0^{\gamma}\norm{\theta_0-\thetas}^2}{\alpha_0}\biggr)
\end{equation}
Moreover, it holds
for any $k\in\{1,\ldots n\}$ and any $p\geq 2$ that
\begin{equation}
\PE^{2/p}[\norm{\theta_k - \thetas}^{p}]\leq p\alpha_k (\rme n)^{2/p}K_2/2\eqsp.
\end{equation}
\end{lemma}
\begin{proof}
The proof is similar to the proof of \cref{lem: high_prob_last_iter_boot} and \Cref{cor: last_iter_boot_p_moment}.
\end{proof}
% \begin{proof}
% Note that
% \begin{align}
% &\norm{\theta_k-\thetas}^2 = \norm{\theta_{k-1}-\thetas}^2 - 2\alpha_k\langle F(\theta_{k-1}, \xi_k), \theta_{k-1}-\thetas \rangle + \alpha_k^2\norm{ F(\theta_{k-1}, \xi_k)}^2 \\&\leq
% \norm{\theta_{k-1}-\thetas}^2 - 2\alpha_k\langle F(\theta_{k-1}, \xi_k), \theta_{k-1}-\thetas \rangle + 2\alpha_k^2\norm{F(\theta_{k-1}, \xi_k)-F(\thetas, \xi_k)}^2 + 2\alpha_k^2\norm{\eta(\xi_k)}^2 \eqsp.
% \end{align}
% Using \Cref{ass:L-co-coercivity-assum}, \Cref{ass:mu-convex} and \Cref{ass:step_size_new_boot}, we have
% \begin{align}
% \norm{\theta_{k}-\thetas}^2 &\leq
% \norm{\theta_{k-1}-\thetas}^2 - (2\alpha_k- 2\alpha_k^2L_3)\langle F(\theta_{k-1}, \xi_k)-F(\thetas, \xi_k), \theta_{k-1}-\thetas \rangle \\
% & \qquad \qquad \qquad \qquad \qquad - 2\alpha_k\langle \eta(\xi_k), \theta_{k-1}-\thetas \rangle + 2\alpha_k^2\norm{\eta(\xi_k)}^2 \\
% &\leq (1-\mu\alpha_k)\norm{\theta_{k-1}-\thetas}^2 - \alpha_k\langle g(\theta_{k-1}, \xi_k)+2\eta(\xi_k), \theta_{k-1}-\thetas \rangle + 2\alpha_k^2\norm{\eta(\xi_k)}^2\eqsp.
% \end{align}
% Finally, using \Cref{ass:bound_noise}, we get
% \begin{equation}
% \label{eq:last_iter_reccurrence_high_prob}
% \norm{\theta_k-\thetas}^2 \leq (1-\mu\alpha_k)\norm{\theta_{k-1}-\thetas}^2 - \alpha_k\langle g(\theta_{k-1}, \xi_k)+2\eta(\xi_k), \theta_{k-1}-\thetas \rangle
% + 2\alpha_k^2C_{1,\xi}^2\eqsp.
% \end{equation}
% Set
% \[
% Y_k = \alpha_k^{-1}\norm{\theta_k-\thetas}^2 \quad \text{ and } \quad \hat X_{k-1} = \frac{\langle g(\theta_{k-1}, \xi_k)+2\eta(\xi_k), \theta_{k-1}-\thetas \rangle}{(C_{2,\xi} + 2C_{1,\xi})\norm{\theta_{k-1}-\thetas}}\eqsp.
% \]
% Then using \eqref{eq:last_iter_reccurrence_high_prob}, we obtain
% \begin{equation}
% Y_k \leq\alpha_k^{-1} \alpha_{k-1}(1-\mu\alpha_k)Y_{k-1} - (C_{2,\xi} + 2C_{1,\xi})\sqrt{\alpha_{k-1}} \hat{X}_{k-1}\sqrt{Y_{k-1}}+ 2\alpha_kC_{1,\xi}^2\eqsp.
% \end{equation}
% Note that
% \begin{align}
% \label{eq:bound_contracting_multiplier_last_iter}
% \frac{\alpha_{k-1}}{\alpha_k}(1-\mu\alpha_k) &= \biggl(\frac{k_0+k}{k_0+k-1}\biggr)^{\gamma} -\mu\frac{\alpha_0}{(k_0+k-1)^{\gamma}}\\&\leq 1 + \frac{\alpha_0(\gamma/\alpha_0)}{k_0+k-1}-\mu\frac{\alpha_0}{(k_0+k-1)^{\gamma}}\\& = 1 - \alpha_{k-1}\biggr( \mu -\frac{(\gamma/\alpha_0)}{(k_0+k-1)^{1-\gamma}}\biggl)
% \end{align}
% Then, since $k_0 \geq (\frac{2\gamma}{\mu\alpha_0})^{1/(1-\gamma)}$, we have
% \begin{equation}
% Y_k \leq (1-\frac{\mu}{2}\alpha_{k-1})Y_{k-1} - (C_{2,\xi} + 2C_{1,\xi})\sqrt{\alpha_{k-1}}\hat{X}_{k-1}\sqrt{Y_{k-1}}+ 2\alpha_kC_{1,\xi}^2\eqsp.
% \end{equation}
% Using \Cref{ass:noise_decomposition} and \Cref{ass:bound_noise}, we get that $X_{k-1}$ is $\F_k$-measurable, and
% \begin{equation}
% \begin{split}
% &\PE[\hat X_{k-1}|\F_{k-1}] = 0\\
% &\norm{\hat X_{k-1}} \leq \frac{(\norm{g(\theta_{k-1}, \xi_k)}+2\norm{\eta(\xi_k)})\norm{\theta_{k-1}-\thetas}}{(C_{2,\xi} + 2C_{1,\xi})\norm{\theta_{k-1}-\thetas}} \leq 1\eqsp.
% \end{split}
% \end{equation}
% Then using \cite[Theorem 4.1]{harvey2019tight}, with probability at least $1-\delta$ it holds for any $k\in\{1,\ldots n\}$ that
% \begin{equation}
% Y_k \leq K \log\biggl(\frac{\rme n}{\delta}\biggr)\eqsp,
% \end{equation}
% where $K = \underset{k\in\{1,\ldots n\}}{\max}\biggl(\frac{8\alpha_k C_{1,\xi}^2}{\mu\alpha_{k-1}}, \frac{4 (C_{1,\xi}+2C_{2,\xi})^2}{\mu}\biggr) \leq \frac{8(C_{1,\xi}+2C_{2,\xi})^2}{\mu}$.
% \end{proof}
% \begin{corollary}
% \label{cor: last_iter_p_moment}
% Under assumptions of \Cref{lem: high_prob_last_iter}, it holds
% for any $k\in\{1,\ldots n\}$ and any $p\geq 2$ that
% \begin{equation}
% \PE^{2/p}[\norm{\theta_k - \thetas}^{p}]\leq p (\rme n)^{2/p} \frac{4(C_{1,\xi}+2C_{2,\xi})^2\alpha_k}{\mu}
% \end{equation}
% \end{corollary}
% \begin{proof}
% Note that from \Cref{lem: high_prob_last_iter_boot} for $\forall k\in\{1,\ldots n\}$ and for any $t\geq 0$ it holds
% \begin{equation}
% \PP\biggl[\norm{\theta_k - \thetas}^2 \geq t\biggr]\leq f(t)\eqsp,
% \end{equation}
% where
% \[
% f(t) = \rme n\exp\biggl\{-\frac{t\mu}{8(C_{1,\xi}+2C_{2,\xi})^2\alpha_k}\biggr\}\eqsp.
% \]
% Then, we have
% \begin{align}
% \PE[\norm{\theta_k - \thetas}^{p}] &= \int_{0}^{+\infty}\PP[\norm{\theta_k - \thetas}^{p} > u]\rmd u \\ &\leq\int_{0}^{+\infty}\rme n\exp\biggl\{-\frac{u^{2/p}\mu}{8(C_{1,\xi}+2C_{2,\xi})^2\alpha_k}\biggr\}\rmd u \\&=\rme n(p/2)\biggl(\frac{8(C_{1,\xi}+2C_{2,\xi})^2\alpha_k}{\mu}\biggr)^{p/2}\int_{0}^{+\infty}\rme^{-x}x^{p/2-1}\rmd x \\&\leq\rme n\biggl(\frac{4p (C_{1,\xi}+2C_{2,\xi})^2\alpha_k}{\mu}\biggr)^{p/2}\eqsp,
% \end{align}
% where in the last inequality we use that $\Gamma(p/2) \leq (p/2)^{p/2 -1}$ (see \cite[Theorem 1.5]{AndersonGammaFunc}).
% \end{proof}
We begin this section with useful technical bounds on sums of coefficients
\[
\sum_{i=m}^{k}\alpha_{i}^{p}\eqsp,
\]
where the step sizes $\alpha_i$ have a form
\[
\alpha_i = \frac{c_0}{(k_0+i)^{\gamma}}\eqsp, \quad 1/2 < \gamma < 1\eqsp, \quad k_0 \geq 1\eqsp.
\]
We also bound other related quantities, which are instrumental to our further analysis.
\begin{lemma}
\label{lem:bounds_on_sum_step_sizes}
Assume \Cref{ass:step_size}. Then
\begin{enumerate}[(a)]
\item \label{eq:sum_alpha_k_p} for any $p\geq 2$, it holds that
\begin{equation}
\sum_{i=1}^{k}\alpha_i^p \leq\frac{c_0^p}{p\gamma-1}\eqsp,
\end{equation}
\item \label{eq:simple_bound_sum_alpha_k}
for any $m\in\{0, \ldots, k\}$, it holds that
\begin{equation}
\sum_{i=m+1}^k\alpha_i \geq \frac{c_0}{2(1-\gamma)}((k+k_0)^{1-\gamma}-(m+k_0)^{1-\gamma})\eqsp,
\end{equation}
\end{enumerate}
\end{lemma}
\begin{proof}
To prove \ref{eq:sum_alpha_k_p}, observe that
\begin{equation}
\sum_{i=1}^{k}\alpha_i^p \leq c_{0}^p \int_{1}^{+\infty}\frac{\rmd x}{x^{p\gamma}} \leq \frac{c_0^p}{p\gamma-1}\eqsp,
\end{equation}
To prove \ref{eq:simple_bound_sum_alpha_k}, note that for any $i\geq 1$ and $k_0 \geq 1$, we have $2(i+k_0)^{-\gamma} \geq (i+k_0-1)^{-\gamma}$. Hence,
\begin{equation}
\sum_{i=m+1}^k\alpha_i \geq \frac{1}{2}\sum_{i=m}^{k-1}\alpha_i \geq \frac{c_0}{2}\int_{m+k_0}^{k+k_0}\frac{\rmd x}{x^{\gamma}} =
\frac{c_0}{2(1-\gamma)}((k+k_0)^{1-\gamma}-(m+k_0)^{1-\gamma})\eqsp.
\end{equation}
\end{proof}
\begin{lemma}
\label{lem:bound_sum_exponent}
For any $A >0$, any $0 \leq i \leq n-1$, and any $\gamma \in (1/2, 1)$ it holds
\begin{equation}
\sum_{j=i}^{n-1}\exp\biggl\{-A(j^{1-\gamma} - i^{1-\gamma})\biggr\} \leq
\begin{cases}
1 + \exp\bigl\{\frac{1}{1-\gamma}\bigr\}\frac{1}{A^{1/(1-\gamma)}(1-\gamma)}\Gamma(\frac{1}{1-\gamma})\eqsp, &\text{ if } Ai^{1-\gamma} \leq \frac{1}{1-\gamma} \text{ and } i \geq 1\eqsp;\\
1 + \frac{1}{A(1-\gamma)^2}i^\gamma\eqsp, &\text{ if } Ai^{1-\gamma} >\frac{1}{1-\gamma} \text{ and } i \geq 1\eqsp;\\
1 + \frac{1}{A^{1/(1-\gamma)}(1-\gamma)}\Gamma(\frac{1}{1-\gamma})\eqsp, &\text{ if } i=0 \eqsp.
\end{cases}
\end{equation}
\end{lemma}
\begin{proof}
Note that
\begin{align}
\sum_{j=i}^{n-1}\exp\biggl\{-A(j^{1-\gamma} - i^{1-\gamma})\biggr\}
&\leq 1 +\exp\biggl\{A i^{1-\gamma}\biggr\}\int_{i}^{+\infty}\exp\biggl\{-Ax^{1-\gamma} \biggr\}\rmd x \\
&= 1 + \exp\biggl\{A i^{1-\gamma}\biggr\}\frac{1}{A^{1/(1-\gamma)}(1-\gamma)}\int_{Ai^{1-\gamma}}^{+\infty}\rme^{-u} u^{\frac{1}{1-\gamma}-1}\rmd u
\end{align}
Applying \cite[Theorem 4.4.3]{gabcke2015neue}, we get
\begin{equation}
\int_{Ai^{1-\gamma}}^{+\infty}\rme^{-u} u^{\frac{1}{1-\gamma}-1} \rmd u\leq
\begin{cases}
\Gamma(\frac{1}{1-\gamma})\eqsp, &\text{ if } Ai^{1-\gamma} < \frac{1}{1-\gamma};\\
\frac{1}{1-\gamma}\exp\{-Ai^{1-\gamma}\} A^{\gamma/(1-\gamma)}i^\gamma\eqsp, &\text{ otherwise.}
\end{cases}
\end{equation}
Combining inequities above, for $i \geq 1$ we obtain
\begin{equation}
\sum_{j=i}^{n-1}\exp\biggl\{-A(j^{1-\gamma} - i^{1-\gamma})\biggr\} \leq
\begin{cases}
1 + \exp\bigl\{\frac{1}{1-\gamma}\bigr\}\frac{1}{A^{1/(1-\gamma)}(1-\gamma)}\Gamma(\frac{1}{1-\gamma})\eqsp, &\text{ if } Ai^{1-\gamma} < \frac{1}{1-\gamma};\\
1 + \frac{1}{A(1-\gamma)^2}i^\gamma\eqsp, &\text{ otherwise.}
\end{cases}
\eqsp,
\end{equation}
and for $i=0$, we have
\begin{equation}
\sum_{j=0}^{n-1}\exp\biggl\{-A j^{1-\gamma} \biggr\} \leq 1 + \frac{1}{A^{1/(1-\gamma)}(1-\gamma)}\Gamma\biggl(\frac{1}{1-\gamma}\biggr)\eqsp.
\end{equation}
\end{proof}
\subsection{Proof of \Cref{lem:bound_Q_i_and_Sigma_n}}
\label{subsec:proof_bound_Q_i_Sigma_n}
\textbf{Formulation of the \Cref{lem:bound_Q_i_and_Sigma_n} with explicit constants.}
\textit{
Assume \Cref{ass:L-smooth} and \Cref{ass:step_size}. Then for any $i \in \{0, \ldots, n-1\}$ it holds that
\begin{equation}
\lambda_{\max}(Q_i) \leq C_Q\eqsp,
\end{equation}
where the constant $C_Q$ is given by \begin{equation}
\label{eq:const_C_Q}
C_Q =\biggl[1+\max\biggl(\exp\biggl\{\frac{1}{1-\gamma}\biggr\}\biggl(\frac{2(1-\gamma)}{\mu c_0}\biggr)^{1/(1-\gamma)}\frac{1}{1-\gamma}\Gamma(\frac{1}{1-\gamma}), \frac{2}{\mu c_0(1-\gamma)}\biggr)\biggr]c_0\eqsp.
\end{equation}
Moreover,
\begin{equation}
\label{eq:Q_i_lower_bound}
\lambda_{\min}(Q_i) \geq C_Q^{\min}\eqsp, \text{ and } \norm{\Sigma_n^{-1/2}} \leq C_{\Sigma}\eqsp,
\end{equation}
where the matrix $\Sigma_n$ is defined in \eqref{eq:sigma_n_def}, and
\begin{equation}
\label{def:C_Q_min}
C_Q^{\min} = \frac{1}{L_1}(1-(1-\alpha_i L_1)^{n-i})\eqsp,
\end{equation}
\begin{equation}
\label{eq:def_C_Sigma}
C_{\Sigma} = \frac{\sqrt{2}L_1}{(1-\exp\{-\frac{\mu c_0L_1}{2(k_0+1)}\})\sqrt{\lambda_{\min}(\Sigma_{\xi})}}\eqsp.
\end{equation}
}
\begin{proof}
Note that using \Cref{lem:bounds_on_sum_step_sizes}\ref{eq:simple_bound_sum_alpha_k}, for $i \geq 0$, it holds that
\begin{align}
\lambda_{\max}(Q_i)
&\leq \alpha_i\sum_{j=i}^{n-1}\prod_{k=i+1}^{j}(1-\alpha_k \mu) \leq \alpha_i\sum_{j=i}^{n-1}\exp\biggl\{-\mu\sum_{k=i+1}^j\alpha_k\biggr\} \\
&\leq \alpha_i\sum_{j=i+k_0}^{n-1+k_0}\exp\biggl\{-\frac{\mu c_0}{2(1-\gamma)}(j^{1-\gamma}-(i+k_0)^{1-\gamma})\biggr\}\eqsp.
\end{align}
Using \Cref{lem:bound_sum_exponent}, we complete the first part with the constant $C_Q$ defined in \eqref{eq:const_C_Q}. In order to prove \eqref{eq:Q_i_lower_bound}, we note that
\begin{equation}
\lambda_{\min}(Q_i) \geq \alpha_i\sum_{j=i}^{n-1}(1-\alpha_i L_1)^{j-i} =\frac{1}{L_1}(1-(1-\alpha_i L_1)^{n-i})\eqsp.
\end{equation}
Then for $i \leq n/2$, we have
\begin{equation}
\lambda_{\min}(Q_i) \geq \frac{1}{L_1}(1-(1-\alpha_i L_1)^{n/2})\geq \frac{1}{L_1}(1-\exp\{-\mu\alpha_i L_1 n/2\}) \geq \frac{1}{L_1}(1-\exp\{-\frac{\mu c_0L_1}{2(k_0+1)}\})\eqsp,
\end{equation}
where the last inequality holds, since $\alpha_i n \geq \alpha_n n \geq \frac{c_0 n}{k_0 + n} \geq \frac{c_0}{1+k_0}$ .
Combining previous inequalities, we get
\begin{equation}
\lambda_{\min}(\Sigma_n) \geq \lambda_{\min}\bigl(n^{-1} \sum_{i=1}^{n/2}Q_i\Sigma_{\xi}Q_i^\top \bigr) \geq \frac{\lambda_{\min}(\Sigma_{\xi})}{2 L_1^2}(1-\exp\{-\frac{\mu c_0L_1}{2(k_0+1)}\})^2\eqsp,
\end{equation}
and \eqref{eq:Q_i_lower_bound} follows.
\end{proof}
\begin{lemma}
\label{lem:H_theta_bound}
Assume \Cref{ass:L-smooth} and \Cref{ass:hessian_Lipschitz_ball}. Then
\begin{equation}
\norm{H(\theta)}\leq L_{H} \norm{\theta-\thetas}^2,
\end{equation}
where $L_{H}= \max(L_3,2L_1/\beta)$.
\end{lemma}
\begin{proof}
Using \Cref{ass:hessian_Lipschitz_ball} and the definition of $H(\theta)$ in \eqref{eq:H_theta_def}, we get
\begin{equation}
\| H(\theta) \| \mathds{1}(\norm{\theta-\thetas}\leq \beta)\leq L_3\norm{\theta-\thetas}^2.
\end{equation}
Since $\mu \Id \preccurlyeq \nabla^2f(\theta) \preccurlyeq L_1 \Id$, we also obtain
\begin{equation}
\norm{H(\theta)} \mathds{1}(\norm{\theta-\thetas} > \beta) \leq 2L_1\mathds{1}(\norm{\theta-\thetas} > \beta)\norm{\theta-\thetas} \leq \frac{2L_1}{\beta}\norm{\theta-\thetas}^2.
\end{equation}
This concludes the proof.
\end{proof}
\subsection{Gaussian comparison lemma}
There are quite a few works devoted to the comparison of Gaussian measures with different covariance matrices and means. Among others, we note \cite{BarUly86}, \cite{Bernolli2019}, and \cite{Devroye2018}. We use the result from \cite[Theorem 1.1]{Devroye2018}, which provides a comparison in terms of the total variation distance. Recall that the total variation distance between probability measures $\mu$ and $\nu$ on a measurable space $(\Xset,\Xsigma)$ is defined as
\[
\mathsf{d}_{\mathsf{TV}}(\mu, \nu) = \sup_{B \in \Xsigma} |\mu(B) - \nu(B)|\eqsp.
\]
With a slight abuse of notation, when $X$ and $Y$ are random vectors with distributions $\mu$ and $\nu$, respectively, we write $\mathsf{d}_{\mathsf{TV}}(X,Y)$ instead of $\mathsf{d}_{\mathsf{TV}}(\mu,\nu)$. The following lemma holds:
\begin{lemma}
\label{Pinsker}
Let $\Sigma_1$ and $\Sigma_2$ be positive definite covariance matrices in $\rset^{d \times d}$. Let $X \sim \mathcal{N}(0, \Sigma_1)$ and $Y \sim \mathcal{N}(0, \Sigma_2)$. Then
\begin{equation}
\mathsf{d}_{\mathsf{TV}}(X, Y) \le \frac{3}{2} \|\Sigma_2^{-1/2} \Sigma_1 \Sigma_2^{-1/2} - \Id_d \|_{\mathsf{F}} \eqsp.
\end{equation}
\end{lemma}
Recall that our primary aim in this paper is to obtain convergence bounds in the convex distance
\[
\kolmogorov(X, Y) = \sup_{B \in \Conv(\rset^{d})}\left|\P\bigl(X \in B\bigr) - \P(Y \in B)\right|\eqsp,
\]
where $\Conv(\rset^{d})$ is a collection of convex sets on $\rset^{d}$. We can immediately obtain the result for convex distance from \Cref{Pinsker}, since
\[
\kolmogorov(X, Y) \leq \mathsf{d}_{\mathsf{TV}}(X, Y)\eqsp.
\]
\begin{thebibliography}{10}
\bibitem{agrawalla2023high}
Bhavya Agrawalla, Krishnakumar Balasubramanian, and Promit Ghosal.
\newblock High-dimensional central limit theorems for linear functionals of
online least-squares sgd.
\newblock {\em arXiv preprint arXiv:2302.09727}, 2023.
\bibitem{pmlr-v99-anastasiou19a}
Andreas Anastasiou, Krishnakumar Balasubramanian, and Murat~A. Erdogdu.
\newblock Normal approximation for stochastic gradient descent via
non-asymptotic rates of martingale {CLT}.
\newblock In Alina Beygelzimer and Daniel Hsu, editors, {\em Proceedings of the
Thirty-Second Conference on Learning Theory}, volume~99 of {\em Proceedings
of Machine Learning Research}, pages 115--137. PMLR, 25--28 Jun 2019.
\bibitem{AndersonGammaFunc}
G.~D. Anderson and S.-L. Qiu.
\newblock A monotoneity property of the gamma function.
\newblock {\em Proceedings of the American Mathematical Society},
125(11):3355--3362, 1997.
\bibitem{ball_reverse_1993}
Keith Ball.
\newblock The reverse isoperimetric problem for {Gaussian} measure.
\newblock {\em Discrete \& Computational Geometry}, 10(4):411--420, October
1993.
\bibitem{BarUly86}
S.~{Barsov} and V.~{Ulyanov}.
\newblock {Estimates for the closeness of Gaussian measures}.
\newblock {\em Dokl. Akad. Nauk SSSR}, 291(2):273--277, 1986.
\bibitem{bentkus2004}
V.~Bentkus.
\newblock On the dependence of the berry–esseen bound on dimension.
\newblock {\em Journal of Statistical Planning and Inference}, 113(2):385--402,
2003.
\bibitem{benveniste2012adaptive}
A.~Benveniste, M.~M{\'e}tivier, and P.~Priouret.
\newblock {\em Adaptive algorithms and stochastic approximations}, volume~22.
\newblock Springer Science \& Business Media, 2012.
\bibitem{bubeck2015convex}
S{\'e}bastien Bubeck et~al.
\newblock Convex optimization: Algorithms and complexity.
\newblock {\em Foundations and Trends{\textregistered} in Machine Learning},
8(3-4):231--357, 2015.
\bibitem{chen2021statistical}
Haoyu Chen, Wenbin Lu, and Rui Song.
\newblock Statistical inference for online decision making via stochastic
gradient descent.
\newblock {\em Journal of the American Statistical Association},
116(534):708--719, 2021.
\bibitem{chen2022online}
Xi~Chen, Zehua Lai, He~Li, and Yichen Zhang.
\newblock Online statistical inference for contextual bandits via stochastic
gradient descent.
\newblock {\em arXiv preprint arXiv:2212.14883}, 2022.
\bibitem{chen2020aos}
Xi~Chen, Jason~D. Lee, Xin~T. Tong, and Yichen Zhang.
\newblock {Statistical inference for model parameters in stochastic gradient
descent}.
\newblock {\em The Annals of Statistics}, 48(1):251 -- 273, 2020.
\bibitem{defazio2014saga}
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien.
\newblock {SAGA}: A fast incremental gradient method with support for
non-strongly convex composite objectives.
\newblock {\em Advances in neural information processing systems}, 27, 2014.
\bibitem{Devroye2018}
Luc Devroye, Abbas Mehrabian, and Tommy Reddad.
\newblock The total variation distance between high-dimensional gaussians with
the same mean.
\newblock {\em arXiv preprint arXiv:1810.08693}, 2018.
\bibitem{durmus2020biassgd}
Aymeric Dieuleveut, Alain Durmus, and Francis Bach.
\newblock {Bridging the gap between constant step size stochastic gradient
descent and Markov chains}.
\newblock {\em The Annals of Statistics}, 48(3):1348 -- 1382, 2020.
\bibitem{efron1992bootstrap}
Bradley Efron.
\newblock Bootstrap methods: another look at the jackknife.
\newblock In {\em Breakthroughs in statistics: Methodology and distribution},
pages 569--593. Springer, 1992.
\bibitem{JMLR:v19:17-370}
Yixin Fang, Jinfeng Xu, and Lei Yang.
\newblock Online bootstrap confidence intervals for the stochastic gradient
descent estimator.
\newblock {\em Journal of Machine Learning Research}, 19(78):1--21, 2018.
\bibitem{fedoryuk1977metod}
M.~V. Fedoryuk.
\newblock {\em Metod perevala}.
\newblock Izdat. ``Nauka'', Moscow, 1977.
\bibitem{gabcke2015neue}
Wolfgang Gabcke.
\newblock Neue herleitung und explizite restabsch{\"a}tzung der
riemann-siegel-formel.
\newblock 2015.
\bibitem{Bernolli2019}
Friedrich G\"{o}tze, Alexey Naumov, Vladimir Spokoiny, and Vladimir Ulyanov.
\newblock Large ball probabilities, {G}aussian comparison and
anti-concentration.
\newblock {\em Bernoulli}, 25(4A):2538--2563, 2019.
\bibitem{harvey2019tight}
Nicholas~JA Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa.
\newblock Tight analyses for non-smooth stochastic gradient descent.
\newblock In {\em Conference on Learning Theory}, pages 1579--1613. PMLR, 2019.
\bibitem{hazan2014beyond}
Elad Hazan and Satyen Kale.
\newblock Beyond the regret minimization barrier: optimal algorithms for
stochastic strongly-convex optimization.
\newblock {\em The Journal of Machine Learning Research}, 15(1):2489--2512,
2014.
\bibitem{kushner2003stochastic}
Harold Kushner and G~George Yin.
\newblock {\em Stochastic approximation and recursive algorithms and
applications}, volume~35.
\newblock Springer Science \& Business Media, 2003.
\bibitem{lan_2020}
Guanghui Lan.
\newblock {\em First-order and Stochastic Optimization Methods for Machine
Learning}.
\newblock 01 2020.
\bibitem{li2022root}
Chris~Junchi Li, Wenlong Mou, Martin Wainwright, and Michael Jordan.
\newblock Root-sgd: Sharp nonasymptotics and asymptotic efficiency in a single
algorithm.
\newblock In Po-Ling Loh and Maxim Raginsky, editors, {\em Proceedings of
Thirty Fifth Conference on Learning Theory}, volume 178 of {\em Proceedings
of Machine Learning Research}, pages 909--981. PMLR, 02--05 Jul 2022.
\bibitem{pmlr-v178-li22b}
Xiang Li, Jiadong Liang, Xiangyu Chang, and Zhihua Zhang.
\newblock Statistical estimation and online inference via local sgd.
\newblock In Po-Ling Loh and Maxim Raginsky, editors, {\em Proceedings of
Thirty Fifth Conference on Learning Theory}, volume 178 of {\em Proceedings
of Machine Learning Research}, pages 1613--1661. PMLR, 02--05 Jul 2022.
\bibitem{moulines2011non}
Eric Moulines and Francis Bach.
\newblock Non-asymptotic analysis of stochastic approximation algorithms for
machine learning.
\newblock {\em Advances in neural information processing systems}, 24:451--459,
2011.
\bibitem{nemirovski2009robust}
Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro.
\newblock Robust stochastic approximation approach to stochastic programming.
\newblock {\em SIAM Journal on optimization}, 19(4):1574--1609, 2009.
\bibitem{nesterov:2004}
Y.~Nesterov.
\newblock {\em Introductory Lectures on Convex Optimization: A Basic Course}.
\newblock Applied Optimization. Springer, 2004.
\bibitem{olver1997asymptotics}
Frank W.~J. Olver.
\newblock {\em Asymptotics and special functions}.
\newblock AKP Classics. A K Peters, Ltd., Wellesley, MA, 1997.
\newblock Reprint of the 1974 original [Academic Press, New York; MR0435697 (55
\#8655)].
\bibitem{oskekowski2012sharp}
Adam Osekowski.
\newblock {\em Sharp martingale and semimartingale inequalities}, volume~72.
\newblock Springer Science \& Business Media, 2012.
\bibitem{polyak1992acceleration}
Boris~T Polyak and Anatoli~B Juditsky.
\newblock Acceleration of stochastic approximation by averaging.
\newblock {\em SIAM journal on control and optimization}, 30(4):838--855, 1992.
\bibitem{qian1999momentum}
Ning Qian.
\newblock On the momentum term in gradient descent learning algorithms.
\newblock {\em Neural networks}, 12(1):145--151, 1999.
\bibitem{rakhlin2012making}
Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan.
\newblock Making gradient descent optimal for strongly convex stochastic
optimization.
\newblock In {\em Proceedings of the 29th International Coference on
International Conference on Machine Learning}, pages 1571--1578, 2012.
\bibitem{ruppert1988efficient}
David Ruppert.
\newblock Efficient estimations from a slowly convergent {R}obbins-{M}onro
process.
\newblock Technical report, Cornell University Operations Research and
Industrial Engineering, 1988.
\bibitem{sadiev2023high}
Abdurakhmon Sadiev, Marina Danilova, Eduard Gorbunov, Samuel Horv{\'a}th,
Gauthier Gidel, Pavel Dvurechensky, Alexander Gasnikov, and Peter
Richt{\'a}rik.
\newblock High-probability bounds for stochastic optimization and variational
inequalities: the case of unbounded variance.
\newblock In {\em International Conference on Machine Learning}, pages
29563--29648. PMLR, 2023.
\bibitem{samsonov2024gaussian}
Sergey Samsonov, Eric Moulines, Qi-Man Shao, Zhuo-Song Zhang, and Alexey
Naumov.
\newblock Gaussian {A}pproximation and {M}ultiplier {B}ootstrap for
{P}olyak-{R}uppert {A}veraged {L}inear {S}tochastic {A}pproximation with
{A}pplications to {TD} {L}earning.
\newblock In {\em Advances in Neural Information Processing Systems},
volume~37, pages 12408--12460. Curran Associates, Inc., 2024.
\bibitem{schmidt2017minimizing}
Mark Schmidt, Nicolas Le~Roux, and Francis Bach.
\newblock Minimizing finite sums with the stochastic average gradient.
\newblock {\em Mathematical Programming}, 162:83--112, 2017.
\bibitem{shao2003mathematical}
Jun Shao.
\newblock {\em Mathematical statistics}.
\newblock Springer Science \& Business Media, 2003.
\bibitem{shao1995jackknife}
Jun Shao and Dongsheng Tu.
\newblock {\em The Jackknife and Bootstrap}.
\newblock Springer Series in Statistics. Springer New York, NY, 1 edition,
1995.
\newblock Springer Book Archive, © Springer Science+Business Media New York
1995.
\bibitem{shao2022berry}
Qi-Man Shao and Zhuo-Song Zhang.
\newblock Berry--{E}sseen bounds for multivariate nonlinear statistics with
applications to {M}-estimators and stochastic gradient descent algorithms.
\newblock {\em Bernoulli}, 28(3):1548--1576, 2022.
\bibitem{SheshukovaICLR}
Marina Sheshukova, Denis Belomestny, Alain Durmus, Eric Moulines, Alexey
Naumov, and Sergey Samsonov.
\newblock Nonasymptotic analysis of stochastic gradient descent with the
richardson-romberg extrapolation.
\newblock {\em arXiv preprint arXiv:2410.05106}, 2024.
\bibitem{srikant2024rates}
R~Srikant.
\newblock Rates of {C}onvergence in the {C}entral {L}imit {T}heorem for
{M}arkov {C}hains, with an {A}pplication to {TD} learning.
\newblock {\em arXiv preprint arXiv:2401.15719}, 2024.
\bibitem{tropp2015introduction}
Joel~A Tropp et~al.
\newblock An introduction to matrix concentration inequalities.
\newblock {\em Foundations and Trends{\textregistered} in Machine Learning},
8(1-2):1--230, 2015.
\bibitem{wu2024statistical}
Weichen Wu, Gen Li, Yuting Wei, and Alessandro Rinaldo.
\newblock {S}tatistical {I}nference for {T}emporal {D}ifference {L}earning with
{L}inear {F}unction {A}pproximation.
\newblock {\em arXiv preprint arXiv:2410.16106}, 2024.
\bibitem{zhong2023online}
Yanjie Zhong, Todd Kuffner, and Soumendra Lahiri.
\newblock Online {B}ootstrap {I}nference with {N}onconvex {S}tochastic
{G}radient {D}escent {E}stimator.
\newblock {\em arXiv preprint arXiv:2306.02205}, 2023.
\bibitem{zhu2023online_cov_matr}
Wanrong Zhu, Xi~Chen, and Wei~Biao Wu.
\newblock Online {C}ovariance {M}atrix {E}stimation in {S}tochastic {G}radient
{D}escent.
\newblock {\em Journal of the American Statistical Association},
118(541):393--404, 2023.
\end{thebibliography}