Noise is All You Need: Private Second-Order Convergence of Noisy SGD
Extracted source text · arXiv 2410.06878
\begin{abstract}
Private optimization is a topic of major interest in machine learning, with differentially private stochastic gradient descent (DP-SGD) playing a key role in both theory and practice. Furthermore, DP-SGD is known to be a powerful tool in contexts beyond privacy, including robustness, machine unlearning, etc.
Existing analyses of DP-SGD either make relatively strong assumptions (e.g., Lipschitz continuity of the loss function, or even convexity) or prove only first-order convergence (and thus might end at a saddle point in the non-convex setting).
At the same time, there has been progress in proving second-order convergence of the non-private version of ``noisy SGD'', as well as progress in designing algorithms that are more complex than DP-SGD and do guarantee second-order convergence.
We revisit DP-SGD and show that ``noise is all you need'': the noise necessary for privacy already implies second-order convergence under the standard smoothness assumptions, even for non-Lipschitz loss functions.
Hence, we get second-order convergence essentially for free: DP-SGD, the workhorse of modern private optimization, under minimal assumptions can be used to find a second-order stationary point.
\end{abstract}
\section{Algorithm and Analysis}
\label{sec:algorithm}
Algorithm~\ref{alg:old_dpsgd} shows standard DP-SGD with clipping~\citep{abadi_DeepLearning_2016}.
Algorithm~\ref{alg:dpsgd} shows our simpler version of DP-SGD without clipping which, as we show next, still guarantees second-order convergence and differential privacy even without Lipschitz assumption.
\begin{algorithm}[t!]
\caption{Differentially private Clipped SGD}
\label{alg:old_dpsgd}
\begin{algorithmic}
\STATE {\bfseries Input:} Clipping threshold $C$, noise variance $\Delta^2$, batch size $B$, number of iterations $T$
\FOR{$t = 0, 1, 2, \ldots, T-1$}
\STATE Sample minibatch $\batch{t}$ of size $B$ \\
\STATE $\g{t}^{(i)} \gets \nabla f_i(\x{t}) / \max \pars{1, \frac{\|\nabla f_i(\x{t})\|}{C}}$ for each $i \in \batch{t}$ \\
\STATE Sample $\xi_t \sim \mathcal{N}(0, \Delta^2 \id)$ \\
\STATE $\x{t + 1} \gets \x{t} - \step \pars{\avg_{i \in \batch{t}} \g{t}^{(i)} + \xi_t}$
\ENDFOR
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[t!]
\caption{Differentially private SGD without clipping }
\label{alg:dpsgd}
\begin{algorithmic}
\STATE {\bfseries Input:} Batch size $B$, number of iterations $T$
\STATE Choose noise scale $\Delta$ as in Theorem~\ref{thm:dp}
\FOR{$t = 0, 1, 2, \ldots, T-1$}
\STATE Uniformly sample minibatch $\batch{t}$ of size $B$ \\
\STATE Sample $\xi_t \sim \mathcal{N}(0, \Delta^2 \id)$ \\
\STATE $\x{t+1} \gets \x{t} - \step \pars{\avg_{i \in \batch{t}} \nabla f_i(\x{t}) + \xi_t}$
\ENDFOR
\end{algorithmic}
\end{algorithm}
%\todo[inline]{Call our algorithm DP-SGD-light?}
%\todo[inline]{Write a story}
We begin in Section~\ref{sec:DP} by showing that Algorithm~\ref{alg:dpsgd} is indeed differentially private.
The proof of second-order convergence is more involved, so we give an outline of our analysis and some of the proofs in Section~\ref{ssec:convergence} (the details can be found in the Appendix).
% \mdnote{supplementary material or appendix?}
We then combine the privacy and convergence analysis to give our main overall result in Section~\ref{ssec:combine}.
In Section~\ref{ssec:high_prob} we describe how to find a second-order stationary point w.h.p.
In this section, $\tOh{\cdot}$ hides polynomial dependence on $L, \rho, \fmax$, and polylogarithmic dependence on all parameters.
\subsection{Analysis of Differential Privacy} \label{sec:DP}
The proof that \cref{alg:dpsgd} is differentially private is quite intuitive: we use the fact from previous work that \cref{alg:old_dpsgd} is differentially private, and then show that, thanks to the way we set the noise in \cref{alg:dpsgd}, clipping is unlikely to happen and thus the two algorithms behave identically. So the only difference in privacy is the small probability that clipping \emph{does} occur, which can be folded into the $\delta$ parameter of the privacy guarantee.
\begin{theorem}[Standard DP for clipped SGD~\citep{abadi_DeepLearning_2016}]
\label{thm:standard_dp}
There exist constants $c_1$ and $c_2$, such that for any $\eps < c_1 T B^2 / n^2$, Algorithm~\ref{alg:old_dpsgd} is $(\eps, \delta)$-DP for any $\delta > 0$ if $\Delta > c_2 \frac{C \sqrt{T \log \nicefrac{1}{\delta}}}{n \eps}$.
\end{theorem}
% \mdnoteinline{Now that the algorithm don't have a loop that ends at $T$, I don't think that it's been defined. Why not have the algorithm loop to $T$?}
%Because the privacy is only dependency on iteration number $T$, and $\Delta > c_2 \frac{C \sqrt{T \log \nicefrac{1}{\delta}}}{n \eps}$.
We will use the following bound on the gradients from~\citet{DBLP:conf/iclr/FangLF023}, who proved first-order convergence for DP-SGD without Lipschitz assumption.
% \todo{Be careful with lemma}
\begin{lemma}[Bound on the gradient; Proposition 4.3 from \citet{DBLP:conf/iclr/FangLF023}, simplified]
\label{lem:grad_bound}
% Let $\vx^*$ be the minimizer of $f$.
% Then $\|\nabla f(\vx)\| \le \sqrt{2L (f(\vx) - f(\vx^*))}$ for all $\vx$.
%\todo[inline]{$f(\x{0})$ might break DP}
Under Assumptions~\ref{ass:main} and~\ref{ass:grad_concentration}, when $\eta \le \min(1/L, 1 / (\sigma \sqrt{T}))$, with probability at least $1 - \delta$, for some absolute constant $c$:
% \[
% \max_{t = 0, ..., T-1}\|\nabla f(\x{t})\|
% \le 2 \sqrt{L \fmax} + c L \sqrt{\log \nicefrac{T}{\delta}}
% ,
% \]
% and, under Assumption:
\begin{align*}
&\max_{i \in [n], t = 0, ..., T-1} \|\nabla f_i(\x{t})\|
\le 2 \sqrt{L \fmax} + \\
&\qquad+ c \pars{L \sqrt{\log \nicefrac{T}{\delta}}
+ \sigma \sqrt{\log (n T)} + \sqrt{\sigma \log \nicefrac{1}{\delta}}
}
.
\end{align*}
\end{lemma}
We can now prove our main differential privacy bound.
\begin{theorem}
\label{thm:dp}
Let $\eta \le \min(1/L, 1 / (\sigma \sqrt{T}))$.
There exist $c_1, c_2 > 0$, such that for any $\delta > 0, \eps < c_1 T B^2 / n^2$, Algorithm~\ref{alg:dpsgd} is $(\eps, 2\delta)$-DP if $\Delta > c_2 \frac{C \sqrt{T \log \nicefrac{1}{\delta}}}{n \eps}$, where \\ $C = 2 \sqrt{L \fmax} + c \pars{L \sqrt{\log \nicefrac{T}{\delta}} + \sigma \sqrt{\log (n T)} + \sqrt{\sigma \log \nicefrac{1}{\delta}}}$.
% Then for $\Delta = c_2 \frac{q C \sqrt{T \log (1/\delta)}}{\eps}$, Algorithm~\ref{alg:dpsgd} is $(\eps, 2 \delta)$-differentially private.
\end{theorem}
\begin{proof}
%
% Note that the only difference between Algorithm~\ref{alg:dpsgd} and Algorithm~\ref{alg:old_dpsgd} is clipping.
From Theorem~\ref{thm:standard_dp}, we know that Algorithm~\ref{alg:old_dpsgd} is $(\eps,\delta)$-DP for our choice of $\Delta$.
For our Algorithm~\ref{alg:dpsgd}, let $\mathcal{E}$ be the event that all gradient norms are less than $C$.
By Lemma~\ref{lem:grad_bound}, all gradient norms are less than $C$ with probability at least $1 - \delta$, so $\mathcal{E}$ holds with high probability $1-\delta$.
% \mdnoteinline{The rest of this proof is weird to me for the reasons in my notes. Why can't we just replace it with the following:
So by \cref{thm:standard_dp}, if we condition on $\mathcal{E}$, then \cref{alg:dpsgd} is $(\epsilon, \delta)$-private.
On the other hand, if $\mathcal{E}$ doesn't hold, then we have no privacy guarantee, but this happens with probability at most $\delta$.
Hence, the overall privacy guarantee is $(\epsilon, 2\delta)$ as claimed.
% }
%
% Then the operation to query the status of $X$ in Algorithm~\ref{alg:dpsgd}\mdnote{There is no such operation in the algorithm} is $(0,\delta)$-differentially private based on the definition of approximate differential privacy.
% Also for our choice of $\Delta$, each iteration is $(\eps'=O(q\eps), q\delta)$-differentially private when $X=1$, namely in total $(\eps=O(\sqrt{T}q\eps),\delta)$-differentially private for $T$ iterations when $X=1$ based on the Moments accountant in~\citet{abadi_DeepLearning_2016}\mdnote{Why are we talking about iterations rather than using Theorem~\ref{thm:standard_dp} as a black box?}.
% So Algorithm~\ref{alg:dpsgd} is $(\eps+0=\eps, \delta+\delta=2\delta)$-DP for any $\delta > 0$ if $\Delta > c_2 \frac{C \sqrt{T \log \nicefrac{1}{\delta}}}{n \eps}$ by sequential composition theorem of differential privacy.
%
% Hence, with probability at least $1-\delta$ clipping never happens, and Algorithm~\ref{alg:dpsgd} coincides with Algorithm~\ref{alg:old_dpsgd}.
% We select the clipping threshold to be larger than the largest potential gradient norm (w.h.p.), and select the noise variance $\Delta^2$ based on this threshold, then the clipping is never needed, and the resulting algorithm is $(\eps,\delta)$-differentially private.
\end{proof}
%\todo{Expand and explain the proof}
\subsection{Outline of the Convergence Analysis}
\label{ssec:convergence}
In this section, we show that, for a fixed $\Delta^2$, for any $\alpha$, and for a certain choice of $\step$ and $T$, Algorithm~\ref{alg:dpsgd} finds an $\alpha$-SOSP.
In Section~\ref{ssec:combine}, we show for which $\alpha$ we guarantee both differential privacy and convergence.
Our main result shows that at least a constant fraction of iterates are $\alpha$-SOSP.
\begin{theorem}
\label{thm:convergence_main}
Under Assumption~\ref{ass:main}, when $\Delta^2 \ge \sigma^2 / B$ and $\step = \tOh{\alpha^2 / d \Delta^2}$
for $T = 64 \fmax / (\step \alpha^2)$, w.h.p. at least half of visited iterates of \Cref{alg:dpsgd} are $\alpha$-SOSP.
% \mdnote{Has $\fmax$ been defined?}
\end{theorem}
% \mdnoteinline{Should the $O$ be $\Omega$ in this theorem?}
%
We would like to point out that the fraction $1/2$ is somewhat arbitrary; we can guarantee that any constant fraction of the iterates are $\alpha$-SOSP, at the cost of increased number of iterations.
The full proof of this theorem can be found in Appendix~\ref{app:convergence_proof}; we outline the proof here.
% Note that the analysis assumes that the $\Delta^2$ is fixed, and in Section~\ref{ssec:combine} we show how to combine differential privacy with .
The proof of convergence follows the general structure of that of~\citet{jin_NonconvexOptimization_2021}.
However, unlike the standard stochastic gradient settings, when we select the noise variance $\Delta^2$ guaranteeing the best convergence rate, for noisy SGD the variance must be chosen according to Theorem~\ref{thm:dp}.
Moreover, by construction, the noise is large compared with the gradients: its norm is proportional to $C \sqrt{d}$, where $C$ upper-bounds gradient norms.
Hence, our analysis must be able to handle the case when the noise is large.
% Compared with the~\citet{jin_NonconvexOptimization_2021}, our analysis requires a significantly larger noise magnitude.
% In~\citet{jin_NonconvexOptimization_2021}, there are two type of noise: ``bad'' stochastic noise and ``good'' artificial Gaussian noise.
% For example, when the ``bad'' stochastic noise is $0$, the ``good'' artificial noise is selected sufficiently small so that the number of iterations depends on $\alpha$ as $\alpha^{-2}$ instead of $\alpha^{-4}$.
% On the other hand, preserving privacy requires a substantial amount of noise, which inevitably affects convergence.
In the large noise settings, we can assume smoothness of the objective function instead of smoothness of each component function.
\citet{jin_NonconvexOptimization_2021} require $\tilde{O}(\sigma^2 / \alpha^4)$ iterations to find an $\alpha$-SOSP when component function smoothness is assumed, and $\tilde{O}(d \sigma^2 / \alpha^4)$ iterations when only the objective function smoothness is assumed.
In our case, due to large noise, we don't lose the $d$ factor in the convergence rate, allowing us to get the standard $\nicefrac{d^{1/4}}{\sqrt{n \eps}}$ bound on $\alpha$.
Similarly, while \Cref{ass:grad_concentration} is required in \Cref{lem:grad_bound}, contrary to \citet{jin_NonconvexOptimization_2021}, in \Cref{thm:convergence_main} the assumption is not necessary when the noise is sufficiently large.
See Appendix~\ref{app:no_nSG} for the details.
% \todo[inline]{Statements}
% \begin{lemma}
% \label{lem:quarter_saddle_main}
% Under Assumptions~\ref{ass:main} and~\ref{ass:grad_concentration},
% for Algorithm~\ref{alg:dpsgd} with $\step = \tilde O \pars{\nicefrac{\alpha^2}{(L \Delta^2)}}$,
% after $T = O \pars{\nicefrac{\fmax}{(\step \alpha^2)}}$ iterations
% w.h.p. we have $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ for at most $\nicefrac{T}{4}$ points $\x{t}$.
% \end{lemma}
% We prove the following convergence rate.
%
\begin{proof}[Proof sketch]
We independently bound the number of points with gradient norm larger than $\alpha$ (Lemma~\ref{lem:quarter_large_grad}) and the number of points with Hessian having an eigenvalue less than $-\sqrtra$ (Lemma~\ref{lem:quarter_saddle}).
In Appendix~\ref{app:convergence_proof}, we prove that there are at most $\nicefrac{T}{4}$ points of the first type, and at most $\nicefrac{T}{4}$ points of the second type, which implies that at least $\nicefrac{T}{2}$ points are $\alpha$-SOSP.
The main idea is to show that, if a point is not an $\alpha$-SOSP, then w.h.p. the objective value decreases by at least $\df = \tOm{\alpha^{3/2}}$ after at most $\escIter = \tOh{\nicefrac{1}{\step \sqrt{\alpha}}}$ iterations.
% \mdnote{What's $\df$?}
It implies that w.h.p., if more than half of the points are not $\alpha$-SOSP, after $T$ iterations the objective value decreases by at least $\frac{\df T}{2\escIter} = \tOm{\frac{\alpha^{3/2} \fmax / (\step \alpha^2)}{1 / (\step \sqrt{\alpha})}} = \tOm{\fmax}$.
We choose our parameters so that this value is strictly greater than $\fmax$, which leads to contradiction.
\paragraph{Large gradients} For the points with gradients larger than $\alpha$, the analysis is similar to standard SGD analysis.
We show that the function value decreases by $\df$ after one iteration w.h.p.
By the folklore Descent Lemma, we have:
\begin{align*}
f(\x{t}) - f(\x{t+1})
&\ge \InnerProd{\nabla f(\x{t}), \x{t} - \x{t+1}}
- \frac{L}{2} \SqrNorm{\x{t} - \x{t+1}}
\end{align*}
Using that $\Exp{\x{t} - \x{t+1}} = \step \nabla f(\x{t})$ and $\ExpSqrNorm{\x{t} - \x{t+1}} = \step^2 (\SqrNorm{\nabla f(\x{t})} + \tsigma^2)$, where $\tsigma^2 = d \Delta^2 + \nicefrac{\sigma^2}{B}$ is the full noise variance, we have
\begin{align*}
\E[f(\x{t}) - f(\x{t+1})] &\ge \pars{\step - L \step^2} \|\nabla f(\x{t})\|^2 - \frac{L \step^2}{2} \tsigma^2
\end{align*}
Hence, for $\step \le \min(1, \nicefrac{\alpha^2}{\tsigma^2}) / (2L)$, if $\|\nabla f(\x{t})\| \ge \alpha$, the function decreases by $\step \alpha^2 / 4$ in expectation.
In Lemma~\ref{lem:descent} we prove a high-probability version of this result.
\paragraph{Saddle points}
% For points whose Hessian has an eigenvalue less than $-\sqrtra$, we use the fact that the gradient descent performs similarly to the power method, up to stochastic and quadratic approximation errors.
We first consider the case when the objective function is quadratic, i.e. $f(\vx) = \vx^\top \vH \vx / 2$.
Then the gradient step is
\begin{align*}
\x{t+1} = \x{t} - \step \nabla f(\x{t}) = (\id - \step \vH) \x{t}
.
\end{align*}
Hence, gradient descent emulates power method for matrix $\id - \step \vH$.
Since $L$ is the smoothness constant, it is an upper bound on eigenvalues of the Hessian $\vH$, and by choosing $\step < \nicefrac{1}{L}$ we guarantee that $\id - \step \vH$ is positive definite.
Hence, the smallest eigenvalue $-\gamma$ of $\vH$ corresponds to the largest eigenvalue $1 + \step \gamma > 1$ of $\id - \step \vH$, with the same eigenvector $\vv_1$.
Hence, we know that after $\escIter$ iterations, the projection of $\x{\escIter}$ on $\vv_1$ increases by a factor of $(1 + \step \gamma)^\escIter \approx \exp(\escIter \step \gamma)$.
Conversely, all positive eigenvalues of $\vH$ correspond to the eigenvalues between $0$ and $1$ of $1 - \step \vH$, and hence the projection on their eigenvectors is vanishing.
% Since our goal
% To conclude, after $\tOm{\escIter}$
The above reasoning shows that gradient descent for quadratic functions escapes from saddle points.
For general functions, we need to take into account stochastic gradients and quadratic approximation error.
The former issue is addressed by selecting a sufficiently small step size $\step \le \nicefrac{\alpha^2}{L \tsigma^2}$ so that the stochastic noise is smoothed.
The latter issue is addressed by ensuring that the objective function improves while iterates stay in a ball where the Hessian doesn't change significantly.
Using that $\gamma \ge \sqrtra$, this motivates the choice of parameters $\df$ and $\escIter$:
\begin{compactitem}
\item
Since the Hessian is $\rho$-Lipschitz and the smallest eigenvalue is at most $-\sqrtra$, by ensuring that we stay in a ball of radius $\escRad = \sqrt{\alpha / 4\rho} = \Theta(\sqrt{\alpha})$, we guarantee that the Hessian has an eigenvalue at most $-\sqrtra / 2$.
\item
In this ball, the best objective improvement $\df$ we can hope for is if we move in the direction of $\vv_1$, which is $\sqrtra \escRad^2 = \Omega(\alpha^{3/2})$.
\item
As in calculations above, to reach the boundary of the ball, we need $\exp(\escIter \step \sqrtra) = \Theta(\escRad)$, leading to $\escIter = \tOh{1 / \step \sqrt{\alpha}}$.
\end{compactitem}
More formally, we use the fact that, if the point moves by distance $\escRad$ within $\escIter$ iterations, then the objective value improves by $\Omega(\df)$ (Corollary~\ref{cor:i_or_l}).
We give the complete proof in Appendix~\ref{app:convergence_proof}.
\end{proof}
\subsection{Differentially Private Escaping from Saddle Points}
\label{ssec:combine}
In this section, we prove our main result, which combines the second-order convergence and differential privacy.
Intuitively, in Section~\ref{ssec:convergence} we showed that for any fixed $\Delta^2$, for any $\alpha$ we can select $\step$ and $T$ which guarantee the second-order convergence.
However, $\Delta^2$ is not fixed and depends on $T$ (see Theorem~\ref{thm:dp}), which leads to circular dependency: the number of iterations $T$ grows as variance $\Delta^2$ grows, and $\Delta^2$ grows as $T$ grows.
Hence, it is not a priori clear that it is possible to guarantee both differential privacy and second-order convergence simultaneously.
Nevertheless, since the number of iterations also depends on $\alpha$, we show that for a sufficiently large (but still quite small) $\alpha$ it is possible to guarantee both differential privacy and convergence to an $\alpha$-SOSP.
\begin{theorem}
\label{thm:combine_main}
Let Assumptions~\ref{ass:main} and~\ref{ass:grad_concentration} be satisfied, and let $\Delta$ satisfy conditions in Theorem~\ref{thm:dp} as well as $\Delta^2 \ge \sigma^2 / B$.
Then for $\step = \tOh{\alpha^2 / (d \Delta^2)}$, Algorithm~\ref{alg:dpsgd} is $(\eps, 2\delta)$-DP and after $T = 64 \fmax / (\step \alpha^2)$ iterations,
% \mdnote{Why not just set $T$ to this?}
w.h.p. at least half of the points are $\alpha$-SOSP for $\alpha = \tilde{\Omega} \pars{\frac{\fmax d^{1/4}}{\sqrt{n \eps}}}$.
\end{theorem}
\begin{proof}
From Theorem~\ref{thm:dp} we know that, to guarantee that Algorithm~\ref{alg:dpsgd} is $(\eps,\delta)$-differentially private, we must select $\Delta > c_2 \frac{C \sqrt{T \log \nicefrac{1}{\delta}}}{n \eps}$, where $C$ is selected as in Theorem~\ref{thm:dp}.
% $
% C = 2 \sqrt{L \fmax} + c \pars{L \sqrt{\log \nicefrac{T}{\delta}} + \theta \sqrt{\log (n T)} + \sqrt{\theta \log \nicefrac{1}{\delta}}}
% .
% $
% For the large value of $T$\mdnote{What does ``large'' mean here?}, the above expression is dominated by the first term, and hence we can bound $C \le 3 \sqrt{L \fmax}$.
% \todo[inline]{Fix this}
We need to choose $\alpha$ so that such $\Delta$ exists.
% By assuming that $\frac{\sigma^2}{B} \le (\sigma^2/B + d \Delta^2)$, we have:
Recall that
\[
T = \frac{64 \fmax}{\step \alpha^2}
= c \frac{L \fmax (\sigma^2/B + d \Delta^2)}{\alpha^4}
.
\]
for some constant $c$.
It suffices to choose $\alpha$ so that for some constant $c$
\[
\Delta > c \frac{C \sqrt{L \fmax (\sigma^2/B + d \Delta^2) \log \nicefrac{1}{\delta}}}{n \eps \alpha^2}
% \Delta > c \frac{L \fmax \cdot \sqrt{d \Delta^2 \log \nicefrac{1}{\delta}}}{n \eps \alpha^2}
,
\]
Using $\Delta^2 \ge \sigma^2 / B$, eliminating $\Delta$ on both sides, and rearranging the terms,
the inequality is satisfied if we choose $\alpha = \tOm{\frac{\fmax d^{1/4}}{\sqrt{n \eps}}}$.
\end{proof}
\subsection{Differentially Private Selection of SOSP}
\label{ssec:high_prob}
While Theorem~\ref{thm:combine_main} shows that a substantial fraction of points are SOSP\footnote{This is common in analysis of SGD for non-convex functions. For example, the folklore analysis bounds $\avg_{t=0}^{T-1} \SqrNorm{\nabla f(\x{t})}$, implying that most of the iterates have small gradients}, we still need to find one of these points.
The simplest approach is to sample random points and check that the point is a SOSP by computing the norm of the gradient and the eigenvalue of the Hessian.
To preserve privacy, we use \textsc{AboveThreshold} algorithm by~\citet{dwork_AlgorithmicFoundations_2014}.
By adding Laplacian noise $\text{Lap}\pars{\nicefrac{C}{n \eps}}$ to the gradient norm and Laplacian noise $\text{Lap}(\nicefrac{L}{n \eps})$ to the smallest eigenvalue of the Hessian, we guarantee privacy.
Since $\alpha$ depends on $n$ as $n^{-1/2}$, $\sqrt{\alpha}$ depends on $n$ as $n^{-1/4}$, and noise depends on $n$ as $n^{-1}$, for a sufficiently large $n$, namely $n = \tOm{(\eps\fmax^2 \sqrt{d})^{-1}}$ we approximate the gradient norm up to additive error $\nicefrac{\alpha}{2}$, and the eigenvalue up to additive error $\nicefrac{\sqrtra}{2}$.
We note that the smallest eigenvalue can be computed efficiently.
Many machine learning frameworks allow efficient computation of stochastic Hessian-vector products~-- that is, for any vector $\vv$, one can compute $\nabla^2 f_i(\vx) \vv$~-- using a constant number of back-propagation passes.
With this, we can find the smallest eigenvalue using stochastic variants of the power method, see e.g.~\citet{oja1985stochastic}.
% For the case when $n \eps > 1$, the noise is less than $ $
% Computing them privately with sufficient precision is possible only when $n$ is very large compared with $\alpha$.
% \todo[inline]{We can actually add noise}
% An alternative solution is based on the idea that, when $\vx$ is a saddle point, then with at least probability $\nicefrac 14$, we move by a certain distance $\escRad$ from $\vx$ after certain number ($\escIter$) of iterations.
\newpage
\section*{NeurIPS Paper Checklist}
%%% END INSTRUCTIONS %%%
\begin{enumerate}
\item {\bf Claims}
\item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: See~\Cref{thm:combine_main} and \Cref{ssec:high_prob}
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the abstract and introduction do not include the claims made in the paper.
\item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
\item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
\item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
\item {\bf Limitations}
\item[] Question: Does the paper discuss the limitations of the work performed by the authors?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: All assumptions are stated in \Cref{sec:pre}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
\item The authors are encouraged to create a separate "Limitations" section in their paper.
\item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
\item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
\item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
\item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
\item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
\item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
\item {\bf Theory Assumptions and Proofs}
\item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: All assumptions are stated in \Cref{sec:pre}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include theoretical results.
\item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
\item All assumptions should be clearly stated or referenced in the statement of any theorems.
\item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
\item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
\item Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
\item {\bf Experimental Result Reproducibility}
\item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: See \Cref{sec:experiments,app:experimental_details}
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
\item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
\item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
\item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
\item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
\item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
\item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
\item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
\item {\bf Open access to data and code}
\item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
\item[] Answer: \answerNo{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: All data is public, and we plan to release the code in the future. For reviewing purposes, the code is provided in the supplementary.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that paper does not include experiments requiring code.
\item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
\item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
\item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
\item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
\item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
\item {\bf Experimental Setting/Details}
\item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We use the datasets as is. The algorithm is as described in the paper. Additional details are in \Cref{app:experimental_details}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
\item The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
\item {\bf Experiment Statistical Significance}
\item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: For all plots, the captions explain what the data points and the error bars mean. We report the mean over the $10$ runs as well as minimum and maximum values over the runs.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
\item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
\item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
\item The assumptions made should be given (e.g., Normally distributed errors).
\item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
\item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
\item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
\item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
\item {\bf Experiments Compute Resources}
\item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We specified the training environment.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
\item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
\item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
\item {\bf Code Of Ethics}
\item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The paper conforms with the Code of Ethics.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
\item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
\item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
\item {\bf Broader Impacts}
\item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We don't expect the paper to have a direct societal impact
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that there is no societal impact of the work performed.
\item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
\item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
\item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
\item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
\item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
\item {\bf Safeguards}
\item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: None of the outlined scenarios applies
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper poses no such risks.
\item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
\item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
\item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
\item {\bf Licenses for existing assets}
\item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We cite all the datasets used in the paper.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not use existing assets.
\item The authors should cite the original paper that produced the code package or dataset.
\item The authors should state which version of the asset is used and, if possible, include a URL.
\item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
\item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
\item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
\item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
\item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
\item {\bf New Assets}
\item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The paper doesn't release new assets
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not release new assets.
\item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
\item The paper should discuss whether and how consent was obtained from people whose asset is used.
\item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
\item {\bf Crowdsourcing and Research with Human Subjects}
\item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The paper does not involve crowdsourcing nor research with human subjects
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
\item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
\item {\bf Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects}
\item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The paper does not involve crowdsourcing nor research with human subjects
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
\item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
\item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
\end{enumerate}
\section{Conclusion}
We have shown that DP-SGD, even without gradient clipping, converges to a second-order stationary point under minimal assumptions; in particular, without assuming that the loss function is Lipschitz. As DP-SGD is the workhorse of modern private optimization, this fundamentally improves our understanding of private optimization.
Moreover, we showed that experimentally, for the values of $\eps$ considered in~\citet{abadi_DeepLearning_2016}, on a standard CIFAR-10 dataset our performance is comparable to the \emph{non}-private classical SGD algorithm.
%\textbf{Some future directions.} An important research direction is proving convergence without the light-tail assumptions.
%Another interesting questions is whether it is possible to further improve the utility bound with light tail assumption.
%\todo[inline]{Open questions?}
%%%%%%%% ICML 2024 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2024} with \usepackage[nohyperref]{icml2024} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
% Use the following line for the initial blind version submitted for review:
\usepackage{icml2024}
% If accepted, instead use the following line for the camera-ready submission:
% \usepackage[accepted]{icml2024}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Submission and Formatting Instructions for ICML 2024}
\begin{document}
\twocolumn[
\icmltitle{Submission and Formatting Instructions for \\
International Conference on Machine Learning (ICML 2024)}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2024
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Firstname1 Lastname1}{equal,yyy}
\icmlauthor{Firstname2 Lastname2}{equal,yyy,comp}
\icmlauthor{Firstname3 Lastname3}{comp}
\icmlauthor{Firstname4 Lastname4}{sch}
\icmlauthor{Firstname5 Lastname5}{yyy}
\icmlauthor{Firstname6 Lastname6}{sch,yyy,comp}
\icmlauthor{Firstname7 Lastname7}{comp}
%\icmlauthor{}{sch}
\icmlauthor{Firstname8 Lastname8}{sch}
\icmlauthor{Firstname8 Lastname8}{yyy,comp}
%\icmlauthor{}{sch}
%\icmlauthor{}{sch}
\end{icmlauthorlist}
\icmlaffiliation{yyy}{Department of XXX, University of YYY, Location, Country}
\icmlaffiliation{comp}{Company Name, Location, Country}
\icmlaffiliation{sch}{School of ZZZ, Institute of WWW, Location, Country}
\icmlcorrespondingauthor{Firstname1 Lastname1}{first1.last1@xxx.edu}
\icmlcorrespondingauthor{Firstname2 Lastname2}{first2.last2@www.uk}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
%\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
\printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
\begin{abstract}
This document provides a basic paper template and submission guidelines.
Abstracts must be a single paragraph, ideally between 4--6 sentences long.
Gross violations will trigger corrections at the camera-ready phase.
\end{abstract}
\section{Electronic Submission}
\label{submission}
Submission to ICML 2024 will be entirely electronic, via a web site
(not email). Information about the submission process and \LaTeX\ templates
are available on the conference web site at:
\begin{center}
\textbf{\texttt{http://icml.cc/}}
\end{center}
The guidelines below will be enforced for initial submissions and
camera-ready copies. Here is a brief summary:
\begin{itemize}
\item Submissions must be in PDF\@.
\item \textbf{New to this year}: If your paper has appendices, submit the appendix together with the main body and the references \textbf{as a single file}. Reviewers will not look for appendices as a separate PDF file. So if you submit such an extra file, reviewers will very likely miss it.
\item Page limit: The main body of the paper has to be fitted to 8 pages, excluding references and appendices; the space for the latter two is not limited. For the final version of the paper, authors can add one extra page to the main body.
\item \textbf{Do not include author information or acknowledgements} in your
initial submission.
\item Your paper should be in \textbf{10 point Times font}.
\item Make sure your PDF file only uses Type-1 fonts.
\item Place figure captions \emph{under} the figure (and omit titles from inside
the graphic file itself). Place table captions \emph{over} the table.
\item References must include page numbers whenever possible and be as complete
as possible. Place multiple citations in chronological order.
\item Do not alter the style template; in particular, do not compress the paper
format by reducing the vertical spaces.
\item Keep your abstract brief and self-contained, one paragraph and roughly
4--6 sentences. Gross violations will require correction at the
camera-ready phase. The title should have content words capitalized.
\end{itemize}
\subsection{Submitting Papers}
\textbf{Paper Deadline:} The deadline for paper submission that is
advertised on the conference website is strict. If your full,
anonymized, submission does not reach us on time, it will not be
considered for publication.
\textbf{Anonymous Submission:} ICML uses double-blind review: no identifying
author information may appear on the title page or in the paper
itself. \cref{author info} gives further details.
\textbf{Simultaneous Submission:} ICML will not accept any paper which,
at the time of submission, is under review for another conference or
has already been published. This policy also applies to papers that
overlap substantially in technical content with conference papers
under review or previously published. ICML submissions must not be
submitted to other conferences and journals during ICML's review
period.
%Authors may submit to ICML substantially different versions of journal papers
%that are currently under review by the journal, but not yet accepted
%at the time of submission.
Informal publications, such as technical
reports or papers in workshop proceedings which do not appear in
print, do not fall under these restrictions.
\medskip
Authors must provide their manuscripts in \textbf{PDF} format.
Furthermore, please make sure that files contain only embedded Type-1 fonts
(e.g.,~using the program \texttt{pdffonts} in linux or using
File/DocumentProperties/Fonts in Acrobat). Other fonts (like Type-3)
might come from graphics files imported into the document.
Authors using \textbf{Word} must convert their document to PDF\@. Most
of the latest versions of Word have the facility to do this
automatically. Submissions will not be accepted in Word format or any
format other than PDF\@. Really. We're not joking. Don't send Word.
Those who use \textbf{\LaTeX} should avoid including Type-3 fonts.
Those using \texttt{latex} and \texttt{dvips} may need the following
two commands:
{\footnotesize
\begin{verbatim}
dvips -Ppdf -tletter -G0 -o paper.ps paper.dvi
ps2pdf paper.ps
\end{verbatim}}
It is a zero following the ``-G'', which tells dvips to use
the config.pdf file. Newer \TeX\ distributions don't always need this
option.
Using \texttt{pdflatex} rather than \texttt{latex}, often gives better
results. This program avoids the Type-3 font problem, and supports more
advanced features in the \texttt{microtype} package.
\textbf{Graphics files} should be a reasonable size, and included from
an appropriate format. Use vector formats (.eps/.pdf) for plots,
lossless bitmap formats (.png) for raster graphics with sharp lines, and
jpeg for photo-like images.
The style file uses the \texttt{hyperref} package to make clickable
links in documents. If this causes problems for you, add
\texttt{nohyperref} as one of the options to the \texttt{icml2024}
usepackage statement.
\subsection{Submitting Final Camera-Ready Copy}
The final versions of papers accepted for publication should follow the
same format and naming convention as initial submissions, except that
author information (names and affiliations) should be given. See
\cref{final author} for formatting instructions.
The footnote, ``Preliminary work. Under review by the International
Conference on Machine Learning (ICML). Do not distribute.'' must be
modified to ``\textit{Proceedings of the
$\mathit{41}^{st}$ International Conference on Machine Learning},
Vienna, Austria, PMLR 235, 2024.
Copyright 2024 by the author(s).''
For those using the \textbf{\LaTeX} style file, this change (and others) is
handled automatically by simply changing
$\mathtt{\backslash usepackage\{icml2024\}}$ to
$$\mathtt{\backslash usepackage[accepted]\{icml2024\}}$$
Authors using \textbf{Word} must edit the
footnote on the first page of the document themselves.
Camera-ready copies should have the title of the paper as running head
on each page except the first one. The running title consists of a
single line centered above a horizontal rule which is $1$~point thick.
The running head should be centered, bold and in $9$~point type. The
rule should be $10$~points above the main text. For those using the
\textbf{\LaTeX} style file, the original title is automatically set as running
head using the \texttt{fancyhdr} package which is included in the ICML
2024 style file package. In case that the original title exceeds the
size restrictions, a shorter form can be supplied by using
\verb|\icmltitlerunning{...}|
just before $\mathtt{\backslash begin\{document\}}$.
Authors using \textbf{Word} must edit the header of the document themselves.
\section{Format of the Paper}
All submissions must follow the specified format.
\subsection{Dimensions}
The text of the paper should be formatted in two columns, with an
overall width of 6.75~inches, height of 9.0~inches, and 0.25~inches
between the columns. The left margin should be 0.75~inches and the top
margin 1.0~inch (2.54~cm). The right and bottom margins will depend on
whether you print on US letter or A4 paper, but all final versions
must be produced for US letter size.
Do not write anything on the margins.
The paper body should be set in 10~point type with a vertical spacing
of 11~points. Please use Times typeface throughout the text.
\subsection{Title}
The paper title should be set in 14~point bold type and centered
between two horizontal rules that are 1~point thick, with 1.0~inch
between the top rule and the top edge of the page. Capitalize the
first letter of content words and put the rest of the title in lower
case.
\subsection{Author Information for Submission}
\label{author info}
ICML uses double-blind review, so author information must not appear. If
you are using \LaTeX\/ and the \texttt{icml2024.sty} file, use
\verb+\icmlauthor{...}+ to specify authors and \verb+\icmlaffiliation{...}+ to specify affiliations. (Read the TeX code used to produce this document for an example usage.) The author information
will not be printed unless \texttt{accepted} is passed as an argument to the
style file.
Submissions that include the author information will not
be reviewed.
\subsubsection{Self-Citations}
If you are citing published papers for which you are an author, refer
to yourself in the third person. In particular, do not use phrases
that reveal your identity (e.g., ``in previous work \cite{langley00}, we
have shown \ldots'').
Do not anonymize citations in the reference section. The only exception are manuscripts that are
not yet published (e.g., under submission). If you choose to refer to
such unpublished manuscripts \cite{anonymous}, anonymized copies have
to be submitted
as Supplementary Material via OpenReview\@. However, keep in mind that an ICML
paper should be self contained and should contain sufficient detail
for the reviewers to evaluate the work. In particular, reviewers are
not required to look at the Supplementary Material when writing their
review (they are not required to look at more than the first $8$ pages of the submitted document).
\subsubsection{Camera-Ready Author Information}
\label{final author}
If a paper is accepted, a final camera-ready copy must be prepared.
%
For camera-ready papers, author information should start 0.3~inches below the
bottom rule surrounding the title. The authors' names should appear in 10~point
bold type, in a row, separated by white space, and centered. Author names should
not be broken across lines. Unbolded superscripted numbers, starting 1, should
be used to refer to affiliations.
Affiliations should be numbered in the order of appearance. A single footnote
block of text should be used to list all the affiliations. (Academic
affiliations should list Department, University, City, State/Region, Country.
Similarly for industrial affiliations.)
Each distinct affiliations should be listed once. If an author has multiple
affiliations, multiple superscripts should be placed after the name, separated
by thin spaces. If the authors would like to highlight equal contribution by
multiple first authors, those authors should have an asterisk placed after their
name in superscript, and the term ``\textsuperscript{*}Equal contribution"
should be placed in the footnote block ahead of the list of affiliations. A
list of corresponding authors and their emails (in the format Full Name
\textless{}email@domain.com\textgreater{}) can follow the list of affiliations.
Ideally only one or two names should be listed.
A sample file with author names is included in the ICML2024 style file
package. Turn on the \texttt{[accepted]} option to the stylefile to
see the names rendered. All of the guidelines above are implemented
by the \LaTeX\ style file.
\subsection{Abstract}
The paper abstract should begin in the left column, 0.4~inches below the final
address. The heading `Abstract' should be centered, bold, and in 11~point type.
The abstract body should use 10~point type, with a vertical spacing of
11~points, and should be indented 0.25~inches more than normal on left-hand and
right-hand margins. Insert 0.4~inches of blank space after the body. Keep your
abstract brief and self-contained, limiting it to one paragraph and roughly 4--6
sentences. Gross violations will require correction at the camera-ready phase.
\subsection{Partitioning the Text}
You should organize your paper into sections and paragraphs to help
readers place a structure on the material and understand its
contributions.
\subsubsection{Sections and Subsections}
Section headings should be numbered, flush left, and set in 11~pt bold
type with the content words capitalized. Leave 0.25~inches of space
before the heading and 0.15~inches after the heading.
Similarly, subsection headings should be numbered, flush left, and set
in 10~pt bold type with the content words capitalized. Leave
0.2~inches of space before the heading and 0.13~inches afterward.
Finally, subsubsection headings should be numbered, flush left, and
set in 10~pt small caps with the content words capitalized. Leave
0.18~inches of space before the heading and 0.1~inches after the
heading.
Please use no more than three levels of headings.
\subsubsection{Paragraphs and Footnotes}
Within each section or subsection, you should further partition the
paper into paragraphs. Do not indent the first line of a given
paragraph, but insert a blank line between succeeding ones.
You can use footnotes\footnote{Footnotes
should be complete sentences.} to provide readers with additional
information about a topic without interrupting the flow of the paper.
Indicate footnotes with a number in the text where the point is most
relevant. Place the footnote in 9~point type at the bottom of the
column in which it appears. Precede the first footnote in a column
with a horizontal rule of 0.8~inches.\footnote{Multiple footnotes can
appear in each column, in the same order as they appear in the text,
but spread them across columns and pages if possible.}
\begin{figure}[ht]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{icml_numpapers}}
\caption{Historical locations and number of accepted papers for International
Machine Learning Conferences (ICML 1993 -- ICML 2008) and International
Workshops on Machine Learning (ML 1988 -- ML 1992). At the time this figure was
produced, the number of accepted papers for ICML 2008 was unknown and instead
estimated.}
\label{icml-historical}
\end{center}
\vskip -0.2in
\end{figure}
\subsection{Figures}
You may want to include figures in the paper to illustrate
your approach and results. Such artwork should be centered,
legible, and separated from the text. Lines should be dark and at
least 0.5~points thick for purposes of reproduction, and text should
not appear on a gray background.
Label all distinct components of each figure. If the figure takes the
form of a graph, then give a name for each axis and include a legend
that briefly describes each curve. Do not include a title inside the
figure; instead, the caption should serve this function.
Number figures sequentially, placing the figure number and caption
\emph{after} the graphics, with at least 0.1~inches of space before
the caption and 0.1~inches after it, as in
\cref{icml-historical}. The figure caption should be set in
9~point type and centered unless it runs two or more lines, in which
case it should be flush left. You may float figures to the top or
bottom of a column, and you may set wide figures across both columns
(use the environment \texttt{figure*} in \LaTeX). Always place
two-column figures at the top or bottom of the page.
\subsection{Algorithms}
If you are using \LaTeX, please use the ``algorithm'' and ``algorithmic''
environments to format pseudocode. These require
the corresponding stylefiles, algorithm.sty and
algorithmic.sty, which are supplied with this package.
\cref{alg:example} shows an example.
\begin{algorithm}[tb]
\caption{Bubble Sort}
\label{alg:example}
\begin{algorithmic}
\STATE {\bfseries Input:} data $x_i$, size $m$
\REPEAT
\STATE Initialize $noChange = true$.
\FOR{$i=1$ {\bfseries to} $m-1$}
\IF{$x_i > x_{i+1}$}
\STATE Swap $x_i$ and $x_{i+1}$
\STATE $noChange = false$
\ENDIF
\ENDFOR
\UNTIL{$noChange$ is $true$}
\end{algorithmic}
\end{algorithm}
\subsection{Tables}
You may also want to include tables that summarize material. Like
figures, these should be centered, legible, and numbered consecutively.
However, place the title \emph{above} the table with at least
0.1~inches of space before the title and the same after it, as in
\cref{sample-table}. The table title should be set in 9~point
type and centered unless it runs two or more lines, in which case it
should be flush left.
% Note use of \abovespace and \belowspace to get reasonable spacing
% above and below tabular lines.
\begin{table}[t]
\caption{Classification accuracies for naive Bayes and flexible
Bayes on various data sets.}
\label{sample-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lcccr}
\toprule
Data set & Naive & Flexible & Better? \\
\midrule
Breast & 95.9$\pm$ 0.2& 96.7$\pm$ 0.2& $\surd$ \\
Cleveland & 83.3$\pm$ 0.6& 80.0$\pm$ 0.6& $\times$\\
Glass2 & 61.9$\pm$ 1.4& 83.8$\pm$ 0.7& $\surd$ \\
Credit & 74.8$\pm$ 0.5& 78.3$\pm$ 0.6& \\
Horse & 73.3$\pm$ 0.9& 69.7$\pm$ 1.0& $\times$\\
Meta & 67.1$\pm$ 0.6& 76.5$\pm$ 0.5& $\surd$ \\
Pima & 75.1$\pm$ 0.6& 73.9$\pm$ 0.5& \\
Vehicle & 44.9$\pm$ 0.6& 61.5$\pm$ 0.4& $\surd$ \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}
Tables contain textual material, whereas figures contain graphical material.
Specify the contents of each row and column in the table's topmost
row. Again, you may float tables to a column's top or bottom, and set
wide tables across both columns. Place two-column tables at the
top or bottom of the page.
\subsection{Theorems and such}
The preferred way is to number definitions, propositions, lemmas, etc. consecutively, within sections, as shown below.
\begin{definition}
\label{def:inj}
A function $f:X \to Y$ is injective if for any $x,y\in X$ different, $f(x)\ne f(y)$.
\end{definition}
Using \cref{def:inj} we immediate get the following result:
\begin{proposition}
If $f$ is injective mapping a set $X$ to another set $Y$,
the cardinality of $Y$ is at least as large as that of $X$
\end{proposition}
\begin{proof}
Left as an exercise to the reader.
\end{proof}
\cref{lem:usefullemma} stated next will prove to be useful.
\begin{lemma}
\label{lem:usefullemma}
For any $f:X \to Y$ and $g:Y\to Z$ injective functions, $f \circ g$ is injective.
\end{lemma}
\begin{theorem}
\label{thm:bigtheorem}
If $f:X\to Y$ is bijective, the cardinality of $X$ and $Y$ are the same.
\end{theorem}
An easy corollary of \cref{thm:bigtheorem} is the following:
\begin{corollary}
If $f:X\to Y$ is bijective,
the cardinality of $X$ is at least as large as that of $Y$.
\end{corollary}
\begin{assumption}
The set $X$ is finite.
\label{ass:xfinite}
\end{assumption}
\begin{remark}
According to some, it is only the finite case (cf. \cref{ass:xfinite}) that is interesting.
\end{remark}
%restatable
\subsection{Citations and References}
Please use APA reference format regardless of your formatter
or word processor. If you rely on the \LaTeX\/ bibliographic
facility, use \texttt{natbib.sty} and \texttt{icml2024.bst}
included in the style-file package to obtain this format.
Citations within the text should include the authors' last names and
year. If the authors' names are included in the sentence, place only
the year in parentheses, for example when referencing Arthur Samuel's
pioneering work \yrcite{Samuel59}. Otherwise place the entire
reference in parentheses with the authors and year separated by a
comma \cite{Samuel59}. List multiple references separated by
semicolons \cite{kearns89,Samuel59,mitchell80}. Use the `et~al.'
construct only for citations with three or more authors or after
listing all authors to a publication in an earlier reference \cite{MachineLearningI}.
Authors should cite their own work in the third person
in the initial version of their paper submitted for blind review.
Please refer to \cref{author info} for detailed instructions on how to
cite your own papers.
Use an unnumbered first-level section heading for the references, and use a
hanging indent style, with the first line of the reference flush against the
left margin and subsequent lines indented by 10 points. The references at the
end of this document give examples for journal articles \cite{Samuel59},
conference publications \cite{langley00}, book chapters \cite{Newell81}, books
\cite{DudaHart2nd}, edited volumes \cite{MachineLearningI}, technical reports
\cite{mitchell80}, and dissertations \cite{kearns89}.
Alphabetize references by the surnames of the first authors, with
single author entries preceding multiple author entries. Order
references for the same authors by year of publication, with the
earliest first. Make sure that each reference includes all relevant
information (e.g., page numbers).
Please put some effort into making references complete, presentable, and
consistent, e.g. use the actual current name of authors.
If using bibtex, please protect capital letters of names and
abbreviations in titles, for example, use \{B\}ayesian or \{L\}ipschitz
in your .bib file.
\section*{Accessibility}
Authors are kindly asked to make their submissions as accessible as possible for everyone including people with disabilities and sensory or neurological differences.
Tips of how to achieve this and what to pay attention to will be provided on the conference website \url{http://icml.cc/}.
\section*{Software and Data}
If a paper is accepted, we strongly encourage the publication of software and data with the
camera-ready version of the paper whenever appropriate. This can be
done by including a URL in the camera-ready copy. However, \textbf{do not}
include URLs that reveal your institution or identity in your
submission for review. Instead, provide an anonymous URL or upload
the material as ``Supplementary Material'' into the OpenReview reviewing
system. Note that reviewers are not required to look at this material
when writing their review.
% Acknowledgements should only appear in the accepted version.
\section*{Acknowledgements}
\textbf{Do not} include acknowledgements in the initial version of
the paper submitted for blind review.
If a paper is accepted, the final camera-ready version can (and
probably should) include acknowledgements. In this case, please
place such acknowledgements in an unnumbered section at the
end of the paper. Typically, this will include thanks to reviewers
who gave useful comments, to colleagues who contributed to the ideas,
and to funding agencies and corporate sponsors that provided financial
support.
% In the unusual situation where you want a paper to appear in the
% references without citing it in the main text, use \nocite
\nocite{langley00}
\bibliography{example_paper}
\bibliographystyle{icml2024}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\onecolumn
\section{You \emph{can} have an appendix here.}
You can have as much text here as you want. The main body must be at most $8$ pages long.
For the final version, one more page can be added.
If you want, you can use an appendix like this one.
The $\mathtt{\backslash onecolumn}$ command above can be kept in place if you prefer a one-column appendix, or can be removed if you prefer a two-column appendix. Apart from this possible change, the style (font size, spacing, margins, page numbering, etc.) should be kept the same as the main body.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
% This document was modified from the file originally made available by
% Pat Langley and Andrea Danyluk for ICML-2K. This version was created
% by Iain Murray in 2018, and modified by Alexandre Bouchard in
% 2019 and 2021 and by Csaba Szepesvari, Gang Niu and Sivan Sabato in 2022.
% Modified again in 2023 and 2024 by Sivan Sabato and Jonathan Scarlett.
% Previous contributors include Dan Roy, Lise Getoor and Tobias
% Scheffer, which was slightly modified from the 2010 version by
% Thorsten Joachims & Johannes Fuernkranz, slightly modified from the
% 2009 version by Kiri Wagstaff and Sam Roweis's 2008 version, which is
% slightly modified from Prasad Tadepalli's 2007 version which is a
% lightly changed version of the previous year's version by Andrew
% Moore, which was in turn edited from those of Kristian Kersting and
% Codrina Lauth. Alex Smola contributed to the algorithmic style files.
\section{Experimental Details}
\label{app:experimental_details}
We perform experiments on NVIDIA RTX A5000.
To improve the performance of our experiments, we use FFCV library~\citep{leclerc2023ffcv}.
We use the Convolutional Neural Network with the following architecture:
\begin{lstlisting}
conv_bn(3, 64, kernel_size=3, stride=1, padding=1),
conv_bn(64, 128, kernel_size=5, stride=2, padding=2),
Residual(Sequential(conv_bn(128, 128), conv_bn(128, 128))),
conv_bn(128, 256, kernel_size=3, stride=1, padding=1),
MaxPool2d(2),
Residual(Sequential(conv_bn(256, 256), conv_bn(256, 256))),
conv_bn(256, 128, kernel_size=3, stride=1, padding=0),
AdaptiveMaxPool2d((1, 1)),
Flatten(),
Linear(128, 10, bias=False),
Mul(0.2)
\end{lstlisting}
where \lstinline{conv_bn}$(c_i, c_o, k, s, p)$ consists of 1) a \lstinline{Conv2d} layer with $c_i$ input channels, $c_o$ output channels, kernel size $k$, stride $s$, and padding $p$, 2) batch normalization layer, and 3) ReLU activation layer.
Aside from $\eps$ and $\delta$, our algorithm requires two hyperparameters: learning rate $\step$ and the Gaussian noise variance $\Delta^2$.
% Aside from the learning rate, another parameter of DP-SGD algorithm is $\Delta^2$, the noise variance.
We select the learning rate to be $0.1$, which is the largest value which still achieves convergence.
The parameter $\Delta^2$ is computed per Theorem~\ref{thm:dp} based on $C$~-- the bound on the gradient norms throughout the algorithm execution.
While $C$ can be computed based on Lemma~\ref{lem:grad_bound}, this bound is pessimistic; in practice, we choose $C$ to be the upper bound on the gradient norm, which, for training a convolutional neural network on CIFAR-10 dataset, equals $2.5$ (as we verify in Figure~\ref{fig:grad_norms}).
% \subsection{Additional Experimental Results}
% In this section, we present experimental results excluded from Section~\ref{sec:experiments}.
In Figure~\ref{fig:testing_C}, we show testing accuracy, but with gradient bound $C$ chosen as in Theorem~\ref{thm:dp}.
The figure shows the trend similar to Figure~\ref{fig:convergence}, with lower accuracies due to using a pessimistic value of $C$.
The experimental setup is the same as in Section~\ref{sec:experiments}, i.e. on CIFAR-10 dataset with $\delta=0.01$.
\begin{figure}[t!]
\centering
\begin{subfigure}[t]{0.45\textwidth}
\includegraphics[width=\textwidth]{pics/grad_norm.png}
\caption{
Training accuracy averaged over $10$ runs on CIFAR-10 dataset, with the shaded area showing the minimum and the maximum values over the runs.
}
\label{fig:grad_norms}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.45\textwidth}
\includegraphics[width=\textwidth]{pics/testing_C.png}
\caption{
Testing accuracy averaged over $10$ runs on CIFAR-10 dataset, with the shaded area showing the minimum and the maximum values over the runs.
Compared with Figure~\ref{fig:convergence}, we bound gradient norms using $C$ from Theorem~\ref{thm:dp}.
}
\label{fig:testing_C}
\end{subfigure}
\label{fig:additional_exp_details}
\caption{Additional results for training a convolutional neural network using DP-SGD on CIFAR-10 dataset}
\end{figure}
\section{Experiments}
\label{sec:experiments}
In this section, we give the experimental results.
We compare convergence of DP-SGD (Algorithm~\ref{alg:dpsgd}) for various choices of $\eps$ with convergence of SGD.
We train a neural network on CIFAR-10 and CIFAR-100 image datasets~\citep{krizhevsky_LearningMultiple_2009}, as well as CoLa corpus~\citep{warstadt_NeuralNetwork_2019}.
For CIFAR-10, we use a convolutional neural network (see Appendix~\ref{app:experimental_details} for the detailed outline of the experimental setup) and a ResNet-18 neural network.
For CIFAR-100, we use pretrained ResNet-18, and for CoLa dataset we use pretrained uncased base BERT model.
All experiments are performed on a single NVIDIA A100 GPU.
\begin{figure*}[t!]
\centering
\begin{subfigure}[t]{\subfigwidth\textwidth}
\includegraphics[width=\textwidth]{pics/DP.png}
\caption{
Convolutional model (see Appendix~\ref{app:experimental_details})
}
\label{fig:cifar10_cnn}
\end{subfigure}
\hfill
\begin{subfigure}[t]{\subfigwidth\textwidth}
\includegraphics[width=\textwidth]{pics/dp_cifar10_resnet_test.png}
\caption{
ResNet-18
}
\label{fig:cifar10_resnet}
\end{subfigure}
\caption{
Testing accuracy of DP-SGD on CIFAR-10 dataset for various choices of $\eps$.
Testing accuracy is averaged over $10$ runs, with the shaded area showing the minimum and the maximum values over the runs.
% For all experiments, $\delta=0.01$
}
\label{fig:experiments_cifar10}
\end{figure*}
\begin{figure*}[t!]
\centering
\begin{subfigure}[t]{\subfigwidth\textwidth}
\includegraphics[width=\textwidth]{pics/dp_cifar100_test.png}
\caption{
CIFAR-100 dataset
}
\label{fig:convergence}
\end{subfigure}
\hfill
\begin{subfigure}[t]{\subfigwidth\textwidth}
\includegraphics[width=\textwidth]{pics/dp_cola_test.png}
\caption{
CoLa dataset
}
\label{fig:train_acc}
\end{subfigure}
\caption{
Testing accuracy of DP-SGD on CIFAR-100 and CoLa datasets.
Testing accuracy is averaged over $10$ runs, with the shaded area showing the minimum and the maximum values.
% over the runs.
% For all experiments, $\delta=0.01$
}
\label{fig:experiments_cola_cifar100}
\end{figure*}
% \begin{figure}[t!]
% \centering
% \includegraphics[width=0.45\textwidth]{pics/DP.png}
% \caption{
% Convergence of DP-SGD for various choices of $\eps$ on CIFAR-10 dataset.
% We show testing accuracy averaged over $10$ runs, with the shaded area showing the minimum and the maximum values over the runs
% }
% \label{fig:convergence}
% \includegraphics[width=0.45\textwidth]{pics/DP_train.png}
% \caption{
% Training accuracy averaged over $10$ runs on CIFAR-10 dataset, with the shaded area showing the minimum and the maximum values over the runs
% }
% \label{fig:train_acc}
% \includegraphics[width=0.45\textwidth]{pics/grad_norm.png}
% \caption{
% Gradient norms averaged over $10$ runs on CIFAR-10 dataset, with the shaded area showing the minimum and the maximum values over the runs.
% For all differentially private settings, the gradient norm doesn't exceed $2.5$.
% }
% \label{fig:grad_norms}
% \end{figure}
% \begin{figure}[t!]
% \centering
% \begin{subfigure}[t]{0.48\textwidth}
% \includegraphics[width=\textwidth]{pics/DP_train.png}
% \caption{
% Training accuracy averaged over $10$ runs on CIFAR-10 dataset, with the shaded area showing the minimum and the maximum values over the runs
% }
% \label{fig:train_acc}
% \end{subfigure}
% \hfill
% \begin{subfigure}[t]{0.48\textwidth}
% \includegraphics[width=\textwidth]{pics/grad_norm.png}
% \caption{
% Gradient norms averaged over $10$ runs on CIFAR-10 dataset, with the shaded area showing the minimum and the maximum values over the runs.
% For all differentially private settings, the gradient norm doesn't exceed $2.5$.
% }
% \label{fig:grad_norms}
% \end{subfigure}
% \end{figure}
For our experiments, we choose error probability $\delta=0.01$ and learning rate $\step=0.1$.
We consider $\eps \in \{2, 4, 8\}$ similarly to~\citet{abadi_DeepLearning_2016}, as well as $\eps \in \{\nicefrac{1}{8}, \nicefrac{1}{4}, \nicefrac{1}{2}, 1\}$.
\Cref{fig:experiments_cifar10,fig:experiments_cifar10} shows testing accuracy for DP-SGD with the above choices of $\eps$, as well as testing accuracy of non-differentially private SGD.
Our results show monotone dependence on $\eps$, matching our prediction in Theorem~\ref{thm:combine_main} that for greater value of $\eps$ we find a better solution.
Moreover, for $\eps \in \{4, 8\}$, the testing accuracy is close to that of the non-differentially private SGD, which means that we obtain strong privacy guarantees without sacrificing the quality of the model.
Recall that the algorithm requires $C$~-- an upper bound on all gradient norms~-- which we choose empirically.
In Appendix~\ref{app:experimental_details} we present additional experimental results, including training accuracy and algorithm convergence when $C$ chosen as in \Cref{thm:dp}.
% Figure~\ref{fig:train_acc} shows training accuracy, and, similarly to Figure~\ref{fig:convergence}, the accuracy increases when $\eps$ increases, with $\eps \in \{4, 8\}$ resulting in the quality similar to the non-private settings.
% We show that the norm increases when $\eps$ increases.
% Recall that the algorithm requires $C$~-- an upper bound on all gradient norms~-- and Figure~\ref{fig:grad_norms} shows that $2.5$ is the upper bound for all differentially private settings in our experiments.
\section{Convergence Proof}
\begin{assumption}
\label{ass:main_new}
The objective function $f$ and the component functions $f_i$ satisfy the following conditions.
\begin{enumerate}
{\color{green}\item (Boundedness)
There exists $\fmax$ such that $\fmax > f(\x{0}) - f(\vx^*)$.}
{\color{green}\item (Smoothness) $f$ is $L$-smooth, i.e. $\norm{\nabla f(\vx) - \nabla f(\vy)} \le L \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.
Consequently, $f$ is also $L$-smooth.}
{\color{green}\item (Lipschitz Hessian) The objective function $f$ has a $\rho$-Lipschitz Hessian, i.e. $\norm{\nabla^2 f(\vx) - \nabla^2 f(\vy)} \le \rho \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.}
{\color{red}\item (Relaxed Stochastic variance) The stochastic variance is bounded as follows: $\avg_{i=1}^N \norm{\nabla f_i(\vx) - \nabla f(\vx)}^2 \le \sigma_1^2 \SqrNorm{\nabla f(\vx)} + \sigma_2^2$.}
\end{enumerate}
\end{assumption}
\todo{Rewrite}
\begin{assumption}
{\color{red}No nSG assumption.}
\end{assumption}
\begin{assumption}
{\color{red}Clipping.}
\end{assumption}
\label{app:convergence_proof}
\todo{Mention that mostly follows Jin et al.}
\begin{definition}
\label{def:notation}
We introduce the following auxiliary notation.
\begin{itemize}
\item Let $\fullVar = \sigma^2 / B + d \Delta^2$ be the total noise variance.
\item Let $\clipC$ be the clipping threshold. Can be set to infinity when there is no clipping.
\item Let $\clip{t}$ be clipping scaling at iteration $t$: $\clip{t} = \max(1, \|\nabla f_{\batch{t}}(\x{t})\| / \clipC)$.
\item Let $\nabla \fclip_{\batch{t}}(\x{t})$ be clipped stochastic gradient at iteration $t$: $\nabla \fclip_{\batch{t}}(\x{t}) = \nabla f_{\batch{t}}(\x{t}) / \clip{t}$.
\end{itemize}
\end{definition}
\begin{definition}
\label{def:params}
Our choice of parameters is the following ($\cstep, \cit, \crad, \cf$ hide polylogarithmic dependence on all parameters):
\begin{equation*}
\begin{aligned}
\text{Step size} && \step &= \cstep \min\pars{\frac{\alpha^2}{L (1 + \tsigma^2)}, \frac{\fmax}{L \tsigma^2}} \\
\text{Total number of iterations} && T &= \frac{64\fmax}{\step \alpha^2} \\
\text{Iterations required for escaping} && \escIter &= \cit \frac {1} {\step \sqrtra} \\
\text{Escaping radius} && \escRad &= \crad \sqrt {\alpha / \rho} \\
\text{Objective decrease after successful escaping} && \df &= \cf \sqrtfacr
% \text{Noise standard deviation} && r &= \cnoise \frac \alpha {\sqrt{\gradLip \step}}
\end{aligned}
% \label{eq:params}
\end{equation*}
\end{definition}
Intuitively, the parameters are chosen as follows.
Assume that the current point is $\vzero$ and the function is quadratic, i.e. $f(\vx) = \vx^T H \vx$.
If $\vx$ is a saddle point, then we know that the smallest eigenvalue of $H$ is $-\eigen < -\sqrtra$ with the corresponding eigenvector $\vv_1$.
Then, the largest function decrease would be in the direction of $\vv_1$, and this direction can be found using power method for the appropriate matrix.
But, if we look at the gradient descent update:
\[
\x{t+1}
= \x{t} - \step \nabla f(\x{t})
= \x{t} - \step 2 H \x{t}
= (\id - \step 2 H) \x{t},
\]
we see that for quadratic functions, gradient descent is exactly the power method for the matrix $\id - \step 2H$, which has the same eigenvectors as $H$.\footnote{More complicated, but similar logic works for gradient descent with clipping}
The eigenvalue $-\eigen$ of $H$ corresponds to the eigenvalue $1 + 2 \step \eigen$ of $1 - \step 2 H$, which is the largest (by absolute value) eigenvalue when $\step \le 1/L$.
Note that power method requires some initial randomness, which explains while randomness is required to escape from saddle points.
The above intuition motivates the choice of parameters.
\begin{itemize}
\item
\textbf{Escaping radius $\escRad$}. Since in general the function is not necessarily quadratic, we want to guarantee that the Hessian doesn't change excessively.
More specifically, we want the smallest eigenvalue to be at most $-\eigen / 2$ for the entire escaping procedure.
By the Hessian-Lipschitz property $\|\nabla^2 f(\vx) - \nabla^2 f(\vy)\| \le \rho \|\vx - \vy\|$, if we work in the radius $\escRad = \eigen /(2\rho)$, then for any point within the radius $\escRad$ of the initial point, the smallest eigenvalue is at most $-\eigen + \eigen / 2 = -\eigen / 2$.
When $\eigen > \sqrtra$, we can lower-bound $\escRad$ as $\sqrtra / (2\rho) = \sqrt{\alpha / (4\rho)}$.
\item
\textbf{Number of iterations required for escaping $\escIter$}.
After $t$ iterations, the length of $\x{t}$ along the $\vv_1$ direction is $(1 + \step \eigen)^t \approx e^{t / (\step \eigen)}$.
Hence, we reach the boundary of the ball of radius $\escRad$ after $(t \log \escRad) / (\step \eigen)$ iterations.
\item
\textbf{Function change after escaping $\df$}.
Since we are working inside the ball of size $\escRad$, the best objective improvement $\df$ we can aim for is achievable by moving by $\escRad \vv_1$.
This improves the objective by approximately $\escRad^2 \eigen$, which, by the choice of the parameters can be lower-bounded as $\alpha / \rho \cdot \sqrtra = \sqrt{\alpha^3 / \rho}$.
\item
\textbf{Total number of iterations $T$}. Every $\escIter$ iterations we improve objective by $\df$.
Since the total objective improvement cannot exceed $\fmax$, the maximum number of iterations is
\[
\frac{\fmax \escIter}{\df}
\le \frac{\fmax}{(\step \sqrtra)\sqrt{\alpha^3 / \rho}}
= \frac{\fmax}{\step \alpha^2}
\]
\end{itemize}
\begin{lemma}[Descent Lemma]
\label{lem:descent}
Assume that $\step < 1/L$, for any $t_0$ and $t$.
Then, with probability at least $1 - \tdelta$, for some constant $c$:
\[
f(\x{t_0}) - f(\x{t_0 + t})
\ge \frac{\step}{8} \sum_{\tau=t_0}^{t_0 + t - 1} \norm{\nabla f(\x{\tau})}^2 - c \step \tsigma^2 (\step L t + \logErr)
.
\]
\end{lemma}
\begin{proof}
W.l.o.g., we consider $t_0 = 0$.
Let $\sgdNoise{\tau} = \nabla f_{\batch{t}}(\x{\tau}) - \nabla f(\x{\tau})$ be the stochastic noise at iteration $\tau$.
By the folklore Descent Lemma,
\begin{align*}
f(\x{\tau}) - f(\x{\tau+1})
% \ge \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \nabla f_{\batch{t}}(\x{\tau})}
% - \frac{L}{2} \SqrNorm{\x{}}
&\ge \step \InnerProd{\nabla f(\x{\tau}), \nabla \fclip_{\batch{t}}(\x{\tau})}
- \frac{L}{2} \SqrNorm{\x{\tau + 1} - \x{\tau}} \\
&= \step \InnerProd{\nabla f(\x{\tau}), \frac{\nabla f(\x{\tau}) + \sgdNoise{\tau}}{\clip{\tau}} + \ourNoise{\tau}}
- \frac{L\step^2}{2} \SqrNorm{\frac{\nabla f(\x{\tau}) + \sgdNoise{\tau}}{\clip{\tau}} + \ourNoise{\tau}} \\
&\ge \frac{\step}{\clip{t}} \SqrNorm{\nabla f(\x{\tau})}
+ \step \InnerProd{\nabla f(\x{\tau}), \frac{\sgdNoise{\tau}}{\clip{\tau}} + \ourNoise{\tau}}
- \frac{\step^2 L}{2} \pars{\frac{3}{2} \SqrNorm{\frac{\nabla f(\x{\tau})}{\clip{\tau}}} + 3 \SqrNorm{\frac{\sgdNoise{\tau}}{\clip{\tau}} + \ourNoise{\tau}}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
,
\end{align*}
where we used inequality $\|a + b\|^2 \le (1 + \nu) \|a\|^2 + (1 + 1/\nu)\|b\|^2$ for $\nu = 1/2$.
By telescoping,
\begin{align*}
f(\x{0}) - f(\x{t})
&\ge \frac{\step}{4} \sum_{\tau=0}^{t-1} \frac{\SqrNorm{\nabla f(\x{\tau})}}{\clip{\tau}}
+ \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \frac{\sgdNoise{\tau}}{\clip{\tau}} + \ourNoise{\tau}}
- \frac{3 \step^2 L}{2} \sum_{\tau=0}^{t-1} \SqrNorm{\frac{\sgdNoise{\tau}}{\clip{\tau}} + \ourNoise{\tau}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
\end{align*}
%
Using a concentration inequality for nSG variables, with probability at least $1 - \tdelta$ for some constant $c$ we have
\todo{Handle that}
\[
\abs{\step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}}
\le \frac{\step}{8} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})} + c \step \fullVar^2 \logErr
\]
\todo{Give reference}
\todo{Avoid norm-subGaussian}
For the last term, with probability at least $1 - \tdelta$ we have
\[
\frac{3 \step^2 L}{2} \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau}}
\le c \step^2 L \fullVar^2 (t + \logErr).
\]
Combining the above inequalities and using $\step \le 1/L$ finishes the proof.
% \todo{TODO}
\end{proof}
\subsection{Large Gradient Case}
\todo{Get rid of frac in the text}
The first application of the Descent Lemma is to bound the number of iterations when the gradient is large.
\begin{lemma}
\label{lem:quarter_large_grad}
When $T \ge \frac{64 \fmax}{\step \alpha^2}$ and other parameters are as in \Cref{def:params}, the number of iterations $t$ with $\|\nabla f(\x{t})\| \ge \alpha$ is at most $\nicefrac T4$.
\end{lemma}
\begin{proof}
For the sake of contradiction, assume that the number of points $\x{t}$ with $\|\nabla f(\x{t})\| \ge \alpha$ is greater than $\nicefrac T4$.
Then, by the Descent Lemma we have
\begin{align*}
f(\x{0}) - f(\x{T})
&\ge \frac{\step}{8} \sum_{\tau=0}^{T - 1} \norm{\nabla f(\x{\tau})}^2 - c \step \tsigma^2 (\step L T + \logErr) \\
&\ge \frac{\step T \alpha^2}{32} - c \step T \cdot \tsigma^2 \pars{\step L + \frac{\logErr}{T}}
\end{align*}
We will show that the right-hand side is at least $\fmax$, leading to contradiction.
For that, it suffices to guarantee that $\frac{\step T \alpha^2}{32} \ge 2\fmax$ and $c \step T \cdot \tsigma^2 \pars{\step L + \frac{\logErr}{T}} \le \fmax$.
The former condition is satisfied by our choice of $T$, and substituting the bound on $\fmax$ into the latter condition shows that it suffices to prove
\[
64 c \tsigma^2 \pars{\step L + \frac{\step \alpha^2 \logErr}{64 \fmax}} \le \alpha^2
\]
It suffices to bound each term with $\alpha^2 / 2$, which is satisfied for
\begin{align*}
\step &\le \frac{\alpha^2}{128 \cdot c \tsigma^2 L} \\
\step &\le \frac{\fmax\logErr}{2 \cdot c \tsigma^2 L}
\end{align*}
Both of these conditions are satisfied by our choice of $\step$, with the appropriate choice of $\cstep$.
\end{proof}
\subsection{Escaping From Saddle Points}
The Descent Lemma also implies the following result, which upper-bounds the distance change in terms of the objective changes.
Its \Cref{cor:i_or_l} states that, if after a bounded number of iterations the point moves far from the original point, then the objective value sufficiently decreases.
%
\begin{lemma}[Improve or localize]
\label{lem:improve_or_localize}
For any $t_0$ and $t$, with probability at least $1 - \tdelta$, for some constant $c$:
\begin{align*}
\SqrNorm{\x{t_0 + t} - \x{t_0}}
\le c \step t \pars{(f(\x{t_0}) - f(\x{t_0 + t}) + \step \tsigma^2 (\step L t + \logErr)}
\end{align*}
\end{lemma}
%
\begin{proof}
W.l.o.g. we consider the case when $t_0 = 0$.
% By the Descent Lemma,
% \[
% f(\x{0}) - f(\x{t})
% \ge \frac{\step}{8} \sum_{\tau=0}^{t - 1} \norm{\nabla f(\x{\tau})}^2 - c \step \tsigma^2 (\step L t + \logErr)
% \]
By the definition of $\x{\tau + 1}$, we have
\begin{align*}
\x{\tau} - \x{\tau + 1}
= \step (\nabla f_{\batch{\tau}}(\x{\tau}) + \ourNoise{\tau})
= \step (\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}),
\end{align*}
where $\sgdNoise{\tau} = \nabla f_{\batch{\tau}}(\x{\tau}) - \nabla f(\x{\tau})$ is stochastic gradient noise.
By telescoping, we have:
\begin{align*}
\|\x{0} - \x{t}\|
&= \step \norm{\sum_{\tau = 0}^{t - 1}(\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau})}
\end{align*}
By Cauchy-Schwarz Inequality, we have:
\begin{align*}
\|\x{0} - \x{t}\|^2
&\le 2 \step^2 \pars{\norm{\sum_{\tau = 0}^{t - 1}\nabla f(\x{\tau})}^2 + \norm{\sum_{\tau = 0}^{t - 1} (\sgdNoise{\tau} + \ourNoise{\tau})}^2} \\
&\le 2 \step^2 \pars{t \sum_{\tau = 0}^{t - 1}\norm{\nabla f(\x{\tau})}^2 + \norm{\sum_{\tau = 0}^{t - 1} (\sgdNoise{\tau} + \ourNoise{\tau})}^2}
\end{align*}
\todo{Elaborate and avoid nSG}
Since $\sgdNoise{\tau} + \ourNoise{\tau}$ is $\mathrm{nSG}(\fullVar^2)$, by applying concentration inequality for $\mathrm{nSG}$ variables, we have with probability at least $1 - \tdelta$:
\begin{align*}
\|\x{0} - \x{t}\|^2
&\le 2 \step^2 \pars{t \sum_{\tau = 0}^{t - 1}\norm{\nabla f(\x{\tau})}^2 + c t \fullVar^2 \logErr}
\end{align*}
Finally, applying the Descent Lemma, we get
\begin{align*}
\|\x{0} - \x{t}\|^2
&\le \step c t \pars{f(\x{0}) - f(\x{t}) + \step \fullVar^2 (\step L t + \logErr)}
\end{align*}
for some constant $c$.
\end{proof}
\begin{corollary}
\label{cor:i_or_l}
For the parameters chosen as in \Cref{def:params}, if for some $t \le \escIter$ we have $\SqrNorm{\x{t_0 + \tau} - \x{t_0}} \ge \escRad$, then $f(\x{t_0}) - f(\x{t_0 + \tau}) \ge \df$ with probability at least $1 - \tdelta$.
\end{corollary}
%
\begin{proof}
From \Cref{lem:improve_or_localize}, we know that for some $c$:
\begin{align*}
f(\x{t_0}) - f(\x{t_0 + t})
&\ge \frac{\escRad^2}{c \step \escIter} - \step \tsigma^2 (\step L \escIter + \logErr) \\
&\ge \frac{\crad^2}{c \cit} \sqrtfacr - \step \tsigma^2 \pars{\frac{\cit L}{\sqrtra} + \logErr}
% &\ge \frac{\SqrNorm{\x{t_0 + t} - \x{t_0}}}{c \step \escIter} - \step \tsigma^2 (\step L \escIter + \log 1/\fullVar)
\end{align*}
We want to guarantee that the right-hand side is at most $\df = \cf \sqrtfacr$.
It suffices to guarantee that
\begin{align*}
\crad^2 / (c \cit) \ge 2 \cf \qquad \text{and} \qquad \step \tsigma^2 (\cit L / \sqrtra + \logErr) \le \cf \sqrtfacr.
\end{align*}
The former holds for the appropriate choice of $\crad$, $\cit$, and $\cf$, and the latter holds by choosing
\[
\step \le (\alpha^2 / L \tsigma^2) \cdot \cf / \pars{\cit + \logErr},
\]
where we assumed $L \ge \sqrtra$, since otherwise any $\alpha$-FOSP is an $\alpha$-SOSP because $L$ is an upper bound on the absolute values of the Hessian.
\end{proof}
\begin{definition}[Coupling sequences]
\label{def:coupling_sequence}
Let $\vv_1$ be the eigenvector corresponding to the smallest eigenvalue of $\nabla^2 f(\x{t_0})$.
The coupling sequences starting from $\x{t_0}$ are defined as follows:
\begin{equation}
\begin{aligned}
\ourNoise{t_0+t} &\sim \mathcal{N}(0, \fullVar^2 \id) \\
\tourNoise{t_0+t} &= \ourNoise{t_0+t} - 2 \InnerProd{ \vv_1, \ourNoise{t_0+t} } \vv_1 \\
% \batch'_{t_0+t} &= \batch_{t_0+t}, \\
\x{t_0 + t+1} &= \x{t_0+t} - \step (\nabla f_{\batch{t}}(\x{t_0 + t}) + \ourNoise{t_0 + t}) \\
\tx{t_0 + t+1} &= \tx{t_0+t} - \step (\nabla f_{\batch{t}}(\tx{t_0 + t}) + \tourNoise{t_0 + t}) \\
\sgdNoise{t_0 + t} &= \nabla f_{\batch{t}}(\x{t_0 + t}) - \nabla f(\x{t_0 + t}) \\
\tsgdNoise{t_0 + t} &= \nabla f_{\batch{t}}(\tx{t_0 + t}) - \nabla f(\tx{t_0 + t}) \\
\end{aligned}\nonumber
\end{equation}
%
Note that:
\begin{itemize}
\item the artificial noise $\tourNoise{t_0 + t}$ has the opposite sign in direction $\vv_1$ compared with $\ourNoise{t_0 + t}$,
\item the stochastic randomness ${\batch{t}}$ is the same for both sequences,\todo{Might be not required, but anyways makes things simpler to define}
\item $\{\x{t_0 + t}\}_t$ and $\{\tx{t_0 + t}\}_t$ are sampled from the same distribution.
\end{itemize}
%
We next define the difference between the coupling sequences:
\begin{equation}
\begin{aligned}
\difourNoise{t} &= \xi_{t_0+t} - \xi'_{t_0+t}
= 2 \InnerProd{ \vv_1, \xi_{t_0+t} } \vv_1 \\
% \batch'_{t_0+t} &= \batch_{t_0+t}, \\
\difx{t_0 + t+1} &= \x{t_0+t} - \tx{t_0+t} \\
\difsgdNoise{t_0 + t} &= \sgdNoise{t_0 + t} - \tsgdNoise{t_0 + t}
\end{aligned}\nonumber
\end{equation}
\end{definition}
% TODO (the rest is standard)
% SGD with variance $\sigma^2$ and batch size $B$, then the number of iterations to find an $\alpha$-FOSP ($\|\nabla f(\vx)\| < \alpha$) is $\frac{\sigma^2}{B \alpha^4}$.
From now on, we assume that we want to escape from point $\x{t_0}$ such that $\lmin(\nabla^2 f(\x{t_0})) < -\sqrtra$.
W.l.o.g. we assume $t_0 = 0$.
Let $H = \nabla^2 f(\x{0})$ be the Hessian of $f$ at point $\x{0}$, let $-\eigen = \lmin(H)$ be the smallest eigenvalue of $H$, and $\vv_1$ be the eigenvector corresponding to $-\eigen$.
\begin{lemma}
For every $t$,
\[
\difx{t} = -\dif{t} - \sgdErr{t} -\quadErr{t},
\]
where
\begin{align*}
\dif{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difourNoise{\tau} \\
\sgdErr{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difsgdNoise{\tau} \\
\quadErr{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \hesInt{\tau} \difx{\tau},
\quad \text{ where } \quad \hesInt{\tau} = \int_0^1 (\nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) - H)\ d\psi
\end{align*}
\end{lemma}
\begin{proof}
Proof by induction.
\begin{align*}
\difx{t + 1}
&= \difx{t} - \step (\difsgdNoise{t} + \difsgdNoise{t}) - \step (\nabla f(\x{t}) - \nabla f(\tx{t})) \\
&= (1 - \step H) \difx{t} - \step (\difsgdNoise{t} + \difsgdNoise{t}) - \step (\nabla f(\x{t}) - \nabla f(\tx{t}) - H \difx{t})
\end{align*}
Using that $\nabla f(\x{t}) - \nabla f(\tx{t}) = \int_0^1 \nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) \difx{\tau}\ d\psi$, by telescoping we complete the proof.
\end{proof}
%
In the following statements, the geometric series $\alpha_t^2 = \sum_{\tau = 0}^{t-1} (1 + \step \eigen)^{2 \tau}$ occurs frequently.
Intuitively, each iteration increases the distance approximately by a factor of $(1 + \step \eigen)$~-- the largest eigenvalue of matrix $I - \step H$.
The noise added at iteration $\tau$ gets amplified by $(1 + \step \eigen)^{t - \tau}$ by iteration $t$, and, if every iteration an independent noise of variance $\Delta^2$ is added, $\Delta^2 \alpha_t^2$ is the total variance of such a noise accumulated over the iterations.
%
\begin{lemma}
\label{lem:alpha_beta}
Let $\alpha_t = \sqrt{\sum_{\tau = 0}^{t-1} (1 + \step \eigen)^{2 \tau}}$ and $\beta_t = (1 + \step \eigen)^t / \sqrt{\step \eigen}$.
If $\step \eigen < 1$, then $\alpha_t \le \beta_t$ for any $t$ and $\alpha_t \ge \beta_t / 3$ for $t \ge 2 / (\step \eigen)$.
\end{lemma}
\begin{proof}
As a sum of geometric progression, we have
\[
\alpha_t^2
= \frac{(1 + \step \eigen)^{2t} - 1}{(1 + \step \eigen)^2 - 1}
= \frac{(1 + \step \eigen)^{2t} - 1}{\step \eigen (2 + \step \eigen)},
\]
and $\alpha_t^2 \le \beta_t^2$ follows immediately.
On the other hand, for $t \ge 1 / (\step \eigen)$ we have $(1 + \step \eigen)^{2t} \ge 2$, and hence $(1 + \step \eigen)^{2t} - 1 \ge (1 + \step \eigen)^{2t} / 2$.
Finally, by our choice of the step size, we have $\step \eigen \le \step L \le 1$, and hence $2 + \step \eigen \le 3$.
This implies $\alpha_t^2 \ge \beta_t^2 / 6$, finishing the proof.
\end{proof}
\begin{lemma}
\label{lem:good_term}
Let $\beta_t = (1 + \step \eigen)^t / \sqrt{\step \eigen}$.
Then for any $t$ and $\tdelta$, for some constant $c$:
\begin{align*}
\P{\|\dif{t}\| \ge c \beta_t \step \Delta \sqrt{\logErr}} &\le \tdelta \\
\P{\|\dif{t}\| \le \frac{\beta_t \step \Delta}{10}} &\le \frac 13
\end{align*}
\end{lemma}
\todo{Check $\sqrt{\logErr}$}
\begin{proof}
Since $\difourNoise{\tau} = 2\InnerProd{\ourNoise{\tau}, \vv_1} \vv_1$ and $\vv_1$ is an eigenvalue of $H$ corresponding to eigenvalue $-\eigen$, we have
\[
\dif{t}
= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difourNoise{\tau}
= \pars{\step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{t - 1 - \tau} \InnerProd{\ourNoise{\tau}, \vv_1}} \vv_1
\]
Hence, $\dif{t}$ is a random variable which is parallel to $\vv_1$.
% Hence, $\dif{t}$ is a Gaussian random variable along direction $\vv_1$ with variance $\step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{t - 1 - \tau} \InnerProd{\ourNoise{\tau}, \vv_1}$
Since each $\ourNoise{t}$ is a Gaussian random variable with the covariance matrix $\Delta^2 \id$, each $\InnerProd{\ourNoise{t}, \vv_1}$ a Gaussian random variable with variance $\Delta^2$.
Hence, $\dif{t}$ is also a Gaussian random variable, and
\[
\Var{\|\dif{t}\|}
= \step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{\tau} \Delta^2
= \Delta^2 \alpha_t^2
\]
The upper bound on $\|\dif{t}\|$ follows from \Cref{lem:alpha_beta} and standard concentration inequality for Guassians.
The lower bound follows from inequality $\P{|X| \le \sigma a} \le a$ for $X \sim \mathcal{N}(0, \sigma^2)$ and any $a$.
\end{proof}
\todo{Be more careful with $\tdelta$}
\todo{Replace $\fullVar$ with $\Delta$ where required}
\todo{Propagate assumption $\Delta^2 \ge \sigma^2 / B$}
\begin{lemma}
\label{lem:error_terms}
Let $\mathcal{E}_t$ be the event ``$\|\x{0} - \x{\tau}\| < \escRad$ and $\|\x{0} - \tx{\tau}\| < \escRad$ for all $\tau < t$''.
Then, when $\Delta^2 \ge \sigma^2 / B$, for some constant $c$ we have
% \todo[inline]{Fix log error}
\[
\P{\mathcal{E}_\escIter
\implies \max\nolimits_{\tau=0}^{\escIter - 1} \norm{\sgdErr{\tau} + \quadErr{\tau}}
\le
c \beta_\tau \step \Delta \sqrt{\logErr}
} \ge 1 - \escIter \tdelta
\]
\end{lemma}
\begin{proof}
By induction on $t$, we will prove that $\P{\mathcal{E}_t \implies \|\difx{t}\| \le c \step \Delta \beta_t} \le 1 - t\fullVar$ and
\[
\P{\mathcal{E}_t
\implies \max\nolimits_{\tau=0}^{t - 1} \norm{\sgdErr{\tau} + \quadErr{\tau}}
\le c \beta_\tau \step \Delta \sqrt{\logErr}
} \ge 1 - t \tdelta.
\]
Base case is $t=0$, in which case $\difx{0} = \vzero$.
For the induction step, since $\difx{t} = -\dif{t} - \sgdErr{t} -\quadErr{t}$, we bound each term separately.
\paragraph{Bounding $\dif{t}$}
By \Cref{lem:good_term}, we have $\P{\|\dif{t}\| \ge c \beta_t \step \Delta \sqrt{\logErr}} \le \tdelta$.
\paragraph{Bounding $\quadErr{t}$}
Assuming $\mathcal{E}_t$, by the Hessian-Lipschitz property we have
\begin{align*}
\|\hesInt{\tau} \difx{\tau}\|
&= \norm{\int_0^1 (\nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) - H)\ d\psi \cdot \difx{\tau}} \\
&\le 2 \rho \escRad \|\difx{\tau}\| \\
&\le 2 \rho \escRad \cdot c \beta_\tau \step \Delta \sqrt{\logErr},
\end{align*}
%
and we bound $\quadErr{t}$ as
%
\begin{align*}
\|\quadErr{t}\|
&= \norm{\step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \hesInt{\tau} \difx{\tau}} \\
&\le \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{t - 1 - \tau} 2 \rho \escRad \cdot c \beta_\tau \step \Delta \sqrt{\logErr} \\
% &\le \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{\tau} 2 \rho \escRad \cdot c \beta_t \step \fullVar^2 \sqrt{\logErr} \\
\end{align*}
Since $(1 + \step \gamma)^{t - 1 - \tau} \beta_\tau = \beta_{t-1}$,
% \[
% \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{\tau}
% = \frac{(1 + \step \gamma)^t - 1}{\step \gamma}
% \le \frac{\beta_t}{\sqrt{\step \gamma}},
% \]
we can bound $\|\quadErr{t}\|$ as
\begin{align*}
\|\quadErr{t}\|
&\le 2 \rho \escRad \cdot c \escIter \beta_t \step \Delta \sqrt{\logErr} \\
&= 2 c \crad \cit \beta_t \Delta \sqrt{\logErr},
\end{align*}
where the last equality is due to our choice of parameters in \Cref{def:params}:
\[
\escRad \escIter
= \crad \sqrt {\alpha / \rho} \cdot \cit \frac {1} {\step \sqrtra}
= \frac{\crad \cit}{\rho \step}
\]
\paragraph{Bounding $\sgdErr{t}$}
% To bound $\sgdErr{t}$, we use that $\difsgdNoise{t}$ is $\mathrm{nSG}(\sigma)$:
% \begin{align*}
% \|\difsgdNoise{t}\|
% = \|\nabla f_{\batch{t}}(\x{t}) - \nabla f_{\batch{t}}(\tx{t})\|
% \le L \|\difx{t}\|
% \end{align*}
%
By \Cref{ass:grad_concentration}, each $\sgdErr{t}$ is $\mathrm{nSG}(\sigma^2 / B)$.
Using the concentration inequality for nSG variables, with probability at least $1 - \tdelta$ we have
\begin{align*}
\|\sgdErr{t}\|
&= \norm{\step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difsgdNoise{\tau}} \\
&\le \step \sqrt{\sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{2(t - 1 - \tau)} \frac{\sigma^2}{B}} \cdot \sqrt{\logErr} \\
&\le \step \sigma \beta_t \sqrt{\logErr} / \sqrt{B} \\
\end{align*}
Using assumption $\Delta^2 \ge \sigma^2 / B$, we get the desired bound.
\end{proof}
\begin{lemma}
\label{lem:bucket_change}
When $\Delta^2 \ge \sigma^2 / B$, with probability at least $1 - \tdelta$, we have
\[
f(\x{0}) - f(\x{\escIter}) > -\df / 100
\]
Moreover, with probability at least $1/3 - \tdelta$, we have:
\[
f(\x{0}) - f(\x{\escIter}) > \df
\]
\end{lemma}
\begin{proof}
The first claim follows from the descent lemma by our choice of parameters:
\[
f(\x{0}) - f(\x{\escIter})
> -c \step \fullVar^2 (\step L \escIter + \logErr)
> -\df / 10
\]
From Corollary~\ref{cor:i_or_l}, we know that with probability at least $1-\tdelta$, if $\|\x{t} - \x{0}\| > \escRad$ for some $t \le \escIter$, then $f(\x{0}) - f(\x{t}) > \df$ with probability at least $1 - \tdelta$.
Since the $\tx{t}$ has the same distribution as $\x{t}$, the same holds for $\|\tx{t} - \x{0}\|$.
Hence, if either $\|\x{t} - \x{0}\| > \escRad$ or $\|\tx{t} - \x{0}\| > \escRad$, then with probability $\frac{1}{2} - 2 \tdelta$ we have $f(\x{0}) - f(\x{t})$.
For the rest of the proof, we can assume that $\|\x{t} - \x{0}\| < \escRad$ and $\|\tx{t} - \x{0}\| < \escRad$ for all $t \le \escIter$.
By Lemmas~\ref{lem:good_term} and~\ref{lem:error_terms} we know that $\|\dif{\escIter}\| \ge \step \beta_t \Delta \sqrt{\logErr} / 10$ and $\|\sgdErr{t} + \quadErr{t}\| \le \step \beta_t \Delta \sqrt{\logErr} / 20$ with probability $1 - \tdelta$.
Hence,
\begin{align*}
\max(\|\x{\escIter} - \x{0}\|, \|\tx{\escIter} - \x{0}\|)
&\ge \frac{1}{2} \|\difx{\escIter}\| \\
&\ge \frac 12 \pars{\|\dif{\escIter}\| - \|\sgdErr{\escIter} + \quadErr{t}\|} \\
&\ge \frac{\step \beta_\escIter \Delta}{40} \sqrt{\logErr} \\
&= (1 + \step \eigen)^\escIter \frac{1}{40} \Delta \sqrt{\frac{\step}{\eigen}} \sqrt{\logErr} \\
&\ge \escRad,
\end{align*}
where the last inequality follows by the appropriate choice of logarithmic term in $\cit$ (note that $(1 + \step \eigen)^\escIter$ dominates other terms).
This contradicts the assumption $\max(\|\x{\escIter} - \x{0}\|, \|\tx{\escIter} - \x{0}\|) \le \escRad$.
\end{proof}
\begin{lemma}
\label{lem:quarter_saddle}
When $\Delta^2 \ge \sigma^2 / B$, for $T = 64 \fmax / (\step \alpha^2)$, with $\step$ chosen as in \Cref{def:params}, with probability at least $1 - \tdelta$ we have $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ for at most $T/4$ points $\x{t}$.
\end{lemma}
\begin{proof}
\todo{Error probability}
Proof by contradiction.
Let's split all iterations $0,...,T-1$ into buckets of size $\escIter$.
There are $T/\escIter = 64\fmax \sqrtra / \cit$ buckets, and at least quarter of the buckets have a point $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ in it.
Note that within each bucket, by Lemma~\ref{lem:bucket_change}, the objective increases by at most $\df / 100$.
Out of such buckets, at least half of them are non-adjacent, and we consider points $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ in these buckets.
With probability at least $1/3 - \tdelta$, we have $f(\x{t}) - f(\x{t + \escIter}) > \df$ for each such $\x{t}$.
% \todo[inline]{Properly handle logs}
Hence,
\begin{align*}
\fmax
&\ge \frac{T \df}{10\cdot \escIter} \\
&= \frac{1}{10} \cdot \frac{64\fmax}{\step \alpha^2} \cdot \cf \sqrtfacr \cdot \frac{\step \sqrtra}{\cit} \\
&= \fmax \frac{64 \cdot \cf}{10 \cdot \cit}
\end{align*}
Hence, by selecting $\cit$ and $\cf$ so that $\frac{64 \cdot \cf}{10 \cdot \cit} > 1$, we arrive at contradiction.
\end{proof}
% \begin{aligned}
% \text{Step size} && \step &= \cstep \frac{1}{L (1 + \tsigma^2)} \\
% \text{Total number of iterations} && T &= \frac{64\fmax}{\step \alpha^2} \\
% \text{Iterations required for escaping} && \escIter &= \cit \frac {1} {\step \sqrtra} \\
% \text{Escaping radius} && \escRad &= \crad \sqrt {\frac {\alpha} \rho} \\
% \text{Objective decrease after successful escaping} && \df &= \cf \sqrtfacr
% % \text{Noise standard deviation} && r &= \cnoise \frac \alpha {\sqrt{\gradLip \step}}
% \end{aligned}\label{eq:params}
% \begin{theorem}[Large-gradient case]
% If $\|\|$
% \end{theorem}
\begin{theorem}
\label{thm:combine}
When $\Delta^2 \ge \sigma^2 / B$, for $T = 64 \fmax / (\step \alpha^2)$, with probability at least $1 - \tdelta$, at least half of visited points are $\alpha$-SOSP.
\end{theorem}
\begin{proof}
From Lemma~\ref{lem:quarter_large_grad}, we know that the number of points with large gradient is at most $T/4$.
From Lemma~\ref{lem:quarter_saddle} we know that the number of points $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ is at most $T/4$.
Hence, at least half of the points are $\alpha$-SOSP.
\end{proof}
\section{Introduction}
%\todo{Clarify that the other assumptions (aside from Lipschitzness remain)}
%\todo{Narrative: focus on the noise}
%\todo{For this narrative: sharp/flat local minima}
%\todo[inline]{Add remark: if we want to use gradient clipping, maybe we can compensate with the step size}
%\todo[inline]{Argue that our algorithm is simpler than Clipped SGD}
In recent years, there has been a growing interest in private data analysis, from academic research to real-world applications.
\emph{Differential privacy}~\citep{dwork_CalibratingNoise_2006} provides strong privacy guarantee for each individual, while allowing to train accurate machine learning models.
Informally, an algorithm is differentially private if, for any person, the outcome of the algorithm is roughly the same regardless of whether or not this person's data is included.
%This preserves the privacy of every individual, since anything that could be learned about the individual from their data could also be learned \emph{without} their data.
Differential privacy has become one of the most popular formalizations of privacy in both theory and practice~\citep{dwork_AlgorithmicFoundations_2014,DBLP:series/synthesis/2016Li}, and has been adopted in a variety of settings including regression, combinatorial optimization, continuous optimization, principal component analysis, matrix completion, reinforcement learning and deep learning~\citep{chaudhuri_PrivacypreservingLogistic_2008,chaudhuri_DifferentiallyPrivate_2011,DBLP:conf/soda/GuptaLMRT10,agarwal_CpSGDCommunicationefficient_2018,ge_MinimaxOptimalPrivacyPreserving_2018,jain_DifferentiallyPrivate_2018, DBLP:conf/icml/VietriBKW20, abadi_DeepLearning_2016}.
% Adding noise is a key technique used to achieve Differential Privacy.
%There are different mechanisms for adding noise, such as Laplace mechanism or Gaussian noise, introduced by ~\citep{DBLP:conf/tcc/DworkMNS06, DBLP:journals/fttcs/DworkR14}, each of those mechanisms adds noise drawn from either a Laplace or a Gaussian distribution, depending on the specific differential privacy algorithm being used. The goal is to balance privacy protection with the utility of the data or analysis results.
%A prominent application of differential privacy is private \emph{optimization}.
%In practice, the machine learning models are commonly trained on some finite dataset, with the quality of the model evaluated using certain loss functions.
A typical machine learning problem can be stated as \emph{empirical risk minimization} (ERM), where we optimize over the parameters of the model in order to minimize the empirical risk.
%(performance of the model on the unseen data is estimated using generalization bounds, such as in~\citet{hardt_TrainFaster_2016}). %The empirical-risk minimization (ERM) is an important concept in machine learning.
%We use the notation
%$\widetilde{\BigOh}(\cdot)$ to hide poly-logarithmic terms.
Formally, ERM is:
\begin{align*}
\minimize_{ w \in \R^d } f( w ) := \frac{1}{n} \sum_{i=1}^n f_i(w),
\end{align*}
where $n$ is the number of samples, $f_i$'s are differentiable functions and $w$ are the model weights.
Since sampled data points typically correspond to information about individuals, it is natural to ask whether we can do ERM \emph{privately}~\citep{bassily_PrivateEmpirical_2014}. While there are various algorithmic approaches for solving ERM, Stochastic Gradient Descent (SGD) plays a key role as a very basic yet widely used practical algorithm. SGD is a first-order optimization algorithm that updates the model parameters based on the gradient of the loss function with respect to these parameters.
Due to the simplicity and efficiency of SGD, one of the most important techniques in private optimization is Differentially Private SGD (DP-SGD) (see e.g.~\citet{abadi_DeepLearning_2016}), which is implemented as a part of TensorFlow library and in PyTorch\footnote{\url{https://github.com/tensorflow/privacy}, \url{https://github.com/pytorch/opacus/}}.
DP-SGD guarantees differential privacy by adding noise to the gradients computed during the training process. By carefully controlling the noise, this can be achieved without substantially sacrificing the overall utility of the trained model.
DP-SGD now plays a central role in private optimization, and has been studied extensively in recent years~\citep{song2013stochastic,bassily_PrivateEmpirical_2014,abadi_DeepLearning_2016,wang2017differentially,bassily2019private,feldman2020private,asi2021private,yang2022normalized,DBLP:conf/iclr/FangLF023}.
SGD converges to a point with a small gradient, which can be a local minimum or a saddle point.
In general, finding a local minimum is NP-hard for non-convex functions~\citep{DBLP:conf/colt/AnandkumarG16}.
Despite this, the problem of finding a local minimum by escaping from saddle points has received significant attention~\citep{DBLP:conf/colt/GeHJY15,DBLP:conf/nips/BhojanapalliNS16, DBLP:conf/icml/0001JZ17, jin_HowEscape_2017,carmon_AnalysisKrylov_2018,jin_AcceleratedGradient_2018,allen-zhu_NatashaFaster_2018,allen-zhu_NEON2Finding_2018,fang_SPIDERNearOptimal_2018,tripuraneni_StochasticCubic_2018,xu_FirstorderStochastic_2018, daneshmand_EscapingSaddles_2018, carmon_FirstOrderMethods_2020,zhou_StochasticRecursive_2020,zhou_StochasticNested_2020, jin_NonconvexOptimization_2021, zhang_EscapeSaddle_2021}.
Many non-convex ERM functions have been shown to be strict saddles~\citep{DBLP:conf/colt/GeHJY15}, meaning that a second-order stationary point (SOSP) (i.e. a point with a small gradient and a bounded Hessian) can be sufficiently close to a local minimum. Similarly to differential privacy, adding noise to the gradient during training has been the key technique for escaping from saddle points~\citep{DBLP:conf/colt/GeHJY15,jin_NonconvexOptimization_2021}.
Since adding noise to gradients during training is the fundamental idea behind both private SGD and second-order convergence of SGD, it is natural to ask whether ``noise is all you need'': is adding noise enough to guarantee both privacy and second-order convergence simultaneously? Or do we need more sophisticated algorithms to simultaneously guarantee privacy and second-order convergence?
There are a number of technical complications that arise when trying to analyze both properties. Most notably, standard versions of DP-SGD~\citep{abadi_DeepLearning_2016} use a technique known as \emph{gradient clipping}, which goes beyond simply adding noise.
We ask a simple yet fundamental question:
\begin{center}\textit{Does noisy SGD converge to a second-order stationary point while also providing privacy?}\end{center}
% While one might hope that the answer is a simple ``yes, noise is enough'', there are a number of technical complications that arise when trying to analyze both properties. Most notably, standard versions of DP-SGD~\citep{abadi_DeepLearning_2016} use a technique known as \emph{gradient clipping}, and so goes beyond simply adding noise. So we still have a simple yet fundamental question:
% \begin{center}\textit{Does noisy SGD converge to a second-order stationary point while also providing privacy?}\end{center}
%In other words: is it true that ``noise is all you need'' for private optimization? Or do we need more sophisticated algorithms to simultaneously guarantee privacy and second-order convergence?
%Hence an alternative way is to find an approximate second-order stationary point (SOSP).
%For the non-convex function of ERM problem,
% a first-order stationary point is a point where the function has small gradient and a second-order stationary point is a point where the function has both small gradient and positive semi-definite Hessian.
%Since finding a local minimum of ERM problem is difficult in general, one usually tries to find an approximate second-order stationary point instead.
\subsection{Our Results}
% \todo[inline]{
% Outline:
% \begin{itemize}
% \item We show that DP-SGD converges privately to a stable point
% \item Compare to directly related recent work
% \item In addition, we relax the necessary assumptions.
% \item Main theorem statement
% \item Overview of analysis: what we did that's nonstandard.
% \end{itemize}
% }
In this paper, we give an affirmative answer the above question: we show that an extremely simple version of noisy SGD is enough to find a second-order stationary point privately.
We consider a version of DP-SGD which does not use gradient clipping~-- the usual technique used to ensure privacy~-- and doesn't assume bounded domain.
Instead, we run vanilla SGD, but with carefully chosen noise added to each stochastic gradient.
Our main result is the following, where $\vx$ is an $\alpha$-second order stationary point ($\alpha$-SOSP) for a twice-differentiable function $f$ if $\|\nabla f(\vx)\| < \alpha$ and $\lmin(\nabla^2 f(\vx)) > -\sqrtra$ (see Definition~\ref{def:sosp}):
\begin{theorem}[Informal, see Theorem~\ref{thm:combine_main}]
% Without the Lipschitz condition, there exists an $(\eps,\delta)$-differentially private algorithm which for any $\alpha = \tilde{\Omega} \pars{\nicefrac{d^{1/4}}{\sqrt{n \eps}}}$ has the property that after $\tilde{\Omega}(\frac{\cdots}{\alpha^4})$ iterations, at least half of the iterates are $\alpha$-second-order stationary points w.h.p.
Under the standard assumptions of non-convex optimization~-- in particular, without the Lipschitz condition~-- noisy SGD (without gradient clipping) is $(\eps,\delta)$-differentially private and for any $\alpha = \tilde{\Omega} \pars{\nicefrac{d^{1/4}}{\sqrt{n \eps}}}$ finds an $\alpha$-second-order stationary point w.h.p. Moreover, the number of stochastic oracle calls with variance $\sigma^2$ is
%$\tOh{(\sigma n \eps)^2/d}$ .
$\tOh{\nicefrac{\sigma^2}{\alpha^4}}$.
% For some constant $c_2$, let $\Delta > c_2 \frac{C \sqrt{T \log \nicefrac{1}{\delta}}}{n \eps}$.
% Then for $\step = \tilde{\Omega}\pars{\nicefrac{\alpha^2}{\Delta^2}}$, Algorithm~\ref{alg:dpsgd} is $(\eps, 2\delta)$-DP and after $T = O\pars{\nicefrac{\fmax}{(\step \alpha^2)}}$ iterations,
% \mdnote{Why not just set $T$ to this?}
% w.h.p. at least half of the points are $\alpha$-SOSP .
\end{theorem}
% In more detail, our algorithm produces iterates $\x{0}, \ldots, \x{T}$, and for any constant $\theta \in (0,1)$, we can guarantee that at least $\theta$-fraction of iterates are $\alpha$-SOSP.
% \mdnoteinline{Discuss how the ``weird'' parts of it are standard in previous work (e.g., that a constant fraction of iterates are SOSP). Also discuss how the value of $\alpha$ we can guarantee and the rate compare to previous work.}
In non-private optimization, an arbitrary small $\alpha$ can be achieved at the cost of larger running time, with $\tOh{\nicefrac{\sigma^2}{\alpha^4}}$ being a standard number of stochastic oracle calls~\citep{arjevani_LowerBounds_2023}.
On the other hand, in differentially private optimization, there is a trade-off between privacy and accuracy, with $\nicefrac{d^{1/4}}{\sqrt{n \eps}}$ being the standard convergence rate.
% best achievable $\alpha$
% ~\citep{bassily_PrivateEmpirical_2014}.\footnote{
% Under standard assumptions, \citet{bassily_PrivateEmpirical_2014} show $\nicefrac{d^{1/2}}{n \eps}$ lower bound on $f(\vx) - f(\vx^*)$ for convex objectives, where $\vx^*$ is the optimum and $\vx$ is the algorithm output.
% This corresponds to the same bound on $\norm{\nabla f(\vx)}^2$.
% % Under stronger assumtions (see e.g.~\citet{DBLP:conf/icml/AroraBGGMU23}), better convergence rates are possible.
% }
% \todo{Check the footnote, might be wrong}
Our result has a number of important differences when compared to recent related work on privately finding second-order stationary points.
\paragraph{Gradient clipping}
Our algorithm is very simple: it is essentially just ``vanilla'' SGD but with noise added to preserve privacy.
Contrary to that, the standard version of DP-SGD (e.g., from~\citet{abadi_DeepLearning_2016}) uses a technique known as \emph{gradient clipping}, where if the algorithm encounters a large gradient, it first scales the gradient to have a small norm before adding noise.
In general, clipping introduces non-vanishing bias term hurting convergence (see \citet{koloskova_RevisitingGradient_2023,chen_UnderstandingGradient_2020} for examples and convergence rates).
In contrast, based on ideas from~\citet{DBLP:conf/iclr/FangLF023}, we design a simpler differentially private algorithm where clipping is not necessary.
\paragraph{More involved algorithms.}
Most notably, there is a recent line of work on this problem (private second-order convergence), with the goal of achieving optimal (or near-optimal) rates.
This requires using more involved algorithms, often based on SPIDER~\citep{fang_SPIDERNearOptimal_2018} or SpiderBoost~\citep{wang_SpiderBoostMomentum_2019}.
In particular,~\citet{DBLP:conf/icml/AroraBGGMU23} introduced a private version SpiderBoost to find second-order stationary points privately, later refined by~\citet{liu2023private}. While these are important results, they require more involved algorithms, which are rarely used in practice.
SGD and DP-SGD are still the workhorses of modern (private) machine learning. Understanding their powers and limitations is the key motivation of our work.
\paragraph{Stronger Assumptions, Weaker Algorithms.}
\citet{wang_DifferentiallyPrivate_2019} analyze simple first-order algorithms for private second-order convergence. They give upper bounds for private ERM in a number of settings, including finding approximate local minima. Our bounds offer two important improvements.
First,~\citet{wang_DifferentiallyPrivate_2019} only consider \emph{non-stochastic} gradients.
In other words, rather than computing stochastic gradients through random minibatches, in every iteration they use the entire dataset to compute an exact gradient.
In practice, computing full gradients is usually prohibitively expensive.
Second, we make less restrictive assumptions on the loss functions: \citet{wang_DifferentiallyPrivate_2019}, as common in differentially private optimization, assume that the loss functions are Lipschitz continuous, while we assume that they are smooth~-- a standard assumption in non-convex optimization~-- and have a Lipschitz Hessian~-- a standard assumption for the second-order convergence~\citep{jin_NonconvexOptimization_2021,fang_SPIDERNearOptimal_2018}.
Replacing a Lipschitz assumption~\citep{Searcoid2006} with less restrictive assumptions has recently been studied for \emph{first}-order convergence of DP-SGD~\citep{DBLP:conf/iclr/FangLF023}; we continue this line of work by generalizing it to second-order convergence.
As in~\citet{DBLP:conf/iclr/FangLF023}, since we are making very limited assumptions about the loss function, it is not possible to show global optimality, unlike in~\citet{wang_DifferentiallyPrivate_2019}, who are able to do so by assuming Lipschitzness and regularization.
\paragraph{Broader Impact}
We show that noisy SGD without clipping finds second-order stationary points, which are good approximations of local minima.
In many cases, local minima are global minima (see e.g.~\citet{sun_CompleteDictionary_2017,ge_MatrixCompletion_2016a,kawaguchi_DeepLearning_2016}), and hence DP-SGD is capable of finding high-quality solutions while preserving privacy.
One of the main privacy applications is federated learning~\citep{kairouz_AdvancesOpen_2021}, where multiple clients cooperate in training a single model.
Our result implies that noisy distributed SGD preserves the privacy of its clients.
Our result shows that noisy SGD is a good choice for major areas with sensitive data, including healthcare~\citep{ficek2021differential,hassan_DifferentialPrivacy_2020}, Internet of Things~\citep{jiang_DifferentialPrivacy_2021}, geolocation~\citep{andres_GeoindistinguishabilityDifferential_2013}, etc.
For certain applications, such as natural language processing, differential privacy is crucial due to models potentially memorizing the inputs~\citep{carlini_SecretSharer_2019a}.
Aside from the above direct applications, differential privacy guarantee are useful for machine unlearning~\citep{sekhari_RememberWhat_2021}, which is an important direction, in particular, due to General Data Protection Regulation.
Since differential privacy guarantees that output is not sensitive to any given example, the model trained by noisy SGD is guaranteed to be able to unlearn a noticeable fraction of inputs.
Similarly, low sensitivity to individual samples makes the model trained with noisy SGD robust~\citep{lecuyer_CertifiedRobustness_2019}, i.e. a significant number of adversarial examples is needed to noticeably affect the model.
Moreover, such models have provable bounds on generalization~\citep{jung_NewAnalysis_2024}.
\subsection{Technical Overview}
As mentioned above, our algorithm is simply a ``vanilla'' SGD with noise.
Despite this simplicity, there are some interesting and important subtleties.
At a very high level, the noise we have to add to preserve privacy depends on the magnitude of the gradients we see in the algorithm.
The standard solution is to use gradient clipping, which guarantees that gradient norm is small compared with noise.
We, on the other hand, use the fact that standard assumptions in non-convex optimization imply that the gradient norms can be uniformly bounded depending on the initial objective value (as pointed out in~\citet{DBLP:conf/iclr/FangLF023} for private first-order convergence) to set the noise in such a way that clipping is essentially not necessary.
% \todo{Check that it works well with previous section}
%\todo[inline]{Some variations, e.g. our main competitor~\citep{wang_DifferentiallyPrivate_2019}, don't use clipping}
The above observation simplifies our analysis significantly: we can use the existing privacy analysis of DP-SGD with clipping almost as a black box, and can focus on proving convergence to a second-order stationary point of ``noisy'' SGD with no complications from clipping. Our analysis of convergence has a number of technical subtleties, but is overall inspired by the non-private analysis of the second-order convergence of noisy SGD. The main difference is that the noise we add for privacy is \emph{significantly} larger than the noise that we would have to add if our only goal was to escape from saddle points.
Hence, proving convergence requires carefully arguing that this added noise is still dominated by the fast convergence of SGD.
%\mdnoteinline{Add more discussion if there's something useful to say. Dmitrii?}
\subsection{Related Work}
\textbf{Convergence Analysis of DP-SGD.}
DP-SGD with gradient clipping was initially proposed by~\citet{abadi_DeepLearning_2016}.
Asymptotically, under the Lipschitzness assumption, DP-SGD was shown to produce a tight utility-privacy tradeoff~\citep{bassily2019private}.
However, the understanding of DP-SGD remains limited without Lipschitz continuity.
The analysis of convergence rate of DP-SGD without Lipschitzness and bounded domain assumptions is a challenging task~\citep{wang2022differentially}.
On one hand, negative examples are shown in ~\citet{chen_UnderstandingGradient_2020,song2021evading} where DP-SGD in general might not converge.
On the other hand, \citet{DBLP:conf/iclr/FangLF023} showed that DP-SGD (with or without clipping) converges in the unbounded domain without Lipschitzness.
The main idea of~\citet{DBLP:conf/iclr/FangLF023} is to show that w.h.p. clipping never actually happens, at least under the widely used light-tail assumption (Assumption~\ref{ass:grad_concentration}).
Another work by~\citet{yang2022normalized} studied the convergence of DP-SGD under the generalized smoothness condition.
The concurrent work by~\citet{bu2023automatic} shows that a small clipping threshold can
yield promising performance in training language models.
The original implementation~\citep{abadi_DeepLearning_2016} of gradient clipping can be computationally inefficient as one needs to calculate the norm of the gradient for each individual sample in every iteration.
Due to its importance and centrality in private optimization and machine learning, there has been significant work on improving the efficiency of DP-SGD from both algorithmic and engineering perspectives; see~\citet{goodfellow2015efficient,abadi_DeepLearning_2016,rochette2019efficient,bu2021fast,subramani2021enabling} for some examples of such work.
\textbf{Analysis of (private) convergence to an $\alpha$-SOSP.}
An \emph{$\alpha$-first-order stationary point} ($\alpha$-FOSP, Definition~\ref{def:fosp}) is point $\vx$ so that $\|\nabla f(\vx)\| \le \alpha$.
Analyses of convergence to an $\alpha$-FOSP are a cornerstone of non-convex optimization (see, e.g., classical texts~\citet{bertsekas_NonlinearProgramming_1997,nocedal_NumericalOptimization_1999}).
Quantitative analysis of convergence to an $\alpha$-second-order stationary point ($\alpha$-SOSP, Definition~\ref{def:sosp}) started with the breakthrough work by~\citet{DBLP:conf/colt/GeHJY15}, further refined by~\citet{jin_HowEscape_2017,carmon_AnalysisKrylov_2018,jin_AcceleratedGradient_2018,carmon_FirstOrderMethods_2020,jin_NonconvexOptimization_2021} and most recently in~\citet{zhang_EscapeSaddle_2021}, who show an almost optimal bound.
Due to the prevalence of SGD in deep learning, stochastic methods have attracted the most attention (see~\citet{allen-zhu_NatashaFaster_2018,allen-zhu_NEON2Finding_2018,fang_SPIDERNearOptimal_2018,tripuraneni_StochasticCubic_2018,xu_FirstorderStochastic_2018,zhou_StochasticRecursive_2020,zhou_StochasticNested_2020} for the case of Lipschitz gradients and~\citet{DBLP:conf/colt/GeHJY15,daneshmand_EscapingSaddles_2018} for non-Lipschitz gradients).
For an in-depth summary of the previous work on unconstrained non-convex optimization, we refer the reader to~\citet{jin_NonconvexOptimization_2021}.
For private second-order convergence, a private version of SpiderBoost was proposed by~\citet{DBLP:conf/icml/AroraBGGMU23} and was then refined by~\citet{liu2023private}.
Since here we are interested in understanding the fundamental power of SGD rather than in obtaining optimal convergence rates, the main baseline we compare with is ~\citet{wang_DifferentiallyPrivate_2019,DBLP:conf/pkdd/WangX20}, which essentially uses noisy gradient descent.
Compared to this result, we allow stochastic gradients and make weaker assumptions about the structure of the problem.
%\todo[inline]{Say that our Algo doesn't need clipping}
%\todo[inline]{Be careful with~\citet{qiu2021unsupervised}}
%\mdnoteinline{I'm not familiar with~\citet{qiu2021unsupervised}, so I'm not sure what the above comment means}
%\todo[inline]{Paper outline}
\textbf{Roadmap.} In Section~\ref{sec:pre} we describe the basic definitions and assumptions. Section~\ref{sec:algorithm} describes our version of DP-SGD without clipping, as well as the privacy analysis and the outline of the convergence analysis. Empirical evaluations are provided in section~\ref{sec:experiments}, with additional experimental results provided in Appendix~\ref{app:experimental_details}.
The proof details excluded from the main body can be found in Appendix~\ref{app:convergence_proof}.
\documentclass[11pt]{article}
\usepackage{fullpage}
%\usepackage{amsmath,amssymb,graphicx,url,amsthm}
\usepackage[algo2e,ruled]{algorithm2e}
%\newtheorem{theorem}{Theorem}[section]
%\usepackage[rgb]{xcolor}
%\usepackage{hyperref}
%\newcommand{\citep}[1]{\cite{#1}}
\usepackage{natbib}
\usepackage[hyphens]{url}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{todonotes}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2024} with \usepackage[nohyperref]{icml2024} above.
\usepackage{hyperref}
\usepackage{nicefrac}
\usepackage{listings}
% \usepackage[ruled,vlined]{algorithm2e}
% \usepackage{algorithm2e}
\usepackage{algorithmic}
%\usepackage{algorithm}
\usepackage{paralist}
% Attempt to make hyperref and algorithmic work together better:
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2024
% ready for submission
%\usepackage{neurips_2024}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2024}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2024}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2024}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% If accepted, instead use the following line for the camera-ready submission:
% \usepackage[accepted]{icml2024}
\usepackage{algorithm}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{subcaption}
\usepackage{graphicx}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
% \newtheorem{assumption}[theorem]{Assumption}
% \newtheorem{assumption}[theorem]{Assumption}
\newtheorem{assumption}{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\setcounter{assumption}{0} %assign desired value to theorem counter
\renewcommand\theassumption{\Alph{assumption}}% locally redefine the representation of the assumption counter
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
% \usepackage[textsize=tiny]{todonotes}
% \usepackage{todonotes}
% \usepackage[disable]{todonotes}
\newcommand{\mdnote}[1]{\todo[color=red!40]{Mike's Note: #1}}
\newcommand{\mdnoteinline}[1]{\todo[inline, size=\normalsize, color=green!40]{Mike's Note: #1}}
%\setuptodonotes{inline}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\input{preamble}
\begin{document}
\title{Noise is All You Need: Private Second-Order Convergence of Noisy SGD}
\author{%
Dmitrii Avdiukhin \thanks{dmitrii.avdiukhin@northwestern.edu,
Northwestern University. }
%\footnotemark[1]
\and
Michael Dinitz \thanks{ mdinitz@cs.jhu.edu, Johns Hopkins University.}
%\footnotemark[2]
\and
Chenglin Fan \thanks{ fanchenglin@gmail.com, Seoul National University. }
%\footnotemark[3]
\and
Grigory Yaroslavtsev %\footnotemark[4]
\thanks{grigory@gmu.edu,
George Mason University.}
}
% \tableofcontents
% \listoftodos
\newpage
\setcounter{page}{1}
\maketitle
%\usepackage{times}
%\footnote{\footnotemark[1]UTD }
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2024
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\input{abstract}
\input{intro}
\input{prelims}
\input{algorithm}
\input{experiments}
\input{conclusion}
%\input{impact_statement}
\bibliography{main}
\bibliographystyle{plainnat}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\onecolumn
% \input{proof_dp}
\input{experimental_details}
\input{proof_convergence}
\input{proofs_no_nSG}
% \input{checklist}
\end{document}
\newcommand{\R}{\mathbb R}
\newcommand{\E}{\mathbb E}
\newcommand{\eps}{\varepsilon}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\SqrNorm}[1]{\left\| #1 \right\|^2}
\newcommand{\pars}[1]{\left( #1 \right)}
\newcommand{\sqbr}[1]{\left[ #1 \right]}
\newcommand{\set}[1]{\left\{ #1 \right\}}
\newcommand{\InnerProd}[1]{\left\langle #1 \right\rangle}
\newcommand{\Exp}[1]{\E \sqbr{#1}}
\newcommand{\Var}[1]{\mathbf{Var} \sqbr{#1}}
\newcommand{\ExpSqrNorm}[1]{\E \norm{#1}^2}
\newcommand{\Oh}[1]{O \pars{#1}}
\newcommand{\tOh}[1]{\tilde{O} \pars{#1}}
\newcommand{\Om}[1]{\Omega \pars{#1}}
\newcommand{\tOm}[1]{\tilde{\Omega} \pars{#1}}
\newcommand{\vzero}{\mathbf{0}}
\newcommand{\va}{\mathbf a}
\newcommand{\vb}{\mathbf b}
\newcommand{\vc}{\mathbf c}
\newcommand{\vg}{\mathbf g}
\newcommand{\vu}{\mathbf u}
\newcommand{\vv}{\mathbf v}
\newcommand{\vw}{\mathbf w}
\newcommand{\vx}{\mathbf x}
\newcommand{\vy}{\mathbf y}
\newcommand{\vz}{\mathbf z}
\newcommand{\vA}{\mathbf A}
\newcommand{\vB}{\mathbf B}
\newcommand{\vH}{\mathbf H}
\DeclareMathOperator*{\avg}{avg}
\DeclareMathOperator{\minimize}{minimize}
\newcommand{\x}[1]{\vx_{#1}}
\newcommand{\tx}[1]{\vx'_{#1}}
\newcommand{\difx}[1]{\bar{\vx}_{#1}}
\newcommand{\ourNoise}[1]{\xi_{#1}}
\newcommand{\tourNoise}[1]{\xi'_{#1}}
\newcommand{\difourNoise}[1]{\bar{\xi}_{#1}}
\newcommand{\sgdNoise}[1]{\zeta_{#1}}
\newcommand{\tsgdNoise}[1]{\zeta'_{#1}}
\newcommand{\difsgdNoise}[1]{\bar{\zeta}_{#1}}
\newcommand{\sgdErr}[1]{\mathrm{err}^{\mathrm{sgd}}_{#1}}
\newcommand{\quadErr}[1]{\mathrm{err}^{\mathrm{Q}}_{#1}}
\newcommand{\dif}[1]{\mathrm{dif}_{#1}}
\newcommand{\hesInt}[1]{\Gamma_{#1}}
\newcommand{\logErr}{\log \nicefrac{1}{\tdelta}}
\newcommand{\fmax}{f_{\max}}
\newcommand{\batch}[1]{\mathcal{B}_{#1}}
\newcommand{\g}[1]{\vg_{#1}}
\newcommand{\id}{\mathbf{I}}
\newcommand{\tdelta}{\tilde \delta}
\newcommand{\tsigma}{\tilde \sigma}
\newcommand{\fullVar}{\tilde \sigma}
\newcommand{\step}{\eta}
\renewcommand{\P}[1]{\mathbb P \sqbr{#1}}
% Algorithm Parameters
\newcommand{\sqrtra}{\sqrt{\rho \alpha}}
% \newcommand{\sqrtfacr}{\sqrt{\nicefrac {\alpha^3}{\rho}}}
\newcommand{\sqrtfacr}{\sqrt{\alpha^3 / \rho}}
\newcommand{\escIter}{\mathcal I}
\newcommand{\df}{\mathcal F}
\newcommand{\escRad}{\mathcal R}
\newcommand{\eigen}{\gamma}
\newcommand{\lmin}{\lambda_{\min}}
\newcommand{\cstep}{c_{\step}}
\newcommand{\cit}{c_{\escIter}}
\newcommand{\crad}{c_{\escRad}}
\newcommand{\cf}{c_{\df}}
% \newcommand{\cnoise}{\c_\Delta}
% Names
\newcommand{\algname}{\textsc{DP-SGD}\xspace}
\newcommand{\clip}[1]{\theta_{#1}}
\newcommand{\clipC}{\mathcal{C}}
\newcommand{\fclip}{f^{(c)}}
\newcommand{\subfigwidth}{0.48}
\section{Preliminaries}
\label{sec:pre}
We use $\| \cdot \|$ to denote the vector 2-norm or matrix operator norm if not otherwise specified.
We use $\avg_{i=1}^n \cdots$ to denote $\frac{1}{n} \sum_{i=1}^n \cdots$.
\begin{definition}[Differential Privacy~\citep{DBLP:conf/eurocrypt/DworkKMMN06}]
If for any two datasets $D$ and $D'$ which differ by a single data point and any $O\subseteq Range(\mathbb{A})$, an algorithm $\mathbb{A}$ satisfies
\[
\P{\mathbb{A}(D)\in O} \leq e^\eps \cdot \P{\mathbb{A}(D')\in O} + \delta
,
\]
then algorithm $\mathbb{A}$ is said to be ($\eps, \delta)$-Differentially Private ($(\eps, \delta)$-DP).
\end{definition}
%\todo{What is $f$ above? Cite DMNS instead of Dwork}
% Throughout this paper, for any positive integer $n$, we denote $[n] := \{1,2,\ldots,n\}$.
We want to minimize function $f(\vx) = \avg_{i=1}^N f_i(\vx)$, where each $f_i$ is a continuously differentiable function $\R^d \to \R$. We make the following assumptions which are widely used for the analysis of high probability convergence of SGD~\cite{DBLP:journals/siamjo/NemirovskiJLS09, DBLP:journals/siamjo/GhadimiL13a, fang_SPIDERNearOptimal_2018, DBLP:conf/colt/HarveyLPR19, feldman2020private,DBLP:conf/iclr/FangLF023}.\footnote{
Assumption~\ref{ass:main}.\ref{ass:stoc} is usually stated as $\Exp{\SqrNorm{g(\vx) - \nabla f(\vx)}} \le \sigma^2$, where $g$ is the stochastic gradient oracle.
Assumptions~\ref{ass:main}.1 - \ref{ass:main}.3 are standard in non-convex settings for convergence to a first-order stationary point, while additional Assumption~\ref{ass:main}.\ref{ass:lip_hes} is standard for convergence to a second-order stationary point.
}
%\todo{Refs that the assumptions are standard}
%
\begin{assumption}
\label{ass:main}
The objective function $f$ and the component functions $f_i$ satisfy the following conditions.
\begin{enumerate}
\item (Boundedness)
There exists $\fmax$ such that $\fmax > f(\x{0}) - f(\vx^*)$.
\item\label{ass:stoc} (Stochastic variance) The stochastic variance is bounded by $\sigma^2$, i.e. $\avg_{i=1}^N \norm{\nabla f_i(\vx) - \nabla f(\vx)}^2 \le \sigma^2$.
\item (Smoothness) The objective function $f$ is $L$-smooth, i.e. $\norm{\nabla f(\vx) - \nabla f(\vy)} \le L \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.
\item\label{ass:lip_hes} (Lipschitz Hessian) The objective function $f$ has a $\rho$-Lipschitz Hessian, i.e. $\norm{\nabla^2 f(\vx) - \nabla^2 f(\vy)} \le \rho \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.
\end{enumerate}
\end{assumption}
%
For a minibatch $\batch{t}$, we define $f_{\batch{t}} = \avg_{i \in \batch{t}} f_i$.
\begin{definition}
\label{def:norm_subgaussian}
A random vector $X$ is norm-subgaussian (or $\text{nSG}(\sigma^2)$) if $\Exp{\exp\pars{\frac{\SqrNorm{X - \Exp{X}}}{\sigma^2}}} \le e$.
\end{definition}
%
We make the following assumption, which is used to obtain high-probability bounds.
% This assumption is required in optional, see Appendix~\ref{app:convergence_proof} for the analysis without this assumption, which results in polynomial dependence on the error probability.
\begin{assumption}[Gradient concentration]
\label{ass:grad_concentration}
For any $\vx$, the stochastic gradient at $\vx$ is $\text{nSG}(\sigma^2)$:
\[
\avg\nolimits_{i=1}^N \exp \pars{\frac{\norm{\nabla f_i(\vx) - \nabla f(\vx)}^2}{\sigma^2}} < e
\]
\end{assumption}
The above assumption is required in \Cref{lem:grad_bound} to get logarithmic dependence on $\delta$, but is not required in our convergence analysis if one is willing to add sufficiently large noise.
%
%Since finding a global, or even a local, minimum of a non-convex function is NP-hard in general~\citep{anandkumar_EfficientApproaches_2016}, the common goal is to find an approximate stationary point.
An approximate first-order stationary point requires that the gradient is small.
\begin{definition}[$\alpha$-FOSP]
\label{def:fosp}
We say that $\vx$ is an $\alpha$-first order stationary point ($\alpha$-FOSP) for a differentiable function $f$ if $\|\nabla f(\vx)\| < \alpha$.
\end{definition}
%
A second-order stationary point additionally requires that the function is almost convex near the point.
%
\begin{definition}[$\alpha$-SOSP~\citep{nesterovP06}]
\label{def:sosp}
We say that $\vx$ is an $\alpha$-second order stationary point ($\alpha$-SOSP) for a twice-differentiable function $f$ if $\|\nabla f(\vx)\| < \alpha$ and $\lmin(\nabla^2 f(\vx)) > -\sqrtra$.
\end{definition}
\section{Convergence Proof}
% \begin{assumption}
% \label{ass:main_new}
% The objective function $f$ and the component functions $f_i$ satisfy the following conditions.
% \begin{enumerate}
% {\color{green}\item (Boundedness)
% There exists $\fmax$ such that $\fmax > f(\x{0}) - f(\vx^*)$.}
% {\color{green}\item (Smoothness) $f$ is $L$-smooth, i.e. $\norm{\nabla f(\vx) - \nabla f(\vy)} \le L \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.\todo{Check that we cak reuse the gradient norm bound}}
% {\color{green}\item (Lipschitz Hessian) The objective function $f$ has a $\rho$-Lipschitz Hessian, i.e. $\norm{\nabla^2 f(\vx) - \nabla^2 f(\vy)} \le \rho \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.}
% {\color{red}\item (Relaxed Stochastic variance) The stochastic variance is bounded as follows: $\avg_{i=1}^N \SqrMorn{\nabla f_i(\vx) - \nabla f(\vx)} \le \sigma_1^2 \SqrNorm{\nabla f(\vx)} + \sigma_2^2$.}
% \end{enumerate}
% \end{assumption}
% \todo{Rewrite}
% \begin{assumption}
% {\color{red}No nSG assumption.}
% \end{assumption}
% \begin{assumption}
% {\color{red}Clipping. (Not doable quickly)}
% \end{assumption}
\label{app:convergence_proof}
% \todo{Mention that mostly follows Jin et al.}
\begin{definition}
\label{def:params}
Let $\fullVar = \nicefrac{\sigma^2}{B} + d \Delta^2$ be total noise variance.
Our choice of parameters is the following ($\cstep, \cit, \crad, \cf$ hide polylogarithmic dependence on all parameters):
\begin{equation*}
\begin{aligned}
\text{Step size} && \step &= \cstep \min\pars{\frac{\alpha^2}{L (1 + \tsigma^2)}, \frac{\fmax}{L \tsigma^2}} \\
\text{Total number of iterations} && T &= \frac{64\fmax}{\step \alpha^2} \\
\text{Iterations required for escaping} && \escIter &= \cit \frac {1} {\step \sqrtra} \\
\text{Escaping radius} && \escRad &= \crad \sqrt {\alpha / \rho} \\
\text{Objective decrease after successful escaping} && \df &= \cf \sqrtfacr
% \text{Noise standard deviation} && r &= \cnoise \frac \alpha {\sqrt{\gradLip \step}}
\end{aligned}
% \label{eq:params}
\end{equation*}
\end{definition}
Intuitively, the parameters are chosen as follows.
Assume that the current point is $\vzero$ and the function is quadratic, i.e. $f(\vx) = \vx^T H \vx$.
If $\vx$ is a saddle point, then we know that the smallest eigenvalue of $H$ is $-\eigen < -\sqrtra$ with the corresponding eigenvector $\vv_1$.
Then, the largest function decrease would be in the direction of $\vv_1$, and this direction can be found using power method for the appropriate matrix.
But, if we look at the gradient descent update:
\[
\x{t+1}
= \x{t} - \step \nabla f(\x{t})
= \x{t} - \step 2 H \x{t}
= (\id - \step 2 H) \x{t},
\]
we see that for quadratic functions, gradient descent is exactly the power method for the matrix $\id - \step 2H$, which has the same eigenvectors as $H$. The eigenvalue $-\eigen$ of $H$ corresponds to the eigenvalue $1 + 2 \step \eigen$ of $1 - \step 2 H$, which is the largest (by absolute value) eigenvalue when $\step \le 1/L$.
Note that power method requires some initial randomness, which explains while randomness is required to escape from saddle points.
The above intuition motivates the choice of parameters.
\begin{itemize}
\item
\textbf{Escaping radius $\escRad$}. Since in general the function is not necessarily quadratic, we want to guarantee that the Hessian doesn't change excessively.
More specifically, we want the smallest eigenvalue to be at most $-\eigen / 2$ for the entire escaping procedure.
By the Hessian-Lipschitz property $\|\nabla^2 f(\vx) - \nabla^2 f(\vy)\| \le \rho \|\vx - \vy\|$, if we work in the radius $\escRad = \eigen /(2\rho)$, then for any point within the radius $\escRad$ of the initial point, the smallest eigenvalue is at most $-\eigen + \eigen / 2 = -\eigen / 2$.
When $\eigen > \sqrtra$, we can lower-bound $\escRad$ as $\sqrtra / (2\rho) = \sqrt{\alpha / (4\rho)}$.
\item
\textbf{Number of iterations required for escaping $\escIter$}.
After $t$ iterations, the length of $\x{t}$ along the $\vv_1$ direction is $(1 + \step \eigen)^t \approx e^{t / (\step \eigen)}$.
Hence, we reach the boundary of the ball of radius $\escRad$ after $(t \log \escRad) / (\step \eigen)$ iterations.
\item
\textbf{Function change after escaping $\df$}.
Since we are working inside the ball of size $\escRad$, the best objective improvement $\df$ we can aim for is achievable by moving by $\escRad \vv_1$.
This improves the objective by approximately $\escRad^2 \eigen$, which, by the choice of the parameters can be lower-bounded as $\alpha / \rho \cdot \sqrtra = \sqrt{\alpha^3 / \rho}$.
\item
\textbf{Total number of iterations $T$}. Every $\escIter$ iterations we improve objective by $\df$.
Since the total objective improvement cannot exceed $\fmax$, the maximum number of iterations is
\[
\frac{\fmax \escIter}{\df}
\le \frac{\fmax}{(\step \sqrtra)\sqrt{\alpha^3 / \rho}}
= \frac{\fmax}{\step \alpha^2}
\]
\end{itemize}
\begin{lemma}[Descent Lemma]
\label{lem:descent}
Assume that $\step < 1/L$, for any $t_0$ and $t$.
Then, with probability at least $1 - \tdelta$, for some constant $c$:
\[
f(\x{t_0}) - f(\x{t_0 + t})
\ge \frac{\step}{8} \sum_{\tau=t_0}^{t_0 + t - 1} \SqrNorm{\nabla f(\x{\tau})} - c \step \tsigma^2 (\step L t + \logErr)
.
\]
\end{lemma}
\begin{proof}
W.l.o.g., we consider $t_0 = 0$.
Let $\sgdNoise{\tau} = \nabla f_{\batch{t}}(\x{\tau}) - \nabla f(\x{\tau})$ be the stochastic noise at iteration $\tau$.
By the folklore Descent Lemma,
\begin{align*}
f(\x{\tau}) - f(\x{\tau+1})
% \ge \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \nabla f_{\batch{t}}(\x{\tau})}
% - \frac{L}{2} \SqrNorm{\x{}}
&\ge \step \InnerProd{\nabla f(\x{\tau}), \nabla f_{\batch{t}}(\x{\tau})}
- \frac{L}{2} \SqrNorm{\x{\tau + 1} - \x{\tau}} \\
&= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}} \\
&\ge \step \SqrNorm{\nabla f(\x{\tau})} + \step \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{\step^2 L}{2} \pars{\frac{3}{2}\SqrNorm{\nabla f(\x{\tau})} + 3 \SqrNorm{\sgdNoise{\tau} + \ourNoise{\tau}}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
,
\end{align*}
where we used inequality $\SqrNorm{a + b} \le (1 + \nu) \SqrNorm{a} + (1 + 1/\nu) \SqrNorm{b}$ for $\nu = 1/2$.
By telescoping, we have:
\begin{align*}
f(\x{0}) - f(\x{t})
&\ge \frac{\step}{4} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})}
+ \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{3 \step^2 L}{2} \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau} + \ourNoise{\tau}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
\end{align*}
%
Using a concentration inequality for nSG variables~\citep{jin_ShortNote_2019}, with probability at least $1 - \tdelta$ for some constant $c$ we have
\[
\abs{\step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}}
\le \frac{\step}{8} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})} + c \step \fullVar^2 \logErr
\]
For the last term, with probability at least $1 - \tdelta$ we have
\[
\frac{3 \step^2 L}{2} \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau} + \ourNoise{\tau}}
\le c \step^2 L \fullVar^2 (t + \logErr).
\]
Combining the above inequalities and using $\step \le 1/L$ finishes the proof.
% \todo{TODO}
\end{proof}
\subsection{Large Gradient Case}
The first application of the Descent Lemma is to bound the number of iterations when the gradient is large.
\begin{lemma}
\label{lem:quarter_large_grad}
When $T \ge \frac{64 \fmax}{\step \alpha^2}$ and other parameters are as in \Cref{def:params}, the number of iterations $t$ with $\|\nabla f(\x{t})\| \ge \alpha$ is at most $\nicefrac T4$.
\end{lemma}
\begin{proof}
For the sake of contradiction, assume that the number of points $\x{t}$ with $\|\nabla f(\x{t})\| \ge \alpha$ is greater than $\nicefrac T4$.
Then, by the Descent Lemma we have
\begin{align*}
f(\x{0}) - f(\x{T})
&\ge \frac{\step}{8} \sum_{\tau=0}^{T - 1} \SqrNorm{\nabla f(\x{\tau})}
- c \step \tsigma^2 (\step L T + \logErr) \\
&\ge \frac{\step T \alpha^2}{32} - c \step T \cdot \tsigma^2 \pars{\step L + \frac{\logErr}{T}}
\end{align*}
We will show that the right-hand side is at least $\fmax$, leading to contradiction.
For that, it suffices to guarantee that $\frac{\step T \alpha^2}{32} \ge 2\fmax$ and $c \step T \cdot \tsigma^2 \pars{\step L + \frac{\logErr}{T}} \le \fmax$.
The former condition is satisfied by our choice of $T$, and substituting the bound on $\fmax$ into the latter condition shows that it suffices to prove
\[
64 c \tsigma^2 \pars{\step L + \frac{\step \alpha^2 \logErr}{64 \fmax}} \le \alpha^2
\]
It suffices to bound each term with $\alpha^2 / 2$, which is satisfied for
\begin{align*}
\step &\le \frac{\alpha^2}{128 \cdot c \tsigma^2 L} \\
\step &\le \frac{\fmax\logErr}{2 \cdot c \tsigma^2 L}
\end{align*}
Both of these conditions are satisfied by our choice of $\step$, with the appropriate choice of $\cstep$.
\end{proof}
\subsection{Escaping From Saddle Points}
The Descent Lemma also implies the following result, which upper-bounds the distance change in terms of the objective changes.
Its \Cref{cor:i_or_l} states that, if after a bounded number of iterations the point moves far from the original point, then the objective value sufficiently decreases.
%
\begin{lemma}[Improve or localize]
\label{lem:improve_or_localize}
For any $t_0$ and $t$, with probability at least $1 - \tdelta$, for some constant $c$:
\begin{align*}
\SqrNorm{\x{t_0 + t} - \x{t_0}}
\le c \step t \pars{(f(\x{t_0}) - f(\x{t_0 + t}) + \step \tsigma^2 (\step L t + \logErr)}
\end{align*}
\end{lemma}
%
\begin{proof}
W.l.o.g. we consider the case when $t_0 = 0$.
% By the Descent Lemma,
% \[
% f(\x{0}) - f(\x{t})
% \ge \frac{\step}{8} \sum_{\tau=0}^{t - 1} \SqrNorm{\nabla f(\x{\tau})} - c \step \tsigma^2 (\step L t + \logErr)
% \]
By the definition of $\x{\tau + 1}$, we have
\begin{align*}
\x{\tau} - \x{\tau + 1}
= \step (\nabla f_{\batch{\tau}}(\x{\tau}) + \ourNoise{\tau})
= \step (\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}),
\end{align*}
where $\sgdNoise{\tau} = \nabla f_{\batch{\tau}}(\x{\tau}) - \nabla f(\x{\tau})$ is stochastic gradient noise.
By telescoping, we have:
\begin{align*}
\|\x{0} - \x{t}\|
&= \step \norm{\sum_{\tau = 0}^{t - 1}(\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau})}
\end{align*}
By Cauchy-Schwarz Inequality, we have:
\begin{align*}
\SqrNorm{\x{0} - \x{t}}
&\le 2 \step^2 \pars{\SqrNorm{\sum_{\tau = 0}^{t - 1}\nabla f(\x{\tau})}
+ \SqrNorm{\sum_{\tau = 0}^{t - 1} (\sgdNoise{\tau} + \ourNoise{\tau})}} \\
&\le 2 \step^2 \pars{t \sum_{\tau = 0}^{t - 1}\SqrNorm{\nabla f(\x{\tau})}
+ \SqrNorm{\sum_{\tau = 0}^{t - 1} (\sgdNoise{\tau} + \ourNoise{\tau})}}
\end{align*}
Since $\sgdNoise{\tau} + \ourNoise{\tau}$ is $\mathrm{nSG}(\fullVar^2)$, by applying concentration inequality for $\mathrm{nSG}$ variables, we have with probability at least $1 - \tdelta$:
\begin{align*}
\SqrNorm{\x{0} - \x{t}}
&\le 2 \step^2 \pars{t \sum_{\tau = 0}^{t - 1}\SqrNorm{\nabla f(\x{\tau})} + c t \fullVar^2 \logErr}
\end{align*}
Finally, applying the Descent Lemma, we get
\begin{align*}
\SqrNorm{\x{0} - \x{t}}
&\le \step c t \pars{f(\x{0}) - f(\x{t}) + \step \fullVar^2 (\step L t + \logErr)}
\end{align*}
for some constant $c$.
\end{proof}
\begin{corollary}
\label{cor:i_or_l}
For the parameters chosen as in \Cref{def:params}, if for some $t \le \escIter$ we have $\SqrNorm{\x{t_0 + \tau} - \x{t_0}} \ge \escRad$, then $f(\x{t_0}) - f(\x{t_0 + \tau}) \ge \df$ with probability at least $1 - \tdelta$.
\end{corollary}
%
\begin{proof}
From \Cref{lem:improve_or_localize}, we know that for some $c$:
\begin{align*}
f(\x{t_0}) - f(\x{t_0 + t})
&\ge \frac{\escRad^2}{c \step \escIter} - \step \tsigma^2 (\step L \escIter + \logErr) \\
&\ge \frac{\crad^2}{c \cit} \sqrtfacr - \step \tsigma^2 \pars{\frac{\cit L}{\sqrtra} + \logErr}
% &\ge \frac{\SqrNorm{\x{t_0 + t} - \x{t_0}}}{c \step \escIter} - \step \tsigma^2 (\step L \escIter + \log 1/\fullVar)
\end{align*}
We want to guarantee that the right-hand side is at most $\df = \cf \sqrtfacr$.
It suffices to guarantee that
\begin{align*}
\crad^2 / (c \cit) \ge 2 \cf \qquad \text{and} \qquad \step \tsigma^2 (\cit L / \sqrtra + \logErr) \le \cf \sqrtfacr.
\end{align*}
The former holds for the appropriate choice of $\crad$, $\cit$, and $\cf$, and the latter holds by choosing
\[
\step \le (\alpha^2 / L \tsigma^2) \cdot \cf / \pars{\cit + \logErr},
\]
where we assumed $L \ge \sqrtra$, since otherwise any $\alpha$-FOSP is an $\alpha$-SOSP because $L$ is an upper bound on the absolute values of the Hessian.
\end{proof}
\begin{definition}[Coupling sequences]
\label{def:coupling_sequence}
Let $\vv_1$ be the eigenvector corresponding to the smallest eigenvalue of $\nabla^2 f(\x{t_0})$.
The coupling sequences starting from $\x{t_0}$ are defined as follows:
\begin{equation}
\begin{aligned}
\ourNoise{t_0+t} &\sim \mathcal{N}(0, \fullVar^2 \id) \\
\tourNoise{t_0+t} &= \ourNoise{t_0+t} - 2 \InnerProd{ \vv_1, \ourNoise{t_0+t} } \vv_1 \\
% \batch'_{t_0+t} &= \batch_{t_0+t}, \\
\x{t_0 + t+1} &= \x{t_0+t} - \step (\nabla f_{\batch{t}}(\x{t_0 + t}) + \ourNoise{t_0 + t}) \\
\tx{t_0 + t+1} &= \tx{t_0+t} - \step (\nabla f_{\batch{t}}(\tx{t_0 + t}) + \tourNoise{t_0 + t}) \\
\sgdNoise{t_0 + t} &= \nabla f_{\batch{t}}(\x{t_0 + t}) - \nabla f(\x{t_0 + t}) \\
\tsgdNoise{t_0 + t} &= \nabla f_{\batch{t}}(\tx{t_0 + t}) - \nabla f(\tx{t_0 + t}) \\
\end{aligned}\nonumber
\end{equation}
%
Note that:
\begin{itemize}
\item the artificial noise $\tourNoise{t_0 + t}$ has the opposite sign in direction $\vv_1$ compared with $\ourNoise{t_0 + t}$,
% \item the stochastic randomness ${\batch{t}}$ is the same for both sequences,\todo{Might be not required, but anyways makes things simpler to define}
\item $\{\x{t_0 + t}\}_t$ and $\{\tx{t_0 + t}\}_t$ are sampled from the same distribution.
\end{itemize}
%
We next define the difference between the coupling sequences:
\begin{equation}
\begin{aligned}
\difourNoise{t} &= \xi_{t_0+t} - \xi'_{t_0+t}
= 2 \InnerProd{ \vv_1, \xi_{t_0+t} } \vv_1 \\
% \batch'_{t_0+t} &= \batch_{t_0+t}, \\
\difx{t_0 + t+1} &= \x{t_0+t} - \tx{t_0+t} \\
\difsgdNoise{t_0 + t} &= \sgdNoise{t_0 + t} - \tsgdNoise{t_0 + t}
\end{aligned}\nonumber
\end{equation}
\end{definition}
% TODO (the rest is standard)
% SGD with variance $\sigma^2$ and batch size $B$, then the number of iterations to find an $\alpha$-FOSP ($\|\nabla f(\vx)\| < \alpha$) is $\frac{\sigma^2}{B \alpha^4}$.
From now on, we assume that we want to escape from point $\x{t_0}$ such that $\lmin(\nabla^2 f(\x{t_0})) < -\sqrtra$.
W.l.o.g. we assume $t_0 = 0$.
Let $H = \nabla^2 f(\x{0})$ be the Hessian of $f$ at point $\x{0}$, let $-\eigen = \lmin(H)$ be the smallest eigenvalue of $H$, and $\vv_1$ be the eigenvector corresponding to $-\eigen$.
\begin{lemma}
For every $t$,
\[
\difx{t} = -\dif{t} - \sgdErr{t} -\quadErr{t},
\]
where
\begin{align*}
\dif{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difourNoise{\tau} \\
\sgdErr{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difsgdNoise{\tau} \\
\quadErr{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \hesInt{\tau} \difx{\tau},
\quad \text{ where } \quad \hesInt{\tau} = \int_0^1 (\nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) - H)\ d\psi
\end{align*}
\end{lemma}
\begin{proof}
Proof by induction.
\begin{align*}
\difx{t + 1}
&= \difx{t} - \step (\difsgdNoise{t} + \difsgdNoise{t}) - \step (\nabla f(\x{t}) - \nabla f(\tx{t})) \\
&= (1 - \step H) \difx{t} - \step (\difsgdNoise{t} + \difsgdNoise{t}) - \step (\nabla f(\x{t}) - \nabla f(\tx{t}) - H \difx{t})
\end{align*}
Using that $\nabla f(\x{t}) - \nabla f(\tx{t}) = \int_0^1 \nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) \difx{\tau}\ d\psi$, by telescoping we complete the proof.
\end{proof}
%
In the following statements, the geometric series $\alpha_t^2 = \sum_{\tau = 0}^{t-1} (1 + \step \eigen)^{2 \tau}$ occurs frequently.
Intuitively, each iteration increases the distance approximately by a factor of $(1 + \step \eigen)$~-- the largest eigenvalue of matrix $I - \step H$.
The noise added at iteration $\tau$ gets amplified by $(1 + \step \eigen)^{t - \tau}$ by iteration $t$, and, if every iteration an independent noise of variance $\Delta^2$ is added, $\Delta^2 \alpha_t^2$ is the total variance of such a noise accumulated over the iterations.
%
\begin{lemma}
\label{lem:alpha_beta}
Let $\alpha_t = \sqrt{\sum_{\tau = 0}^{t-1} (1 + \step \eigen)^{2 \tau}}$ and $\beta_t = (1 + \step \eigen)^t / \sqrt{\step \eigen}$.
If $\step \eigen < 1$, then $\alpha_t \le \beta_t$ for any $t$ and $\alpha_t \ge \beta_t / 3$ for $t \ge 2 / (\step \eigen)$.
\end{lemma}
\begin{proof}
As a sum of geometric progression, we have
\[
\alpha_t^2
= \frac{(1 + \step \eigen)^{2t} - 1}{(1 + \step \eigen)^2 - 1}
= \frac{(1 + \step \eigen)^{2t} - 1}{\step \eigen (2 + \step \eigen)},
\]
and $\alpha_t^2 \le \beta_t^2$ follows immediately.
On the other hand, for $t \ge 1 / (\step \eigen)$ we have $(1 + \step \eigen)^{2t} \ge 2$, and hence $(1 + \step \eigen)^{2t} - 1 \ge (1 + \step \eigen)^{2t} / 2$.
Finally, by our choice of the step size, we have $\step \eigen \le \step L \le 1$, and hence $2 + \step \eigen \le 3$.
This implies $\alpha_t^2 \ge \beta_t^2 / 6$, finishing the proof.
\end{proof}
\begin{lemma}
\label{lem:good_term}
Let $\beta_t = (1 + \step \eigen)^t / \sqrt{\step \eigen}$.
Then for any $t$ and $\tdelta$, for some constant $c$:
\begin{align*}
\P{\|\dif{t}\| \ge c \beta_t \step \Delta \sqrt{\logErr}} &\le \tdelta \\
\P{\|\dif{t}\| \le \frac{\beta_t \step \Delta}{10}} &\le \frac 13
\end{align*}
\end{lemma}
\begin{proof}
Since $\difourNoise{\tau} = 2\InnerProd{\ourNoise{\tau}, \vv_1} \vv_1$ and $\vv_1$ is an eigenvalue of $H$ corresponding to eigenvalue $-\eigen$, we have
\[
\dif{t}
= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difourNoise{\tau}
= \pars{\step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{t - 1 - \tau} \InnerProd{\ourNoise{\tau}, \vv_1}} \vv_1
\]
Hence, $\dif{t}$ is a random variable which is parallel to $\vv_1$.
% Hence, $\dif{t}$ is a Gaussian random variable along direction $\vv_1$ with variance $\step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{t - 1 - \tau} \InnerProd{\ourNoise{\tau}, \vv_1}$
Since each $\ourNoise{t}$ is a Gaussian random variable with the covariance matrix $\Delta^2 \id$, each $\InnerProd{\ourNoise{t}, \vv_1}$ a Gaussian random variable with variance $\Delta^2$.
Hence, $\dif{t}$ is also a Gaussian random variable, and
\[
\Var{\|\dif{t}\|}
= \step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{\tau} \Delta^2
= \Delta^2 \alpha_t^2
\]
The upper bound on $\|\dif{t}\|$ follows from \Cref{lem:alpha_beta} and standard concentration inequality for Guassians.
The lower bound follows from inequality $\P{|X| \le \sigma a} \le a$ for $X \sim \mathcal{N}(0, \sigma^2)$ and any $a$.
\end{proof}
\begin{lemma}
\label{lem:error_terms}
Let $\mathcal{E}_t$ be the event ``$\|\x{0} - \x{\tau}\| < \escRad$ and $\|\x{0} - \tx{\tau}\| < \escRad$ for all $\tau < t$''.
Then, when $\Delta^2 \ge \sigma^2 / B$, for some constant $c$ we have
% \todo[inline]{Fix log error}
\[
\P{\mathcal{E}_\escIter
\implies \max\nolimits_{\tau=0}^{\escIter - 1} \norm{\sgdErr{\tau} + \quadErr{\tau}}
\le
c \beta_\tau \step \Delta \sqrt{\logErr}
} \ge 1 - \escIter \tdelta
\]
\end{lemma}
\begin{proof}
By induction on $t$, we will prove that $\P{\mathcal{E}_t \implies \|\difx{t}\| \le c \step \Delta \beta_t} \le 1 - t\fullVar$ and
\[
\P{\mathcal{E}_t
\implies \max\nolimits_{\tau=0}^{t - 1} \norm{\sgdErr{\tau} + \quadErr{\tau}}
\le c \beta_\tau \step \Delta \sqrt{\logErr}
} \ge 1 - t \tdelta.
\]
Base case is $t=0$, in which case $\difx{0} = \vzero$.
For the induction step, since $\difx{t} = -\dif{t} - \sgdErr{t} -\quadErr{t}$, we bound each term separately.
\paragraph{Bounding $\dif{t}$}
By \Cref{lem:good_term}, we have $\P{\|\dif{t}\| \ge c \beta_t \step \Delta \sqrt{\logErr}} \le \tdelta$.
\paragraph{Bounding $\quadErr{t}$}
Assuming $\mathcal{E}_t$, by the Hessian-Lipschitz property we have
\begin{align*}
\|\hesInt{\tau} \difx{\tau}\|
&= \norm{\int_0^1 (\nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) - H)\ d\psi \cdot \difx{\tau}} \\
&\le 2 \rho \escRad \|\difx{\tau}\| \\
&\le 2 \rho \escRad \cdot c \beta_\tau \step \Delta \sqrt{\logErr},
\end{align*}
%
and we bound $\quadErr{t}$ as
%
\begin{align*}
\|\quadErr{t}\|
&= \norm{\step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \hesInt{\tau} \difx{\tau}} \\
&\le \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{t - 1 - \tau} 2 \rho \escRad \cdot c \beta_\tau \step \Delta \sqrt{\logErr} \\
% &\le \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{\tau} 2 \rho \escRad \cdot c \beta_t \step \fullVar^2 \sqrt{\logErr} \\
\end{align*}
Since $(1 + \step \gamma)^{t - 1 - \tau} \beta_\tau = \beta_{t-1}$,
% \[
% \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{\tau}
% = \frac{(1 + \step \gamma)^t - 1}{\step \gamma}
% \le \frac{\beta_t}{\sqrt{\step \gamma}},
% \]
we can bound $\|\quadErr{t}\|$ as
\begin{align*}
\|\quadErr{t}\|
&\le 2 \rho \escRad \cdot c \escIter \beta_t \step \Delta \sqrt{\logErr} \\
&= 2 c \crad \cit \beta_t \Delta \sqrt{\logErr},
\end{align*}
where the last equality is due to our choice of parameters in \Cref{def:params}:
\[
\escRad \escIter
= \crad \sqrt {\alpha / \rho} \cdot \cit \frac {1} {\step \sqrtra}
= \frac{\crad \cit}{\rho \step}
\]
\paragraph{Bounding $\sgdErr{t}$}
% To bound $\sgdErr{t}$, we use that $\difsgdNoise{t}$ is $\mathrm{nSG}(\sigma)$:
% \begin{align*}
% \|\difsgdNoise{t}\|
% = \|\nabla f_{\batch{t}}(\x{t}) - \nabla f_{\batch{t}}(\tx{t})\|
% \le L \|\difx{t}\|
% \end{align*}
%
By \Cref{ass:grad_concentration}, each $\sgdErr{t}$ is $\mathrm{nSG}(\sigma^2 / B)$.
Using the concentration inequality for nSG variables, with probability at least $1 - \tdelta$ we have
\begin{align*}
\|\sgdErr{t}\|
&= \norm{\step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difsgdNoise{\tau}} \\
&\le \step \sqrt{\sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{2(t - 1 - \tau)} \frac{\sigma^2}{B}} \cdot \sqrt{\logErr} \\
&\le \step \sigma \beta_t \sqrt{\logErr} / \sqrt{B} \\
\end{align*}
Using assumption $\Delta^2 \ge \sigma^2 / B$, we get the desired bound.
\end{proof}
\begin{lemma}
\label{lem:bucket_change}
When $\Delta^2 \ge \sigma^2 / B$, with probability at least $1 - \escIter \tdelta$, we have
\[
f(\x{0}) - f(\x{\escIter}) > -\df / 100
\]
Moreover, with probability at least $1/3 - \escIter \tdelta$, we have:
\[
f(\x{0}) - f(\x{\escIter}) > \df
\]
\end{lemma}
\begin{proof}
The first claim follows from the descent lemma by our choice of parameters:
\[
f(\x{0}) - f(\x{\escIter})
> -c \step \fullVar^2 (\step L \escIter + \logErr)
> -\df / 10
\]
From Corollary~\ref{cor:i_or_l}, we know that with probability at least $1-\tdelta$, if $\|\x{t} - \x{0}\| > \escRad$ for some $t \le \escIter$, then $f(\x{0}) - f(\x{t}) > \df$ with probability at least $1 - \tdelta$.
Since the $\tx{t}$ has the same distribution as $\x{t}$, the same holds for $\|\tx{t} - \x{0}\|$.
Hence, if either $\|\x{t} - \x{0}\| > \escRad$ or $\|\tx{t} - \x{0}\| > \escRad$, then with probability $\frac{1}{2} - 2 \tdelta$ we have $f(\x{0}) - f(\x{t})$.
For the rest of the proof, we can assume that $\|\x{t} - \x{0}\| < \escRad$ and $\|\tx{t} - \x{0}\| < \escRad$ for all $t \le \escIter$.
By Lemmas~\ref{lem:good_term} and~\ref{lem:error_terms} we know that $\|\dif{\escIter}\| \ge \step \beta_t \Delta \sqrt{\logErr} / 10$ and $\|\sgdErr{t} + \quadErr{t}\| \le \step \beta_t \Delta \sqrt{\logErr} / 20$ with probability $1 - \tdelta$.
Hence,
\begin{align*}
\max(\|\x{\escIter} - \x{0}\|, \|\tx{\escIter} - \x{0}\|)
&\ge \frac{1}{2} \|\difx{\escIter}\| \\
&\ge \frac 12 \pars{\|\dif{\escIter}\| - \|\sgdErr{\escIter} + \quadErr{t}\|} \\
&\ge \frac{\step \beta_\escIter \Delta}{40} \sqrt{\logErr} \\
&= (1 + \step \eigen)^\escIter \frac{1}{40} \Delta \sqrt{\frac{\step}{\eigen}} \sqrt{\logErr} \\
&\ge \escRad,
\end{align*}
where the last inequality follows by the appropriate choice of logarithmic term in $\cit$ (note that $(1 + \step \eigen)^\escIter$ dominates other terms).
This contradicts the assumption $\max(\|\x{\escIter} - \x{0}\|, \|\tx{\escIter} - \x{0}\|) \le \escRad$.
\end{proof}
\begin{lemma}
\label{lem:quarter_saddle}
When $\Delta^2 \ge \sigma^2 / B$, for $T = 64 \fmax / (\step \alpha^2)$, with $\step$ chosen as in \Cref{def:params}, with probability at least $1 - T \tdelta$ we have $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ for at most $T/4$ points $\x{t}$.
\end{lemma}
\begin{proof}
Proof by contradiction.
Let's split all iterations $0,...,T-1$ into buckets of size $\escIter$.
There are $T/\escIter = 64\fmax \sqrtra / \cit$ buckets, and at least quarter of the buckets have a point $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ in it.
Note that within each bucket, by Lemma~\ref{lem:bucket_change}, the objective increases by at most $\df / 100$.
Out of such buckets, at least half of them are non-adjacent, and we consider points $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ in these buckets.
With probability at least $1/3 - \tdelta$, we have $f(\x{t}) - f(\x{t + \escIter}) > \df$ for each such $\x{t}$.
% \todo[inline]{Properly handle logs}
Hence,
\begin{align*}
\fmax
&\ge \frac{T \df}{10\cdot \escIter} \\
&= \frac{1}{10} \cdot \frac{64\fmax}{\step \alpha^2} \cdot \cf \sqrtfacr \cdot \frac{\step \sqrtra}{\cit} \\
&= \fmax \frac{64 \cdot \cf}{10 \cdot \cit}
\end{align*}
Hence, by selecting $\cit$ and $\cf$ so that $\frac{64 \cdot \cf}{10 \cdot \cit} > 1$, we arrive at contradiction.
\end{proof}
% \begin{aligned}
% \text{Step size} && \step &= \cstep \frac{1}{L (1 + \tsigma^2)} \\
% \text{Total number of iterations} && T &= \frac{64\fmax}{\step \alpha^2} \\
% \text{Iterations required for escaping} && \escIter &= \cit \frac {1} {\step \sqrtra} \\
% \text{Escaping radius} && \escRad &= \crad \sqrt {\frac {\alpha} \rho} \\
% \text{Objective decrease after successful escaping} && \df &= \cf \sqrtfacr
% % \text{Noise standard deviation} && r &= \cnoise \frac \alpha {\sqrt{\gradLip \step}}
% \end{aligned}\label{eq:params}
% \begin{theorem}[Large-gradient case]
% If $\|\|$
% \end{theorem}
\begin{theorem}
\label{thm:combine}
When $\Delta^2 \ge \sigma^2 / B$, for $T = 64 \fmax / (\step \alpha^2)$, with probability at least $1 - \delta$, at least half of visited points are $\alpha$-SOSP.
\end{theorem}
\begin{proof}
We choose $\tdelta = \delta / T$.
Since dependence on $1/\tdelta$ is poly-logarithmic, this results in poly-logarithmic dependence on $1/\delta$ and $T$.
From Lemma~\ref{lem:quarter_large_grad}, we know that the number of points with large gradient is at most $T/4$.
From Lemma~\ref{lem:quarter_saddle} we know that the number of points $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ is at most $T/4$.
Hence, at least half of the points are $\alpha$-SOSP.
\end{proof}
\section{Convergence Without Gradient Concentration Assumption}
\label{app:no_nSG}
In this section, we prove that DP-SGD finds a SOSP even without stochastic gradient concentration assumption (\Cref{ass:grad_concentration}).
We show that we recover the same convergence rate (w.r.t $\Delta$), although a substantially larger $\Delta$ might be required.
As shown in \Cref{thm:combine_main}, assuming that the bound on the norm of the gradients doesn't change, we find the $\alpha$-SOSP for the same value of $\alpha$.
Most of the proof remains the same, and in this section we focus on the required changes.
\begin{lemma}[Descent Lemma]
\label{lem:descent_no_nSG}
Assume that $\step < 1/L$, for any $t_0$ and $t$.
When $d \Delta^2 > \sigma^2 / (\tdelta B)$, with probability at least $1 - \tdelta$, for some constant $c$:
\[
f(\x{t_0}) - f(\x{t_0 + t})
\ge \frac{\step}{16} \sum_{\tau=t_0}^{t_0 + t - 1} \SqrNorm{\nabla f(\x{\tau})} - c \step d \Delta^2 (\step L t + \logErr).
\]
\end{lemma}
\begin{proof}
W.l.o.g., we consider $t_0 = 0$.
Let $\sgdNoise{\tau} = \nabla f_{\batch{t}}(\x{\tau}) - \nabla f(\x{\tau})$ be the stochastic noise at iteration $\tau$.
By the folklore Descent Lemma,
\begin{align*}
f(\x{\tau}) - f(\x{\tau+1})
% \ge \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \nabla f_{\batch{t}}(\x{\tau})}
% - \frac{L}{2} \SqrNorm{\x{}}
&\ge \step \InnerProd{\nabla f(\x{\tau}), \nabla f_{\batch{t}}(\x{\tau})}
- \frac{L}{2} \SqrNorm{\x{\tau + 1} - \x{\tau}} \\
&= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}} \\
&\ge \step \SqrNorm{\nabla f(\x{\tau})} + \step \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{\step^2 L}{2} \pars{\frac{3}{2}\SqrNorm{\nabla f(\x{\tau})} + 3 \SqrNorm{\sgdNoise{\tau} + \ourNoise{\tau}}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
,
\end{align*}
where we used inequality $\SqrNorm{a + b} \le (1 + \nu) \SqrNorm{a} + (1 + 1/\nu) \SqrNorm{b}$ for $\nu = 1/2$.
By telescoping, we have:
\begin{align*}
f(\x{0}) - f(\x{t})
&\ge \frac{\step}{4} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})}
+ \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau}}
+ \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \ourNoise{\tau}}
- 3 \step^2 L \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau}}
- 3 \step^2 L \sum_{\tau=0}^{t-1} \SqrNorm{\ourNoise{\tau}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
\end{align*}
%
The terms containing $\ourNoise{\tau}$ can be bounded as in \Cref{lem:descent}, giving
\begin{align*}
f(\x{0}) - f(\x{t})
&\ge \frac{\step}{8} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})}
+ \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau}}
- 3 \step^2 L \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau}}
- c \step d \Delta^2 (\step L t + \logErr)
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
\end{align*}
with probability at least $1 - \tdelta$.
It remains to bound the terms containing $\sgdNoise{\tau}$.
\paragraph{Bounding $3 \step^2 L \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau}}$.}
Since $\ExpSqrNorm{\sgdNoise{\tau}} \le \sigma^2 / B$, by the Markov's inequality, with probability at least $1-\tdelta$ we have
\[
3 \step^2 L \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau}}
\le 3 \step^2 L t \sigma^2 / (\tdelta B)
\]
\paragraph{Bounding $\step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau}}$.}
We have $\Var{\InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau}} \mid \x{\tau}} \le \SqrNorm{\nabla f(\x{\tau})} \cdot \sigma^2 / B$.
By the Chebyshev's inequality, with probability at least $1 - \tdelta$, we have
\begin{align*}
\abs{\step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau}}}
&\le \step \sqrt{\sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})}} \sigma / \sqrt{\tdelta B} \\
&\le \frac{\step}{\nu} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})} + \step \nu \sigma^2 / (\tdelta B)
\end{align*}
for any $\nu$.
Taking $\nu = 16$ and using $d \Delta^2 / (\tdelta B)$ finishes the proof.
% \[
% \abs{\step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau}}}
% \le \frac{\step}{16} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})} + 4 \step \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau}}
% \]
% Using a concentration inequality for nSG variables, with probability at least $1 - \tdelta$ for some constant $c$ we have
% \[
% \abs{\step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}}
% \le \frac{\step}{8} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})} + c \step \fullVar^2 \logErr
% \]
% For the last term, with probability at least $1 - \tdelta$ we have
% \[
% \frac{3 \step^2 L}{2} \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau} + \ourNoise{\tau}}
% \le c \step^2 L \fullVar^2 (t + \logErr).
% \]
% Combining the above inequalities and using $\step \le 1/L$ finishes the proof.
% % \todo{TODO}
\end{proof}
\begin{lemma}
\label{lem:error_terms_no_nSG}
Let $\mathcal{E}_t$ be the event ``$\|\x{0} - \x{\tau}\| < \escRad$ and $\|\x{0} - \tx{\tau}\| < \escRad$ for all $\tau < t$''.
Then, when $\tdelta \Delta^2 \ge \sigma^2 / B$, for some constant $c$ we have
% \todo[inline]{Fix log error}
\[
\P{\mathcal{E}_\escIter
\implies \max\nolimits_{\tau=0}^{\escIter - 1} \norm{\sgdErr{\tau} + \quadErr{\tau}}
\le
c \beta_\tau \step \Delta \sqrt{\logErr}
} \ge 1 - \escIter \tdelta
\]
\end{lemma}
\begin{proof}
As in \Cref{lem:error_terms}, the proof is by induction on $t$, and we bound $\dif{t}$ and $\quadErr{t}$ as in \Cref{lem:error_terms}.
It remains to bound $\sgdErr{t}$.
% To bound $\sgdErr{t}$, we use that $\difsgdNoise{t}$ is $\mathrm{nSG}(\sigma)$:
% \begin{align*}
% \|\difsgdNoise{t}\|
% = \|\nabla f_{\batch{t}}(\x{t}) - \nabla f_{\batch{t}}(\tx{t})\|
% \le L \|\difx{t}\|
% \end{align*}
%
By \Cref{ass:grad_concentration}, each $\sgdErr{t}$ has variance $\sigma^2 / B$.
Hence, by the Chebyshev's inequality, with probability at least $1 - \tdelta$ we have
\begin{align*}
\norm{\sgdErr{t}}
&= \norm{\step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difsgdNoise{\tau}} \\
&\le \step \sqrt{\sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{2(t - 1 - \tau)} \frac{\sigma^2}{B \tdelta}} \\
&\le \step \beta_t \cdot \Delta,
\end{align*}
where the last inequality follows from assumption $\Delta^2 \ge \sigma^2 / {B \tdelta}$.
\end{proof}
% \begin{lemma}[Variation of Proposition 4.3 from \citet{DBLP:conf/iclr/FangLF023}]
% \label{lem:grad_bound_no_nSG}
% Under Assumptions~\ref{ass:main} and~\ref{ass:grad_concentration}, when $\eta \le \min(1/L, 1 / (\sigma \sqrt{T}))$ and $d \Delta^2 \ge \sigma^2 / (\tdelta B)$, with probability at least $1 - \delta$, for some absolute constant $c$:
% \begin{align*}
% \max_{t = 0, \ldots, T-1} \max_{i \in \batch{t}} \|\nabla f_i(\x{t})\|
% \le c \pars{
% \sqrt{L \fmax \log \nicefrac{1}{\delta}}
% + \sqrt{B T \sigma^2 / \tdelta}
% }
% \end{align*}
% \end{lemma}
% \todo{TODO}
% \begin{proof}
% Since $f$ is $L$-smooth, we have $\SqrNorm{\nabla f(\vx)} \le L (f(\vx) - f(\vx^*))$, where $\vx^*$ is the minimizer of $f$.
% By the Checyshev's inequality and the union bound, with probability $\tdelta$ we have
% \[
% \max_{t = 0, \ldots, T-1} \max_{i \in \batch{t}} \SqrNorm{\nabla f_i(\x{t}) - \nabla f(\x{t})}
% < B T \sigma^2 / \tdelta
% \]
% and hence
% \[
% \max_{t = 0, \ldots, T-1} \max_{i \in \batch{t}} \norm{\nabla f_i(\x{t}) - \nabla f(\x{t})}
% < \sqrt{B T \sigma^2 / \tdelta} + \max_{t = 0, \ldots, T-1} \sqrt{L (f(\x{t}) - f(\vx^*))}
% \]
% Therefore, it suffices to bound $\max_{t = 0, \ldots, T-1} f(\x{t}) - f(\vx^*)$.
% Note that we can assume that $\fmax \ge \df$, since otherwise $\x{0}$ is an $\alpha$-SOSP.
% Recall that, by \Cref{lem:bucket_change}, for any $\delta'$, we have
% \begin{align*}
% \P{f(\x{0}) - f(\x{\escIter}) < -\df / 100} &< \escIter \delta' \\
% \P{f(\x{0}) - f(\x{\escIter}) > \df} &> 1/3 - \escIter \delta'
% \end{align*}
% The first event never happens with probability at least $1 - T \delta'$.
% By the Azuma's inequality, for all $t$ we have $f(\x{t}) < f(\x{0}) + c \fmax \sqrt{\logErr}$ with probability at least $1 - \tdelta$ for some constant $c$.
% \end{proof}
\section{Convergence Proof}
\begin{assumption}
\label{ass:main_new}
The objective function $f$ and the component functions $f_i$ satisfy the following conditions.
\begin{enumerate}
{\color{green}\item (Boundedness)
There exists $\fmax$ such that $\fmax > f(\x{0}) - f(\vx^*)$.}
{\color{green}\item (Smoothness) $f$ is $L$-smooth, i.e. $\norm{\nabla f(\vx) - \nabla f(\vy)} \le L \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.\todo{Check that we cak reuse the gradient norm bound}}
{\color{green}\item (Lipschitz Hessian) The objective function $f$ has a $\rho$-Lipschitz Hessian, i.e. $\norm{\nabla^2 f(\vx) - \nabla^2 f(\vy)} \le \rho \norm{\vx - \vy}$ for all $\vx, \vy \in \R^d$.}
{\color{red}\item (Relaxed Stochastic variance) The stochastic variance is bounded as follows: $\avg_{i=1}^N \norm{\nabla f_i(\vx) - \nabla f(\vx)}^2 \le \sigma_1^2 \SqrNorm{\nabla f(\vx)} + \sigma_2^2$.}
\end{enumerate}
\end{assumption}
\todo{Rewrite}
\begin{assumption}
{\color{red}No nSG assumption.}
\end{assumption}
\begin{assumption}
{\color{red}Clipping. (Not doable quickly)}
\end{assumption}
\label{app:convergence_proof}
\todo{Mention that mostly follows Jin et al.}
\begin{definition}
\label{def:params}
Let $\fullVar = \nicefrac{\sigma^2}{B} + d \Delta^2$ be total noise variance.
Our choice of parameters is the following ($\cstep, \cit, \crad, \cf$ hide polylogarithmic dependence on all parameters):
\begin{equation*}
\begin{aligned}
\text{Step size} && \step &= \cstep \min\pars{\frac{\alpha^2}{L (1 + \tsigma^2)}, \frac{\fmax}{L \tsigma^2}} \\
\text{Total number of iterations} && T &= \frac{64\fmax}{\step \alpha^2} \\
\text{Iterations required for escaping} && \escIter &= \cit \frac {1} {\step \sqrtra} \\
\text{Escaping radius} && \escRad &= \crad \sqrt {\alpha / \rho} \\
\text{Objective decrease after successful escaping} && \df &= \cf \sqrtfacr
% \text{Noise standard deviation} && r &= \cnoise \frac \alpha {\sqrt{\gradLip \step}}
\end{aligned}
% \label{eq:params}
\end{equation*}
\end{definition}
Intuitively, the parameters are chosen as follows.
Assume that the current point is $\vzero$ and the function is quadratic, i.e. $f(\vx) = \vx^T H \vx$.
If $\vx$ is a saddle point, then we know that the smallest eigenvalue of $H$ is $-\eigen < -\sqrtra$ with the corresponding eigenvector $\vv_1$.
Then, the largest function decrease would be in the direction of $\vv_1$, and this direction can be found using power method for the appropriate matrix.
But, if we look at the gradient descent update:
\[
\x{t+1}
= \x{t} - \step \nabla f(\x{t})
= \x{t} - \step 2 H \x{t}
= (\id - \step 2 H) \x{t},
\]
we see that for quadratic functions, gradient descent is exactly the power method for the matrix $\id - \step 2H$, which has the same eigenvectors as $H$. The eigenvalue $-\eigen$ of $H$ corresponds to the eigenvalue $1 + 2 \step \eigen$ of $1 - \step 2 H$, which is the largest (by absolute value) eigenvalue when $\step \le 1/L$.
Note that power method requires some initial randomness, which explains while randomness is required to escape from saddle points.
The above intuition motivates the choice of parameters.
\begin{itemize}
\item
\textbf{Escaping radius $\escRad$}. Since in general the function is not necessarily quadratic, we want to guarantee that the Hessian doesn't change excessively.
More specifically, we want the smallest eigenvalue to be at most $-\eigen / 2$ for the entire escaping procedure.
By the Hessian-Lipschitz property $\|\nabla^2 f(\vx) - \nabla^2 f(\vy)\| \le \rho \|\vx - \vy\|$, if we work in the radius $\escRad = \eigen /(2\rho)$, then for any point within the radius $\escRad$ of the initial point, the smallest eigenvalue is at most $-\eigen + \eigen / 2 = -\eigen / 2$.
When $\eigen > \sqrtra$, we can lower-bound $\escRad$ as $\sqrtra / (2\rho) = \sqrt{\alpha / (4\rho)}$.
\item
\textbf{Number of iterations required for escaping $\escIter$}.
After $t$ iterations, the length of $\x{t}$ along the $\vv_1$ direction is $(1 + \step \eigen)^t \approx e^{t / (\step \eigen)}$.
Hence, we reach the boundary of the ball of radius $\escRad$ after $(t \log \escRad) / (\step \eigen)$ iterations.
\item
\textbf{Function change after escaping $\df$}.
Since we are working inside the ball of size $\escRad$, the best objective improvement $\df$ we can aim for is achievable by moving by $\escRad \vv_1$.
This improves the objective by approximately $\escRad^2 \eigen$, which, by the choice of the parameters can be lower-bounded as $\alpha / \rho \cdot \sqrtra = \sqrt{\alpha^3 / \rho}$.
\item
\textbf{Total number of iterations $T$}. Every $\escIter$ iterations we improve objective by $\df$.
Since the total objective improvement cannot exceed $\fmax$, the maximum number of iterations is
\[
\frac{\fmax \escIter}{\df}
\le \frac{\fmax}{(\step \sqrtra)\sqrt{\alpha^3 / \rho}}
= \frac{\fmax}{\step \alpha^2}
\]
\end{itemize}
\begin{lemma}[Descent Lemma]
\label{lem:descent}
Assume that $\step < 1/L$, for any $t_0$ and $t$.
Then, with probability at least $1 - \tdelta$, for some constant $c$:
\[
f(\x{t_0}) - f(\x{t_0 + t})
\ge \frac{\step}{8} \sum_{\tau=t_0}^{t_0 + t - 1} \norm{\nabla f(\x{\tau})}^2 - c \step \tsigma^2 (\step L t + \logErr)
.
\]
\end{lemma}
\begin{proof}
W.l.o.g., we consider $t_0 = 0$.
Let $\sgdNoise{\tau} = \nabla f_{\batch{t}}(\x{\tau}) - \nabla f(\x{\tau})$ be the stochastic noise at iteration $\tau$.
By the folklore Descent Lemma,
\begin{align*}
f(\x{\tau}) - f(\x{\tau+1})
% \ge \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \nabla f_{\batch{t}}(\x{\tau})}
% - \frac{L}{2} \SqrNorm{\x{}}
&\ge \step \InnerProd{\nabla f(\x{\tau}), \nabla f_{\batch{t}}(\x{\tau})}
- \frac{L}{2} \SqrNorm{\x{\tau + 1} - \x{\tau}} \\
&= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}} \\
&\ge \step \SqrNorm{\nabla f(\x{\tau})} + \step \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{\step^2 L}{2} \pars{\frac{3}{2}\SqrNorm{\nabla f(\x{\tau})} + 3 \SqrNorm{\sgdNoise{\tau} + \ourNoise{\tau}}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
,
\end{align*}
where we used inequality $\|a + b\|^2 \le (1 + \nu) \|a\|^2 + (1 + 1/\nu)\|b\|^2$ for $\nu = 1/2$.
By telescoping, we have:
\begin{align*}
f(\x{0}) - f(\x{t})
&\ge \frac{\step}{4} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})} + \step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}
- \frac{3 \step^2 L}{2} \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau} + \ourNoise{\tau}}
% &= \step \InnerProd{\nabla f(\x{\tau}), \nabla f(\x{\tau}) + \sgdNoise{\tau}}
% - \frac{L\step^2}{2} \SqrNorm{\nabla f(\x{\tau}) + \sgdNoise{\tau}} \\
\end{align*}
%
Using a concentration inequality for nSG variables, with probability at least $1 - \tdelta$ for some constant $c$ we have
\[
\abs{\step \sum_{\tau=0}^{t-1} \InnerProd{\nabla f(\x{\tau}), \sgdNoise{\tau} + \ourNoise{\tau}}}
\le \frac{\step}{8} \sum_{\tau=0}^{t-1} \SqrNorm{\nabla f(\x{\tau})} + c \step \fullVar^2 \logErr
\]
\todo{Give reference}
\todo{Avoid norm-subGaussian}
For the last term, with probability at least $1 - \tdelta$ we have
\[
\frac{3 \step^2 L}{2} \sum_{\tau=0}^{t-1} \SqrNorm{\sgdNoise{\tau}}
\le c \step^2 L \fullVar^2 (t + \logErr).
\]
Combining the above inequalities and using $\step \le 1/L$ finishes the proof.
% \todo{TODO}
\end{proof}
\subsection{Large Gradient Case}
The first application of the Descent Lemma is to bound the number of iterations when the gradient is large.
\begin{lemma}
\label{lem:quarter_large_grad}
When $T \ge \frac{64 \fmax}{\step \alpha^2}$ and other parameters are as in \Cref{def:params}, the number of iterations $t$ with $\|\nabla f(\x{t})\| \ge \alpha$ is at most $\nicefrac T4$.
\end{lemma}
\begin{proof}
For the sake of contradiction, assume that the number of points $\x{t}$ with $\|\nabla f(\x{t})\| \ge \alpha$ is greater than $\nicefrac T4$.
Then, by the Descent Lemma we have
\begin{align*}
f(\x{0}) - f(\x{T})
&\ge \frac{\step}{8} \sum_{\tau=0}^{T - 1} \norm{\nabla f(\x{\tau})}^2 - c \step \tsigma^2 (\step L T + \logErr) \\
&\ge \frac{\step T \alpha^2}{32} - c \step T \cdot \tsigma^2 \pars{\step L + \frac{\logErr}{T}}
\end{align*}
We will show that the right-hand side is at least $\fmax$, leading to contradiction.
For that, it suffices to guarantee that $\frac{\step T \alpha^2}{32} \ge 2\fmax$ and $c \step T \cdot \tsigma^2 \pars{\step L + \frac{\logErr}{T}} \le \fmax$.
The former condition is satisfied by our choice of $T$, and substituting the bound on $\fmax$ into the latter condition shows that it suffices to prove
\[
64 c \tsigma^2 \pars{\step L + \frac{\step \alpha^2 \logErr}{64 \fmax}} \le \alpha^2
\]
It suffices to bound each term with $\alpha^2 / 2$, which is satisfied for
\begin{align*}
\step &\le \frac{\alpha^2}{128 \cdot c \tsigma^2 L} \\
\step &\le \frac{\fmax\logErr}{2 \cdot c \tsigma^2 L}
\end{align*}
Both of these conditions are satisfied by our choice of $\step$, with the appropriate choice of $\cstep$.
\end{proof}
\subsection{Escaping From Saddle Points}
The Descent Lemma also implies the following result, which upper-bounds the distance change in terms of the objective changes.
Its \Cref{cor:i_or_l} states that, if after a bounded number of iterations the point moves far from the original point, then the objective value sufficiently decreases.
%
\begin{lemma}[Improve or localize]
\label{lem:improve_or_localize}
For any $t_0$ and $t$, with probability at least $1 - \tdelta$, for some constant $c$:
\begin{align*}
\SqrNorm{\x{t_0 + t} - \x{t_0}}
\le c \step t \pars{(f(\x{t_0}) - f(\x{t_0 + t}) + \step \tsigma^2 (\step L t + \logErr)}
\end{align*}
\end{lemma}
%
\begin{proof}
W.l.o.g. we consider the case when $t_0 = 0$.
% By the Descent Lemma,
% \[
% f(\x{0}) - f(\x{t})
% \ge \frac{\step}{8} \sum_{\tau=0}^{t - 1} \norm{\nabla f(\x{\tau})}^2 - c \step \tsigma^2 (\step L t + \logErr)
% \]
By the definition of $\x{\tau + 1}$, we have
\begin{align*}
\x{\tau} - \x{\tau + 1}
= \step (\nabla f_{\batch{\tau}}(\x{\tau}) + \ourNoise{\tau})
= \step (\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau}),
\end{align*}
where $\sgdNoise{\tau} = \nabla f_{\batch{\tau}}(\x{\tau}) - \nabla f(\x{\tau})$ is stochastic gradient noise.
By telescoping, we have:
\begin{align*}
\|\x{0} - \x{t}\|
&= \step \norm{\sum_{\tau = 0}^{t - 1}(\nabla f(\x{\tau}) + \sgdNoise{\tau} + \ourNoise{\tau})}
\end{align*}
By Cauchy-Schwarz Inequality, we have:
\begin{align*}
\|\x{0} - \x{t}\|^2
&\le 2 \step^2 \pars{\norm{\sum_{\tau = 0}^{t - 1}\nabla f(\x{\tau})}^2 + \norm{\sum_{\tau = 0}^{t - 1} (\sgdNoise{\tau} + \ourNoise{\tau})}^2} \\
&\le 2 \step^2 \pars{t \sum_{\tau = 0}^{t - 1}\norm{\nabla f(\x{\tau})}^2 + \norm{\sum_{\tau = 0}^{t - 1} (\sgdNoise{\tau} + \ourNoise{\tau})}^2}
\end{align*}
\todo{Elaborate and avoid nSG}
Since $\sgdNoise{\tau} + \ourNoise{\tau}$ is $\mathrm{nSG}(\fullVar^2)$, by applying concentration inequality for $\mathrm{nSG}$ variables, we have with probability at least $1 - \tdelta$:
\begin{align*}
\|\x{0} - \x{t}\|^2
&\le 2 \step^2 \pars{t \sum_{\tau = 0}^{t - 1}\norm{\nabla f(\x{\tau})}^2 + c t \fullVar^2 \logErr}
\end{align*}
Finally, applying the Descent Lemma, we get
\begin{align*}
\|\x{0} - \x{t}\|^2
&\le \step c t \pars{f(\x{0}) - f(\x{t}) + \step \fullVar^2 (\step L t + \logErr)}
\end{align*}
for some constant $c$.
\end{proof}
\begin{corollary}
\label{cor:i_or_l}
For the parameters chosen as in \Cref{def:params}, if for some $t \le \escIter$ we have $\SqrNorm{\x{t_0 + \tau} - \x{t_0}} \ge \escRad$, then $f(\x{t_0}) - f(\x{t_0 + \tau}) \ge \df$ with probability at least $1 - \tdelta$.
\end{corollary}
%
\begin{proof}
From \Cref{lem:improve_or_localize}, we know that for some $c$:
\begin{align*}
f(\x{t_0}) - f(\x{t_0 + t})
&\ge \frac{\escRad^2}{c \step \escIter} - \step \tsigma^2 (\step L \escIter + \logErr) \\
&\ge \frac{\crad^2}{c \cit} \sqrtfacr - \step \tsigma^2 \pars{\frac{\cit L}{\sqrtra} + \logErr}
% &\ge \frac{\SqrNorm{\x{t_0 + t} - \x{t_0}}}{c \step \escIter} - \step \tsigma^2 (\step L \escIter + \log 1/\fullVar)
\end{align*}
We want to guarantee that the right-hand side is at most $\df = \cf \sqrtfacr$.
It suffices to guarantee that
\begin{align*}
\crad^2 / (c \cit) \ge 2 \cf \qquad \text{and} \qquad \step \tsigma^2 (\cit L / \sqrtra + \logErr) \le \cf \sqrtfacr.
\end{align*}
The former holds for the appropriate choice of $\crad$, $\cit$, and $\cf$, and the latter holds by choosing
\[
\step \le (\alpha^2 / L \tsigma^2) \cdot \cf / \pars{\cit + \logErr},
\]
where we assumed $L \ge \sqrtra$, since otherwise any $\alpha$-FOSP is an $\alpha$-SOSP because $L$ is an upper bound on the absolute values of the Hessian.
\end{proof}
\begin{definition}[Coupling sequences]
\label{def:coupling_sequence}
Let $\vv_1$ be the eigenvector corresponding to the smallest eigenvalue of $\nabla^2 f(\x{t_0})$.
The coupling sequences starting from $\x{t_0}$ are defined as follows:
\begin{equation}
\begin{aligned}
\ourNoise{t_0+t} &\sim \mathcal{N}(0, \fullVar^2 \id) \\
\tourNoise{t_0+t} &= \ourNoise{t_0+t} - 2 \InnerProd{ \vv_1, \ourNoise{t_0+t} } \vv_1 \\
% \batch'_{t_0+t} &= \batch_{t_0+t}, \\
\x{t_0 + t+1} &= \x{t_0+t} - \step (\nabla f_{\batch{t}}(\x{t_0 + t}) + \ourNoise{t_0 + t}) \\
\tx{t_0 + t+1} &= \tx{t_0+t} - \step (\nabla f_{\batch{t}}(\tx{t_0 + t}) + \tourNoise{t_0 + t}) \\
\sgdNoise{t_0 + t} &= \nabla f_{\batch{t}}(\x{t_0 + t}) - \nabla f(\x{t_0 + t}) \\
\tsgdNoise{t_0 + t} &= \nabla f_{\batch{t}}(\tx{t_0 + t}) - \nabla f(\tx{t_0 + t}) \\
\end{aligned}\nonumber
\end{equation}
%
Note that:
\begin{itemize}
\item the artificial noise $\tourNoise{t_0 + t}$ has the opposite sign in direction $\vv_1$ compared with $\ourNoise{t_0 + t}$,
\item the stochastic randomness ${\batch{t}}$ is the same for both sequences,\todo{Might be not required, but anyways makes things simpler to define}
\item $\{\x{t_0 + t}\}_t$ and $\{\tx{t_0 + t}\}_t$ are sampled from the same distribution.
\end{itemize}
%
We next define the difference between the coupling sequences:
\begin{equation}
\begin{aligned}
\difourNoise{t} &= \xi_{t_0+t} - \xi'_{t_0+t}
= 2 \InnerProd{ \vv_1, \xi_{t_0+t} } \vv_1 \\
% \batch'_{t_0+t} &= \batch_{t_0+t}, \\
\difx{t_0 + t+1} &= \x{t_0+t} - \tx{t_0+t} \\
\difsgdNoise{t_0 + t} &= \sgdNoise{t_0 + t} - \tsgdNoise{t_0 + t}
\end{aligned}\nonumber
\end{equation}
\end{definition}
% TODO (the rest is standard)
% SGD with variance $\sigma^2$ and batch size $B$, then the number of iterations to find an $\alpha$-FOSP ($\|\nabla f(\vx)\| < \alpha$) is $\frac{\sigma^2}{B \alpha^4}$.
From now on, we assume that we want to escape from point $\x{t_0}$ such that $\lmin(\nabla^2 f(\x{t_0})) < -\sqrtra$.
W.l.o.g. we assume $t_0 = 0$.
Let $H = \nabla^2 f(\x{0})$ be the Hessian of $f$ at point $\x{0}$, let $-\eigen = \lmin(H)$ be the smallest eigenvalue of $H$, and $\vv_1$ be the eigenvector corresponding to $-\eigen$.
\begin{lemma}
For every $t$,
\[
\difx{t} = -\dif{t} - \sgdErr{t} -\quadErr{t},
\]
where
\begin{align*}
\dif{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difourNoise{\tau} \\
\sgdErr{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difsgdNoise{\tau} \\
\quadErr{t} &= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \hesInt{\tau} \difx{\tau},
\quad \text{ where } \quad \hesInt{\tau} = \int_0^1 (\nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) - H)\ d\psi
\end{align*}
\end{lemma}
\begin{proof}
Proof by induction.
\begin{align*}
\difx{t + 1}
&= \difx{t} - \step (\difsgdNoise{t} + \difsgdNoise{t}) - \step (\nabla f(\x{t}) - \nabla f(\tx{t})) \\
&= (1 - \step H) \difx{t} - \step (\difsgdNoise{t} + \difsgdNoise{t}) - \step (\nabla f(\x{t}) - \nabla f(\tx{t}) - H \difx{t})
\end{align*}
Using that $\nabla f(\x{t}) - \nabla f(\tx{t}) = \int_0^1 \nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) \difx{\tau}\ d\psi$, by telescoping we complete the proof.
\end{proof}
%
In the following statements, the geometric series $\alpha_t^2 = \sum_{\tau = 0}^{t-1} (1 + \step \eigen)^{2 \tau}$ occurs frequently.
Intuitively, each iteration increases the distance approximately by a factor of $(1 + \step \eigen)$~-- the largest eigenvalue of matrix $I - \step H$.
The noise added at iteration $\tau$ gets amplified by $(1 + \step \eigen)^{t - \tau}$ by iteration $t$, and, if every iteration an independent noise of variance $\Delta^2$ is added, $\Delta^2 \alpha_t^2$ is the total variance of such a noise accumulated over the iterations.
%
\begin{lemma}
\label{lem:alpha_beta}
Let $\alpha_t = \sqrt{\sum_{\tau = 0}^{t-1} (1 + \step \eigen)^{2 \tau}}$ and $\beta_t = (1 + \step \eigen)^t / \sqrt{\step \eigen}$.
If $\step \eigen < 1$, then $\alpha_t \le \beta_t$ for any $t$ and $\alpha_t \ge \beta_t / 3$ for $t \ge 2 / (\step \eigen)$.
\end{lemma}
\begin{proof}
As a sum of geometric progression, we have
\[
\alpha_t^2
= \frac{(1 + \step \eigen)^{2t} - 1}{(1 + \step \eigen)^2 - 1}
= \frac{(1 + \step \eigen)^{2t} - 1}{\step \eigen (2 + \step \eigen)},
\]
and $\alpha_t^2 \le \beta_t^2$ follows immediately.
On the other hand, for $t \ge 1 / (\step \eigen)$ we have $(1 + \step \eigen)^{2t} \ge 2$, and hence $(1 + \step \eigen)^{2t} - 1 \ge (1 + \step \eigen)^{2t} / 2$.
Finally, by our choice of the step size, we have $\step \eigen \le \step L \le 1$, and hence $2 + \step \eigen \le 3$.
This implies $\alpha_t^2 \ge \beta_t^2 / 6$, finishing the proof.
\end{proof}
\begin{lemma}
\label{lem:good_term}
Let $\beta_t = (1 + \step \eigen)^t / \sqrt{\step \eigen}$.
Then for any $t$ and $\tdelta$, for some constant $c$:
\begin{align*}
\P{\|\dif{t}\| \ge c \beta_t \step \Delta \sqrt{\logErr}} &\le \tdelta \\
\P{\|\dif{t}\| \le \frac{\beta_t \step \Delta}{10}} &\le \frac 13
\end{align*}
\end{lemma}
\todo{Check $\sqrt{\logErr}$}
\begin{proof}
Since $\difourNoise{\tau} = 2\InnerProd{\ourNoise{\tau}, \vv_1} \vv_1$ and $\vv_1$ is an eigenvalue of $H$ corresponding to eigenvalue $-\eigen$, we have
\[
\dif{t}
= \step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difourNoise{\tau}
= \pars{\step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{t - 1 - \tau} \InnerProd{\ourNoise{\tau}, \vv_1}} \vv_1
\]
Hence, $\dif{t}$ is a random variable which is parallel to $\vv_1$.
% Hence, $\dif{t}$ is a Gaussian random variable along direction $\vv_1$ with variance $\step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{t - 1 - \tau} \InnerProd{\ourNoise{\tau}, \vv_1}$
Since each $\ourNoise{t}$ is a Gaussian random variable with the covariance matrix $\Delta^2 \id$, each $\InnerProd{\ourNoise{t}, \vv_1}$ a Gaussian random variable with variance $\Delta^2$.
Hence, $\dif{t}$ is also a Gaussian random variable, and
\[
\Var{\|\dif{t}\|}
= \step \sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{\tau} \Delta^2
= \Delta^2 \alpha_t^2
\]
The upper bound on $\|\dif{t}\|$ follows from \Cref{lem:alpha_beta} and standard concentration inequality for Guassians.
The lower bound follows from inequality $\P{|X| \le \sigma a} \le a$ for $X \sim \mathcal{N}(0, \sigma^2)$ and any $a$.
\end{proof}
\todo{Be more careful with $\tdelta$}
\todo{Replace $\fullVar$ with $\Delta$ where required}
\todo{Propagate assumption $\Delta^2 \ge \sigma^2 / B$}
\begin{lemma}
\label{lem:error_terms}
Let $\mathcal{E}_t$ be the event ``$\|\x{0} - \x{\tau}\| < \escRad$ and $\|\x{0} - \tx{\tau}\| < \escRad$ for all $\tau < t$''.
Then, when $\Delta^2 \ge \sigma^2 / B$, for some constant $c$ we have
% \todo[inline]{Fix log error}
\[
\P{\mathcal{E}_\escIter
\implies \max\nolimits_{\tau=0}^{\escIter - 1} \norm{\sgdErr{\tau} + \quadErr{\tau}}
\le
c \beta_\tau \step \Delta \sqrt{\logErr}
} \ge 1 - \escIter \tdelta
\]
\end{lemma}
\begin{proof}
By induction on $t$, we will prove that $\P{\mathcal{E}_t \implies \|\difx{t}\| \le c \step \Delta \beta_t} \le 1 - t\fullVar$ and
\[
\P{\mathcal{E}_t
\implies \max\nolimits_{\tau=0}^{t - 1} \norm{\sgdErr{\tau} + \quadErr{\tau}}
\le c \beta_\tau \step \Delta \sqrt{\logErr}
} \ge 1 - t \tdelta.
\]
Base case is $t=0$, in which case $\difx{0} = \vzero$.
For the induction step, since $\difx{t} = -\dif{t} - \sgdErr{t} -\quadErr{t}$, we bound each term separately.
\paragraph{Bounding $\dif{t}$}
By \Cref{lem:good_term}, we have $\P{\|\dif{t}\| \ge c \beta_t \step \Delta \sqrt{\logErr}} \le \tdelta$.
\paragraph{Bounding $\quadErr{t}$}
Assuming $\mathcal{E}_t$, by the Hessian-Lipschitz property we have
\begin{align*}
\|\hesInt{\tau} \difx{\tau}\|
&= \norm{\int_0^1 (\nabla^2 f(\psi \x{\tau} + (1 - \psi) \tx{\tau}) - H)\ d\psi \cdot \difx{\tau}} \\
&\le 2 \rho \escRad \|\difx{\tau}\| \\
&\le 2 \rho \escRad \cdot c \beta_\tau \step \Delta \sqrt{\logErr},
\end{align*}
%
and we bound $\quadErr{t}$ as
%
\begin{align*}
\|\quadErr{t}\|
&= \norm{\step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \hesInt{\tau} \difx{\tau}} \\
&\le \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{t - 1 - \tau} 2 \rho \escRad \cdot c \beta_\tau \step \Delta \sqrt{\logErr} \\
% &\le \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{\tau} 2 \rho \escRad \cdot c \beta_t \step \fullVar^2 \sqrt{\logErr} \\
\end{align*}
Since $(1 + \step \gamma)^{t - 1 - \tau} \beta_\tau = \beta_{t-1}$,
% \[
% \sum_{\tau = 0}^{t - 1} (1 + \step \gamma)^{\tau}
% = \frac{(1 + \step \gamma)^t - 1}{\step \gamma}
% \le \frac{\beta_t}{\sqrt{\step \gamma}},
% \]
we can bound $\|\quadErr{t}\|$ as
\begin{align*}
\|\quadErr{t}\|
&\le 2 \rho \escRad \cdot c \escIter \beta_t \step \Delta \sqrt{\logErr} \\
&= 2 c \crad \cit \beta_t \Delta \sqrt{\logErr},
\end{align*}
where the last equality is due to our choice of parameters in \Cref{def:params}:
\[
\escRad \escIter
= \crad \sqrt {\alpha / \rho} \cdot \cit \frac {1} {\step \sqrtra}
= \frac{\crad \cit}{\rho \step}
\]
\paragraph{Bounding $\sgdErr{t}$}
% To bound $\sgdErr{t}$, we use that $\difsgdNoise{t}$ is $\mathrm{nSG}(\sigma)$:
% \begin{align*}
% \|\difsgdNoise{t}\|
% = \|\nabla f_{\batch{t}}(\x{t}) - \nabla f_{\batch{t}}(\tx{t})\|
% \le L \|\difx{t}\|
% \end{align*}
%
By \Cref{ass:grad_concentration}, each $\sgdErr{t}$ is $\mathrm{nSG}(\sigma^2 / B)$.
Using the concentration inequality for nSG variables, with probability at least $1 - \tdelta$ we have
\begin{align*}
\|\sgdErr{t}\|
&= \norm{\step \sum_{\tau = 0}^{t - 1} (\id - \step H)^{t - 1 - \tau} \difsgdNoise{\tau}} \\
&\le \step \sqrt{\sum_{\tau = 0}^{t - 1} (1 + \step \eigen)^{2(t - 1 - \tau)} \frac{\sigma^2}{B}} \cdot \sqrt{\logErr} \\
&\le \step \sigma \beta_t \sqrt{\logErr} / \sqrt{B} \\
\end{align*}
Using assumption $\Delta^2 \ge \sigma^2 / B$, we get the desired bound.
\end{proof}
\begin{lemma}
\label{lem:bucket_change}
When $\Delta^2 \ge \sigma^2 / B$, with probability at least $1 - \tdelta$, we have
\[
f(\x{0}) - f(\x{\escIter}) > -\df / 100
\]
Moreover, with probability at least $1/3 - \tdelta$, we have:
\[
f(\x{0}) - f(\x{\escIter}) > \df
\]
\end{lemma}
\begin{proof}
The first claim follows from the descent lemma by our choice of parameters:
\[
f(\x{0}) - f(\x{\escIter})
> -c \step \fullVar^2 (\step L \escIter + \logErr)
> -\df / 10
\]
From Corollary~\ref{cor:i_or_l}, we know that with probability at least $1-\tdelta$, if $\|\x{t} - \x{0}\| > \escRad$ for some $t \le \escIter$, then $f(\x{0}) - f(\x{t}) > \df$ with probability at least $1 - \tdelta$.
Since the $\tx{t}$ has the same distribution as $\x{t}$, the same holds for $\|\tx{t} - \x{0}\|$.
Hence, if either $\|\x{t} - \x{0}\| > \escRad$ or $\|\tx{t} - \x{0}\| > \escRad$, then with probability $\frac{1}{2} - 2 \tdelta$ we have $f(\x{0}) - f(\x{t})$.
For the rest of the proof, we can assume that $\|\x{t} - \x{0}\| < \escRad$ and $\|\tx{t} - \x{0}\| < \escRad$ for all $t \le \escIter$.
By Lemmas~\ref{lem:good_term} and~\ref{lem:error_terms} we know that $\|\dif{\escIter}\| \ge \step \beta_t \Delta \sqrt{\logErr} / 10$ and $\|\sgdErr{t} + \quadErr{t}\| \le \step \beta_t \Delta \sqrt{\logErr} / 20$ with probability $1 - \tdelta$.
Hence,
\begin{align*}
\max(\|\x{\escIter} - \x{0}\|, \|\tx{\escIter} - \x{0}\|)
&\ge \frac{1}{2} \|\difx{\escIter}\| \\
&\ge \frac 12 \pars{\|\dif{\escIter}\| - \|\sgdErr{\escIter} + \quadErr{t}\|} \\
&\ge \frac{\step \beta_\escIter \Delta}{40} \sqrt{\logErr} \\
&= (1 + \step \eigen)^\escIter \frac{1}{40} \Delta \sqrt{\frac{\step}{\eigen}} \sqrt{\logErr} \\
&\ge \escRad,
\end{align*}
where the last inequality follows by the appropriate choice of logarithmic term in $\cit$ (note that $(1 + \step \eigen)^\escIter$ dominates other terms).
This contradicts the assumption $\max(\|\x{\escIter} - \x{0}\|, \|\tx{\escIter} - \x{0}\|) \le \escRad$.
\end{proof}
\begin{lemma}
\label{lem:quarter_saddle}
When $\Delta^2 \ge \sigma^2 / B$, for $T = 64 \fmax / (\step \alpha^2)$, with $\step$ chosen as in \Cref{def:params}, with probability at least $1 - \tdelta$ we have $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ for at most $T/4$ points $\x{t}$.
\end{lemma}
\begin{proof}
\todo{Error probability}
Proof by contradiction.
Let's split all iterations $0,...,T-1$ into buckets of size $\escIter$.
There are $T/\escIter = 64\fmax \sqrtra / \cit$ buckets, and at least quarter of the buckets have a point $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ in it.
Note that within each bucket, by Lemma~\ref{lem:bucket_change}, the objective increases by at most $\df / 100$.
Out of such buckets, at least half of them are non-adjacent, and we consider points $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ in these buckets.
With probability at least $1/3 - \tdelta$, we have $f(\x{t}) - f(\x{t + \escIter}) > \df$ for each such $\x{t}$.
% \todo[inline]{Properly handle logs}
Hence,
\begin{align*}
\fmax
&\ge \frac{T \df}{10\cdot \escIter} \\
&= \frac{1}{10} \cdot \frac{64\fmax}{\step \alpha^2} \cdot \cf \sqrtfacr \cdot \frac{\step \sqrtra}{\cit} \\
&= \fmax \frac{64 \cdot \cf}{10 \cdot \cit}
\end{align*}
Hence, by selecting $\cit$ and $\cf$ so that $\frac{64 \cdot \cf}{10 \cdot \cit} > 1$, we arrive at contradiction.
\end{proof}
% \begin{aligned}
% \text{Step size} && \step &= \cstep \frac{1}{L (1 + \tsigma^2)} \\
% \text{Total number of iterations} && T &= \frac{64\fmax}{\step \alpha^2} \\
% \text{Iterations required for escaping} && \escIter &= \cit \frac {1} {\step \sqrtra} \\
% \text{Escaping radius} && \escRad &= \crad \sqrt {\frac {\alpha} \rho} \\
% \text{Objective decrease after successful escaping} && \df &= \cf \sqrtfacr
% % \text{Noise standard deviation} && r &= \cnoise \frac \alpha {\sqrt{\gradLip \step}}
% \end{aligned}\label{eq:params}
% \begin{theorem}[Large-gradient case]
% If $\|\|$
% \end{theorem}
\begin{theorem}
\label{thm:combine}
When $\Delta^2 \ge \sigma^2 / B$, for $T = 64 \fmax / (\step \alpha^2)$, with probability at least $1 - \tdelta$, at least half of visited points are $\alpha$-SOSP.
\end{theorem}
\begin{proof}
From Lemma~\ref{lem:quarter_large_grad}, we know that the number of points with large gradient is at most $T/4$.
From Lemma~\ref{lem:quarter_saddle} we know that the number of points $\x{t}$ with $\lmin(\nabla^2 f(\x{t})) < -\sqrtra$ is at most $T/4$.
Hence, at least half of the points are $\alpha$-SOSP.
\end{proof}
\begin{thebibliography}{77}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abadi et~al.(2016)Abadi, Chu, Goodfellow, McMahan, Mironov, Talwar, and Zhang]{abadi_DeepLearning_2016}
Martin Abadi, Andy Chu, Ian Goodfellow, H.~Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li~Zhang.
\newblock Deep {{Learning}} with {{Differential Privacy}}.
\newblock In \emph{Proceedings of the 2016 {{ACM SIGSAC Conference}} on {{Computer}} and {{Communications Security}}}, {{CCS}} '16, pages 308--318, {New York, NY, USA}, October 2016. {Association for Computing Machinery}.
\newblock ISBN 978-1-4503-4139-4.
\newblock \doi{10.1145/2976749.2978318}.
\bibitem[Agarwal et~al.(2018)Agarwal, Suresh, Yu, Kumar, and McMahan]{agarwal_CpSGDCommunicationefficient_2018}
Naman Agarwal, Ananda~Theertha Suresh, Felix Yu, Sanjiv Kumar, and H.~Brendan McMahan.
\newblock {{cpSGD}}: Communication-efficient and differentially-private distributed {{SGD}}.
\newblock In \emph{Proceedings of the 32nd {{International Conference}} on {{Neural Information Processing Systems}}}, {{NIPS}}'18, pages 7575--7586, {Red Hook, NY, USA}, December 2018. {Curran Associates Inc.}
\bibitem[{Allen-Zhu}(2018)]{allen-zhu_NatashaFaster_2018}
Zeyuan {Allen-Zhu}.
\newblock Natasha 2: {{Faster Non-Convex Optimization Than SGD}}.
\newblock In S.~Bengio, H.~Wallach, H.~Larochelle, K.~Grauman, N.~{Cesa-Bianchi}, and R.~Garnett, editors, \emph{Advances in {{Neural Information Processing Systems}}}, volume~31. {Curran Associates, Inc.}, 2018.
\bibitem[{Allen-Zhu} and Li(2018)]{allen-zhu_NEON2Finding_2018}
Zeyuan {Allen-Zhu} and Yuanzhi Li.
\newblock {{NEON2}}: {{Finding Local Minima}} via {{First-Order Oracles}}.
\newblock In S.~Bengio, H.~Wallach, H.~Larochelle, K.~Grauman, N.~{Cesa-Bianchi}, and R.~Garnett, editors, \emph{Advances in {{Neural Information Processing Systems}}}, volume~31. {Curran Associates, Inc.}, 2018.
\bibitem[Anandkumar and Ge(2016)]{DBLP:conf/colt/AnandkumarG16}
Animashree Anandkumar and Rong Ge.
\newblock Efficient approaches for escaping higher order saddle points in non-convex optimization.
\newblock In \emph{Proceedings of the 29th Conference on Learning Theory, {COLT} 2016, New York, USA, June 23-26, 2016}, pages 81--102, 2016.
\bibitem[Andr{\'e}s et~al.(2013)Andr{\'e}s, Bordenabe, Chatzikokolakis, and Palamidessi]{andres_GeoindistinguishabilityDifferential_2013}
Miguel~E. Andr{\'e}s, Nicol{\'a}s~E. Bordenabe, Konstantinos Chatzikokolakis, and Catuscia Palamidessi.
\newblock Geo-indistinguishability: Differential privacy for location-based systems.
\newblock In \emph{Proceedings of the 2013 {{ACM SIGSAC}} Conference on Computer \& Communications Security}, Ccs '13, pages 901--914, New York, NY, USA, 2013. Association for Computing Machinery.
\newblock ISBN 978-1-4503-2477-9.
\newblock \doi{10.1145/2508859.2516735}.
\bibitem[Arjevani et~al.(2023)Arjevani, Carmon, Duchi, Foster, Srebro, and Woodworth]{arjevani_LowerBounds_2023}
Yossi Arjevani, Yair Carmon, John~C. Duchi, Dylan~J. Foster, Nathan Srebro, and Blake Woodworth.
\newblock Lower bounds for non-convex stochastic optimization.
\newblock \emph{Mathematical Programming}, 199\penalty0 (1):\penalty0 165--214, May 2023.
\newblock ISSN 1436-4646.
\newblock \doi{10.1007/s10107-022-01822-7}.
\bibitem[Arora et~al.(2023)Arora, Bassily, Gonz{\'{a}}lez, Guzm{\'{a}}n, Menart, and Ullah]{DBLP:conf/icml/AroraBGGMU23}
Raman Arora, Raef Bassily, Tom{\'{a}}s Gonz{\'{a}}lez, Crist{\'{o}}bal Guzm{\'{a}}n, Michael Menart, and Enayat Ullah.
\newblock Faster rates of convergence to stationary points in differentially private optimization.
\newblock In \emph{International Conference on Machine Learning, {ICML} 2023, 23-29 July 2023, Honolulu, Hawaii, {USA}}, volume 202 of \emph{Proceedings of Machine Learning Research}, pages 1060--1092. {PMLR}, 2023.
\bibitem[Asi et~al.(2021)Asi, Duchi, Fallah, Javidbakht, and Talwar]{asi2021private}
Hilal Asi, John~C. Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar.
\newblock Private adaptive gradient methods for convex optimization.
\newblock In \emph{Proceedings of the 38th International Conference on Machine Learning (ICML)}, pages 383--392, Virtual Event, 2021.
\bibitem[Bassily et~al.(2014)Bassily, Smith, and Thakurta]{bassily_PrivateEmpirical_2014}
Raef Bassily, Adam Smith, and Abhradeep Thakurta.
\newblock Private {{Empirical Risk Minimization}}: {{Efficient Algorithms}} and {{Tight Error Bounds}}.
\newblock In \emph{2014 {{IEEE}} 55th {{Annual Symposium}} on {{Foundations}} of {{Computer Science}}}, pages 464--473, October 2014.
\newblock \doi{10.1109/FOCS.2014.56}.
\bibitem[Bassily et~al.(2019)Bassily, Feldman, Talwar, and Thakurta]{bassily2019private}
Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep~Guha Thakurta.
\newblock Private stochastic convex optimization with optimal rates.
\newblock In \emph{Advances in Neural Information Processing Systems (NeurIPS)}, pages 11279--11288, Vancouver, Canada, 2019.
\bibitem[Bertsekas(1997)]{bertsekas_NonlinearProgramming_1997}
Dimitri~P Bertsekas.
\newblock Nonlinear programming.
\newblock \emph{Journal of the Operational Research Society}, 48\penalty0 (3):\penalty0 334--334, 1997.
\bibitem[Bhojanapalli et~al.(2016)Bhojanapalli, Neyshabur, and Srebro]{DBLP:conf/nips/BhojanapalliNS16}
Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro.
\newblock Global optimality of local search for low rank matrix recovery.
\newblock In \emph{Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain}, pages 3873--3881, 2016.
\bibitem[Bu et~al.(2021)Bu, Gopi, Kulkarni, Lee, Shen, and Tantipongpipat]{bu2021fast}
Zhiqi Bu, Sivakanth Gopi, Janardhan Kulkarni, Yin~Tat Lee, Judy~Hanwen Shen, and Uthaipon Tantipongpipat.
\newblock Fast and memory efficient differentially private-{SGD} via {JL} projections.
\newblock In \emph{Advances in Neural Information Processing Systems (NeurIPS)}, pages 19680--19691, virtual, 2021.
\bibitem[Bu et~al.(2023)Bu, Wang, Zha, and Karypis]{bu2023automatic}
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis.
\newblock Automatic clipping: Differentially private deep learning made easier and stronger.
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\bibitem[Carlini et~al.(2019)Carlini, Liu, Erlingsson, Kos, and Song]{carlini_SecretSharer_2019a}
Nicholas Carlini, Chang Liu, {\'U}lfar Erlingsson, Jernej Kos, and Dawn Song.
\newblock The secret sharer: {{Evaluating}} and testing unintended memorization in neural networks.
\newblock In \emph{28th {{USENIX}} Security Symposium ({{USENIX}} Security 19)}, pages 267--284, Santa Clara, CA, August 2019. USENIX Association.
\newblock ISBN 978-1-939133-06-9.
\bibitem[Carmon and Duchi(2018)]{carmon_AnalysisKrylov_2018}
Yair Carmon and John~C Duchi.
\newblock Analysis of {{Krylov Subspace Solutions}} of {{Regularized Non-Convex Quadratic Problems}}.
\newblock In \emph{Advances in {{Neural Information Processing Systems}}}, volume~31. {Curran Associates, Inc.}, 2018.
\bibitem[Carmon and Duchi(2020)]{carmon_FirstOrderMethods_2020}
Yair Carmon and John~C. Duchi.
\newblock First-{{Order Methods}} for {{Nonconvex Quadratic Minimization}}.
\newblock \emph{SIAM Review}, 62\penalty0 (2):\penalty0 395--436, January 2020.
\newblock ISSN 0036-1445.
\newblock \doi{10.1137/20M1321759}.
\bibitem[Chaudhuri and Monteleoni(2008)]{chaudhuri_PrivacypreservingLogistic_2008}
Kamalika Chaudhuri and Claire Monteleoni.
\newblock Privacy-preserving logistic regression.
\newblock In \emph{Advances in {{Neural Information Processing Systems}}}, volume~21. {Curran Associates, Inc.}, 2008.
\bibitem[Chaudhuri et~al.(2011)Chaudhuri, Monteleoni, and Sarwate]{chaudhuri_DifferentiallyPrivate_2011}
Kamalika Chaudhuri, Claire Monteleoni, and Anand~D. Sarwate.
\newblock Differentially {{Private Empirical Risk Minimization}}.
\newblock \emph{The Journal of Machine Learning Research}, 12\penalty0 (null):\penalty0 1069--1109, July 2011.
\newblock ISSN 1532-4435.
\bibitem[Chen et~al.(2020)Chen, Wu, and Hong]{chen_UnderstandingGradient_2020}
Xiangyi Chen, Steven~Z. Wu, and Mingyi Hong.
\newblock Understanding {{Gradient Clipping}} in {{Private SGD}}: {{A Geometric Perspective}}.
\newblock In \emph{Advances in {{Neural Information Processing Systems}}}, volume~33, pages 13773--13782. Curran Associates, Inc., 2020.
\bibitem[Daneshmand et~al.(2018)Daneshmand, Kohler, Lucchi, and Hofmann]{daneshmand_EscapingSaddles_2018}
Hadi Daneshmand, Jonas Kohler, Aurelien Lucchi, and Thomas Hofmann.
\newblock Escaping {{Saddles}} with {{Stochastic Gradients}}.
\newblock In \emph{Proceedings of the 35th {{International Conference}} on {{Machine Learning}}}, pages 1155--1164. {PMLR}, July 2018.
\bibitem[Dwork and Roth(2014)]{dwork_AlgorithmicFoundations_2014}
Cynthia Dwork and Aaron Roth.
\newblock The {{Algorithmic Foundations}} of {{Differential Privacy}}.
\newblock \emph{Foundations and Trends{\textregistered} in Theoretical Computer Science}, 9\penalty0 (3{\textendash}4):\penalty0 211--407, August 2014.
\newblock ISSN 1551-305X, 1551-3068.
\newblock \doi{10.1561/0400000042}.
\bibitem[Dwork et~al.(2006{\natexlab{a}})Dwork, Kenthapadi, McSherry, Mironov, and Naor]{DBLP:conf/eurocrypt/DworkKMMN06}
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor.
\newblock Our data, ourselves: Privacy via distributed noise generation.
\newblock In \emph{Advances in Cryptology - {EUROCRYPT} 2006, 25th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28 - June 1, 2006, Proceedings}, volume 4004 of \emph{Lecture Notes in Computer Science}, pages 486--503. Springer, 2006{\natexlab{a}}.
\bibitem[Dwork et~al.(2006{\natexlab{b}})Dwork, McSherry, Nissim, and Smith]{dwork_CalibratingNoise_2006}
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith.
\newblock Calibrating noise to sensitivity in private data analysis.
\newblock In \emph{Proceedings of the {{Third}} Conference on {{Theory}} of {{Cryptography}}}, {{TCC}}'06, pages 265--284, {Berlin, Heidelberg}, March 2006{\natexlab{b}}. {Springer-Verlag}.
\newblock ISBN 978-3-540-32731-8.
\newblock \doi{10.1007/11681878_14}.
\bibitem[Fang et~al.(2018)Fang, Li, Lin, and Zhang]{fang_SPIDERNearOptimal_2018}
Cong Fang, Chris~Junchi Li, Zhouchen Lin, and Tong Zhang.
\newblock {{SPIDER}}: {{Near-Optimal Non-Convex Optimization}} via {{Stochastic Path-Integrated Differential Estimator}}.
\newblock In S.~Bengio, H.~Wallach, H.~Larochelle, K.~Grauman, N.~{Cesa-Bianchi}, and R.~Garnett, editors, \emph{Advances in {{Neural Information Processing Systems}}}, volume~31. {Curran Associates, Inc.}, 2018.
\bibitem[Fang et~al.(2023)Fang, Li, Fan, and Li]{DBLP:conf/iclr/FangLF023}
Huang Fang, Xiaoyun Li, Chenglin Fan, and Ping Li.
\newblock Improved convergence of differential private {SGD} with gradient clipping.
\newblock In \emph{The Eleventh International Conference on Learning Representations, {ICLR} 2023, Kigali, Rwanda, May 1-5, 2023}, 2023.
\bibitem[Feldman et~al.(2020)Feldman, Koren, and Talwar]{feldman2020private}
Vitaly Feldman, Tomer Koren, and Kunal Talwar.
\newblock Private stochastic convex optimization: optimal rates in linear time.
\newblock In \emph{Proccedings of the 52nd Annual {ACM} {SIGACT} Symposium on Theory of Computing (STOC)}, pages 439--449, Chicago, IL, 2020.
\bibitem[Ficek et~al.(2021)Ficek, Wang, Chen, Dagne, and Daley]{ficek2021differential}
Joseph Ficek, Wei Wang, Henian Chen, Getachew Dagne, and Ellen Daley.
\newblock Differential privacy in health research: A scoping review.
\newblock \emph{Journal of the American Medical Informatics Association}, 28\penalty0 (10):\penalty0 2269--2276, 2021.
\bibitem[Ge et~al.(2018)Ge, Wang, Wang, and Liu]{ge_MinimaxOptimalPrivacyPreserving_2018}
Jason Ge, Zhaoran Wang, Mengdi Wang, and Han Liu.
\newblock Minimax-{{Optimal Privacy-Preserving Sparse PCA}} in {{Distributed Systems}}.
\newblock In \emph{Proceedings of the {{Twenty-First International Conference}} on {{Artificial Intelligence}} and {{Statistics}}}, pages 1589--1598. {PMLR}, March 2018.
\bibitem[Ge et~al.(2015)Ge, Huang, Jin, and Yuan]{DBLP:conf/colt/GeHJY15}
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan.
\newblock Escaping from saddle points - online stochastic gradient for tensor decomposition.
\newblock In \emph{Proceedings of The 28th Conference on Learning Theory, {COLT} 2015, Paris, France, July 3-6, 2015}, pages 797--842, 2015.
\bibitem[Ge et~al.(2016)Ge, Lee, and Ma]{ge_MatrixCompletion_2016a}
Rong Ge, Jason~D Lee, and Tengyu Ma.
\newblock Matrix completion has no spurious local minimum.
\newblock In D.~Lee, M.~Sugiyama, U.~Luxburg, I.~Guyon, and R.~Garnett, editors, \emph{Advances in Neural Information Processing Systems}, volume~29. Curran Associates, Inc., 2016.
\bibitem[Ge et~al.(2017)Ge, Jin, and Zheng]{DBLP:conf/icml/0001JZ17}
Rong Ge, Chi Jin, and Yi~Zheng.
\newblock No spurious local minima in nonconvex low rank problems: {A} unified geometric analysis.
\newblock In \emph{Proceedings of the 34th International Conference on Machine Learning, {ICML} 2017, Sydney, NSW, Australia, 6-11 August 2017}, pages 1233--1242, 2017.
\bibitem[Ghadimi and Lan(2013)]{DBLP:journals/siamjo/GhadimiL13a}
Saeed Ghadimi and Guanghui Lan.
\newblock Stochastic first- and zeroth-order methods for nonconvex stochastic programming.
\newblock \emph{{SIAM} J. Optim.}, 23\penalty0 (4):\penalty0 2341--2368, 2013.
\bibitem[Goodfellow(2015)]{goodfellow2015efficient}
Ian Goodfellow.
\newblock Efficient per-example gradient computations.
\newblock \emph{arXiv preprint arXiv:1510.01799}, 2015.
\bibitem[Gupta et~al.(2010)Gupta, Ligett, McSherry, Roth, and Talwar]{DBLP:conf/soda/GuptaLMRT10}
Anupam Gupta, Katrina Ligett, Frank McSherry, Aaron Roth, and Kunal Talwar.
\newblock Differentially private combinatorial optimization.
\newblock In \emph{Proceedings of the Twenty-First Annual {ACM-SIAM} Symposium on Discrete Algorithms, {SODA} 2010, Austin, Texas, USA, January 17-19, 2010}, pages 1106--1125, 2010.
\bibitem[Harvey et~al.(2019)Harvey, Liaw, Plan, and Randhawa]{DBLP:conf/colt/HarveyLPR19}
Nicholas J.~A. Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa.
\newblock Tight analyses for non-smooth stochastic gradient descent.
\newblock In \emph{Conference on Learning Theory, {COLT} 2019, 25-28 June 2019, Phoenix, AZ, {USA}}, volume~99 of \emph{Proceedings of Machine Learning Research}, pages 1579--1613. {PMLR}, 2019.
\bibitem[Hassan et~al.(2020)Hassan, Rehmani, and Chen]{hassan_DifferentialPrivacy_2020}
Muneeb~Ul Hassan, Mubashir~Husain Rehmani, and Jinjun Chen.
\newblock Differential privacy techniques for cyber physical systems: A survey.
\newblock \emph{IEEE Communications Surveys \& Tutorials}, 22\penalty0 (1):\penalty0 746--789, 2020.
\newblock \doi{10.1109/COMST.2019.2944748}.
\bibitem[Jain et~al.(2018)Jain, Thakkar, and Thakurta]{jain_DifferentiallyPrivate_2018}
Prateek Jain, Om~Dipakbhai Thakkar, and Abhradeep Thakurta.
\newblock Differentially {{Private Matrix Completion Revisited}}.
\newblock In \emph{Proceedings of the 35th {{International Conference}} on {{Machine Learning}}}, pages 2215--2224. {PMLR}, July 2018.
\bibitem[Jiang et~al.(2021)Jiang, Li, Yue, and Song]{jiang_DifferentialPrivacy_2021}
Bin Jiang, Jianqiang Li, Guanghui Yue, and Houbing Song.
\newblock Differential {{Privacy}} for {{Industrial Internet}} of {{Things}}: {{Opportunities}}, {{Applications}}, and {{Challenges}}.
\newblock \emph{IEEE Internet of Things Journal}, 8\penalty0 (13):\penalty0 10430--10451, July 2021.
\newblock ISSN 2327-4662.
\newblock \doi{10.1109/JIOT.2021.3057419}.
\bibitem[Jin et~al.(2017)Jin, Ge, Netrapalli, Kakade, and Jordan]{jin_HowEscape_2017}
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham~M. Kakade, and Michael~I. Jordan.
\newblock How to {{Escape Saddle Points Efficiently}}.
\newblock In \emph{Proceedings of the 34th {{International Conference}} on {{Machine Learning}}}, pages 1724--1732. {PMLR}, July 2017.
\bibitem[Jin et~al.(2018)Jin, Netrapalli, and Jordan]{jin_AcceleratedGradient_2018}
Chi Jin, Praneeth Netrapalli, and Michael~I. Jordan.
\newblock Accelerated {{Gradient Descent Escapes Saddle Points Faster}} than {{Gradient Descent}}.
\newblock In \emph{Proceedings of the 31st {{Conference On Learning Theory}}}, pages 1042--1085. {PMLR}, July 2018.
\bibitem[Jin et~al.(2019)Jin, Netrapalli, Ge, Kakade, and Jordan]{jin_ShortNote_2019}
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham~M. Kakade, and Michael~I. Jordan.
\newblock A {{Short Note}} on {{Concentration Inequalities}} for {{Random Vectors}} with {{SubGaussian Norm}}, February 2019.
\bibitem[Jin et~al.(2021)Jin, Netrapalli, Ge, Kakade, and Jordan]{jin_NonconvexOptimization_2021}
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham~M. Kakade, and Michael~I. Jordan.
\newblock On {{Nonconvex Optimization}} for {{Machine Learning}}: {{Gradients}}, {{Stochasticity}}, and {{Saddle Points}}.
\newblock \emph{Journal of the ACM}, 68\penalty0 (2):\penalty0 11:1--11:29, February 2021.
\newblock ISSN 0004-5411.
\newblock \doi{10.1145/3418526}.
\bibitem[Jung et~al.(2024)Jung, Ligett, Neel, Roth, {Sharifi-Malvajerdi}, and Shenfeld]{jung_NewAnalysis_2024}
Christopher Jung, Katrina Ligett, Seth Neel, Aaron Roth, Saeed {Sharifi-Malvajerdi}, and Moshe Shenfeld.
\newblock A {{New Analysis}} of {{Differential Privacy}}'s {{Generalization Guarantees}}, June 2024.
\bibitem[Kairouz et~al.(2021)Kairouz, McMahan, Avent, Bellet, Bennis, Bhagoji, Bonawitz, Charles, Cormode, Cummings, D'Oliveira, Eichner, Rouayheb, Evans, Gardner, Garrett, Gasc{\'o}n, Ghazi, Gibbons, Gruteser, Harchaoui, He, He, Huo, Hutchinson, Hsu, Jaggi, Javidi, Joshi, Khodak, Konecn{\'y}, Korolova, Koushanfar, Koyejo, Lepoint, Liu, Mittal, Mohri, Nock, {\"O}zg{\"u}r, Pagh, Qi, Ramage, Raskar, Raykova, Song, Song, Stich, Sun, Suresh, Tram{\`e}r, Vepakomma, Wang, Xiong, Xu, Yang, Yu, Yu, and Zhao]{kairouz_AdvancesOpen_2021}
Peter Kairouz, H.~Brendan McMahan, Brendan Avent, Aur{\'e}lien Bellet, Mehdi Bennis, Arjun~Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G.~L. D'Oliveira, Hubert Eichner, Salim~El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adri{\`a} Gasc{\'o}n, Badih Ghazi, Phillip~B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecn{\'y}, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancr{\`e}de Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer {\"O}zg{\"u}r, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian~U. Stich, Ziteng Sun, Ananda~Theertha Suresh, Florian Tram{\`e}r, Praneeth Vepakomma, Jianyu Wang, Li~Xiong, Zheng Xu, Qiang Yang, Felix~X. Yu, Han Yu, and Sen Zhao.
\newblock Advances and {{Open Problems}} in {{Federated Learning}}.
\newblock \emph{Foundations and Trends{\textregistered} in Machine Learning}, 14\penalty0 (1--2):\penalty0 1--210, June 2021.
\newblock ISSN 1935-8237, 1935-8245.
\newblock \doi{10.1561/2200000083}.
\bibitem[Kawaguchi(2016)]{kawaguchi_DeepLearning_2016}
Kenji Kawaguchi.
\newblock Deep learning without poor local minima.
\newblock In D.~Lee, M.~Sugiyama, U.~Luxburg, I.~Guyon, and R.~Garnett, editors, \emph{Advances in Neural Information Processing Systems}, volume~29. Curran Associates, Inc., 2016.
\bibitem[Koloskova et~al.(2023)Koloskova, Hendrikx, and Stich]{koloskova_RevisitingGradient_2023}
Anastasia Koloskova, Hadrien Hendrikx, and Sebastian~U Stich.
\newblock Revisiting {{Gradient Clipping}}: {{Stochastic}} bias and tight convergence guarantees.
\newblock In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, \emph{Proceedings of the 40th International Conference on Machine Learning}, volume 202 of \emph{Proceedings of Machine Learning Research}, pages 17343--17363. PMLR, 2023.
\bibitem[Krizhevsky and Hinton(2009)]{krizhevsky_LearningMultiple_2009}
Alex Krizhevsky and Geoffrey Hinton.
\newblock Learning multiple layers of features from tiny images.
\newblock 2009.
\newblock Publisher: Toronto, ON, Canada.
\bibitem[Leclerc et~al.(2023)Leclerc, Ilyas, Engstrom, Park, Salman, and Madry]{leclerc2023ffcv}
Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung~Min Park, Hadi Salman, and Aleksander Madry.
\newblock {FFCV}: Accelerating training by removing data bottlenecks.
\newblock In \emph{Computer Vision and Pattern Recognition (CVPR)}, 2023.
\newblock \url{https://github.com/libffcv/ffcv/}. commit xxxxxxx.
\bibitem[Lecuyer et~al.(2019)Lecuyer, Atlidakis, Geambasu, Hsu, and Jana]{lecuyer_CertifiedRobustness_2019}
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana.
\newblock Certified robustness to adversarial examples with differential privacy.
\newblock In \emph{2019 {{IEEE}} Symposium on Security and Privacy ({{SP}})}, pages 656--672, 2019.
\newblock \doi{10.1109/SP.2019.00044}.
\bibitem[Li et~al.(2016)Li, Lyu, Su, and Yang]{DBLP:series/synthesis/2016Li}
Ninghui Li, Min Lyu, Dong Su, and Weining Yang.
\newblock \emph{Differential Privacy: From Theory to Practice}.
\newblock Synthesis Lectures on Information Security, Privacy, {\&} Trust. Morgan {\&} Claypool Publishers, 2016.
\bibitem[Liu et~al.(2023)Liu, Ganesh, Oh, and Thakurta]{liu2023private}
Daogao Liu, Arun Ganesh, Sewoong Oh, and Abhradeep~Guha Thakurta.
\newblock Private (stochastic) non-convex optimization revisited: Second-order stationary points and excess risks.
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\bibitem[Nemirovski et~al.(2009)Nemirovski, Juditsky, Lan, and Shapiro]{DBLP:journals/siamjo/NemirovskiJLS09}
Arkadi Nemirovski, Anatoli~B. Juditsky, Guanghui Lan, and Alexander Shapiro.
\newblock Robust stochastic approximation approach to stochastic programming.
\newblock \emph{{SIAM} J. Optim.}, 19\penalty0 (4):\penalty0 1574--1609, 2009.
\bibitem[Nesterov and Polyak(2006)]{nesterovP06}
Yurii Nesterov and Boris~T Polyak.
\newblock Cubic regularization of newton method and its global performance.
\newblock \emph{Mathematical Programming}, 108\penalty0 (1):\penalty0 177--205, 2006.
\bibitem[Nocedal and Wright(1999)]{nocedal_NumericalOptimization_1999}
Jorge Nocedal and Stephen~J. Wright, editors.
\newblock \emph{Numerical {{Optimization}}}.
\newblock Springer {{Series}} in {{Operations Research}} and {{Financial Engineering}}. {Springer-Verlag}, {New York}, 1999.
\newblock ISBN 978-0-387-98793-4.
\newblock \doi{10.1007/b98874}.
\bibitem[Oja and Karhunen(1985)]{oja1985stochastic}
Erkki Oja and Juha Karhunen.
\newblock On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix.
\newblock \emph{Journal of mathematical analysis and applications}, 106\penalty0 (1):\penalty0 69--84, 1985.
\bibitem[O'Searcoid(2006)]{Searcoid2006}
Mícheál O'Searcoid.
\newblock \emph{Metric Spaces}.
\newblock Springer Undergraduate Mathematics. Springer, 2006.
\bibitem[Rochette et~al.(2019)Rochette, Manoel, and Tramel]{rochette2019efficient}
Gaspar Rochette, Andre Manoel, and Eric~W Tramel.
\newblock Efficient per-example gradient computations in convolutional neural networks.
\newblock \emph{arXiv preprint arXiv:1912.06015}, 2019.
\bibitem[Sekhari et~al.(2021)Sekhari, Acharya, Kamath, and Suresh]{sekhari_RememberWhat_2021}
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda~Theertha Suresh.
\newblock Remember what you want to forget: {{Algorithms}} for machine unlearning.
\newblock In M.~Ranzato, A.~Beygelzimer, Y.~Dauphin, P.S. Liang, and J.~Wortman Vaughan, editors, \emph{Advances in Neural Information Processing Systems}, volume~34, pages 18075--18086. Curran Associates, Inc., 2021.
\bibitem[Song et~al.(2013)Song, Chaudhuri, and Sarwate]{song2013stochastic}
Shuang Song, Kamalika Chaudhuri, and Anand~D. Sarwate.
\newblock Stochastic gradient descent with differentially private updates.
\newblock In \emph{Proceedings of the {IEEE} Global Conference on Signal and Information Processing (GlobalSIP)}, pages 245--248, Austin, TX, 2013.
\bibitem[Song et~al.(2021)Song, Steinke, Thakkar, and Thakurta]{song2021evading}
Shuang Song, Thomas Steinke, Om~Thakkar, and Abhradeep Thakurta.
\newblock Evading the curse of dimensionality in unconstrained private glms.
\newblock In \emph{Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS)}, pages 2638--2646, Virtual Event, 2021.
\bibitem[Subramani et~al.(2021)Subramani, Vadivelu, and Kamath]{subramani2021enabling}
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath.
\newblock Enabling fast differentially private {SGD} via just-in-time compilation and vectorization.
\newblock In \emph{Advances in Neural Information Processing Systems (NeurIPS)}, pages 26409--26421, 2021.
\bibitem[Sun et~al.(2017)Sun, Qu, and Wright]{sun_CompleteDictionary_2017}
Ju~Sun, Qing Qu, and John Wright.
\newblock Complete dictionary recovery over the sphere {{I}}: {{Overview}} and the geometric picture.
\newblock \emph{IEEE Transactions on Information Theory}, 63\penalty0 (2):\penalty0 853--884, 2017.
\newblock \doi{10.1109/TIT.2016.2632162}.
\bibitem[Tripuraneni et~al.(2018)Tripuraneni, Stern, Jin, Regier, and Jordan]{tripuraneni_StochasticCubic_2018}
Nilesh Tripuraneni, Mitchell Stern, Chi Jin, Jeffrey Regier, and Michael~I Jordan.
\newblock Stochastic {{Cubic Regularization}} for {{Fast Nonconvex Optimization}}.
\newblock In \emph{Advances in {{Neural Information Processing Systems}}}, volume~31. {Curran Associates, Inc.}, 2018.
\bibitem[Vietri et~al.(2020)Vietri, Balle, Krishnamurthy, and Wu]{DBLP:conf/icml/VietriBKW20}
Giuseppe Vietri, Borja Balle, Akshay Krishnamurthy, and Zhiwei~Steven Wu.
\newblock Private reinforcement learning with {PAC} and regret guarantees.
\newblock In \emph{Proceedings of the 37th International Conference on Machine Learning, {ICML} 2020, 13-18 July 2020, Virtual Event}, pages 9754--9764, 2020.
\bibitem[Wang and Xu(2020)]{DBLP:conf/pkdd/WangX20}
Di~Wang and Jinhui Xu.
\newblock Escaping saddle points of empirical risk privately and scalably via dp-trust region method.
\newblock In \emph{Machine Learning and Knowledge Discovery in Databases - European Conference, {ECML} {PKDD} 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part {III}}, volume 12459 of \emph{Lecture Notes in Computer Science}, pages 90--106. Springer, 2020.
\bibitem[Wang et~al.(2017)Wang, Ye, and Xu]{wang2017differentially}
Di~Wang, Minwei Ye, and Jinhui Xu.
\newblock Differentially private empirical risk minimization revisited: Faster and more general.
\newblock In \emph{Advances in Neural Information Processing Systems (NIPS)}, pages 2722--2731, 2017.
\bibitem[Wang et~al.(2019{\natexlab{a}})Wang, Chen, and Xu]{wang_DifferentiallyPrivate_2019}
Di~Wang, Changyou Chen, and Jinhui Xu.
\newblock Differentially {{Private Empirical Risk Minimization}} with {{Non-convex Loss Functions}}.
\newblock In \emph{Proceedings of the 36th {{International Conference}} on {{Machine Learning}}}, pages 6526--6535. {PMLR}, May 2019{\natexlab{a}}.
\bibitem[Wang et~al.(2022)Wang, Lei, Ying, and Zhang]{wang2022differentially}
Puyu Wang, Yunwen Lei, Yiming Ying, and Hai Zhang.
\newblock Differentially private sgd with non-smooth losses.
\newblock \emph{Applied and Computational Harmonic Analysis}, 56:\penalty0 306--336, 2022.
\bibitem[Wang et~al.(2019{\natexlab{b}})Wang, Ji, Zhou, Liang, and Tarokh]{wang_SpiderBoostMomentum_2019}
Zhe Wang, Kaiyi Ji, Yi~Zhou, Yingbin Liang, and Vahid Tarokh.
\newblock {{SpiderBoost}} and {{Momentum}}: {{Faster Variance Reduction Algorithms}}.
\newblock In \emph{Advances in {{Neural Information Processing Systems}}}, volume~32. {Curran Associates, Inc.}, 2019{\natexlab{b}}.
\bibitem[Warstadt et~al.(2019)Warstadt, Singh, and Bowman]{warstadt_NeuralNetwork_2019}
Alex Warstadt, Amanpreet Singh, and Samuel~R. Bowman.
\newblock Neural network acceptability judgments.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 7:\penalty0 625--641, 2019.
\newblock \doi{10.1162/tacl_a_00290}.
\bibitem[Xu et~al.(2018)Xu, Jin, and Yang]{xu_FirstorderStochastic_2018}
Yi~Xu, Rong Jin, and Tianbao Yang.
\newblock First-order {{Stochastic Algorithms}} for {{Escaping From Saddle Points}} in {{Almost Linear Time}}.
\newblock In S.~Bengio, H.~Wallach, H.~Larochelle, K.~Grauman, N.~{Cesa-Bianchi}, and R.~Garnett, editors, \emph{Advances in {{Neural Information Processing Systems}}}, volume~31. {Curran Associates, Inc.}, 2018.
\bibitem[Yang et~al.(2022)Yang, Zhang, Chen, and Liu]{yang2022normalized}
Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu.
\newblock Normalized/clipped {SGD} with perturbation for differentially private non-convex optimization.
\newblock \emph{arXiv preprint arXiv:2206.13033}, 2022.
\bibitem[Zhang and Li(2021)]{zhang_EscapeSaddle_2021}
Chenyi Zhang and Tongyang Li.
\newblock Escape saddle points by a simple gradient-descent based algorithm.
\newblock In \emph{Advances in {{Neural Information Processing Systems}}}, volume~34, pages 8545--8556. {Curran Associates, Inc.}, 2021.
\bibitem[Zhou and Gu(2020)]{zhou_StochasticRecursive_2020}
Dongruo Zhou and Quanquan Gu.
\newblock Stochastic {{Recursive Variance-Reduced Cubic Regularization Methods}}.
\newblock In \emph{Proceedings of the {{Twenty Third International Conference}} on {{Artificial Intelligence}} and {{Statistics}}}, pages 3980--3990. {PMLR}, June 2020.
\bibitem[Zhou et~al.(2020)Zhou, Xu, and Gu]{zhou_StochasticNested_2020}
Dongruo Zhou, Pan Xu, and Quanquan Gu.
\newblock Stochastic {{Nested Variance Reduction}} for {{Nonconvex Optimization}}.
\newblock \emph{Journal of Machine Learning Research}, 21\penalty0 (103):\penalty0 1--63, 2020.
\newblock ISSN 1533-7928.
\end{thebibliography}