ABrA-GD: Adaptive Bregman Accelerated Gradient Descent for Relatively Smooth and (Strongly-)Convex Optimization

Damien Scieur Affiliation: Samsung SAIL Montreal Email: damien.scieur@gmail.com

Abstract

We propose ABrA-GD (Adaptive Bregman Accelerated Gradient Descent), an adaptive algorithm for relatively smooth convex optimization. The algorithm is derived from a computable primal–dual certificate that guarantees progress by comparing the objective value with a lower bound on the minimum of a regularized objective. This certificate enables adaptation to both smoothness and geometry, using only the relative strong convexity constant as a problem-dependent input. We introduce the dual Bregman Length Distortion Factor (BLDF), which measures how dual Bregman lengths change under an anchor shift or rescaling. Under bounded local dual BLDF, ABrA-GD achieves an accelerated $O(1/k^{2})$ rate for convex objectives and an accelerated linear rate for relatively strongly convex objectives.

1 Setting and Introduction

We consider optimization problems of the form

\phi_{\star}=\min_{x\in\mathcal{X}}\phi(x)\;\coloneqq\;f(x)+\mu D_{d}(x,x_{0}),\quad D_{d}(x,y):=d(x)-d(y)-\langle\nabla d(y),x-y\rangle.

(1)

The term $D_{d}(x,y)$ is the Bregman divergence generated by $d$ [6, 7, 8]. Throughout the paper we work under the following hypotheses.

Assumption 1 (Problem setting).

$d$ is a Legendre function, $\mathcal{X}=\operatorname{cl}(\dom d)$ , and $x_{0}\in\operatorname{int}(\dom d)$ . The function $f:\mathcal{X}\to\mathbb{R}$ is convex, differentiable, and relatively $(L-\mu)$ -smooth with respect to $d$ [4, 19], with $0\leq\mu<L$ .

Assumption 1 is equivalent to requiring $\phi$ to be $L$ -smooth and $\mu$ -strongly convex relative to $d$ (appendices A and A.1). The domain of $d$ determines the feasible set. The analysis uses interior minimizers, for which the optimality conditions are stationarity equations. Affine equality constraints are also allowed: after a change of coordinates, the problem satisfies the same assumptions (section A.2).

Acceleration in Euclidean optimization.

For smooth convex objectives, accelerated gradient methods improve the $O(1/k)$ rate of gradient descent to $O(1/k^{2})$ [21]. Several viewpoints explain this improvement, including estimate sequences, Lyapunov functions, and the coupling of gradient and mirror-descent steps [26, 1]. Primal–dual interpretations express progress through a certificate that combines an objective upper bound with a lower model [11].

Beyond Euclidean smoothness.

Relative smoothness replaces the Euclidean quadratic upper model with a Bregman divergence [4, 19]. This allows objectives whose gradients are not globally Lipschitz, including log-determinant, entropy, and Poisson-type objectives with singular behavior near the domain boundary. Bregman gradient descent attains an $O(1/k)$ rate in this setting. Unlike the Euclidean case, this rate cannot be improved over the full relatively smooth class [12]. Acceleration therefore requires additional structure.

Conditions for Bregman acceleration.

Accelerated Bregman Proximal Gradient (ABPG) methods obtain $O(k^{-\gamma})$ rates under a Triangle Scaling Property (TSP) with exponent $\gamma$ and gain $1$ [13, 24]. For standard geometries such as entropy and Burg, this exponent is at most $1$ , so this guarantee gives no acceleration. The accelerated variants instead fix $\gamma=2$ and backtrack on the gain, so their rate depends on the gains accepted during the run. Chen et al. [10] obtain an accelerated linear rate under a generalized Cauchy–Schwarz condition, whose constant is an input of their method. Liu and Liu [18] additionally assume relative strong convexity, but their rate does not improve under this assumption.

Contributions

1.

A primal–dual formulation for acceleration (section 2). We formulate a descent requirement using the Bregman–Moreau envelope and certify it through its dual problem. This framework provides a common starting point for the algorithm and its analysis; appendix B recovers Bregman gradient descent and dual averaging from it.
2.

ABrA-GD (section 3). We derive ABrA-GD (algorithm 2), in which one gradient serves two steps: A Bregman gradient step gives an upper bound on the objective, and a dual-averaging step gives a lower bound on the minimum of a regularized objective. ABrA-GD backtracks on smoothness and acceleration parameters, and we establish a generic convergence bound in terms of the accepted acceleration parameters (Theorem 2).
3.

A local measure of geometric distortion (section 4). We introduce the dual Bregman Length Distortion Factor (BLDF) to compare the progress of gradient and dual-averaging steps. We give explicit formulas for standard separable kernels and relate this local condition to the triangle-scaling conditions. The algorithm adapts without evaluating the BLDF.
4.

Accelerated rates under bounded distortion (section 5). Under bounded local distortion, a condition implied by a uniform triangle-scaling gain (section E.2), we obtain an accelerated $O(1/k^{2})$ rate for convex objectives and an accelerated linear rate for relatively strongly convex objectives. ABrA-GD’s only problem-constant input is $\mu$ . To our knowledge, it is the first method with these inputs that has an accelerated linear rate for this class.

Limitations.

The accelerated analysis does not cover an additional nonsmooth term, unlike the ABPG analyses of Hanzely et al. [13], Savchuk et al. [24]. The envelope descent requirement extends to composite objectives (appendix F), but an accelerated BLDF analysis remains open.

2 The Primal–Dual Envelope Framework

For $\eta>0$ , define the regularized objective

\phi_{\star}^{\eta}:=\min_{x}\phi^{\eta}(x),\;\;\phi^{\eta}(x):=\phi(x)+\frac{1}{\eta}D_{d}(x,x_{0})=f(x)+\left(\mu+\tfrac{1}{\eta}\right)D_{d}(x,x_{0}),\qquad\eta>0.

(2)

The value $\phi^{\eta}_{\star}$ is the Bregman–Moreau envelope of $\phi$ anchored at $x_{0}$ [5], which approaches $\phi_{\star}$ as $\eta\to\infty$ . Consider the implicit envelope descent scheme (algorithm 1).

Algorithm 1 Implicit envelope descent (conceptual algorithm, used only as convergence tool)

Input:

\phi,x_{0}

. Initialization:

k\leftarrow 0

\eta_{0}\leftarrow 0

1 for $k=0,1,2,\dots$ do

2 Choose

\eta_{k+1}\geq\eta_{k}

and find

x_{k+1}

such that

\phi(x_{k+1})\leq\phi_{\star}^{\eta_{k+1}}

(defined in eq. 2).

Each iterate must satisfy $\phi(x_{k+1})\leq\phi_{\star}^{\eta_{k+1}}$ , but it need not minimize the regularized objective (fig. 2). The scheme does not specify how to compute $x_{k+1}$ or verify this requirement, since the envelope value is itself defined by a minimization. It does, however, give an immediate error bound.

Theorem 1.

Under Assumption 1, algorithm 1 with inputs $\phi,\,x_{0}$ satisfies

\phi(x_{k})-\phi_{\star}\leq\eta_{k}^{-1}D_{d}(x_{\star},x_{0}).

(3)

To prove the bound, evaluate the regularized objective at $x_{\star}$ (section C.1). The growth of $\eta_{k}$ is therefore the central choice. Faster growth gives a faster rate in eq. 3, but it also lowers $\phi_{\star}^{\eta_{k}}$ toward $\phi_{\star}$ , so each iteration must make more progress.

Link with proximal methods.

The proximal point method [15] centers each subproblem at the current iterate, and Accelerated Hybrid Proximal Extragradient [20] and Catalyst [16, 17] solve these subproblems approximately. In algorithm 1, the anchor $x_{0}$ stays fixed and no subproblem is solved.

2.1 Envelope Dual Problem and Lower Bound

The envelope value $\phi_{\star}^{\eta_{k+1}}$ is defined by a minimization, so the condition $\phi(x_{k+1})\leq\phi_{\star}^{\eta_{k+1}}$ cannot be checked directly. We certify it through the dual problem and provide a sufficient condition for the requirement of algorithm 1. For any $\lambda$ , the Fenchel–Young inequality gives the affine lower bound

\ell_{\lambda}(x):=\langle x,\lambda\rangle-f^{*}(\lambda)\leq f(x),

(4)

where $f^{*}$ is the Fenchel conjugate of $f$ , with equality if and only if $\lambda=\nabla f(x)$ . Adding the regularization defines the dual value and its associated primal point:

	$\displaystyle\mathcal{D}^{\eta}(\lambda)$	$\displaystyle:=\min_{x}\left\{\ell_{\lambda}(x)+(\mu+\eta^{-1})D_{d}(x,x_{0})\right\},$		(5)
	$\displaystyle z^{\eta}(\lambda)$	$\displaystyle:=\arg\min_{x}\left\{\ell_{\lambda}(x)+(\mu+\eta^{-1})D_{d}(x,x_{0})\right\}=\nabla d^{*}\!\left(\nabla d(x_{0})-\frac{\lambda}{\mu+\eta^{-1}}\right).$		(6)

By weak duality, we have $\ell_{\lambda}\leq f$ , and therefore $\mathcal{D}^{\eta}(\lambda)\leq\phi_{\star}^{\eta}\leq\phi^{\eta}(x)$ for every $x$ . Although $z^{\eta}(\lambda)$ is computable without evaluating $f^{*}(\lambda)$ , the dual value $\mathcal{D}^{\eta}(\lambda)$ generally is not. The next section shows how to certify $\phi(x_{k+1})\leq\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})$ without evaluating $f^{*}$ .

3 Deriving ABrA-GD and Its Accepted-Parameter Rate

Figure 1: Envelope descent requires that

\phi(x_{k+1})\leq\phi_{\star}^{\eta}

, without minimizing the regularized objective

\phi^{\eta}

Figure 2: The model minimizers are

x_{k+1}

and

z_{k+1}

. An upper model eq. 9 whose minimum value is below

\min_{x}\mathcal{L}_{k+1}(x)

eq. 16 certifies envelope descent.

Algorithm 2 ABrA-GD — Adaptive Bregman Accelerated Gradient Descent

Input:

x_{0}

\mu\geq 0

\phi

, initial estimates

L_{\mathrm{cur}}=L_{0}>0

and

M_{-1}>0

(default

L_{0}=1

M_{-1}=L_{0}

)

Initialization:

x_{0}^{\text{best}}=z_{0}=x_{0}

\lambda_{0}=0

\eta_{0}=0

\phi^{\rm best}_{0}=\phi(x_{0}),\,t_{0}=1;

1 for $k=0,1,2,\dots$ do

M_{k}\leftarrow\max\{M_{k-1}/4,\mu\},\quad L_{\mathrm{cur}}\leftarrow\max\{L_{\mathrm{cur}}/2,\,\mu\},\quad\phi^{\rm best}_{k+1}\leftarrow\phi^{\rm best}_{k},\quad x^{\rm best}_{k+1}\leftarrow x^{\rm best}_{k}

;

3 repeat

	$\displaystyle M_{k}$	$\displaystyle\leftarrow 2M_{k},$	$\displaystyle t_{k\geq 1}$	$\displaystyle=\min\left\{1,\,\frac{2(\eta_{k}^{-1}+\mu)}{\eta_{k}^{-1}+\sqrt{\eta_{k}^{-2}+4M_{k}(\eta_{k}^{-1}+\mu)}}\right\},$
	$\displaystyle\eta_{k+1}^{-1}$	$\displaystyle=M_{k}t_{k}^{2}-\mu,$	$\displaystyle\tau_{k}$	$\displaystyle=\frac{t_{k}-\mu/\sqrt{M_{k}L_{\mathrm{cur}}}}{1-\mu/\sqrt{M_{k}L_{\mathrm{cur}}}}.$

(x_{k+1},z_{k+1},\lambda_{k+1},L_{\mathrm{cur}},\phi^{\rm low}_{k})=\textsc{Breg-PD-step}(x_{k}^{\text{best}},z_{k},\lambda_{k},\eta_{k+1},L_{\mathrm{cur}},t_{k},\tau_{k}).

4 if $\phi^{\rm best}_{k+1}>\phi(x_{k+1})$ then

\phi^{\rm best}_{k+1}\leftarrow\phi(x_{k+1})

x^{\rm best}_{k+1}\leftarrow x_{k+1}

;

5 until $\phi^{\rm best}_{k+1}\leq\mathcal{L}_{k+1}(z_{k+1}):=(1-t_{k})\phi^{\rm best}_{k}+t_{k}\phi^{\rm low}_{k}-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1})$ ;

Algorithm 3 Breg-PD-step

(x,z,\lambda,\eta,L_{\mathrm{cur}},t,\tau)

Input:

x,z,\lambda,\eta,L_{\mathrm{cur}},t,\tau

y=(1-\tau)x+\tau z,\qquad\phi^{\mathrm{low}}=\phi(y)+\langle\nabla\phi(y),z-y\rangle+\mu D_{d}(z,y).

Update

$\displaystyle(x_{\mathrm{new}},L_{\mathrm{cur}})$	$\displaystyle=\textsc{Backtracking-GD}\bigl(y,\phi(y),\nabla\phi(y),L_{\mathrm{cur}}\bigr),$	(Primal Step)
$\displaystyle\lambda_{\mathrm{new}}$	$\displaystyle=(1-t)\lambda+t\Big(\nabla\phi(y)-\mu(\nabla d(y)-\nabla d(x_{0}))\Big),$	(Dual Step)
$\displaystyle z_{\mathrm{new}}$	$\displaystyle=\arg\min_{u}\left\{\langle\lambda_{\mathrm{new}},u\rangle+\left(\mu+\eta^{-1}\right)D_{d}(u,x_{0})\right\}.$	(Primal Map)

return

(x_{\mathrm{new}},z_{\mathrm{new}},\lambda_{\mathrm{new}},L_{\mathrm{cur}},\phi^{\mathrm{low}})

;

Algorithm 4 Backtracking-GD

(y,\phi(y),\nabla\phi(y),L_{\mathrm{cur}})

Input:

y

\phi(y)

\nabla\phi(y)

, trial constant

L_{\mathrm{cur}}>0

Initialization:

L_{\mathrm{new}}=L_{\mathrm{cur}}/2

1 repeat

L_{\mathrm{new}}\leftarrow 2L_{\mathrm{new}},\quad x_{\mathrm{new}}=\arg\min_{x}\left\{\langle\nabla\phi(y),x\rangle+L_{\mathrm{new}}D_{d}(x,y)\right\},

3 until $\phi(x_{\mathrm{new}})\leq\phi(y)+\langle\nabla\phi(y),x_{\mathrm{new}}-y\rangle+L_{\mathrm{new}}D_{d}(x_{\mathrm{new}},y)=\phi(y)-L_{\mathrm{new}}D_{d}(y,x_{\mathrm{new}})$ ;

4 return

(x_{\mathrm{new}},L_{\mathrm{new}})

;

Algorithm 1 is conceptual. The dual formulation eq. 5 says what to certify, $\phi(x_{k+1})\leq\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})$ , but not how to construct $x_{k+1}$ , $\lambda_{k+1}$ (hence $z_{k+1}=z^{\eta_{k+1}}(\lambda_{k+1})$ ), or $\eta_{k+1}$ . ABrA-GD constructs all of them, with $\eta_{k}\to\infty$ , and maintains the invariant

\phi_{k}^{\mathrm{best}}\leq\mathcal{D}^{\eta_{k}}(\lambda_{k})\leq\phi_{\star}^{\eta_{k}}.

(7)

Each trial of an iteration evaluates one gradient at a point $y_{k}$ between $x_{k}^{\mathrm{best}}$ and $z_{k}$ . This gradient gives a primal candidate $x_{k+1}$ and a dual update $\lambda_{k+1}$ . A computable lower bound then tests whether the invariant holds at step $k+1$ ; if not, backtracking increases $M_{k}$ and repeats the trial.

1. Common gradient evaluation.

Both model updates use one common gradient evaluation at the point $y_{k}$ , defined below. Here $x_{k}^{\mathrm{best}}$ is the best retained point and $z_{k}$ is the current primal map; the analysis specifies $\tau_{k}$ (lemma 7):

y_{k}=(1-\tau_{k})x_{k}^{\mathrm{best}}+\tau_{k}z_{k}.

(8)

2. Primal upper model.

The upper model at $y_{k}$ is

\mathcal{U}(x,y_{k}):=\phi(y_{k})+\langle\nabla\phi(y_{k}),x-y_{k}\rangle+L_{\mathrm{cur}}D_{d}(x,y_{k}).

(9)

Since $\phi$ is $L$ -smooth relative to $d$ (Assumption 1), $\phi\leq\mathcal{U}(\cdot,y_{k})$ when $L_{\mathrm{cur}}\geq L$ . The Primal Step in algorithm 3 minimizes this model:

x_{k+1}=\arg\min_{x}\mathcal{U}(x,y_{k}).

(10)

Backtracking-GD (algorithm 4) adjusts $L_{\mathrm{cur}}$ until $\phi(x_{k+1})\leq\mathcal{U}(x_{k+1},y_{k})$ , and terminates whenever $L_{\mathrm{cur}}\geq L$ .

3. Dual step and primal map.

The Dual Averaging Step updates $\lambda_{k}$ with the same gradient:

\lambda_{k+1}=(1-t_{k})\lambda_{k}+t_{k}\nabla f(y_{k}),\quad t_{k}\in[0,1].

(11)

Here $\nabla f(y_{k})=\nabla\phi(y_{k})-\mu(\nabla d(y_{k})-\nabla d(x_{0}))$ , so no additional objective-gradient evaluation is needed. The relation to dual best responses and conditional gradients [3] is discussed in section B.3. The Primal Map then gives

z_{k+1}=z^{\eta_{k+1}}(\lambda_{k+1}).

(12)

Both updates in eqs. 11 and 12 appear in algorithm 3.

4. Parameters $t_{k}$ , $\eta_{k}$ , and $M_{k}$ .

For $k\geq 1$ , the weight $t_{k}$ and regularization $\eta_{k+1}$ satisfy

\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1},\qquad\mu+\eta_{k+1}^{-1}=M_{k}t_{k}^{2}.

(13)

The first identity matches the weight on the previous dual model, enabling the recursive certificate below. Together, the identities give $M_{k}t_{k}^{2}=(1-t_{k})(\mu+\eta_{k}^{-1})+t_{k}\mu$ , the quadratic relation in Nesterov’s method [21]. ABrA-GD selects $M_{k}$ by backtracking and computes the positive root $t_{k}$ . The accepted $M_{k}$ controls the rate; see Theorem 2.

5. A computable certificate.

Evaluating $\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})$ requires $f^{*}(\lambda_{k+1})$ , which is not tractable in general. This step builds a computable lower bound $\mathcal{L}_{k+1}(z_{k+1})\leq\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})$ . The exit condition of algorithm 2 checks $\phi_{k+1}^{\mathrm{best}}\leq\mathcal{L}_{k+1}(z_{k+1})$ , which gives eq. 7 at step $k+1$ .

Lemma 3 gives the construction; we sketch it. The affine bound $\ell_{\lambda}$ of eq. 4 is concave in $\lambda$ , so the dual update eq. 11 and the first relation in eq. 13 split $\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})$ into two terms. Relative strong convexity gives the first, and the Fenchel equality with the three-point identity gives the second:

	$\displaystyle\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})\geq\min_{x}\Big\{$	$\displaystyle(1-t_{k}){\fcolorbox{certdual}{certdual!10}{$\displaystyle\mathcal D^{\eta_k}(\lambda_k)\!+\!(\mu+\eta_k^{-1})D_d(x,z_k)$}}$		(14)
		$\displaystyle+t_{k}{\fcolorbox{certgrad}{certgrad!10}{$\displaystyle\phi(y_k)\!+\!\langle\nabla\phi(y_k),x-y_k\rangle\!+\!\mu D_d(x,y_k)$}}\Big\}.$		(14)

By the invariant eq. 7 at step $k$ , $\mathcal{D}^{\eta_{k}}(\lambda_{k})\geq\phi_{k}^{\mathrm{best}}$ . Replacing $\mathcal{D}^{\eta_{k}}(\lambda_{k})$ by $\phi_{k}^{\mathrm{best}}$ in the first box of eq. 14 therefore gives $\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})\geq\min_{x}\mathcal{L}_{k+1}(x)$ , where $\mathcal{L}_{k+1}$ is the lower model (fig. 2)

\hskip-15.00002pt\mathcal{L}_{k+1}(x):=(1-t_{k}){\fcolorbox{certdual}{certdual!10}{$\displaystyle\phi_k^{\mathrm{best}}\!+\!(\mu+\eta_k^{-1})D_d(x,z_k)$}}+t_{k}{\fcolorbox{certgrad}{certgrad!10}{$\displaystyle\phi(y_k)\!+\!\langle\nabla\phi(y_k),x-y_k\rangle\!+\!\mu D_d(x,y_k)$}}.

(15)

Both boxes use only $\phi_{k}^{\mathrm{best}}$ , $z_{k}$ and one gradient at $y_{k}$ , so $\mathcal{L}_{k+1}$ is computable. Finally, lemma 3 shows that $\mathcal{L}_{k+1}$ is minimized at $z_{k+1}$ . Writing $\phi_{k}^{\mathrm{low}}$ for the second term of eq. 15 at $x=z_{k}$ , we obtain the right-hand side of the exit condition of algorithm 2:

\mathcal{L}_{k+1}(z_{k+1})=(1-t_{k})\phi_{k}^{\mathrm{best}}+t_{k}\phi_{k}^{\mathrm{low}}-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1}).

(16)

Convergence in terms of the accepted parameters

The exit condition preserves the envelope descent requirement. The following theorem expresses the resulting rate in terms of the accepted acceleration parameters $M_{k}$ , without imposing an additional condition on the geometry. The theorem does not bound the accepted parameters themselves, nor show that the inner loop terminates. Sections 4 and 5 give accelerated rates under bounded local distortion.

Theorem 2 (A posteriori convergence of ABrA-GD).

Under Assumption 1, assume that the inner loop of algorithm 2 exits at every iteration. Let $M_{k}>\mu$ and choose $t_{k}\in(0,1]$ by

\mu+\eta_{k+1}^{-1}=M_{k}t_{k}^{2}\quad(k\geq 0),\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}\quad(k\geq 1),\qquad t_{0}=1,\quad\eta_{1}^{-1}=M_{0}-\mu.

Then, for any $x_{\star}\in\argmin_{x}\phi(x)$ ,

\phi_{k+1}^{\mathrm{best}}\leq\phi_{k}^{\mathrm{best}},\qquad\phi_{k+1}^{\mathrm{best}}\leq\phi_{\star}^{\eta_{k+1}},\qquad\phi_{k+1}^{\mathrm{best}}-\phi_{\star}\leq\eta_{k+1}^{-1}D_{d}(x_{\star},x_{0}).

Moreover, the decay of $\eta_{k}^{-1}$ is controlled by the acceleration parameters $M_{k}$ :

\begin{array}[]{c|c|c}\textbf{Case}&\textbf{No uniform bound on $M_{k}$}&\textbf{If $M_{i}\leq M$ for all $i\geq K$}\\ \hline\cr\mu=0&\displaystyle\eta_{k}^{-1}\leq\frac{4}{\left(\sum_{i=0}^{k-1}M_{i}^{-1/2}\right)^{2}}&\displaystyle\eta_{k}^{-1}\leq\frac{\eta_{K}^{-1}}{\left(1+\frac{k-K}{2\sqrt{M\eta_{K}}}\right)^{2}}\\[13.99995pt] \mu>0&\displaystyle\eta_{k}^{-1}\leq(M_{0}-\mu)\prod_{i=1}^{k-1}\left(1-\sqrt{\frac{\mu}{M_{i}}}\right)&\displaystyle\eta_{k}^{-1}\leq\eta_{K}^{-1}\left(1-\sqrt{\frac{\mu}{M}}\right)^{k-K}.\end{array}

4 Dual Bregman Length Distortion

We identify the local geometric distortion in the comparison between the gradient and dual-averaging steps. We define the dual BLDF, relate it to local norms, and give formulas for standard kernels. Section 5 uses this condition to obtain accelerated rates and compares them with ABPG.

4.1 Comparing the gradient and dual-averaging steps (convex case)

The exit condition requires $\phi(x_{k+1})\leq\mathcal{L}_{k+1}(z_{k+1})$ . We compare how far the upper model and the lower bound decrease from the common value $\phi(y_{k})$ . Here $\mu=0$ and $\tau_{k}=t_{k}$ . The primal step (line 4 of algorithm 4) and the exit test (line 2 of algorithm 2) give

\displaystyle\phi(x_{k+1})\leq\phi(y_{k})-{\color[rgb]{0.1211,0.3047,0.4727}\underbrace{L_{\mathrm{cur}}D_{d}(y_{k},x_{k+1})}_{\text{primal decrease}}},\quad\mathcal{L}_{k+1}(z_{k+1})\geq\phi(y_{k})-{\color[rgb]{0.7227,0.5273,0.043}\underbrace{\eta_{k+1}^{-1}D_{d}(z_{k},z_{k+1})}_{\text{dual decrease}}},

where the second inequality uses $y_{k}=(1-t_{k})x_{k}^{\mathrm{best}}+t_{k}z_{k}$ and convexity, $(1-t_{k})\phi_{k}^{\mathrm{best}}+t_{k}\phi_{k}^{\mathrm{low}}\geq\phi(y_{k})$ (lemma 8). The exit condition is therefore guaranteed to hold when the upper model decreases at least as much as the lower bound:

{\color[rgb]{0.7227,0.5273,0.043}\eta_{k+1}^{-1}D_{d}(z_{k},z_{k+1})}\leq{\color[rgb]{0.1211,0.3047,0.4727}L_{\mathrm{cur}}D_{d}(y_{k},x_{k+1})}\quad\Rightarrow\quad\phi(x_{k+1})\leq\mathcal{L}_{k+1}(z_{k+1}).

(17)

When $d=\tfrac{1}{2}\|\cdot\|^{2}$ , both decreases are multiples of $\|\nabla\phi(y_{k})\|^{2}$ : ${\color[rgb]{0.1211,0.3047,0.4727}L_{\mathrm{cur}}^{-1}\|\nabla\phi(y_{k})\|^{2}/2}$ and ${\color[rgb]{0.7227,0.5273,0.043}t_{k}^{2}\eta_{k+1}\|\nabla\phi(y_{k})\|^{2}/2}$ . Equality in eq. 17 gives $\eta_{k+1}^{-1}=L_{\mathrm{cur}}t_{k}^{2}$ , the Nesterov coupling (appendix D). For a general $d$ , the two decreases are Bregman divergences taken at different points and with different step lengths, so they cannot be compared term by term. In dual coordinates, however, with $g_{k}=\nabla\phi(y_{k})$ , the two steps are segments along the same direction:

\nabla d(x_{k+1})=\nabla d(y_{k})-L_{\mathrm{cur}}^{-1}g_{k},\qquad\nabla d(z_{k+1})=\nabla d(z_{k})-t_{k}\eta_{k+1}g_{k}.

Since $D_{d}(x,y)=D_{d^{*}}(\nabla d(y),\nabla d(x))$ , the condition in eq. 17 reads

{\color[rgb]{0.7227,0.5273,0.043}\eta_{k+1}^{-1}D_{d^{*}}\big(\nabla d(z_{k})-t_{k}\eta_{k+1}g_{k},\,\nabla d(z_{k})\big)}\leq{\color[rgb]{0.1211,0.3047,0.4727}L_{\mathrm{cur}}D_{d^{*}}\big(\nabla d(y_{k})-L_{\mathrm{cur}}^{-1}g_{k},\,\nabla d(y_{k})\big)}.

Both sides measure the Bregman length of a segment along the same direction $-g_{k}$ , from different anchors, $\nabla d(z_{k})$ and $\nabla d(y_{k})$ , and with different scales, $t_{k}\eta_{k+1}$ and $L_{\mathrm{cur}}^{-1}$ .

Strongly convex case.

When $\mu>0$ , the two dual segments point toward $\nabla d(q_{k+1})$ , with $q_{k+1}$ as in lemma 9, and are no longer parallel (fig. 3(b)). The proof of proposition 6 makes two comparisons, each between two segments with the same anchor and direction but different scales: from $\nabla d(y_{k})$ to $\nabla d(x_{k+1})$ and to $\nabla d(q_{k+1})$ , and likewise from $\nabla d(z_{k})$ .

Figure 3: The two steps in dual coordinates

u=\nabla d(x)

for the Burg kernel. The ellipses are the unit balls

\{\xi:\tfrac{1}{2}\langle\xi,\nabla^{2}d^{*}(w)\,\xi\rangle\leq 1\}

of the local dual norm (definition 8) at the anchors

w

, drawn at a common scale; they change across the region, and

\rho_{d^{*}}

bounds this change on the shaded convex set, which contains

\Omega_{k}

(definition 3). (a) When

\mu=0

, the primal and dual steps are parallel displacements along

-\nabla\phi(y_{k})

, from different anchors and with different scales. (b) When

\mu>0

\nabla d(x_{k+1})

and

\nabla d(z_{k+1})

lie on two segments whose extensions meet at

\nabla d(q_{k+1})

; the shaded set contains the triangle they form with

\nabla d(y_{k})

and

\nabla d(z_{k})

4.2 The dual Bregman length distortion factor

In both cases, the comparison is between Bregman lengths of segments along a common direction, after a change of anchor, of scale, or both. The local dual Bregman length distortion factor bounds this change.

Definition 1 (Local Dual Bregman Length Distortion Factor (Dual BLDF)).

Let $\Omega\subset\operatorname{int}(\dom d^{*})$ , $s>0$ , and $\gamma>0$ . The dual BLDF of $d^{*}$ on $\Omega$ is

c_{d^{*}}(s,\Omega;\gamma):=\sup_{u_{1},u_{2},v\neq 0}\left(\frac{D_{d^{*}}(u_{2}+sv,u_{2})}{s^{\gamma}D_{d^{*}}(u_{1}+v,u_{1})}\right)^{1/2}\quad\text{s.t.}\quad\begin{cases}u_{1}+[0,1]v\subset\Omega,\\ u_{2}+[0,s]v\subset\Omega.\end{cases}

The numerator uses the displacement $sv$ at anchor $u_{2}$ , while the denominator uses $v$ at anchor $u_{1}$ . At $\gamma=2$ , the factor $s^{2}$ removes the quadratic scaling, leaving the change due to the geometry. Our convergence analysis uses $\gamma=2$ ; other exponents serve only the comparison with triangle scaling (section E.2). If $d^{*}$ is $L_{d^{*}}$ -smooth and $\mu_{d^{*}}$ -strongly convex on $\Omega$ , then $c_{d^{*}}(s,\Omega;2)\leq\sqrt{L_{d^{*}}/\mu_{d^{*}}}$ (lemma 5), so our rate then matches that of Auslender and Teboulle [2] up to a constant. If $d^{*}$ is twice differentiable, the dual BLDF is bounded by the variation of the dual local norm over $\Omega$ (proposition 1).

Definition 2 (Dual norm distortion factor).

Let $\Omega\subset\operatorname{int}(\dom d^{*})$ , and assume that $d^{*}$ is twice differentiable on $\Omega$ . The dual norm distortion factor of $d^{*}$ on $\Omega$ is

\rho_{d^{*}}(\Omega):=\sup_{w_{1},w_{2}\in\Omega}\;\sup_{\xi\neq 0}\left(\frac{\langle\xi,\nabla^{2}d^{*}(w_{2})\,\xi\rangle}{\langle\xi,\nabla^{2}d^{*}(w_{1})\,\xi\rangle}\right)^{1/2}=\underbrace{\sup_{w_{1},w_{2}\in\Omega}\max_{1\leq i\leq n}\left(\frac{(d_{i}^{*})^{\prime\prime}((w_{2})_{i})}{(d_{i}^{*})^{\prime\prime}((w_{1})_{i})}\right)^{1/2}}_{\text{if $d^{*}$ is separable (\lx@cref{creftype~refnum}{cor:rho_separable})}}.

The factor $\rho_{d^{*}}(\Omega)$ measures the largest change in local norm across the region, after taking the supremum over directions. It has no scaling parameter and bounds the dual BLDF at $\gamma=2$ .

Proposition 1.

Assume that $d^{*}$ is twice differentiable on $\Omega\subset\operatorname{int}(\dom d^{*})$ . Then, for every $s>0$ ,

c_{d^{*}}(s,\Omega;2)\leq\rho_{d^{*}}(\Omega).

If $\Omega$ is open and $\nabla^{2}d^{*}$ is continuous and positive definite on $\Omega$ , then $c_{d^{*}}(s,\Omega;2)=\rho_{d^{*}}(\Omega)$ for every $s>0$ . More generally, $c_{d^{*}}(s,\Omega;\gamma)=s^{1-\gamma/2}\rho_{d^{*}}(\Omega)$ for every $s>0$ and $\gamma>0$ (proposition 4).

The rates of section 5 use $\rho_{d^{*}}$ . Proposition 1 gives simple expressions for common geometries (table 2). For instance, for the Burg entropy, $\rho_{d^{*}}(\nabla d(\mathcal{X}))\leq b/a$ whenever $a\leq x\leq b$ for all $x\in\mathcal{X}$ .

5 Accelerated Rates under Bounded Distortion

The accepted-parameter bound of Theorem 2 becomes an accelerated rate when the accepted $M_{k}$ remain bounded. We now bound them using the local distortion $\rho_{k}$ of each iteration.

Definition 3 (Local distortion).

Let $\Omega_{k}$ be the convex hull of the dual segments of all trials of the $M_{k}$ -search in iteration $k$ of algorithm 2. The local distortion of iteration $k$ is $\rho_{k}:=\rho_{d^{*}}(\Omega_{k})$ .

5.1 Convergence and iteration complexity

Using the dual BLDF, propositions 5 and 6 show that every trial with $M_{k}\geq\rho_{k}^{2}L$ ( $\mu=0$ ), or $M_{k}\geq\rho_{k}^{4}L$ ( $\mu>0$ ), passes the exit test of algorithm 2. Hence, if $\rho_{k}\leq\rho$ from some iteration $K$ on, backtracking eventually keeps $M_{k}$ below $4\rho^{2}L$ , respectively $4\rho^{4}L$ .

Theorem 3 (Accelerated rate under bounded distortion).

Under Assumption 1, assume $d^{*}$ is twice differentiable, and let $\rho_{k}$ be as in definition 3. If there exist $K\geq 0$ and $\rho$ such that the inner loop of algorithm 2 exits at every iteration $k<K$ and $\rho_{k}\leq\rho$ for all $k\geq K$ , then it exits at every iteration, and there is a finite $\bar{K}\geq K$ such that, for all $k\geq\bar{K}$ , we have $M_{k}\leq 4\rho^{2}L$ ( $\mu=0$ ), respectively $M_{k}\leq 4\rho^{4}L$ ( $\mu>0$ ). By Theorem 2, this implies $\phi_{k}^{\mathrm{best}}-\phi_{\star}\leq\eta_{k}^{-1}D_{d}(x_{\star},x_{0})$ , with

\eta_{k}^{-1}\leq\frac{\eta_{\bar{K}}^{-1}}{\big(1+\frac{k-\bar{K}}{4\rho\sqrt{L\eta_{\bar{K}}}}\big)^{2}}\;{\color[rgb]{0.5,0.5,0.5}(\mu=0)},\qquad\eta_{k}^{-1}\leq\eta_{\bar{K}}^{-1}\Big(1-\sqrt{\tfrac{\mu}{4\rho^{4}L}}\Big)^{k-\bar{K}}\;{\color[rgb]{0.5,0.5,0.5}(\mu>0)}.

Corollary 1 (Iteration complexity).

Under Theorem 3, $\phi_{k}^{\mathrm{best}}-\phi_{\star}\leq\varepsilon$ after $N$ iterations from $\bar{K}$ :

N=O\!\left(\rho\sqrt{\frac{L\,D_{d}(x_{\star},x_{0})}{\varepsilon}}\right)\quad(\mu=0),\qquad N=O\!\left(\rho^{2}\sqrt{\frac{L}{\mu}}\,\log\frac{\eta_{\bar{K}}^{-1}D_{d}(x_{\star},x_{0})}{\varepsilon}\right)\quad(\mu>0).

When $\rho=1$ , these bounds match the iteration complexities of Nesterov’s fast gradient method up to an absolute constant; appendix D describes the Euclidean specialization.

Oracle complexity.

Backtracking incurs constant amortized oracle overhead per accepted iteration, up to logarithmic terms for underestimated initial parameters. The complexity above therefore translates into an oracle complexity bound with these additional terms; see section C.4.

5.2 Comparison with existing methods

Inputs.

The rates of Theorem 3 depend on $\rho$ , but ABrA-GD uses neither $\rho$ nor $\Omega_{k}$ : it finds $M_{k}$ by backtracking on the exit condition, and the only problem constant it takes as input is $\mu$ . The accelerated linear rate of Chen et al. [10] requires the constant of their condition as input. To our knowledge, Theorem 3 gives the first accelerated linear rate for relatively smooth and relatively strongly convex objectives obtained by a method with these inputs. Table 1 summarizes the comparison.

Triangle scaling and a posteriori bounds.

ABPG converges at rate $O(k^{-\gamma})$ under a triangle-scaling property with exponent $\gamma$ and gain $1$ [13, 24]. For standard non-Euclidean kernels, this exponent is at most $1$ : $\gamma=1$ for jointly convex divergences, and $\gamma\leq 1/2$ for the Burg kernel (appendix E). This guarantee therefore gives no acceleration, and some additional condition is necessary, since $O(1/k)$ is optimal for the full relatively smooth class [12]. ABrA-GD shares this limit: if $M_{k}$ grows without bound, $y_{k}$ approaches $x_{k}^{\mathrm{best}}$ , the iterations approach Bregman gradient steps, and the safeguard of appendix G retains the $O(1/k)$ rate.

The accelerated variants instead fix $\gamma=2$ and backtrack on the gain, so their rate $O(\bar{G}_{k}/k^{2})$ depends on the accepted gains and holds a posteriori, as in Theorem 2. Bounded distortion is implied by a uniform gain: if triangle scaling holds with exponent $2$ and a gain $G$ uniform over triples, then $\rho_{d^{*}}^{2}\leq G$ on the corresponding dual region (section E.2). Hence, whenever ABPG has a uniform gain, Theorem 3 also gives ABrA-GD an accelerated rate.

Method	Smooth. adaptive	Geometry adaptive	Convex rate	Acc. $\mu>0$
BPG / BPG-LS	✓ (BPG-LS)	–	$O(1/k)$	–
ABPG	–	–	$O(k^{-\gamma})$ , global $\gamma$	–
ABPG-e	–	✓ (exponent search)	$O(k^{-\gamma_{k}})$	–
ABPG-g	–	✓ (gain search)	$O(\bar{G}_{k}/k^{2})$	–
Acc-MD (GCS)	–	–	$O(C/k^{2})$	✓
ABrA-GD	✓	✓ (via $M_{k}$ )	$O\!\left((\sum_{i}M_{i}^{-1/2})^{-2}\right)$	✓

Table 1: Comparison in the relatively smooth setting. BPG-LS denotes Bregman proximal gradient (-LS: with line search). ABPG-e uses an exponent search while ABPG-g uses a gain search. Acc-MD [10] takes the constant

C

of its generalized Cauchy–Schwarz condition as input.

6 Numerical Experiments

Figure 4: D-optimal design on the housing dataset. Panels show the objective residual

F-F_{\star}

against iterations and oracle calls. Methods are BPG, BPG with line search (BPG-LS), ABPG, its exponent-adaptive and gain-adaptive variants (ABPG-e and ABPG-g), and ABrA-GD. ABPG and ABPG-g use

\gamma=2

. Top: convex case (

\mu=0

), where ABrA-GD performs similarly to the best ABPG variants. Bottom: relatively strongly convex case (

\mu=10^{-4}

), where ABrA-GD reaches a lower residual and its log-residual decreases approximately linearly. Additional datasets and accepted parameters are reported in figs. 6 and 7; the

\gamma=1

comparison appears in fig. 5.

We evaluate ABrA-GD on D-optimal design and Poisson linear inverse problems. The formulations, reference functions, datasets, baselines, and initialization follow Hanzely et al. [13]. Further protocol details are given in section G.1, and additional comparisons appear in appendix G. Oracle calls count first-order oracle calls, including those of trials rejected by backtracking.

For D-optimal design, we use $\gamma=2$ in the ABPG comparisons, following the experimental choices of Hanzely et al. [13]. With $\gamma=1$ , ABPG has only an $O(1/k)$ guarantee and converged more slowly than BPG in their experiments; fig. 5 reproduces this comparison.

Figure 4 shows the housing dataset. In the convex case, ABrA-GD improves on BPG and BPG-LS and performs similarly to the best ABPG variants over the plotted range. With $\mu=10^{-4}$ , its log-residual decreases approximately linearly, and it reaches a lower residual with fewer oracle calls than the displayed baselines. The other three D-optimal design datasets show the same pattern (appendix G). Figures 6 and 7 also plot the accepted $M_{k}$ values. The D-optimal design instances are scaled so that $L=1$ [13], and on these instances $L_{\mathrm{cur}}$ stays between $1/4$ and $2$ .

On the Poisson problems, ABrA-GD improves on the nonaccelerated baselines. Its performance relative to the accelerated methods depends on the instance and variant. We report Poisson L1 with $(m,n)=(200,100)$ and Poisson L2 with $(m,n)=(100,1000)$ in appendix G.

7 Conclusion

ABrA-GD combines a Bregman gradient step with a dual lower model to satisfy an envelope descent requirement. Its convergence bound depends on the acceleration parameters accepted during the run. The dual BLDF measures the local geometric distortion in the comparison between the two steps, but the algorithm does not need to evaluate it.

Under bounded distortion, the analysis gives an accelerated $O(1/k^{2})$ rate for convex objectives and an accelerated linear rate for relatively strongly convex objectives. In the D-optimal design experiments, ABrA-GD performs similarly to adaptive ABPG in the convex case and reaches lower residuals in the relatively strongly convex case. Performance on the Poisson problems depends on the instance.

Future directions include sharper local distortion bounds, an accelerated analysis for composite objectives (appendix F), and stochastic variants.

References

[1] Zeyuan Allen-Zhu and Lorenzo Orecchia. Linear coupling: An ultimate unification of gradient and mirror descent. arXiv preprint arXiv:1407.1537, 2014.
[2] Alfred Auslender and Marc Teboulle. Interior gradient and proximal methods for convex and conic optimization. SIAM Journal on Optimization, 16(3):697–725, 2006.
[3] Francis Bach. Duality between subgradient and conditional gradient methods. SIAM Journal on Optimization, 25(1):115–129, 2015.
[4] Heinz H. Bauschke, Jérôme Bolte, and Marc Teboulle. A descent lemma beyond lipschitz gradient continuity: first-order methods revisited and applications. Mathematics of Operations Research, 42(2):330–348, 2017.
[5] Heinz H Bauschke, Minh N Dao, and Scott B Lindstrom. Regularizing with bregman–moreau envelopes. SIAM Journal on Optimization, 28(4):3208–3228, 2018.
[6] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200–217, 1967.
[7] Yair Censor and Arnold Lent. An iterative row-action method for interval convex programming. Journal of Optimization theory and Applications, 34(3):321–353, 1981.
[8] Yair Censor and Stavros Andrea Zenios. Proximal minimization algorithm with d-functions. Journal of Optimization Theory and Applications, 73(3):451–464, 1992.
[9] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):1–27, 2011.
[10] Long Chen, Hao Luo, Jingrong Wei, Zeyi Xu, and Yuan Yao. Accelerated mirror descent method through variable and operator splitting. arXiv preprint arXiv:2601.19038, 2026.
[11] Jelena Diakonikolas and Lorenzo Orecchia. The approximate duality gap technique: A unified theory of first-order methods. SIAM Journal on Optimization, 29(1):660–689, 2019.
[12] Radu-Alexandru Dragomir, Adrien B Taylor, Alexandre d’Aspremont, and Jérôme Bolte. Optimal complexity and certification of bregman first-order methods. Mathematical Programming, 194(1):41–83, 2022.
[13] Filip Hanzely, Peter Richtarik, and Lin Xiao. Accelerated bregman proximal gradient methods for relatively smooth convex optimization. Computational Optimization and Applications, 79(2):405–440, 2021.
[14] Sai Praneeth Karimireddy, Sebastian U. Stich, and Martin Jaggi. Global linear convergence of Newton’s method without strong-convexity or Lipschitz gradients. arXiv preprint arXiv:1806.00413, 2018.
[15] Claude Lemaréchal and Claudia Sagastizábal. Practical aspects of the moreau–yosida regularization: Theoretical preliminaries. SIAM journal on optimization, 7(2):367–385, 1997.
[16] Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. Advances in neural information processing systems, 28, 2015.
[17] Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. Catalyst acceleration for first-order convex optimization: from theory to practice. Journal of Machine Learning Research, 18(212):1–54, 2018.
[18] Jin-Zan Liu and Xin-Wei Liu. A dual bregman proximal gradient method for relatively-strongly convex optimization. Numerical Algebra, Control and Optimization, 12(4):679–692, 2022. doi: 10.3934/naco.2021028.
[19] Haihao Lu, Robert M Freund, and Yurii Nesterov. Relatively smooth convex optimization by first-order methods, and applications. SIAM Journal on Optimization, 28(1):333–354, 2018.
[20] Renato DC Monteiro and Benar Fux Svaiter. An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods. SIAM Journal on Optimization, 23(2):1092–1125, 2013.
[21] Yurii Nesterov. Introductory lectures on convex optimization. Springer, 2004.
[22] Yurii Nesterov and Boris T Polyak. Cubic regularization of Newton method and its global performance. Mathematical Programming, 108(1):177–205, 2006.
[23] R Tyrrell Rockafellar. Convex analysis, volume 28. Princeton university press, 1997.
[24] OS Savchuk, MS Alkousa, AS Shushko, AA Vyguzov, FS Stonyakin, DA Pasechnyuk, and AV Gasnikov. Accelerated bregman gradient methods for relatively smooth and relatively lipschitz continuous minimization problems. arXiv preprint arXiv:2411.16743, 2024.
[25] Paul Tseng. On accelerated proximal gradient methods for convex-concave optimization. submitted to SIAM Journal on Optimization, 2(3), 2008.
[26] Ashia C Wilson, Ben Recht, and Michael I Jordan. A lyapunov analysis of accelerated methods in optimization. Journal of Machine Learning Research, 22(113):1–34, 2021.

Appendix A Notations and definitions

Definition 4 (Legendre function).

A function $d$ is Legendre if it is proper, closed and convex, and is both essentially smooth and essentially strictly convex: it is differentiable on $\operatorname{int}(\operatorname{dom}d)$ with $\|\nabla d(x_{k})\|\to\infty$ whenever $x_{k}\to\partial\operatorname{dom}d$ , and it is strictly convex on every convex subset of $\operatorname{dom}\partial d$ . Essential strict convexity is what makes $\nabla d$ injective. In this case, $\nabla d$ is a bijection between $\operatorname{int}(\operatorname{dom}d)$ and $\operatorname{int}(\operatorname{dom}d^{*})$ with inverse $\nabla d^{*}$ .

Definition 5 (Bregman Divergence [6]).

The Bregman divergence of a Legendre function $d$ is

D_{d}(x,y):=d(x)-d(y)-\langle\nabla d(y),\,x-y\rangle,\qquad x\in\operatorname{dom}d,\;y\in\operatorname{int}(\mathcal{X}).

Definition 6 (Relative smoothness and strong convexity).

We say that $f$ is $\mu$ -strongly convex (convex if $\mu=0$ ) and $L$ -smooth relative to a Legendre function $d$ if there exists $0\leq\mu\leq L$ s.t.

\mu D_{d}(x,y)\leq f(x)-f(y)-\langle\nabla f(y),x-y\rangle\leq LD_{d}(x,y)\qquad\forall x,y\in\operatorname{int}(\mathcal{X}).

Definition 7 (Fenchel–Legendre conjugate [23, §12]).

Let $f$ be a proper function. Its conjugate is

f^{*}(\lambda):=\sup_{x\in\mathbb{R}^{n}}\{\langle\lambda,x\rangle-f(x)\}.

Definition 8 (Primal and dual local norms).

Let $d$ be a twice differentiable Legendre function whose Hessian is positive definite on $\operatorname{int}(\dom d)$ . For $x\in\operatorname{int}(\dom d)$ , $w\in\operatorname{int}(\dom d^{*})$ , $v\in\mathcal{X}$ , and $s\in\mathcal{X}^{*}$ , define

\|v\|_{x}^{2}:=\tfrac{1}{2}\langle\nabla^{2}d(x)\,v,\,v\rangle,\qquad\|s\|_{w,*}^{2}:=\tfrac{1}{2}\langle s,\,\nabla^{2}d^{*}(w)\,s\rangle.

Positive definiteness is a hypothesis, not a consequence of the rest: $d(x)=x^{4}/4$ is Legendre and twice differentiable on $\mathbb{R}$ with $d^{\prime\prime}(0)=0$ , so the first form is degenerate at the origin and $d^{*}$ has no finite second derivative there. Under the hypothesis, $\nabla^{2}d^{*}(w)=\nabla^{2}d(\nabla d^{*}(w))^{-1}$ is positive definite as well and both forms are norms. Every statement below that uses local norms or ratios of Hessians carries this hypothesis on the region where it is applied.

The Bregman divergence admits an integral representation in terms of the local norms.

Proposition 2 (Bregman divergence and local norms).

Let $d$ be a Legendre function of class $C^{2}$ with positive definite Hessian on $\operatorname{int}(\dom d)$ . For any $x_{1},x_{2}\in\operatorname{int}(\dom d)$ , there is a point $z$ on the segment $[x_{1},x_{2}]$ such that

D_{d}(x_{1},x_{2})=\|x_{1}-x_{2}\|_{z}^{2}.

Proof.

Since $\operatorname{int}(\dom d)$ is convex, the segment $[x_{1},x_{2}]$ lies in it, and $g(t):=d\big(x_{2}+t(x_{1}-x_{2})\big)$ is twice continuously differentiable on a neighbourhood of $[0,1]$ . Taylor’s theorem with the Lagrange remainder gives a $\theta\in(0,1)$ with $g(1)=g(0)+g^{\prime}(0)+\tfrac{1}{2}g^{\prime\prime}(\theta)$ , that is, writing $z:=x_{2}+\theta(x_{1}-x_{2})$ ,

d(x_{1})-d(x_{2})-\langle\nabla d(x_{2}),\,x_{1}-x_{2}\rangle=\tfrac{1}{2}\big\langle\nabla^{2}d(z)(x_{1}-x_{2}),\,x_{1}-x_{2}\big\rangle.

The left-hand side is $D_{d}(x_{1},x_{2})$ and the right-hand side is $\|x_{1}-x_{2}\|_{z}^{2}$ by definition 8. ∎

A.1 The anchored decomposition

For a differentiable $g$ , write

D_{g}(x,y):=g(x)-g(y)-\langle\nabla g(y),x-y\rangle,

which is the Bregman divergence already defined for $d$ . In this notation, $g$ is $L$ -smooth relative to $d$ when $D_{g}\leq LD_{d}$ and $\mu$ -strongly convex relative to $d$ when $D_{g}\geq\mu D_{d}$ , both on $\operatorname{int}(\dom d)$ .

The anchor $x\mapsto\mu D_{d}(x,x_{0})$ differs from $\mu d$ by an affine function, and an affine function has zero Bregman divergence. So the anchor contributes exactly $\mu D_{d}$ , and $\phi=f+\mu D_{d}(\cdot,x_{0})$ satisfies

D_{\phi}(x,y)=D_{f}(x,y)+\mu D_{d}(x,y),\qquad x,y\in\operatorname{int}(\dom d).

(18)

Let $\phi$ be $L$ -smooth and $\mu$ -strongly convex relative to $d$ , that is $\mu D_{d}\leq D_{\phi}\leq LD_{d}$ . Strong convexity says precisely that $\phi-\mu D_{d}(\cdot,x_{0})$ is convex, so put

f:=\phi-\mu D_{d}(\cdot,x_{0}).

Subtracting $\mu D_{d}$ throughout eq. 18 gives $0\leq D_{f}\leq(L-\mu)D_{d}$ : the function $f$ is convex and $(L-\mu)$ -smooth relative to $d$ , which is Assumption 1, and $\phi$ is back in the form eq. 1.

Conversely, if $f$ satisfies Assumption 1 then $0\leq D_{f}\leq(L-\mu)D_{d}$ , and eq. 18 returns $\mu D_{d}\leq D_{\phi}\leq LD_{d}$ .

The two hypotheses are therefore the same one. All of the relative strong convexity of $\phi$ sits in the anchor term, and the anchor is what raises the relative smoothness constant from $L-\mu$ to $L$ .

A.2 Affine equality constraints

The analysis also applies when the feasible set includes affine equality constraints. Let $\mathcal{A}\subseteq\mathbb{R}^{n}$ be an affine subspace, assume that $x_{0}\in\mathcal{A}\cap\operatorname{int}(\dom d)$ , and consider the feasible set

\mathcal{X}=\operatorname{cl}(\dom d)\cap\mathcal{A}.

We can eliminate the equality constraints by choosing coordinates on $\mathcal{A}$ .

Let the columns of $U$ form an orthonormal basis for the direction space of $\mathcal{A}$ . Every $x\in\mathcal{A}$ has a unique representation $x=x_{0}+Uu$ . Define

\widetilde{f}(u)=f(x_{0}+Uu),\qquad\widetilde{d}(u)=d(x_{0}+Uu).

The Bregman divergence is preserved under this change of variables:

D_{\widetilde{d}}(u,v)=D_{d}(x_{0}+Uu,x_{0}+Uv).

The constrained problem therefore becomes

\min_{u\in\operatorname{cl}(\dom\widetilde{d})}\widetilde{f}(u)+\mu D_{\widetilde{d}}(u,0).

The relative smoothness and strong convexity inequalities carry over directly, with the same constants $L$ and $\mu$ .

The restricted kernel $\widetilde{d}$ is Legendre. To see this, write $V=\operatorname{range}(U)$ . Since $\mathcal{A}$ intersects $\operatorname{int}(\dom d)$ , the subdifferential sum rule gives

\partial(d+\delta_{\mathcal{A}})(x)=\partial d(x)+V^{\perp}

[23, Thm. 23.8]. At a boundary point of the restricted domain, relative to $\mathcal{A}$ , the right-hand side is empty because $d$ is Legendre. This gives essential smoothness in the reduced coordinates. Essential strict convexity is inherited from $d$ .

Thus the problem in the variable $u$ satisfies Assumption 1. All updates and proofs apply in these coordinates, so affine equality constraints require no change to the analysis.

Appendix B Recovering Bregman Gradient Descent and Dual Averaging

The implicit envelope of section 2 specializes to the two classical methods. Both derivations are given here: they motivate the accelerated step of algorithm 2 but are not needed to read it.

B.1 Mirror Gradient Descent

Since $f$ is relatively smooth, for all $x_{k}$ , we have the following upper model $\mathcal{U}(x,x_{k})$ for $f$ ,

f(x)\leq\mathcal{U}(x,x_{k}):=f(x_{k})+\langle\nabla f(x_{k}),x-x_{k}\rangle+(L-\mu)D_{d}(x,x_{k}).

By definition of $\phi$ , it is sufficient to find an iterate $x_{k+1}$ and a stepsize $\eta_{k+1}$ such that

\mathcal{U}(x_{k+1},x_{k})+\mu D_{d}(x_{k+1},x_{0})\leq\phi_{\star}^{\eta_{k+1}}

Naturally, we can take the minimum of the upper bound and adjust $\eta_{k+1}$ accordingly,

x_{k+1}=\argmin_{x}\mathcal{U}(x,x_{k})+\mu D_{d}(x,x_{0})=\nabla d^{*}\Big(\nabla d(x_{k})-\frac{1}{L}\nabla\phi(x_{k})\Big).

This is exactly gradient descent, and it fits algorithm 1 with a certain schedule for $\eta_{k}$ .

Proposition 3.

The Bregman gradient descent iterates satisfy algorithm 1 with

\eta_{k}^{-1}=L\min\left\{\frac{1}{k},\frac{\kappa}{(1+\kappa)^{k}-1}\right\},\qquad\kappa=\frac{\mu}{L},

where the second term is omitted when $\mu=0$ .

The two branches are the known rates of the Bregman gradient method: the $O(1/k)$ rate under relative smoothness [4, 19], and the linear rate under relative strong convexity [19]. What the proposition adds is the translation into the envelope schedule, that is, the $\eta_{k}$ for which the method satisfies the requirement of algorithm 1.

B.2 Saddle formulation of the envelope

Define

S^{\eta}(x,\lambda):=\langle x,\lambda\rangle-f^{*}(\lambda)+\left(\mu+\frac{1}{\eta}\right)D_{d}(x,x_{0}).

(19)

The Fenchel representation of $f$ and the definition of $\mathcal{D}^{\eta}$ give

\phi^{\eta}(x)=\sup_{\lambda}S^{\eta}(x,\lambda),\qquad\mathcal{D}^{\eta}(\lambda)=\min_{x}S^{\eta}(x,\lambda).

Weak duality therefore yields

\sup_{\lambda}\mathcal{D}^{\eta}(\lambda)\leq\min_{x}\sup_{\lambda}S^{\eta}(x,\lambda)=\phi_{\star}^{\eta}.

For $\lambda=\nabla f(y)$ , Fenchel equality gives

S^{\eta}(x,\nabla f(y))=f(y)+\langle\nabla f(y),x-y\rangle+\left(\mu+\frac{1}{\eta}\right)D_{d}(x,x_{0}),

With $\eta^{-1}=0$ , this is the second term of the lower model $\mathcal{L}_{k+1}$ of eq. 15, at $y=y_{k}$ .

This formulation is helpful for proving the convergence rate of Dual Averaging, and therefore, AbrA-GD.

B.3 Dual Averaging

Even when $S^{\eta}(x,\lambda)$ cannot be evaluated explicitly, we can still update the dual variable through the best-response (BR) maps

	$\displaystyle z^{\eta}(\lambda)$	$\displaystyle\in\arg\min_{x}S^{\eta}(x,\lambda)=\nabla d^{*}\!\left(\nabla d(x_{0})-\frac{1}{\mu+\eta^{-1}}\lambda\right),$		(Primal BR)
	$\displaystyle\lambda(x)$	$\displaystyle\in\arg\max_{\lambda}S^{\eta}(x,\lambda)=\nabla f(x).$		(Dual BR)

The dual-averaging update is the damped best-response-to-best-response iteration

	$\displaystyle\lambda_{k+1}$	$\displaystyle=(1-t_{k})\lambda_{k}+t_{k}\lambda(z_{k})=(1-t_{k})\lambda_{k}+t_{k}\nabla f(z_{k}),$
	$\displaystyle z_{k+1}$	$\displaystyle=z^{\eta_{k+1}}(\lambda_{k+1}).$		(Dual Avg.)

This can be viewed as a damped alternating $\min$ - $\max$ scheme. Its main advantage is that it avoids computing $\nabla f^{*}(\lambda)$ , which is usually intractable.

Theorem 4.

The update eq. Dual Avg., initialized with $\eta_{0}=0$ , $\lambda_{0}=0$ and $z_{0}=x_{0}$ , and using

t_{k}=\frac{\mu+\eta_{k}^{-1}}{L+\eta_{k}^{-1}},\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1},

satisfies algorithm 1 with

x_{k}\in\argmin_{1\leq i\leq k}\phi(z_{i}),\quad\eta_{k}^{-1}\leq L\min\left\{\frac{1}{k},\frac{\kappa}{(1+\kappa)^{k}-1}\right\},\qquad\kappa=\frac{\mu}{L},

where the second term is omitted when $\mu=0$ .

Appendix C Technical Lemmas

Lemma 1 (Three-point property of Tseng [25]).

Let $g:\mathcal{X}\to\mathbb{R}\cup\{+\infty\}$ be a convex function, and let $D_{d}(\cdot,\cdot)$ be the Bregman divergence induced by $d$ . For a given $z\in\mathcal{X}$ , define

z_{+}\;:=\;\arg\min_{x\in\mathcal{X}}\big\{g(x)+D_{d}(x,z)\big\}.

Then, for all $x\in\mathcal{X}$ ,

g(x)+D_{d}(x,z)\;\geq\;g(z_{+})+D_{d}(z_{+},z)+D_{d}(x,z_{+}).

Note that when the minimization is unconstrained and $g$ is affine, the inequality holds with equality,

g(x)+D_{d}(x,z)\;=\;g(z_{+})+D_{d}(z_{+},z)+D_{d}(x,z_{+}).

(20)

Lemma 2 (Growth around the minimizer).

Let $g$ be differentiable and $\alpha$ -strongly convex relative to $d$ , and let $z:=\arg\min_{x}g(x)$ lie in the interior of the domain of $d$ . Then, for all $x$ ,

g(x)\;\geq\;g(z)+\alpha D_{d}(x,z).

If $g-\alpha d$ is affine, the inequality holds with equality.

Proof.

The function $h:=g-\alpha d$ is convex, so $h(x)\geq h(z)+\langle\nabla h(z),x-z\rangle$ , with equality when $h$ is affine. Since $z$ is an interior minimizer of $g$ , $\nabla g(z)=0$ and $\nabla h(z)=-\alpha\nabla d(z)$ . Adding $\alpha d(x)=\alpha d(z)+\alpha\langle\nabla d(z),x-z\rangle+\alpha D_{d}(x,z)$ gives the result. ∎

The following generic lemma is important in the proof of Dual Averaging and Accelerated Gradient Descent.

Lemma 3 (Dual lower-bound recursion).

Under Assumption 1, so that $\phi$ is $L$ -smooth and $\mu$ -strongly convex relative to $d$ , assume that, for some $b_{k}$ ,

b_{k}\leq\mathcal{D}^{\eta_{k}}(\lambda_{k}),\qquad\lambda_{k+1}=(1-t_{k})\lambda_{k}+t_{k}\nabla f(y_{k}),\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1},

and let

z_{k}\in\arg\min_{x}S^{\eta_{k}}(x,\lambda_{k}),\qquad z_{k+1}\in\arg\min_{x}S^{\eta_{k+1}}(x,\lambda_{k+1}),

where $S^{\eta}$ is the saddle point problem defined in eq. 19. Define

\phi_{k}^{\mathrm{low}}:=\phi(y_{k})+\langle\nabla\phi(y_{k}),z_{k}-y_{k}\rangle+\mu D_{d}(z_{k},y_{k}).

Then

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})\geq(1-t_{k})b_{k}+t_{k}\phi_{k}^{\mathrm{low}}-\left(\mu+\eta_{k+1}^{-1}\right)D_{d}(z_{k},z_{k+1}).

If $t_{k}=1$ , the same conclusion holds with the term $(1-t_{k})b_{k}$ omitted, for any $\eta_{k+1}>0$ , and without the hypothesis $b_{k}\leq\mathcal{D}^{\eta_{k}}(\lambda_{k})$ or the recursion $\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}$ :

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})\geq\phi_{k}^{\mathrm{low}}-\left(\mu+\eta_{k+1}^{-1}\right)D_{d}(z_{k},z_{k+1}).

Proof.

By concavity of $S^{\eta}(x,\cdot)$ , evaluated at $x=z_{k+1}$ ,

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})=S^{\eta_{k+1}}(z_{k+1},\lambda_{k+1})\geq(1-t_{k})S^{\eta_{k+1}}(z_{k+1},\lambda_{k})+t_{k}S^{\eta_{k+1}}(z_{k+1},\nabla f(y_{k})).

Using

S^{\eta_{k+1}}(x,\lambda_{k})=S^{\eta_{k}}(x,\lambda_{k})+(\eta_{k+1}^{-1}-\eta_{k}^{-1})D_{d}(x,x_{0}),

S^{\eta_{k}}(x,\lambda_{k})\geq b_{k}+(\mu+\eta_{k}^{-1})D_{d}(x,z_{k}),

where the last inequality follows directly from the optimality of $z_{k}$ . Indeed, with $\alpha_{k}:=\mu+\eta_{k}^{-1}$ ,

	$\displaystyle S^{\eta_{k}}(x,\lambda_{k})-S^{\eta_{k}}(z_{k},\lambda_{k})$	$\displaystyle=\left\langle\lambda_{k}+\alpha_{k}\bigl(\nabla d(z_{k})-\nabla d(x_{0})\bigr),x-z_{k}\right\rangle+\alpha_{k}D_{d}(x,z_{k})$
		$\displaystyle\geq\alpha_{k}D_{d}(x,z_{k}),$

and $S^{\eta_{k}}(z_{k},\lambda_{k})=\mathcal{D}^{\eta_{k}}(\lambda_{k})\geq b_{k}$ . This is the only use of the hypothesis on $b_{k}$ , and it enters the conclusion through the factor $(1-t_{k})$ . Finally, since $\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}$ , the coefficients of $D_{d}(x,x_{0})$ sum to

(1-t_{k})(\eta_{k+1}^{-1}-\eta_{k}^{-1})+t_{k}(\mu+\eta_{k+1}^{-1})=t_{k}\mu.

The identity $D_{d}(x,x_{0})=D_{d}(y_{k},x_{0})+\langle\nabla d(y_{k})-\nabla d(x_{0}),x-y_{k}\rangle+D_{d}(x,y_{k})$ turns $t_{k}\big(f(y_{k})+\langle\nabla f(y_{k}),x-y_{k}\rangle+\mu D_{d}(x,x_{0})\big)$ into $t_{k}\big(\phi(y_{k})+\langle\nabla\phi(y_{k}),x-y_{k}\rangle+\mu D_{d}(x,y_{k})\big)$ . Hence

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})\geq(1-t_{k})b_{k}+t_{k}\phi(y_{k})+H_{k}(z_{k+1}),

where

H_{k}(x):=(1-t_{k})(\mu+\eta_{k}^{-1})D_{d}(x,z_{k})+t_{k}\mu D_{d}(x,y_{k})+t_{k}\langle\nabla\phi(y_{k}),x-y_{k}\rangle.

We claim that $z_{k+1}\in\arg\min_{x}H_{k}(x)$ . Indeed, using

\lambda_{k}+(\mu+\eta_{k}^{-1})(\nabla d(z_{k})-\nabla d(x_{0}))=0,

\lambda_{k+1}=(1-t_{k})\lambda_{k}+t_{k}\nabla f(y_{k}),\qquad\nabla f(y_{k})=\nabla\phi(y_{k})-\mu(\nabla d(y_{k})-\nabla d(x_{0})),

the optimality condition of

z_{k+1}\in\arg\min_{x}S^{\eta_{k+1}}(x,\lambda_{k+1})

is equivalent to

\displaystyle 0=(1-t_{k})(\mu+\eta_{k}^{-1})\bigl(\nabla d(z_{k+1})-\nabla d(z_{k})\bigr)+t_{k}\mu\bigl(\nabla d(z_{k+1})-\nabla d(y_{k})\bigr)+t_{k}\nabla\phi(y_{k}),

which is exactly the optimality condition for $z_{k+1}\in\arg\min_{x}H_{k}(x)$ .

Moreover, $H_{k}$ has Bregman curvature

(1-t_{k})(\mu+\eta_{k}^{-1})+t_{k}\mu=\mu+\eta_{k+1}^{-1}.

Therefore $H_{k}$ is an affine function plus a weighted sum of Bregman divergences in its first argument, with total weight $\mu+\eta_{k+1}^{-1}$ , and $z_{k+1}$ is its unconstrained minimizer. Equation 20 applies and gives, at $x=z_{k}$ ,

H_{k}(z_{k+1})=H_{k}(z_{k})-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1}).

The three-point property of lemma 1 would give this inequality in the opposite direction; here it holds as an equality because the minimization is unconstrained. Since

H_{k}(z_{k})=t_{k}\mu D_{d}(z_{k},y_{k})+t_{k}\langle\nabla\phi(y_{k}),z_{k}-y_{k}\rangle,

we obtain

	$\displaystyle\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})\geq$	$\displaystyle(1-t_{k})b_{k}+t_{k}\phi(y_{k})+t_{k}\langle\nabla\phi(y_{k}),z_{k}-y_{k}\rangle$
		$\displaystyle+t_{k}\mu D_{d}(z_{k},y_{k})-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1}).$

By definition,

\phi_{k}^{\mathrm{low}}=\phi(y_{k})+\langle\nabla\phi(y_{k}),z_{k}-y_{k}\rangle+\mu D_{d}(z_{k},y_{k}),

and therefore

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})\geq(1-t_{k})b_{k}+t_{k}\phi_{k}^{\mathrm{low}}-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1}).

For the case $t_{k}=1$ , argue directly. Then $\lambda_{k+1}=\nabla f(y_{k})$ , so the concavity step above is unnecessary and neither $\lambda_{k}$ nor $\eta_{k}$ appears. Writing $\alpha_{k+1}=\mu+\eta_{k+1}^{-1}$ and using that the conjugate is attained at $y_{k}$ ,

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})=S^{\eta_{k+1}}(z_{k+1},\nabla f(y_{k}))=f(y_{k})+\langle\nabla f(y_{k}),z_{k+1}-y_{k}\rangle+\alpha_{k+1}D_{d}(z_{k+1},x_{0}).

Substituting $\nabla f(y_{k})=\nabla\phi(y_{k})-\mu(\nabla d(y_{k})-\nabla d(x_{0}))$ and $f(y_{k})=\phi(y_{k})-\mu D_{d}(y_{k},x_{0})$ , and expanding $\mu D_{d}(z_{k+1},x_{0})$ by the three-point identity around $y_{k}$ , the terms anchored at $x_{0}$ combine into $\mu D_{d}(z_{k+1},y_{k})$ :

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})=\phi(y_{k})+\widetilde{H}_{k}(z_{k+1}),\qquad\widetilde{H}_{k}(x):=\eta_{k+1}^{-1}D_{d}(x,x_{0})+\mu D_{d}(x,y_{k})+\langle\nabla\phi(y_{k}),x-y_{k}\rangle.

The optimality condition for $z_{k+1}\in\arg\min_{x}S^{\eta_{k+1}}(x,\lambda_{k+1})$ is exactly the one for $z_{k+1}\in\arg\min_{x}\widetilde{H}_{k}(x)$ , and $\widetilde{H}_{k}$ is affine plus Bregman divergences of total weight $\eta_{k+1}^{-1}+\mu=\alpha_{k+1}$ , so eq. 20 at $x=z_{k}$ gives $\widetilde{H}_{k}(z_{k+1})=\widetilde{H}_{k}(z_{k})-\alpha_{k+1}D_{d}(z_{k},z_{k+1})$ . Since $\phi(y_{k})+\widetilde{H}_{k}(z_{k})=\phi_{k}^{\mathrm{low}}+\eta_{k+1}^{-1}D_{d}(z_{k},x_{0})$ and $D_{d}\geq 0$ ,

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})=\phi_{k}^{\mathrm{low}}+\eta_{k+1}^{-1}D_{d}(z_{k},x_{0})-\alpha_{k+1}D_{d}(z_{k},z_{k+1})\geq\phi_{k}^{\mathrm{low}}-\alpha_{k+1}D_{d}(z_{k},z_{k+1}).

At $k=0$ the initialization gives $z_{0}=x_{0}$ , so the middle term is zero and the bound holds with equality. ∎

C.1 Convergence rate of gradient descent and dual averaging

Proof of Theorem 1.

By construction, algorithm 1 produces $x_{k}$ satisfying $\phi(x_{k})\leq\phi_{\star}^{\eta_{k}}$ . Bounding the minimum in eq. 2 by its value at $x^{\star}$ ,

\phi_{\star}^{\eta_{k}}=\min_{x}\left\{\phi(x)+\frac{1}{\eta_{k}}D_{d}(x,x_{0})\right\}\;\leq\;\phi(x^{\star})+\frac{1}{\eta_{k}}D_{d}(x^{\star},x_{0})=\phi_{\star}+\frac{D_{d}(x_{\star},x_{0})}{\eta_{k}}.

Chaining the two inequalities gives eq. 3. ∎

Proof of proposition 3.

The optimality condition of the Bregman gradient step is

\nabla\phi(x_{k})+L\big(\nabla d(x_{k+1})-\nabla d(x_{k})\big)=0.

The objective of this step is $\langle\nabla f(x_{k}),x\rangle+LD_{d}(x,w_{k})$ up to a constant, with $\nabla d(w_{k})=(1-\kappa)\nabla d(x_{k})+\kappa\nabla d(x_{0})$ and $\kappa=\mu/L$ . By relative $(L-\mu)$ -smoothness of $f$ , convexity of $f$ , and eq. 20, we obtain

\phi(x_{k+1})\leq f(x)+\mu D_{d}(x,x_{0})+(L-\mu)D_{d}(x,x_{k})-LD_{d}(x,x_{k+1}).

(21)

Let

r:=\frac{L}{L-\mu}=\frac{1}{1-\kappa}.

Multiplying eq. 21 by $r^{k+1}$ and using

(L-\mu)r^{k+1}=Lr^{k},

gives

r^{k+1}\phi(x_{k+1})\leq r^{k+1}\bigl(f(x)+\mu D_{d}(x,x_{0})\bigr)+L\bigl(r^{k}D_{d}(x,x_{k})-r^{k+1}D_{d}(x,x_{k+1})\bigr).

Summing from $i=0$ to $k$ yields

\sum_{i=0}^{k}r^{i+1}\phi(x_{i+1})\leq\left(\sum_{i=0}^{k}r^{i+1}\right)\bigl(f(x)+\mu D_{d}(x,x_{0})\bigr)+L\bigl(D_{d}(x,x_{0})-r^{k+1}D_{d}(x,x_{k+1})\bigr).

Dropping the last negative term and using monotonicity of $\phi(x_{i})$ , we get

\phi(x_{k+1})\leq f(x)+\left(\mu+\frac{L}{\sum_{i=0}^{k}r^{i+1}}\right)D_{d}(x,x_{0}).

Therefore $x_{k+1}$ satisfies algorithm 1 with

\eta_{k+1}^{-1}=\frac{L}{\sum_{i=0}^{k}r^{i+1}}.

If $\mu=0$ , then $r=1$ , so

\eta_{k+1}^{-1}=\frac{L}{k+1}.

If $0<\mu<L$ , then

\eta_{k+1}^{-1}=L\frac{\kappa}{r^{k+1}-1}=L\frac{\kappa(1-\kappa)^{k+1}}{1-(1-\kappa)^{k+1}}.

Since $r=(1-\kappa)^{-1}\geq 1+\kappa$ , we have

\eta_{k+1}^{-1}\leq L\frac{\kappa}{(1+\kappa)^{k+1}-1}.

Thus the simpler valid rate is

\eta_{k+1}^{-1}\leq L\min\left\{\frac{1}{k+1},\frac{\kappa}{(1+\kappa)^{k+1}-1}\right\}.

∎

Lemma 4 (Dual-averaging one-step recursion).

Let

\mathcal{D}_{k}:=\mathcal{D}^{\eta_{k}}(\lambda_{k}),\qquad z_{k}=z^{\eta_{k}}(\lambda_{k}),

and assume

\lambda_{k+1}=(1-t_{k})\lambda_{k}+t_{k}\nabla f(z_{k}),\qquad z_{k+1}=z^{\eta_{k+1}}(\lambda_{k+1}).

Choose

t_{k}=\frac{\mu+\eta_{k}^{-1}}{L+\eta_{k}^{-1}},\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}.

Then

\mathcal{D}_{k+1}\geq(1-t_{k})\mathcal{D}_{k}+t_{k}\phi(z_{k+1}).

Moreover, if $t_{k}\in[0,1]$ , $x_{k+1}\in\argmin\{\phi(x_{k}),\phi(z_{k+1})\}$ , and

\phi(x_{k})\leq\mathcal{D}_{k},

then

\phi(x_{k+1})\leq\mathcal{D}_{k+1}.

Proof.

By concavity of $S^{\eta}(x,\cdot)$ , for every $x$ ,

S^{\eta_{k+1}}(x,\lambda_{k+1})\geq(1-t_{k})S^{\eta_{k+1}}(x,\lambda_{k})+t_{k}S^{\eta_{k+1}}(x,\nabla f(z_{k})).

Using

S^{\eta_{k+1}}(x,\lambda_{k})=S^{\eta_{k}}(x,\lambda_{k})+(\eta_{k+1}^{-1}-\eta_{k}^{-1})D_{d}(x,x_{0}),

and $z_{k}=z^{\eta_{k}}(\lambda_{k})$ , we have

S^{\eta_{k}}(x,\lambda_{k})\geq\mathcal{D}_{k}+(\mu+\eta_{k}^{-1})D_{d}(x,z_{k}).

By relative $(L-\mu)$ -smoothness of $f$ ,

f(z_{k})+\langle\nabla f(z_{k}),x-z_{k}\rangle\geq f(x)-(L-\mu)D_{d}(x,z_{k}),

hence

S^{\eta_{k+1}}(x,\nabla f(z_{k}))\geq\phi(x)+\eta_{k+1}^{-1}D_{d}(x,x_{0})-(L-\mu)D_{d}(x,z_{k}).

Combining gives

	$\displaystyle S^{\eta_{k+1}}(x,\lambda_{k+1})\geq$	$\displaystyle(1-t_{k})\mathcal{D}_{k}+t_{k}\phi(x)$
		$\displaystyle+\Big[(1-t_{k})(\mu+\eta_{k}^{-1})-t_{k}(L-\mu)\Big]D_{d}(x,z_{k})$
		$\displaystyle+\Big[\eta_{k+1}^{-1}-(1-t_{k})\eta_{k}^{-1}\Big]D_{d}(x,x_{0}).$

With the chosen $t_{k}$ and $\eta_{k+1}$ , both Bregman coefficients vanish, so

S^{\eta_{k+1}}(x,\lambda_{k+1})\geq(1-t_{k})\mathcal{D}_{k}+t_{k}\phi(x).

Taking $x=z_{k+1}=z^{\eta_{k+1}}(\lambda_{k+1})$ gives

\mathcal{D}_{k+1}\geq(1-t_{k})\mathcal{D}_{k}+t_{k}\phi(z_{k+1}).

Now assume $\phi(x_{k})\leq\mathcal{D}_{k}$ . Since

x_{k+1}\in\argmin\{\phi(x_{k}),\phi(z_{k+1})\},

and $t_{k}\in[0,1]$ ,

\phi(x_{k+1})\leq(1-t_{k})\phi(x_{k})+t_{k}\phi(z_{k+1})\leq(1-t_{k})\mathcal{D}_{k}+t_{k}\phi(z_{k+1})\leq\mathcal{D}_{k+1}.

∎

Proof of Theorem 4.

Let

\mathcal{D}_{k}:=\mathcal{D}^{\eta_{k}}(\lambda_{k}).

The updates in the statement are the hypotheses of lemma 4, whose dual step is taken at $z_{k}$ . Its remaining hypothesis $z_{k}=z^{\eta_{k}}(\lambda_{k})$ holds for $k\geq 1$ because that is how eq. Dual Avg. produced $z_{k}$ , and at $k=0$ because $\eta_{0}=0$ and $\lambda_{0}=0$ give $z^{\eta_{0}}(\lambda_{0})=x_{0}=z_{0}$ . It gives

\mathcal{D}_{k+1}\geq(1-t_{k})\mathcal{D}_{k}+t_{k}\phi(z_{k+1}).

With the limiting initialization $\eta_{0}=0$ , we have $t_{0}=1$ , so

\mathcal{D}_{1}\geq\phi(z_{1}).

Since $x_{1}\in\argmin_{1\leq i\leq 1}\phi(z_{i})$ , this gives

\phi(x_{1})\leq\mathcal{D}_{1}.

Assume now that

\phi(x_{k})\leq\mathcal{D}_{k}.

Because

x_{k+1}\in\argmin_{1\leq i\leq k+1}\phi(z_{i}),

we have

\phi(x_{k+1})\leq(1-t_{k})\phi(x_{k})+t_{k}\phi(z_{k+1}).

Thus

\phi(x_{k+1})\leq(1-t_{k})\mathcal{D}_{k}+t_{k}\phi(z_{k+1})\leq\mathcal{D}_{k+1}.

By induction,

\phi(x_{k})\leq\mathcal{D}_{k}\qquad\forall k\geq 1.

Finally,

\mathcal{D}_{k}=\min_{x}S^{\eta_{k}}(x,\lambda_{k})\leq\phi_{\star}^{\eta_{k}},

\phi(x_{k})\leq\phi_{\star}^{\eta_{k}}.

Thus the update satisfies algorithm 1.

It remains to estimate $\eta_{k}^{-1}$ . If $\mu=0$ , then

t_{k}=\frac{\eta_{k}^{-1}}{L+\eta_{k}^{-1}},\qquad\eta_{k+1}^{-1}=\frac{L\eta_{k}^{-1}}{L+\eta_{k}^{-1}}.

Equivalently,

\eta_{k+1}=\eta_{k}+\frac{1}{L}.

With $\eta_{0}=0$ , this gives

\eta_{k}^{-1}=\frac{L}{k}.

If $\mu>0$ , then, with $\kappa:=\mu/L$ ,

\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}=\frac{(L-\mu)\eta_{k}^{-1}}{L+\eta_{k}^{-1}}.

Hence

\frac{L}{\eta_{k+1}^{-1}}=\frac{1}{1-\kappa}\left(\frac{L}{\eta_{k}^{-1}}+1\right).

With the limiting initialization $\eta_{0}^{-1}=+\infty$ , this yields

\frac{L}{\eta_{k}^{-1}}=\sum_{i=1}^{k}(1-\kappa)^{-i}=\frac{(1-\kappa)^{-k}-1}{\kappa}.

Thus

\eta_{k}^{-1}=L\frac{\kappa}{(1-\kappa)^{-k}-1}\leq L\frac{\kappa}{(1+\kappa)^{k}-1},

where we used

(1-\kappa)^{-1}\geq 1+\kappa.

Each term of the sum $\sum_{i=1}^{k}(1-\kappa)^{-i}$ above is at least $1$ , so $L/\eta_{k}^{-1}\geq k$ , that is,

\eta_{k}^{-1}\leq\frac{L}{k}.

Therefore

\eta_{k}^{-1}\leq L\min\left\{\frac{1}{k},\frac{\kappa}{(1+\kappa)^{k}-1}\right\}.

∎

C.2 Technical results on BLDF

Geometry	$d_{i}(x)$	Dual form	Primal form ( $\Omega=\nabla d(\mathcal{X}^{\prime})$ )
Euclidean	$\tfrac{1}{2}x^{2}$	$1$	$1$
KL	$x\log x-x$	$\displaystyle\sup_{w_{1},w_{2}\in\Omega}\max_{i}\left(\frac{e^{(w_{2})_{i}}}{e^{(w_{1})_{i}}}\right)^{1/2}$	$\displaystyle\sup_{x_{1},x_{2}\in\mathcal{X}^{\prime}}\max_{i}\left(\frac{(x_{2})_{i}}{(x_{1})_{i}}\right)^{1/2}$
Burg	$-\log x$	$\displaystyle\sup_{w_{1},w_{2}\in\Omega}\max_{i}\frac{(w_{1})_{i}}{(w_{2})_{i}}$	$\displaystyle\sup_{x_{1},x_{2}\in\mathcal{X}^{\prime}}\max_{i}\frac{(x_{2})_{i}}{(x_{1})_{i}}$
Power-type	$\tfrac{\|x\|^{p}}{p}$	$\displaystyle\sup_{w_{1},w_{2}\in\Omega}\max_{i}\left(\frac{\|(w_{2})_{i}\|}{\|(w_{1})_{i}\|}\right)^{\frac{2-p}{2(p-1)}}$	$\displaystyle\sup_{x_{1},x_{2}\in\mathcal{X}^{\prime}}\max_{i}\left(\frac{\|(x_{1})_{i}\|}{\|(x_{2})_{i}\|}\right)^{\frac{p-2}{2}}$

Table 2: Dual norm distortion factor

\rho_{d^{*}}(\Omega)

for standard separable geometries, in dual and primal coordinates, where

\mathcal{X}^{\prime}\subset\operatorname{int}(\dom d)

is the primal region with

\nabla d(\mathcal{X}^{\prime})=\Omega

. The factor is finite when the coordinate ratios over

\Omega

are bounded, but diverges as a coordinate approaches the boundary of the domain for the KL and Burg geometries, or the origin for the power-type geometry. In the Euclidean case, it equals

1

on every

\Omega

Lemma 5 (BLDF under smoothness and strong convexity).

Assume that $d^{*}$ is $L_{d^{*}}$ -smooth and $\mu_{d^{*}}$ -strongly convex on a convex set $\Omega\subset\operatorname{int}(\dom d^{*})$ . Then, for every $s>0$ ,

c_{d^{*}}(s,\Omega;2)\leq\sqrt{\frac{L_{d^{*}}}{\mu_{d^{*}}}}.

In particular, if $d^{*}(u)=\frac{1}{2}\|u\|^{2}$ , then

c_{d^{*}}(s,\Omega;2)=1.

Proof.

By $L_{d^{*}}$ -smoothness and $\mu_{d^{*}}$ -strong convexity, for all admissible $u_{1},u_{2},v$ ,

D_{d^{*}}(u_{2}+sv,u_{2})\leq\frac{L_{d^{*}}}{2}\|sv\|^{2}=s^{2}\frac{L_{d^{*}}}{2}\|v\|^{2},

and

D_{d^{*}}(u_{1}+v,u_{1})\geq\frac{\mu_{d^{*}}}{2}\|v\|^{2}.

Therefore

\frac{D_{d^{*}}(u_{2}+sv,u_{2})}{s^{2}D_{d^{*}}(u_{1}+v,u_{1})}\leq\frac{L_{d^{*}}}{\mu_{d^{*}}}.

Taking the square root and then the supremum over all admissible $u_{1},u_{2},v$ gives

c_{d^{*}}(s,\Omega;2)\leq\sqrt{\frac{L_{d^{*}}}{\mu_{d^{*}}}}.

For $d^{*}(u)=\frac{1}{2}\|u\|^{2}$ ,

D_{d^{*}}(u_{2}+sv,u_{2})=\frac{1}{2}s^{2}\|v\|^{2},\qquad D_{d^{*}}(u_{1}+v,u_{1})=\frac{1}{2}\|v\|^{2},

so the ratio is exactly $1$ , hence

c_{d^{*}}(s,\Omega;2)=1.

∎

Corollary 2 (Separable case).

Assume that $d^{*}$ is twice differentiable and separable on $\Omega$ , i.e.,

d^{*}(u)=\sum_{i=1}^{n}d_{i}^{*}(u_{i}).

Then the dual norm distortion factor satisfies

\rho_{d^{*}}(\Omega)=\sup_{w_{1},w_{2}\in\Omega}\max_{1\leq i\leq n}\left(\frac{(d_{i}^{*})^{\prime\prime}((w_{2})_{i})}{(d_{i}^{*})^{\prime\prime}((w_{1})_{i})}\right)^{1/2}.

Proof.

For separable $d^{*}$ , definition 8 gives

\|\xi\|_{w,*}^{2}=\tfrac{1}{2}\sum_{i=1}^{n}(d_{i}^{*})^{\prime\prime}(w_{i})\,\xi_{i}^{2},

and the factor $\tfrac{1}{2}$ cancels in the ratio below. Hence, for any $w_{1},w_{2}\in\Omega$ ,

\sup_{\xi\neq 0}\frac{\|\xi\|_{w_{2},*}}{\|\xi\|_{w_{1},*}}=\max_{1\leq i\leq n}\left(\frac{(d_{i}^{*})^{\prime\prime}((w_{2})_{i})}{(d_{i}^{*})^{\prime\prime}((w_{1})_{i})}\right)^{1/2},

and the result follows by taking the supremum over $w_{1},w_{2}\in\Omega$ . ∎

See 1

Proof.

Fix admissible $u_{1},u_{2}\in\Omega$ and $v\neq 0$ . Since $u_{1}+[0,1]v\subset\Omega$ and $u_{2}+[0,s]v\subset\Omega$ , the integral representation of the Bregman divergence gives

D_{d^{*}}(u_{1}+v,u_{1})=\int_{0}^{1}(1-r)\,v^{\top}\nabla^{2}d^{*}(u_{1}+rv)\,v\,dr,

and this holds under pointwise twice differentiability alone. Indeed, $g(r):=d^{*}(u_{1}+rv)$ is convex, so $g^{\prime}$ is nondecreasing, and $g^{\prime}$ is differentiable everywhere on $[0,1]$ with $g^{\prime\prime}\geq 0$ and $\int_{0}^{1}g^{\prime\prime}\leq g^{\prime}(1)-g^{\prime}(0)<\infty$ ; a function differentiable everywhere with integrable derivative is absolutely continuous, so the fundamental theorem of calculus applies to $g^{\prime}$ , and one integration by parts produces the weight $(1-r)$ . Note also that $D_{d^{*}}(u_{1}+v,u_{1})>0$ : $d$ is Legendre, hence so is $d^{*}$ , hence $d^{*}$ is strictly convex on the convex set $\Omega\subseteq\operatorname{int}(\dom d^{*})$ , so the denominators below do not vanish. Explicitly,

D_{d^{*}}(u_{1}+v,u_{1})=\int_{0}^{1}(1-r)\,v^{\top}\nabla^{2}d^{*}(u_{1}+rv)\,v\,dr,

and

D_{d^{*}}(u_{2}+sv,u_{2})=s^{2}\int_{0}^{1}(1-r)\,v^{\top}\nabla^{2}d^{*}(u_{2}+rsv)\,v\,dr.

Both integrands are nonnegative and $\int_{0}^{1}(1-r)\,dr=\tfrac{1}{2}$ , so each divergence lies between the extreme values of $\|v\|_{\cdot,*}^{2}$ on its segment,

\inf_{w\in u_{1}+[0,1]v}\|v\|_{w,*}^{2}\;\leq\;D_{d^{*}}(u_{1}+v,u_{1})\;\leq\;\sup_{w\in u_{1}+[0,1]v}\|v\|_{w,*}^{2},

and likewise $D_{d^{*}}(u_{2}+sv,u_{2})\in s^{2}[\inf,\sup]$ over $u_{2}+[0,s]v$ . No continuity of $\nabla^{2}d^{*}$ is required. Therefore,

\left(\frac{D_{d^{*}}(u_{2}+sv,u_{2})}{s^{\gamma}D_{d^{*}}(u_{1}+v,u_{1})}\right)^{1/2}\leq s^{1-\gamma/2}\left(\frac{\sup_{w_{2}\in u_{2}+[0,s]v}\|v\|_{w_{2},*}^{2}}{\inf_{w_{1}\in u_{1}+[0,1]v}\|v\|_{w_{1},*}^{2}}\right)^{1/2}\leq s^{1-\gamma/2}\rho_{d^{*}}(\Omega),

the last step because both segments lie in $\Omega$ . Taking the supremum over all admissible $u_{1},u_{2},v$ yields the claim. ∎

With the following additional regularity assumptions, the two factors are equal.

Proposition 4.

Assume that $\Omega$ is open and that $\nabla^{2}d^{*}$ is continuous and positive definite on $\Omega$ . Then

c_{d^{*}}(s,\Omega;2)=\rho_{d^{*}}(\Omega)\quad\text{for every }s>0,\qquad c_{d^{*}}(s,\Omega;\gamma)=s^{1-\gamma/2}\rho_{d^{*}}(\Omega)\quad\text{for every }\gamma>0.

Proof.

The proof of proposition 1 gives $\leq$ at every $\gamma$ , although the proposition is stated at $\gamma=2$ ; only the reverse direction is at issue. Fix $u_{1},u_{2}\in\Omega$ and $\xi\neq 0$ , and take $v=\varepsilon\xi$ . Since $\Omega$ is open, both $u_{1}+[0,1]v$ and $u_{2}+[0,s]v$ lie in $\Omega$ once $\varepsilon$ is small enough, so the pair is admissible. Here $\nabla^{2}d^{*}$ is continuous, so the weighted mean-value theorem applies to the integral representation used in the proof of proposition 1 and gives points $\bar{w}_{1}(\varepsilon)\in u_{1}+[0,\varepsilon]\xi$ and $\bar{w}_{2}(\varepsilon)\in u_{2}+[0,s\varepsilon]\xi$ with $D_{d^{*}}(u_{1}+v,u_{1})=\|v\|_{\bar{w}_{1}(\varepsilon),*}^{2}$ and $D_{d^{*}}(u_{2}+sv,u_{2})=s^{2}\|v\|_{\bar{w}_{2}(\varepsilon),*}^{2}$ . Both points converge to $u_{1}$ and $u_{2}$ as $\varepsilon\downarrow 0$ , and $\nabla^{2}d^{*}$ is positive definite there, so

\left(\frac{D_{d^{*}}(u_{2}+sv,u_{2})}{s^{\gamma}D_{d^{*}}(u_{1}+v,u_{1})}\right)^{1/2}=s^{1-\gamma/2}\frac{\|\xi\|_{\bar{w}_{2}(\varepsilon),*}}{\|\xi\|_{\bar{w}_{1}(\varepsilon),*}}\xrightarrow[\varepsilon\downarrow 0]{}s^{1-\gamma/2}\frac{\|\xi\|_{u_{2},*}}{\|\xi\|_{u_{1},*}}.

The left-hand side is bounded above by $c_{d^{*}}(s,\Omega;\gamma)$ for every $\varepsilon$ , so the limit is too. Taking the supremum over $u_{1},u_{2}\in\Omega$ and $\xi\neq 0$ gives $c_{d^{*}}(s,\Omega;\gamma)\geq s^{1-\gamma/2}\rho_{d^{*}}(\Omega)$ . ∎

C.3 Technical results on Accelerated Method

Throughout this subsection, $\alpha_{k}:=\mu+\eta_{k}^{-1}$ .

Lemma 6 (Implicit recursive accelerated bound).

Assume that $t_{k}=1$ , or that

\phi_{k}^{\mathrm{best}}\leq\mathcal{D}^{\eta_{k}}(\lambda_{k}).

If the inner loop of Algorithm 2 exits, i.e.,

\phi_{k+1}^{\mathrm{best}}\leq(1-t_{k})\phi_{k}^{\mathrm{best}}+t_{k}\phi_{k}^{\mathrm{low}}-\left(\mu+\eta_{k+1}^{-1}\right)D_{d}(z_{k},z_{k+1}),

then

\phi_{k+1}^{\mathrm{best}}=\phi(x_{k+1}^{\mathrm{best}})\leq\phi_{\star}^{\eta_{k+1}}.

Proof.

Apply lemma 3 with $b_{k}=\phi_{k}^{\mathrm{best}}$ ; when $t_{k}=1$ , apply instead its second conclusion, which carries no hypothesis on $\phi_{k}^{\mathrm{best}}$ and whose right-hand side is the same because $(1-t_{k})=0$ . The exit condition gives

\phi_{k+1}^{\mathrm{best}}\leq\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1}).

Since $S^{\eta_{k+1}}(x,\lambda_{k+1})\leq\phi^{\eta_{k+1}}(x)$ for all $x$ ,

\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})=\min_{x}S^{\eta_{k+1}}(x,\lambda_{k+1})\leq\min_{x}\phi^{\eta_{k+1}}(x)=\phi_{\star}^{\eta_{k+1}}.

Therefore

\phi_{k+1}^{\mathrm{best}}\leq\phi_{\star}^{\eta_{k+1}}.

∎

See 2

Proof.

By lemma 6, the invariant and the exit condition imply

\phi_{k+1}^{\mathrm{best}}\leq\phi_{\star}^{\eta_{k+1}}.

Thus the iterates satisfy the implicit descent condition. Applying Theorem 1 with $x_{k+1}=x_{k+1}^{\mathrm{best}}$ gives

\phi_{k+1}^{\mathrm{best}}-\phi_{\star}\leq\eta_{k+1}^{-1}D_{d}(x_{\star},x_{0}).

The monotonicity

\phi_{k+1}^{\mathrm{best}}\leq\phi_{k}^{\mathrm{best}}

follows from the definition of the best value.

It remains to estimate $\eta_{k}^{-1}$ . First assume $\mu=0$ . Then

\eta_{k+1}^{-1}=M_{k}t_{k}^{2},\qquad t_{k}=1-\frac{\eta_{k}}{\eta_{k+1}}.

Hence

\eta_{k+1}=M_{k}(\eta_{k+1}-\eta_{k})^{2},

\eta_{k+1}-\eta_{k}=\sqrt{\frac{\eta_{k+1}}{M_{k}}}.

Therefore

\sqrt{\eta_{k+1}}-\sqrt{\eta_{k}}=\frac{\eta_{k+1}-\eta_{k}}{\sqrt{\eta_{k+1}}+\sqrt{\eta_{k}}}\geq\frac{1}{2\sqrt{M_{k}}}.

With the limiting initialization $\eta_{0}=0$ , summing gives

\sqrt{\eta_{k}}\geq\frac{1}{2}\sum_{i=0}^{k-1}M_{i}^{-1/2},

hence

\eta_{k}^{-1}\leq\frac{4}{\left(\sum_{i=0}^{k-1}M_{i}^{-1/2}\right)^{2}}.

If $M_{i}\leq M$ for all $i\geq K$ , then

\sqrt{\eta_{k}}\geq\sqrt{\eta_{K}}+\frac{k-K}{2\sqrt{M}},

and thus

\eta_{k}^{-1}\leq\frac{\eta_{K}^{-1}}{\left(1+\frac{k-K}{2\sqrt{M\eta_{K}}}\right)^{2}}.

Now assume $\mu>0$ . Since

\mu+\eta_{k+1}^{-1}=M_{k}t_{k}^{2},

we have

t_{k}\geq\sqrt{\frac{\mu}{M_{k}}}.

Thus

\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}\leq\left(1-\sqrt{\frac{\mu}{M_{k}}}\right)\eta_{k}^{-1}.

With the limiting initialization $\eta_{0}=0$ , we have $t_{0}=1$ , hence

\eta_{1}^{-1}=M_{0}-\mu.

Iterating from $i=1$ to $k-1$ gives

\eta_{k}^{-1}\leq(M_{0}-\mu)\prod_{i=1}^{k-1}\left(1-\sqrt{\frac{\mu}{M_{i}}}\right).

If $M_{i}\leq M$ for all $i\geq K$ , then

1-\sqrt{\frac{\mu}{M_{i}}}\leq 1-\sqrt{\frac{\mu}{M}},

and therefore

\eta_{k}^{-1}\leq\eta_{K}^{-1}\left(1-\sqrt{\frac{\mu}{M}}\right)^{k-K}.

∎

Lemma 7 (Generic accelerated bound).

Assume that $t_{k}=1$ , or that

\phi(x_{k})\leq\mathcal{D}^{\eta_{k}}(\lambda_{k}),\qquad\lambda_{k+1}=(1-t_{k})\lambda_{k}+t_{k}\nabla f(y_{k}),\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1},

and set

y_{k}=(1-\tau_{k})x_{k}+\tau_{k}z_{k}.

Let

\mathcal{L}_{k+1}(z_{k+1}):=(1-t_{k})\phi(x_{k})+t_{k}\phi_{k}^{\mathrm{low}}-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1}),

with $\phi_{k}^{\mathrm{low}}$ as in lemma 3. This is the right-hand side of the exit condition of algorithm 2 with $\phi_{k}^{\mathrm{best}}=\phi(x_{k})$ , and $\mathcal{L}_{k+1}(z_{k+1})\leq\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})$ by lemma 3. Then $x_{k+1}$ in Algorithm 3 satisfies

	$\displaystyle\mathcal{L}_{k+1}(z_{k+1})\geq$	$\displaystyle\phi(x_{k+1})+LD_{d}(y_{k},x_{k+1})-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1})$
		$\displaystyle+(1-t_{k})\mu D_{d}(x_{k},y_{k})+t_{k}\mu D_{d}(z_{k},y_{k})$
		$\displaystyle+(\tau_{k}-t_{k})\langle\nabla\phi(y_{k}),x_{k}-z_{k}\rangle.$

Proof.

The bound $\mathcal{L}_{k+1}(z_{k+1})\leq\mathcal{D}^{\eta_{k+1}}(\lambda_{k+1})$ is lemma 3 with $b_{k}=\phi(x_{k})$ , or its second conclusion when $t_{k}=1$ . When $t_{k}=1$ , every term below carrying the factor $(1-t_{k})$ — including the strong convexity substitution for $\phi(x_{k})$ — is absent. Expanding $\phi_{k}^{\mathrm{low}}$ gives

	$\displaystyle\mathcal{L}_{k+1}(z_{k+1})=$	$\displaystyle(1-t_{k})\phi(x_{k})+t_{k}\phi(y_{k})+t_{k}\langle\nabla\phi(y_{k}),z_{k}-y_{k}\rangle$
		$\displaystyle+t_{k}\mu D_{d}(z_{k},y_{k})-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1}).$

By relative $\mu$ -strong convexity of $\phi$ ,

\phi(x_{k})\geq\phi(y_{k})+\langle\nabla\phi(y_{k}),x_{k}-y_{k}\rangle+\mu D_{d}(x_{k},y_{k}).

Substituting this lower bound yields

	$\displaystyle\mathcal{L}_{k+1}(z_{k+1})\geq$	$\displaystyle\phi(y_{k})+(1-t_{k})\mu D_{d}(x_{k},y_{k})+t_{k}\mu D_{d}(z_{k},y_{k})$
		$\displaystyle+\left\langle\nabla\phi(y_{k}),(1-t_{k})(x_{k}-y_{k})+t_{k}(z_{k}-y_{k})\right\rangle$
		$\displaystyle-(\mu+\eta_{k+1}^{-1})D_{d}(z_{k},z_{k+1}).$

Since

y_{k}=(1-\tau_{k})x_{k}+\tau_{k}z_{k},

we have

(1-t_{k})(x_{k}-y_{k})+t_{k}(z_{k}-y_{k})=(\tau_{k}-t_{k})(x_{k}-z_{k}).

Finally, the descent step gives

\phi(x_{k+1})\leq\phi(y_{k})-LD_{d}(y_{k},x_{k+1}),

which gives the claim. ∎

Lemma 8 (Convex specialization).

Assume $\mu=0$ and choose $\tau_{k}=t_{k}$ . Then

\mathcal{L}_{k+1}(z_{k+1})\geq\phi(x_{k+1})+LD_{d}(y_{k},x_{k+1})-\alpha_{k+1}D_{d}(z_{k},z_{k+1}).

Proof.

Apply lemma 7. Since $\mu=0$ , the two terms multiplied by $\mu$ vanish. Since $\tau_{k}=t_{k}$ , the scalar-product term also vanishes. ∎

Proposition 5 (Convex one-step bound under dual norm distortion).

Under Assumption 1, assume $\mu=0$ . Let $\rho_{k}\geq 1$ be such that the relevant dual segments of iteration $k$ are contained in a convex set $\Omega_{k}$ satisfying

\rho_{d^{*}}(\Omega_{k})\leq\rho_{k}.

Let $M_{k}\geq\rho_{k}^{2}L$ and choose

\tau_{k}=t_{k},\qquad\eta_{k+1}^{-1}=M_{k}t_{k}^{2},\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}.

Then the accelerated step satisfies

\mathcal{L}_{k+1}(z_{k+1})\geq\phi(x_{k+1}).

Hence the exit condition holds at every acceleration parameter $M_{k}\geq\rho_{k}^{2}L$ , with $\mathcal{L}_{k+1}(z_{k+1})$ as in lemma 7.

Proof.

By lemma 8,

\mathcal{L}_{k+1}(z_{k+1})\geq\phi(x_{k+1})+LD_{d}(y_{k},x_{k+1})-\eta_{k+1}^{-1}D_{d}(z_{k},z_{k+1}).

Since $\mu=0$ ,

\nabla d(x_{k+1})=\nabla d(y_{k})-\frac{1}{L}\nabla\phi(y_{k}),\qquad\nabla d(z_{k+1})=\nabla d(z_{k})-\frac{t_{k}}{\eta_{k+1}^{-1}}\nabla\phi(y_{k}).

Thus the two dual displacements are parallel, with relative scaling

\frac{Lt_{k}}{\eta_{k+1}^{-1}}.

Using proposition 1 and $\rho_{d^{*}}(\Omega_{k})\leq\rho_{k}$ ,

D_{d}(z_{k},z_{k+1})\leq\rho_{k}^{2}\left(\frac{Lt_{k}}{\eta_{k+1}^{-1}}\right)^{2}D_{d}(y_{k},x_{k+1}).

Therefore

\displaystyle LD_{d}(y_{k},x_{k+1})-\eta_{k+1}^{-1}D_{d}(z_{k},z_{k+1})\geq\left(L-\frac{\rho_{k}^{2}L^{2}t_{k}^{2}}{\eta_{k+1}^{-1}}\right)D_{d}(y_{k},x_{k+1})\geq 0,

where the last inequality uses $\eta_{k+1}^{-1}=M_{k}t_{k}^{2}\geq\rho_{k}^{2}Lt_{k}^{2}$ . Hence

\mathcal{L}_{k+1}(z_{k+1})\geq\phi(x_{k+1}).

∎

Lemma 9 (Strongly convex specialization).

Assume $\mu>0$ . Define

q_{k+1}:=\nabla d^{*}\!\left(\nabla d(y_{k})-\frac{1}{\mu}\nabla\phi(y_{k})\right).

Assume that $\nabla d(y_{k})-\frac{1}{\mu}\nabla\phi(y_{k})\in\operatorname{int}(\dom d^{*})$ , so that $q_{k+1}$ is well defined, and that $\tau_{k}<1$ . Then, if

t_{k}\leq 1,\qquad\frac{t_{k}-\tau_{k}}{1-\tau_{k}}\leq t_{k},

we have

	$\displaystyle\mathcal{L}_{k+1}(z_{k+1})\geq$	$\displaystyle\phi(x_{k+1})+\Bigg[LD_{d}(y_{k},x_{k+1})-\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(y_{k},q_{k+1})\Bigg]$
		$\displaystyle+\Bigg[\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(z_{k},q_{k+1})-\alpha_{k+1}D_{d}(z_{k},z_{k+1})\Bigg].$

Proof.

By definition of $q_{k+1}$ ,

\nabla\phi(y_{k})=\mu\big(\nabla d(y_{k})-\nabla d(q_{k+1})\big).

Since

y_{k}=(1-\tau_{k})x_{k}+\tau_{k}z_{k},

we have

x_{k}-z_{k}=\frac{y_{k}-z_{k}}{1-\tau_{k}}.

Therefore

\displaystyle(\tau_{k}-t_{k})\langle\nabla\phi(y_{k}),x_{k}-z_{k}\rangle

\displaystyle=\frac{\mu(\tau_{k}-t_{k})}{1-\tau_{k}}\left\langle\nabla d(y_{k})-\nabla d(q_{k+1}),y_{k}-z_{k}\right\rangle.

By the three-point identity,

\left\langle\nabla d(y_{k})-\nabla d(q_{k+1}),y_{k}-z_{k}\right\rangle=D_{d}(y_{k},q_{k+1})+D_{d}(z_{k},y_{k})-D_{d}(z_{k},q_{k+1}).

Hence

	$\displaystyle(\tau_{k}-t_{k})\langle\nabla\phi(y_{k}),x_{k}-z_{k}\rangle=$	$\displaystyle-\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(y_{k},q_{k+1})$
		$\displaystyle-\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(z_{k},y_{k})$
		$\displaystyle+\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(z_{k},q_{k+1}).$

Substituting this identity into lemma 7 and writing $\alpha_{k+1}=\mu+\eta_{k+1}^{-1}$ gives

	$\displaystyle\mathcal{L}_{k+1}(z_{k+1})\geq$	$\displaystyle\phi(x_{k+1})+LD_{d}(y_{k},x_{k+1})-\alpha_{k+1}D_{d}(z_{k},z_{k+1})$
		$\displaystyle+(1-t_{k})\mu D_{d}(x_{k},y_{k})+t_{k}\mu D_{d}(z_{k},y_{k})$
		$\displaystyle-\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(y_{k},q_{k+1})-\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(z_{k},y_{k})$
		$\displaystyle+\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(z_{k},q_{k+1}).$

Rearranging,

	$\displaystyle\mathcal{L}_{k+1}(z_{k+1})\geq$	$\displaystyle\phi(x_{k+1})+\Bigg[LD_{d}(y_{k},x_{k+1})-\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(y_{k},q_{k+1})\Bigg]$
		$\displaystyle+\Bigg[\mu\frac{t_{k}-\tau_{k}}{1-\tau_{k}}D_{d}(z_{k},q_{k+1})-\alpha_{k+1}D_{d}(z_{k},z_{k+1})\Bigg]$
		$\displaystyle+(1-t_{k})\mu D_{d}(x_{k},y_{k})+\mu\left(t_{k}-\frac{t_{k}-\tau_{k}}{1-\tau_{k}}\right)D_{d}(z_{k},y_{k}).$

The last two terms are nonnegative under

t_{k}\leq 1,\qquad\frac{t_{k}-\tau_{k}}{1-\tau_{k}}\leq t_{k}.

Dropping them gives the result. ∎

Proposition 6 (Strongly convex one-step bound under dual norm distortion).

Under Assumption 1, assume $\mu>0$ and that $q_{k+1}$ of lemma 9 is well defined. Let $\rho_{k}\geq 1$ be such that the relevant dual segments of iteration $k$ , including the dual segments joining $\nabla d(y_{k})$ to $\nabla d(q_{k+1})$ and joining $\nabla d(z_{k})$ to $\nabla d(q_{k+1})$ , are contained in a convex set $\Omega_{k}$ satisfying

\rho_{d^{*}}(\Omega_{k})\leq\rho_{k}.

Let $M_{k}\geq\rho_{k}^{4}L$ , and let $t_{k}\in(0,1]$ be chosen by

\mu+\eta_{k+1}^{-1}=M_{k}t_{k}^{2},\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1}.

Equivalently, with $\alpha_{k}:=\mu+\eta_{k}^{-1}$ , $t_{k}$ is the positive root of

M_{k}t_{k}^{2}+(\alpha_{k}-\mu)t_{k}-\alpha_{k}=0.

Set

\tau_{k}=\frac{t_{k}-\mu/\sqrt{M_{k}L}}{1-\mu/\sqrt{M_{k}L}},\qquad y_{k}=(1-\tau_{k})x_{k}+\tau_{k}z_{k}.

Throughout, $L$ is the constant the gradient step of algorithm 3 actually uses, which is the same constant that forms $\tau_{k}$ . Then the accelerated step satisfies

\mathcal{L}_{k+1}(z_{k+1})\geq\phi(x_{k+1}).

Hence the exit condition holds at every acceleration parameter $M_{k}\geq\rho_{k}^{4}L$ , with $\mathcal{L}_{k+1}(z_{k+1})$ as in lemma 7.

Proof.

Assume first $t_{k}<1$ ; the case $t_{k}=1$ , which the initialization imposes at $k=0$ , is treated at the end.

Set

\alpha_{k+1}:=\mu+\eta_{k+1}^{-1},\qquad\delta_{k}:=\frac{t_{k}-\tau_{k}}{1-\tau_{k}}.

By construction,

\delta_{k}=\frac{\mu}{\sqrt{M_{k}L}}.

Moreover, $\alpha_{k+1}=M_{k}t_{k}^{2}$ .

By lemma 9,

	$\displaystyle\mathcal{L}_{k+1}(z_{k+1})\geq$	$\displaystyle\phi(x_{k+1})+\Big[LD_{d}(y_{k},x_{k+1})-\mu\delta_{k}D_{d}(y_{k},q_{k+1})\Big]$
		$\displaystyle+\Big[\mu\delta_{k}D_{d}(z_{k},q_{k+1})-\alpha_{k+1}D_{d}(z_{k},z_{k+1})\Big],$

provided $t_{k}\leq 1$ and $\delta_{k}\leq t_{k}$ .

We first show that the two brackets are nonnegative. The two uses of the dual BLDF are separate and each compares displacements along the same dual ray. For the first bracket, set

u_{y}:=\nabla d(y_{k}),\qquad v_{y}:=\nabla d(x_{k+1})-\nabla d(y_{k}).

The primal gradient step and the definition of $q_{k+1}$ give

\nabla d(x_{k+1})=u_{y}+v_{y},\qquad\nabla d(q_{k+1})=u_{y}+\frac{L}{\mu}v_{y}.

Hence

D_{d}(y_{k},x_{k+1})=D_{d^{*}}(u_{y}+v_{y},u_{y}),\qquad D_{d}(y_{k},q_{k+1})=D_{d^{*}}\left(u_{y}+\frac{L}{\mu}v_{y},u_{y}\right).

Applying the dual BLDF on this same ray, with scaling $L/\mu$ , gives

D_{d}(y_{k},x_{k+1})\geq\frac{1}{\rho_{k}^{2}}\left(\frac{\mu}{L}\right)^{2}D_{d}(y_{k},q_{k+1}),

hence

LD_{d}(y_{k},x_{k+1})-\mu\delta_{k}D_{d}(y_{k},q_{k+1})\geq\left(\frac{\mu^{2}}{\rho_{k}^{2}L}-\mu\delta_{k}\right)D_{d}(y_{k},q_{k+1})\geq 0,

since $\sqrt{M_{k}L}\geq\rho_{k}^{2}L$ . For the second bracket, set

u_{z}:=\nabla d(z_{k}),\qquad v_{z}:=\nabla d(q_{k+1})-\nabla d(z_{k}).

Using the optimality conditions of the dual map and $\alpha_{k+1}=\mu+\eta_{k+1}^{-1}$ , we get

\nabla d(z_{k+1})=u_{z}+\frac{t_{k}\mu}{\alpha_{k+1}}v_{z}.

Thus

D_{d}(z_{k},z_{k+1})=D_{d^{*}}\left(u_{z}+\frac{t_{k}\mu}{\alpha_{k+1}}v_{z},u_{z}\right),\qquad D_{d}(z_{k},q_{k+1})=D_{d^{*}}(u_{z}+v_{z},u_{z}),

and the dual BLDF on this ray gives

D_{d}(z_{k},z_{k+1})\leq\rho_{k}^{2}\left(\frac{t_{k}\mu}{\alpha_{k+1}}\right)^{2}D_{d}(z_{k},q_{k+1}),

\displaystyle\mu\delta_{k}D_{d}(z_{k},q_{k+1})-\alpha_{k+1}D_{d}(z_{k},z_{k+1})\geq\left(\mu\delta_{k}-\frac{\rho_{k}^{2}t_{k}^{2}\mu^{2}}{\alpha_{k+1}}\right)D_{d}(z_{k},q_{k+1})\geq 0,

where the last inequality uses $\delta_{k}=\mu/\sqrt{M_{k}L}$ and $\alpha_{k+1}=M_{k}t_{k}^{2}$ , so that

\mu\delta_{k}-\frac{\rho_{k}^{2}t_{k}^{2}\mu^{2}}{\alpha_{k+1}}=\mu^{2}\left(\frac{1}{\sqrt{M_{k}L}}-\frac{\rho_{k}^{2}}{M_{k}}\right)\geq 0\quad\text{since }M_{k}\geq\rho_{k}^{4}L.

Thus both brackets are nonnegative, and therefore

\mathcal{L}_{k+1}(z_{k+1})\geq\phi(x_{k+1}).

It remains to check the side conditions of lemma 9. Since

\alpha_{k+1}=M_{k}t_{k}^{2}=\mu+\eta_{k+1}^{-1}\geq\mu,

we have

t_{k}\geq\sqrt{\frac{\mu}{M_{k}}}\geq\sqrt{\frac{\mu}{M_{k}}}\sqrt{\frac{\mu}{L}}=\frac{\mu}{\sqrt{M_{k}L}}=\delta_{k},

where we used $\mu\leq L$ . Also $t_{k}\leq 1$ : the polynomial

p(t)=M_{k}t^{2}+(\alpha_{k}-\mu)t-\alpha_{k}

satisfies $p(0)=-\alpha_{k}<0$ and

p(1)=M_{k}-\mu\geq 0,

so its positive root lies in $(0,1]$ . Hence the assumptions of lemma 9 hold.

The case $t_{k}=1$ .

This case is outside lemma 9, which requires $\tau_{k}<1$ : the definition of $\tau_{k}$ gives $\tau_{k}=1$ when $t_{k}=1$ , and $\delta_{k}=(t_{k}-\tau_{k})/(1-\tau_{k})$ is then $0/0$ . It arises only at $k=0$ , where algorithm 2 imposes $t_{0}=1$ from outside the root formula above, and it does not need the virtual point $q_{k+1}$ at all. The initialization sets $x_{0}^{\mathrm{best}}=z_{0}=x_{0}$ , so $y_{0}=x_{0}$ for every $\tau_{0}$ , and $\nabla f(y_{0})=\nabla\phi(x_{0})$ . The two dual displacements therefore leave the single anchor $\nabla d(x_{0})$ along $-\nabla\phi(x_{0})$ :

\nabla d(x_{1})=\nabla d(x_{0})-\tfrac{1}{L}\nabla\phi(x_{0}),\qquad\nabla d(z_{1})=\nabla d(x_{0})-\tfrac{1}{\alpha_{1}}\nabla\phi(x_{0}),

which is the configuration of proposition 5. At $y_{0}=z_{0}=x_{0}$ , the $t_{k}=1$ case of lemma 3 gives $\mathcal{D}^{\eta_{1}}(\lambda_{1})\geq\min_{x}\{\phi(x_{0})+\langle\nabla\phi(x_{0}),x-x_{0}\rangle+\alpha_{1}D_{d}(x,x_{0})\}$ . The minimizer is $z_{1}$ , and $\nabla\phi(x_{0})=\alpha_{1}(\nabla d(x_{0})-\nabla d(z_{1}))$ gives $\langle\nabla\phi(x_{0}),z_{1}-x_{0}\rangle=-\alpha_{1}\big(D_{d}(z_{1},x_{0})+D_{d}(x_{0},z_{1})\big)$ . Hence

\mathcal{D}^{\eta_{1}}(\lambda_{1})\geq\phi(x_{0})-\alpha_{1}D_{d}(x_{0},z_{1})=\phi(x_{0})-\alpha_{1}D_{d}(z_{0},z_{1})=\mathcal{L}_{1}(z_{1}),

since $\phi_{0}^{\mathrm{low}}=\phi(x_{0})$ at $y_{0}=z_{0}=x_{0}$ . The same computation with $L$ in place of $\alpha_{1}$ and $x_{1}$ in place of $z_{1}$ , together with the relative smoothness of $\phi$ , gives $\phi(x_{1})\leq\phi(x_{0})-LD_{d}(x_{0},x_{1})$ . One comparison of the two displacements at the scale $s=L/\alpha_{1}$ , through definition 1 with $\rho_{d^{*}}(\Omega_{0})\leq\rho_{0}$ , gives $\alpha_{1}D_{d}(z_{0},z_{1})\leq\rho_{0}^{2}L^{2}\alpha_{1}^{-1}D_{d}(y_{0},x_{1})$ . The exit condition $\mathcal{L}_{1}(z_{1})\geq\phi(x_{1})$ follows whenever $\alpha_{1}\geq\rho_{0}^{2}L$ , and the choice in the statement gives $\alpha_{1}=M_{0}t_{0}^{2}=M_{0}\geq\rho_{0}^{4}L\geq\rho_{0}^{2}L$ since $\rho_{0}\geq 1$ . The first iteration therefore passes the exit condition at every acceleration parameter $M_{0}\geq\rho_{0}^{4}L$ . ∎

See 3

Proof.

We prove the convex case; the strongly convex case is identical, replacing $\rho^{2}L$ by $\rho^{4}L$ .

For every $k\geq K$ , proposition 5 with $\rho_{k}\leq\rho$ and $L_{\mathrm{cur}}\leq 2L$ implies that any trial value $M\geq 2\rho^{2}L$ is sufficient for acceptance. Since each trial doubles $M_{k}$ , the inner loop exits at every $k\geq K$ ; with the hypothesis for $k<K$ , it exits at every iteration, so Theorem 2 applies. By the update rule in algorithm 2, the search starts from a reduced value of the previously accepted $M_{k-1}$ and then doubles until acceptance. Hence, if the search starts below $2\rho^{2}L$ , the accepted value is at most $4\rho^{2}L$ . If it starts above $2\rho^{2}L$ , the first trial is accepted, and the next iteration again starts from a fixed reduction of that accepted value. Thus any overestimate inherited from earlier iterations decreases geometrically until it falls below $4\rho^{2}L$ . Therefore there exists $\bar{K}\geq K$ such that

M_{k}\leq 4\rho^{2}L\qquad\forall k\geq\bar{K}.

Applying Theorem 2 from iteration $\bar{K}$ with the uniform bound $M_{k}\leq 4\rho^{2}L$ gives

\eta_{k}^{-1}\leq\frac{\eta_{\bar{K}}^{-1}}{\left(1+\frac{k-\bar{K}}{2\sqrt{4\rho^{2}L\eta_{\bar{K}}}}\right)^{2}}=\frac{\eta_{\bar{K}}^{-1}}{\left(1+\frac{k-\bar{K}}{4\rho\sqrt{L\eta_{\bar{K}}}}\right)^{2}}.

The strongly convex case follows similarly from $M_{k}\leq 4\rho^{4}L$ , yielding

\eta_{k}^{-1}\leq\eta_{\bar{K}}^{-1}\left(1-\sqrt{\frac{\mu}{4\rho^{4}L}}\right)^{k-\bar{K}}.

∎

See 1

Proof.

Write $D:=D_{d}(x_{\star},x_{0})$ and let $k\geq\bar{K}$ .

Case $\mu=0$ . By Theorem 3 it suffices that

\frac{\eta_{\bar{K}}^{-1}D}{\left(1+s\right)^{2}}\leq\varepsilon,\qquad s:=\frac{k-\bar{K}}{4\rho\sqrt{L\eta_{\bar{K}}}},

that is $1+s\geq\sqrt{\eta_{\bar{K}}^{-1}D/\varepsilon}$ . Dropping the $1$ on the left and substituting $s$ gives the sufficient condition

k-\bar{K}\;\geq\;4\rho\sqrt{L\eta_{\bar{K}}}\,\sqrt{\frac{\eta_{\bar{K}}^{-1}D}{\varepsilon}}\;=\;4\rho\sqrt{\frac{LD}{\varepsilon}},

in which $\eta_{\bar{K}}$ cancels.

Case $\mu>0$ . Set $q:=\sqrt{\mu/(4\rho^{4}L)}\in(0,1)$ . By Theorem 3 it suffices that $\eta_{\bar{K}}^{-1}D(1-q)^{k-\bar{K}}\leq\varepsilon$ , i.e.

(k-\bar{K})\log\frac{1}{1-q}\;\geq\;\log\frac{\eta_{\bar{K}}^{-1}D}{\varepsilon}.

Since $\log\frac{1}{1-q}\geq q$ , the condition $k-\bar{K}\geq q^{-1}\log\!\left(\eta_{\bar{K}}^{-1}D/\varepsilon\right)$ is sufficient, and $q^{-1}=2\rho^{2}\sqrt{L/\mu}$ . ∎

C.4 Oracle complexity

Backtracking uses a variable number of oracle calls per accepted iteration. Let $L$ denote the relative smoothness constant, let $L_{0}$ and $M_{-1}$ be the initial estimates, and let

M_{\max}:=\max_{0\leq i<K}M_{i}

be the largest accepted value in the first $K$ outer iterations. Each iteration reduces the current estimates before the searches double them. This gives constant amortized overhead, with logarithmic terms for underestimated initial values. Writing $(a)_{+}:=\max\{a,0\}$ , the total number of trials satisfies

N_{\mathrm{trials}}\leq 4K+\left\lceil\log_{2}\!\left(\frac{M_{\max}}{M_{-1}}\right)_{+}\right\rceil+\left\lceil\log_{2}\!\left(\frac{L}{L_{0}}\right)_{+}\right\rceil.

This is the standard backtracking tradeoff [22]: the additional trials allow the method to use local estimates instead of a fixed worst-case constant.

Appendix D Euclidean specialization and comparison with Nesterov

We show that, in the Euclidean geometry, ABrA-GD reduces to the standard estimate-sequence form of Nesterov acceleration. Let

d(x)=\frac{1}{2}\|x\|^{2},\qquad D_{d}(x,y)=\frac{1}{2}\|x-y\|^{2},\qquad\nabla d(x)=x.

Assume for this comparison that the line searches accept the sharp constants, namely the smoothness estimate is $L$ and the accepted acceleration parameter is $M_{k}=L$ . Then the BLDF is equal to one, so no geometric distortion is introduced by the Bregman geometry.

One difference from the ordinary trajectory remains. Algorithm 2 passes $x_{k}^{\mathrm{best}}$ , not $x_{k}$ , into the next step, so the identities below are the classical estimate-sequence identities with $x_{k}$ read as $x_{k}^{\mathrm{best}}$ . The correspondence is therefore between the updates. The two trajectories coincide whenever the gradient step never increases the objective, and ABrA-GD is otherwise a monotone variant of the classical method.

Convex case.

When $\mu=0$ , one has $\tau_{k}=t_{k}$ and $\eta_{k+1}^{-1}=Lt_{k}^{2}$ . Writing $g_{k}=\nabla\phi(y_{k})$ , the ABrA-GD updates become

y_{k}=(1-t_{k})x_{k}+t_{k}z_{k},\qquad x_{k+1}=y_{k}-\frac{1}{L}g_{k},\qquad z_{k+1}=z_{k}-\frac{1}{Lt_{k}}g_{k}.

Therefore

x_{k+1}=(1-t_{k})x_{k}+t_{k}z_{k+1},

which is exactly the usual accelerated estimate-sequence coupling. Moreover, from

\eta_{k+1}^{-1}=Lt_{k}^{2},\qquad\eta_{k+1}^{-1}=(1-t_{k})\eta_{k}^{-1},

we recover the standard Nesterov/FISTA scalar recursion

t_{k}^{2}=(1-t_{k})t_{k-1}^{2}.

Theorem 2 gives

\phi(x_{k}^{\rm best})-\phi_{\star}\leq\eta_{k}^{-1}D_{d}(x_{\star},x_{0})\leq\frac{4L}{k^{2}}D_{d}(x_{\star},x_{0})=\frac{2L}{k^{2}}\|x_{\star}-x_{0}\|^{2},

which matches the classical accelerated $O(L\|x_{\star}-x_{0}\|^{2}/k^{2})$ rate up to the standard indexing convention.

Relatively strongly convex case.

Let $\mu>0$ and set

\alpha_{k}:=\mu+\eta_{k}^{-1}.

With $M_{k}=L$ , the parameters satisfy

\alpha_{k+1}=Lt_{k}^{2},\qquad\alpha_{k+1}=(1-t_{k})\alpha_{k}+t_{k}\mu,

and

\tau_{k}=\frac{t_{k}-\mu/L}{1-\,\mu/L}.

Define the Euclidean mirror point

q_{k+1}:=y_{k}-\frac{1}{\mu}\nabla\phi(y_{k}).

The dual/primal-averaging update can be written as

z_{k+1}=\left(1-\frac{t_{k}\mu}{\alpha_{k+1}}\right)z_{k}+\frac{t_{k}\mu}{\alpha_{k+1}}q_{k+1}.

The specific choice of $\tau_{k}$ gives again

x_{k+1}=(1-t_{k})x_{k}+t_{k}z_{k+1}.

Thus the Euclidean strongly convex specialization is the standard Nesterov estimate-sequence update written in the primal–dual variables. The generic rate becomes

\phi(x_{k}^{\rm best})-\phi_{\star}\leq\eta_{k}^{-1}D_{d}(x_{\star},x_{0}),\qquad\eta_{k}^{-1}\leq\eta_{K}^{-1}\left(1-\sqrt{\frac{\mu}{L}}\right)^{k-K},

once the accepted parameter is $M_{k}=L$ . This recovers the classical accelerated linear contraction $1-\sqrt{\mu/L}$ , up to constants and indexing. If the line search accepts values $M_{k}>L$ , the updates no longer coincide with Nesterov’s, and the rate is the one Theorem 2 states in terms of the accepted $M_{k}$ .

Appendix E Comparison with ABPG and triangle scaling

E.1 Comparison with ABPG

ABPG controls acceleration through a triangle-scaling exponent or gain. The adaptive variants of Hanzely et al. [13] search over the exponent or gain, while Savchuk et al. [24] fix the exponent and backtrack the relative smoothness constant. ABrA-GD backtracks both a smoothness parameter and an acceleration parameter. Its search over $M_{k}$ tests the primal–dual bound directly and responds to both smoothness and local geometric distortion.

Both approaches give bounds in terms of quantities accepted during the run (table 1). For ABrA-GD, the same framework also gives a conditional accelerated linear rate when $\mu>0$ . The cited TSP-based analyses do not establish such a rate. Hanzely et al. [13] report faster convergence with restarted ABPG on a D-optimal design instance described as relatively strongly convex, but give no accelerated linear guarantee for that experiment.

For gain-adaptive ABPG at $\gamma=2$ , the rate is $O(\bar{G}_{k}/k^{2})$ , where $\bar{G}_{k}$ depends on the accepted gains. A uniform bound on the gain can be obtained from a Hessian ratio over the full region $\mathcal{X}$ [13, §2.2, eq. (16)]. ABrA-GD uses the corresponding local quantity $\rho_{d^{*}}(\Omega_{k})$ to bound its dual divergence comparisons. For twice-differentiable geometries, proposition 1 relates these quantities, and table 2 gives formulas for standard kernels.

E.2 Relation to the triangle-scaling property

A uniform triangle-scaling gain at exponent $2$ bounds the dual norm distortion. Let $\mathcal{X}$ be convex with $d$ twice differentiable and $\nabla^{2}d\succ 0$ , let $\Omega=\nabla d(\mathcal{X})$ , and suppose the triangle-scaling property holds on $\mathcal{X}$ with exponent $\gamma=2$ and a gain $G$ uniform over triples. Writing both divergences in integral form along the common direction, letting $\theta\to 0$ and shrinking the pair to a point $q$ gives

G\;\geq\;\sup_{x,q\in\mathcal{X}}\lambda_{\max}\!\left(\nabla^{2}d(q)^{-1}\nabla^{2}d(x)\right)\;=\;\rho_{d^{*}}(\Omega)^{2},

using $\nabla^{2}d^{*}(\nabla d(x))=\nabla^{2}d(x)^{-1}$ . Thus $\rho_{d^{*}}^{2}\leq G$ on the corresponding dual region. This Hessian ratio appears in the triangle-scaling bound of Hanzely et al. [13, §2.2, eq. (16)] and in the Hessian stability condition of Karimireddy et al. [14]. Here, it directly controls the local divergence comparisons, including those needed when $\mu>0$ .

The implication requires a gain uniform over triples. It does not concern the intrinsic triangle-scaling exponent defined by a pointwise limit and used by adaptive variants in practice.

For a jointly convex Bregman divergence, $\gamma=1$ is a uniform triangle-scaling exponent at gain $1$ , giving an $O(1/k)$ rate. The entropy kernel has this property. For $d=\tfrac{1}{p}\|x\|^{p}$ with $p>2$ , the global exponent can also be below $1$ . The kernel-specific constructions of Hanzely et al. [13] modify the reference function and restrict the domain. The formulas in table 2 express the corresponding local distortion through coordinate ratios: approaching the origin can make the distortion unbounded, while bounded ratios give finite distortion.

The Burg kernel.

The Burg kernel $d(x)=-\sum_{i}\log x_{i}$ is not jointly convex, and Hanzely et al. [13] show that it has no uniform exponent $\gamma>1/2$ at gain $1$ . The global guarantee of ABPG for this kernel is therefore at best $O(k^{-1/2})$ , which is slower than the $O(1/k)$ rate of Bregman gradient descent. By table 2, the dual distortion factor of the Burg kernel is

\rho_{d^{*}}(\Omega)=\sup_{x_{1},x_{2}\in\mathcal{X}}\max_{i}\frac{(x_{2})_{i}}{(x_{1})_{i}},\qquad\Omega=\nabla d(\mathcal{X}),

which is finite whenever the coordinate ratios are bounded on $\mathcal{X}$ . For example, let $\mathcal{X}_{a,b}=\{x:\ a\leq x_{i}\leq b\ \text{for all }i\}$ with $0<a<b$ . Its image $\nabla d(\mathcal{X}_{a,b})$ is a box, hence convex, and $\rho_{d^{*}}(\nabla d(\mathcal{X}_{a,b}))=b/a$ . If, from some iteration on, the points that define the dual segments of definition 3 lie in $\mathcal{X}_{a,b}$ , then $\rho_{k}\leq b/a$ . Theorem 3 then gives an $O(1/k^{2})$ rate, and corollary 1 gives $O\big((b/a)\sqrt{L\,D_{d}(x_{\star},x_{0})/\varepsilon}\big)$ iterations when $\mu=0$ and an accelerated linear rate when $\mu>0$ . For the Burg kernel, the distortion factor thus gives an accelerated rate where the uniform triangle-scaling exponent gives a rate slower than $O(1/k)$ .

The comparison with a uniform gain at exponent $2$ is closer. On $\mathcal{X}_{a,b}$ , the display above gives $G\geq(b/a)^{2}$ , so the $O(G/k^{2})$ bound of a uniform gain is not smaller than the $O(\rho^{2}/k^{2})$ bound of Theorem 3, up to absolute constants. The distortion factor $\rho_{k}$ is measured on the region of iteration $k$ only, so it can be smaller than $b/a$ when the iterates concentrate. The guarantee of Theorem 3 is conditional: it requires the bound on $\rho_{k}$ to hold along the iterates, and the paper does not prove this bound a priori.

Appendix F Composite objectives and the subgradient envelope

The implicit envelope viewpoint does not require differentiability of the whole objective. Consider the composite problem

\Phi_{\star}=\min_{x\in\mathcal{X}}\Phi(x):=f(x)+r(x)+\mu D_{d}(x,x_{0}),

where $f$ is differentiable and relatively smooth, while $r$ is a proper closed convex term for which a Bregman proximal step is available. For $\eta>0$ , define

\Phi^{\eta}_{\star}:=\min_{x}\left\{f(x)+r(x)+\left(\mu+\eta^{-1}\right)D_{d}(x,x_{0})\right\}.

If an algorithm produces iterates satisfying the requirement

\Phi(x_{k})\leq\Phi^{\eta}_{\star},

then the same envelope argument gives

\Phi(x_{k})-\Phi_{\star}\leq\eta^{-1}D_{d}(x_{\star},x_{0}),\qquad x_{\star}\in\argmin_{x}\Phi(x),

because

\Phi^{\eta}_{\star}\leq\Phi^{\eta}(x_{\star})=\Phi_{\star}+\eta^{-1}D_{d}(x_{\star},x_{0}).

Thus the envelope descent requirement survives unchanged in the composite setting.

A corresponding lower model can be written either with a primal nonsmooth term or with a subgradient. Dualizing only the smooth part gives

S_{r}^{\eta}(x,\lambda):=\langle\lambda,x\rangle-f^{*}(\lambda)+r(x)+\left(\mu+\eta^{-1}\right)D_{d}(x,x_{0})\leq\Phi^{\eta}(x),

and therefore

\mathcal{D}_{r}^{\eta}(\lambda):=\min_{x}S_{r}^{\eta}(x,\lambda)\leq\Phi^{\eta}_{\star}.

Equivalently, for any $s_{y}\in\partial r(y)$ , convexity of $r$ gives the computable subgradient lower model

	$\displaystyle\Phi_{y}^{\rm low}(z):=$	$\displaystyle\Phi(y)+\left\langle\nabla f(y)+s_{y}+\mu\big(\nabla d(y)-\nabla d(x_{0})\big),z-y\right\rangle+\mu D_{d}(z,y)$
	$\displaystyle\leq$	$\displaystyle\Phi(z).$

The proximal primal step also remains well defined through

x^{+}\in\argmin_{x}\left\{\langle\nabla f(y),x-y\rangle+r(x)+LD_{d}(x,y)+\mu D_{d}(x,x_{0})\right\}.

This shows that the envelope and lower-model mechanisms extend naturally to nonsmooth composite terms. What does not follow automatically is the accelerated BLDF proof in the main text: the cancellation there uses a single continuous gradient-generated dual direction, whereas the composite model introduces a set-valued subgradient or a nonsmooth proximal residual. A full accelerated composite analysis would therefore require additional control of this residual.

Appendix G Numerical Experiments

G.1 Experimental protocol

The code is the toolbox of Hanzely et al. [13]¹¹ 1 https://github.com/linxiaolx/accbpg with an implementation of ABrA-GD added as one more method; the feasible domain and its boundary handling, the datasets, the baseline methods, and the initialization are theirs, unchanged. The objectives, reference functions and initial points below are stated from that source.

D-optimal design.

For $H\in\mathbb{R}^{m\times n}$ with $m<n$ ,

f(x)=-\log\det\big(H\operatorname{diag}(x)H^{\top}\big),\qquad\mathcal{X}=\Big\{x\geq 0:\ \textstyle\sum_{i}x^{(i)}=1\Big\},

and $d$ is the Burg entropy $d(x)=-\sum_{i}\log x^{(i)}$ restricted to that simplex. Here $f$ is $1$ -smooth relative to $d$ [19], so $L=1$ . The simplex constraint is the affine equality of section A.2. The restriction makes $\dom d$ bounded, so $\dom d^{*}=\mathbb{R}^{n}$ and the virtual point $q_{k+1}$ of lemma 9 exists at every iterate on these instances. Every method starts at the simplex center $x_{0}=(1/n)\mathbf{1}$ . The instances are built from the LIBSVM regression datasets abalone ( $n=4177$ , $m=8$ ), bodyfat ( $252$ , $14$ ), mpg ( $392$ , $7$ ) and housing ( $506$ , $13$ ) [9].

The relatively strongly convex instances are formed by the anchor term already in eq. 1: $\phi(x)=f(x)+\mu D_{d}(x,x_{0})$ , with the same $f$ , $d$ , and $x_{0}$ as the convex instance and $\mu=10^{-4}$ .

Poisson linear inverse problem.

$f(x)=D_{\mathrm{KL}}(b,Ax)$ on $\mathcal{X}=\mathbb{R}^{n}_{+}$ , with $A\in\mathbb{R}^{m\times n}$ drawn entrywise uniform on $[0,1]$ and column-normalized, and $b=A\bar{x}+\text{noise}$ for a sparse nonnegative $\bar{x}$ . The reference function is again the Burg entropy, now on the orthant, and $f$ is $\|b\|_{1}$ -smooth relative to it [4].

Parameters.

ABrA-GD starts every run at $L_{0}=1$ and $M_{-1}=L_{0}$ on both problems. This differs from Hanzely et al. [13], who supply the theoretical constant: $L=1$ for D-optimal design and $L=\|b\|_{1}$ for the Poisson problem. The baselines are run through their own interface with the constants that interface expects, as in Hanzely et al. [13]: BPG and ABPG receive $L$ , and BPG-LS, ABPG-e and ABPG-g adjust a gain online starting from it. No relative strong convexity constant is supplied to any baseline, because that interface takes no $\mu$ ; on the $\mu>0$ instances the anchor term enters through $\phi$ itself.

G.2 Safeguards and ablations

Restart strategies.

As in adaptive ABPG [13], we test two standard restart heuristics as ablations. The first is a function-value restart, triggered when the candidate objective value increases. The second is a gradient restart, triggered when the accelerated direction becomes misaligned with the current descent direction. After a restart, we keep the current best primal point and reset the accelerated state by setting

z_{k}=x_{k}=x_{k}^{\mathrm{best}},

resetting the dual averaging state, and continuing with the same adaptive searches for $L_{\mathrm{cur}}$ and $M_{k}$ .

These restart rules are heuristic and are not part of ABrA-GD’s convergence analysis. They are included only to test whether explicit momentum resets add anything beyond the exit condition of algorithm 2. The experiments below show that restarts do not provide a consistent advantage; in several cases they increase the variability of the accepted acceleration parameter $M_{k}$ or reduce oracle-call efficiency.

Gradient-descent safeguard.

The accelerated step may become inefficient when the accepted acceleration parameter $M_{k}$ is very large. To make the method never worse than Bregman gradient descent, one can compare the certificate progress of the accelerated step with the nonaccelerated dual-averaging update. This safeguard uses the true relative smoothness constant $L$ only as a theoretical comparison. Define

t_{k}^{\rm gd}=\frac{\mu+\eta_{k}^{-1}}{L+\eta_{k}^{-1}},\qquad\bar{\eta}_{k+1}^{-1}=(1-t_{k}^{\rm gd})\eta_{k}^{-1}.

After the accelerated line search has produced a candidate

\eta_{k+1}^{-1}=M_{k}t_{k}^{2}-\mu,

we accept it only if

\eta_{k+1}^{-1}\leq\bar{\eta}_{k+1}^{-1}.

Otherwise, we perform the certified nonaccelerated primal–dual step obtained by setting

y_{k}=z_{k},\qquad t_{k}=t_{k}^{\rm gd},\qquad\eta_{k+1}^{-1}=\bar{\eta}_{k+1}^{-1}.

This fallback is exactly a Bregman gradient step in primal variables, but written in the primal–dual variables so that the envelope certificate is preserved. Consequently,

\phi(x_{k}^{\rm best})-\phi^{\star}\leq\eta_{k,\rm gd}^{-1}D_{d}(x^{\star},x_{0}),

so the safeguarded method has at least the convergence rate of Bregman gradient descent while retaining accelerated steps whenever their certificate progress is better.

In all reported experiments, the safeguard was never triggered: every accepted ABrA-GD step already produced at least as much certificate progress as the nonaccelerated update. Thus, the safeguard is included only to make the worst-case comparison with Bregman gradient descent explicit and does not affect the numerical results.

G.3 Additional numerical experiments

We report additional experiments on D-optimal design and Poisson linear inverse problems. For D-optimal design, we test both the convex case and a relatively strongly convex variant with $\mu=10^{-4}$ . For the Poisson inverse problem, we test two variants, denoted Poisson L1 and Poisson L2.

Across the convex D-optimal design instances, ABrA-GD improves on nonaccelerated Bregman gradient methods and matches the best ABPG variants over the plotted range. In the relatively strongly convex setting, ABrA-GD’s log-residual decreases approximately linearly over the plotted range and reaches a lower residual than the competing methods. On the Poisson problems, ABrA-GD again improves on the nonaccelerated baselines, but which method reaches the lowest residual depends more strongly on the instance and the variant. The restart comparisons below show that explicit restarts do not improve on the plain method in these instances.

G.3.1 D-optimal design, convex, $\gamma=1$ for ABPG

Figure 5: D-optimal design on the housing dataset with ABPG run at the triangle-scaling exponent

\gamma=1

, one point in Hanzely et al. [13]’s own sweep over

\gamma\in\{1.0,1.5,2.0,2.2\}

; they report that at

\gamma=1

ABPG converges at

O(k^{-1})

but is slower than BPG. They fix

\gamma=2

for every ABPG variant in their own comparisons, which is why we use

\gamma=2

in all other comparisons.

G.3.2 D-optimal design, convex

Figure 6: Convex D-optimal design experiments on the abalone, bodyfat, mpg, and housing datasets. For each dataset, we report the residual

F-F_{\star}

versus iterations, the same residual versus oracle calls, and the accepted values of

M_{k}

for ABrA-GD. ABrA-GD improves over BPG and BPG-LS on all datasets and matches the best accelerated ABPG variants over the plotted range, while adapting

M_{k}

locally during the run.

G.3.3 D-optimal design, strongly convex

Figure 7: Relatively strongly convex D-optimal design experiments with

\mu=10^{-4}

. The left and middle columns show convergence in iterations and oracle calls, while the right column reports the accepted values of

M_{k}

. In this regime, ABrA-GD’s log-residual decreases approximately linearly over the plotted range and reaches a lower residual than BPG, BPG-LS, and the ABPG variants. The accepted values of

M_{k}

can fluctuate;

\phi^{\rm best}_{k}

is non-increasing by construction (algorithm 2) regardless.

G.3.4 Poisson problem

Figure 8: Poisson linear inverse problem, L1 variant

(m=200,n=100)

. ABrA-GD improves on the nonaccelerated Bregman methods and matches the best ABPG variants over the plotted range, in both iteration count and oracle-call count.

Figure 9: Poisson linear inverse problem, L2 variant

(m=100,n=1000)

. The accelerated methods reach a lower residual than BPG and BPG-LS. ABrA-GD tracks the best accelerated baseline over the plotted range; which method reaches the lowest final residual is more sensitive to the instance and the adaptive strategy than in the D-optimal design experiments.

G.3.5 Restart comparison: D-optimal design, convex

Figure 10: Restart comparison for convex D-optimal design. We compare ABrA-GD without restart, with gradient restart (g-RS), and with function-value restart (f-RS). Restarts do not improve performance consistently: the plain ABrA-GD variant is usually comparable or better in oracle calls, while the restarted variants can introduce additional variability in

M_{k}

G.3.6 Restart comparison: D-optimal design, strongly convex

Figure 11: Restart comparison for relatively strongly convex D-optimal design with

\mu=10^{-4}

. The non-restarted ABrA-GD variant typically reaches the lowest final residual across the datasets. The restarted variants often plateau earlier or require larger accepted values of

M_{k}

G.3.7 Restart comparison: Poisson problem

Figure 12: Restart comparison for the Poisson linear inverse problems. Restart is neutral or worse on both instances: the plain ABrA-GD update is not improved upon by either restarted variant on either instance. The restarted variants both stall on the Poisson L2 instance.

ABrA-GD: Adaptive Bregman Accelerated Gradient Descent for Relatively Smooth and (Strongly-)Convex Optimization

Abstract

1 Setting and Introduction

Assumption 1 (Problem setting).

Acceleration in Euclidean optimization.

Beyond Euclidean smoothness.

Conditions for Bregman acceleration.

Contributions

Limitations.

2 The Primal–Dual Envelope Framework

Theorem 1.

Link with proximal methods.

2.1 Envelope Dual Problem and Lower Bound

3 Deriving ABrA-GD and Its Accepted-Parameter Rate

1. Common gradient evaluation.

2. Primal upper model.

3. Dual step and primal map.

4. Parameters tkt_{k}, ηk\eta_{k}, and MkM_{k}.

5. A computable certificate.

Convergence in terms of the accepted parameters

Theorem 2 (A posteriori convergence of ABrA-GD).

4 Dual Bregman Length Distortion

4.1 Comparing the gradient and dual-averaging steps (convex case)

Strongly convex case.

4.2 The dual Bregman length distortion factor

Definition 1 (Local Dual Bregman Length Distortion Factor (Dual BLDF)).

Definition 2 (Dual norm distortion factor).

Proposition 1.

5 Accelerated Rates under Bounded Distortion

Definition 3 (Local distortion).

5.1 Convergence and iteration complexity

Theorem 3 (Accelerated rate under bounded distortion).

Corollary 1 (Iteration complexity).

Oracle complexity.

5.2 Comparison with existing methods

Inputs.

Triangle scaling and a posteriori bounds.

6 Numerical Experiments

7 Conclusion

References

Appendix A Notations and definitions

Definition 4 (Legendre function).

Definition 5 (Bregman Divergence [6]).

Definition 6 (Relative smoothness and strong convexity).

Definition 7 (Fenchel–Legendre conjugate [23, §12]).

Definition 8 (Primal and dual local norms).

Proposition 2 (Bregman divergence and local norms).

Proof.

A.1 The anchored decomposition

A.2 Affine equality constraints

Appendix B Recovering Bregman Gradient Descent and Dual Averaging

B.1 Mirror Gradient Descent

Proposition 3.

B.2 Saddle formulation of the envelope

B.3 Dual Averaging

Theorem 4.

Appendix C Technical Lemmas

Lemma 1 (Three-point property of Tseng [25]).

Lemma 2 (Growth around the minimizer).

Proof.

Lemma 3 (Dual lower-bound recursion).

Proof.

C.1 Convergence rate of gradient descent and dual averaging

Proof of Theorem 1.

Proof of proposition 3.

Lemma 4 (Dual-averaging one-step recursion).

Proof.

Proof of Theorem 4.

C.2 Technical results on BLDF

Lemma 5 (BLDF under smoothness and strong convexity).

Proof.

Corollary 2 (Separable case).

Proof.

Proof.

Proposition 4.

Proof.

C.3 Technical results on Accelerated Method

Lemma 6 (Implicit recursive accelerated bound).

Proof.

Proof.

4. Parameters $t_{k}$ , $\eta_{k}$ , and $M_{k}$ .

The case $t_{k}=1$ .

G.3.1 D-optimal design, convex, $\gamma=1$ for ABPG