UGM: A Unified Framework and New Perspectives for Accelerated Gradient Methods in Smooth and Strongly Convex Optimization

Danqing Zhou¹¹ 1 Email: dqkyo@foxmail.com. Shiqian Ma²² 2 Data Science and AI Institute, and Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21218. (Email: sqma@jhu.edu.) Junfeng Yang³³ 3 School of Mathematics, Nanjing University, Nanjing, Jiangsu 210093, People’s Republic of China. (Email: jfyang@nju.edu.cn).

August 27, 2026

Abstract

In this paper, we propose a unified framework for accelerated gradient methods, dubbed UGM, which subsumes a wide range of accelerated and conventional gradient-type methods designed for minimizing $L$ -smooth and $\mu$ -strongly convex functions. We demonstrate that the iteration update of the proposed framework can be intrinsically interpreted as a hybrid combination of the heavy-ball method and vanilla gradient descent. This interpretation reveals that classical accelerated gradient methods essentially integrate a conservative gradient descent step into the fast yet unstable heavy-ball dynamics, which achieves a favorable trade-off between acceleration and stability. We further establish a unified convergence analysis using Lyapunov functions. Guided by our analysis, we develop a family of enhanced accelerated gradient algorithms that leverage the inner product relationship between gradient information and iterative variables to optimize iterative updates. Extensive numerical experiments on unconstrained quadratic optimization and logistic regression validate that the proposed algorithms achieve superior performance compared with existing baseline methods under typical structural conditions.

Mathematical Subject Classifications: 15A18, 15A69, 65F15, 90C33

1 Introduction

Let $\mu$ and $L$ be constants satisfying $0<\mu<L<\infty$ , and let $\mathcal{F}_{\mu,L}$ be the set of $L$ -smooth and $\mu$ -strongly convex functions on $\mathbb{R}^{n}$ . In this paper, we consider the following unconstrained minimization problem

\min_{x\in\mathbb{R}^{n}}f(x),\quad f\in\mathcal{F}_{\mu,L}.

(1)

Throughout this paper, we denote $\kappa:=L/\mu$ as the condition number, $x^{*}$ as the unique minimizer of $f$ , and $f_{*}:=f(x^{*})$ . Gradient descent (GD) method is a standard algorithm for solving (1). With fixed step size $\alpha>0$ and initial point $x^{0}\in\mathbb{R}^{n}$ , the GD method iterates as

x^{k+1}=x^{k}-\alpha\nabla f(x^{k}),\quad k\geq 0.

(2)

A typical choice of the step size that is optimal for maximizing the per-iteration decrease in objective value is $\alpha=1/L$ , leading to the update

x^{k+1}=x^{k}-\frac{1}{L}\nabla f(x^{k}),\quad k\geq 0,

(3)

whereas $\alpha=2/(L+\mu)$ optimizes the decay of the squared optimality gap $\|x^{k}-x^{*}\|^{2}$ [25, 21]; these two step sizes yield convergence rates of $\mathcal{O}((1-{1}/{\kappa})^{k})$ and $\mathcal{O}((\kappa-1)/(\kappa+1))^{k})$ , respectively. Moreover, momentum-based accelerated algorithms have been extensively developed to solve the problem (1). The key principle of such methods lies in incorporating historical iterative information alongside the instantaneous gradient to construct the update direction, which enables improved convergence speed. Representative algorithms within this category contain Polyak’s classic Heavy-Ball (HB) accelerated scheme [28], Nesterov’s Fast Gradient Method (FGM) [24, 25], as well as numerous subsequent variants proposed in [7, 13, 17, 33, 27, 35, 32, 10]. While the body of related work is extensive, we focus, throughout this paper, our discussion on the FGM [24, 25], the Strongly Convex Optimized Gradient Method (SC-OGM) [27], the Triple Momentum (TM) method [33], and the Accelerated Over-Relaxation Heavy Ball method (AOR-HB) [35].

Given $x^{0}\in\mathbb{R}^{n}$ , $\upsilon,\varsigma>0$ , and $x^{-1}=x^{0}$ , the general form of Polyak’s HB method iterates via

x^{k+1}=x^{k}-\upsilon\nabla f(x^{k})+\varsigma(x^{k}-x^{k-1}),\quad k\geq 0.

(gHB)

Setting $\upsilon=4/{(\sqrt{L}+\sqrt{\mu})^{2}}$ and $\varsigma={(\sqrt{L}-\sqrt{\mu})^{2}}/{(\sqrt{L}+\sqrt{\mu})^{2}}$ yields

x^{k+1}=x^{k}-\frac{4}{(\sqrt{L}+\sqrt{\mu})^{2}}\nabla f(x^{k})+\frac{(\sqrt{L}-\sqrt{\mu})^{2}}{(\sqrt{L}+\sqrt{\mu})^{2}}(x^{k}-x^{k-1}),

(HB)

which is theoretically proven optimal for convex quadratic minimization with the convergence rate $\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}$ , since its convergence complexity attains the known lower complexity bounds [23]. However, (HB) is not guaranteed to converge for $f\in\mathcal{F}_{\mu,L}$ ; see counterexamples given in [19, 16]. Moreover, it is shown in [16] that for any $\kappa,\upsilon,\varsigma>0$ , the general HB method (gHB) either fails to attain an accelerated convergence rate (i.e., slower than $1-\mathcal{O}(1/\kappa)$ ) over strongly convex quadratic functions, or admits some $f\in\mathcal{F}_{\mu,L}$ together with an initialization for which the iteration diverges. Compared with the GD method (2), FGM achieves the faster convergence rates $\mathcal{O}(1/k^{2})$ on $\mathcal{F}_{0,L}$ and $\mathcal{O}((1-1/\sqrt{\kappa})^{k})$ on $\mathcal{F}_{\mu,L}$ for $\mu>0$ ; by contrast, convergence of (HB) is guaranteed only for strongly convex quadratic functions. The convergence rate of FGM over $\mathcal{F}_{\mu,L}$ is further improved to $\mathcal{O}((1-2/\sqrt{\kappa})^{k})$ in [33, 32], which matches the exact lower complexity bound in [12]. Moreover, the idea of accelerating gradient methods via momentum was extended to the proximal point method in [5] and to composite optimization problems in [7, 26]. Furthermore, the fundamental question “Why can FGM achieve accelerated convergence?” has attracted substantial research attention, and its acceleration mechanism has been interpreted from diverse viewpoints; we review several representative interpretations in what follows.

•

ODE-based interpretation. Su et al. [31] established an approximate equivalence between FGM and a second-order ordinary differential equation (ODE). Based on this ODE model, they found that oscillations in late iterations degrade the algorithm’s convergence. Accordingly, they proposed an adaptive restart strategy for cases where the gradient direction conflicts with the momentum direction. A large body of follow-up research has adopted the ODE perspective to analyze and design accelerated algorithms, including [6, 22, 11, 20, 29, 35].
•

Linear-coupling-based interpretation. Zhu et al. [3] attributed the acceleration of FGM to the linear coupling of two basic algorithms: GD (3) and mirror descent. The combination integrates their respective strengths to attain accelerated convergence. Later, this linear-coupling framework was extended to design faster and more concise algorithms for packing and covering linear programs [2, 34], semidefinite programming solvers [1], and coordinate descent methods [4].
•

Dissipativity theory from the control perspective. Lessard et al. [19] modeled FGM as a dynamical system and adopted the integral quadratic constraints (IQC) framework from robust control theory to rigorously analyze its accelerated convergence rate for strongly convex problems. Their work established a unified paradigm for automated algorithm design and performance evaluation. Later, this framework was extended to general convex settings via non-quadratic Lyapunov functions [14].

1.1 Our proposed framework and contributions

Given $x^{0}\in\mathbb{R}^{n}$ , $q\in(0,1)$ and $\theta\neq-1$ , we initialize $z^{0}=x^{0}$ . In this paper, we propose the following unified framework, dubbed UGM, for both non-accelerated and accelerated gradient methods


$\displaystyle\quad y^{k+1}$	$\displaystyle=x^{k}-\frac{1}{L}\nabla f(x^{k}),$	(4a)
$\displaystyle\quad w^{k+1}$	$\displaystyle=x^{k}-\frac{1}{\mu}\nabla f(x^{k}),$	(4b)
$\displaystyle\quad z^{k+1}$	$\displaystyle=qz^{k}+(1-q)w^{k+1},$	(4c)
$\displaystyle\quad x^{k+1}$	$\displaystyle=\frac{\theta}{1+\theta}y^{k+1}+\frac{1}{1+\theta}z^{k+1}.$	(4d)

A more compact yet equivalent formulation of the UGM update (4) is given in (49), which matches the general form of the optimized gradient method presented in [17, 18], [32, Eq. (2)]. Our main contributions are summarized below.

(i)

We show that UGM can be interpreted as a combination of (HB) and the GD method (3). Moreover, a broad class of popular accelerated and non-accelerated methods is unified within the UGM framework. Hence, accelerated methods such as FGM can be interpreted as integrating a conservative GD step (3) into the fast yet unstable HB method (HB) via an appropriate combination of coefficients, thereby balancing acceleration and convergence. Furthermore, we elucidate the roles of the parameters $q$ and $\theta$ : $q$ mainly regulates the convergence rate, while $\theta$ controls how closely the method resembles (3). These findings provide useful insights for constructing Lyapunov functions and are discussed in detail in Section 2.
(ii)

In Section 3, we conduct a unified convergence analysis for a broad class of popular accelerated methods with fixed parameters $q$ and $\theta$ under the UGM framework, using intuitively structured Lyapunov functions. The main convergence results are summarized in the fixed-parameter portion of Table 1. Under our analysis framework, we develop new $Q$ -linear convergence guarantees for FGM, TM, and AOR-HB methods.
(iii)

Since our analysis indicates that the choice of $\theta$ is, to some extent, independent of $q$ , we propose new accelerated UGM algorithms with varying parameters. We also develop adaptive variants, UGM-II and UGM-III, which select $\theta$ based on the inner product $\langle\nabla f(x^{k+1}),x^{k+1}\!-z^{k+1}\rangle$ to improve the iteration updates. To the best of our knowledge, these adaptive selection mechanisms are new to the literature. Full details are given in Section 4, and the main convergence results are summarized in the varying parameters portion of Table 1.

Param.	Method	Lyapunov function¹¹ 1 In this column, we only present the Lyapunov functions $V_{k}$ for Q-linear convergence, which satisfy $V_{k+1}\leq qV_{k}$ for some $q\in(0,1)$ .	Rate	Type	Ref.
fixed	FGM	—	$1-1/\sqrt{\kappa}$	R²² 2 In [25], convergence is established via the estimate sequence $\{\phi_{k}^{}\}$ , which is lower bounded by $\{f(x^{k})\}$ , leading to the result $f(x^{k})-f_{}\leq\phi_{k}^{}-f_{}\leq\left(1-1/\sqrt{\kappa}\right)^{k}\left(f(x^{0})-f_{}+\frac{\mu}{2}\\|x^{0}-x^{}\\|^{2}\right)$ .	[25]
	FGM	(19) with $a=1$	$1-1/\sqrt{\kappa}$	Q	this work
	SC-OGM	(19) with $a=1$	$1-\sqrt{2}/\sqrt{\kappa}$	Q	[27]
	SC-OGM	(19) with $a=1$	$1-\sqrt{2}/\sqrt{\kappa}$	Q	this work
	TM method	—	$1-2/\sqrt{\kappa}$	R	[33]
	TM method	(19) with $a=2\sqrt{\frac{\sqrt{\kappa}(\kappa+1)}{(\kappa-1)(\sqrt{\kappa}-1)}}$	$1-1/(2\sqrt{\kappa})$	Q	this work
	AOR-HB	see footnote³³ 3 Here, the Lyapunov function is $f(x^{k})\!-f_{}\!+{\mu\over 2}\\|p^{k}\!-\!x^{}\\|^{2}+\sqrt{\kappa}\,\langle\nabla f(x^{k}),p^{k}\!-\!x^{*}\rangle$ , where $p^{k}=x^{k+1}+(x^{k+1}-x^{k})/\sqrt{\kappa}$ . The authors also provide $1-1/\sqrt{\kappa}$ convergence analysis, but with a more complex Lyapunov function. In contrast, our analysis achieves the same rate using a simple Lyapunov function.	$1-1/(2\sqrt{\kappa})$	Q	[35]
	AOR-HB	(18) with $a=1\!+\!\frac{4\sqrt{\kappa}\!+1\!+\!(2\sqrt{\kappa}\!+\!1)\sqrt{4\sqrt{\kappa}\!+\!5}}{2\kappa}$	$1-1/\sqrt{\kappa}$	Q	this work
varying	ITEM	—	$1-2/\sqrt{\kappa}$	R	[32]
	UGM-I	(33)	$\big[\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1},\frac{\kappa-1}{\kappa+1}\big)$	Q	this work
	UGM-II
	UGM-III

Table 1: Convergence results for the main algorithms discussed in this paper. Parameters (Param. in the table) are classified as fixed or varying. The column “Type” denotes the type of linear convergence: R-linear (R) or Q-linear (Q).

1.2 Preliminaries and notation

Throughout the paper, we adopt the following notation. Let $\mathbb{R}^{n}$ be the $n$ -dimensional Euclidean space, with inner product and induced norm denoted by $\langle\cdot,\cdot\rangle$ and $\|\cdot\|=\sqrt{\langle\cdot,\cdot\rangle}$ , respectively. A function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is said to be $L$ -smooth if it is differentiable and satisfies $\|\nabla f(x)-\nabla f(y)\|\leq L\|x-y\|$ for all $x,y\in\mathbb{R}^{n}$ . Following [25, Thm. 2.1.5], for convex $f$ , $L$ -smoothness is equivalent to either of the following statements:

	$\displaystyle f(x)-f(y)-\langle\nabla f(y),x-y\rangle$	$\displaystyle\geq\tfrac{1}{2L}\\|\nabla f(x)-\nabla f(y)\\|^{2},\quad\forall x,y\in\mathbb{R}^{n};$		(5)
	$\displaystyle\langle\nabla f(x)-\nabla f(y),x-y\rangle$	$\displaystyle\geq{\tfrac{1}{L}}\\|\nabla f(x)-\nabla f(y)\\|^{2},\quad\forall x,y\in\mathbb{R}^{n}.$		(6)

A differentiable function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is said to be $\mu$ -strongly convex if

f(x)-f(y)-\langle\nabla f(y),x-y\rangle\geq\frac{\mu}{2}\|x-y\|^{2},\quad\forall x,y\in\mathbb{R}^{n}.

For any $f\in\mathcal{F}_{\mu,L}$ , there holds

\langle\nabla f(x)-\nabla f(y),x-y\rangle\geq\frac{\mu L}{\mu+L}\|x-y\|^{2}+\frac{1}{\mu+L}\|\nabla f(x)-\nabla f(y)\|^{2},\quad\forall x,y\in\mathbb{R}^{n}.

(7)

For any $u,v\in\mathbb{R}^{n}$ and $\alpha\in\mathbb{R}$ , the following identity holds:

\|\alpha u+(1-\alpha)v\|^{2}=\alpha\|u\|^{2}+(1-\alpha)\|v\|^{2}-\alpha(1-\alpha)\|u-v\|^{2}.

(8)

All algebraic identities and equivalent reformulations in this paper have been verified by manual calculation and symbolic computation using SymPy. The verification notebook is available at https://github.com/dq-kyo/UGM-a-Unified-Framework-for-Gradient-Method. Statements whose proofs rely on these code-verified algebraic identities are marked with footnotes in the text.

2 UGM unifies diverse gradient methods

We now explain the motivation behind UGM (4) from two perspectives.

Perspective 1: Embedding the GD step (3) into (HB).

First, the UGM iteration can be viewed as a combination of (HB) and the GD method (3). To see this, we can rewrite (HB) as


$\displaystyle\quad w^{k+1}$	$\displaystyle=x^{k}-\frac{1}{\mu}\nabla f\left(x^{k}\right),$	(9a)
$\displaystyle\quad z^{k+1}$	$\displaystyle=qz^{k}+(1-q)w^{k+1},$	(9b)
$\displaystyle\quad x^{k+1}$	$\displaystyle=\frac{\theta}{1+\theta}x^{k}+\frac{1}{1+\theta}z^{k+1}.$	(9c)

Here $z^{0}=x^{0}$ , $q=\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}$ and $\theta=\frac{\sqrt{\kappa}-1}{2}$ . This formulation shows that (HB) consists of a large GD step with step size $1/\mu$ , followed by two moving-average updates; the proof of equivalence between (HB) and (9) is provided in Appendix A. As shown in [19, 16], (HB) fails to converge for general strongly convex and smooth functions. To address this issue, we introduce the standard GD update (3), i.e., $y^{k+1}=x^{k}-\frac{1}{L}\nabla f(x^{k})$ , into the second moving-average step (9c), which yields the UGM iteration (4). By properly tuning the combination coefficients for (3) and (HB), existing accelerated methods (e.g., FGM) can be naturally embedded into the UGM framework. Hence, these methods essentially integrate the stable GD step (3) into the fast but possibly divergent scheme (HB), achieving a favorable trade-off between acceleration and convergence guarantee.

Perspective 2: Unifying many existing gradient methods.

Second, the UGM framework unifies a large family of classic accelerated and non-accelerated gradient methods. This unified perspective helps us clarify the roles of parameters $q$ and $\theta$ . Such understanding further guides the construction of Lyapunov functions and the design of new algorithms. Below, we list some representative algorithms encompassed by the proposed UGM framework.

1.

GD with step size $\alpha=1/L$ , namely, (3): UGM reduces to (3) when setting $q=1-\frac{1}{\kappa}$ and any $\theta\neq 1$ ; see Lemma B.2 for the equivalence proof. Here we also have $y^{k}=x^{k}=z^{k}$ for all $k$ ;
2.

GD with step size $\alpha=2/(L+\mu)$ : UGM reduces to this scheme when setting $q=\frac{\kappa-1}{\kappa+1}$ and $\theta=0$ ; see Lemma B.3 for the equivalence proof. Here we also have $x^{k}=z^{k}$ for all $k$ ;
3.

FGM [25]: This method iterates as

$y^{k+1}=x^{k}-\frac{1}{L}\nabla f(x^{k}),\;\;x^{k+1}=y^{k+1}+\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}(y^{k+1}-y^{k}),\quad k\geq 0,$ (10)

with $y^{0}=x^{0}$ . UGM reduces to this method when setting $q=1-\frac{1}{\sqrt{\kappa}}$ and $\theta=\sqrt{\kappa}$ ; see Lemma B.4 for the equivalence proof.

TM method [33]: This method iterates as


$\displaystyle\quad\xi^{k+1}$	$\displaystyle=(1+\lambda)\xi^{k}-\lambda\xi^{k-1}-\nu\nabla f(x^{k}),$	(11a)
$\displaystyle\quad x^{k}$	$\displaystyle=(1+\gamma)\xi^{k}-\gamma\xi^{k-1},$	(11b)
$\displaystyle\quad z^{k}$	$\displaystyle=(1+\delta)\xi^{k}-\delta\xi^{k-1},$	(11c)

where $(\nu,\lambda,\gamma,\delta)=(\frac{1+\rho}{L},\frac{\rho^{2}}{2-\rho},\frac{\rho^{2}}{(1+\rho)(2-\rho)},\frac{\rho^{2}}{1-\rho^{2}})$ and $\rho=1-\frac{1}{\sqrt{\kappa}}$ . UGM reduces to this method when setting $q=\rho=1-\frac{1}{\sqrt{\kappa}}$ and $\theta=\frac{\rho}{2(1-\rho)}$ ; see Lemma B.5 for the equivalence proof.

SC-OGM [27]: This method iterates as

y^{k+1}=x^{k}-\frac{1}{L}\nabla f(x^{k}),\;\;x^{k+1}=y^{k+1}+\frac{1}{2\gamma+1}(y^{k+1}-y^{k})+\frac{1}{2\gamma+1}(y^{k+1}-x^{k}),\quad k\geq 0,

(12)

with $y^{0}=x^{0}$ and $\gamma=\frac{\sqrt{8\kappa+1}+3}{2\kappa-2}$ . UGM reduces to this method when setting $q=\frac{1}{\gamma+1}$ and $\theta=\frac{\gamma+1}{\gamma}$ ; see Lemma B.6 for the equivalence proof.

AOR-HB [35]: This method iterates as

x^{k+1}=x^{k}-\frac{1}{(\sqrt{L}+\sqrt{\mu})^{2}}(2\nabla f(x^{k})-\nabla f(x^{k-1}))+\frac{L}{(\sqrt{L}+\sqrt{\mu})^{2}}(x^{k}-x^{k-1}),\quad k\geq 1.

(13)

UGM reduces to this method when setting $q=\frac{2\kappa+2\sqrt{\kappa}-1+\sqrt{4\sqrt{\kappa}+5}}{2(\sqrt{\kappa}+1)^{2}}$ and $\theta=\frac{\kappa}{2}\cdot\frac{2\sqrt{\kappa}+3-\sqrt{4\sqrt{\kappa}+5}}{\kappa+\sqrt{4\sqrt{\kappa}+5}-2}$ ; see Lemma B.7 for the equivalence proof.

Remark 2.1 (The role of $q$ and $\theta$ ).

Note that for the HB method (9), GD method (2) with $\alpha=1/L$ or $2/(L+\mu)$ , and FGM, the associated $q$ exactly matches their respective convergence rates. On the other hand, the convergence rates of SC-OGM, AOR-HB, and TM methods approximate their corresponding $q$ values. From this observation, we conclude that $q$ primarily characterizes the convergence rate. On the other hand, the parameter $\theta$ governs how similar the iteration behaves to the GD method (3). In fact, as can be seen from (4d), taking $\theta$ sufficiently large forces $x^{k+1}$ to mimic the GD iteration (3).

3 Unified analysis via Lyapunov functions

In this section, we derive the accelerated convergence rates for all the aforementioned accelerated algorithms via an intuitive unified Lyapunov function framework.

When conducting convergence analysis for gradient-type methods over the class $\mathcal{F}_{\mu,L}$ with $\mu>0$ , it is standard to study two quantities: the objective gap $f(x^{k})-f_{*}$ and the squared distance to the optimal solution $\|x^{k}-x^{*}\|^{2}$ . Standard proofs for Nesterov’s FGM require rather intricate technical arguments to balance these two quantities.

In our UGM analysis, we instead track the objective gap $f(x^{k})-f_{*}$ and the squared distance $\|z^{k}-x^{*}\|^{2}$ separately, and then balance these two terms. Our choice to employ $\|z^{k}-x^{*}\|^{2}$ draws inspiration from (2) with step sizes $\alpha=1/L$ and $\alpha=2/(L+\mu)$ , for which $z^{k}=x^{k}$ holds for all $k$ when regarded as special instances of UGM. We further derive an inductive relation for $f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ , motivated by [27] and the inductive relation for $f(x^{k})-f_{*}$ in (14).

Lemma 3.1 (Inductive relation for $f(x^{k})-f_{}$ and $f(x^{k})-f_{}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ ).

Let $\varrho\in(0,1)$ and $\theta\neq 0$ . For all $k\geq 0$ , the update $x^{k+1}$ defined in (4) satisfies

	$\displaystyle f(x^{k})-f_{*}$	$\displaystyle-\frac{1}{\varrho}\big(f(x^{k+1})-f_{*}\big)\geq\frac{1}{2L}\\|\nabla f(x^{k})\\|^{2}+\frac{1}{2L}\\|\nabla f(x^{k+1})\\|^{2}$		(14)
		$\displaystyle+\frac{1}{\theta}\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle-\frac{1-\varrho}{\varrho}\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right].$		(14)

Moreover, we also have

	$\displaystyle f(x^{k})-f_{*}$	$\displaystyle-\frac{1}{2L}\\|\nabla f(x^{k})\\|^{2}-\frac{1}{\varrho}\left[f(x^{k+1})-f_{*}-\frac{1}{2L}\\|\nabla f(x^{k+1})\\|^{2}\right]\geq\frac{\varrho+1}{2L\varrho}\\|\nabla f(x^{k+1})\\|^{2}$		(15)
		$\displaystyle+\frac{1}{\theta}\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle-\frac{1-\varrho}{\varrho}\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right].$		(15)

Proof.

Since $f$ is $L$ -smooth and $\mu$ -strongly convex, there hold

	$\displaystyle f(x^{k})-f(x^{k+1})$	$\displaystyle\geq\langle\nabla f(x^{k+1}),x^{k}-x^{k+1}\rangle+\frac{1}{2L}\\|\nabla f(x^{k})-\nabla f(x^{k+1})\\|^{2},$		(16)
	$\displaystyle f(x^{k+1})-f_{*}$	$\displaystyle\leq\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}.$		(16)

As a result, we have

		$\displaystyle f(x^{k})-f_{}-\frac{1}{\varrho}\big(f(x^{k+1})-f_{}\big)=f(x^{k})-f(x^{k+1})-\frac{1-\varrho}{\varrho}\big(f(x^{k+1})-f_{*}\big)$
	$\displaystyle\stackrel{{\scriptstyle\eqref{jy-a}}}{{\geq}}$	$\displaystyle\frac{1}{2L}\\|\nabla f(x^{k})-\nabla f(x^{k+1})\\|^{2}+\langle\nabla f(x^{k+1}),x^{k}-x^{k+1}\rangle-\frac{1-\varrho}{\varrho}\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right]$
	$\displaystyle\stackrel{{\scriptstyle\eqref{ugm-x}}}{{=}}$	$\displaystyle\text{right-hand side of~}\eqref{ef}.$

By subtracting $\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ and adding $\frac{1}{2L\varrho}\|\nabla f(x^{k+1})\|^{2}$ from both sides of (14), we obtain (15). ∎

Lemma 3.2 (Inductive relation for $\|z^{k+1}-x^{*}\|^{2}$ ).

For all $k\geq 0$ , the update $z^{k+1}$ defined in (4) satisfies

	$\displaystyle q\\|z^{k}-x^{*}\\|^{2}$	$\displaystyle-\\|z^{k+1}-x^{}\\|^{2}=\frac{2(1-q)}{\mu}\left[\langle\nabla f(x^{k}),x^{k}-x^{}\rangle-\frac{\mu}{2}\\|x^{k}-x^{*}\\|^{2}\right]$		(17)
		$\displaystyle+q(1-q)\\|x^{k}-z^{k}\\|^{2}-\frac{(1-q)^{2}}{\mu^{2}}\\|\nabla f(x^{k})\\|^{2}-\frac{2q(1-q)}{\mu}\langle\nabla f(x^{k}),x^{k}-z^{k}\rangle.$		(17)

Proof.

It follows from the UGM framework (4) and the identity (8) that

	$\displaystyle\\|z^{k+1}$	$\displaystyle-x^{}\\|^{2}\stackrel{{\scriptstyle\eqref{ugm-w},\eqref{ugm-z}}}{{=}}\big\\|(1-q)x^{k}+qz^{k}-x^{}-\frac{1-q}{\mu}\nabla f(x^{k})\big\\|^{2}$
	$\displaystyle=$	$\displaystyle\\|(1-q)x^{k}+qz^{k}-x^{}\\|^{2}+\frac{(1-q)^{2}}{\mu^{2}}\\|\nabla f(x^{k})\\|^{2}-\frac{2(1-q)}{\mu}\langle(1-q)x^{k}+qz^{k}-x^{},\nabla f(x^{k})\rangle$
	$\displaystyle\stackrel{{\scriptstyle\eqref{eq:convex-id}}}{{=}}$	$\displaystyle(1-q)\\|x^{k}-x^{}\\|^{2}+q\\|z^{k}-x^{}\\|^{2}-q(1-q)\\|x^{k}-z^{k}\\|^{2}+\frac{(1-q)^{2}}{\mu^{2}}\\|\nabla f(x^{k})\\|^{2}$
		$\displaystyle-\frac{2(1-q)}{\mu}\langle x^{k}-x^{*},\nabla f(x^{k})\rangle+\frac{2q(1-q)}{\mu}\langle x^{k}-z^{k},\nabla f(x^{k})\rangle.$

Rearranging the terms in the above equality, we obtain (17). ∎

Next, we introduce the following Lyapunov functions, which play a central role in our linear convergence analysis:

	$\displaystyle E_{k}(a)$	$\displaystyle:=f(x^{k})-f_{}+\frac{a\mu}{2}\\|z^{k+1}-x^{}\\|^{2},$		(18)
	$\displaystyle F_{k}(a)$	$\displaystyle:=f(x^{k})-f_{}-\frac{1}{2L}\\|\nabla f(x^{k})\\|^{2}+\frac{a\mu}{2}\\|z^{k+1}-x^{}\\|^{2}.$		(19)

Here $a>0$ denotes an undetermined parameter serving as a balancing weight between the squared distance term $\|z^{k+1}-x^{*}\|^{2}$ and either the objective gap $f(x^{k})-f_{*}$ or $f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ . For $a=1$ , we write $E_{k}=E_{k}(1)$ and $F_{k}=F_{k}(1)$ for brevity. By combining the inductive bounds for $f(x^{k})-f_{*}$ from (14), for $f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ from (15), and for $\|z^{k+1}-x^{*}\|^{2}$ from (17), we arrive at the following theorem.

Theorem 3.1.

For the iterates defined in (4), suppose $0<q\leq\varrho<1$ and $a>0$ . Let $E_{k}(a)$ and $F_{k}(a)$ be defined in (18) and (19), respectively. Then, for all $k\geq 0$ , we have

		$\displaystyle E_{k}(a)-\frac{1}{\varrho}E_{k+1}(a)\geq\frac{1}{2L}\\|\nabla f(x^{k})\\|^{2}+\left(\frac{1-q}{q}a-\frac{1-\varrho}{\varrho}\right)\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right]$		(20)
		$\displaystyle+\left(\frac{1}{2L}-\frac{a(1-q)^{2}}{2q\mu}\right)\\|\nabla f(x^{k+1})\\|^{2}+\left(\frac{1}{\theta}-(1-q)a\right)\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle+\frac{\mu(1-q)a}{2}\\|x^{k+1}-z^{k+1}\\|^{2},$

and

		$\displaystyle F_{k}(a)-\frac{1}{\varrho}F_{k+1}(a)\geq\left(\frac{1-q}{q}a-\frac{1-\varrho}{\varrho}\right)\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right]$		(21)
		$\displaystyle+\left(\frac{\varrho+1}{2L\varrho}-\frac{a(1-q)^{2}}{2q\mu}\right)\\|\nabla f(x^{k+1})\\|^{2}+\left(\frac{1}{\theta}-(1-q)a\right)\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle+\frac{\mu(1-q)a}{2}\\|x^{k+1}-z^{k+1}\\|^{2}.$

Proof.

Let $k\geq 0$ . It follows from the definition of $E_{k}(a)$ in (18) and the condition $\varrho\geq q$ that

E_{k}(a)-\frac{1}{\varrho}E_{k+1}(a)\geq\Big(f(x^{k})-f_{*}-\frac{1}{\varrho}\big(f(x^{k+1})-f_{*}\big)\Big)+\frac{a\mu}{2\varrho}\Big(q\|z^{k+1}-x^{*}\|^{2}-\|z^{k+2}-x^{*}\|^{2}\Big).

(22)

Using (14) to bound the first term on the right-hand side of (22), substituting (17) (with $k$ replaced by $(k+1)$ ) into the second term, and rearranging all terms immediately yields the desired result (20). Similarly, (21) can be derived using (19), (15) and (17). ∎

Remark 3.1.

Note that the only non-trivial term appearing in both (20) and (21) is $\langle x^{k+1}-z^{k+1},\nabla f(x^{k+1})\rangle$ , whose sign is indeterminate. In particular, this term vanishes when $x^{k+1}=z^{k+1}$ . In the subsequent analysis, we either select appropriate parameters to eliminate this term or bound it via the weighted AM-GM inequality.

Remark 3.2.

Equations (18) and (19) each define a Lyapunov function by combining a function-value-based error term with the squared distance $\|z^{k}-x^{*}\|^{2}$ . Specifically, (18) adopts the standard objective gap $f(x^{k})-f_{*}$ , whereas (19) employs the modified residual term $f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ . Note that the nonnegativity of $f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ follows directly from the $L$ -smoothness of $f$ . Alternatively, one may replace the function-value error term with its strongly convex counterpart $f(x^{k})-f_{*}-\frac{\mu}{2}\|x^{k}-x^{*}\|^{2}\geq 0$ , or adopt the tighter inequality $f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}-\frac{\mu}{2(1-\mu/L)}\|x^{k}-\tfrac{1}{L}\nabla f(x^{k})-x^{*}\|^{2}\geq 0$ as studied in [35, 32]. Nonnegativity of these two expressions hinges on the $\mu$ -strong convexity of $f$ . Moreover, both expressions explicitly couple the objective gap and squared distance, which conflicts with our aim to decouple these two quantities.

Based on (20) and (21), we subsequently present new convergence analysis for FGM, SC-OGM, TM method and AOR-HB, using the unified Lyapunov functions.

(i)

FGM [24, 25]: As established in the analysis of Section 2, FGM is a special case of UGM with $\theta=\sqrt{\kappa}$ and $q=1-\frac{1}{\sqrt{\kappa}}$ . Under this parameter setup, the identities $\frac{(1-q)^{2}}{2q\mu}=\frac{1}{2Lq}$ and $\frac{1}{\theta}=1-q$ hold; these equalities have been verified both analytically and via the symbolic computation script check_proof.ipynb. Set $a=1$ and $\varrho=q$ . Substituting these into (21), we obtain

F_{k}-\frac{1}{q}F_{k+1}\geq\frac{1}{2L}\|\nabla f(x^{k+1})\|^{2}+\frac{\mu(1-q)}{2}\|x^{k+1}-z^{k+1}\|^{2}\geq 0.

From this inequality, we conclude that Nesterov’s FGM attains linear convergence with convergence rate $1-1/\sqrt{\kappa}$ .

(ii)

SC-OGM [27]: SC-OGM is a special case of UGM with $\gamma=\frac{\sqrt{8\kappa+1}+3}{2\kappa-2}$ , $q=\frac{1}{\gamma+1}$ and $\theta=\frac{\gamma+1}{\gamma}$ . With these parameter choices, the equalities $\frac{\gamma+2}{L}=\frac{\gamma^{2}}{(\gamma+1)\mu}$ , $\frac{q+1}{2Lq}=\frac{(1-q)^{2}}{2q\mu}$ and $\frac{1}{\theta}=1-q$ hold. Again, these identities have been verified both analytically and using the symbolic computation script check_proof.ipynb. Set $a=1$ and $\varrho=q$ . Substituting these into (21), we obtain

$F_{k}-\frac{1}{q}F_{k+1}\geq\frac{\mu(1-q)}{2}\|x^{k+1}-z^{k+1}\|^{2}\geq 0.$

Hence, SC-OGM achieves linear convergence with rate $\frac{1}{1+\gamma}\sim 1-\sqrt{2/\kappa}$ .

(iii)

TM method [33]: The TM method is a special instance of UGM with $q=\rho=1-\frac{1}{\sqrt{\kappa}}$ and $\theta=\frac{\rho}{2(1-\rho)}$ . Then, using $\frac{1}{\theta}=\frac{2(1-q)}{q}$ and applying the weighted AM-GM inequality, we obtain

\left(\frac{1}{\theta}-(1-q)a\right)\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\geq-\frac{\mu(1-q)a}{2}\|x^{k+1}-z^{k+1}\|^{2}-\frac{1-q}{2\mu a}\left(\frac{2}{q}-a\right)^{2}\|\nabla f(x^{k+1})\|^{2}.

Furthermore, invoking (7) with $x=x^{k+1}$ and $y=x^{*}$ , and using $\|\nabla f(x^{k+1})\|\leq L\|x^{k+1}-x^{*}\|$ , we derive

\langle\nabla f(x^{k+1}),x^{k+1}-x^{*}\rangle-\frac{\mu}{2}\|x^{k+1}-x^{*}\|^{2}\geq\left(\frac{1}{\mu+L}+\frac{\mu(L-\mu)}{2(\mu+L)L^{2}}\right)\|\nabla f(x^{k+1})\|^{2}\geq\frac{1}{\mu+L}\|\nabla f(x^{k+1})\|^{2}.

For convenience, we define

	$\displaystyle A(a,\varrho)$	$\displaystyle:=\frac{\varrho+1}{2L\varrho}-\frac{a(1-q)^{2}}{2q\mu}-\frac{1-q}{2\mu a}\left(\frac{2}{q}-a\right)^{2}+\frac{1}{\mu+L}\left(\frac{(1-q)a}{q}-\frac{1-\varrho}{\varrho}\right)$		(23)
		$\displaystyle=\left(\frac{1}{2L}+\frac{1}{\mu+L}+\frac{2(1-q)}{\mu q}\right)+\left(\frac{1}{2L}-\frac{1}{\mu+L}\right)\frac{1}{\varrho}-\left[\frac{(L-\mu)(1-q)}{2q\mu(\mu+L)}a+\frac{2(1-q)}{\mu q^{2}}\frac{1}{a}\right].$		(24)

If inequality

\frac{1-q}{q}a-\frac{1-\varrho}{\varrho}\geq 0,

(25)

holds, then from (21) we obtain

F_{k}(a)-\frac{1}{\varrho}F_{k+1}(a)\geq A(a,\varrho)\|\nabla f(x^{k+1})\|^{2},

(26)

where $A(a,\varrho)$ is defined in (23). Next, we show that there exists a pair $(a,\varrho)$ satisfying (25) and such that $A(a,\varrho)\geq 0$ . We focus on the nontrivial case $L>\mu$ . First, we set $a_{*}=2\sqrt{\frac{L+\mu}{L-\mu}\frac{1}{q}}$ to minimize the expression $\frac{(L-\mu)(1-q)}{2q\mu(\mu+L)}a+\frac{2(1-q)}{\mu q^{2}}\frac{1}{a}$ . Since $q=1-\frac{1}{\sqrt{\kappa}}<\frac{L-\mu}{L+\mu}$ , it follows that $a_{*}\geq\frac{2(L+\mu)}{L-\mu}>2$ . Substituting the variable transformation $t=1/\sqrt{\kappa}$ into (24), along with the relations $q=1-t$ and $a_{*}=2\sqrt{\frac{1+t^{2}}{(1+t)(1-t)^{2}}}$ , and performing elementary algebraic computations, we obtain

A(a_{*},\varrho)=\frac{1}{\mu}\left(\frac{t^{2}}{2}+\frac{t^{2}}{1+t^{2}}+\frac{2t}{1-t}-\frac{2t\sqrt{1+t}}{(1-t)\sqrt{1+t^{2}}}-\frac{t^{2}(1-t^{2})}{2(1+t^{2})}\frac{1}{\varrho}\right)=\frac{t^{2}}{\mu}\Big(P(t)-\frac{1}{\varrho}Q(t)\Big).

(27)

where $P(t):=\frac{1}{2}+\frac{1}{1+t^{2}}-\frac{2}{\sqrt{1+t^{2}}(\sqrt{1+t}+\sqrt{1+t^{2}})}$ and $Q(t):=\frac{1-t^{2}}{2(1+t^{2})}$ . Next, we choose $\varrho_{*}$ such that $A(a_{*},\varrho_{*})=0$ , i.e., $\varrho_{*}=Q(t)\big/P(t)$ . From Lemma C.1, we have $\frac{1}{2}\leq P(t)\leq\frac{1+t}{2(1+t^{2})}$ . Hence, via elementary algebraic calculations, we derive $q=1-t\leq\varrho_{*}\leq\frac{1-t^{2}}{1+t^{2}}<1$ . Since $a_{*}>1$ and $\varrho_{*}\geq q$ , we get $\frac{1-q}{q}a_{*}-\frac{1-\varrho_{*}}{\varrho_{*}}\geq 0$ , which verifies that the condition (25) holds for $(a,\varrho)=(a_{*},\varrho_{*})$ . Substituting $(a,\varrho)=(a_{*},\varrho_{*})$ into (26) and noting that $A(a_{*},\varrho_{*})=0$ , we derive the linear convergence result $F_{k+1}(a_{*})\leq\varrho_{*}F_{k}(a_{*})$ . Finally, we have

\varrho_{*}=\frac{Q(t)}{P(t)}=1-\frac{2(t^{2}+1)\sqrt{1+t}+2(t^{2}-1)\sqrt{1+t^{2}}}{(t^{2}+3)\sqrt{1+t}+(t^{2}-1)\sqrt{1+t^{2}}}\sim 1-\frac{t+\frac{11}{4}t^{2}+\mathcal{O}(t^{3})}{2+\frac{3}{2}t+\frac{9}{8}t^{2}+\mathcal{O}(t^{3})}\sim 1-\frac{t}{2}=1-\frac{1}{2\sqrt{\kappa}},

(28)

where the Taylor expansion $\sqrt{1+v}=1+\frac{v}{2}-\frac{v^{2}}{8}+\mathcal{O}(v^{3})$ is adopted for the asymptotic approximation. Accordingly, we establish a Q-linear convergence rate of $1-{1}/{(2\sqrt{\kappa})}$ , while the original result in [33] achieves an R-linear convergence rate of $1-{2}/{\sqrt{\kappa}}$ . Note that the accelerated convergence behavior is retained. Finally, we note that all algebraic identities: the expression for $A(a,\varrho)$ in (23)-(24), the form of $A(a_{*},\varrho)$ in (27), and the alternative representation of $\varrho_{*}$ in (28) have been formally verified in the script check_proof.ipynb.

(iv)

AOR-HB [35]: From the discussion in Section 2, AOR-HB is a special instance of UGM with parameters $q=\frac{2\kappa+2\sqrt{\kappa}-1+\sqrt{4\sqrt{\kappa}+5}}{2(\sqrt{\kappa}+1)^{2}}$ and $\theta=\frac{\kappa}{2}\cdot\frac{2\sqrt{\kappa}+3-\sqrt{4\sqrt{\kappa}+5}}{\kappa+\sqrt{4\sqrt{\kappa}+5}-2}$ . Lemma B.7 guarantees $q\in[\frac{3}{4},1)$ and $\theta>0$ . Let $a=\frac{2q-1}{\kappa(1-q)^{2}}$ and $\varrho=q$ . One can manually verify the relation $\frac{1}{\theta}=\frac{2q-1}{\kappa(1-q)}=a(1-q)$ . Alternatively, these equalities are also validated via symbolic computation in check_proof.ipynb. Hence, we have

\frac{1}{L}-\frac{a(1-q)^{2}}{q\mu}=\frac{1}{L}-\frac{2q-1}{q\kappa\mu}=\frac{1}{L}\Big(1-\frac{2q-1}{q}\Big)=\frac{1-q}{qL}\text{~~and~~}a\mu(1-q)=\frac{\mu(2q-1)}{\kappa(1-q)}=\frac{\mu}{\theta}.

(29)

For the given $q$ , the following equivalence holds:

a=\frac{2q-1}{\kappa(1-q)^{2}}>1\text{~~if and only if~~}4\kappa^{7/2}+21\kappa^{3}+48\kappa^{5/2}+63\kappa^{2}+52\kappa^{3/2}+27\kappa+8\sqrt{\kappa}+1>0.

This equivalence is verified via symbolic computation in check_proof.ipynb. Given $\kappa>0$ , the polynomial on the right-hand side is obviously strictly positive, which directly yields $a>1$ . Recall our choice $\varrho=q$ . We thus have $\frac{1-q}{q}a-\frac{1-\varrho}{\varrho}=\frac{1-q}{q}(a-1)>0$ . Meanwhile, $\mu$ -strongly convexity of $f$ delivers the bound $\langle\nabla f(x^{k+1}),x^{k+1}-x^{*}\rangle\geq\mu\|x^{k+1}-x^{*}\|^{2}$ . This observation guarantees nonnegativity of the second term appearing on the right-hand side of (20). Combining the relations (20), (29), alongside the identity $\frac{1}{\theta}=a(1-q)$ , we deduce that

E_{k}(a)-\frac{1}{q}E_{k+1}(a)\geq\frac{1}{2L}\|\nabla f(x^{k})\|^{2}+\frac{1-q}{2qL}\|\nabla f(x^{k+1})\|^{2}+\frac{\mu}{2\theta}\|x^{k+1}-z^{k+1}\|^{2}\geq 0.

Hence, we have established the linear convergence result $E_{k+1}(a)\leq qE_{k}(a)$ . Furthermore, one may check that our selected $q$ satisfies $q\sim 1-1/\sqrt{\kappa}$ . In comparison, the original analysis of the AOR-HB algorithm in [35]⁴⁴ 4 The authors also mention a rate $1-1/\sqrt{\kappa}$ in their slides, yet the corresponding Lyapunov function is far more intricate than the one constructed here. establishes a linear convergence rate of $1-1/(2\sqrt{\kappa})$ .

4 UGM with varying parameters $q_{k}$ and $\theta_{k}$

Let $x^{0}=z^{0}\in\mathbb{R}^{n}$ . This section investigates the UGM framework (4) equipped with iteration-dependent parameters $q_{k}$ and $\theta_{k}$ :

\left\{\quad\begin{aligned} y^{k+1}&=x^{k}-\frac{1}{L}\nabla f(x^{k}),\\ w^{k+1}&=x^{k}-\frac{1}{\mu}\nabla f(x^{k}),\\ z^{k+1}&=q_{k}z^{k}+(1-q_{k})w^{k+1},\\ x^{k+1}&=\frac{\theta_{k}}{1+\theta_{k}}y^{k+1}+\frac{1}{1+\theta_{k}}z^{k+1}.\end{aligned}\right.

(30)

In particular, Subsection 4.1 presents simple iteration-varying parameter choices, while Subsections 4.2–4.3 further develop adaptive, problem-dependent parameter variants. First, using analogous arguments based on (15) and (17), we derive the following iterate estimates for the updates in (30).

Lemma 4.1.

For iterates generated by (30), the following two relations hold for all $k\geq 0$ :

	$\displaystyle f(x^{k})-f_{*}$	$\displaystyle-\frac{1}{2L}\\|\nabla f(x^{k})\\|^{2}-\frac{1}{\varrho}\left[f(x^{k+1})-f_{*}-\frac{1}{2L}\\|\nabla f(x^{k+1})\\|^{2}\right]\geq\frac{\varrho+1}{2L\varrho}\\|\nabla f(x^{k+1})\\|^{2}$		(31)
		$\displaystyle+\frac{1}{\theta_{k}}\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle-\frac{1-\varrho}{\varrho}\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right],$		(31)

and

	$\displaystyle\\|z^{k}-x^{*}\\|^{2}$	$\displaystyle-\frac{1}{q_{k}}\\|z^{k+1}-x^{}\\|^{2}=\frac{2(1-q_{k})}{\mu q_{k}}\langle x^{k}-x^{},\nabla f(x^{k})\rangle-\frac{1-q_{k}}{q_{k}}\\|x^{k}-x^{*}\\|^{2}$		(32)
		$\displaystyle+(1-q_{k})\\|x^{k}-z^{k}\\|^{2}-\frac{(1-q_{k})^{2}}{q_{k}\mu^{2}}\\|\nabla f(x^{k})\\|^{2}-\frac{2(1-q_{k})}{\mu}\langle\nabla f(x^{k}),x^{k}-z^{k}\rangle.$		(32)

Theorem 4.1.

Let $0<\varrho<1$ and $a_{k}>0$ satisfy $a_{k+1}\leq\frac{\varrho a_{k}}{q_{k+1}}$ for all $k\geq 0$ . Define the Lyapunov function $G_{k}(a_{k})$ by

G_{k}(a_{k}):=f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}+\frac{a_{k}\mu}{2}\|z^{k+1}-x^{*}\|^{2}.

(33)

Then, for the iterates generated by (30), it holds for all $k\geq 0$ that

$\displaystyle G_{k}(a_{k})-$	$\displaystyle\frac{1}{\varrho}G_{k+1}(a_{k+1})\geq\left(\frac{1-q_{k+1}}{q_{k+1}}a_{k}-\frac{1-\varrho}{\varrho}\right)\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right]$	(34)
	$\displaystyle+\left(\frac{\varrho+1}{2L\varrho}\!-\!\frac{a_{k}(1\!-\!q_{k+1})^{2}}{2q_{k+1}\mu}\right)\\|\nabla f(x^{k+1})\\|^{2}+\left(\frac{1}{\theta_{k}}\!-\!(1\!-\!q_{k+1})a_{k}\right)\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$
	$\displaystyle+\frac{\mu(1\!-\!q_{k+1})a_{k}}{2}\\|x^{k+1}\!-\!z^{k+1}\\|^{2}.$

Proof.

Let $k\geq 0$ . Since $a_{k+1}\leq\frac{\varrho a_{k}}{q_{k+1}}$ , we have

\frac{a_{k}\mu}{2}\|z^{k+1}-x^{*}\|^{2}-\frac{a_{k+1}\mu}{2\varrho}\|z^{k+2}-x^{*}\|^{2}\geq\frac{a_{k}\mu}{2}\left[\|z^{k+1}-x^{*}\|^{2}-\frac{1}{q_{k+1}}\|z^{k+2}-x^{*}\|^{2}\right]

(35)

Substituting (32) (with $k$ replaced by $k+1$ ) into the right-hand side of (35), summing both sides of the derived inequality with those of (31), and invoking the definition of $G_{k}(a_{k})$ from (33), we arrive at the target estimate (34). ∎

4.1 UGM-I: Parameter rule $(1-q_{k+1})a_{k}=1/\theta_{k}$

Consider the four terms on the right-hand side of (34). Among them and aside from their coefficients, the only term with an undetermined sign is “ $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ ”, whose coefficient is $\left(1/\theta_{k}\!-\!(1\!-\!q_{k+1})a_{k}\right)$ . We therefore set the coefficient preceding this term to zero, yielding $(1-q_{k+1})a_{k}=1/\theta_{k}$ , and select the parameters involved in the iterative scheme (30) and Lyapunov function (33) as follows:

UGM-I. Choose

\varrho\in(0,1)

and any sequence

\{a_{k},q_{k+1},\theta_{k}\}_{k\geq 0}

such that

a_{k}>0

\theta_{k}\in(0,1)

, and

q_{k+1}\in(0,1)

for all

k\geq 0

, and

\displaystyle a_{k+1}\leq\frac{\varrho a_{k}}{q_{k+1}},

(36a)

\displaystyle\frac{1-q_{k+1}}{q_{k+1}}a_{k}-\frac{1-\varrho}{\varrho}\geq 0,

(36b)

\displaystyle\frac{\varrho+1}{2L\varrho}-\frac{a_{k}(1\!-\!q_{k+1})^{2}}{2q_{k+1}\mu}+\left(\frac{1-q_{k+1}}{q_{k+1}}a_{k}-\frac{1-\varrho}{\varrho}\right)\frac{1}{\mu+L}\geq 0,

(36c)

\displaystyle 1/\theta_{k}=(1-q_{k+1})a_{k}.

(36d)

Theorem 4.2.

For the iterates generated by (30) with parameters satisfying (36), it holds for all $k\geq 0$ that $G_{k+1}\leq\varrho G_{k}$ .

Proof.

For parameters satisfying (36), inequality (34) holds. Furthermore, since $f$ is $L$ -smooth and $\mu$ -strongly convex, we have $\langle\nabla f(x^{k+1}),x^{k+1}-x^{*}\rangle-\frac{\mu}{2}\|x^{k+1}-x^{*}\|^{2}\geq\frac{1}{\mu+L}\|\nabla f(x^{k+1})\|^{2}$ . By further exploiting the parameter conditions in (36), we conclude that the right-hand side of (34) is nonnegative, which establishes the desired result. ∎

We next provide concrete parameter choices that satisfy the conditions in (36).

UGM-I (explicit). Choose

\varrho

satisfying

\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1}\leq\varrho<\frac{\kappa-1}{\kappa+1}

, and set

a_{0}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}

. The iteration-dependent parameters are updated via the following explicit rules:

\displaystyle\theta_{k}=\dfrac{\varrho}{1-\varrho}+\dfrac{1}{a_{k}},

(37a)

\displaystyle a_{k+1}=\varrho a_{k}+1-\varrho,

(37b)

\displaystyle q_{k+1}=\dfrac{\varrho a_{k}}{\varrho a_{k}+1-\varrho}.

(37c)

To verify that the parameters in (37) satisfy the constraints in (36), we first introduce a useful lemma.

Lemma 4.2.

Let $\{a_{k}\}_{k\geq 0}$ be a sequence satisfying $a_{k+1}=\varrho a_{k}+1-\varrho$ for all $k\geq 0$ , with $a_{0}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}$ . If $\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1}\leq\varrho<\frac{\kappa-1}{\kappa+1}$ , then for all $k\geq 0$ , we have

a_{k}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}.

(38)

Moreover, for $q_{k+1}=\dfrac{\varrho a_{k}}{\varrho a_{k}+1-\varrho}$ , we have $\min\left\{\varrho,\frac{\varrho a_{0}}{\varrho a_{0}+1-\varrho}\right\}\leq q_{k+1}<1$ .

Proof.

For $q_{k+1}=\tfrac{\varrho a_{k}}{\varrho a_{k}+1-\varrho}$ , the inequality $q_{k+1}<1$ trivially holds due to $\varrho<1$ . Furthermore, solving the recurrence relation $a_{k+1}=\varrho a_{k}+1-\varrho$ for all $k\geq 0$ yields the closed-form expression $a_{k}=1+\varrho^{k}(a_{0}-1)$ . The subsequent proof proceeds by considering two separate cases: $a_{0}\geq 1$ and $a_{0}<1$ .

(i)

If $a_{0}\geq 1$ , the sequence $\{a_{k}\}$ is decreasing and converges to $1$ , which implies $a_{k}\geq 1$ for all $k\geq 0$ . The condition $\varrho\geq\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1}$ ensures $\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}\leq 1$ (see Lemma C.2 for a rigorous verification). Consequently, $a_{k}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}$ holds for all $k\geq 0$ . In addition, since the function $x\mapsto\frac{\varrho x}{\varrho x+1-\varrho}$ is monotonically increasing and $a_{k}\geq 1$ , we further obtain $q_{k+1}\geq\varrho$ .
(ii)

If $a_{0}<1$ , the sequence $\{a_{k}\}$ is increasing and converges to $1$ , so $a_{k}\geq a_{0}$ for all $k\geq 0$ . This further implies $a_{k}\geq a_{0}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}$ and $q_{k+1}\geq\frac{\varrho a_{0}}{\varrho a_{0}+1-\varrho}$ .

Thus, we have shown that $a_{k}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}$ and $q_{k+1}\geq\min\left\{\varrho,\frac{\varrho a_{0}}{\varrho a_{0}+1-\varrho}\right\}$ for all $k\geq 0$ . ∎

Lemma 4.3.

The parameters given in (37) satisfy (36).

Proof.

First, direct computation using (37c) yields $\frac{1-q_{k+1}}{q_{k+1}}a_{k}=\frac{1-\varrho}{\varrho}$ , which verifies condition (36b). Second, it is straightforward to verify from (37) that $a_{k+1}=\varrho a_{k}/q_{k+1}$ and $(1-q_{k+1})a_{k}=1/\theta_{k}$ , which validates (36a) and (36d). Finally, using the foregoing relations and the update rules in (37), one can directly derive that

\text{LHS of~}\eqref{con-3}=\frac{\varrho+1}{2L\varrho}-\frac{1-\varrho}{2\varrho\mu}\frac{1-\varrho}{1-\varrho+\varrho a_{k}}\geq\frac{\varrho+1}{2L\varrho}-\frac{1-\varrho}{2\varrho\mu}\frac{\varrho+1}{\kappa(1-\varrho)}=0,

where the inequality follows from (38). This verifies condition (36c). ∎

4.2 UGM-II: $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ -aware parameter rule

In UGM-I, we choose $\theta_{k}$ so that the coefficient of $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ vanishes, since this inner-product term carries an indefinite sign. A natural question arises: if, under suitable conditions, we can determine the sign of $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ , do we gain greater flexibility in selecting $\{\theta_{k}\}_{k\geq 0}$ ? Suppose $\theta_{k}\neq 0$ for all $k\geq 0$ . Substituting $x=x^{k}$ and $y=x^{k+1}$ into (5) and invoking (30), we can derive

\displaystyle f(x^{k})-f(x^{k+1})\geq\frac{1}{2L}\|\nabla f(x^{k})\|^{2}+\frac{1}{2L}\|\nabla f(x^{k+1})\|^{2}+\frac{1}{\theta_{k}}\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle.

(39)

Hence, the term $\frac{1}{\theta_{k}}\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle$ is regarded as a “good” term whenever it is nonnegative, since (39) then guarantees a reduction in the objective function value.

Below, we derive two estimates for $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle$ . First, we have

\mu\|x^{k+1}-z^{k+1}\|^{2}\leq\langle\nabla f(x^{k+1})-\nabla f(z^{k+1}),x^{k+1}-z^{k+1}\rangle\leq L\|x^{k+1}-z^{k+1}\|^{2}.

(40)

Second, rearranging $x^{k+1}=\frac{\theta_{k}}{1+\theta_{k}}y^{k+1}+\frac{1}{1+\theta_{k}}z^{k+1}$ yields $x^{k+1}-z^{k+1}=\frac{\theta_{k}}{1+\theta_{k}}(y^{k+1}-z^{k+1})$ . Substituting this identity into (40) and rearranging the terms, we obtain

	$\displaystyle$	$\displaystyle\frac{\mu\theta_{k}^{2}}{(1+\theta_{k})^{2}}\\|y^{k+1}-z^{k+1}\\|^{2}+\frac{\theta_{k}}{1+\theta_{k}}\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\rangle$		(41)
	$\displaystyle\leq$	$\displaystyle\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\leq\frac{L\theta_{k}^{2}}{(1+\theta_{k})^{2}}\\|y^{k+1}-z^{k+1}\\|^{2}+\frac{\theta_{k}}{1+\theta_{k}}\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\rangle.$		(41)

Lemma 4.4.

Let $\bar{\theta}>\underline{\theta}>0$ be fixed constants. Suppose $\theta_{k}\in[\underline{\theta},\bar{\theta}]$ for all $k\geq 0$ . Then the following hold:

(i)

If $\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\rangle\leq-\frac{L\bar{\theta}}{1+\bar{\theta}}\|y^{k+1}-z^{k+1}\|^{2}$ , then $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\leq 0$ ;
(ii)

If $\langle\nabla f(z^{k+1}),z^{k+1}-y^{k+1}\rangle\leq\frac{\mu\underline{\theta}}{1+\underline{\theta}}\|y^{k+1}-z^{k+1}\|^{2}$ , then $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\geq 0$ .

Proof.

Fix $k\geq 0$ . Since $\theta_{k}\leq\bar{\theta}$ , the condition of (i) implies

\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\rangle\leq-\frac{L\bar{\theta}}{1+\bar{\theta}}\|y^{k+1}-z^{k+1}\|^{2}\leq-\frac{L\theta_{k}}{1+\theta_{k}}\|y^{k+1}-z^{k+1}\|^{2}.

It then follows from the second inequality in (41) that $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\leq 0$ . Similarly, since $\theta_{k}\geq\underline{\theta}$ , the condition of (ii) implies

\langle\nabla f(z^{k+1}),z^{k+1}-y^{k+1}\rangle\leq\frac{\mu\underline{\theta}}{1+\underline{\theta}}\|y^{k+1}-z^{k+1}\|^{2}\leq\frac{\mu\theta_{k}}{1+\theta_{k}}\|y^{k+1}-z^{k+1}\|^{2}.

Then, $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\geq 0$ follows from the first inequality from (41). ∎

In view of (34), if $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\geq 0$ , then it is desirable to choose $\theta_{k}$ satisfying $1/\theta_{k}\geq(1-q_{k+1})a_{k}$ . Furthermore, (39) suggests that a smaller $\theta_{k}$ may be preferable, as it tends to yield a larger decrease in the objective function value. Conversely, if $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\leq 0$ , then it is desirable to choose $\theta_{k}$ so that the reverse inequality $1/\theta_{k}\leq(1-q_{k+1})a_{k}$ holds. On the other hand, the parameters $\{(q_{k+1},a_{k})\}_{k\geq 0}$ and $\varrho$ are selected following the scheme UGM-I (explicit) presented in (37). Recall that from (37) and Lemma 4.2, for all $k\geq 0$ we have

\min\left\{\varrho,\frac{\varrho a_{0}}{\varrho a_{0}+1-\varrho}\right\}\leq q_{k+1}<1.

(42)

Then, we have the following lemma.

Lemma 4.5.

Choose $\{(q_{k+1},a_{k})\}_{k\geq 0}$ and $\varrho$ by following the scheme UGM-I (explicit) presented in (37). Assume $\bar{\theta}>\underline{\theta}>0$ be fixed constants satisfying $\underline{\theta}\leq\frac{\varrho}{1-\varrho}$ and $\bar{\theta}\geq\max\left\{\frac{1}{1-\varrho},\frac{\varrho a_{0}+1-\varrho}{(1-\varrho)a_{0}}\right\}$ . For $k\geq 0$ , define $\theta_{k}$ via

\theta_{k}=\left\{\begin{array}[]{ll}\bar{\theta},&\text{if }\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\rangle\leq-\frac{L\bar{\theta}}{1+\bar{\theta}}\|y^{k+1}-z^{k+1}\|^{2},\\ \vskip 3.0pt plus 1.0pt minus 1.0pt\underline{\theta},&\text{if }\langle\nabla f(z^{k+1}),z^{k+1}-y^{k+1}\rangle\leq\frac{\mu\underline{\theta}}{1+\underline{\theta}}\|y^{k+1}-z^{k+1}\|^{2},\\ \vskip 3.0pt plus 1.0pt minus 1.0pt\dfrac{\varrho}{1-\varrho}+\dfrac{1}{a_{k}},&\text{otherwise.}\end{array}\right.

(43)

Then, we have $\theta_{k}\in[\underline{\theta},\bar{\theta}]$ for all $k\geq 0$ .

Proof.

By the definition of $\theta_{k}$ in (43), we only need to treat the “otherwise” case. Note that in the “otherwise” case, from (37c) we have $\theta_{k}=\frac{\varrho}{1-\varrho}\!+\!\frac{1}{a_{k}}=\frac{\varrho}{1-\varrho}\frac{1}{q_{k+1}}$ . Then, it follows from (42) that

\underline{\theta}\leq\frac{\varrho}{1-\varrho}\stackrel{{\scriptstyle\eqref{jy-0718b}}}{{<}}\theta_{k}\equiv\frac{\varrho}{1-\varrho}\frac{1}{q_{k+1}}\stackrel{{\scriptstyle\eqref{jy-0718b}}}{{\leq}}\max\left\{\frac{1}{1-\varrho},\frac{\varrho a_{0}+1-\varrho}{(1-\varrho)a_{0}}\right\}\leq\bar{\theta}.

This completes the proof. ∎

Based on Lemma 4.5, we are now ready to present our $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ -aware parameter rule as follows.

UGM-II:

\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle

-aware parameter rule. Choose parameters

(\varrho,a_{0},\underline{\theta},\bar{\theta})

such that

0<\underline{\theta}\leq\frac{\varrho}{1-\varrho}

\bar{\theta}\geq\max\left\{\frac{1}{1-\varrho},\frac{\varrho a_{0}+1-\varrho}{(1-\varrho)a_{0}}\right\}

\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1}\leq\varrho<\frac{\kappa-1}{\kappa+1}

a_{0}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}

. For all

k\geq 0

, compute

\theta_{k}

a_{k+1}

and

q_{k+1}

via (43), (37b) and (37c), respectively.

Theorem 4.3.

For the iterations generated by (30) with the parameters specified in UGM-II, we have $G_{k+1}\leq\varrho G_{k}$ for all $k\geq 0$ .

Proof.

It is straightforward to verify that, in all three cases defining $\theta_{k}$ in (43), the right-hand side of (34) is nonnegative, which gives the desired conclusion. ∎

4.3 UGM-III: $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ - and $\|x^{k+1}-z^{k+1}\|^{2}$ -aware parameter rule

We further note the term $\|x^{k+1}-z^{k+1}\|^{2}$ on the right-hand side of (34) also depends implicitly on the choice of $\theta_{k}$ , since $x^{k+1}-z^{k+1}=\frac{\theta_{k}}{1+\theta_{k}}(y^{k+1}-z^{k+1})$ . In this subsection, we select $\theta_{k}$ by relating it to this previously neglected term; specifically, we seek to find $\theta_{k}$ such that

\left(1/\theta_{k}\!-\!(1\!-\!q_{k+1})a_{k}\right)\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle+\frac{\mu(1\!-\!q_{k+1})a_{k}}{2}\|x^{k+1}-z^{k+1}\|^{2}\geq 0.

(44)

Next, we focus on the case $1/\theta_{k}-(1\!-\!q_{k+1})a_{k}\geq 0$ .

Lemma 4.6.

Let $\{(a_{k},q_{k+1})\}_{k\geq 0}$ be any sequence satisfying $a_{k}>0$ and $q_{k+1}\in(0,1)$ for all $k\geq 0$ . Define $A_{k}:=(1-q_{k+1})a_{k}$ , $B_{k}:=\|y^{k+1}-z^{k+1}\|^{2}$ , and $C_{k}:=\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\rangle$ for all $k\geq 0$ . Suppose that $\sup_{k\geq 0}A_{k}<\infty$ and $\underline{\theta}$ satisfies $0<\underline{\theta}\leq 1/\sup_{k\geq 0}A_{k}$ . For any $k\geq 0$ , if $B_{k}=0$ , then inequality (44) holds trivially, and if $B_{k}>0$ then

\tilde{\theta}_{k}:=\dfrac{-2C_{k}}{\mu B_{k}+(1-A_{k})C_{k}+\sqrt{\mu^{2}B_{k}^{2}+2\mu B_{k}C_{k}+(1+A_{k})^{2}C_{k}^{2}}}\leq\frac{1}{A_{k}},

(45)

and inequality (44) holds for any $\theta_{k}$ satisfying $\max\{\underline{\theta},\tilde{\theta}_{k}\}\leq\theta_{k}\leq 1/A_{k}$ .

Proof.

Fix any $k\geq 0$ . Recall that we always have $A_{k}>0$ and the identity $x^{k+1}-z^{k+1}=\frac{\theta_{k}}{1+\theta_{k}}(y^{k+1}-z^{k+1})$ . If $B_{k}=0$ , then $y^{k+1}=z^{k+1}$ , and hence $x^{k+1}-z^{k+1}=0$ , which implies that (44) holds. Next, we assume $B_{k}>0$ . From the definitions of $A_{k}$ , $B_{k}$ , $C_{k}$ , the relation $x^{k+1}-z^{k+1}=\frac{\theta_{k}}{1+\theta_{k}}(y^{k+1}-z^{k+1})$ , and the left-hand side inequality in (41), we obtain

	$\displaystyle\text{LHS of~}\eqref{sat-con}=$	$\displaystyle\left(\frac{1}{\theta_{k}}\!-\!A_{k}\right)\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle+\frac{\mu A_{k}}{2}\frac{\theta_{k}^{2}}{(1+\theta_{k})^{2}}\\|y^{k+1}-z^{k+1}\\|^{2}$
	$\displaystyle\geq$	$\displaystyle\left(\frac{1}{\theta_{k}}\!-\!A_{k}\right)\left(\frac{\mu\theta_{k}^{2}B_{k}}{(1+\theta_{k})^{2}}\!+\!\frac{\theta_{k}C_{k}}{1+\theta_{k}}\right)+\frac{\mu A_{k}B_{k}}{2}\frac{\theta_{k}^{2}}{(1+\theta_{k})^{2}}=\frac{I_{1}(\theta_{k})}{(1+\theta_{k})^{2}},$

where $I_{1}(\theta):=-\frac{1}{2}A_{k}(2C_{k}+\mu B_{k})\theta^{2}+(C_{k}-A_{k}C_{k}+\mu B_{k})\theta+C_{k}$ is a quadratic function of $\theta$ . Note that $I_{1}(1/A_{k})=\frac{\mu B_{k}}{2A_{k}}>0$ , and the axis of symmetry of $I_{1}(\theta)$ is $\frac{C_{k}-A_{k}C_{k}+\mu B_{k}}{A_{k}(2C_{k}+\mu B_{k})}$ . Next, we split the discussion into three mutually disjoint cases.

(i)

$2C_{k}+\mu B_{k}>0$ . In this case, the coefficient of the quadratic term in $I_{1}(\theta)$ is negative, and straightforward calculations show that $\tilde{\theta}_{k}$ defined in (45) is the smaller root of the quadratic equation $I_{1}(\theta)=0$ . Since $I_{1}(1/A_{k})>0$ , we conclude that $\tilde{\theta}_{k}\leq 1/A_{k}$ , and for all $\theta_{k}\in[\tilde{\theta}_{k},1/A_{k}]$ , we have $I_{1}(\theta_{k})\geq 0$ , hence (44) follows;
(ii)

$2C_{k}+\mu B_{k}<0$ . In this case, the coefficient of the quadratic term in $I_{1}(\theta)$ is positive, and straightforward calculations show that $\tilde{\theta}_{k}$ defined in (45) is the larger root of the quadratic equation $I_{1}(\theta)=0$ . Since $C_{k}<0$ and $A_{k}>0$ , we have $C_{k}-A_{k}C_{k}+\mu B_{k}>2C_{k}+\mu B_{k}$ . Combined with $2C_{k}+\mu B_{k}<0$ , this implies that the axis of symmetry of $I_{1}(\theta)$ , namely $\frac{C_{k}-A_{k}C_{k}+\mu B_{k}}{A_{k}(2C_{k}+\mu B_{k})}$ , is bounded above $1/A_{k}$ . Recalling $I_{1}(1/A_{k})>0$ , we further obtain $\tilde{\theta}_{k}\leq 1/A_{k}$ , and $I_{1}(\theta_{k})\geq 0$ for all $\theta_{k}\in[\tilde{\theta}_{k},1/A_{k}]$ , so (44) holds;
(iii)

$2C_{k}+\mu B_{k}=0$ . In this case, $C_{k}<0$ and straightforward calculations yields $\tilde{\theta}_{k}=1/(A_{k}+1)$ , which solves the linear equation $I_{1}(\theta)=\frac{1}{2}\mu B_{k}\big((A_{k}+1)\theta-1\big)=0$ . Hence, $I_{1}(\theta_{k})\geq 0$ for all $\theta_{k}\in[\tilde{\theta}_{k},1/A_{k}]$ , and thus (44) follows.

Thus, we complete the proof. ∎

Based on Lemma 4.6 and in analogy with (43), we now propose the following more flexible update rule for selecting $\theta_{k}$ :

\theta_{k}\in\left\{\begin{array}[]{ll}\{\bar{\theta}\},&\text{if }\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\rangle\leq-\frac{L\bar{\theta}}{1+\bar{\theta}}\|y^{k+1}-z^{k+1}\|^{2},\\ \vskip 3.0pt plus 1.0pt minus 1.0pt\{\underline{\theta}\},&\text{if }\langle\nabla f(z^{k+1}),z^{k+1}-y^{k+1}\rangle\leq\frac{\mu\underline{\theta}}{1+\underline{\theta}}\|y^{k+1}-z^{k+1}\|^{2},\\ \vskip 3.0pt plus 1.0pt minus 1.0pt\left[\max\big\{\underline{\theta},\tilde{\theta}_{k}\big\},\dfrac{\varrho}{1-\varrho}+\dfrac{1}{a_{k}}\right],&\text{otherwise.}\end{array}\right.

(46)

We are now ready to present our $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ - and $\|x^{k+1}-z^{k+1}\|^{2}$ -aware parameter rule as follows.

UGM-III:

\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle

\|x^{k+1}-z^{k+1}\|^{2}

-aware parameter rule. Choose parameters

(\varrho,a_{0},\underline{\theta},\bar{\theta})

such that

0<\underline{\theta}\leq\frac{\varrho}{1-\varrho}

\bar{\theta}\geq\max\left\{\frac{1}{1-\varrho},\frac{\varrho a_{0}+1-\varrho}{(1-\varrho)a_{0}}\right\}

\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1}\leq\varrho<\frac{\kappa-1}{\kappa+1}

a_{0}\geq\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}

. For all

k\geq 0

, compute

\theta_{k}

a_{k+1}

and

q_{k+1}

via (46), (37b) and (37c), respectively.

Theorem 4.4.

For the iterations generated by (30) with the parameters specified in UGM-III, we have $G_{k+1}\leq\varrho G_{k}$ for all $k\geq 0$ .

Proof.

It is straightforward to verify that, in all three cases defining $\theta_{k}$ in (46), the right-hand side of (34) is nonnegative, which gives the desired conclusion. ∎

5 Numerical experiments

This section presents three groups of numerical experiments to comprehensively validate the performance and practical merits of the proposed algorithms. First, we test the proposed methods on unconstrained quadratic optimization problems to explore their favorable application scenarios and analyze the functional role of the parameter $\varrho$ . Second, we evaluate the algorithm performance on the logistic regression task, verifying that our methods achieve competitive results in practical real-world learning problems. Third, we conduct experiments on softmax regression to investigate whether the iteration efficiency advantages of UGM-II and UGM-III can be converted into CPU time savings in distributed environments, leveraging cached-logit computation and scalar aggregation techniques. All experiments are implemented in Python 3.11.0 and run on a personal computer equipped with an AMD Ryzen 7 5800H processor, Radeon Graphics, and 16 GB of RAM.

5.1 Unconstrained quadratic optimization

For the first experiment, we consider the unconstrained optimization of quadratic functions. Let $A\in\mathbb{R}^{n\times n}$ be a symmetric positive definite matrix and $b\in\mathbb{R}^{n}$ . Consider minimizing $f(x)=\frac{1}{2}x^{\top}Ax-b^{\top}x$ over $x\in\mathbb{R}^{n}$ . We evaluate four accelerated gradient methods in this setting, including the classical FGM algorithm [25] and our three proposed schemes, namely UGM-I, UGM-II, and UGM-III, which are developed in Section 4. To simplify the experimental setup, we construct $A$ and $b$ in a controlled manner. We first randomly generate an optimal solution $x^{*}$ whose entries are uniformly sampled from $[0,1]$ , and then set $b=Ax^{*}$ . The matrix $A$ is defined as the diagonal matrix $A=\operatorname{diag}(\lambda_{1},\lambda_{2},\dots,\lambda_{n})$ with ordered eigenvalues $\lambda_{1}\leq\lambda_{2}\leq\cdots\leq\lambda_{n}$ . For a fixed dimension $n$ , we set the extreme eigenvalues to $\lambda_{1}=\mu$ and $\lambda_{n}=L$ , which ensures that $f$ is $\mu$ -strongly convex and $L$ -smooth. The intermediate eigenvalues $\lambda_{2},\ldots,\lambda_{n-1}$ are sampled from five predefined distributions, rescaled to the interval $[\mu,L]$ , and finally sorted in ascending order. The detailed configurations of the five eigenvalue distributions are presented as follows.

(Case 1)

Geometric progression: set $\lambda_{i}=\mu r^{\,i-1}$ for $i=2,\ldots,n-1$ with $r=(L/\mu)^{1/(n-1)}$ .
(Case 2)

U-shaped distribution: generate $n-2$ $i.i.d.$ samples from $\texttt{Beta}(0.01,0.01)$ .
(Case 3)

Central-concentrated distribution: generate $n-2$ $i.i.d.$ samples from $\texttt{Beta}(50,50)$ .
(Case 4)

Lower-end skewed distribution: generate $n-2$ $i.i.d.$ samples from $\texttt{Beta}(1,20)$ .
(Case 5)

Upper-end skewed distribution: generate $n-2$ $i.i.d.$ samples from $\texttt{Beta}(20,1)$ .

For Cases 2-5, all samples are linearly rescaled to the interval $[\mu,L]$ via the mapping $x\mapsto\mu+(L-\mu)x$ , and the resulting values are then sorted in ascending order to form the intermediate eigenvalues. In this experiment, we set $n=1000$ . The five eigenvalue distributions are visualized in Figure 1.

Refer to caption — Figure 1: Illustration of eigenvalue distributions for Cases 1-5 with $\mu=1$ and $L=10^{3}$ .

We consider five combinations of strong convexity and smoothness parameters $(\mu,L)$ : $(1,10^{2})$ , $(1,10^{3})$ , $(1,10^{6})$ , $(10,10^{2})$ , and $(10^{2},10^{4})$ . These configurations produce condition numbers $\kappa=L/\mu$ ranging from $10$ to $10^{6}$ . To ensure fair comparison across different conditioning levels, the maximum iteration number $K$ is adaptively determined according to $\kappa$ . Specifically, we specify three anchor points $(\log_{10}\kappa,K)=(1,50),\;(3,300),\;(6,5000)$ and adopt linear interpolation on the $\log_{10}\kappa$ scale to compute the iteration budget for each tested condition number.

For the proposed UGMs, we adopt the following parameter settings. We set $\varrho=1-1/\sqrt{\kappa}$ for all the three UGMs, $q_{0}=\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}$ for UGM-I and $q_{0}=\varrho$ for both UGM-II and UGM-III. The lower bound parameter $\underline{\theta}$ is set to $1$ for $\kappa\geq 10^{3}$ and $0.01$ otherwise, which satisfies the theoretical constraint $\underline{\theta}<\frac{\varrho}{1-\varrho}$ . Furthermore, we set $a_{0}=\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}$ and define the upper bound parameter $\bar{\theta}=\max\left\{\frac{1}{1-\varrho},\frac{\varrho a_{0}+1-\varrho}{(1-\varrho)a_{0}}\right\}=\sqrt{\kappa}$ . All compared algorithms are initialized at the origin.

Comparison results.

Each $(\mu,L)$ configuration is run five times with independent random seeds. For each eigenvalue distribution and each tested $(\mu,L)$ pair, we report the objective suboptimality $f(x^{k})-f_{*}$ against iteration count and CPU time (s), alongside the evolution of the extrapolation parameter $\theta_{k}$ for UGM-II and UGM-III. Numerical results for Cases 1–5 are presented respectively in Figures 2–6.

In terms of iteration counts, UGM‑II and UGM‑III attain slightly lower suboptimality than FGM. However, since UGM‑II and UGM‑III require one extra gradient evaluation per iteration, they do not offer a clear advantage in CPU time in most test cases. Some exceptions include Figure 4 (column (c)), and Figure 6 (column (a), (c), (d)), where the faster convergence of UGM‑II and UGM‑III in terms of iterations offsets the additional gradient evaluations, rendering them competitive in CPU‑time performance. Across all test cases, the proposed UGMs achieve performance comparable to FGM.

The role of $\varrho$ in UGM‑I.

We next investigate the effect of the parameter $\varrho$ on the performance of UGM‑I. We select four representative values of $\varrho$ from the interval $[\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1},\frac{\kappa-1}{\kappa+1})$ . The smallest value is taken at the lower bound endpoint, while the largest is set to $0.99(\kappa-1)/(\kappa+1)$ to stay safely below the strict upper limit. The remaining two values are equally spaced between these two extremes. We compare the four resulting variants of UGM‑I on two contrasting eigenvalue distributions, Case 1 and Case 5, using three $(\mu,L)$ configurations: $(1,10^{2})$ , $(1,10^{4})$ , and $(10^{2},10^{4})$ . The numerical results are presented in Figure 7.

It is clear from the results in Figure 7 that the parameter $\varrho$ dictates the convergence speed of UGM‑I: the smaller the value of $\varrho$ , the steeper the decrease of the suboptimality $f(x^{k})-f_{*}$ .

5.2 Logistic regression

In this subsection, we consider the regularized logistic regression problem

\min_{x\in\mathbb{R}^{n}}f(x)=-\frac{1}{m}\sum_{i=1}^{m}\left(y_{i}\log\big(s(a_{i}^{\top}x)\big)+(1-y_{i})\log\big(1-s(a_{i}^{\top}x)\big)\right)+\frac{\gamma}{2}\|x\|^{2},

where $\{(a_{i},y_{i})\in\mathbb{R}^{n}\times\{0,1\}:i=1,\ldots,m\}$ is the dataset, $s(z)=1/\bigl(1+\exp(-z)\bigr)$ denotes the sigmoid function, $m$ is the number of samples, and $\gamma>0$ is the regularization parameter. For this problem, the gradient of $f$ is given by $\nabla f(x)=\frac{1}{m}\sum_{i=1}^{m}a_{i}\big(s(a_{i}^{\top}x)-y_{i}\big)+\gamma x$ . Consequently, $f$ is $L$ -smooth with Lipschitz constant $L=\lambda_{\max}(A^{\top}A)/4+\gamma$ , where $A=(a_{1}^{\top},\ldots,a_{m}^{\top})^{\top}$ and $\lambda_{\max}(A^{\top}A)$ denotes the largest eigenvalue of $A^{\top}A$ . In this experiment, all competing algorithms are run for $3000$ iterations. We test on the mushrooms dataset from LIBSVM [9], which has sample size $m=8124$ and feature dimension $n=112$ . The Lipschitz constant is $L=2.59$ , and the regularization parameter is set to $\gamma=L/m=3.18\times 10^{-5}$ . Consistent with the experiments in Section 5.1, every algorithm is initialized at the origin. For the proposed UGMs, the parameters are chosen as follows. We take $\varrho=1-1/\sqrt{\kappa}$ , matching the convergence‑rate parameter of the FGM method. We further set $q_{0}=(\sqrt{\kappa}-1)/(\sqrt{\kappa}+1)$ for UGM‑I and $q_{0}=\varrho$ for UGM‑II and UGM‑III. We then let $\underline{\theta}=1$ , $a_{0}=\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}$ , and $\bar{\theta}=\max\big\{\frac{1}{1-\varrho},\frac{\varrho a_{0}+1-\varrho}{(1-\varrho)a_{0}}\big\}=\sqrt{\kappa}$ .

In Figure 8, we plot the objective suboptimality $f(x^{k})-f_{*}$ against iteration count and CPU time (in seconds), along with the evolution of the extrapolation parameter $\theta_{k}$ for UGM‑II and UGM‑III. From Figures 8(a) and 8(b), we observe that both UGM‑II and UGM‑III consistently outperform FGM in terms of both iteration efficiency and computational cost on the tested logistic regression instance. This performance gain arises from the adaptive updating rule of the extrapolation parameter $\theta_{k}$ . As shown in Figure 8(c), $\theta_{k}$ eventually converges to its lower bound $\underline{\theta}$ after a certain number of iterations, which suppresses parameter oscillations and substantially accelerates the decay of objective suboptimality.

5.3 Softmax regression

The numerical results above demonstrate that UGM‑II and UGM‑III achieve faster convergence in terms of iteration count, yet this benefit does not always yield a corresponding wall‑clock improvement, owing to the extra gradient evaluation required at each iteration. In this subsection, we examine whether their iteration‑wise advantage can be converted into better computational efficiency when the inner‑product scalar $\langle\nabla f(z^{k+1}),z^{k+1}-y^{k+1}\rangle$ is computed without constructing an additional full‑gradient vector. To this end, we consider the following $\ell_{2}$ -regularized multiclass logistic regression problem, commonly referred to as softmax regression:

\min_{W\in\mathbb{R}^{d\times K}}f(W)=\frac{1}{m}\sum_{i=1}^{m}\left[\log\!\sum\nolimits_{c=1}^{K}\exp\bigl((XW)_{i,c}\bigr)-(XW)_{i,y_{i}}\right]+\frac{\gamma}{2}\|W\|_{F}^{2}.

(47)

Here $(XW)_{i,c}$ denotes the score of sample $i$ for class $c$ , $(XW)_{i,y_{i}}$ is the score corresponding to its true class label $y_{i}$ , $\|\cdot\|_{F}$ denotes the Frobenius norm, and $\gamma>0$ . Given a prescribed condition number $\kappa=3000$ , we set $\gamma:=L_{\mathrm{data}}/(\kappa-1)$ , with $L_{\mathrm{data}}:=\frac{1}{2m}\|X\|_{2}^{2}$ . With this choice, the objective function $f$ in (47) is $L$ -smooth with Lipschitz constant $L=L_{\mathrm{data}}+\gamma$ and $\gamma$ -strongly convex. Its resulting condition number is therefore $L/\gamma=\kappa$ .

We investigate two ways to convert UGM‑II and UGM‑III’s iteration‑count advantage into CPU‑time savings. The first uses cached logits to evaluate $\langle\nabla f(z^{k+1}),z^{k+1}-y^{k+1}\rangle$ without computing an additional full gradient. The second employs scalar aggregation in a distributed setting, replacing communication of high‑dimensional gradient matrices with a single scalar transmitted from each worker. Both experiments use the Digits dataset from sklearn.datasets.load_digits, which contains $m=1797$ grayscale images belonging to $10$ digit classes. Each $8\times 8$ image is flattened into a 64‑dimensional vector with pixel values in $[0,16]$ . We standardize the original $64$ features and use all samples to construct the finite‑sum problem. The two experiments differ only in the nonlinear feature map subsequently applied to the standardized data. Furthermore, the UGM variants in these experiments adopt the same parameter settings for $\varrho$ , $q_{0}$ , $\underline{\theta}$ , $a_{0}$ , and $\bar{\theta}$ as specified in Section 5.2.

Scalar evaluation using cached logits.

Let $P(z^{k+1})\in\mathbb{R}^{n\times K}$ denote the matrix of softmax probabilities corresponding to the logits $Xz^{k+1}$ , with entries

\bigl[P(z^{k+1})\bigr]_{i,c}=\exp\bigl((Xz^{k+1})_{i,c}\bigr)\big/\sum\nolimits_{j=1}^{K}\exp\bigl((Xz^{k+1})_{i,j}\bigr).

Then, the scalar required by the update can be expressed as

\displaystyle\left\langle\nabla f(z^{k+1}),y^{k+1}-z^{k+1}\right\rangle=\frac{1}{n}\left\langle P(z^{k+1})-Y,X\bigl(y^{k+1}-z^{k+1}\bigr)\right\rangle+\mu\left\langle z^{k+1},y^{k+1}-z^{k+1}\right\rangle,

(48)

where $Y\in\mathbb{R}^{n\times K}$ is the one-hot encoding matrix of the class labels. Here, for matrix variables, $\langle\cdot,\cdot\rangle$ denotes the trace inner product. The quantity $X(y^{k+1}-z^{k+1})$ can be computed directly from the cached logits $Xy^{k+1}$ and $Xz^{k+1}$ . Thus, this directional implementation only evaluates the Frobenius inner products in (48), without explicitly constructing the full‑gradient term $X^{\top}(P(z^{k+1})-Y)$ . The directional and full‑gradient implementations produce identical iterates up to floating‑point round‑off error, yet they incur different per‑iteration computational costs.

Our first experiment maps the $1797$ handwritten‑digit samples to $d=768$ random Fourier features (Digits‑RFF) and considers $K=10$ classes.

The experimental results are presented in Figure 9. In our tests, UGM‑II and UGM‑III attain the stopping criterion $f(x^{k})-f_{*}\leq 10^{-6}$ in $255$ and $230$ iterations, respectively, compared with $400$ iterations for FGM and $390$ iterations for UGM‑I. With the cached‑logit directional implementation, their CPU times are $1.619$ and $1.340$ seconds. Compared with FGM, these results yield speedups of $1.52\times$ and $1.84\times$ . These findings demonstrate that the cached‑logit implementation successfully converts the iteration‑count advantages of UGM‑II and UGM‑III into substantial CPU‑time savings.

Digits‑Poly2 with scalar aggregation.

Our second experiment expands the same samples into $d=2144$ standardized degree‑2 polynomial features (Digits‑Poly2), again with $K=10$ classes, to further evaluate the benefits of scalar aggregation in a distributed setting. The samples are partitioned across four logical workers. For the directional computation, each worker returns a single double‑precision scalar, resulting in a total communication payload of $32$ bytes. In contrast, full‑gradient aggregation requires transmitting four $2144\times 10$ matrices, corresponding to $686080$ bytes. Therefore, scalar aggregation reduces the communication payload by a factor of $21440$ .

The experimental results are presented in Figure 10. In our tests, UGM‑II and UGM‑III attain the stopping criterion $f(x^{k})-f_{*}\leq 10^{-6}$ in $255$ and $240$ iterations, while FGM and UGM‑I require $400$ and $395$ iterations. With scalar aggregation across the distributed workers, their CPU times are reduced to $4.480$ and $3.918$ seconds, whereas FGM requires $6.128$ seconds. Relative to FGM, this yields speedups of $1.37\times$ and $1.56\times$ , respectively. These findings demonstrate that scalar aggregation can convert the lower iteration counts of UGM‑II and UGM‑III into measurable computational gains in distributed settings.

Moreover, the $\theta_{k}$ panels (last plots in Figures 9 and 10) show a brief initial adaptive transient phase, after which the algorithm stably selects the lower bound $\underline{\theta}=1$ . Therefore, the observed improvement in iteration‑count performance does not arise merely from employing a fixed momentum parameter. Instead, it stems from the branch‑aware update rule, while cached‑logit evaluation and scalar aggregation reduce the overhead of this extra selection information sufficiently to achieve an end‑to‑end runtime advantage.

6 Concluding remarks

In this paper, we study a unified first‑order method that encompasses a broad family of accelerated and non‑accelerated algorithms for minimizing $f\in\mathcal{F}_{\mu,L}$ . We show that the proposed method can be naturally interpreted as a combination of the heavy‑ball method and gradient descent. This reveals that classical acceleration schemes effectively incorporate a conservative gradient‑descent step into the fast yet potentially divergent heavy‑ball dynamics to balance acceleration We then present a unified convergence analysis built upon a Lyapunov‑function framework, which further motivates the design of problem‑dependent accelerated algorithms and their variants. Applications to unconstrained quadratic optimization, logistic regression, and softmax regression demonstrate that the proposed methods can outperform existing approaches under specific structural conditions.

The following directions are worthy of further investigation.

(i)

One interesting direction is to estimate the sign of $\langle\nabla\!f(x^{k+1}),x^{k+1}-z^{k+1}\rangle$ without relying on Lemma 4.4, which requires either additional full-gradient evaluations or the corresponding directional computations. Directly determining the sign of this inner product would reduce computational overhead and improve practicality. Such a technique is consistent with our unified Lyapunov framework and may support the design of additional accelerated optimization variants.

(ii)

Estimating the $L$ -smoothness and $\mu$ -strong convexity parameters without prior problem information remains challenging [8, 36]. It is worth investigating whether adaptive iteration-dependent parameters $L_{k}$ and $\mu_{k}$ can be incorporated into the UGM framework (4) to achieve accelerated convergence, following the parameter strategy of Nesterov’s FGM [25]. The corresponding update scheme is provided below:

\left\{\quad\begin{aligned} y^{k+1}&=x^{k}-\frac{1}{L_{k}}\nabla f(x^{k}),\\ w^{k+1}&=x^{k}-\frac{1}{\mu_{k}}\nabla f(x^{k}),\\ z^{k+1}&=\Big(1-\sqrt{\mu_{k}/L_{k}}\Big)z^{k}+\sqrt{\mu_{k}/L_{k}}w^{k+1},\\ x^{k+1}&=\frac{\sqrt{L_{k}/\mu_{k}}}{1+\sqrt{L_{k}/\mu_{k}}}y^{k+1}+\frac{1}{1+\sqrt{L_{k}/\mu_{k}}}z^{k+1}.\end{aligned}\right.

(iii)

We note that if $\langle\nabla f(x^{k+1}),x^{k+1}-z^{k+1}\rangle\leq 0$ , any negative $\theta_{k}$ satisfies the condition (44), which allows negative momentum while maintaining convergence guarantees. Previous studies have shown that negative momentum may cause oscillation and instability in minimax optimization [15, 30]. Examining its behavior under the current algorithmic setting constitutes another interesting research direction.

References

[ALO16a] Z. Allen-Zhu, Y. T. Lee, and L. Orecchia (2016) Using optimization to obtain a width-independent, parallel, simpler, and faster positive sdp solver. In Proceedings of the twenty-seventh annual ACM-SIAM symposium on Discrete algorithms, pp. 1824–1831. Cited by: 2nd item.
[AO15] Z. Allen-Zhu and L. Orecchia (2015) Nearly-linear time positive lp solver with faster convergence rate. In STOC, pp. 229–236. Cited by: 2nd item.
[AO17] Z. Allen-Zhu and L. Orecchia (2017) Linear coupling: an ultimate unification of gradient and mirror descent. In ITCS, pp. 3:1–3:22. Cited by: 2nd item.
[AQR+16b] Z. Allen-Zhu, Z. Qu, P. Richtárik, and Y. Yuan (2016) Even faster accelerated coordinate descent using non-uniform sampling. In ICML, pp. 1110–1119. Cited by: 2nd item.
[AA01] F. Alvarez and H. Attouch (2001) An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping. Set-Valued Anal. 9 (1-2), pp. 3–11. Note: Wellposedness in optimization and related topics (Gargnano, 1999) External Links: ISSN 0927-6947,1572-932X, Document, MathReview Entry Cited by: §1.
[ACP+18] H. Attouch, Z. Chbani, J. Peypouquet, and P. Redont (2018) Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Math. Program. 168 (1), pp. 123–175. Cited by: 1st item.
[BT09] A. Beck and M. Teboulle (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2 (1), pp. 183–202. External Links: Document, MathReview Entry Cited by: §1, §1.
[CLW25] J. V. Cavalcanti, L. Lessard, and A. C. Wilson (2025) Adaptive acceleration without strong convexity priors or restarts. arXiv preprint arXiv:2506.13033. Cited by: item (ii).
[CL11] C. Chang and C. Lin (2011) LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2 (3), pp. 27. External Links: ISSN 2157-6904, Link, Document Cited by: §5.2.
[CL19] L. Chen and H. Luo (2019) First order optimization methods based on hessian-driven nesterov accelerated gradient flow. arXiv preprint arXiv:1912.09276. Cited by: §1.
[CL21] L. Chen and H. Luo (2021) A unified convergence analysis of first order convex optimization methods via strong lyapunov functions. arXiv preprint arXiv:2108.00132. Cited by: 1st item.
[DT22] Y. Drori and A. Taylor (2022) On the oracle complexity of smooth strongly convex minimization. Journal of Complexity 68, pp. 101590. Cited by: §1.
[DT14] Y. Drori and M. Teboulle (2014) Performance of first-order methods for smooth convex minimization: a novel approach. Math. Program. 145 (1), pp. 451–482. Cited by: §1.
[FRM+18] M. Fazlyab, A. Ribeiro, M. Morari, and V. M. Preciado (2018) Analysis of optimization algorithms via integral quadratic constraints: nonstrongly convex problems. SIAM J. Optim. 28 (3), pp. 2654–2689. Cited by: 3rd item.
[GHP+19] G. Gidel, R. A. Hemmat, M. Pezeshki, R. Le Priol, G. Huang, S. Lacoste-Julien, and I. Mitliagkas (2019) Negative momentum for improved game dynamics. In AISTATS, pp. 1802–1811. Cited by: item (iii).
[GTD25] B. Goujaud, A. Taylor, and A. Dieuleveut (2025) Provable non-accelerations of the heavy-ball method: b. goujaud et al.. Math. Program., pp. 1–59. Cited by: §1, §2.
[KF16] D. Kim and J. A. Fessler (2016) Optimized first-order methods for smooth convex minimization. Math. Program. 159 (1), pp. 81–107. Cited by: §1.1, §1.
[KF18] D. Kim and J. A. Fessler (2018) Generalizing the optimized gradient method for smooth convex minimization. SIAM J. Optim. 28 (2), pp. 1920–1950. Cited by: §1.1.
[LRP16] L. Lessard, B. Recht, and A. Packard (2016) Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J. Optim. 26 (1), pp. 57–95. Cited by: 3rd item, §1, §2.
[LC22] H. Luo and L. Chen (2022) From differential equation solvers to accelerated first-order methods for convex optimization. Math. Program. 195 (1), pp. 735–781. Cited by: 1st item.
[MM20] Y. Malitsky and K. Mishchenko (2020) Adaptive gradient descent without descent. In ICML, pp. 6702–6712. Cited by: §1.
[MJ19] M. Muehlebach and M. Jordan (2019) A dynamical systems perspective on nesterov acceleration. In ICML, pp. 4656–4662. Cited by: 1st item.
[NEM95] A. Nemirovski (1995) Information-based complexity of convex programming. Lecture notes 834. Cited by: §1.
[NES83] Y. Nesterov (1983) A method for solving the convex programming problem with convergence rate $O(1/k^{2})$ . Dokl. Akad. Nauk SSSR 269 (3), pp. 543–547. External Links: ISSN 0002-3264, MathReview (R. Şerban) Cited by: §1, item (i).
[NES04] Y. Nesterov (2004) Introductory lectures on convex optimization. Applied Optimization, Vol. 87, Kluwer Academic Publishers, Boston, MA. Note: A basic course External Links: MathReview Entry Cited by: §1.2, Table 1, §1, item 3, item (i), §5.1, item (ii), footnote 2.
[NES13] Y. Nesterov (2013) Gradient methods for minimizing composite functions. Math. Program. 140 (1), pp. 125–161. External Links: ISSN 0025-5610,1436-4646, Document, MathReview (Dana Simian) Cited by: §1.
[PPR23] C. Park, J. Park, and E. K. Ryu (2023) Factor- $\sqrt{2}$ acceleration of accelerated gradient methods. Applied Mathematics & Optimization 88 (3), pp. 77. Cited by: Table 1, §1, item 5, item (ii), §3.
[POL64] B. T. Polyak (1964) Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics 4 (5), pp. 1–17. Cited by: §1.
[SDJ+22] B. Shi, S. S. Du, M. I. Jordan, and W. J. Su (2022) Understanding the acceleration phenomenon via high-resolution differential equations. Math. Program. 195 (1), pp. 79–148. Cited by: 1st item.
[SWA26] H. Shugart, S. Wang, and J. M. Altschuler (2026) Negative momentum for convex-concave optimization. arXiv preprint arXiv:2604.17145. Cited by: item (iii).
[SBC16] W. Su, S. Boyd, and E. J. Candes (2016) A differential equation for modeling nesterov’s accelerated gradient method: theory and insights. JMLR 17 (153), pp. 1–43. Cited by: 1st item.
[TD23] A. Taylor and Y. Drori (2023) An optimal gradient method for smooth strongly convex minimization. Math. Program. 199 (1), pp. 557–594. Cited by: §1.1, Table 1, §1, §1, Remark 3.2.
[VFL17] B. Van Scoy, R. A. Freeman, and K. M. Lynch (2017) The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters 2 (1), pp. 49–54. Cited by: Table 1, §1, §1, item 4, item (iii), item (iii).
[WRM15] D. Wang, S. Rao, and M. W. Mahoney (2015) Unified acceleration method for packing and covering problems via diameter reduction. arXiv preprint arXiv:1508.02439. Cited by: 2nd item.
[WC25] J. Wei and L. Chen (2025) Accelerated over-relaxation heavy-ball method: achieving global accelerated convergence with broad generalization. In ICLR, Cited by: 1st item, Table 1, §1, item 6, item (iv), item (iv), Remark 3.2.
[XC26] Z. Xu and L. Chen (2026) Adaptive accelerated gradient descent methods for convex optimization. arXiv preprint arXiv:2601.19013. Cited by: item (ii).

Appendix A Proof on the equivalence between (HB) and (9)

Lemma A.1.

The iteration scheme (HB) is equivalent to (9) with $z^{0}=x^{0}$ , $q=\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}$ , and $\theta=\frac{\sqrt{\kappa}-1}{2}$ .

Proof.

First, it follows from (9c) that $z^{i+1}=(1+\theta)x^{i+1}-\theta x^{i}$ for all $i\geq 0$ . Substituting this relation with $i=k$ and $i=k+1$ into (9b) and using (9a), we obtain

	$\displaystyle(1+\theta)x^{k+1}-\theta x^{k}=z^{k+1}$	$\displaystyle{\displaystyle\mathrel{\to}^{\eqref{hb-z},\,\eqref{hb-w}}}q\left((1+\theta)x^{k}-\theta x^{k-1}\right)+(1-q)\big(x^{k}-\mu^{-1}\nabla f(x^{k})\big)$
		$\displaystyle=x^{k}+q\theta(x^{k}-x^{k-1})-(1-q)\mu^{-1}\nabla f(x^{k}).$

By reformulating the above equality, we obtain $x^{k+1}=x^{k}-\frac{1-q}{\mu(1+\theta)}\nabla f(x^{k})+\frac{q\theta}{1+\theta}(x^{k}-x^{k-1})$ . By setting $q=\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}$ and $\theta=\frac{\sqrt{\kappa}-1}{2}$ , one can readily verify that

1-q=\frac{1}{1+\theta}=\frac{2\sqrt{\mu}}{\sqrt{L}+\sqrt{\mu}},\quad\frac{1-q}{\mu(1+\theta)}=\frac{4}{(\sqrt{L}+\sqrt{\mu})^{2}}\text{~~and~~}\frac{q\theta}{1+\theta}=\frac{(\sqrt{L}-\sqrt{\mu})^{2}}{(\sqrt{L}+\sqrt{\mu})^{2}}.

Therefore, the two schemes (HB) and (9) are equivalent. ∎

Appendix B Reformulation of several algorithms

In this section, we present equivalent formulations of several well‑known algorithms within the UGM framework (4). Note that all cumbersome identities used in the proofs of Lemmas B.4, B.5, B.6 and B.7 have been validated by hand-written derivation and symbolic computation, where check_proof.ipynb is used exclusively for the symbolic-computation verification.

Lemma B.1.

The UGM framework (4) with initialization $z^{0}=x^{0}$ is equivalent to


$\displaystyle y^{k+1}$	$\displaystyle=x^{k}-\frac{1}{L}\nabla f(x^{k}),$	(49a)
$\displaystyle x^{k+1}$	$\displaystyle=\frac{\theta+(1-q)\kappa}{1+\theta}y^{k+1}+\frac{1+q\theta-(1-q)\kappa}{1+\theta}x^{k}-\frac{q\theta}{1+\theta}y^{k},$	(49b)

with initialization $y^{0}=x^{0}$ .

Proof.

First, (4a) is the same as (49a), from which we obtain $\nabla f(x^{k})=L(x^{k}-y^{k+1})$ . Substituting this relation into (4b) yields

w^{k+1}=x^{k}-\kappa(x^{k}-y^{k+1})=\left(1-\kappa\right)x^{k}+\kappa y^{k+1}.

(50)

Additionally, from (4d), we have $z^{i}=(1+\theta)x^{i}-\theta y^{i}$ for all $i\geq 1$ . Substituting this relation with $i=k$ and $i=k+1$ into (4c) and using (50), we obtain

	$\displaystyle(1+\theta)x^{k+1}-\theta y^{k+1}$	$\displaystyle=z^{k+1}{\mathrel{\to}^{\eqref{ugm-z},\eqref{jy-0808a}}}q\big((1+\theta)x^{k}-\theta y^{k}\big)+(1-q)\big((1-\kappa)x^{k}+\kappa y^{k+1}\big)$
		$\displaystyle=\big(q(1+\theta)+(1-q)(1-\kappa)\big)x^{k}-q\theta y^{k}+(1-q)\kappa y^{k+1}$
		$\displaystyle=\big(1+q\theta-(1-q)\kappa\big)x^{k}-q\theta y^{k}+(1-q)\kappa y^{k+1}.$

By rearranging the above equality to explicitly solve for $x^{k+1}$ , we obtain (49b), which establishes the desired equivalence. ∎

B.1 Gradient descent methods

Lemma B.2.

With the initial condition $z^{0}=x^{0}$ and parameters $q=1-1/\kappa$ and arbitrary $\theta\neq-1$ , the UGM framework (4) is equivalent to GD with step size $1/L$ , i.e., (3).

Proof.

Let $\{(y^{k},w^{k},z^{k},x^{k}):k\geq 1\}$ be the sequence generated by UGM (4) with initialization $z^{0}=x^{0}$ and parameters $q=1-1/\kappa$ and arbitrary $\theta\neq-1$ . We prove by induction that $z^{k}=x^{k}$ for all $k\geq 1$ . Assume that $z^{k}=x^{k}$ for some $k\geq 0$ . Then, from (4c), (4b) and (4a), we obtain

z^{k+1}{\mathrel{\to}^{\eqref{ugm-z},\eqref{ugm-w}}}qx^{k}+(1-q)\big(x^{k}-\mu^{-1}\nabla f(x^{k})\big)=x^{k}-L^{-1}\nabla f(x^{k})\stackrel{{\scriptstyle\eqref{ugm-y}}}{{=}}y^{k+1},

where the second “ $=$ ” follows from $q=1-\frac{1}{\kappa}=1-\frac{\mu}{L}$ . By plugging $z^{k+1}=y^{k+1}$ into (4d), we obtain $x^{k+1}=y^{k+1}$ . Hence, $z^{k+1}=y^{k+1}=x^{k+1}=x^{k}-L^{-1}\nabla f(x^{k})$ . The proof is completed by induction. ∎

Lemma B.3.

With the initial condition $z^{0}=x^{0}$ and parameters $q=\frac{\kappa-1}{\kappa+1}$ and $\theta=0$ , the UGM framework (4) is equivalent to GD (2) with step size $\alpha=2/(L+\mu)$ .

Proof.

Since $\theta=0$ and $z^{0}=x^{0}$ , (4d) yields $x^{k}=z^{k}$ for all $k\geq 0$ . Hence, for every $k\geq 0$ , we obtain

x^{k+1}=z^{k+1}{\mathrel{\to}^{\eqref{ugm-z},\eqref{ugm-w}}}qx^{k}+(1-q)\big(x^{k}-\mu^{-1}\nabla f(x^{k})\big)=x^{k}-\frac{2}{L+\mu}\nabla f(x^{k}),

where the final equality follows from $\frac{1-q}{\mu}=\frac{2}{L+\mu}$ , which holds with $q=\frac{\kappa-1}{\kappa+1}$ . This completes the proof. ∎

B.2 Nesterov’s FGM

Lemma B.4.

With the initial condition $z^{0}=x^{0}$ and parameters $q=1-\frac{1}{\sqrt{\kappa}}$ and $\theta=\sqrt{\kappa}$ , the UGM framework (4) is equivalent to (10).

Proof.

To prove the equivalence between (4) and (10), we only need to verify that the update $x^{k+1}$ in (4d) reduces to that given in (10). Utilizing the given values of $q$ and $\theta$ , the following identities hold:

\frac{\theta+(1-q)\kappa}{1+\theta}=\frac{2\sqrt{\kappa}}{\sqrt{\kappa}+1},\quad\frac{1+q\theta-(1-q)\kappa}{1+\theta}=0\text{~~and~~}\frac{q\theta}{1+\theta}=\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}.

(51)

We have validated these equalities through manual derivation, while the symbolic‑computation check is also implemented in the script check_proof.ipynb. It then follows from (49b) and (51) that

x^{k+1}=\frac{2\sqrt{\kappa}}{\sqrt{\kappa}+1}y^{k+1}-\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}y^{k}=y^{k+1}+\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}(y^{k+1}-y^{k}).

This exactly matches the update of $x^{k+1}$ in (10), which completes the proof. ∎

B.3 TM method

Lemma B.5.

With the initial condition $z^{0}=x^{0}$ and parameters $q=\rho=1-\frac{1}{\sqrt{\kappa}}$ and $\theta=\frac{\rho}{2(1-\rho)}$ , the sequence $\{(z^{k},x^{k}):k\geq 1\}$ generated by the UGM framework (4) coincides with the sequence generated by the TM method (11).

Proof.

To begin with, we note that all cumbersome identities used in this proof have been checked by manual derivation, while symbolic‑computation verification is carried out in check_proof.ipynb.

First, we establish the equivalence between (11b) and (4d). To prove the equivalence, it suffices to show that $x^{k+1}$ from both (11b) and (4d) satisfies

x^{k+1}-x^{k}=\frac{\rho^{2}}{2-\rho}(y^{k+1}-y^{k})+(1+\rho)(y^{k+1}-x^{k}),

(52)

where $y^{k+1}=x^{k}-\frac{1}{L}\nabla f(x^{k})$ .

(i)

Consider $\{x^{k}:k\geq 1\}$ generated by the TM method (11). From (11b), the left-hand side of (52) is equal to

\displaystyle x^{k+1}-x^{k}\stackrel{{\scriptstyle\eqref{tm-x}}}{{=}}(1+\gamma)\xi^{k+1}-\gamma\xi^{k}-\big((1+\gamma)\xi^{k}-\gamma\xi^{k-1}\big)=(1+\gamma)\xi^{k+1}-(2\gamma+1)\xi^{k}+\gamma\xi^{k-1}.

(53)

On the other hand, from the parameter settings of (11), we obtain $\lambda/\gamma=1+\rho$ and (11b) yields $\xi^{k}-\xi^{k-1}=(x^{k}-\xi^{k})/\gamma$ . We then derive

	$\displaystyle\xi^{k+1}$	$\displaystyle\stackrel{{\scriptstyle\eqref{tm-xi}}}{{=}}\xi^{k}+\lambda(\xi^{k}-\xi^{k-1})-\nu\nabla f(x^{k})\stackrel{{\scriptstyle\eqref{tm-x}}}{{=}}\xi^{k}+(1+\rho)(x^{k}-\xi^{k})-\nu\nabla f(x^{k})$
		$\displaystyle=-\rho\xi^{k}+(1+\rho)(x^{k}-\tfrac{1}{L}\nabla f(x^{k}))\stackrel{{\scriptstyle\eqref{ugm-y}}}{{=}}-\rho\xi^{k}+(1+\rho)y^{k+1},$

where the third “ $=$ ” follows from $\nu={(1+\rho)}/{L}$ . Thus, we obtain $y^{k+1}=(\xi^{k+1}+\rho\xi^{k})/(1+\rho)$ . Using this relation together with $\gamma=\frac{\rho^{2}}{(1+\rho)(2-\rho)}$ , we compute the two terms on the right-hand side of (52) as follows:


$\displaystyle y^{k+1}-y^{k}$	$\displaystyle=\frac{1}{1+\rho}\big(\xi^{k+1}+\rho\xi^{k}\big)-\frac{1}{1+\rho}\big(\xi^{k}+\rho\xi^{k-1}\big)=\frac{1}{1+\rho}\big(\xi^{k+1}+(\rho-1)\xi^{k}-\rho\xi^{k-1}\big),$	(54a)
$\displaystyle y^{k+1}-x^{k}$	$\displaystyle\stackrel{{\scriptstyle\eqref{tm-x}}}{{=}}\frac{1}{1+\rho}\big(\xi^{k+1}+\rho\xi^{k}\big)-(1\!+\!\gamma)\xi^{k}+\gamma\xi^{k-1}=\frac{1}{1+\rho}\Big[\xi^{k+1}\!-\!\frac{\rho^{2}\!-\!\rho\!+\!2}{2\!-\!\rho}\xi^{k}+\frac{\rho^{2}}{2\!-\!\rho}\xi^{k-1}\Big].$	(54b)

Utilizing $\gamma=\frac{\rho^{2}}{(1+\rho)(2-\rho)}$ again, the following identities can be verified:

\frac{\rho^{2}(\rho-1)}{(1+\rho)(2-\rho)}-\frac{\rho^{2}-\rho+2}{2-\rho}=-(2\gamma+1)\text{~~and~~}\frac{-\rho^{3}}{(1+\rho)(2-\rho)}+\frac{\rho^{2}}{2-\rho}=\gamma.

(55)

Then, substituting (54) into the right-hand side of (52) and using the identities in (55), we derive

\displaystyle\frac{\rho^{2}}{2-\rho}(y^{k+1}-y^{k})+(1+\rho)(y^{k+1}-x^{k})=(1+\gamma)\xi^{k+1}-(2\gamma+1)\xi^{k}+\gamma\xi^{k-1}.

(56)

By comparing (53) and (56), we confirm that (52) is satisfied by (11).

(ii)

Consider $\{x^{k}:k\geq 1\}$ generated by the UGM framework (4). Utilizing the given parameters $q=\rho=1-\frac{1}{\sqrt{\kappa}}$ and $\theta=\frac{\rho}{2(1-\rho)}$ , it can be verified that

\frac{\theta+(1-q)\kappa}{1+\theta}=\frac{2+\rho}{2-\rho},\quad\frac{1+q\theta-(1-q)\kappa}{1+\theta}=-\rho\text{~~and~~}\frac{q\theta}{1+\theta}=\frac{\rho^{2}}{2-\rho}.

It then follows from the above identities and (49b) that $x^{k+1}=\frac{2+\rho}{2-\rho}y^{k+1}-\rho x^{k}-\frac{\rho^{2}}{2-\rho}y^{k}$ , from which it is trivial to derive (52).

This completes the proof of the equivalence between (11b) and (4d).

We next establish the equivalence between (11c) and (4c) by verifying that $z^{k+1}$ from both (11c) and (4c) satisfies

z^{k+1}-x^{k+1}=\frac{\rho}{2(1-\rho)}(x^{k+1}-y^{k+1}).

(57)

(i)

Consider $\{z^{k}:k\geq 1\}$ generated by the TM method (11). Subtracting each‑hand‑side of (54b) from the corresponding side of (53) and utilizing $\gamma=\frac{\rho^{2}}{(1+\rho)(2-\rho)}$ alongside elementary algebraic manipulations, we arrive at $x^{k+1}-y^{k+1}=\frac{2\rho}{(1+\rho)(2-\rho)}(\xi^{k+1}-\xi^{k})$ . Using (11b) and (11c), together with this relation, we further derive

z^{k+1}-x^{k+1}{\mathrel{\to}^{\eqref{tm-x},\eqref{tm-z}}}(\delta-\gamma)(\xi^{k+1}-\xi^{k})=(\delta-\gamma)\frac{(1+\rho)(2-\rho)}{2\rho}(x^{k+1}-y^{k+1})=\frac{\rho(x^{k+1}-y^{k+1})}{2(1-\rho)},

where the last equality follows from the parameter settings of $\delta$ and $\gamma$ in (11) and elementary computations. This confirms (57).

(ii)

Consider $\{z^{k}:k\geq 1\}$ generated by the UGM framework (4). It follows straightforwardly from (4d) and $\theta=\frac{\rho}{2(1-\rho)}$ that $z^{k+1}-x^{k+1}\stackrel{{\scriptstyle\eqref{ugm-x}}}{{=}}\theta(x^{k+1}-y^{k+1})=\frac{\rho}{2(1-\rho)}(x^{k+1}-y^{k+1})$ , i.e., (57) is confirmed.

This completes the proof of the equivalence between (11c) and (4c). ∎

B.4 SC-OGM

Lemma B.6.

With the initial condition $z^{0}=x^{0}$ and parameters $q=\frac{1}{\gamma+1}$ and $\theta=\frac{\gamma+1}{\gamma}$ , the UGM framework (4) is equivalent to the SC-OGM scheme (12).

Proof.

To begin with, we note that all cumbersome identities used in this proof have been checked by manual derivation, while symbolic‑computation verification is carried out in check_proof.ipynb.

To establish this equivalence, it suffices to verify that the update for $x^{k+1}$ given in (4d) satisfies

x^{k+1}=y^{k+1}+\frac{1}{2\gamma+1}(y^{k+1}-y^{k})+\frac{1}{2\gamma+1}(y^{k+1}-x^{k}),

(58)

which exactly matches the iteration rule for $x^{k+1}$ in (12). With the given parameters $q=\frac{1}{\gamma+1}$ , $\theta=\frac{\gamma+1}{\gamma}$ , $\gamma=\frac{\sqrt{8\kappa+1}+3}{2\kappa-2}$ , and noting that $\kappa=1+3/\gamma+2/\gamma^{2}$ , we can verify the following identities:

\frac{\theta+(1-q)\kappa}{1+\theta}=\frac{2\gamma+3}{2\gamma+1},\quad\frac{1+q\theta-(1-q)\kappa}{1+\theta}=-\frac{1}{2\gamma+1}\text{~~and~~}\frac{q\theta}{1+\theta}=\frac{1}{2\gamma+1}.

It then follows from (49b) that $x^{k+1}=\frac{2\gamma+3}{2\gamma+1}y^{k+1}-\frac{1}{2\gamma+1}x^{k}-\frac{1}{2\gamma+1}y^{k}$ , which implies (58) immediately. ∎

B.5 AOR-HB

Lemma B.7.

With the initial condition $z^{0}\!=\!x^{0}$ , and parameters $q\!=\!\frac{2\kappa+2\sqrt{\kappa}\!-\!1+\sqrt{4\sqrt{\kappa}+5}}{2(\sqrt{\kappa}+1)^{2}}$ and $\theta=\frac{\kappa\left(2\sqrt{\kappa}+3\!-\!\sqrt{4\sqrt{\kappa}+5}\right)}{2\left(\kappa+\sqrt{4\sqrt{\kappa}+5}-2\right)}$ , the UGM framework (4) is equivalent to the AOR-HB scheme (13). Moreover, we have $q\in[\frac{3}{4},1)$ and $\theta>0$ .

Proof.

To begin with, we note that all cumbersome identities used in this proof have been checked by manual derivation, while symbolic‑computation verification is carried out in check_proof.ipynb. To prove the equivalence, we only need to verify that the update for $x^{k+1}$ from both (4d) and (13) satisfies

x^{k+1}=\frac{2\kappa}{(\sqrt{\kappa}+1)^{2}}y^{k+1}+\frac{2\sqrt{\kappa}+1}{(\sqrt{\kappa}+1)^{2}}x^{k}-\frac{\kappa}{(\sqrt{\kappa}+1)^{2}}y^{k},

(59)

where $y^{k+1}=x^{k}-\frac{1}{L}\nabla f(x^{k})$ . Using the given values for $q$ and $\theta$ , it can be verified that

\frac{\theta+(1-q)\kappa}{1+\theta}=\frac{2\kappa}{(\sqrt{\kappa}+1)^{2}},\quad\frac{1+q\theta-(1-q)\kappa}{1+\theta}=\frac{2\sqrt{\kappa}+1}{(\sqrt{\kappa}+1)^{2}}\text{~~and~~}\frac{q\theta}{1+\theta}=\frac{\kappa}{(\sqrt{\kappa}+1)^{2}}.

Hence, for the UGM framework (4), (59) is implied by (49b), together with the above identities. On the other hand, for the AOR-HB scheme (13), noting that $\nabla f(x^{k})=L(x^{k}-y^{k+1})$ , we obtain

	$\displaystyle x^{k+1}$	$\displaystyle\stackrel{{\scriptstyle\eqref{aor-hb}}}{{=}}x^{k}-\frac{1}{(\sqrt{L}+\sqrt{\mu})^{2}}\big(2\nabla f(x^{k})-\nabla f(x^{k-1})\big)+\frac{L}{(\sqrt{L}+\sqrt{\mu})^{2}}(x^{k}-x^{k-1})$
		$\displaystyle=x^{k}-\frac{L}{(\sqrt{L}+\sqrt{\mu})^{2}}\big(2(x^{k}-y^{k+1})-(x^{k-1}-y^{k})\big)+\frac{L}{(\sqrt{L}+\sqrt{\mu})^{2}}(x^{k}-x^{k-1})$
		$\displaystyle=\frac{2\kappa}{(\sqrt{\kappa}+1)^{2}}y^{k+1}+\frac{2\sqrt{\kappa}+1}{(\sqrt{\kappa}+1)^{2}}x^{k}-\frac{\kappa}{(\sqrt{\kappa}+1)^{2}}y^{k}.$

Here, elementary computations are involved in deriving the last equality. This confirms (59) and completes the proof of the equivalence.

To complete the proof for the remaining claims of this lemma, we set $t=\sqrt{\kappa}\geq 1$ , which yields $q=q(t):=\frac{2t^{2}+2t-1+\sqrt{4t+5}}{2(t+1)^{2}}$ . Direct computation yields $q^{\prime}(t)=\frac{(t+2)\sqrt{4t+5}-(3t+4)}{\sqrt{4t+5}(t+1)^{3}}$ . Given $t\geq 1$ , we obtain $(t+2)\sqrt{4t+5}\geq 3(t+2)>3t+4$ . Hence, $q^{\prime}(t)>0$ holds for every $t\geq 1$ . Accordingly, $q(t)$ is strictly-increasing over $[1,+\infty)$ , and $\frac{3}{4}=q(1)\leq q(t)<\lim_{t\rightarrow+\infty}q(t)=1$ . Finally, $\theta=\frac{t^{2}\left(2t+3-\sqrt{4t+5}\right)}{2\left(t^{2}+\sqrt{4t+5}-2\right)}>0$ holds trivially since $t\geq 1$ . ∎

Appendix C Supplementary proofs for two inequalities

In the analysis for the TM-method in Section 3, the following inequality (60) is used, and we now provide its proof.

Lemma C.1.

Let $P(t)=\frac{1}{2}+\frac{1}{1+t^{2}}-\frac{2}{\sqrt{1+t^{2}}(\sqrt{1+t}+\sqrt{1+t^{2}})}$ as introduced right below (27). Then, for any $t\in(0,1)$ , we have

\frac{1}{2}\leq P(t)\leq\frac{1+t}{2(1+t^{2})}.

(60)

Proof.

First, we show that for any $t\in(0,1)$ , there holds

1<\frac{\sqrt{1+t}}{\sqrt{1+t^{2}}}<\frac{2+t-t^{2}}{2-t+t^{2}}.

(61)

The left-hand-side inequality in (61) holds trivially since $t>t^{2}$ for $t\in(0,1)$ . To verify the right-hand-side inequality, it is sufficient to show that $(2+m)\sqrt{1+t^{2}}>(2-m)\sqrt{1+t}$ for $t\in(0,1)$ , where $m:=t(1-t)$ . By noting that $m=t(1-t)\in(0,1/4]$ for $t\in(0,1)$ , one can readily show that $(2+m)\sqrt{1+t^{2}}>(2-m)\sqrt{1+t}$ is equivalent to $4-m^{2}+4(t^{2}+t)>0$ , and the latter inequality obviously holds since $m^{2}\leq 1/16$ . This confirms that (61) holds for any $t\in(0,1)$ .

Now, we prove (60). On the one hand, it follows from the right-hand-side inequality in (61) that

P(t)\leq\frac{1}{2}+\frac{1}{1+t^{2}}-\frac{2}{\sqrt{1+t^{2}}(\sqrt{1+t^{2}}\frac{2+t-t^{2}}{2-t+t^{2}}+\sqrt{1+t^{2}})}=\frac{1+t}{2(1+t^{2})}.

On the other hand, the left-hand-side inequality in (61) implies

P(t)\geq\frac{1}{2}+\frac{1}{1+t^{2}}-\frac{2}{\sqrt{1+t^{2}}(\sqrt{1+t^{2}}+\sqrt{1+t^{2}})}=\frac{1}{2}.

The two inequalities right above confirm (60). ∎

The following inequality is employed in the proof of Lemma 4.2, and we now present its proof.

Lemma C.2.

If $\varrho$ satisfies $\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1}\leq\varrho<1$ , then $\frac{(1-\varrho)[\kappa(1-\varrho)-\varrho-1]}{(\varrho+1)\varrho}\leq 1$ .

Proof.

Since $\frac{\sqrt{1+8\kappa}-3}{\sqrt{1+8\kappa}+1}\leq\varrho<1$ , we have $\varrho+3\geq\sqrt{1+8\kappa}(1-\varrho)>0$ . By squaring both sides, it is straightforward to show that the latter inequality is equivalent to $\varrho+1\geq\kappa(1-\varrho)^{2}$ . Subtracting $(1-\varrho^{2})$ from both-hand sides yields $\varrho(\varrho+1)\geq(1-\varrho)\bigl[\kappa(1-\varrho)-\varrho-1\bigr]$ , which establishes the desired inequality. ∎

	$\displaystyle q\\|z^{k}-x^{*}\\|^{2}$	$\displaystyle-\\|z^{k+1}-x^{}\\|^{2}=\frac{2(1-q)}{\mu}\left[\langle\nabla f(x^{k}),x^{k}-x^{}\rangle-\frac{\mu}{2}\\|x^{k}-x^{*}\\|^{2}\right]$		(17)
		$\displaystyle+q(1-q)\\|x^{k}-z^{k}\\|^{2}-\frac{(1-q)^{2}}{\mu^{2}}\\|\nabla f(x^{k})\\|^{2}-\frac{2q(1-q)}{\mu}\langle\nabla f(x^{k}),x^{k}-z^{k}\rangle.$		(17)

	$\displaystyle\\|z^{k+1}$	$\displaystyle-x^{}\\|^{2}\stackrel{{\scriptstyle\eqref{ugm-w},\eqref{ugm-z}}}{{=}}\big\\|(1-q)x^{k}+qz^{k}-x^{}-\frac{1-q}{\mu}\nabla f(x^{k})\big\\|^{2}$
	$\displaystyle=$	$\displaystyle\\|(1-q)x^{k}+qz^{k}-x^{}\\|^{2}+\frac{(1-q)^{2}}{\mu^{2}}\\|\nabla f(x^{k})\\|^{2}-\frac{2(1-q)}{\mu}\langle(1-q)x^{k}+qz^{k}-x^{},\nabla f(x^{k})\rangle$
	$\displaystyle\stackrel{{\scriptstyle\eqref{eq:convex-id}}}{{=}}$	$\displaystyle(1-q)\\|x^{k}-x^{}\\|^{2}+q\\|z^{k}-x^{}\\|^{2}-q(1-q)\\|x^{k}-z^{k}\\|^{2}+\frac{(1-q)^{2}}{\mu^{2}}\\|\nabla f(x^{k})\\|^{2}$
		$\displaystyle-\frac{2(1-q)}{\mu}\langle x^{k}-x^{*},\nabla f(x^{k})\rangle+\frac{2q(1-q)}{\mu}\langle x^{k}-z^{k},\nabla f(x^{k})\rangle.$

	$\displaystyle\\|z^{k}-x^{*}\\|^{2}$	$\displaystyle-\frac{1}{q_{k}}\\|z^{k+1}-x^{}\\|^{2}=\frac{2(1-q_{k})}{\mu q_{k}}\langle x^{k}-x^{},\nabla f(x^{k})\rangle-\frac{1-q_{k}}{q_{k}}\\|x^{k}-x^{*}\\|^{2}$		(32)
		$\displaystyle+(1-q_{k})\\|x^{k}-z^{k}\\|^{2}-\frac{(1-q_{k})^{2}}{q_{k}\mu^{2}}\\|\nabla f(x^{k})\\|^{2}-\frac{2(1-q_{k})}{\mu}\langle\nabla f(x^{k}),x^{k}-z^{k}\rangle.$		(32)

$\displaystyle G_{k}(a_{k})-$	$\displaystyle\frac{1}{\varrho}G_{k+1}(a_{k+1})\geq\left(\frac{1-q_{k+1}}{q_{k+1}}a_{k}-\frac{1-\varrho}{\varrho}\right)\left[\langle\nabla f(x^{k+1}),x^{k+1}-x^{}\rangle-\frac{\mu}{2}\\|x^{k+1}-x^{}\\|^{2}\right]$	(34)
	$\displaystyle+\left(\frac{\varrho+1}{2L\varrho}\!-\!\frac{a_{k}(1\!-\!q_{k+1})^{2}}{2q_{k+1}\mu}\right)\\|\nabla f(x^{k+1})\\|^{2}+\left(\frac{1}{\theta_{k}}\!-\!(1\!-\!q_{k+1})a_{k}\right)\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$
	$\displaystyle+\frac{\mu(1\!-\!q_{k+1})a_{k}}{2}\\|x^{k+1}\!-\!z^{k+1}\\|^{2}.$

UGM: A Unified Framework and New Perspectives for Accelerated Gradient Methods in Smooth and Strongly Convex Optimization

Abstract

1 Introduction

1.1 Our proposed framework and contributions

1.2 Preliminaries and notation

2 UGM unifies diverse gradient methods

Perspective 1: Embedding the GD step (3) into (HB).

Perspective 2: Unifying many existing gradient methods.

Remark 2.1 (The role of qq and θ\theta).

3 Unified analysis via Lyapunov functions

Lemma 3.1 (Inductive relation for f⁡(xk)−f∗f(x^{k})-f_{*} and f⁡(xk)−f∗−12​L​‖∇f​(xk)‖2f(x^{k})-f_{*}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}).

Proof.

Lemma 3.2 (Inductive relation for ‖zk+1−x∗‖2\|z^{k+1}-x^{*}\|^{2}).

Proof.

Theorem 3.1.

Proof.

Remark 3.1.

Remark 3.2.

4 UGM with varying parameters qkq_{k} and θk\theta_{k}

Lemma 4.1.

Theorem 4.1.

Proof.

4.1 UGM-I: Parameter rule (1−qk+1)​ak=1/θk(1-q_{k+1})a_{k}=1/\theta_{k}

Theorem 4.2.

Proof.

Lemma 4.2.

Proof.

Lemma 4.3.

Proof.

4.2 UGM-II: ⟨∇f​(xk+1),xk+1−zk+1⟩\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle-aware parameter rule

Lemma 4.4.

Proof.

Lemma 4.5.

Proof.

Theorem 4.3.

Proof.

4.3 UGM-III: ⟨∇f​(xk+1),xk+1−zk+1⟩\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle- and ‖xk+1−zk+1‖2\|x^{k+1}-z^{k+1}\|^{2}-aware parameter rule

Lemma 4.6.

Proof.

Theorem 4.4.

Proof.

5 Numerical experiments

5.1 Unconstrained quadratic optimization

Comparison results.

The role of ϱ\varrho in UGM‑I.

5.2 Logistic regression

5.3 Softmax regression

Scalar evaluation using cached logits.

Digits‑Poly2 with scalar aggregation.

6 Concluding remarks

References

Appendix A Proof on the equivalence between (HB) and (9)

Lemma A.1.

Proof.

Appendix B Reformulation of several algorithms

Lemma B.1.

Proof.

B.1 Gradient descent methods

Lemma B.2.

Proof.

Lemma B.3.

Proof.

B.2 Nesterov’s FGM

Lemma B.4.

Proof.

B.3 TM method

Lemma B.5.

Proof.

B.4 SC-OGM

Lemma B.6.

Proof.

B.5 AOR-HB

Lemma B.7.

Proof.

Appendix C Supplementary proofs for two inequalities

Lemma C.1.

Proof.

Lemma C.2.

Proof.

Remark 2.1 (The role of $q$ and $\theta$ ).

Lemma 3.1 (Inductive relation for $f(x^{k})-f_{}$ and $f(x^{k})-f_{}-\frac{1}{2L}\|\nabla f(x^{k})\|^{2}$ ).

Lemma 3.2 (Inductive relation for $\|z^{k+1}-x^{*}\|^{2}$ ).

4 UGM with varying parameters $q_{k}$ and $\theta_{k}$

4.1 UGM-I: Parameter rule $(1-q_{k+1})a_{k}=1/\theta_{k}$

4.2 UGM-II: $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ -aware parameter rule

4.3 UGM-III: $\langle\nabla f(x^{k+1}),x^{k+1}\!-\!z^{k+1}\rangle$ - and $\|x^{k+1}-z^{k+1}\|^{2}$ -aware parameter rule

The role of $\varrho$ in UGM‑I.