Nonsmooth Optimization via Orthogonalized Momentum

Lexiao Lai^† and Tianyi Lin^⋄ and Jiayu Zhang^⋄

Department of Industrial Engineering and Operations Research^⋄

Columbia University

Department of Mathematics, The University of Hong Kong^†

September 12, 2026

Abstract

Modern real application problems involve matrix-valued parameters, yet conventional optimizers treat them as vectors, thereby motivating matrix-aware methods that exploit input-output geometry, such as Muon which orthogonalizes the momentum matrices before parameter updates. Its empirical success raises a conceptual question: can orthogonalized momentum remain effective beyond smooth optimization? This paper studies this question for locally Lipschitz functions using a generalized derivative framework compatible with backpropagation. Our first contribution is to identify a key limitation: for every fixed momentum factor $\beta\in[0,1)$ , Muon can fail to approach the global optimal solution of a convex Lipschitz objective from almost every initialization, when step sizes adapt to the full gradient history. The failure can occur even along bounded iterates. Our example is inspired by the one of Parshakova et al. (2026) which only covers $\beta\in[0,\frac{1}{2})$ . Then, we show that the obstruction lies in fixed momentum rather than orthogonalization. Indeed, when the momentum factor is adaptive and approaches $1$ together with a vanishing step size, Muon recovers asymptotic convergence for nonconvex nonsmooth optimization under the boundedness and regularity conditions. Moreover, we propose a MAGD method, which combines orthogonalized momentum with gradient and weights them based on their relative progress. MAGD retains asymptotic convergence in nonconvex settings and achieves a convergence rate of $O(\min\{m,n\}\epsilon^{-2})$ in convex settings. A lower bound is established to show the optimal dimension dependence. Experiments on synthetic problems, image classification, and LLM pretraining show that our method is a simple and practical alternative to Muon. Together, our results characterize when orthogonalized momentum fails without smoothness and how it can be made reliable and we hope that the analysis may be useful more broadly.

1 Introduction

A common geometric principle underlying first-order methods is that first-order information is converted into an update through a chosen norm or regularizer. Classical algorithms have adopted Euclidean or coordinate-wise geometry after vectorizing the decision variables. This framework includes stochastic gradient descent (SGD), momentum methods, and adaptive methods such as AdaGrad, RMSProp, and Adam (Robbins and Monro, 1951; Polyak, 1964; Nesterov, 1983; Sutskever et al., 2013; Duchi et al., 2011; Tieleman and Hinton, 2012; Kingma and Ba, 2015). For matrix-valued variables, however, vectorization obscures their input-output structure. The operator norm instead directly measures how strongly a matrix amplifies its inputs. This observation has motivated matrix-aware methods whose geometry respects the role of a matrix as a linear map (Large et al., 2024; Bernstein and Newhouse, 2024; Bernstein and Newhouse, 2025; Pethick et al., 2025). Neural network training is a prominent application because most trainable parameters are naturally organized as matrix-valued layers, but the underlying problem is more general: first-order optimization in matrix-norm geometry.

A simple matrix-aware first-order method forms a momentum matrix and orthogonalizes it before taking a main step. This update is known as Muon (Jordan et al., 2024; Liu et al., 2025). For a matrix variable $W_{t}\in\mathbb{R}^{m\times n}$ , the inexact Muon recursion is

G_{t}\in\mathcal{D}_{f}(W_{t}),\quad M_{t+1}=\beta_{t}M_{t}+(1-\beta_{t})G_{t},\quad\widetilde{M}_{t+1}=\tfrac{M_{t+1}}{\|M_{t+1}\|_{F}+\sigma_{t}}\quad W_{t+1}\in W_{t}-\eta_{t}\operatorname{polar}_{\delta_{t}}(\widetilde{M}_{t+1}),

where $\mathcal{D}_{f}$ represents the first-order information returned by backpropagation and $\operatorname{polar}_{\delta}(\widetilde{M})$ is the set of all inexact orthogonalizations of $\widetilde{M}\in\mathbb{R}^{m\times n}$ within an inexactness factor $\delta\in[0,1)$ . In practice, we apply a small number of polynomial iterations to $\widetilde{M}$ to realize $\operatorname{polar}_{\delta}$ , which preserves the singular subspaces of $M$ while pushing its nonzero singular values toward $1$ (Jordan et al., 2024; Amsel et al., 2026a).

The empirical success of Muon raises a conceptual question: can orthogonalized momentum remain effective for nonsmooth optimization? Recent analyses have shown the convergence of inexact Muon with finitely many inner iterations for smooth optimization (Shulgin et al., 2026; Kim and Oh, 2026). Smoothness provides a local upper bound on the objective, ensuring that a sufficiently small step along a well-aligned direction produces a predictable decrease. This argument is unavailable for locally Lipschitz objectives, whose behavior may change abruptly at nonsmooth points (Wang, 2005). Orthogonalization creates an additional difficulty since it removes the magnitude information from the momentum matrix. Even an arbitrarily small nonzero momentum can produce a full operator-norm step. It is thus unclear whether momentum stabilizes these normalized updates or leads to nonconvergence.

The recent work of Parshakova et al. (2026) gives a negative answer for convex objectives. Indeed, for diagonal cases, Muon reduces to signed momentum (Bernstein et al., 2018), allowing the counterexamples for sign-based methods to be embedded into the matrix recursion. Based on this insight, they construct convex Lipschitz objectives on which Muon with fixed momentum does not converge. One result covers every fixed momentum factor $\beta\in[0,1)$ but restricts the step size schedule and initialization, while another result allows the step size chosen from the full gradient information but only covers $\beta\in[0,\frac{1}{2})$ . They also combine error feedback with Muon and restores the convergence for convex objectives. Their results establish a novel obstruction but leave its precise source unresolved. In particular, does almost everywhere failure extend to every fixed factor $\beta\in[0,1)$ ? Can an adaptive momentum schedule restore convergence, even when orthogonalization is computed inexactly? Can one obtain a finite-time guarantee with the optimal dimension dependence while retaining the effectiveness of orthogonalized momentum?

We study these questions in a nonconvex and locally Lipschitz setting. We formulate our analysis using conservative fields, set-valued mappings that encode generalized first-order information for nonsmooth functions. (Bolte and Pauwels, 2021). This framework accommodates the outputs of backpropagation while retaining a chain rule. We identify the nonconvergence counterexamples and derive convergence results using conservative fields. The convex Lipschitz function class is a simple and clean benchmark: every stationary point is globally optimal, Lipschitz continuity bounds the subgradients, and Euclidean subgradient descent has the optimal $O(\epsilon^{-2})$ worst-case dependence (Nemirovski and Yudin, 1983; Beck, 2017; Nesterov, 2018). Nonconvergence counterexample is established on this class and cannot be attributed to a nonconvex landscape or unbounded first-order information.

The negative result comes from the interaction between the fixed momentum and orthogonalization. At a nonsmooth point, the first-order information can change abruptly. Then, a fixed momentum factor transmits this change at a rate independent of the step size, while orthogonalization turns every nonzero momentum into a normalized direction. Our construction exploits this behavior to preserve a nonzero component of the objective, regardless of chosen step sizes. For the positive results, we synchronize the momentum and parameter updates by setting

\beta_{t}=1-\tau\eta_{t},\quad\eta_{t}>0,\quad\eta_{t}\to 0,\quad\sum_{t=0}^{+\infty}\eta_{t}=+\infty.

(1.1)

Related one-timescale couplings appear in stochastic heavy-ball schemes (Gadat et al., 2018, Eq. (4)), nonsmooth Adam-family methods (Xiao et al., 2024), and, for $\tau=1$ , a variance-reduced Muon analysis (Chang et al., 2025, Theorem 4.2). Under the schedule (1.1), the momentum and parameter updates evolve on the same time scale and controlled approximation errors vanish asymptotically. This yields a Lyapunov function $\varphi(W,M)=f(W)+\tau^{-1}\|M\|_{\mathrm{nuc}}$ , for which the conservative field chain rule and operator-nuclear norm duality yield $\frac{\mathrm{d}}{\mathrm{d}s}\varphi(W(s),M(s))=-\|M(s)\|_{\mathrm{nuc}}$ . Coupling the two updates restores a dissipative structure which is absent under fixed momentum and yields the asymptotic convergence. To obtain a finite-time guarantee for convex objectives, we propose MAGD, which combines orthogonalized momentum with gradient descent and weights them based on their relative progress. The weighting rule is inspired by the one of Cesa-Bianchi and Lugosi (2006) and allows our method to inherit the safeguard without modifying the Muon direction.

Contributions.

The contribution of this paper consists in studying when orthogonalized momentum fails and how it can be made reliable beyond smoothness, and in establishing matching upper and lower complexity bounds for convex Lipschitz optimization in operator-norm geometry. In further detail:

1.

We establish a sharp separation between fixed and adaptive momentum beyond smoothness. For every fixed $\beta\in[0,1)$ , Muon fails to approach the global optimal solution of a convex and Lipschitz objective from almost every initialization, even when the step sizes adapt to full gradient history. The failure can occur even along bounded iterates. By using the coupled schedule in Eq. (1.1) and controlled inner approximation, inexact Muon converges asymptotically and the objective values converge to the global optimal value for convex objectives.
2.

We establish asymptotic and finite-time guarantees for MAGD. Under the conservative field model, an appropriate diminishing step size schedule yields stationarity of every cluster point as well as convergence of the objective values. For convex Lipschitz objectives on $\mathbb{R}^{m\times n}$ , MAGD can return an $\epsilon$ -optimal point within $O(\min\{m,n\}\epsilon^{-2})$ iterations. A lower bound is then established to show the optimal dimension dependence. Experiments on synthetic problems, image classification, and LLM pretraining show that our method is a simple and practical alternative to Muon.

Further related work.

In addition to the aforementioned works, we provide a few additional remarks regarding related work on Muon and related methods, nonsmooth optimization, lower bound analysis, and normalized and sign methods.

The existing theory of Muon and related scale-invariant methods has developed along two lines. Early analyses consider the exact polar update and establish convergence for several variants involving momentum, constraints, star-convexity, weight decay and variance reduction (Li and Hong, 2025; Shen et al., 2025; Sato et al., 2025; Sfyraki and Wang, 2025; Chang et al., 2025). Recent works have moved toward the inexact orthogonalization used in practice. Shulgin et al. (2026) quantified how approximation error interacts with the step size and momentum, Kim and Oh (2026) proved convergence with finite Newton-Schulz iterations, Qian and Richtárik (2026) studied inexact Muon in the degenerate spectral regime under generalized smoothness condition, and Zhang and Lin (2026) established optimal rates for stochastic scale-invariant methods in operator-norm geometry and implemented the resulting matrix updates using inexact polar computations. Complementary analyses examine spectral updates through one-step progress, local curvature models, and non-Euclidean trust regions (Davis and Drusvyatskiy, 2025; Su, 2025; Kovalev, 2025). Alongside this theory, a rapidly growing literature develops practical Muon variants and scalable implementations, including large-scale Muon and MuonClip, NorMuon and Muon², adaptive and Gram-based orthogonalization and full-stack distributed systems (Liu et al., 2025; Team, 2025; Li et al., 2026; Liu et al., 2026; Zhuang et al., 2026; Zhang et al., 2026; Khona et al., 2026; Amsel et al., 2026b). Existing convergence theory remains tied to smoothness or generalized smoothness, whereas we ask what remains true for locally Lipschitz objectives.

The framework used here belongs to a line of work on set-valued first-order dynamics. Differential inclusion methods connect stochastic approximation with continuous-time limits (Benaïm et al., 2005) and stochastic subgradient methods converge on tame objectives under some regularity conditions (Davis et al., 2020). For automatic differentiation, Bolte and Pauwels (2021) has introduced conservative fields and developed their calculus, while Lewis and Tian (2021) characterized their specific structure on semialgebraic functions through Whitney stratifications, building on earlier variational analysis of stratifiable functions (Bolte et al., 2007). Related tools have since been applied to adaptive first-order methods (Xiao et al., 2024; Hu et al., 2025; Ding and Toh, 2025; Ding et al., 2025), while Jiang et al. (2026) obtained the finite-time guarantees for nonconvex nonsmooth optimization using matrix online learning methods through smoothing. Our analysis instead applies conservative field dynamics directly to orthogonalized momentum for nonsmooth optimization.

Beyond the aforementioned classical convex lower bounds, the information-theoretic arguments give minimax bounds for stochastic first-order oracles, while local oracle constructions extend such results to randomized algorithms and general normed spaces (Agarwal et al., 2012; Braun et al., 2017; Braun et al., 2024). In smooth nonconvex settings, Carmon et al. (2020); Carmon et al. (2021) established deterministic oracle lower bounds for finding stationary points, and Arjevani et al. (2023) developed corresponding stochastic lower bounds. In nonsmooth nonconvex settings, the notion of Clarke stationarity (Clarke, 1990) yields impossibility and lower bound results for first-order methods (Zhang et al., 2020; Kornowski and Shamir, 2022). Our lower bound is derived for convex first-order optimization in matrix-norm geometry. Indeed, we adapt hidden-direction arguments to operator-norm geometry using nearly orthogonal rank-one matrices and operator-nuclear norm duality.

Finally, the diagonal reduction of polar updates connects Muon with normalized and sign methods, which likewise discard magnitude information (Balles and Hennig, 2018; Bernstein et al., 2018; Cutkosky and Mehta, 2020). In this context, Karimireddy et al. (2019) has shown that sign subgradient descent can fail on a convex Lipschitz objective and that error feedback can restore convergence. Error feedback has since been developed for biased compression and for Muon and related methods (Beznosikov et al., 2023; Demidovich et al., 2023; Gruntkowska et al., 2026; Parshakova et al., 2026). Our MAGD method follows a different route by leaving the inexact Muon direction unchanged and controlling its influence through a separate gradient branch, yielding an improved performance across different tasks.

Organization.

The remainder is organized as follows. In Section 2, we review conservative fields, Muon, and black-box first-order oracle model. In Section 3, we present the counterexample, establish asymptotic convergence for inexact Muon under increasing momentum, introduce MAGD, and prove the matching upper and lower bounds. In Section 4, we conduct the experiments on synthetic and real data and the numerical results show the efficiency of our method. We conclude this paper in Section 5.

2 Preliminaries

We first review tools from nonsmooth analysis and state the assumptions used throughout the paper. Then, we introduce the inexact Muon and its operator-norm geometry, followed by the first-order oracle model used in our subsequent analysis.

2.1 Nonsmooth analysis and conservative fields

We consider the matrix optimization problem given by

\min_{X\in\mathbb{R}^{m\times n}}f(X).

Throughout, we assume that $f$ is locally Lipschitz, meaning that it is Lipschitz on a neighborhood of every point in $\mathbb{R}^{m\times n}$ . Rademacher’s theorem guarantees that such a function is differentiable almost everywhere and the Clarke subdifferential of $f$ (Clarke, 1990) at $W\in\mathbb{R}^{m\times n}$ is defined by

\partial f(W):=\textnormal{conv}\left\{V\in\mathbb{R}^{m\times n}:\exists W_{k}\to W\text{ such that $f$ is differentiable at every $W_{k}$ and }\nabla f(W_{k})\to V\right\},

where $\textnormal{conv}(A)$ denotes the convex hull of $A$ for any $A\subseteq\mathbb{R}^{m\times n}$ . However, backpropagation might not output an element of the Clarke subdifferential during neural network training (Bolte and Pauwels, 2021, Remark 13). To capture what backpropagation actually returns, a more general model of calculus, called conservative field, was introduced in (Bolte and Pauwels, 2021). In particular, a conservative field of $f$ maps each $W\in\mathbb{R}^{m\times n}$ to a set of potential differentials of $f$ at $W$ , and generalizes $\partial f$ to accommodate the operations and compositions that are standard in neural networks. It becomes almost everywhere the singleton consisting of the gradient of $f$ (Bolte and Pauwels, 2021, Theorem 1).

We abbreviate any set-valued mapping $D:A\to 2^{B}$ by $D:A\rightrightarrows B$ , where $A,B$ are sets and $2^{B}$ is the power set of $B$ . The graph of $D$ is defined as $\{(x,z):x\in A,z\in D(x)\}$ . Let $I\subseteq\mathbb{R}$ be a closed and possibly unbounded interval. A function $x:I\to\mathbb{R}^{m\times n}$ is absolutely continuous if it is differentiable almost everywhere and satisfies $x(t_{2})-x(t_{1})=\int_{t_{1}}^{t_{2}}\dot{x}(s)ds$ for all $t_{1},t_{2}\in I$ . Formally, we have

Definition 2.1

Let $f\colon\mathbb{R}^{m\times n}\to\mathbb{R}$ be locally Lipschitz, and let $\mathcal{D}_{f}\colon\mathbb{R}^{m\times n}\rightrightarrows\mathbb{R}^{m\times n}$ be a set-valued mapping with closed graph and nonempty and compact values. We say that $\mathcal{D}_{f}$ is a conservative field of $f$ if, for every absolutely continuous $\gamma\colon[0,1]\to\mathbb{R}^{m\times n}$ and every measurable $v\colon[0,1]\to\mathbb{R}^{m\times n}$ with $v(t)\in\mathcal{D}_{f}(\gamma(t))$ for almost every $t\in[0,1]$ , the following statement holds,

f(\gamma(1))-f(\gamma(0))=\int_{0}^{1}\big\langle\dot{\gamma}(t),v(t)\big\rangle dt.

For an objective $f$ represented by a finite computational graph whose elementary functions admit conservative fields, the possible outputs of backpropagation form a conservative field for $f$ (Bolte and Pauwels, 2021, Theorem 8). We impose the following assumption on $f$ and its conservative field $\mathcal{D}_{f}$ , which has become recognized as standard in the literature (Bolte and Pauwels, 2021; Davis et al., 2020; Ding et al., 2025).

Assumption 2.2

The objective function $f$ admits a convex-valued conservative field $\mathcal{D}_{f}:\mathbb{R}^{m\times n}\rightrightarrows\mathbb{R}^{m\times n}$ for which the weak Sard condition holds. Equivalently, we have (i) $f$ is locally Lipschitz, (ii) $\mathcal{D}_{f}$ is a convex-valued conservative field for $f$ , and (iii) $\{f(W):0\in\mathcal{D}_{f}(W)\}$ has empty interior in $\mathbb{R}$ .

The objective functions obtained from standard architectures, including convolutional networks (Goodfellow et al., 2016, Chapter 9) and transformers (Vaswani et al., 2017), satisfy Assumption 2.2. Indeed, the automatic differentiation construction of Bolte and Pauwels (2021, Section 5.2 and Corollary 6) yields the conservative field containing the outputs of backpropagation. Taking its convex hull can preserve conservativity and does not exclude any such output (Bolte and Pauwels, 2021, Remark 3(e)), while the nonsmooth Morse-Sard theorem gives the weak Sard condition (Bolte and Pauwels, 2021, Theorem 5). To obtain finite-time guarantees for our algorithm, we impose the additional structure.

Assumption 2.3

The objective function $f$ is convex, and $\mathcal{D}_{f}=\partial f$ .

For convex $f$ , the Clarke subdifferential coincides with the convex subdifferential, so $\partial f$ can be understood in either sense (Clarke, 1990, Proposition 2.2.7) without abuse of notation.

2.2 Muon and operator-norm geometry

We study Muon in the operator-norm geometry. For $A\in\mathbb{R}^{m\times n}$ , let $\|A\|_{\rm op}:=\max_{\|x\|_{2}\leq 1}\|Ax\|_{2}$ , where $\|\cdot\|_{2}$ denotes the Euclidean norm, and let $\|A\|_{\rm nuc}$ denote the nuclear norm. These norms are dual under the Frobenius inner product.

Newton-Schulz (Jordan et al., 2024) and PolarExpress (Amsel et al., 2026a) are standard subroutines for Muon to approximate orthogonalization. At each iteration of inexact Muon, these methods start from a normalized momentum matrix and apply a finite number of polynomial iterations. Given that $M$ is a momentum matrix and $\sigma\geq 0$ is a parameter in normalization, the resulting approximation has the following form of

\widetilde{M}\textnormal{poly}(\widetilde{M}^{\top}\widetilde{M}),\quad\textnormal{where }\widetilde{M}=\begin{cases}\tfrac{M}{\|M\|_{F}+\sigma},&M\not=0,\\ 0,&M=0,\end{cases}

where $\|\cdot\|_{F}$ denotes the Frobenius norm and $\textnormal{poly}(\cdot)$ is the scalar polynomial determined by the chosen method and number of polynomial iterations. These subroutines preserve the nonzero singular subspaces and modify only the singular values. Motivated by these properties, we use the following abstract model. For $\delta\in[0,1)$ , $Z\in\mathbb{R}^{m\times n}$ with $\|Z\|_{F}\leq 1$ and $r:=\operatorname{rank}(Z)$ , we define $\operatorname{polar}_{\delta}(0):=\{0\}$ and, for $Z\neq 0$ ,

\operatorname{polar}_{\delta}(Z):=\left\{U\operatorname{Diag}(\phi_{1},\ldots,\phi_{r})V^{\top}\ \middle|\ \begin{array}[]{l}Z=U\operatorname{Diag}(\sigma_{1},\dots,\sigma_{r})V^{\top}\text{ is a compact SVD},\\ 0\leq\phi_{j}\leq 1+\delta\quad\textnormal{for every }j,\\ \phi_{j}\geq 1-\delta\quad\textnormal{whenever }\sigma_{j}\geq\delta\end{array}\right\}.

(2.1)

Given a sequence $\{\delta_{t}\}_{t\geq 0}\subseteq[0,1)$ and $\{\sigma_{t}\}_{t\geq 0}\subseteq[0,+\infty)$ , the inexact Muon recursion is

G_{t}\in\mathcal{D}_{f}(W_{t}),\quad M_{t+1}=\beta_{t}M_{t}+(1-\beta_{t})G_{t},\quad\widetilde{M}_{t+1}=\tfrac{M_{t+1}}{\|M_{t+1}\|_{F}+\sigma_{t}},\quad W_{t+1}\in W_{t}-\eta_{t}\operatorname{polar}_{\delta_{t}}(\widetilde{M}_{t+1})

(2.2)

with the convention of $\widetilde{M}_{t+1}=0$ when $M_{t+1}=0$ and $\sigma_{t}=0$ . We note that $\operatorname{polar}_{0}(Z)$ is a singleton set for any $Z\in\mathbb{R}^{m\times n}$ such that $\|Z\|_{F}\leq 1$ (Higham, 2008, Theorem 8.1). Therefore, if $\delta_{t}\equiv 0$ and $M_{t+1}=U_{t+1}\Sigma_{t+1}V_{t+1}^{\top}$ is a compact SVD, the inexact Muon scheme (2.2) reduces to the exact Muon recursion

\displaystyle W_{t+1}\in W_{t}-\eta_{t}\operatorname{polar}_{0}\left(\tfrac{M_{t+1}}{\|M_{t+1}\|_{F}+\sigma_{t}}\right),\quad\textnormal{equivalently},\quad W_{t+1}=W_{t}-\eta_{t}U_{t+1}V_{t+1}^{\top}.

The exact Muon recursion thus minimizes a linear model over an operator-norm ball centered at $W_{t}$ (Pethick et al., 2025), given by, $W_{t+1}\in\mathop{\rm argmin}_{\|W-W_{t}\|_{\rm op}\leq\eta_{t}}\langle W-W_{t},M_{t+1}\rangle$ .

2.3 Oracle complexity of convex nonsmooth optimization

We use the black-box first-order oracle model (Nemirovski and Yudin, 1983; Braun et al., 2017) to measure the number of oracle calls needed to optimize in the operator-norm geometry, where the unknown objective $f$ belongs to a known class $\mathcal{F}$ of convex Lipschitz functions on $\mathbb{R}^{m\times n}$ , and accuracy is measured relative to its minimum over a nonempty compact convex set $\mathcal{X}$ . At each iteration, an algorithm queries a point using the preceding oracle replies, and the oracle returns the function value and a subgradient at that point. Given $\epsilon>0$ , the algorithm seeks to output $\widehat{W}$ satisfying $f(\widehat{W})\leq\min_{W\in\mathcal{X}}f(W)+\epsilon$ . This paradigm includes projected subgradient descent and mirror descent (Beck, 2017), conditional gradient (Jaggi, 2013) and Muon (Jordan et al., 2024).

More specifically, given $\mathcal{F}$ and $\mathcal{X}$ , an oracle $\mathsf{O}$ supplies partial information $\mathsf{O}_{f}(W)$ about an unknown instance $f\in\mathcal{F}$ at a queried point $W\in\mathbb{R}^{m\times n}$ . The algorithm may query and report anywhere in $\mathbb{R}^{m\times n}$ . We consider a first-order oracle $\mathsf{O}$ , which answers a query $W\in\mathbb{R}^{m\times n}$ by

\mathsf{O}:\mathcal{F}\times\mathbb{R}^{m\times n}\to\mathbb{R}\times\mathbb{R}^{m\times n},\quad\mathsf{O}_{f}(W)=(f(W),g),\quad g\in\partial f(W),

where $f(W)$ is the function value and $g$ is a subgradient of $f$ at $W$ . We also require $\mathsf{O}$ to be local: if $f_{1},f_{2}\in\mathcal{F}$ agree on a neighborhood of $W$ , we have $\mathsf{O}_{f_{1}}(W)=\mathsf{O}_{f_{2}}(W)$ . Thus, the oracle reply at $W$ depends only on the behavior of $f$ near $W$ .

Let $\mathcal{A}(\mathsf{O})$ denote the set of deterministic algorithms based on $\mathsf{O}$ . Each algorithm proceeds by maintaining $\Pi_{q}:=\left((X^{1},\mathsf{O}_{f}(X^{1})),\ldots,(X^{q},\mathsf{O}_{f}(X^{q}))\right)$ for $q\geq 0$ (with $\Pi_{0}:=\emptyset$ ) and applying a decision rule that either outputs a point $\widehat{X}\in\mathbb{R}^{m\times n}$ or continues to select the next query $X^{q+1}\in\mathbb{R}^{m\times n}$ . Given $\mathsf{A}\in\mathcal{A}(\mathsf{O})$ , $f\in\mathcal{F}$ and a tolerance $\epsilon>0$ , we write $\mathsf{T}_{\mathsf{A}}(f,\epsilon)$ for the number of queries that $\mathsf{A}$ makes to reach an $\epsilon$ -optimal solution. Equivalently, $\mathsf{T}_{\mathsf{A}}(f,\epsilon)=q$ if $\mathsf{A}$ stops after $q$ completed calls with an output satisfying $f(\widehat{X})-\min_{X\in\mathcal{X}}f(X)\leq\epsilon$ , and $\mathsf{T}_{\mathsf{A}}(f,\epsilon)=+\infty$ otherwise. The worst-case oracle complexity is defined by

\mathfrak{m}_{\rm wcc}(\mathcal{X},\mathcal{F},\mathsf{O},\epsilon):=\inf_{\mathsf{A}\in\mathcal{A}(\mathsf{O})}\sup_{f\in\mathcal{F}}\mathsf{T}_{\mathsf{A}}(f,\epsilon).

(2.3)

Let $\Delta(\mathcal{F})$ denote the set of probability distributions on $\mathcal{F}$ , we define the distributional complexity as

\mathfrak{m}_{\rm dc}(\mathcal{X},\mathcal{F},\mathsf{O},\epsilon):=\sup_{\mathcal{P}\in\Delta(\mathcal{F})}\ \inf_{\mathsf{A}\in\mathcal{A}(\mathsf{O})}\mathbb{E}_{f\sim\mathcal{P}}[\mathsf{T}_{\mathsf{A}}(f,\epsilon)].

(2.4)

Bounding this quantity yields stronger lower bounds. Indeed, Yao’s minimax principle (Yao, 1977) yields

\mathfrak{m}_{\rm dc}(\mathcal{X},\mathcal{F},\mathsf{O},\epsilon)\leq\mathfrak{m}_{\rm wcc}(\mathcal{X},\mathcal{F},\mathsf{O},\epsilon).

(2.5)

which implies that a lower bound for $\mathfrak{m}_{\rm dc}$ is also the lower bound for $\mathfrak{m}_{\rm wcc}$ .

3 Main Results

We first show that Muon fails to converge to a global optimal solution for some convex and Lipschitz functions when the momentum factor is constant. Then, we show that inexact Muon with a particular momentum schedule and vanishing polar error converges asymptotically in general nonconvex settings. We also propose a new algorithm based on Muon and proves that it achieves the asymptotic convergence in nonconvex settings and the optimal convergence rate in convex settings.

3.1 Convergence behavior of Muon

Parshakova et al. (2026, Counterexample 3.3) provides a convex and Lipschitz function for which Muon with $\beta_{t}\equiv\beta\in[0,1)$ and a nonincreasing step size sequence fails to converge from some starting points, and Parshakova et al. (2026, Counterexample 3.4) strengthens this conclusion to almost every starting point and adaptive step sizes based on the full gradient history, but only for $\beta_{t}\equiv\beta\in[0,\frac{1}{2})$ . Our first result closes the gap by proving this almost everywhere failure for every $\beta\in[0,1)$ .

Theorem 3.1

For any fixed $m,n\geq 2$ and $\beta\in[0,1)$ , there exists a convex and Lipschitz function $f:\mathbb{R}^{m\times n}\to\mathbb{R}$ such that the iterates generated by Muon in Eq. (2.2) with $\delta_{t}\equiv 0$ , $\mathcal{D}_{f}=\partial f$ , $\beta_{t}\equiv\beta$ , $M_{0}=G_{0}$ , and arbitrary adaptive step sizes $\{\eta_{t}\}_{t\geq 0}$ based on the full subgradient history satisfy

\inf_{t\geq 0}f(W_{t})>f^{\star}:=\inf_{W\in\mathbb{R}^{m\times n}}f(W).

for almost every $W_{0}\in\mathbb{R}^{m\times n}$ ,

Proof. For any matrix $X$ , we denote $[X]_{ij}$ as its $(i,j)$ -th entry. We define

f(W)=A_{1}|[W]_{11}+[W]_{22}|+A_{2}\max\{[W]_{11}-[W]_{22},0\}+\max\{[W]_{22}-[W]_{11},0\}

where $A_{1},A_{2}$ are positive constants and depend only on $\beta$ . For simplicity, we denote the set-valued sign function by $\operatorname{sgn}:\mathbb{R}\rightrightarrows\mathbb{R}$ as

\operatorname{sgn}(x):=\begin{cases}\{-1\},&\textnormal{if }x<0,\\ [-1,1],&\textnormal{if }x=0,\\ \{1\},&\textnormal{otherwise}.\end{cases}

Step 1: We can rewrite Muon’s updates in the coordinate system of $[W]_{11}+[W]_{22}$ and $[W]_{11}-[W]_{22}$ using the form of $f$ . Indeed, we define $p_{t}=[W_{t}]_{11}+[W_{t}]_{22}$ , $q_{t}=[W_{t}]_{11}-[W_{t}]_{22}$ , $\alpha_{t}=\frac{1}{2}([M_{t}]_{11}+[M_{t}]_{22})$ and $\zeta_{t}=\frac{1}{2}([M_{t}]_{11}-[M_{t}]_{22})$ . Since $f$ depends only on $[W]_{11}$ and $[W]_{22}$ , both subgradient and momentum are supported only on the diagonal entries as follows,

[M_{t+1}]_{11}=(\alpha_{t+1}+\zeta_{t+1}),\quad[M_{t+1}]_{22}=(\alpha_{t+1}-\zeta_{t+1}),\quad[M_{t+1}]_{ij}=0\textnormal{ if }i\neq j.

Since $M_{t+1}=U_{t+1}\Sigma_{t+1}V_{t+1}^{\top}$ is a compact SVD, $P_{t+1}=U_{t+1}V_{t+1}^{\top}$ depends on the same two entries and all other entries are $0$ . We write $u_{t+1}:=[P_{t+1}]_{11}$ and $v_{t+1}:=[P_{t+1}]_{22}$ . By definition, we have

\begin{array}[]{ll}u_{t+1}\in\operatorname{sgn}([M_{t+1}]_{11})=\operatorname{sgn}(\alpha_{t+1}+\zeta_{t+1}),&p_{t+1}=p_{t}-\eta_{t}(u_{t+1}+v_{t+1}),\\ v_{t+1}\in\operatorname{sgn}([M_{t+1}]_{22})=\operatorname{sgn}(\alpha_{t+1}-\zeta_{t+1}),&q_{t+1}=q_{t}-\eta_{t}(u_{t+1}-v_{t+1}).\end{array}

In addition, we have $\alpha_{t+1}=\beta\alpha_{t}+(1-\beta)a_{t}$ and $\zeta_{t+1}=\beta\zeta_{t}+(1-\beta)b_{t}$ where

a_{t}\in A_{1}\operatorname{sgn}(p_{t}),\quad b_{t}\in\begin{cases}\{A_{2}\},&q_{t}>0,\\ \{-1\},&q_{t}<0,\\ [-1,A_{2}],&q_{t}=0.\end{cases}

If $|\zeta_{t+1}|>|\alpha_{t+1}|$ , we have $u_{t+1}$ and $v_{t+1}$ have opposite signs and $\frac{1}{2}(u_{t+1}-v_{t+1})\in\operatorname{sgn}(\zeta_{t+1})$ . Thus, we have

p_{t+1}=p_{t},\quad q_{t+1}=q_{t}-2\eta_{t}d_{t},\quad\textnormal{where}\quad d_{t}\in\operatorname{sgn}(\zeta_{t+1}).

(3.1)

Step 2: We define $T_{0}:=\inf\{t\geq 0:q_{t}=0\}$ and show that there are $A_{1},A_{2}>0$ such that $|\zeta_{t+1}|>|\alpha_{t+1}|$ for all $t<T_{0}$ . Note that we will show in Step 3 that $T_{0}=+\infty$ for almost every $W_{0}\in\mathbb{R}^{m\times n}$ .

We consider the case of $\beta=0$ . Indeed, we set $A_{1}=\frac{1}{4}$ and $A_{2}=1$ . For every $t<T_{0}$ , we have $q_{t}\neq 0$ . This implies $\zeta_{t+1}=b_{t}\in\{-1,1\}$ and $|\alpha_{t+1}|=|a_{t}|\leq A_{1}$ . Hence, $|\zeta_{t+1}|=1>|\alpha_{t+1}|$ .

We proceed to the general case of $\beta\in(0,1)$ . Here, the construction of $A_{1}$ and $A_{2}$ is implicit. Indeed, we define $g:(1,\frac{1}{\beta})\to\mathbb{R}$ by $g(x):=\inf_{i\geq 0,j\geq 1}|\beta^{i}-x(1-\beta^{j})|$ and show that there exists $\mu\in(1,\frac{1}{\beta})$ such that $g(\mu)>0$ . Indeed, for any $x\in(1,\frac{1}{\beta})$ , we define

\mathcal{A}=\{\beta^{i}:i\geq 0\}\cup\{0\},\quad\mathcal{B}_{x}:=\{x(1-\beta^{j}):j\geq 1\}\cup\{x\},\quad\mathcal{B}=\left\{\tfrac{\beta^{i}}{1-\beta^{j}}:i\geq 0,j\geq 1\right\}.

Both $\mathcal{A}$ and $\mathcal{B}_{x}$ are compact. Since $x>1$ , we have $0\notin\mathcal{B}_{x}$ and $x\notin\mathcal{A}$ . Thus, if $x\notin\mathcal{B}$ , then $\mathcal{A}\cap\mathcal{B}_{x}=\varnothing$ and $g(x)=\operatorname{dist}(\mathcal{A},\mathcal{B}_{x})>0$ . Since $\mathcal{B}$ is countable, we can choose $\mu\in(1,\frac{1}{\beta})\setminus\mathcal{B}$ , which gives $g(\mu)>0$ . Then, we set $K=\max\{\lceil\log_{1/\beta}(2/\delta)\rceil-1,\lceil\log_{1/\beta}(2/(1-\beta))\rceil\}$ where $\delta=g(\mu)$ , and define

A_{1}=\tfrac{1}{2}\min\left\{\tfrac{1}{2},\tfrac{\delta}{2},1-\beta\mu\right\}>0,\quad A_{2}=\tfrac{\mu}{\beta^{K}}-1>0.

Then, we show $|\zeta_{t+1}|>|\alpha_{t+1}|$ for all $t<T_{0}$ . By using the definition of $u_{t+1}$ and $v_{t+1}$ and $\operatorname{sgn}=\partial|\cdot|$ , we have $0\leq(u_{t+1}-v_{t+1})((\alpha_{t+1}+\zeta_{t+1})-(\alpha_{t+1}-\zeta_{t+1}))=2\zeta_{t+1}(u_{t+1}-v_{t+1})$ . In each of the cases $\zeta_{t+1}>0$ , $\zeta_{t+1}=0$ , and $\zeta_{t+1}<0$ , the $q$ -update can be written as

q_{t+1}=q_{t}-\lambda_{t}s_{t+1}\textnormal{ for }\lambda_{t}\in[0,2\eta_{t}]\textnormal{ and }s_{t+1}\in\operatorname{sgn}(\zeta_{t+1}).

(3.2)

Since $M_{0}=G_{0}$ , we have $|\alpha_{0}|=|a_{0}|\leq A_{1}$ and $\zeta_{0}=b_{0}\in[-1,A_{2}]$ . By induction, we have $|\alpha_{t}|\leq A_{1}$ and $\zeta_{t}\in[-1,A_{2}]$ for all $t\geq 0$ . Thus, it remains to show that $|\zeta_{t+1}|\geq 2A_{1}$ for all $t<T_{0}$ .

Before $T_{0}$ , we call a maximal consecutive block of indices on which $q_{t}<0$ a negative run, and define a positive run analogously. Fix $t<T_{0}$ . If $q_{t}>0$ , then $t$ is in a positive run starting at some $t^{\prime}\leq t$ and

\zeta_{t+1}=(1-\beta^{t+1-t^{\prime}})A_{2}+\beta^{t+1-t^{\prime}}\zeta_{t^{\prime}}\geq(1-\beta^{t+1-t^{\prime}})A_{2}-\beta^{t+1-t^{\prime}}\geq(1-\beta)A_{2}-\beta.

Since $K\geq\lceil\log_{1/\beta}(2/(1-\beta))\rceil$ , we have $\beta^{K}\leq\frac{1-\beta}{2}$ . By definition of $A_{2}$ , we have $(1-\beta)A_{2}-\beta>1\geq 2A_{1}$ . Thus, we have $\zeta_{t+1}\geq 2A_{1}$ . Then, it remains to show $q_{t}<0$ implies $|\zeta_{t+1}|\geq 2A_{1}$ .

We let $t^{\prime}\leq t$ start the negative run containing $t$ and define

\mathcal{E}:=\{-1,A_{2}\}\cup\bigcup_{r\geq 1}I_{r},\textnormal{ where }I_{r}:=[(1-\beta^{r})A_{2}-\beta^{r},(1-\beta^{r})A_{2}]\textnormal{ for all }r\geq 1.

We claim that $\zeta_{t^{\prime}}\in\mathcal{E}$ . Indeed, we first consider $t^{\prime}=0$ and obtain that $q_{0}<0$ and $\zeta_{t^{\prime}}=b_{0}=-1$ . Furthermore, we consider $t^{\prime}>0$ and and the preceding positive run starts at $t=0$ . This implies $\zeta_{i}=A_{2}$ for all $i\leq t^{\prime}$ . Thus, we have $\zeta_{t^{\prime}}=A_{2}$ . Finally, we consider $t^{\prime}>0$ and the preceding positive run starts at $t=\ell>0$ . This implies $q_{\ell-1}<0<q_{\ell}$ , which together with Eq. (3.2) yields $\lambda_{\ell-1}>0$ and $s_{\ell}<0$ . Since $s_{\ell}\in\operatorname{sgn}(\zeta_{\ell})$ , we have $\zeta_{\ell}\leq 0$ . Thus, we have $q_{\ell},\ldots,q_{t^{\prime}-1}>0$ and $q_{t^{\prime}}<0$ . This implies $\zeta_{t^{\prime}}=(1-\beta^{r})A_{2}+\beta^{r}\zeta_{\ell}\in I_{r}$ where $r=t^{\prime}-\ell\geq 1$ . Therefore, we have

|\zeta_{t+1}|=|\beta^{k}\zeta_{t^{\prime}}-(1-\beta^{k})|\geq\beta^{k}\min\left\{\inf_{r\geq 1}\operatorname{dist}(I_{r},\tfrac{1}{\beta^{k}}-1),|-1-(\tfrac{1}{\beta^{k}}-1)|,|A_{2}-(\tfrac{1}{\beta^{k}}-1)|\right\}.

where $k=t+1-t^{\prime}$ is the length of the run up to $t$ .

It remains to prove $D_{0}^{(k)}=|1-(1+A_{2})\beta^{k}|\geq 2A_{1}$ and $D_{r}^{(k)}=\beta^{k}\operatorname{dist}(I_{r},\frac{1}{\beta^{k}}-1)\geq 2A_{1}$ for all $r\geq 1$ . Indeed, we consider the case of $k\leq K$ . Then, we have

\inf_{r\geq 1}D_{r}^{(k)}=\inf_{r\geq 1}\inf_{s\in[0,\beta^{r}]}|1-(1-\beta^{r})\beta^{k}(A_{2}+1)-s\beta^{k}|\geq\inf_{r\geq 1}\ |1-(1-\beta^{r})\beta^{k}(A_{2}+1)|-\beta^{k+r}.

For all $r\geq 1$ , we have $\beta^{k+r}\leq\beta^{k+1}\leq\frac{\delta}{2}\beta^{k-K}$ . By the definition of $K$ , we have

|1-(1-\beta^{r})\beta^{k}(A_{2}+1)|=|1-(1-\beta^{r})\mu\beta^{k-K}|\geq\beta^{k-K}\delta.

Thus, we have $\inf_{r\geq 1}D_{r}^{(k)}\geq\frac{\delta}{2}\geq 2A_{1}$ and

D_{0}^{(k)}=|1-\mu\beta^{k-K}|\geq(\mu-1)\beta^{k-K}\geq\mu-1\geq\delta\geq 2A_{1}.

Then, we consider the case of $k>K$ . Since $\frac{1}{\beta^{k}}-1\geq\frac{1}{\beta^{K+1}}-1>\frac{\mu}{\beta^{K}}-1=A_{2}$ , we have

\inf_{r\geq 1}D_{r}^{(k)}\geq D_{0}^{(k)}=|1-(1+A_{2})\beta^{k}|=1-\mu\beta^{k-K}\geq 1-\mu\beta\geq 2A_{1}.

Since $|\alpha_{t+1}|\leq A_{1}<2A_{1}\leq|\zeta_{t+1}|$ , Eq. (3.1) holds for all $t<T_{0}$ .

Step 3: We show that $T_{0}=+\infty$ for almost every $W_{0}$ . Indeed, we let $\mathcal{G}_{\beta}$ be the set of subgradients of $f$ at points $\{W:|[W]_{11}|\not=|[W]_{22}|\}$ and $\eta_{t}=\phi_{t}(G_{0},\ldots,G_{t})$ where $\phi_{t}(\cdot)$ is the adaptive step-size rule. Then, we define

\mathcal{H}_{\phi}:=\left\{2\sum_{t=0}^{T-1}\phi_{t}(H_{0},\ldots,H_{t})\varepsilon_{t}:T\geq 0,H_{t}\in\mathcal{G}_{\beta},\varepsilon_{t}\in\{\pm 1\}\right\}.

Thanks to the particular form of $f$ , the set $\mathcal{G}_{\beta}$ is finite even though $\{W:|[W]_{11}|\not=|[W]_{22}|\}$ is infinite. Since there are finitely many choices for each $T$ , the set $\mathcal{H}_{\phi}$ is countable. Note that $\{W_{0}:q_{0}\in\mathcal{H}_{\phi}\}$ is a countable union of affine hyperplanes. Thus, its union with $\{p_{0}q_{0}=0\}$ is a zero-measure set. If $q_{0}\notin\mathcal{H}_{\phi}$ , $p_{0}q_{0}\neq 0$ , and $T_{0}<+\infty$ , Eq. (3.1) before $T_{0}$ gives $p_{t}=p_{0}\neq 0$ . Thus, all subgradients before $T_{0}$ belong to $\mathcal{G}_{\beta}$ and $q_{0}\in\mathcal{H}_{\phi}$ . This yields a contradiction. Thus, $T_{0}=+\infty$ and $p_{t}=p_{0}$ for every $t$ for almost every $W_{0}$ , which implies $f(W_{t})\geq A_{1}|p_{0}|>0=\inf_{W\in\mathbb{R}^{m\times n}}f(W)$ . This completes the proof. $\Box$

We remark that the failure of Muon in Theorem 3.1 is not due to unbounded iterates. Indeed, the proof has shown that the iterates $\{W_{t}\}_{t\geq 0}$ remain bounded whenever the step sizes $\{\eta_{t}\}_{t\geq 0}$ are bounded. Our second result further shows that this failure stems from fixed momentum rather than orthogonalization.

Theorem 3.2

Suppose that Assumption 2.2 holds, $\tau>0$ and $\{(W_{t},M_{t})\}_{t\geq 0}$ is generated by Eq. (2.2) with $\beta_{t}=1-\tau\eta_{t}$ , $\eta_{t}>0$ , $\eta_{t}\to 0$ , $\sum_{t=0}^{\infty}\eta_{t}=+\infty$ , $\delta_{t}\to 0$ , and $\sup_{t\geq 0}\sigma_{t}<+\infty$ . If $\sup_{t}\|W_{t}\|_{F}<+\infty$ , every cluster point of $\{W_{t}\}_{t\geq 0}$ is a $\mathcal{D}_{f}$ -stationary point, $M_{t}\to 0$ , and the sequence $\{f(W_{t})\}_{t\geq 0}$ converges. In addition, if Assumption 2.3 holds, $f(W_{t})\to\inf_{W}f(W)$ as $t\to+\infty$ .

We first define the $\varepsilon$ -graph enlargement of the mapping $D:\mathbb{R}^{d}\rightrightarrows\mathbb{R}^{d}$ as $D^{\varepsilon}(x):=\{v:\exists y\textnormal{ with }\|y-x\|_{2}\leq\varepsilon\textnormal{ and }\inf_{z\in D(y)}\|v-z\|_{2}\leq\varepsilon\}$ . Here, $D$ is locally bounded if for any $x\in\mathbb{R}^{d}$ there exists a neighborhood $V_{x}\subseteq\mathbb{R}^{d}$ such that $\cup_{y\in V_{x}}D(y)\subseteq\mathbb{R}^{d}$ is bounded. The typical examples of locally bounded mappings are conservative fields (Bolte and Pauwels, 2021, Remark 3). We then provide a general sufficient condition for the asymptotic convergence of updates that approximate a differential inclusion, which is the key to our subsequent analysis. For the sake of completeness, we provide the detailed proof.

Lemma 3.3

Let $D:\mathbb{R}^{d}\rightrightarrows\mathbb{R}^{d}$ be locally bounded, has a closed graph, and have nonempty convex values. We consider the update $x_{t+1}=x_{t}+\eta_{t}d_{t}$ where $d_{t}\in D^{\varepsilon_{t}}(x_{t})$ , $\eta_{t}>0$ , $\eta_{t}\to 0$ , $\sum_{t=0}^{\infty}\eta_{t}=\infty$ , $\varepsilon_{t}\geq 0$ , $\varepsilon_{t}\to 0$ and $\sup_{t\geq 0}\|x_{t}\|_{2}<+\infty$ . Let $B\subseteq\mathbb{R}^{d}$ , $\varphi:\mathbb{R}^{d}\to\mathbb{R}$ be continuous and $L$ be the set of limit points of the sequence $\{x_{t}\}_{t\geq 0}$ as defined by

L:=\bigcap_{N\geq 0}\overline{\{x_{t}:t\geq N\}}.

(3.3)

Suppose that $\varphi(B\cap L)$ has an empty interior and $\varphi$ is a Lyapunov function for $D$ (every absolutely continuous solution $\gamma:[0,+\infty)\to L$ of $\dot{\gamma}(s)\in D(\gamma(s))$ for almost every $s\geq 0$ satisfies $\varphi(\gamma(s))\leq\varphi(\gamma(0))$ for all $s\geq 0$ , with $<$ for every $s>0$ whenever $\gamma(0)\notin B$ ). Then, we have $L\subseteq B$ , $\varphi$ becomes a constant on $L$ , and $\{\varphi(x_{t})\}_{t\geq 0}$ converges. In addition, every cluster point of $\{x_{t}\}_{t\geq 0}$ lies in $B$ .

Proof. To apply the arguments from Benaïm et al. (2005), we construct the continuous-time version of our updates. Let $\lambda_{0}=0$ and $\lambda_{t}=\sum_{k=0}^{t-1}\eta_{k}$ for $t\geq 1$ . Since $\lambda_{t}\to+\infty$ , we define $w:[0,+\infty)\to\mathbb{R}^{d}$ by

w(\lambda_{t}+s)=x_{t}+s\tfrac{x_{t+1}-x_{t}}{\lambda_{t+1}-\lambda_{t}}=x_{t}+sd_{t},\quad\textnormal{for }s\in[0,\eta_{t})\textnormal{ and }t\geq 0.

Accordingly, we define $L_{w}:=\bigcap_{T\geq 0}\overline{\{w(s):s\geq T\}}$ . We claim that $L=L_{w}$ . Since $\|x_{t+1}-x_{t}\|_{2}\leq\eta_{t}\|d_{t}\|_{2}$ and $\eta_{t}\to 0$ , it suffices to show $\sup_{t\geq 0}\|d_{t}\|_{2}<+\infty$ . Indeed, by the definition of $\varepsilon$ -graph enlargement, there exist $\{(y_{t},v_{t})\}_{t\geq 0}$ such that $\|y_{t}-x_{t}\|_{2}\leq\varepsilon_{t}$ , $v_{t}\in D(y_{t})$ and $\|v_{t}-d_{t}\|_{2}\leq\varepsilon_{t}$ . Since $\{x_{t}\}_{t\geq 0}$ is uniformly bounded and $\varepsilon_{t}\to 0$ , $\{y_{t}\}_{t\geq 0}$ is uniformly bounded. It follows from the local boundedness of $D$ and $v_{t}\in D(y_{t})$ that $\{v_{t}\}_{t\geq 0}$ is uniformly bounded, which implies $\sup_{t\geq 0}\|d_{t}\|_{2}<+\infty$ .

The desired result holds if $L_{w}$ is internally chain transitive (ICT) for $D$ . Indeed, since $L=L_{w}$ and $L_{w}$ is ICT, Benaïm et al. (2005, Lemma 3.5 and Proposition 3.27) and an empty interior of $\varphi(B\cap L)$ gives $L\subseteq B$ and $\varphi|_{L}=c$ for some constant $c$ . If $\{\varphi(x_{t})\}_{t\geq 0}$ does not converge to $c$ , the boundedness of $\{x_{t}\}_{t\geq 0}$ guarantees a subsequence converging to some $\bar{x}\in L$ satisfying $\varphi(\bar{x})\neq c$ . This contradicts the continuity of $\varphi$ . Since every cluster point of $\{x_{t}\}$ belongs to $L$ , every cluster point lies in $B$ .

To apply (Benaïm et al., 2005, Theorem 3.6) to show that $L_{w}$ is ICT, we show that $w$ is a bounded perturbed solution of some differential inclusion $\dot{x}\in\widetilde{D}(x)$ where $\widetilde{D}:\mathbb{R}^{d}\rightrightarrows\mathbb{R}^{d}$ is globally bounded, convex-valued, has a closed graph, and coincides with $D$ on a neighborhood of $\overline{\{x_{t}:t\geq 0\}}$ . Indeed, we define $\widetilde{D}(x):=\{\chi(x)v:v\in D(x)\}$ , where $\chi:\mathbb{R}^{d}\to[0,1]$ is a continuous compactly supported function that equals one on a neighborhood of $\overline{\{x_{t},y_{t}:t\geq 0\}}$ . Since $D$ is locally bounded and $\chi$ is supported on a compact set, $\widetilde{D}$ is globally bounded. $\widetilde{D}$ is convex-valued since $D$ is convex-valued. Moreover, we show that $\widetilde{D}$ has a closed graph. If $x_{t}\to x$ and $\chi(x_{t})v_{t}\to z$ with $v_{t}\in D(x_{t})$ , the local boundedness of $D$ implies that $\{v_{t}\}_{t\geq 0}$ is bounded. Passing to a subsequence, $v_{t}\to v\in D(x)$ by the closed graph of $D$ , and thus $z=\chi(x)v\in\widetilde{D}(x)$ using the continuity of $\chi$ . We then show that $w(s)$ is a bounded perturbed solution of $\dot{x}\in\widetilde{D}(x)$ . Indeed, for $s\in(\lambda_{t},\lambda_{t+1})$ , we have $\dot{w}(s)=d_{t}$ and $\|w(s)-x_{t}\|_{2}\leq\eta_{t}\|d_{t}\|_{2}$ . This implies $\dot{w}(s)\in\widetilde{D}^{\varepsilon_{t}+\eta_{t}\|d_{t}\|_{2}}(w(s))$ where $\varepsilon_{t}+\eta_{t}\|d_{t}\|_{2}\to 0$ . In addition, $\{w(s)\}_{s\geq 0}$ is uniformly bounded since $\{x_{t}\}_{t\geq 0}$ is uniformly bounded. $\Box$

Proof of Theorem 3.2. By choosing $P_{t+1}\in\operatorname{polar}_{\delta_{t}}(M_{t+1}/(\|M_{t+1}\|_{F}+\sigma_{t}))$ , we can rewrite Eq. (2.2) as $(W_{t+1},M_{t+1})=(W_{t},M_{t})+\eta_{t}(-P_{t+1},\tau(G_{t}-M_{t}))$ . Letting $Z_{t}=(W_{t},M_{t})$ and $D_{t}=(-P_{t+1},\tau(G_{t}-M_{t}))$ , we have $Z_{t+1}=Z_{t}+\eta_{t}D_{t}$ . Thus, we define the mapping $\mathcal{F}:\mathbb{R}^{m\times n}\times\mathbb{R}^{m\times n}\rightrightarrows\mathbb{R}^{m\times n}\times\mathbb{R}^{m\times n}$ as $\mathcal{F}(W,M):=\{(-P,\tau(G-M)):G\in\mathcal{D}_{f}(W),P\in\partial(\|\cdot\|_{\textnormal{nuc}})(M)\}$ , where $\|\cdot\|_{\textnormal{nuc}}$ is the nuclear norm as a function $\mathbb{R}^{m\times n}\to\mathbb{R}$ . Then, it suffices to show that the vectorized version of $Z_{t}$ , $D_{t}$ , and $\mathcal{F}$ satisfy the conditions of Lemma 3.3.

We first show that $\{Z_{t}\}_{t\geq 0}$ is bounded. Indeed, $\{W_{t}\}_{t\geq 0}$ is bounded. Since $\mathcal{D}_{f}$ is locally bounded, $\{G_{t}\}_{t\geq 0}$ is bounded. In addition, $M_{t+1}=(1-\tau\eta_{t})M_{t}+\tau\eta_{t}G_{t}$ and $0\leq\tau\eta_{t}\leq 1$ for all large $t$ . Thus, $\{M_{t}\}$ is bounded. Putting these pieces together yields the desired result.

We obtain that $\mathcal{F}$ satisfies the properties from Lemma 3.3 since $\partial\operatorname{\|\cdot\|}$ and $\mathcal{D}_{f}$ are locally bounded, have closed graphs, and have nonempty convex values. To apply Lemma 3.3, we have to prove that (i) $D_{t}\in\mathcal{F}^{\varepsilon_{t}}(Z_{t})$ with $\varepsilon_{t}\to 0$ and (ii) there exists a function $\varphi(Z)$ satisfying the desired properties.

We show (i). Indeed, we construct $\widehat{Z}_{t}=(W_{t},\widehat{M}_{t+1})$ and $\widehat{D}_{t}=(-\widehat{P}_{t+1},\tau(G_{t}-\widehat{M}_{t+1}))$ so that $\widehat{D}_{t}\in\mathcal{F}(\widehat{Z}_{t})$ , $\|\widehat{Z}_{t}-Z_{t}\|_{2}\leq\varepsilon_{t}$ , and $\|\widehat{D}_{t}-D_{t}\|_{2}\leq\varepsilon_{t}$ with $\varepsilon_{t}\to 0$ . If $M_{t+1}=0$ , we set $\widehat{M}_{t+1}=\widehat{P}_{t+1}=0$ . Otherwise, we let $k_{t+1}:=\operatorname{rank}(M_{t+1})$ , $M_{t+1}=U\operatorname{Diag}(s_{1},\ldots,s_{k_{t+1}})V^{\top}$ and $P_{t+1}=U\operatorname{Diag}(\phi_{1},\ldots,\phi_{k_{t+1}})V^{\top}$ using SVD that outputs $P_{t+1}$ in Eq. (2.1). We define

I_{t}:=\{1\leq j\leq k_{t+1}:s_{j}\geq\delta_{t}(\|M_{t+1}\|_{F}+\sigma_{t})\}

and

\begin{array}[]{ll}\widehat{M}_{t+1}:=U\operatorname{Diag}(\widehat{s}_{1},\ldots,\widehat{s}_{k_{t+1}})V^{\top},&\quad\widehat{s}_{j}=\begin{cases}s_{j},&j\in I_{t},\\ 0,&j\notin I_{t}.\end{cases}\\ \widehat{P}_{t+1}:=U\operatorname{Diag}(\widehat{\phi}_{1},\ldots,\widehat{\phi}_{k_{t+1}})V^{\top},&\quad\widehat{\phi}_{j}=\begin{cases}1,&j\in I_{t},\\ \min\{\phi_{j},1\},&j\notin I_{t}.\end{cases}\end{array}

which implies $\|\widehat{M}_{t+1}-M_{t+1}\|_{\textnormal{op}}\leq\delta_{t}(\|M_{t+1}\|_{F}+\sigma_{t})$ and $\|\widehat{P}_{t+1}-P_{t+1}\|_{\textnormal{op}}\leq\delta_{t}$ . Here, we claim that $\widehat{P}_{t+1}\in\partial(\|\cdot\|_{\textnormal{nuc}})(\widehat{M}_{t+1})$ . Indeed, since $\|\widehat{P}_{t+1}\|_{\textnormal{op}}\leq 1$ and $\langle\widehat{P}_{t+1},\widehat{M}_{t+1}\rangle=\|\widehat{M}_{t+1}\|_{\textnormal{nuc}}$ , we have $\|Y\|_{\textnormal{nuc}}\geq\langle\widehat{P}_{t+1},Y\rangle=\|\widehat{M}_{t+1}\|_{\textnormal{nuc}}+\langle\widehat{P}_{t+1},Y-\widehat{M}_{t+1}\rangle$ for all $Y\in\mathbb{R}^{m\times n}$ . Therefore, we have $D_{t}\in\mathcal{F}^{\varepsilon_{t}}(Z_{t})$ , where $\varepsilon_{t}=\max\{\|\widehat{M}_{t+1}-M_{t+1}\|_{F}+\|M_{t+1}-M_{t}\|_{F},\|\widehat{P}_{t+1}-P_{t+1}\|_{F}+\tau(\|\widehat{M}_{t+1}-M_{t+1}\|_{F}+\|M_{t+1}-M_{t}\|_{F})\}$ . Since $\{G_{t}\}_{t\geq 0}$ and $\{M_{t}\}_{t\geq 0}$ are bounded, $\|M_{t+1}-M_{t}\|_{F}=\tau\eta_{t}\|G_{t}-M_{t}\|_{F}\to 0$ . Since $\delta_{t}\to 0$ and $\sup_{t\geq 0}\sigma_{t}<+\infty$ , we have $\|\widehat{M}_{t+1}-{M}_{t+1}\|_{F}\to 0,\|\widehat{P}_{t+1}-P_{t+1}\|_{F}\to 0$ , which further implies $\varepsilon_{t}\to 0$ . Putting these pieces together yields the desired result.

We show (ii). Indeed, we define the function $\varphi(Z):=f(W)+\tau^{-1}\|M\|_{\textnormal{nuc}}$ and show that it satisfies the desired properties. We set $L$ as the limit set of $\{Z_{t}\}_{t\geq 0}$ in Eq. (3.3) and $B:=\{Z:0\in\mathcal{D}_{f}(W),M=0\}$ . By Assumption 2.2, $\varphi(B\cap L)\subseteq\{f(W):0\in\mathcal{D}_{f}(W)\}$ has the empty interior. Let $Z(s)=(W(s),M(s))$ be a solution of $\dot{Z}\in\mathcal{F}(Z)$ whose range lies in $L$ . For $s$ almost everywhere, the matrices $P(s):=-\dot{W}(s)$ and $G(s):=M(s)+\tau^{-1}\dot{M}(s)$ satisfy $P(s)\in\partial(\|\cdot\|_{\textnormal{nuc}})(M(s))$ and $G(s)\in\mathcal{D}_{f}(W(s))$ . By using the conservative field chain rule for $f$ and the chain rule for the nuclear norm, we have

\tfrac{d}{ds}\varphi(W(s),M(s))=\langle G(s),\dot{W}(s)\rangle+\tau^{-1}\langle P(s),\dot{M}(s)\rangle=-\|M(s)\|_{\textnormal{nuc}}.

Thus, $\varphi$ is nonincreasing along $Z(s)$ . The decrease is strict if $Z(0)\notin B$ . Indeed, if $\varphi(Z(T))=\varphi(Z(0))$ for some $T>0$ , $\varphi$ is constant on $[0,T]$ and $M(s)=0$ for almost everywhere $s\in[0,T]$ . The continuity of $M$ gives $M(s)=0$ for all $s\in[0,T]$ . Thus, $\dot{M}=\tau(G-M)$ gives $G(s)=0$ for almost everywhere $s\in[0,T]$ . By using the closed graph of $\mathcal{D}_{f}$ , we obtain that $0\in\mathcal{D}_{f}(W(0))$ , which implies $Z(0)\in B$ . Putting these pieces together yields the desired result.

By Lemma 3.3, $L\subseteq B$ and $\{\varphi(Z_{t})\}_{t\geq 0}$ converges, which implies that every cluster point of $\{W_{t}\}_{t\geq 0}$ is $\mathcal{D}_{f}$ -stationary and every cluster point of $\{M_{t}\}_{t\geq 0}$ is $0$ . As such, $f(W_{t})=\varphi(Z_{t})-\tau^{-1}\|M_{t}\|_{\textnormal{nuc}}$ converges. In addition, if Assumption 2.3 holds, any cluster point is globally optimal. Since $f(W_{t})$ converges and $f$ is continuous, it follows that $\lim_{t\to\infty}f(W_{t})=\inf_{W\in\mathbb{R}^{m\times n}}f(W)$ . This completes the proof. $\Box$

3.2 Muon advised by gradient descent

Muon with fixed momentum can fail on convex Lipschitz objectives, whereas Muon with an appropriately coupled momentum schedule can converge asymptotically. Whether the latter admits a finite-time complexity guarantee remains unclear. Error feedback with momentum (Parshakova et al., 2026) modifies Muon to obtain a finite-time rate. It tracks the discrepancy introduced by orthogonalization and feeds this error into the input of the orthogonalization at the next iteration. However, convergence of this mechanism has not been established for general nonsmooth, nonconvex objectives, and experiments suggest that it can perform worse than unmodified Muon (Parshakova et al., 2026, Section 5).

We propose Muon Advised by Gradient Descent (MAGD) as an alternative modification of Muon for nonsmooth objectives. Inspired by prediction with expert advice (Cesa-Bianchi and Lugosi, 2006), MAGD evaluates a gradient at $W_{t}$ and maintains two branches: $U_{t}$ takes the Muon update, while $S_{t}$ takes the gradient descent update. MAGD forms each iterate by mixing the two branches and adapts the mixing factor by comparing their decreases under the linearized loss. This rule assigns more weight to the branch with the larger predicted decrease, and hence MAGD remains close to Muon whenever the Muon branch performs better.

Algorithm 1 Muon Advised by Gradient Descent (MAGD)

1: Input:

W_{0}\in\mathbb{R}^{m\times n}

\eta_{t}\geq 0

\gamma>0

\rho\in(0,1)

\beta_{t}^{\rm M}\in[0,1]

\eta_{t}^{\rm M}\geq 0

\delta_{t}^{\rm M}\in[0,1)

\sigma_{t}^{\rm M}\geq 0

2: Initialization:

U_{0}=S_{0}=W_{0}

M_{0}=0

z_{0}=\log(\rho/(1-\rho))

3: for

t=0,1,\ldots

\alpha_{t}=(1+\exp(-z_{t}))^{-1}

W_{t}=(1-\alpha_{t})U_{t}+\alpha_{t}S_{t}

G_{t}\in\mathcal{D}_{f}(W_{t})

z_{t+1}=z_{t}-\gamma\eta_{t}\langle G_{t},S_{t}-U_{t}\rangle

S_{t+1}=S_{t}-\eta_{t}G_{t}

M_{t+1}=\beta_{t}^{\rm M}M_{t}+(1-\beta_{t}^{\rm M})G_{t}

10:

U_{t+1}\in U_{t}-\eta_{t}^{\rm M}\operatorname{polar}_{\delta_{t}^{\rm M}}\left(\frac{M_{t+1}}{\|M_{t+1}\|_{F}+\sigma_{t}^{\rm M}}\right)

, with

\frac{0}{0}:=0

We next establish the asymptotic convergence of MAGD. The theorem covers two regimes. In the first regime, the GD branch asymptotically dominates the Muon branch, while the Muon branch may use a fixed or arbitrary momentum schedule. In the second regime, neither branch needs to dominate, but the momentum factor of the Muon branch approaches one as in Theorem 3.2.

Theorem 3.4

Suppose that Assumption 2.2 holds. Run Algorithm 1 with $\gamma>0$ , and suppose that a closed ball $K\subseteq\mathbb{R}^{m\times n}$ contains $\{U_{t}\}_{t\geq 0}$ and $\{S_{t}\}_{t\geq 0}$ . Set $L_{K}:=\sup\{\|G\|_{F}:W\in K,G\in\mathcal{D}_{f}(W)\}$ and $R:=\sup_{t\geq 0}\|S_{t}-U_{t}\|_{F}$ . Assume that one of the following regimes holds.

(i)

We have $\eta_{t}>0$ , $\eta_{t}\to 0$ , $\sum_{t}\eta_{t}=+\infty$ , and $(1-\alpha_{t})\eta_{t}^{\rm M}/\eta_{t}\to 0$ .
(ii)

For some $\tau>0$ , we have $\delta_{t}^{\rm M}\to 0$ , $\delta_{t}^{\rm M}\sigma_{t}^{\rm M}\to 0$ , $\beta_{t}^{\rm M}=1-\tau\eta_{t}^{\rm M}\in[0,1]$ , $\eta_{t}^{\rm M}>0$ , $\eta_{t}^{\rm M}\to 0$ , $\sum_{t}\eta_{t}^{\rm M}=+\infty$ , and $\eta_{t}\geq 0$ with $\eta_{t}/\eta_{t}^{\rm M}\to r\in[0,+\infty)$ . In addition, $\gamma rL_{K}R\leq\tau$ . If $r=0$ , we also assume that $\limsup_{t\to\infty}\alpha_{t}<1$ .

Then, every cluster point of $\{W_{t}\}$ lies in $\{W:0\in\mathcal{D}_{f}(W)\}$ , and $\{f(W_{t})\}$ converges. In regime (ii), we additionally have $M_{t}\to 0$ .

We prove two regimes separately. Let $D_{t}:=S_{t}-U_{t}$ , $E_{t}:=S_{t}-W_{t}$ , and $P_{t+1}\in\operatorname{polar}_{\delta_{t}^{\rm M}}\left(\frac{M_{t+1}}{\|M_{t+1}\|_{F}+\sigma_{t}^{\rm M}}\right)$ such that $U_{t+1}=U_{t}-\eta_{t}^{\rm M}P_{t+1}$ . Since $K$ is convex, $W_{t}\in K$ . Local boundedness of $\mathcal{D}_{f}$ gives $L_{K}<+\infty$ . Hence, $\sup_{t\geq 0}\|D_{t}\|_{F}\leq R$ , $\sup_{t\geq 0}\|G_{t}\|_{F}\leq L_{K}$ , and $\{E_{t}\}_{t\geq 0}$ is uniformly bounded. Eq. (2.1) gives $\|P_{t+1}\|_{\textnormal{op}}<2$ , and thus $\|P_{t+1}\|_{F}<2\sqrt{\min\{m,n\}}$ . In addition,

$\displaystyle D_{t+1}-D_{t}$	$\displaystyle=\eta_{t}^{\rm M}P_{t+1}-\eta_{t}G_{t},$	(3.4)
$\displaystyle W_{t+1}-W_{t}$	$\displaystyle=(\alpha_{t+1}-\alpha_{t})D_{t}-(1-\alpha_{t+1})\eta_{t}^{\rm M}P_{t+1}-\alpha_{t+1}\eta_{t}G_{t},$
$\displaystyle E_{t+1}-E_{t}$	$\displaystyle=-(\alpha_{t+1}-\alpha_{t})D_{t}+(1-\alpha_{t+1})(\eta_{t}^{\rm M}P_{t+1}-\eta_{t}G_{t}).$

In both regimes, $\eta_{t}\to 0$ and $z_{t+1}-z_{t}=-\gamma\eta_{t}\langle G_{t},D_{t}\rangle=O(\eta_{t})$ . The sigmoid function has a globally bounded second derivative, and thus we have

\alpha_{t+1}-\alpha_{t}=-\gamma\eta_{t}\alpha_{t}(1-\alpha_{t})\langle G_{t},D_{t}\rangle+O(\eta_{t}^{2}).

(3.5)

Proof of regime (i). We smooth the update of $\alpha_{t}$ . Let $\Pi_{R}$ denote the Euclidean projection onto the Frobenius ball of radius $R$ centered at origin. For $E,G\in\mathbb{R}^{m\times n}$ and $\alpha\in\mathbb{R}$ , we define

\mathcal{H}_{R}(E,\alpha,G):=\begin{cases}(1-\alpha)\left\langle G,\Pi_{R}\left(E/(1-\alpha)\right)\right\rangle\Pi_{R}\left(E/(1-\alpha)\right),&\alpha\neq 1,\\ 0,&\alpha=1.\end{cases}

Since $E_{t}=(1-\alpha_{t})D_{t}$ and $\|D_{t}\|_{F}\leq R$ , $\mathcal{H}_{R}(E_{t},\alpha_{t},G_{t})=(1-\alpha_{t})\langle G_{t},D_{t}\rangle D_{t}$ . For $X=(W,E,\alpha)$ , let

\mathcal{F}_{\textnormal{S}}(X):=\bigl\{(-\alpha G-\gamma\alpha\mathcal{H}_{R}(E,\alpha,G),-(1-\alpha)G+\gamma\alpha\mathcal{H}_{R}(E,\alpha,G),-\gamma\alpha\langle G,E\rangle):G\in\mathcal{D}_{f}(W)\bigr\}.

(3.6)

We obtain that $\mathcal{F}_{\textnormal{S}}$ satisfies the properties from Lemma 3.3 since $\mathcal{H}_{R}$ is continuous in $(E,\alpha,G)$ and linear in $G$ and $\mathcal{D}_{f}$ is locally bounded, has a closed graph, and has nonempty convex values. Let $X_{t}=(W_{t},E_{t},\alpha_{t})$ and $Y_{t}=\frac{X_{t+1}-X_{t}}{\eta_{t}}$ . To apply Lemma 3.3, we have to prove that (a) $Y_{t}\in\mathcal{F}_{\textnormal{S}}^{\varepsilon_{t}}(X_{t})$ with $\varepsilon_{t}\to 0$ and (b) there exists a function $\varphi_{\textnormal{S}}(X)$ satisfying the desired properties.

We show (a). Let $V_{t}\in\mathcal{F}_{\textnormal{S}}(X_{t})$ where $G=G_{t}\in\mathcal{D}_{f}(W_{t})$ . We set $\xi_{t}:=\frac{\alpha_{t+1}-\alpha_{t}}{\eta_{t}}+\gamma\alpha_{t}\langle G_{t},E_{t}\rangle$ . It follows from Eq. (3.5) that $\xi_{t}\to 0$ . In addition,

\displaystyle Y_{t}-V_{t}=\bigl(\xi_{t}D_{t}-(1-\alpha_{t+1})\tfrac{\eta_{t}^{\rm M}}{\eta_{t}}P_{t+1}-(\alpha_{t+1}-\alpha_{t})G_{t},-\xi_{t}D_{t}+(1-\alpha_{t+1})\tfrac{\eta_{t}^{\rm M}}{\eta_{t}}P_{t+1}+(\alpha_{t+1}-\alpha_{t})G_{t},\xi_{t}\bigr).

By $e^{-|z_{t+1}-z_{t}|}\leq\tfrac{1-\alpha_{t+1}}{1-\alpha_{t}}=\tfrac{1+e^{z_{t}}}{1+e^{z_{t}+z_{t+1}-z_{t}}}\leq e^{|z_{t+1}-z_{t}|}$ and $|z_{t+1}-z_{t}|=O(\eta_{t})$ , we have $\varepsilon_{t}:=\|Y_{t}-V_{t}\|_{2}\to 0$ .

We show (b). Indeed, we define the continuous function $\varphi_{\textnormal{S}}(X)=f(W)$ and show that it satisfies the desired properties. Let $L_{\textnormal{S}}$ be the limit set of $\{X_{t}\}_{t\geq 0}$ defined in Eq. (3.3) and $B_{\textnormal{S}}:=\{(W,E,\alpha):0\in\mathcal{D}_{f}(W)\}$ . Since $\varphi_{\textnormal{S}}(B_{\textnormal{S}}\cap L_{\textnormal{S}})\subseteq f(\{W:0\in\mathcal{D}_{f}(W)\})$ , $\varphi_{\textnormal{S}}(B_{\textnormal{S}}\cap L_{\textnormal{S}})$ has empty interior. Let $X(s)=(W(s),E(s),\alpha(s))$ be a solution of $\dot{X}\in\mathcal{F}_{\textnormal{S}}(X)$ whose range lies in $L_{\textnormal{S}}$ . For $s$ almost everywhere, the matrix $G(s):=-(\dot{W}(s)+\dot{E}(s))\in\mathcal{D}_{f}(W(s))$ . The conservative field chain rule and Eq. (3.6) give

\tfrac{\mathrm{d}}{\mathrm{d}s}\varphi_{\textnormal{S}}(X(s))=-\alpha\|G\|_{F}^{2}-\gamma\alpha\langle G,\mathcal{H}_{R}(E,\alpha,G)\rangle\leq 0,

(3.7)

Thus, $\varphi_{\textnormal{S}}$ is nonincreasing along $X(s)$ . To show strict decrease outside $B_{\textnormal{S}}$ , solving $\dot{\alpha}=-\gamma\alpha\langle G,E\rangle$ gives

\alpha(s)=\alpha(0)\exp\left(-\gamma\int_{0}^{s}\langle G(u),E(u)\rangle\,\mathrm{d}u\right).

(3.8)

Thus, $\alpha(0)>0$ implies $\alpha(s)>0$ for every finite $s\geq 0$ . If $\varphi_{\textnormal{S}}(X(T))=\varphi_{\textnormal{S}}(X(0))$ for some $T>0$ , Eq. (3.7) implies that $G(s)=0$ for almost every $s\in[0,T]$ . The closed graph of $\mathcal{D}_{f}$ gives $0\in\mathcal{D}_{f}(W(0))$ . It remains to consider $\alpha(0)=0$ . Eq. (3.8) gives $\alpha(s)=0$ and $W(s)=W(0)$ for all $s\geq 0$ , and hence $\dot{E}(s)=-G(s)$ for $s$ almost everywhere. Suppose that $0\notin\mathcal{D}_{f}(W(0))$ . $\mathcal{D}_{f}(W(0))$ is nonempty, compact, and convex. Let $G_{\star}$ be the Euclidean projection of $0$ onto $\mathcal{D}_{f}(W(0))$ . Then, $G_{\star}\neq 0$ and $\langle G_{\star},G\rangle\geq\|G_{\star}\|_{F}^{2}$ for every $G\in\mathcal{D}_{f}(W(0))$ . Consequently, we have

\langle G_{\star},E(s)\rangle\leq\langle G_{\star},E(0)\rangle-s\|G_{\star}\|_{F}^{2}.

This contradicts the boundedness of $E(s)$ . Hence, no solution with range in $L_{\textnormal{S}}$ starts in the set $\{(W,E,0)\in L_{\textnormal{S}}:0\notin\mathcal{D}_{f}(W)\}$ . Therefore, every solution with range in $L_{\textnormal{S}}$ that starts outside $B_{\textnormal{S}}$ has $\alpha(0)>0$ , and $\varphi_{\textnormal{S}}(X(s))<\varphi_{\textnormal{S}}(X(0))$ for every $s>0$ . This proves the required properties of $\varphi_{\textnormal{S}}$ .

By Lemma 3.3, $\{\varphi_{\textnormal{S}}(X_{t})\}_{t\geq 0}=\{f(W_{t})\}_{t\geq 0}$ converges. In addition, if $W_{t_{j}}\to\overline{W}$ , there exists $(W_{t_{j_{k}}},E_{t_{j_{k}}},\alpha_{t_{j_{k}}})_{k\geq 0}$ that converges to some point in $L_{\textnormal{S}}\subseteq B_{\textnormal{S}}$ . Hence, we have $0\in\mathcal{D}_{f}(\overline{W})$ . $\Box$

Proof of regime (ii). We set $r_{t}:=\eta_{t}/\eta_{t}^{\rm M}$ , $r:=\lim_{t\to\infty}r_{t}$ , and $b_{t}:=(\alpha_{t+1}-\alpha_{t})/\eta_{t}^{\rm M}$ . For $X_{t}:=(W_{t},D_{t},M_{t},\alpha_{t})$ , Eq. (3.4) gives $X_{t+1}=X_{t}+\eta_{t}^{\rm M}Y_{t}$ , where

Y_{t}:=\bigl(b_{t}D_{t}-(1-\alpha_{t+1})P_{t+1}-r_{t}\alpha_{t+1}G_{t},P_{t+1}-r_{t}G_{t},\tau(G_{t}-M_{t}),b_{t}\bigr).

Since $M_{t+1}=(1-\tau\eta_{t}^{\rm M})M_{t}+\tau\eta_{t}^{\rm M}G_{t}$ and $0\leq\tau\eta_{t}^{\rm M}\leq 1$ , $\{M_{t}\}_{t\geq 0}$ and hence $\{X_{t}\}_{t\geq 0}$ are uniformly bounded. For $X=(W,D,M,\alpha)$ , we define

\begin{split}\mathcal{F}_{\rm M}(X):=\bigl\{(&-(1-\alpha)P-r\alpha G-\gamma r\alpha(1-\alpha)\langle G,D\rangle D,P-rG,\\ &\tau(G-M),-\gamma r\alpha(1-\alpha)\langle G,D\rangle):G\in\mathcal{D}_{f}(W),P\in\partial(\|\cdot\|_{\textnormal{nuc}})(M)\bigr\}.\end{split}

We obtain that $\mathcal{F}_{\textnormal{M}}$ satisfies the properties of Lemma 3.3 since $\partial(\|\cdot\|_{\textnormal{nuc}})$ and $\mathcal{D}_{f}$ are locally bounded, have closed graphs and nonempty and convex values. To apply Lemma 3.3, we have to prove that (a) $Y_{t}\in\mathcal{F}_{\textnormal{M}}^{\varepsilon_{t}}(X_{t})$ with $\varepsilon_{t}\to 0$ and (b) there exists a function $\varphi_{\textnormal{M}}(X)$ satisfying the desired properties.

We show (a). We apply the singular value truncation used in the proof of Theorem 3.2. It gives $\widehat{P}_{t+1}\in\partial(\|\cdot\|_{\textnormal{nuc}})(\widehat{M}_{t+1})$ such that $\|\widehat{M}_{t+1}-M_{t+1}\|_{\textnormal{op}}\leq\delta_{t}^{\rm M}(\|M_{t+1}\|_{F}+\sigma_{t}^{\rm M})$ and $\|\widehat{P}_{t+1}-P_{t+1}\|_{\textnormal{op}}\leq\delta_{t}^{\rm M}$ . Since $\delta_{t}^{\rm M}\to 0$ and $\delta_{t}^{\rm M}\sigma_{t}^{\rm M}\to 0$ , we have $\|\widehat{M}_{t+1}-M_{t+1}\|_{F}\to 0$ , $\|\widehat{P}_{t+1}-P_{t+1}\|_{F}\to 0$ , and $\|M_{t+1}-M_{t}\|_{F}=\tau\eta_{t}^{\rm M}\|G_{t}-M_{t}\|_{F}\to 0$ . In addition, Eq. (3.5) gives $\alpha_{t+1}-\alpha_{t}=O(\eta_{t})\to 0$ . We set $\widehat{X}_{t}:=(W_{t},D_{t},\widehat{M}_{t+1},\alpha_{t})$ and let $\widehat{Y}_{t}\in\mathcal{F}_{\rm M}(\widehat{X}_{t})$ be realized by $(G_{t},\widehat{P}_{t+1})$ . Let $e_{t}:=b_{t}+\gamma r\alpha_{t}(1-\alpha_{t})\langle G_{t},D_{t}\rangle$ . Dividing Eq. (3.5) by $\eta_{t}^{\rm M}$ , we obtain $b_{t}=-\gamma r_{t}\alpha_{t}(1-\alpha_{t})\langle G_{t},D_{t}\rangle+o(1)$ and hence $e_{t}\to 0$ . Denoting the four components of $Y_{t}$ and $\widehat{Y}_{t}$ by the subscripts $W,D,M,\alpha$ , direct subtraction gives

	$\displaystyle Y_{t,W}-\widehat{Y}_{t,W}={}$	$\displaystyle e_{t}D_{t}+(1-\alpha_{t})(\widehat{P}_{t+1}-P_{t+1})+(\alpha_{t+1}-\alpha_{t})P_{t+1}+\bigl((r-r_{t})\alpha_{t}-r_{t}(\alpha_{t+1}-\alpha_{t})\bigr)G_{t},$
	$\displaystyle Y_{t,D}-\widehat{Y}_{t,D}={}$	$\displaystyle P_{t+1}-\widehat{P}_{t+1}+(r-r_{t})G_{t},$
	$\displaystyle Y_{t,M}-\widehat{Y}_{t,M}={}$	$\displaystyle\tau(\widehat{M}_{t+1}-M_{t}),$
	$\displaystyle Y_{t,\alpha}-\widehat{Y}_{t,\alpha}={}$	$\displaystyle e_{t}.$

Every term on the right tends to zero. In addition, since $\|\widehat{M}_{t+1}-M_{t+1}\|_{F}\to 0$ and $\|{M}_{t+1}-M_{t}\|_{F}\to 0$ , we have $\|\widehat{X}_{t}-X_{t}\|\to 0$ . After vectorizing the matrices, we set $\varepsilon_{t}:=\|\widehat{X}_{t}-X_{t}\|_{2}+\|\widehat{Y}_{t}-Y_{t}\|_{2}\to 0$ , and thus $Y_{t}\in\mathcal{F}_{\rm M}^{\varepsilon_{t}}(X_{t})$ .

We show (b). If $r>0$ , we set $\zeta:=0$ . If $r=0$ , the assumption of regime (ii) lets us choose $\zeta\in(0,1)$ such that $\alpha_{t}\leq 1-\zeta$ eventually. Let $L$ be the limit set of $\{X_{t}\}_{t\geq 0}$ defined in Lemma 3.3. Since $K$ is closed, every $X\in L$ satisfies $W\in K$ , $\|D\|_{F}\leq R$ , and $0\leq\alpha\leq 1-\zeta$ . We set

B:=\{X:M=0,0\in\mathcal{D}_{f}(W)\},\quad\varphi_{\textnormal{M}}(X):=f(W)+\tfrac{1-\alpha}{\tau}\|M\|_{\textnormal{nuc}}+\tfrac{r\alpha}{2\tau}\|M\|_{F}^{2}.

We show that $\varphi_{\textnormal{M}}$ satisfies the desired properties. Since $\varphi_{\textnormal{M}}(B\cap L)\subseteq f(\{W:0\in\mathcal{D}_{f}(W)\})$ , $\varphi_{\textnormal{M}}(B\cap L)$ has empty interior. In addition, $\varphi_{\textnormal{M}}$ is continuous. Let $X(s)=(W(s),D(s),M(s),\alpha(s))$ be a solution of $\dot{X}\in\mathcal{F}_{\rm M}(X)$ whose range lies in $L$ . For $s$ almost everywhere, the matrices $G:=M+\tau^{-1}\dot{M}$ and $P:=\dot{D}+rG$ belong to $\mathcal{D}_{f}(W)$ and $\partial(\|\cdot\|_{\textnormal{nuc}})(M)$ , respectively, and realize the inclusion. Then

\dot{\alpha}=-\gamma r\alpha(1-\alpha)\langle G,D\rangle,\quad\dot{W}=-(1-\alpha)P-r\alpha G-\gamma r\alpha(1-\alpha)\langle G,D\rangle D,\quad\dot{M}=\tau(G-M).

Using the conservative field chain rule for $f$ and the chain rule for the nuclear norm, we obtain

	$\displaystyle\tfrac{d}{ds}\varphi_{\textnormal{M}}(X(s))={}$	$\displaystyle-(1-\alpha)\\|M\\|_{\textnormal{nuc}}-r\alpha\\|G\\|_{F}^{2}+r\alpha\langle M,G\rangle-r\alpha\\|M\\|_{F}^{2}$
		$\displaystyle-\gamma r\alpha(1-\alpha)\langle G,D\rangle^{2}+\tfrac{\gamma r\alpha(1-\alpha)}{\tau}\langle G,D\rangle\\|M\\|_{\textnormal{nuc}}-\tfrac{\gamma r^{2}\alpha(1-\alpha)}{2\tau}\langle G,D\rangle\\|M\\|_{F}^{2}.$

Letting $\theta:=\gamma rL_{K}R/\tau\in[0,1]$ , we obtain

\displaystyle\tfrac{d}{ds}\varphi_{\textnormal{M}}(X(s))

\displaystyle\leq-(1-\alpha)^{2}\|M\|_{\textnormal{nuc}}-\tfrac{r\alpha}{2}\|G\|_{F}^{2}-\tfrac{r\alpha^{2}}{2}\|M\|_{F}^{2}\leq 0.

(3.9)

Thus, $\varphi_{\textnormal{M}}$ is nonincreasing along $X(s)$ .

The decrease is strict outside $B$ . Indeed, if $\varphi_{\textnormal{M}}(X(T))=\varphi_{\textnormal{M}}(X(0))$ for some $T>0$ , then $\varphi_{\textnormal{M}}$ is constant on $[0,T]$ . If $r=0$ , we have $1-\alpha\geq\zeta>0$ , and thus the first term in Eq. (3.9) gives $M(s)=0$ for almost every $s\in[0,T]$ . We then consider the case where $r>0$ . If $\alpha(s)<1$ , the first term gives $M(s)=0$ . If $\alpha(s)=1$ , the last term gives $M(s)=0$ . Hence, $M(s)=0$ for almost every $s\in[0,T]$ in either case. By continuity, $M(s)=0$ throughout $[0,T]$ , and $\dot{M}=\tau(G-M)$ then gives $G(s)=0$ for almost every $s\in[0,T]$ . The closed graph of $\mathcal{D}_{f}$ gives $0\in\mathcal{D}_{f}(W(0))$ . Thus, $X(0)\in B$ .

By Lemma 3.3, $L\subseteq B$ and $\{\varphi_{\textnormal{M}}(X_{t})\}_{t\geq 0}$ converges. Letting $\overline{W}$ be a cluster point of $\{W_{t}\}_{t\geq 0}$ , we choose a subsequence $W_{t_{j}}\to\overline{W}$ . Boundedness of $\{(D_{t_{j}},M_{t_{j}},\alpha_{t_{j}})\}$ gives indices $j_{k}\to\infty$ such that $X_{t_{j_{k}}}$ converges to a point of $L\subseteq B$ . Hence $0\in\mathcal{D}_{f}(\overline{W})$ . As such, every cluster point of the bounded sequence $\{M_{t}\}_{t\geq 0}$ is $0$ . Thus, $\varphi_{\textnormal{M}}(X_{t})-f(W_{t})=\tfrac{1-\alpha_{t}}{\tau}\|M_{t}\|_{\textnormal{nuc}}+\tfrac{r\alpha_{t}}{2\tau}\|M_{t}\|_{F}^{2}\to 0$ , completing the proof. $\Box$

Besides asymptotic convergence, MAGD also admits a finite-time convergence rate for convex Lipschitz objectives under Assumption 2.3.

Theorem 3.5

Suppose that Assumption 2.3 holds and let $W^{\star}\in\mathbb{R}^{m\times n}$ satisfy $\|W_{0}-W^{\star}\|_{\textnormal{op}}\leq R$ . Assume that there exists $G>0$ such that $|f(X)-f(Y)|\leq G\|X-Y\|_{\textnormal{op}}$ for any $X,Y\in\mathbb{R}^{m\times n}$ satisfying $\|X-W^{\star}\|_{\textnormal{op}}\leq R$ and $\|Y-W^{\star}\|_{\textnormal{op}}\leq R$ . If we run Algorithm 1 with

\bar{\eta}_{t}>0,\quad\eta_{t}=\begin{cases}\bar{\eta}_{t}/\|G_{t}\|_{F}&G_{t}\neq 0,\\ 0&G_{t}=0,\end{cases}\quad\text{and}\quad 0\leq\eta_{t}^{\rm M}\leq\tfrac{\bar{\eta}_{t}}{2\sqrt{\min\{m,n\}}},

then, for every $t\geq 1$ ,

\min_{0\leq s<t}f(W_{s})-f(W^{\star})\leq\tfrac{G}{2\sum_{s=0}^{t-1}\bar{\eta}_{s}}\left[{\min\{m,n\}R^{2}}+\tfrac{2}{\gamma}\log(1/\rho)+\sum_{s<t}\bar{\eta}_{s}^{2}+\gamma\sum_{s<t}\bar{\eta}_{s}^{2}\left(\textstyle\sum_{q=0}^{s-1}\bar{\eta}_{q}\right)^{2}\right].

(3.10)

As a consequence, for any $\epsilon>0$ , if we choose $T:=\left\lceil 4\min\{m,n\}(\frac{GR}{\epsilon})^{2}\right\rceil$ , $\gamma:=3/(\min\{m,n\}R^{2}T)$ , $\rho:=\exp(-3/(2T))$ , $\bar{\eta}_{t}=R\sqrt{\min\{m,n\}/T}$ , and $\eta_{t}^{\rm M}\in[0,R/(2\sqrt{T})]$ , then Algorithm 1 visits a time step $\hat{t}\in\mathop{\rm argmin}_{0\leq t<T}f(W_{t})$ such that $f(W_{\hat{t}})-f(W^{\star})\leq\epsilon$ , and the number of subgradient evaluations is

O\left(\min\{m,n\}\left(\tfrac{GR}{\epsilon}\right)^{2}\right).

Proof. Let $r:=\min\{m,n\}$ and $A_{s}:=\sum_{q=0}^{s-1}\bar{\eta}_{q}$ , with $A_{0}:=0$ . Let $\widetilde{G}_{s}:=G_{s}/\|G_{s}\|_{F}$ with $0/0:=0$ . Letting $P_{s+1}\in\operatorname{polar}_{\delta_{s}^{\rm M}}\left(\frac{M_{s+1}}{\|M_{s+1}\|_{F}+\sigma_{s}^{\rm M}}\right)$ so that $U_{s+1}=U_{s}-\eta_{s}^{\rm M}P_{s+1}$ , we get $\|P_{s+1}\|_{\textnormal{op}}\leq 2$ and $\|P_{s+1}\|_{F}\leq 2\sqrt{r}$ . Since $S_{0}=U_{0}$ , $S_{s+1}=S_{s}-\bar{\eta}_{s}\widetilde{G}_{s}$ , and $2\sqrt{r}\,\eta_{s}^{\rm M}\leq\bar{\eta}_{s}$ , induction gives $\|S_{s}-U_{s}\|_{F}\leq 2\sum_{q=0}^{s-1}\bar{\eta}_{q}=2A_{s}$ .

We bound the linearized function gap. Letting $\Phi(z):=\log(1+e^{-z})$ , we have $-\Phi^{\prime}(z_{s})=1-\alpha_{s}$ and $0\leq\Phi^{\prime\prime}\leq 1/4$ , and thus $\Phi(z_{s+1})\leq\Phi(z_{s})+\gamma\bar{\eta}_{s}(1-\alpha_{s})\langle\widetilde{G}_{s},S_{s}-U_{s}\rangle+\tfrac{\gamma^{2}\bar{\eta}_{s}^{2}}{8}\langle\widetilde{G}_{s},S_{s}-U_{s}\rangle^{2}$ . Since $|\langle\widetilde{G}_{s},S_{s}-U_{s}\rangle|\leq 2A_{s}$ , summing from $s=0$ to $t-1$ , we have

-\sum_{s<t}\bar{\eta}_{s}(1-\alpha_{s})\langle\widetilde{G}_{s},S_{s}-U_{s}\rangle\leq\tfrac{\Phi(z_{0})-\Phi(z_{t})}{\gamma}+\tfrac{\gamma}{8}\sum_{s<t}\bar{\eta}_{s}^{2}\langle\widetilde{G}_{s},S_{s}-U_{s}\rangle^{2}\leq\tfrac{\log(1/\rho)}{\gamma}+\tfrac{\gamma}{2}\sum_{s<t}\bar{\eta}_{s}^{2}A_{s}^{2}.

(3.11)

It follows from $S_{s+1}=S_{s}-\bar{\eta}_{s}\widetilde{G}_{s}$ that $\bar{\eta}_{s}\langle\widetilde{G}_{s},S_{s}-W^{\star}\rangle\leq\tfrac{1}{2}\left(\|S_{s}-W^{\star}\|_{F}^{2}-\|S_{s+1}-W^{\star}\|_{F}^{2}\right)+\tfrac{\bar{\eta}_{s}^{2}}{2}$ . Summing over $s=0,\ldots,t-1$ , we obtain

\sum_{s<t}\bar{\eta}_{s}\langle\widetilde{G}_{s},S_{s}-W^{\star}\rangle\leq\tfrac{1}{2}\|W_{0}-W^{\star}\|_{F}^{2}-\tfrac{1}{2}\|S_{t}-W^{\star}\|_{F}^{2}+\tfrac{1}{2}\sum_{s<t}\bar{\eta}_{s}^{2}\leq\tfrac{1}{2}\|W_{0}-W^{\star}\|_{F}^{2}+\tfrac{1}{2}\sum_{s<t}\bar{\eta}_{s}^{2}.

Since $W_{s}=S_{s}-(1-\alpha_{s})(S_{s}-U_{s})$ , combining the above inequality with Eq. (3.11) gives

	$\displaystyle\sum_{s<t}\bar{\eta}_{s}\langle\widetilde{G}_{s},W_{s}-W^{\star}\rangle$	$\displaystyle=\sum_{s<t}\bar{\eta}_{s}\langle\widetilde{G}_{s},S_{s}-W^{\star}\rangle-\sum_{s<t}\bar{\eta}_{s}(1-\alpha_{s})\langle\widetilde{G}_{s},S_{s}-U_{s}\rangle$
		$\displaystyle\leq\tfrac{r}{2}\\|W_{0}-W^{\star}\\|_{\textnormal{op}}^{2}+\tfrac{\log(1/\rho)}{\gamma}+\tfrac{1}{2}\sum_{s<t}\bar{\eta}_{s}^{2}+\tfrac{\gamma}{2}\sum_{s<t}\bar{\eta}_{s}^{2}A_{s}^{2}=:C_{t}.$

We convert the inequality above to Eq. (3.10). Let $j<t$ minimize $v_{s}:=\langle\widetilde{G}_{s},W_{s}-W^{\star}\rangle$ . Then, $v_{j}\leq C_{t}/A_{t}$ . If $C_{t}/A_{t}\geq R$ , $\min_{s<t}f(W_{s})-f(W^{\star})\leq f(W_{0})-f(W^{\star})\leq GR\leq GC_{t}/A_{t}$ . We therefore suppose that $C_{t}/A_{t}<R$ . If $v_{j}<0$ , the convexity of $f$ gives $f(W_{j})-f(W^{\star})<0$ . If $G_{j}=0$ , $W_{j}$ is a minimizer of $f$ . Thus, it remains to consider $v_{j}\geq 0$ and $G_{j}\neq 0$ . We choose $D_{j}\in\partial(\|\cdot\|_{\textnormal{nuc}})(G_{j})$ , $\theta_{j}:=\|G_{j}\|_{F}v_{j}/\|G_{j}\|_{\textnormal{nuc}}$ , and $Y_{j}:=W^{\star}+\theta_{j}D_{j}$ so that $\langle G_{j},W_{j}-Y_{j}\rangle=0$ . We then have

0\leq\theta_{j}\leq v_{j}\leq C_{t}/A_{t}<R,\quad\|Y_{j}-W^{\star}\|_{\textnormal{op}}\leq R.

Convexity and Lipschitz continuity therefore proves Eq. (3.10) by

f(W_{j})\leq f(Y_{j})\leq f(W^{\star})+G\theta_{j}\leq f(W^{\star})+GC_{t}/A_{t}.

We derive the complexity bound using Eq. (3.10). For the stated choices of $T,\gamma,\rho,\bar{\eta}_{t}$ , we have $A_{s}=sR\sqrt{r/T}$ . Hence, it follows that $\sum_{s<T}\bar{\eta}_{s}^{2}A_{s}^{2}=R^{4}\frac{r^{2}}{T^{2}}\sum_{s<T}s^{2}\leq\frac{1}{3}R^{4}r^{2}T$ . We also have $\sum_{s<T}\bar{\eta}_{s}=R\sqrt{rT}$ , $\sum_{s<T}\bar{\eta}_{s}^{2}=rR^{2}$ , and $\log(1/\rho)/\gamma=rR^{2}/2$ . Thus, each of the four terms defining $C_{T}$ is at most $rR^{2}/2$ and $C_{T}\leq 2rR^{2}$ . It follows from Eq. (3.10) that $\min_{s<T}f(W_{s})-f(W^{\star})\leq 2GR\sqrt{\tfrac{r}{T}}\leq\epsilon$ . $\Box$

3.3 Oracle complexity in operator-norm geometry

While Theorem 3.5 guarantees that MAGD attains an $\epsilon$ -optimal function value in $O(\min\{m,n\}(GR/\epsilon)^{2})$ subgradient evaluations under Assumption 2.3, is such dimension-dependent factor intrinsic to operator-norm geometry? We provide an affirmative answer by showing that this factor is unavoidable for first-order methods when $\max\{m,n\}$ is sufficiently large. Our analysis is based on the local first-order oracle model and notation of Section 2.3. We recall $\mathfrak{m}_{\textnormal{wcc}}$ and $\mathfrak{m}_{\textnormal{dc}}$ (see Eq. (2.3) and (2.4)) and define

\mathcal{X}_{\textnormal{op}}(R):=\{W\in\mathbb{R}^{m\times n}:\|W\|_{\textnormal{op}}\leq R\}.

and

\mathcal{F}_{\textnormal{op}}(G,R):=\left\{f:\mathbb{R}^{m\times n}\to\mathbb{R}\ \middle|\ \begin{array}[]{l}f\textnormal{ is convex and }G\textnormal{-Lipschitz with respect to}\\ \|\cdot\|_{\textnormal{op}}\textnormal{ on an open set containing }\mathcal{X}_{\textnormal{op}}(R)\end{array}\right\}.

Theorem 3.6

If $\max\{m,n\}>(GR/\epsilon)^{2}>1$ , we have

\Omega\left(\min\{m,n\}\left(\tfrac{GR}{\epsilon}\right)^{2}\right)=\mathfrak{m}_{\textnormal{dc}}(\mathcal{X}_{\textnormal{op}}(R),\mathcal{F}_{\textnormal{op}}(G,R),\mathsf{O},\epsilon)\leq\mathfrak{m}_{\textnormal{wcc}}(\mathcal{X}_{\textnormal{op}}(R),\mathcal{F}_{\textnormal{op}}(G,R),\mathsf{O},\epsilon)=O\left(\min\{m,n\}\left(\tfrac{GR}{\epsilon}\right)^{2}\right),

where $\mathsf{O}$ is a local first-order oracle on $\mathcal{F}_{\textnormal{op}}(G,R)$ .

We divide the proof into two steps: (i) construct the hard instances in operator-norm geometry and reformulate the resulting optimization problem as the string guessing problem (Braun et al., 2017, Oracle III.2 and Proposition III.3), and (ii) apply the lower bound for solving the string guessing problem to yield the desired result.

Definition 3.7 (string guessing oracle)

Fixing $M\geq 1$ and any hidden sign vector $S\in\{\pm 1\}^{M}$ , a query to the string guessing oracle on $S$ is $(a_{i},g_{i})_{i=1}^{\ell}$ , where $0\leq\ell\leq M$ , indices $\{a_{i}\}_{i=1}^{\ell}\subseteq\{1,2,\ldots,M\}$ , and guesses $\{g_{i}\}_{i=1}^{\ell}\in\{\pm 1\}^{\ell}$ . If there exists $1\leq i\leq\ell$ such that $S_{a_{i}}\neq g_{i}$ , the oracle returns the first mismatch $\inf\{1\leq i\leq\ell:S_{a_{i}}\neq g_{i}\}$ . Otherwise, the oracle returns equal.

For simplicity, we define $[N]:=\{1,\ldots,N\}$ for any integer $N\geq 1$ . A string guessing algorithm recovers $S\in\{\pm 1\}^{M}$ by querying the string guessing oracle $\mathcal{O}$ . For $q\geq 1$ , $t\in[q]$ , $0\leq\ell_{t}\leq M$ , $\{a_{i}^{t}\}_{i=1}^{\ell_{t}}\subseteq[M]$ , $\{g_{i}^{t}\}_{i=1}^{\ell_{t}}\in\{\pm 1\}^{\ell_{t}}$ and $Q_{t}=(a_{i}^{t},g_{i}^{t})_{i=1}^{\ell_{t}}$ , it keeps $H_{q}=(Q_{t},\mathcal{O}(Q_{t}))_{t\in[q]}$ with $H_{0}=\emptyset$ , and uses a decision rule based on $H_{q}$ that either returns some $\widehat{S}\in\{\pm 1\}^{M}$ or the next query $(a_{i}^{q+1},g_{i}^{q+1})_{i=1}^{\ell_{q+1}}$ . A consequence of (Braun et al., 2017, Proposition III.3) shows that $\mathcal{O}$ hides useful information from any randomized algorithm attempting to recover $S\sim\textnormal{Unif}(\{\pm 1\}^{M})$ . We define the binary entropy by

\mathbf{H}:[0,1]\to\mathbb{R},\quad\mathbf{H}(p):=-p\log_{2}p-(1-p)\log_{2}(1-p).

We summarize this result in the following lemma.

Lemma 3.8

Let $M\geq 1$ and $S\sim\textnormal{Unif}(\{\pm 1\}^{M})$ . For any string guessing algorithm $\mathsf{A}$ that accesses $S$ only through the string guessing oracle and stops with an output $\widehat{S}(\mathsf{A},S)$ after $\mathcal{T}_{\mathsf{A}}(S)$ queries, we have

\mathbb{E}[\mathcal{T}_{\mathsf{A}}(S)]\geq\tfrac{1}{2}((1-\mathbb{P}\{\widehat{S}(\mathsf{A},S)\neq S\})M-\mathbf{H}(\mathbb{P}\{\widehat{S}(\mathsf{A},S)\neq S\})).

As a consequence, the lower bound on $\mathbb{E}_{S}[\mathcal{T}_{\mathsf{A}}(S)]$ increases in $M$ for given $\mathbb{P}\{\widehat{S}(\mathsf{A},S)\neq S\}\leq\frac{1}{2}$ fixed, and decreases in $\mathbb{P}\{\widehat{S}(\mathsf{A},S)\neq S\}\leq\frac{1}{2}$ given $M$ fixed.

The following proposition shows that optimizing a max-of-affine function whose $M$ affine pieces encode a hidden vector $S\sim\operatorname{Unif}(\{\pm 1\}^{M})$ is at least as hard as recovering $S$ through the string guessing oracle. We recall from Section 2.3 that $\mathcal{A}(\mathsf{O})$ is the set of deterministic algorithms using oracle $\mathsf{O}$ , and $\mathsf{T}_{\mathsf{A}}(f,\epsilon)$ is the number of queries made by $\mathsf{A}\in\mathcal{A}(\mathsf{O})$ to reach an $\epsilon$ -optimal solution of $f$ .

Proposition 3.9

Let $M\geq 1$ , $\tau>0$ , and $A=(A_{a})_{a\in[M]}\in(\mathbb{R}^{m\times n})^{M}$ such that $\|A_{a}\|_{\textnormal{nuc}}\leq 1$ for every $a\in[M]$ . For $s\in\{\pm 1\}^{M}$ and $\theta\in[0,\tau]^{M}$ , let $F_{s,\theta}:\mathbb{R}^{m\times n}\to\mathbb{R}$ , $F_{s,\theta}(X):=\max_{a\in[M]}[s_{a}\langle A_{a},X\rangle+\theta_{a}]$ . We draw $S\sim\operatorname{Unif}(\{\pm 1\}^{M})$ and $\Theta\sim\operatorname{Unif}([0,\tau]^{M})$ independently. For any $\delta_{0}>0$ , any local first-order oracle $\mathsf{O}$ on $\mathcal{F}_{\textnormal{op}}(1,1)$ , and any $\mathsf{A}\in\mathcal{A}(\mathsf{O})$ such that $\mathbb{P}(\mathsf{T}_{\mathsf{A}}(F_{S,\Theta},\delta_{0})<+\infty)=1$ , for almost every $\theta\in[0,\tau]^{M}$ , there exists a string guessing algorithm $\mathsf{B}_{\theta}$ such that the string guessing oracle responses that $\mathsf{B}_{\theta}$ receives on any $s\in\{\pm 1\}^{M}$ determine the queries, oracle responses, and output of $\mathsf{A}$ on $F_{s,\theta}$ with oracle $\mathsf{O}$ , $\mathcal{T}_{\mathsf{B}_{\theta}}(s)\leq\mathsf{T}_{\mathsf{A}}(F_{s,\theta},\delta_{0})$ , and $\mathbb{P}\{\widehat{S}(\mathsf{B}_{\theta},S)\neq S\}\leq 1-|\mathcal{G}_{A,4\tau+\delta_{0}}|/2^{M}$ , where

\mathcal{G}_{A,\gamma}:=\{s\in\{\pm 1\}^{M}:\textnormal{ there exists }X\in\mathcal{X}_{\textnormal{op}}(1)\textnormal{ such that }s_{a}\langle A_{a},X\rangle\leq-\gamma\textnormal{ for all }a\in[M]\}.

Proof. For $a\in[M]$ , $s\in\{\pm 1\}^{M}$ , and $\theta\in[0,\tau]^{M}$ , we define the function $F_{s,\theta}(X)=\max_{a\in[M]}\phi_{a}^{s,\theta}(X)$ where $\phi_{a}^{s,\theta}(X):=s_{a}\langle A_{a},X\rangle+\theta_{a}$ . Accordingly, we define $I_{s,\theta}(X):=\{a\in[M]:\phi_{a}^{s,\theta}(X)=F_{s,\theta}(X)\}$ . Given any algorithm $\mathsf{A}\in\mathcal{A}(\mathsf{O})$ , we define $I^{t}_{s,\theta}$ as the index of the affine function for its $t^{\textnormal{th}}$ query $X^{t}_{s,\theta}$ , i.e., $I^{t}_{s,\theta}:=I_{s,\theta}(X^{t}_{s,\theta})$ . If $\mathsf{A}$ stops before $t^{\textnormal{th}}$ iteration, $I^{t}_{s,\theta}:=\emptyset$ . We also define $J^{t}_{s,\theta}=\bigcup_{q=1}^{t}I^{q}_{s,\theta}$ with $J^{0}=\emptyset$ .

In what follows, we define $\mathcal{E}:=\{\theta\in[0,\tau]^{M}:|I^{t}_{s,\theta}\setminus J^{t-1}_{s,\theta}|\leq 1,\mathsf{T}_{\mathsf{A}}(F_{s,\theta},\delta_{0})<+\infty,\forall s\in\{\pm 1\}^{M},\forall t\geq 1\}$ and show that (i) for $\forall\theta\in\mathcal{E}$ , there exists a string guessing algorithm $\mathsf{B}_{\theta}$ with an algorithm $\mathsf{A}^{\prime}$ such that the string guessing oracle responses that $\mathsf{B}_{\theta}$ receives produces the oracle responses that $\mathsf{A}^{\prime}$ receives. $\mathsf{A}^{\prime}$ and $\mathsf{A}$ are based on different oracles but have the same oracle responses if we run $\mathsf{A}$ on $F_{s,\theta}$ with oracle $\mathsf{O}$ for every $s\in\{\pm 1\}^{M}$ ; (ii) $\mathcal{T}_{\mathsf{B}_{\theta}}(s)\leq\mathsf{T}_{\mathsf{A}}(F_{s,\theta},\delta_{0})$ and $\mathbb{P}\{\widehat{S}(\mathsf{B}_{\theta},S)\neq S\}\leq 1-|\mathcal{G}_{A,4\tau+\delta_{0}}|/2^{M}$ for every $s\in\{\pm 1\}^{M}$ ; and (iii) $\mathbb{P}(\mathcal{E})=1$ . For $z\in\mathbb{R}$ , we define $\operatorname{sgn}_{0}(z)=1$ if $z\geq 0$ and $\operatorname{sgn}_{0}(z)=-1$ otherwise.

Figure 1: Reduction of a convex nonsmooth optimization algorithm

\mathsf{A}

to a string-guessing algorithm

\mathsf{B}

We prove (i) by characterizing $\mathsf{B}_{\theta}$ and $\mathsf{A}^{\prime}$ as shown in Figure 1 and showing the desired properties. Indeed, we denote by $D^{t-1}\subseteq[M]$ the indices of the signs that are known to $\mathsf{B}_{\theta}$ before $\mathsf{A}^{\prime}$ makes its $t^{\textnormal{th}}$ query with $D^{0}=\emptyset$ . For $a\notin D^{t-1}$ and a query $X^{t}_{s,\theta}$ , we define $z_{a}^{t}:=\langle A_{a},X^{t}_{s,\theta}\rangle$ , the sign of the lower sign and higher value $\ell_{a}^{t}=-\operatorname{sgn}_{0}(z_{a}^{t})$ and $h_{a}^{t}=\theta_{a}+|z_{a}^{t}|$ . Suppose that the $t^{\textnormal{th}}$ decision is to query $X^{t}_{s,\theta}$ . Then, we order the unknown indices as $(b_{1}^{t},\ldots,b_{M-|D^{t-1}|}^{t})$ according to non-increasing values $h_{b_{i}^{t}}^{t}$ . If $|D^{t-1}|=M$ , we set $D^{t}=D^{t-1}$ . If $|D^{t-1}|<M$ , $\mathsf{B}_{\theta}$ queries the string guessing oracle with $((b_{i}^{t},\ell_{b_{i}^{t}}^{t}))_{i=1}^{M-|D^{t-1}|}$ . A response equal gives $s_{b_{i}^{t}}=\ell_{b_{i}^{t}}^{t}$ for every $i\in[M-|D^{t-1}|]$ and $D^{t}=[M]$ . A first mismatch at position $j_{t}$ gives $s_{b_{i}^{t}}=\ell_{b_{i}^{t}}^{t}$ for $i<j_{t}$ and $s_{b_{j_{t}}^{t}}=-\ell_{b_{j_{t}}^{t}}^{t}$ . Finally, we update $D^{t}:=D^{t-1}\cup\{b_{1}^{t},\ldots,b_{j_{t}}^{t}\}$ and define $V^{t}:=\mathsf{O}_{F_{\bar{s}^{t},\theta}}(X^{t}_{s,\theta})$ where

\bar{s}_{a}^{t}:=\begin{cases}s_{a},&\textnormal{if }a\in D^{t},\\ \ell_{a}^{t},&\textnormal{otherwise},\end{cases}

We feed $V^{t}$ to $\mathsf{A}^{\prime}$ and $(X^{t}_{s,\theta},V^{t})$ to an ordered list $\Pi^{t-1}$ to get $\Pi^{t}$ that stores previous queries and oracle responses with $\Pi^{0}=\emptyset$ . If the $t^{\textnormal{th}}$ decision is to output $\widehat{X}$ , we output $\widehat{s}_{a}:=-\operatorname{sgn}_{0}(\langle A_{a},\widehat{X}\rangle)$ for $a\in[M]$ . Since $\mathsf{A}^{\prime}$ receives $V^{t}$ , we proceed to the $(t+1)^{\textnormal{th}}$ decision of $\mathsf{A}^{\prime}$ to query $X^{t+1}$ or stop with some $\widehat{X}$ .

By definition, $\mathsf{A}^{\prime}$ and $\mathsf{A}$ are based on different oracles. Then, we show that they have the same oracle responses if we run $\mathsf{A}$ on $F_{s,\theta}$ with oracle $\mathsf{O}$ for every $s\in\{\pm 1\}^{M}$ . It suffices to show that $J^{t}_{s,\theta}\subseteq D^{t}$ and $\Pi^{t}$ coincides with the queries and oracle responses that running $\mathsf{A}$ on $F_{s,\theta}$ with oracle $\mathsf{O}$ would receive. We first prove $J^{t}_{s,\theta}\subseteq D^{t}$ by induction. Indeed, $J^{0}\subseteq D^{0}$ since $J^{0}=D^{0}=\emptyset$ . If $D^{t-1}=[M]$ or the oracle replies equal, we have $J^{t}_{s,\theta}\subseteq D^{t}=[M]$ . Thus, we consider the case when the oracle replies with a mismatch $b:=b_{j_{t}}^{t}$ . By definition, $s_{b}=-\ell_{b}^{t}$ and $\phi_{b}^{s,\theta}(X^{t}_{s,\theta})=h_{b}^{t}$ . We claim that $I^{t}_{s,\theta}\subseteq D^{t}$ . Indeed, for any $a\notin D^{t}$ , we have $h_{a}^{t}\leq h_{b}^{t}$ . By definition, we have $\phi_{a}^{s,\theta}(X^{t}_{s,\theta})\leq h_{a}^{t}$ . If $\phi_{b}^{s,\theta}(X^{t}_{s,\theta})<F_{s,\theta}(X^{t}_{s,\theta})$ , we have $h_{b}^{t}<F_{s,\theta}(X^{t}_{s,\theta})$ . Thus, $\phi_{a}^{s,\theta}(X^{t}_{s,\theta})\leq h_{a}^{t}\leq h_{b}^{t}<F_{s,\theta}(X^{t}_{s,\theta})$ which implies $a\notin I^{t}_{s,\theta}$ . If $\phi_{b}^{s,\theta}(X^{t}_{s,\theta})=F_{s,\theta}(X^{t}_{s,\theta})$ , we have $b\in I^{t}_{s,\theta}\setminus J^{t-1}_{s,\theta}$ since $J^{t-1}_{s,\theta}\subseteq D^{t-1}$ . Since $\theta\in\mathcal{E}$ , we have $|I^{t}_{s,\theta}\setminus J^{t-1}_{s,\theta}|\leq 1$ which implies $a\notin I^{t}_{s,\theta}$ . Since $J^{t}_{s,\theta}=J^{t-1}_{s,\theta}\cup I^{t}_{s,\theta}$ , we have $J^{t}_{s,\theta}\subseteq D^{t}$ . We then prove that $\Pi^{t}$ coincides with the queries and oracle responses that running $\mathsf{A}$ on $F_{s,\theta}$ with oracle $\mathsf{O}$ would receive by induction. Indeed, we have

\phi_{a}^{\bar{s}^{t},\theta}(X^{t}_{s,\theta})=\theta_{a}-|z_{a}^{t}|\leq\phi_{a}^{s,\theta}(X^{t}_{s,\theta})<F_{s,\theta}(X^{t}_{s,\theta}),\textnormal{ for any }a\notin D^{t}.

which implies that $F_{s,\theta}=F_{\bar{s}^{t},\theta}$ on the neighborhood of $X^{t}_{s,\theta}$ and $\mathsf{O}_{F_{s,\theta}}(X^{t}_{s,\theta})=\mathsf{O}_{F_{\bar{s}^{t},\theta}}(X^{t}_{s,\theta})=V^{t}$ .

We prove (ii). By definition, $\mathsf{B}_{\theta}$ makes at most one query per query of $\mathsf{A}$ . Combining it with $\theta\in\mathcal{E}$ yields that $\mathcal{T}_{\mathsf{B}_{\theta}}(s)\leq\mathsf{T}_{\mathsf{A}}(F_{s,\theta},\delta_{0})<+\infty$ for all $s\in\{\pm 1\}^{M}$ . We fix $s\in\mathcal{G}_{A,4\tau+\delta_{0}}$ and choose $X_{s}\in\mathcal{X}_{\textnormal{op}}(1)$ from the definition of $\mathcal{G}_{A,4\tau+\delta_{0}}$ . Since $\theta_{a}\leq\tau$ , we have $F_{s,\theta}(X_{s})\leq-3\tau-\delta_{0}$ . If $\widehat{X}$ is the $\delta_{0}$ -optimal point output by $\mathsf{A}$ , we have $F_{s,\theta}(\widehat{X})\leq-3\tau<0$ . Since $\theta_{a}\geq 0$ , this implies $s_{a}\langle A_{a},\widehat{X}\rangle<0$ for every $a\in[M]$ , and $\widehat{s}_{a}=-\operatorname{sgn}_{0}(\langle A_{a},\widehat{X}\rangle)=s_{a}$ . Therefore, $\mathsf{B}_{\theta}$ recovers every $s\in\mathcal{G}_{A,4\tau+\delta_{0}}$ , and we have $\mathbb{P}\{\widehat{S}(\mathsf{B}_{\theta},S)\neq S\}\leq 1-|\mathcal{G}_{A,4\tau+\delta_{0}}|/2^{M}$ .

We prove (iii). Since $\mathbb{P}(\mathsf{T}_{\mathsf{A}}(F_{S,\Theta},\delta_{0})<+\infty)=1$ and $S\in\{\pm 1\}^{M}$ is a finite-valued random variable, $\mathsf{T}_{\mathsf{A}}(F_{s,\theta},\delta_{0})<+\infty$ for any $s\in\{\pm 1\}^{M}$ and almost every $\theta$ . It suffices to show that $\mathbb{P}\{|I^{t}_{s,\Theta}\setminus J^{t-1}_{s,\Theta}|\leq 1\textnormal{ for all }t\geq 1\}=1$ for any $s\in\{\pm 1\}^{M}$ .

Fixing $t\geq 1$ , $s\in\{\pm 1\}^{M}$ and $a\neq b$ , we define $\Theta_{-a}:=(\Theta_{1},\ldots,\Theta_{a-1},\Theta_{a+1},\ldots,\Theta_{M})$ and claim that $\mathbb{P}(a,b\in(I^{t}_{s,\Theta}\setminus J^{t-1}_{s,\Theta})\mid\Theta_{-a})=0$ . For $u\in[0,\tau]$ , we let $\theta(u)\in[0,\tau]^{M}$ satisfy $\theta_{a}(u)=u$ and $\theta_{-a}(u)=\Theta_{-a}$ , $X^{q}(u):=X^{q}_{s,\theta(u)}$ and define $E=\{u\in[0,\tau]:a,b\in I^{t}_{s,\theta(u)}\setminus J^{t-1}_{s,\theta(u)}\}$ . In what follows, we show that $E$ is countable. Indeed, by induction over $t$ , we show that $X^{t}(u)$ is locally constant for $u\in E$ . Indeed, suppose that $X^{q}(u^{\prime})=X^{q}(u)$ for every $u^{\prime}$ in a neighborhood of $u$ for some $q<t$ . Since $a\notin J^{t-1}_{s,\theta(u)}$ , we have $\phi^{s,\theta(u)}_{a}(X^{q}(u))<F_{s,\theta(u)}(X^{q}(u))$ for every $q<t$ . This together with the continuity implies that, after shrinking this neighborhood if necessary, $\phi^{s,\theta(u^{\prime})}_{a}(X)<\max_{c\neq a}\phi^{s,\theta(u^{\prime})}_{c}(X)$ for every $u^{\prime}$ near $u$ and every $X$ in a neighborhood of $X^{q}(u)$ . For every $c\neq a$ , we have $\phi^{s,\theta(u^{\prime})}_{c}=\phi^{s,\theta(u)}_{c}$ since $\theta_{c}(u^{\prime})=\theta_{c}(u)=\Theta_{c}$ . This implies $F_{s,\theta(u^{\prime})}=F_{s,\theta(u)}$ on the neighborhood of $X^{q}(u)$ . By locality of $\mathsf{O}$ , the oracle response at $X^{q}(u^{\prime})=X^{q}(u)$ remains unchanged, and thus the $(q+1)^{\textnormal{th}}$ query also remains unchanged. Since the first query is independent of $u$ , this induction shows that $X^{t}(\cdot)$ is constant on a neighborhood of $u$ . This yields the desired result and further implies that $\Theta_{b}+s_{b}\langle A_{b},X^{t}(\cdot)\rangle-s_{a}\langle A_{a},X^{t}(\cdot)\rangle$ is constant on this neighborhood. If $u\in E$ , $a,b\in I^{t}_{s,\theta(u)}\setminus J^{t-1}_{s,\theta(u)}$ implies $u=\Theta_{b}+s_{b}\langle A_{b},X^{t}(u)\rangle-s_{a}\langle A_{a},X^{t}(u)\rangle$ . Thus, at most one point of $E$ satisfies this equality in this neighborhood. Putting these pieces together yields that $E$ is countable. Since $\Theta_{a}$ is conditionally uniform on $[0,\tau]$ , we obtain $\mathbb{P}(\Theta_{a}\in E\mid\Theta_{-a})=0$ , which proves $\mathbb{P}(a,b\in I^{t}_{s,\Theta}\setminus J^{t-1}_{s,\Theta}\mid\Theta_{-a})=0$ .

Finally, we have $\mathbb{P}(a,b\in(I^{t}_{s,\Theta}\setminus J^{t-1}_{s,\Theta}))=0$ for any $s\in\{\pm 1\}^{M}$ . This implies that $\mathbb{P}\{|I^{t}_{s,\Theta}\setminus J^{t-1}_{s,\Theta}|\leq 1\textnormal{ for all }t\geq 1\}=1$ for any $s\in\{\pm 1\}^{M}$ and completes the proof. $\Box$

Proof of Theorem 3.6. Eq. (2.5) and Theorem 3.5 guarantee $\mathfrak{m}_{\textnormal{dc}}\leq\mathfrak{m}_{\textnormal{wcc}}=O(\min\{m,n\}(\frac{GR}{\epsilon})^{2})$ . Thus, it suffices to prove $\mathfrak{m}_{\textnormal{dc}}(\mathcal{X}_{\textnormal{op}}(R),\mathcal{F}_{\textnormal{op}}(G,R),\mathsf{O},\epsilon)=\Omega(\min\{m,n\}(\tfrac{GR}{\epsilon})^{2})$ .

To apply Proposition 3.9, we claim that, for any $\delta\in(\frac{1}{\sqrt{\max\{m,n\}}},1)$ , there exists a universal constant $c>0$ such that there exist $\tau>0$ , $M\geq\max\{1,c\min\{m,n\}\delta^{-2}\}$ , and $A=(A_{a})_{a\in[M]}\in(\mathbb{R}^{m\times n})^{M}$ with $\|A_{a}\|_{\textnormal{nuc}}\leq 1$ and $1-|\mathcal{G}_{A,4\tau+\delta}|/2^{M}\leq\frac{1}{8}$ . Indeed, the claim implies

\mathfrak{m}_{\textnormal{dc}}(\mathcal{X}_{\textnormal{op}}(1),\mathcal{F}_{\textnormal{op}}(1,1),{\mathsf{O}},\delta)=\Omega(\min\{m,n\}\delta^{-2})

(3.12)

for any local oracle ${\mathsf{O}}$ on $\mathcal{F}_{\textnormal{op}}(1,1)$ and any $\delta\in(\frac{1}{\sqrt{\max\{m,n\}}},1)$ using the hard function distribution $F_{S,\Theta}$ over $\mathcal{F}_{\textnormal{op}}(1,1)$ defined in Proposition 3.9. For any $\mathsf{A}\in\mathcal{A}({\mathsf{O}})$ , if $\mathsf{T}_{\mathsf{A}}(F_{S,\Theta},\delta)=+\infty$ with any positive probability, $\mathbb{E}[\mathsf{T}_{\mathsf{A}}(F_{S,\Theta},\delta)]=+\infty$ and the lower bound holds. Otherwise, Proposition 3.9 gives, for almost every $\theta$ , a string guessing algorithm $\mathsf{B}_{\theta}$ reproducing $\mathsf{A}$ ’s queries and responses and satisfying $\mathcal{T}_{\mathsf{B}_{\theta}}(s)\leq\mathsf{T}_{\mathsf{A}}(F_{s,\theta},\delta)$ for all $s\in\{\pm 1\}^{M}$ and $\mathbb{P}\{\widehat{S}(\mathsf{B}_{\theta},S)\neq S\}\leq\frac{1}{8}$ , and Lemma 3.8 gives $\mathbb{E}_{S}[\mathcal{T}_{\mathsf{B}_{\theta}}(S)]\geq\frac{1}{2}(\frac{7}{8}M-\mathbf{H}(\frac{1}{8}))$ . Taking expectation over $\Theta$ yields $\mathbb{E}[\mathsf{T}_{\mathsf{A}}(F_{S,\Theta},\delta)]\geq(\frac{7}{16}-\frac{1}{2}\mathbf{H}(\tfrac{1}{8}))c\min\{m,n\}\delta^{-2}$ . This holds for any $\mathsf{A}\in\mathcal{A}({\mathsf{O}})$ , proving Eq. (3.12).

In what follow, we prove the desired claim. Indeed, without loss of generality, we assume $m\leq n$ . We derive a key probability inequality to separate low and high accuracy regimes. Let $e_{i}^{(d)}$ denote the $i^{\rm th}$ standard basis vector in $\mathbb{R}^{d}$ . The uniform distribution on the $\mathbb{R}^{n}$ sphere with radius $\sqrt{n}$ is isotropic and has a uniformly bounded sub-gaussian norm (Vershynin, 2018, Theorem 3.4.6). Hence, (Vershynin, 2018, Theorem 4.6.1) gives an absolute constant $C>0$ such that every random $m\times n$ matrix $B$ with independent rows uniform on the unit sphere satisfies

\mathbb{P}\{\|B\|_{\textnormal{op}}\leq\sqrt{m/n}+2C\}\geq 1-2e^{-n}.

(3.13)

We set $c=(8c_{0}^{2})^{-1}$ where $c_{0}=1+2C$ and consider two cases. First, we assume $\delta\geq 1/(2c_{0})$ . By setting $M:=m$ , $A_{i}:=e_{i}^{(m)}(e_{i}^{(n)})^{\top}$ and $X_{s}:=-\sum_{i=1}^{m}s_{i}A_{i}$ for $\forall s\in\{\pm 1\}^{m}$ , we have $\|A_{i}\|_{\textnormal{nuc}}=1$ , $\|X_{s}\|_{\textnormal{op}}=1$ and $s_{i}\langle A_{i},X_{s}\rangle=-1$ . By setting $\tau:=(1-\delta)/4$ , we have $\mathcal{G}_{A,4\tau+\delta}=\{\pm 1\}^{M}$ and $M\geq cm\delta^{-2}$ . Second, we assume $1/\sqrt{n}<\delta<1/(2c_{0})$ . By setting $k=\lfloor(2c_{0}\delta)^{-2}\rfloor$ , we have $k<n$ , $n>4c_{0}^{2}\geq 4$ and $1-2e^{-n}\geq\frac{7}{8}$ . The existence of $A_{i}$ can be established by the following argument. Indeed, letting $Q_{1},\ldots,Q_{m}\in\mathbb{R}^{n\times n}$ be independent orthogonal matrices distributed uniformly with respect to the Haar measure, we set $u_{ij}:=Q_{i}e_{j}^{(n)}$ , $A_{ij}:=e_{i}^{(m)}u_{ij}^{\top}$ , and $B_{s}:=\tfrac{1}{\sqrt{k}}\sum_{i=1}^{m}\sum_{j=1}^{k}s_{ij}A_{ij}$ . For any given $s\in\{\pm 1\}^{m\times k}$ , the $i^{\textnormal{th}}$ row of $B_{s}$ is $(Q_{i}w_{i})^{\top}$ , where $w_{i}:=k^{-1/2}\sum_{j=1}^{k}s_{ij}e_{j}^{(n)}$ is a unit vector. Thus, the rows of $B_{s}$ are independent and uniform on the unit sphere. Since $\sqrt{m/n}+2C\leq c_{0}$ , Eq. (3.13) guarantees $\mathbb{P}_{Q}\{\|B_{s}\|_{\textnormal{op}}\leq c_{0}\}\geq\frac{7}{8}$ which implies $\mathbb{E}_{Q}[2^{-mk}|\{s\in\{\pm 1\}^{m\times k}:\|B_{s}\|_{\textnormal{op}}\leq c_{0}\}|]\geq\frac{7}{8}$ . Therefore, some deterministic realization of $Q_{1},\ldots,Q_{m}$ makes the quantity inside the expectation at least $\frac{7}{8}$ . We fix such a deterministic $Q_{1},\dots,Q_{m}$ and let $M=mk$ . Then, we define $X_{s}:=-\frac{B_{s}}{c_{0}}$ where

\mathcal{G}:=\{s\in\{\pm 1\}^{m\times k}:\|B_{s}\|_{\textnormal{op}}\leq c_{0}\}.

Thus, $|\mathcal{G}|/2^{M}\geq\frac{7}{8}$ and $\|X_{s}\|_{\textnormal{op}}\leq 1$ for all $s\in\mathcal{G}$ . In addition, we have $\|A_{ij}\|_{\textnormal{nuc}}=1$ and $\langle A_{ij},A_{\ell q}\rangle=\langle e_{i}^{(m)},e_{\ell}^{(m)}\rangle\langle u_{ij},u_{\ell q}\rangle=\mathbf{1}_{\{i=\ell,j=q\}}$ . As a consequence, $s_{ij}\langle A_{ij},X_{s}\rangle=-\frac{1}{c_{0}\sqrt{k}}$ . Letting $\tau:=\frac{1}{4c_{0}\sqrt{k}}-\frac{1}{4}\delta$ , we obtain that any $s\in\mathcal{G}$ belongs to $\mathcal{G}_{A,4\tau+\delta}$ which implies $|\mathcal{G}_{A,4\tau+\delta}|\geq|\mathcal{G}|\geq\frac{7}{8}\cdot 2^{M}$ . Moreover, we have $M=mk\geq cm\delta^{-2}$ . Mapping $\{\pm 1\}^{m\times k}$ to $\{\pm 1\}^{M}$ yields the claim. Therefore, we conclude that

\mathfrak{m}_{\textnormal{dc}}(\mathcal{X}_{\textnormal{op}}(1),\mathcal{F}_{\textnormal{op}}(1,1),{\mathsf{O}},\delta)=\Omega(\min\{m,n\}\delta^{-2})

for any local oracle ${\mathsf{O}}$ on $\mathcal{F}_{\textnormal{op}}(1,1)$ and any $\delta\in(\frac{1}{\sqrt{\max\{m,n\}}},1)$ .

It suffices to prove that, for any local oracle $\mathsf{O}$ on $\mathcal{F}_{\textnormal{op}}(G,R)$ there exists a local oracle $\widetilde{\mathsf{O}}$ on $\mathcal{F}_{\textnormal{op}}(1,1)$ such that for any $\epsilon>0$ , we have $\mathfrak{m}_{\textnormal{dc}}(\mathcal{X}_{\textnormal{op}}(R),\mathcal{F}_{\textnormal{op}}(G,R),\mathsf{O},\epsilon)\geq\mathfrak{m}_{\textnormal{dc}}(\mathcal{X}_{\textnormal{op}}(1),\mathcal{F}_{\textnormal{op}}(1,1),\widetilde{\mathsf{O}},\epsilon/GR)$ . Indeed, we define $\mathscr{S}_{G,R}:\mathcal{F}_{\textnormal{op}}(1,1)\to\mathcal{F}_{\textnormal{op}}(G,R)$ , $\mathscr{S}_{G,R}(F)(W):=GRF(W/R)$ . This is a bijection and guarantees that $X\mapsto RX$ maps $\mathcal{X}_{\textnormal{op}}(1)$ bijectively onto $\mathcal{X}_{\textnormal{op}}(R)$ and $\widehat{X}$ is $(\epsilon/GR)$ -optimal for $F$ if and only if $R\widehat{X}$ is $\epsilon$ -optimal for $\mathscr{S}_{G,R}(F)$ . Given a local first-order oracle $\mathsf{O}$ on $\mathcal{F}_{\textnormal{op}}(G,R)$ , we define

\widetilde{\mathsf{O}}_{F}(X)=\left(\tfrac{v}{GR},\tfrac{g}{G}\right),\textnormal{ where }(v,g)=\mathsf{O}_{\mathscr{S}_{G,R}(F)}(RX).

Since $g\in\partial\mathscr{S}_{G,R}(F)(RX)=G\cdot\partial F(X)$ , we obtain that $\widetilde{\mathsf{O}}$ is the first-order oracle that locality follows from $\mathsf{O}$ . Given any $\mathsf{A}\in\mathcal{A}(\mathsf{O})$ , we construct $\widetilde{\mathsf{A}}\in\mathcal{A}(\widetilde{\mathsf{O}})$ that simulates $\mathsf{A}$ by mapping every query $W$ to $W/R$ , rescaling the resulting oracle reply by $(GR,G)$ , and mapping the terminal output $\widehat{W}$ to $\widehat{W}/R$ . Thus, for every $F\in\mathcal{F}_{\textnormal{op}}(1,1)$ , $\mathsf{T}_{\widetilde{\mathsf{A}}}(F,\epsilon/GR)=\mathsf{T}_{\mathsf{A}}(\mathscr{S}_{G,R}(F),\epsilon)$ . Taking the infimum over algorithms and the supremum over distributions proves the desired result. $\Box$

4 Experiments

We conduct extensive experiments with nonsmooth hard instance, quadratic optimization, convolutional neural networks (CNNs) and LLM pretraining. The candidate approaches include SGD, inexact Muon (with either Newton-Schulz or PolarExpress) and MAGD (with either Newton-Schulz or PolarExpress). In each experiment, we tune the hyperparameters by grid search for all methods and also keep the step size and momentum schedules same for Muon and the Muon branch of MAGD.

4.1 Nonsmooth hard instance

We consider a differentiable approximation of hard instances from Theorem 3.1 parameterized by $\delta\geq 0$ . For $\delta=0$ , they reduce to the original instances. For $\delta>0$ , letting $p=W_{11}+W_{22}$ and $q=W_{11}-W_{22}$ , we define

f_{\delta}(W)=A_{1}\left(\sqrt{p^{2}+\delta^{2}}-\delta\right)+(A_{2}+1)\delta\log\left(\tfrac{A_{2}+e^{q/\delta}}{A_{2}+1}\right)-q.

where the first term approximates $A_{1}|p|$ , while the second term approximates $\max\{A_{2}q,0\}+\max\{-q,0\}$ . We use a fixed momentum factor $\beta_{t}\equiv\beta=0.7$ , $\mu=1.2671$ from the proof of Theorem 3.1, $A_{1}=0.009283$ , and $A_{2}=63.08$ . We initialize $W_{11}$ and $W_{22}$ independently from standard normal distributions and set all of other entries as $0$ .

We compare gradient descent (GD), fixed-momentum Muon, adaptive-momentum Muon, and MAGD for 50000 iterations over 100 initializations, using the exact polar updates. We search over step size schedules $x_{t},y_{t}$ as follows,

x_{t}=\tfrac{x_{0}}{2}\left[1+\cos\left(\tfrac{\pi t}{T-1}\right)\right],\quad y_{t}=\tfrac{y_{0}}{(1+t)^{a}},

where $x_{0}\in\{0.003,0.005,0.008,0.01,0.015,0.02,0.03,0.05,0.08\}$ , $a\in\{0.5,0.6,0.75,0.9,1\}$ and $y_{0}\in\{0.1,0.2,0.3,0.5,0.8\}$ . For MAGD, we additionally tune $\bar{\eta}_{t}$ to follow the same schedules with $x_{0},y_{0}\in\{0.02,0.03,0.035,0.04,0.05\}$ , $\rho=0.5$ , and $\gamma=1$ , and use normalized gradient descent for the GD branch as in Theorem 3.5. For every method and value of $\delta$ , we select the configuration that maximizes the fraction of initializations attaining an objective gap below $10^{-6}$ .

Table 1 shows that fixed-momentum Muon cannot reach an objective gap below $10^{-6}$ for all runs when $\delta=0$ . In contrast, adaptive-momentum Muon, GD and MAGD achieve improved performance. It is also worth noting that the performance of all methods improves as the parameter $\delta$ increases.

$\delta$	Muon (fixed $\beta_{t}=0.7$ )	Muon ( $\beta_{t}=\max\{1-256\eta_{t},0\}$ )	GD	MAGD (fixed $\beta_{t}^{\rm{M}}=0.7$ )
$0$	$0\%$	$85\%$	$100\%$	$100\%$
$10^{-4}$	$55\%$	$100\%$	$100\%$	$100\%$
$10^{-3}$	$100\%$	$100\%$	$100\%$	$100\%$
$10^{-2}$	$100\%$	$100\%$	$100\%$	$100\%$
$10^{-1}$	$100\%$	$100\%$	$100\%$	$100\%$

Table 1: Percentage of the 100 initializations for which the objective gap is below

10^{-6}

4.2 Quadratic optimization

We consider the quadratic optimization problems used in Gonon et al. (2026). Taking $n=100$ , the convex quadratic function is defined by

f(W)=\tfrac{1}{2}\langle W,AW\rangle,

where $A=Q\operatorname{diag}(s_{1},\ldots,s_{n})Q^{\top}$ is positive definite and $Q$ is random and orthogonal. After vectorizing $W$ , the Hessian of $f$ is $I_{n}\otimes A$ . Each eigenvalue $s_{i}$ of $A$ appears $n$ times in the spectrum of $f$ . In Table 2, we summarize seven spectrum families, where all of them use the same endpoints, $s_{\min}=10^{-3}$ and $s_{\max}=10$ . Thus, every problem has condition number $10^{4}$ and only the shape of the eigenvalue distribution varies. We sort the eigenvalues in decreasing order and exactly enforce the endpoints.

Spectrum	Construction
max_spiked	One eigenvalue at $s_{\min}$ , all others at $s_{\max}$ .
min_spiked	One eigenvalue at $s_{\max}$ , all others at $s_{\min}$ .
uniform	$s_{i}$ sampled uniformly on $[s_{\min},s_{\max}]$ .
gaussian	$s_{i}$ sampled from a clipped Gaussian centered in $[s_{\min},s_{\max}]$ .
linear_decay_to_max	$s_{i}=s_{\max}-(s_{\max}-s_{\min})\sqrt{u_{i}}$ , $u_{i}\sim\mathrm{Unif}[0,1]$ .
geometric_decay_to_max	$s_{i}=\max\{s_{\min},s_{\max}\cdot 0.9^{i-1}\}$ .
u_shaped	$s_{i}=s_{\min}+(s_{\max}-s_{\min})z_{i}$ , $z_{i}\sim\mathrm{Beta}(0.2,0.2)$ .

Table 2: Spectrum families used in the quadratic optimization experiment.

Refer to caption — Figure 2: Orders of magnitude of loss decrease after 500 iterations on the quadratic functions.

We compare GD, Muon, and MAGD using PolarExpress (Amsel et al., 2026a) for orthogonalization. We sample $[W_{0}]_{ij}\sim\mathcal{N}(0,0.01)$ and run each method for $T=500$ iterations over 100 initializations on an NVIDIA 4060 Laptop GPU with 8 GB of VRAM. We then report the mean number of orders of magnitude by which the loss decreases (i.e., $\log_{10}\left(\tfrac{f(W_{0})}{\min_{0\leq t\leq T}f(W_{t})}\right)$ ) over the random initializations, together with confidence intervals.

For hyperparameter tuning, we disable Muon momentum and perform a grid search over the constant step sizes in $\{0.1,0.01,0.001\}$ for both Muon and GD. The best step size is $0.1$ for GD and $0.01$ for Muon as observed in Gonon et al. (2026). We run MAGD using these step sizes for its GD and Muon branches, respectively, with $\rho=0.7$ and $\gamma=0.03$ .

We present the results in Figure 2. GD is strong on several instances whose spectra are concentrated near the largest eigenvalues. Muon is robust on spectra for which GD is slow and performs extremely well on min_spiked. MAGD combines these behaviors by adaptively mixing a GD branch and a Muon branch, thereby exhibiting best-of-both-worlds behavior on this benchmark. It outperforms the best of Muon and GD on five of the seven spectra. On each of the two spectra with a single outlier eigenvalue, it retains some of the gains achieved by the stronger baseline.

4.3 Convolutional neural network

We train a 2-million-parameter CIFARNET (Jordan, 2024) on CIFAR-10 (Krizhevsky, 2009) for image classification using one NVIDIA GPU with 48 GB of VRAM. We split the training set into 40000 training samples and 10000 validation samples and evaluate the models on the test set of 10000 samples. Each run uses 200 optimization steps with a batch size of 2000, corresponding to 10 passes over the training set. We train with cross-entropy loss and label smoothing and report unsmoothed cross-entropy loss. We compare Muon, MAGD, and stochastic gradient descent with momentum (SGDM) for training the main convolutional layers, where none of them uses weight decay. All remaining trainable parameters other than the main convolutional layers are optimized by SGDM. When tuning the step size and momentum for these methods, we use 3 Newton-Schulz iterations for orthogonalization. The optimization problem is nonsmooth because the CNN uses max pooling.

We select the hyperparameters by maximizing validation accuracy averaged over 50 seeds. For Muon, we test the pair of step size and momentum factor in

		$\displaystyle\{0.005,0.01,0.02,0.05,0.10,0.18,0.24,0.30,0.36,0.40\}\times$
		$\displaystyle\{0.40,0.50,0.55,0.60,0.65,0.70,0.80,0.85,0.90,0.95\}.$

For SGDM, we test step sizes in $\{0.0001,0.0002,0.0004,0.0008,0.001,0.002,0.004,0.008,0.01,0.02\}$ with the same momentum grid. For MAGD, we retain Muon’s selected step size and momentum factor and tune the MAGD-specific parameters under the same validation criterion. The selected step size and momentum factor are $0.001$ and $0.85$ for SGDM, and $0.24$ and $0.65$ for Muon. MAGD uses the selected Muon settings together with $\bar{\eta}_{t}=0.095$ , $\gamma=0.19$ , and $\rho=0.003$ .

In Table 3, we report the validation and test losses and accuracies averaged over 100 random seeds. MAGD performs competitively with Muon, and its performance improves as the approximation becomes more accurate. Moreover, both MAGD and Muon outperform SGDM in this setting.

Method	Val. Loss	Val. Accuracy	Test Loss	Test Accuracy
MAGD (NS3)	$0.4312\pm 0.0002$	$0.9253\pm 0.0002$	$0.4271\pm 0.0003$	$0.9277\pm 0.0002$
Muon (NS3)	$0.4313\pm 0.0003$	$0.9253\pm 0.0002$	$0.4271\pm 0.0003$	$0.9279\pm 0.0002$
MAGD (NS5)	$0.4310\pm 0.0003$	$0.9260\pm 0.0002$	$0.4271\pm 0.0002$	$0.9282\pm 0.0002$
Muon (NS5)	$0.4310\pm 0.0003$	$0.9259\pm 0.0002$	$0.4273\pm 0.0002$	$0.9282\pm 0.0001$
MAGD (PolarExpress)	$0.4291\pm 0.0002$	$0.9266\pm 0.0002$	$0.4249\pm 0.0002$	$0.9288\pm 0.0002$
Muon (PolarExpress)	$0.4294\pm 0.0003$	$0.9266\pm 0.0002$	$0.4253\pm 0.0003$	$0.9286\pm 0.0002$
SGDM	$0.5337\pm 0.0007$	$0.8885\pm 0.0004$	$0.5322\pm 0.0007$	$0.8901\pm 0.0003$

Table 3: Results are the mean

\pm

standard error over 100 seeds for both validation and test. NS3 and NS5 mean three and five Newton-Schulz steps, respectively. PolarExpress uses five steps.

4.4 LLM pretraining

We train 286M- and 537M-parameter nanochat transformers (Vaswani et al., 2017; Karpathy, 2025) on the NVIDIA ClimbMix dataset (Diao et al., 2025) for 1680 and 3584 steps, respectively. Both settings use a vocabulary of 32768 tokens, 42M validation tokens, a global batch size of $524288=2^{19}$ tokens, and a sequence length of 2048. Each run for the 286M-parameter model uses one NVIDIA GPU with 48 GB of VRAM and 32 gradient-accumulation steps. Each run for the 537M-parameter model uses four 40 GB NVIDIA A100 GPUs and 4 gradient-accumulation steps.

We use decoder-only transformers. Each transformer block contains self-attention with rotary position embeddings (RoPE) (Su et al., 2024) and $d\times 4d$ MLP layers with squared-ReLU activations. The 286M-parameter model has 12 layers, hidden dimension $d=768$ , and 6 attention heads, whereas the 537M-parameter model has 16 layers, hidden dimension $d=1024$ , and 8 attention heads. Both models use a head dimension of 128, learned value embeddings in alternating layers, and an untied head.

We compare Muon and MAGD for training the attention and MLP layers, using peak step sizes in $\{0.01,0.02,0.03,0.04\}$ with a warmup-stable-decay schedule. The weight decay used in Muon follows a cosine schedule, and the Muon branch of MAGD uses the same schedule. We use five PolarExpress steps for orthogonalization. All remaining trainable parameters are optimized using AdamW (Loshchilov and Hutter, 2019) with fine-tuned step sizes and weight decay factors. For MAGD, we use $\bar{\eta}_{t}=0.01$ , $\gamma=0.25$ , and $\rho=10^{-4}$ . Figure 3 shows that MAGD is competitive with Muon and exhibits similar robustness across step sizes.

5 Conclusion

We studied orthogonalized momentum for locally Lipschitz matrix optimization and established a sharp distinction between fixed and coupled momentum schedules. For every fixed momentum factor $\beta\in[0,1)$ , Muon can fail to approach the global optimal solution of a convex Lipschitz objective from almost every initialization, when the step sizes adapt to the full gradient history. In contrast, coupling the momentum factor to a diminishing step size through $\beta_{t}=1-\tau\eta_{t}$ can restore asymptotic convergence for inexact Muon under boundedness and vanishing polar error. In particular, every cluster point is stationary, the momentum converges to $0$ , and the objective values converge. Under convexity, this value coincides with the global minimum. These results identify the interaction between fixed momentum and orthogonalization as a fundamental obstruction in nonsmooth optimization.

To obtain a finite-time guarantee, we proposed MAGD, which adaptively combines an inexact Muon branch with a gradient descent branch. It retains stationary cluster points and convergent objective values for nonconvex objectives and finds an $\epsilon$ -optimal iterate within $O(\min\{m,n\}\epsilon^{-2})$ iterations for convex objectives. We derive the lower bound to demonstrate the optimal dependence on $\min\{m,n\}$ and $\epsilon$ . The numerical results on the synthetic problems, image classification and LLM pretraining illustrate that MAGD safeguards the orthogonalized update while retaining its empirical performance.

Several questions remain open. First, it is unclear whether Muon itself, without an auxiliary gradient branch, admits a uniform finite-time guarantee under an adaptive momentum and step-size schedule, even for convex Lipschitz objectives. Second, our asymptotic analysis requires the polar approximation error to vanish, whereas practical implementations use a fixed number of Newton-Schulz or PolarExpress iterations. Developing a theory that allows persistent approximation error would narrow the gap between theory and practice. Finally, extending the upper and lower bounds to other matrix-norm geometries would clarify which phenomena are specific to operator-norm optimization and which reflect more general principles of matrix-aware methods.

Acknowledgments

The first author is partially supported by a start-up fund at the University of Hong Kong and by the Hong Kong Research Grants Council under ECS project 27301425. The second author is partially supported by a start-up fund and the Early Career Scholarship Support Grant at Columbia University. JZ thanks Derek Long for useful discussions. This work used GPU at NCSA and Indiana Jetstream2 through allocation CIS261175 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program (Boerner et al., 2023), which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We acknowledge the GPU support from Columbia University’s Insomnia computer cluster.

References

Agarwal et al. (2012) A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J. Wainwright Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory 58 (5), pp. 3235–3249. Cited by: §1.
Amsel et al. (2026a) N. Amsel, D. Persson, C. Musco, and R. M. Gower The Polar Express: optimal matrix sign methods and their application to the Muon algorithm. In ICLR, External Links: Link Cited by: §1, §2.2, §4.2.
Amsel et al. (2026b) N. Amsel, J. Zhang, K. Ahn, A. Naeimi, A. Feng, B. Chen, T. Dao, and J. Langford Dion3: full-stack orthogonal updates. ArXiv Preprint: 2608.11612. Cited by: §1.
Arjevani et al. (2023) Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth Lower bounds for non-convex stochastic optimization. Mathematical Programming 199 (1), pp. 165–214. Cited by: §1.
Balles and Hennig (2018) L. Balles and P. Hennig Dissecting Adam: the sign, magnitude and variance of stochastic gradients. In ICML, pp. 404–413. Cited by: §1.
Beck (2017) A. Beck First-order methods in optimization. SIAM. Cited by: §1, §2.3.
Benaïm et al. (2005) M. Benaïm, J. Hofbauer, and S. Sorin Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization 44 (1), pp. 328–348. Cited by: §1, §3.1, §3.1, §3.1.
Bernstein and Newhouse (2024) J. Bernstein and L. Newhouse Old optimizer, new norm: an anthology. In NeurIPS Workshop on Optimization for Machine Learning, External Links: Link Cited by: §1.
Bernstein and Newhouse (2025) J. Bernstein and L. Newhouse Modular duality in deep learning. In ICML, pp. 3920–3930. Cited by: §1.
Bernstein et al. (2018) J. Bernstein, Y-X. Wang, K. Azizzadenesheli, and A. Anandkumar SignSGD: compressed optimisation for non-convex problems. In ICML, pp. 560–569. Cited by: §1, §1.
Beznosikov et al. (2023) A. Beznosikov, S. Horváth, P. Richtárik, and M. Safaryan On biased compression for distributed learning. The Journal of Machine Learning Research 24 (276), pp. 1–50. Cited by: §1.
Boerner et al. (2023) T. J. Boerner, S. Deems, T. R. Furlani, S. L. Knuth, and J. Towns ACCESS: advancing innovation: NSF’s advanced cyberinfrastructure coordination ecosystem: services & support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, pp. 173–176. Cited by: Acknowledgments.
Bolte et al. (2007) J. Bolte, A. Daniilidis, A. S. Lewis, and M. Shiota Clarke subgradients of stratifiable functions. SIAM Journal on Optimization 18 (2), pp. 556–572. Cited by: §1.
Bolte and Pauwels (2021) J. Bolte and E. Pauwels Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming 188 (1), pp. 19–51. Cited by: §1, §1, §2.1, §2.1, §2.1, §3.1.
Braun et al. (2017) G. Braun, C. Guzmán, and S. Pokutta Lower bounds on the oracle complexity of nonsmooth convex optimization via information theory. IEEE Transactions on Information Theory 63 (7), pp. 4709–4724. Cited by: §1, §2.3, §3.3, §3.3.
Braun et al. (2024) G. Braun, C. Guzmán, and S. Pokutta Corrections to “lower bounds on the oracle complexity of nonsmooth convex optimization via information theory”. IEEE Transactions on Information Theory 70 (7), pp. 5408–5409. Cited by: §1.
Carmon et al. (2020) Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford Lower bounds for finding stationary points I. Mathematical Programming 184 (1-2), pp. 71–120. Cited by: §1.
Carmon et al. (2021) Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford Lower bounds for finding stationary points II: first-order methods. Mathematical Programming 185 (1-2), pp. 315–355. Cited by: §1.
Cesa-Bianchi and Lugosi (2006) N. Cesa-Bianchi and G. Lugosi Prediction, learning, and games. Vol. 1, Cambridge University Press. Cited by: §1, §3.2.
Chang et al. (2025) D. Chang, Y. Liu, and G. Yuan On the convergence of Muon and beyond. ArXiv Preprint: 2509.15816. Cited by: §1, §1.
Clarke (1990) F. H. Clarke Optimization and nonsmooth analysis. SIAM. Cited by: §1, §2.1, §2.1.
Cutkosky and Mehta (2020) A. Cutkosky and H. Mehta Momentum improves normalized SGD. In ICML, pp. 2260–2268. Cited by: §1.
Davis et al. (2020) D. Davis, D. Drusvyatskiy, S. Kakade, and J. D. Lee Stochastic subgradient method converges on tame functions. Foundations of Computational Mathematics 20 (1), pp. 119–154. Cited by: §1, §2.1.
Davis and Drusvyatskiy (2025) D. Davis and D. Drusvyatskiy When do spectral gradient updates help in deep learning?. ArXiv Preprint: 2512.04299. Cited by: §1.
Demidovich et al. (2023) Y. Demidovich, G. Malinovsky, I. Sokolov, and P. Richtárik A guide through the zoo of biased SGD. In NeurIPS, pp. 23158–23171. Cited by: §1.
Diao et al. (2025) S. Diao, Y. Yang, Y. Fu, X. Dong, D. Su, M. Kliegl, Z. Chen, P. Belcak, Y. Suhara, H. Yin, M. Patwary, Y. C. Lin, J. Kautz, and P. Molchano Nemotron-CLIMB: clustering-based iterative data mixture bootstrapping for language model pre-training. In NeurIPS Datasets and Benchmarks Track, External Links: Link Cited by: §4.4.
Ding and Toh (2025) K. Ding and K-C. Toh Stochastic Bregman subgradient methods for nonsmooth nonconvex optimization problems. Journal of Optimization Theory and Applications 206 (3), pp. 67. Cited by: §1.
Ding et al. (2025) K. Ding, N. Xiao, and K-C. Toh Adam-family methods with decoupled weight decay in deep learning. Transactions on Machine Learning Research (TMLR). Note: External Links: ISSN 2835-8856, Link Cited by: §1, §2.1.
Duchi et al. (2011) J. Duchi, E. Hazan, and Y. Singer Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research 12 (7), pp. 2121–2159. Cited by: §1.
Gadat et al. (2018) S. Gadat, F. Panloup, and S. Saadane Stochastic heavy ball. Electronic Journal of Statistics 12 (1), pp. 461–529. External Links: Document Cited by: §1.
Gonon et al. (2026) A. Gonon, A-A. Muşat, and N. Boumal Insights on Muon from simple quadratics. ArXiv Preprint: 2602.11948. Cited by: §4.2, §4.2.
Goodfellow et al. (2016) I. Goodfellow, Y. Bengio, and A. Courville Deep learning. MIT Press. External Links: Link Cited by: §2.1.
Gruntkowska et al. (2026) K. Gruntkowska, A. Gaponov, Z. Tovmasyan, and P. Richtárik Error feedback for Muon and friends. In ICLR, External Links: Link Cited by: §1.
Higham (2008) N. J. Higham Functions of matrices. Society for Industrial and Applied Mathematics. External Links: Document Cited by: §2.2.
Hu et al. (2025) X. Hu, N. Xiao, X. Liu, and K-C. Toh Learning-rate-free momentum SGD with reshuffling converges in nonsmooth nonconvex optimization. Journal of Scientific Computing 102 (3), pp. 85. Cited by: §1.
Jaggi (2013) M. Jaggi Revisiting Frank–Wolfe: projection-free sparse convex optimization. In ICML, Vol. 28, pp. 427–435. Cited by: §2.3.
Jiang et al. (2026) R. Jiang, Z. Mhammedi, M. Mohri, and A. Mokhtari Adaptive matrix online learning through smoothing with guarantees for nonsmooth nonconvex optimization. In COLT, pp. 3782–3824. Cited by: §1.
Jordan et al. (2024) K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein Muon: an optimizer for hidden layers in neural networks. External Links: Link Cited by: §1, §1, §2.2, §2.3.
Jordan (2024) K. Jordan 94% on CIFAR-10 in 3.29 seconds on a single GPU. ArXiv Preprint: 2404.00498. Cited by: §4.3.
Karimireddy et al. (2019) S. P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi Error feedback fixes signSGD and other gradient compression schemes. In ICML, pp. 3252–3261. Cited by: §1.
Karpathy (2025) A. Karpathy NanoChat: the best ChatGPT that $100 can buy. GitHub. External Links: Link Cited by: §4.4.
Khona et al. (2026) M. Khona, A. Vavre, B. Wang, D. Fu, H. Wu, M. Chrzanowski, B. Catanzaro, D. Mudigere, J. Pool, M. Lightstone, M. Shoeybi, M. Patwary, N. Tajbakhsh, and T. Blankevoort SOAP, Muon, and beyond: pushing LLM pretraining scales. ArXiv Preprint: 2607.20548. Cited by: §1.
Kim and Oh (2026) G. Y. Kim and M-h. Oh Convergence of Muon with Newton-Schulz. In ICLR, External Links: Link Cited by: §1, §1.
Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In ICLR, External Links: Link Cited by: §1.
Kornowski and Shamir (2022) G. Kornowski and O. Shamir Oracle complexity in nonsmooth nonconvex optimization. The Journal of Machine Learning Research 23 (314), pp. 1–44. Cited by: §1.
Kovalev (2025) D. Kovalev Understanding gradient orthogonalization for deep learning via non-Euclidean trust-region optimization. ArXiv Preprint: 2503.12645. Cited by: §1.
Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. Technical Report Department of Computer Science, University of Toronto. External Links: Link Cited by: §4.3.
Large et al. (2024) T. Large, Y. Liu, M. Huh, P. Isola, H. Bahng, and J. Bernstein Scalable optimization in the modular norm. In NeurIPS, pp. 73501–73548. Cited by: §1.
Lewis and Tian (2021) A. S. Lewis and T. Tian The structure of conservative gradient fields. SIAM Journal on Optimization 31 (3), pp. 2080–2083. Cited by: §1.
Li and Hong (2025) J. Li and M. Hong A note on the convergence of Muon. ArXiv Preprint: 2502.02900. Cited by: §1.
Li et al. (2026) Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao NorMuon: making Muon more efficient and scalable. In ICML, External Links: Link Cited by: §1.
Liu et al. (2025) J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, et al. Muon is scalable for LLM training. ArXiv Preprint: 2502.16982. Cited by: §1, §1.
Liu et al. (2026) Z. Liu, R. Zhang, Z. Wang, Y. Zhao, Y. Su, Z. Yang, and Z. Zhang Muon ${}^{2}$ : boosting Muon via adaptive second-moment preconditioning. ArXiv Preprint: 2604.09967. Cited by: §1.
Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In ICLR, External Links: Link Cited by: §4.4.
Nemirovski and Yudin (1983) A. S. Nemirovski and D. B. Yudin Problem complexity and method efficiency in optimization. J. Wiley. Cited by: §1, §2.3.
Nesterov (1983) Y. E. Nesterov A method of solving a convex programming problem with convergence rate $O(1/k^{2})$ . In Doklady Akademii Nauk, pp. 543–547. Cited by: §1.
Nesterov (2018) Y. Nesterov Lectures on convex optimization. Vol. 137, Springer. Cited by: §1.
Parshakova et al. (2026) T. Parshakova, A. Khaled, M. Crawshaw, G. Garrigos, and R. M. Gower Muon does not converge on convex Lipschitz functions. ArXiv Preprint: 2605.08980. Cited by: §1, §1, §3.1, §3.2, Abstract.
Pethick et al. (2025) T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher Training deep learning models with norm-constrained LMOs. In ICML, pp. 49069–49104. Cited by: §1, §2.2.
Polyak (1964) B. T. Polyak Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4 (5), pp. 1–17. Cited by: §1.
Qian and Richtárik (2026) X. Qian and P. Richtárik Convergence analysis of Muon-type methods with inexact LMO in the degenerate case. ArXiv Preprint: 2606.21581. Cited by: §1.
Robbins and Monro (1951) H. Robbins and S. Monro A stochastic approximation method. The Annals of Mathematical Statistics, pp. 400–407. Cited by: §1.
Sato et al. (2025) N. Sato, H. Naganuma, and H. Iiduka Convergence bound and critical batch size of Muon optimizer. ArXiv Preprint: 2507.01598. Cited by: §1.
Sfyraki and Wang (2025) M-E. Sfyraki and J-K. Wang Lions and Muons: optimization via stochastic Frank-Wolfe. ArXiv Preprint: 2506.04192. Cited by: §1.
Shen et al. (2025) W. Shen, R. Huang, M. Huang, C. Shen, and J. Zhang On the convergence analysis of Muon. ArXiv Preprint: 2505.23737. Cited by: §1.
Shulgin et al. (2026) E. Shulgin, S. AlRashed, P. Richtárik, and F. Orabona Beyond the ideal: analyzing the inexact Muon update. In AISTATS, External Links: Link Cited by: §1, §1.
Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu RoFormer: enhanced transformer with rotary position embedding. Neurocomputing 568, pp. 127063. Cited by: §4.4.
Su (2025) W. Su Isotropic curvature model for understanding deep learning optimization: is gradient orthogonalization optimal?. ArXiv Preprint: 2511.00674. Cited by: §1.
Sutskever et al. (2013) I. Sutskever, J. Martens, G. Dahl, and G. E. Hinton On the importance of initialization and momentum in deep learning. In ICML, pp. 1139–1147. Cited by: §1.
Team (2025) K. Team Kimi k2: open agentic intelligence. ArXiv Preprint: 2507.20534. Cited by: §1.
Tieleman and Hinton (2012) T. Tieleman and G. E. Hinton Neural networks for machine learning, Lecture 6.5 - RMSProp. Note: COURSERA: Neural Networks for Machine Learning Cited by: §1.
Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In NIPS, pp. 5998–6008. Cited by: §2.1, §4.4.
Vershynin (2018) R. Vershynin High-dimensional probability: an introduction with applications in data science. Vol. 47, Cambridge University Press. Cited by: §3.3.
Wang (2005) X. Wang Subdifferentiability of real functions. Real Analysis Exchange 30 (1), pp. 137–171. Cited by: §1.
Xiao et al. (2024) N. Xiao, X. Hu, X. Liu, and K-C. Toh Adam-family methods for nonsmooth optimization with convergence guarantees. The Journal of Machine Learning Research 25 (48), pp. 1–53. Cited by: §1, §1.
Yao (1977) A. C. Yao Probabilistic computations: toward a unified measure of complexity. In 18th Annual Symposium on Foundations of Computer Science (sfcs 1977), pp. 222–227. Cited by: §2.3.
Zhang et al. (2026) J. Zhang, N. Amsel, B. Chen, and T. Dao Gram Newton-Schulz. External Links: Link Cited by: §1.
Zhang et al. (2020) J. Zhang, H. Lin, S. Jegelka, S. Sra, and A. Jadbabaie Complexity of finding stationary points of nonconvex nonsmooth functions. In ICML, pp. 11173–11182. Cited by: §1.
Zhang and Lin (2026) J. Zhang and T. Lin Scale-invariant neural network optimization: norm geometry and heavy-tailed noise. ArXiv Preprint: 2605.18528. Cited by: §1.
Zhuang et al. (2026) X. Zhuang, P. Ouyang, Y. Li, J. Shi, Y. Chen, S. Liu, Y. Qian, W. Liu, H. Zhang, and I. Razzak AMO: adaptive Muon orthogonalization. ArXiv Preprint: 2605.17806. Cited by: §1.