Matrix-free stochastic calculation of operator norms without using adjoints
Extracted source text · arXiv 2410.08297
\documentclass{article}
\usepackage{amsmath,amsthm,amsfonts,amssymb}
\usepackage{mathtools}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{proposition}[theorem]{Proposition}
\theoremstyle{definition}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{assumption}[theorem]{Assumption}
%\usepackage[style=numeric,maxbibnames=99]{biblatex}
%\addbibresource{literature.bib}
\usepackage{enumerate}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage{hyperref}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{pgfplots}
\usepackage{tikz}
\usepackage{tikz-3dplot}
\usetikzlibrary{3d,calc}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{listings}
% Code to typeset python listings (taken from https://tex.stackexchange.com/a/83883/7573)
% Default fixed font does not support bold face
\DeclareFixedFont{\ttb}{T1}{txtt}{bx}{n}{12} % for bold
\DeclareFixedFont{\ttm}{T1}{txtt}{m}{n}{12} % for normal
% Style for code listings
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\lstdefinestyle{mystyle}{
%backgroundcolor=\color{backcolour},
commentstyle=\color{codegreen},
keywordstyle=\color{magenta},
numberstyle=\tiny\color{codegray},
stringstyle=\color{codepurple},
basicstyle=\ttfamily\footnotesize,
breakatwhitespace=false,
breaklines=true,
captionpos=b,
keepspaces=true,
%numbers=left,
%numbersep=5pt,
showspaces=false,
showstringspaces=false,
showtabs=false,
tabsize=2
}
\lstset{style=mystyle}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\sphere}{\mathbb{S}}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\cD}{\mathcal{D}}
\newcommand{\cN}{\mathcal{N}}
\newcommand{\bigO}{\mathcal{O}}
\newcommand{\mgeq}{\succcurlyeq}
\newcommand{\scp}[2]{\left\langle{#1}, {#2}\right\rangle}
\DeclarePairedDelimiter{\abs}{\vert}{\vert}
\DeclarePairedDelimiter{\norm}{\|}{\|}
\DeclareMathOperator{\tr}{tr}
\DeclareMathOperator{\Var}{Var}
\DeclareMathOperator{\prox}{prox}
\DeclareMathOperator{\Unif}{Unif}
\DeclareMathOperator{\range}{range}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\diag}{diag}
\let\dim\relax
\DeclareMathOperator{\dim}{dim}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\dirk}[1]{{\color{brown}\textbf{Dirk:} #1}}
\newcommand{\jonas}[1]{{\color{cyan}\textbf{Jonas:} #1}}
\newcommand{\Max}[1]{{\color{blue}\textbf{Max:} #1}}
\newcommand{\checklater}[1]{{\color{red} #1}}
\title{Matrix-free stochastic calculation of operator norms without using adjoints}
\author{Jonas Bresch\thanks{Institute of Mathematics, Technical University Berlin, Straße des 17. Juni 136, 10623 Berlin, Germany,
\href{mailto:bresch@math.tu-berlin.de}{bresch@math.tu-berlin.de}. \url{https://www.tu.berlin/imageanalysis}.} \and Dirk A. Lorenz\thanks{Center for Industrial Mathematics, Fachbereich 3, University of Bremen, Postfach 33 04 40, 28334 Bremen, Germany, \href{mailto:d.lorenz@uni-bremen.de}{d.lorenz@uni-bremen.de}.} \and Felix Schneppe\thanks{Center for Industrial Mathematics, Fachbereich 3, University of Bremen, Postfach 33 04 40, 28334 Bremen, Germany, \href{mailto:schneppe@uni-bremen.de}{schneppe@uni-bremen.de}.} \and Maximilian Winkler\thanks{Center for Industrial Mathematics, Fachbereich 3, University of Bremen, Postfach 33 04 40, 28334 Bremen, Germany, \href{mailto:maxwin@uni-bremen.de}{maxwin@uni-bremen.de}.}}
\begin{document}
\maketitle
\begin{abstract}
This paper considers the problem of computing the operator norm
of a linear map between finite dimensional Hilbert spaces
when only evaluations of the linear map are available
and under restrictive storage assumptions.
We propose a stochastic method of random search type
to maximize the Rayleigh quotient
and employ an exact line search in the random search directions.
Moreover,
we show that the proposed algorithm converges
to the global maximum (the operator norm) almost surely,
show a sublinear convergence behavior for the corresponding
eigenvector and eigenvalue equation,
and illustrate the performance of the method with numerical experiments.
\end{abstract}
\noindent\textbf{AMS Classification:}
65F35, % Numerical computation of matrix norms, conditioning, scaling
15A60, % Norms of matrices, numerical range, applications of functional analysis to matrix theory
68W20% Randomized algorithms
\noindent\textbf{Keywords:}
spectral norm, operator norm,
stochastic algorithm,
stochastic gradient method
\section{Introduction}
\label{sec:intro}
In this paper, we revisit the problem of the computation of the operator norm $\norm{A}$ of a linear map $A$ between Hilbert spaces of dimension $d$ and $m$, respectively.
We reconsider this problem under restrictive conditions for which we could not find a satisfactory solution in the literature.
First, we assume that the map $A$ is only available through a black-box oracle that returns $Av$ for a given input $v$,
e.g. there is precompiled code available which evaluates a linear map.
Moreover, we consider that it is not possible to store a large number of vectors (of either the input or the output dimension),
which means that we assume that the method has a storage requirement of $\bigO(\max(m,d))$.
Finally, we are interested in a method that is able to approximate the value $\norm{A}$ to any accuracy and also is able to make further incremental improvements if the output at some stage is not good enough.
\subsection{Motivation}
\label{sec:motivation}
Operator norms are needed in many places, e.g. to guarantee convergence of optimization methods
(e.g. the Lipschitz constant of the objective $\norm{Ax-y}^{2}$ is $\norm{A}^{2}$ and gradient descent or the forward-backward method need a stepsize $0<\tau<2/\norm{A}^{2}$ for convergence \cite{ryu2022large},
or the Chambolle-Pock method \cite{chambolle2011first} to minimize $f(x) + g(Ax)$ for convex, closed and proper functions $f$ and $g$ and a linear map $A$ needs two stepsizes $\sigma,\tau$ that fulfill $\sigma\tau\leq 1/\norm{A}^{2}$ \cite{bredies2022degenerate}).
When the operator $A$ is not given as a matrix, one usually resorts to the power iteration to compute $\norm{A}$ and when $A$ is not self-adjoint, one computes $\sqrt{\norm{A^{*}A}}$ by power iteration.
However, this poses an additional challenge: One needs the adjoint of $A$ as well.
If $A$ comes from the discretization of some operator, the discretization of the adjoint operator is usually not the adjoint of the discretized operator. This is discussed for numerical methods for partial differential equations,
e.g., in~\cite{fikl2016comprehensive} or \cite[Section 2.9]{gunzburger2002perspectives}.
The same problem also appears in computerized tomography where forward back-projector pairs are frequently not adjoint, see e.g.~\cite{bredies2021convergence,huber2025convergence}.\footnote{If the adjoint is not available or inexact, the optimization methods can not be applied or are not guaranteed to converge. This is another problem and discussed, e.g. in\cite{lorenz2023chambolle,lorenz2023randomdescent,chouzenoux2023convergence,Chouzenout2021pgm-adjoint}.}
\paragraph{A curious toy example: Rotations.}
We provide a very simple illustration of the problem: Consider a function on the two dimensional disk $\mathbb{D} \coloneqq \{ x\in\RR^{2}\ \mid\ \norm{x}_{2}\leq 1 \}$ and the operator $A_{\alpha}$ that takes a function $f:\mathbb{D}\to \RR$ and maps it to the rotated function $A_{\alpha}f(x) \coloneqq f(R_{\alpha}x)$ where $R_{\alpha} \coloneqq \left(
\begin{smallmatrix}
\cos(\alpha) & \sin(\alpha)\\-\sin(\alpha) & \cos(\alpha)
\end{smallmatrix}\right)
$. The adjoint operator is $A_{\alpha}^{*} = A_{-\alpha}$ and is also the inverse (the operator is orthogonal). When applied to functions that are given on a grid, e.g. an image, this rotation uses interpolation and we can neither expect that the inverse rotation is actually the inverse, nor that it is the adjoint.
This does affect the value of the operator norm: While the continuous operator fulfills $\norm{A_{\alpha}}=1$ for all $\alpha$, it is not clear if this is also fulfilled by the discretized map. Let us denote the discretized map with $\mathbf{A}_{\alpha}$. This is implemented in Python in \texttt{scipy.ndimage}, e.g. as
\begin{center}
\texttt{rotate(x, angle, reshape=False, order=order)}
\end{center}
where \texttt{x} is the input array (and we assume that it is square and zero outside of a centered circle to avoid additional boundary effects) and \texttt{order} is the order of the interpolation routine (nearest neighbor, bilinear,\dots).
Table~\ref{tab:table-norm-rotation} reports results where we computed the norm of $\mathbf{A}_{\alpha}$ by computing $\sqrt{\norm{\mathbf{A}_{-\alpha}\mathbf{A}_{\alpha}}}$ with the power iteration and $\norm{\mathbf{A}_{\alpha}}$ with the method we propose in this paper for some values of $n$ and $\alpha$ and bicubic interpolation.
Note that the values larger than $1$ for the operator norm are not numerical errors: Our method constructs an array $\mathbf{v}$ with norm $1$ such that $\norm{\mathbf{A}_{\alpha}\mathbf{v}}$ has the values in the last column.
Notably, the error in the norm can be as large as $17\%$ for $\alpha = 45^{\circ}$.
Even more drastically we get a norm of $1.4142$ for every angle different from $0^{\circ}$ and $90^{\circ}$ and any $n$ for the nearest neighbor interpolation.
\begin{table}[htb]
\centering
\begin{tabular}{rrrr}
\toprule
$n$ & $\alpha$ & $\sqrt{\norm{\mathbf{A}_{-\alpha}\mathbf{A}_{\alpha}}}$ & $\norm{\mathbf{A}_{\alpha}}$\\\midrule
25 & 10 & 0.99999 & 1.00463 \\
25 & 30 & 1.00002 & 1.05930 \\
25 & 45 & 0.99996 & 1.17387 \\
50 & 10 & 1.00080 & 0.99804 \\
50 & 30 & 0.99998 & 1.05314 \\
50 & 45 & 1.00000 & 1.16840 \\
\bottomrule
\end{tabular}
\caption{Numerical computation of operator norms for a rotation operation. The values in the column $\sqrt{\norm{\mathbf{A}_{-\alpha}\mathbf{A}_{\alpha}}}$ are obtained by power iteration while the values in the column $\norm{\mathbf{A}_{\alpha}}$ are computed with the method we propose in this paper.}
\label{tab:table-norm-rotation}
\end{table}
\subsection{Related work}
\label{sec:related-work}
Our restrictive assumptions rule out existing methods:
The storage requirement rules out the brute force approach of building a full matrix representation for the linear map by evaluating $Ae_{i}$ for all $i$ and then calculating the singular value decomposition of this representation.
Methods based on sketching of the map, as done in~\cite{martinsson2002randomized} for the Schatten $p$-norm for even $p$ or in \cite{magdon2011note} for the spectral norm (which is the Schatten $\infty$-norm),
lead to lower bounds with probabilistic guarantees,
but these depend on the size of the sketch and further improvements of the estimate need a refinement of the sketch and another computation of the norm of the sketched matrix.
In \cite{kong2017spectralestimation} the estimation of the spectrum of a data set is considered
and which includes the estimation of the operator norm as Schatten $\infty$-norm also as limit of the $k$-th moments.
From the main theorem (cf.~\cite[Thm.~1]{kong2017spectralestimation})
we would need the limit $k\to\infty$ of the moments,
where the estimated bound behaves super-exponential in $k$.
Choosing the moment just large enough might be not effective
and depends on the actual values of the singular values.
The work \cite{li2014sketching} also considers the estimation of the Schatten $\infty$-norm by linear and bilinear sketching.
Crucially, they show that any approximation of the spectral norm needs a sketch that is basically almost as large as the full matrix, i.e. of order $\Omega(n^{2})$.
%\dirk{Add discussion of paper mentioned by Alex Townend here.}
Moreover, sketching methods do not meet our storage requirements.
Another popular method for the computation of the spectral norm is the power iteration~\cite{kuczynski1992estimating} which, in its plain form, works for maps for which the input and output dimensions are equal.
To calculate the norm of a map with different input and output dimensions, one could in principle apply the power method to the map $A^{*}A$, however,
this needs that a routine that applies the adjoint map $A^{*}$ is available, which we do not assume here.
If the evaluation of $A$ is done by a computer program that has been written in a way such that automatic differentiation is applicable,
we could simply use that to calculate the derivative of the map $v\mapsto \norm{Av}^{2}$
(and we could even calculate $A^{*}w$ for some $w$ by applying automatic differentiation with respect to $v$ to the function $v\mapsto \scp{w}{Av}$).
However, this is not always the case, especially for precompiled code.
Moreover, it is known that the power method converges slowly if the spectral gap (i.e. the distance between the two largest eigenvalues) is small.
Other approaches to operator norms such as in \cite{kenney1998} (focusing on condition estimation) and \cite{dudley2022} (using Monte Carlo techniques for Schatten-$p$ norms) also only consider maps with equal input and output dimension.
Methods in numerical linear algebra that avoid using adjoints (or transposes) have been studied since at least the 1990s and are known as ``adjoint-free (or transpose-free) methods'', see~\cite{chan1998transpose} for a motivation to consider such methods.
Some examples are the transpose-free quasi minimal residual method (TFQMR) from~\cite{freund1993tfqmr} and Lanczos type methods \cite{brezinski1998transpose,chan1998transpose} which lead to the \emph{conjugate gradient squared method} (CGS)~\cite{sonneveld1989cgs} for the solution of linear systems, and for least squares problems there is the method of random descent from~\cite{lorenz2023randomdescent}.
The recent paper \cite{boulle2024operator} treated the problem of how to learn a linear map if only evaluations of the map are available and no queries of the adjoint are available.
In this article, we focus on the case where $A$ maps $\RR^{d}$ to $\RR^{m}$ and we work with the standard Euclidean inner product and norm. However, what we develop can be generalized to more general finite dimensional Hilbert spaces as well as to spaces over the complex numbers.
\subsection{Contribution}
\label{sec:contribution}
The contribution of this work is:
\begin{itemize}
\item We propose a stochastic algorithm that produces a sequence that converges to the operator norm
\begin{align*}
%\label{prob:OpNorm}
\max_{\norm{v}=1} \norm{A v}^2
\end{align*}
almost surely only using evaluations of the operator $A$, and calculation of norms and inner products.
\item Our method works for any linear map between finite dimensional Hilbert spaces, i.e. we do not need any assumptions on the dimension of the spaces or on the structure of the linear map.
\item The method has the minimal possible storage requirement $\bigO(\max(m,d))$, i.e., we only need to store two vectors of the input dimension $d$ and two vectors of the output dimension $m$ during the iteration.
\end{itemize}
Additionally, our method produces an approximation of a right singular vector for the largest singular value and can be extended to compute further right singular vectors. Curiously, the method also allows us to check if a map $A$ is orthogonal in the sense that $A^{*}A = cI$ for some constant $c \geq 0$, where $I$ denotes the identity on the input space of $A$.
\subsection{Notation}
\label{sec:notation}
We use $\norm{x}$ and $\scp{x}{y}$ for the standard Euclidean vector norm and inner product, respectively, $\norm{A}$ for the induced operator norm and $I_{d}$ for the identity map on $\RR^{d}$. With $\cN(\mu,\Sigma)$ we denote the normal distribution with mean $\mu$ and covariance $\Sigma$ and with $\sphere^{d-1} \coloneqq \{x\in\RR^{d}\mid \norm{x}=1\}$ we denote the unit sphere in $d$-dimensional space. With $\PP(B)$ we denote the probability of an event $B$ (and the underlying distribution will be clear from the context).
In this paper $\sigma_{k}$ will always denote the $k$-th largest singular value of the linear map induced by $A$, i.e. $\sigma_{1} = \sigma_{\max} = \norm{A}$. For a vector $v \in \RR^d$, we denote by $\{v\}^{\bot} \coloneqq \left\{ w \in \RR^d \mid \scp{w}{v}=0 \right\}$ the hyperplane orthogonal to $v$ and $B_{\varepsilon}(v) \coloneqq \left\{ w \in \RR^d \mid \norm{w-v}<\varepsilon \right\}$ is the $\varepsilon$-ball around $v$.
By $A^*$,
we denote the adjoint operator of $A$ defined by $\scp{x}{A^{*}y} = \scp{Ax}{y}$. We will occasionally abbreviate ``almost surely'' with a.s. which means that the described event will occur with probability one. By $\mathbf{1}$, we denote a vector of all ones (and the length will be clear from the context) and $\mathbbm{1}_{B}$ is used for the characteristic function of the set $B$.
\section{Adjoint free calculation of $\norm{A}$}
\label{sec:adjoint-free-A}
We propose a stochastic method to approximate the operator norm $\norm{A}$ for some $A$ that maps from $\RR^{d}$ to $\RR^{m}$. The method relies on the Rayleigh principle namely $\norm{A}^{2} = \max_{\norm{v}=1}\norm{Av}^{2}$. Within our assumptions, we can not apply a simple projected gradient ascent for this problem as this would require to compute the vector $A^{*}Av$ and we do not have access to the adjoint.
The method we propose is a random search method and operates fairly simple:
\begin{enumerate}
\item Initialize some $v^{0}\in\RR^{d}$ with $\norm{v^{0}}=1$ and set $k=0$.
\item Generate $y^{k}\sim \cN(0,I_{d})$.
\item Project $y^{k}$ onto the space orthogonal to $v_k$ and normalize it
\begin{align*}
x^{k} = \frac{y^{k} - \scp{y^{k}}{v^{k}}v^{k}}{\norm{y^{k} - \scp{y^{k}}{v^{k}}v^{k}}}
\end{align*}
to get a random normalized search direction perpendicular to $v^{k}$.
\item Calculate a stepsize $\tau_{k}$ by\footnote{It may happen that there exists no maximizer. However, we will argue that this case occurs with probability zero, see Proposition~\ref{prop:stepsize-formula} and Corollary~\ref{cor:no-sing-vectors-during-iteration}.}
\begin{align}\label{eq:argmax-tau}
\tau_{k}
\in
\argmax_{\tau}
\frac{\norm{Av^{k} + \tau Ax^{k}}^{2}}{\norm{v^{k}+\tau x_{k}}^{2}},
\end{align}
\item Update $v^{k+1} = \frac{v^{k}+\tau_kx^{k}}{\norm{v^{k}+\tau_{k}x^{k}}}$ and go to step 2.
\end{enumerate}
\begin{remark}[Normalization of $v^{k}$ and $x^{k}$]
\label{rem:norm_const}
By design we always have that $\norm{v^{k}} = \norm{x^{k}}=1$ and $\scp{v^{k}}{x^{k}} = 0$.
\end{remark}
\begin{remark}[Critical points]
By Rayleigh's principle, the critical points of the problem $\max_{\norm{v}=1}\norm{Av}$ are exactly the right singular vectors of $A$ and the values at these points are exactly the singular values.
More precisely, the intersection of the eigenspace of the largest (or smallest) singular value gives exactly the global maxima (or minima). All other eigenspaces lead to saddle points.
We will show in the course of the paper that the method we propose will (almost surely) avoid all saddle points and will converge (almost surely) to the global maximum.
\end{remark}
A particularity of this method is that the search direction is quite arbitrary,
but the stepsize is determined exactly (and note that it is allowed to be negative).
This is different from many other stochastic methods which estimate update directions and then take a step according to a fixed stepsize schedule.
In the following, we will show that this method is simple to execute (and especially discuss how~\eqref{eq:argmax-tau} is solved in practice) and indeed uses only evaluations of $A$.
We will also show that the method will produce a sequence $v^{k}$ such that $\norm{Av^{k}}$ converges to $\norm{A}$ almost surely.
\begin{remark}[Distribution of the $x^{k}$]\label{rem:distribution-x}
The random vector $x^{k}$ is distributed as follows:
Since an orthogonal projection of normally distributed random vectors is again normally distributed, the projection $y^{k } - \scp{y^{k}}{v^{k}}v^{k}$ is normally distributed on the tangent space on the unit sphere $\sphere^{d-1}$ at $v^{k}$. Since $x^{k}$ is the result of the normalization of this vector, $x^{k}$ is uniformly distributed on the unit sphere in this tangent space, i.e. on a rotated version of $\sphere^{d-2}$.
\end{remark}
\subsection{Details of the algorithm}
\label{sec:details}
We start with a discussion for the computation of the stepsize $\tau_{k}$ from \eqref{eq:argmax-tau}.
Utilizing Remark~\ref{rem:norm_const}, we observe that
$\norm{v^{k}+\tau x^{k}}^{2}
= \norm{v^{k}}^{2} + 2\tau\scp{v^{k}}{x^{k}} + \tau^{2}\norm{x^{k}}^{2} = 1+\tau^{2}$
and hence,
the objective in the minimization problem~\eqref{eq:argmax-tau} in the $k$th iteration reads as
\begin{align}\label{eq:h}
h_k(\tau) \coloneqq \frac{\norm{Av^{k} + \tau Ax^{k}}^{2}}{1+\tau^{2}}
\end{align}
Its derivative is
\begin{align*}
h_k'(\tau)
& = \frac{(2\scp{Av^{k}}{Ax^{k}} + 2\tau\norm{Ax^{k}}^{2})(1+\tau^{2})}{(1+\tau^{2})^{2}} \\
& \qquad - \frac{(\norm{Av^{k}}^{2} + 2\tau\scp{Av^{k}}{Ax^{k}} + \tau^{2}\norm{Ax^{k}}^{2})\cdot2\tau}{(1+\tau^{2})^{2}}\\
& = 2\frac{\scp{Av^{k}}{Ax^{k}} + (\norm{Ax^{k}}^{2}- \norm{Av^{k}}^{2})\tau - \scp{Av^{k}}{Ax^{k}}\tau^{2}}{(1+\tau^{2})^{2}}\\
& = 2\frac{a_{k} + b_{k}\tau - a_{k}\tau^{2}}{(1+\tau^{2})^{2}}
\end{align*}
where we defined
\begin{align}
\label{eq:def-ak-bk}
a_{k} \coloneqq \scp{Av^{k}}{Ax^{k}}\quad \text{and}\quad b_{k}\coloneqq \norm{Ax^{k}}^{2} - \norm{Av^{k}}^{2}.
\end{align}
The shape of the objective $h_k$ depends on the signs of $a_{k}$ and $b_{k}$ and we illustrate the cases in Figure~\ref{fig:shape-h}.
It is clear that
\begin{align*}
h_k(0) = \norm{Av^{k}}^{2}\quad \text{and}\quad
\lim_{\tau\to\pm\infty} h_k(\tau) = \norm{Ax^{k}}^{2}.
\end{align*}
% settings for the plots of h
\pgfplotsset{width=5cm,height=4cm,
xlabel style={xlabel=$\tau$},
samples = 50,
every axis plot post/.append style={orange,mark=none},
every axis post/.append style={xmin=-5, xmax=5, ymin=0, ymax = 1.9, axis lines=middle, xtick=\empty},
every tick label/.append style={font=\tiny}
}
\begin{figure}[htb]
\centering
\begin{tabular}{ccc}
\begin{tikzpicture}
\begin{axis}[ytick={1,1.5},yticklabels={$\norm{Av^{k}}^{2}$,$\norm{Ax^{k}}^{2}$}]
\addplot{(1+x*1+1.5*x*x)/(1+x*x)};
\addplot[thin,dotted] {1.5};
\end{axis}
\end{tikzpicture}&
\begin{tikzpicture}
\begin{axis}[ytick={1,1.5},yticklabels={$\norm{Av^{k}}^{2}$,$\norm{Ax^{k}}^{2}$}]
\addplot{(1-x*1+1.5*x*x)/(1+x*x)};
\addplot[thin,dotted] {1.5};
\end{axis}
\end{tikzpicture}&
\begin{tikzpicture}
\begin{axis}[ytick={1,1.5},yticklabels={$\norm{Av^{k}}^{2}$,$\norm{Ax^{k}}^{2}$}]
\addplot{(1+1.5*x*x)/(1+x*x)};
\addplot[thin,dotted] {1.5};
\end{axis}
\end{tikzpicture}\\\bigskip
$a_{k}>0$, $b_{k}>0$ &
$a_{k}<0$, $b_{k}>0$ &
$a_{k}=0$, $b_{k}>0$\\
\begin{tikzpicture}
\begin{axis}[ytick={1,1.5},yticklabels={$\norm{Ax^{k}}^{2}$,$\norm{Av^{k}}^{2}$}]
\addplot{(1.5+x*1+1*x*x)/(1+x*x)};
\addplot[thin,dotted] {1};
\end{axis}
\end{tikzpicture}&
\begin{tikzpicture}
\begin{axis}[ytick={1,1.5},yticklabels={$\norm{Ax^{k}}^{2}$,$\norm{Av^{k}}^{2}$}]
\addplot{(1.5-x*1+1*x*x)/(1+x*x)};
\addplot[thin,dotted] {1};
\end{axis}
\end{tikzpicture}&
\begin{tikzpicture}
\begin{axis}[ytick={1,1.5},yticklabels={$\norm{Ax^{k}}^{2}$,$\norm{Av^{k}}^{2}$}]
\addplot{(1.5+1*x*x)/(1+x*x)};
\addplot[thin,dotted] {1};
\end{axis}
\end{tikzpicture}\\
$a_{k}>0$, $b_{k}<0$ &
$a_{k}<0$, $b_{k}<0$ &
$a_{k}=0$, $b_{k}<0$\\
\end{tabular}
\caption{Shape of $h_k$ from~\eqref{eq:h} for different cases of the signs of $a_k$ and $b_k$.}
\label{fig:shape-h}
\end{figure}
\begin{proposition}
\label{prop:stepsize-formula}
Let $h_k$ be defined by~\eqref{eq:h} and $a_{k}$ and $b_{k}$ are defined by~\eqref{eq:def-ak-bk}.
If $a_k \neq 0$,
the $\tau_{k}$ that solves~\eqref{eq:argmax-tau} is unique and given by
\begin{align*}
\tau_{k}
=
\sign(a_k) \left(\tfrac{b_{k}}{2\abs{a_{k}}} + \sqrt{\tfrac{b_{k}^{2}}{4a_{k}^{2}}+1}\right).
\end{align*}
If $a_k = 0$, we have: $h_k$ attains its maximum at $\tau = 0$ if $b_{k}<0$, does not attain its maximum if $b_{k}>0$, and $h$ is constant if $b_{k}=0$.
\end{proposition}
\begin{proof}
The local extrema of $h_k$ are at the roots of the enumerator of $h_k'$ which are
\begin{align*}
\tau_{\pm}
\coloneqq
\tfrac{b_{k}}{2a_{k}} \pm \sqrt{\tfrac{b_k^2}{4a_k^2}+1}
\end{align*}
and it holds that $\tau_{+}>0$ and $\tau_{-}<0$. These are the only local extrema.
If $a_k = \scp{Av^{k}}{Ax^{k}}>0$ we have $h_k'(0)>0$ and one can check that $\tau_{+}$
is the global maximizer of $h_k$.
If $a_k < 0$, we have $h_k'(0)<0$ and one can check that
the global maximum of $h_k$ is attained at
\begin{equation*}
\tau_{-}
=
\tfrac{b_{k}}{2a_{k}} - \sqrt{\tfrac{b_k^2}{4a_k^2}+1}
= - \left( \tfrac{b_{k}}{2 |a_{k}|} + \sqrt{\tfrac{b_k^2}{4 a_k^2}+1} \right).
\qedhere
\end{equation*}
\end{proof}
With the previous result concerning the optimal stepsize,
we give the full algorithm in detail as Algorithm~\ref{alg:mafno-orth-exact}.
Note that the algorithm would break down
if $a_{k}=0$ since we would have to divide by zero.
We will show that this will almost surely not occur,
but before we do so,
we provide few remarks regarding the algorithm.
\begin{remark}[Convergence of the sequence $\norm{Av^{k}}$]\label{rem:conv-normAv}
As long as $a_{k}\neq0$, the algorithm leads to a sequence $\norm{Av^{k}}$ which is strictly increasing. Since we have $\norm{Av^{k}}\leq \norm{A}$, the sequence $\norm{Av^{k}}$ is convergence as long as we have $a_{k}\neq 0$ throughout the iteration.
\end{remark}
\begin{remark}[Algorithm~\ref{alg:mafno-orth-exact} is a stochastic projected gradient method]
Although we sample $x^{k}$ uniformly from the unit directions orthogonal to $v^{k}$, the method is still a stochastic gradient method in some sense. This is because the stepsize $\tau_{k}$ is positive for $a_k = \scp{Av^k}{Ax^k}>0$ and negative for $a_{k}<0$, and thus, we effectively move in direction $\sign(a_{k})x_{k}$. This means that the direction in which the algorithm moves is uniformly distributed on a half-sphere in the space orthogonal to $v^{k}$.
This half-sphere is defined by the inequality $a_{k}>0$ which is $0<\scp{Av^{k}}{Ax^{k}} = \scp{A^{*}Av^{k}}{x^{k}}$. Due to the symmetry of the distribution, the expected direction $\EE(\sign(a_{k})x^{k})$ is a positive multiple of the projection of $A^{*}Av^{k}$ on the space orthogonal to $v^{k}$. We conclude that Algorithm~\ref{alg:mafno-orth-exact} is a stochastic ascent algorithm in the sense that the direction is a multiple of the projection on the gradient $2A^{*}Av^{k}$ of the objective $\norm{Av}^{2}$.
\end{remark}
\begin{algorithm}[tb]
\caption{Matrix- and adjoint free operator norms}\label{alg:mafno-orth-exact}
\begin{algorithmic}[1]
\State Initialize $v^{0}\in\RR^{d}$ with $\norm{v^{0}}=1$
\For{$k=0,1,2,\dots$}
\State Sample $y^{k}\sim \cN(0,I_{d})$
\State Project
\begin{align*}
x^{k} = \frac{y^{k} - \scp{y^{k}}{v^{k}}v^{k}}{\norm{y^{k} - \scp{y^{k}}{v^{k}}v^{k}}}
\end{align*}
and calculate
\begin{align*}
a_{k} & = \scp{Av^{k}}{Ax^{k}},& b_{k} & = \norm{Ax^k}^2 - \norm{Av^{k}}^2
\end{align*}
\State Calculate stepsize $\tau_{k}$ as
\begin{align*}
\tau_{k} = \sign(a_{k})\left(\tfrac{b_{k}}{2\abs{a_{k}}} + \sqrt{\tfrac{b_{k}^{2}}{4a_{k}^{2}}+1}\right)
\end{align*}
\State Update
\begin{align}
\label{eq:v-update-step}
v^{k+1}
=
\frac{v^{k} + \tau_{k}x_{k}}{\sqrt{1+\tau_{k}^{2}}}
\end{align}
%\State \textbf{end if}
\EndFor
\State \textbf{return} estimate of $\norm{A}$:
\[
\norm{Av^{k}}
\]
\end{algorithmic}
\end{algorithm}
Now
we show that $a_{k}=0$ only occurs if $v^{k}$ is a right singular vector
and that this will (almost surely) never occur during the algorithm.
\begin{lemma}
\label{lem:a=0-at-sing-vecs}
If $v^{k}$ is not a right singular vector of $A$ and $x^{k}$ is sampled according to step 3 and 4 of Algorithm~\ref{alg:mafno-orth-exact}, then $a_{k} = \scp{Av^{k}}{Ax^{k}} \neq 0$ almost surely.
\end{lemma}
\begin{proof}
By construction it holds that $\scp{x^{k}}{v^{k}}=0$ and if we also have that $a_{k}=0$, then it holds that $0 = \scp{Av^{k}}{Ax^{k}} = \scp{A^{*}Av^{k}}{x^{k}}$. Hence, $a_{k}=0$ only holds if $x^{k}$ is orthogonal to $v^{k}$ and to $A^{*}Av^{k}$ and since $v^{k}$ is not an eigenvector of $A^{*}A$, this means that the vector $x^{k}$ is in a subspace of dimension at most $d-2$. Recalling Remark~\ref{rem:distribution-x} on the distribution of $x^{k}$, we see this restricts $x^{k}$ to a subset of measure zero on the unit sphere in the $d-1$-dimensional tangent space of $\sphere^{d-1}$ at $v^{k}$ and hence, this happens with probability zero.
\end{proof}
By the previous lemma,
we know that $a_{k}=0$ only happens (a.s.)
if $v^{k}$ is a right singular vector.
Now we show that if $v^{k}$ is not a right singular vector,
the same is true (a.s.) for $v^{k+1}$.
We start with this observation:
\begin{lemma}
\label{lem:dimension-directions}
Let $A\in\RR^{m\times d}$, $\sigma^2$ be an eigenvalue of $A^{*}A$ with geometric multiplicity $r<d$, $v\in\RR^{d}$ be no eigenvector for the eigenvalue $\sigma^{2}$ and denote by $E = \left\{ x \mid A^{*}Ax = \sigma^2 x\right\}$ the respective eigenspace. Then it holds that the set of directions $x\in \{v\}^{\bot}$ for which $v+x\in E$ holds is an affine subspace of dimension $r$ or $r-1$.
\end{lemma}
\begin{proof}
Since $\dim(E) = r$ we can write $E = \left\{ x \mid Cx = 0 \right\}$ with $C\in \RR^{d-r\times d}$ having rank $d-r$. We assume without loss of generality (by applying a suitable rotation) that $v = e_{1} = (1,0,\dots,0)^{*}$, hence $x\in \{v\}^{\bot}$ means $x = (0,x')$ with $x' \in \RR^{d-1}$. We have that $v+x\in E$ exactly if $C(v+x)=0$ which is equivalent to $Cx = -Cv$. Denoting $C = \left[ c_1, C' \right]$ with $c_{1}\in\RR^{d-r}$ and $C' \in \RR^{d-r\times d-1}$ we can rewrite the latter system of equation as $C' x' = -c_{1}$. The matrix $C'$ has rank either $d-r$ or $d-r-1$ and hence, this system has an affine solution space of dimension either $d-1 - (d-r) = r-1$ or $d-1 - (d-r-1) = r$.
\end{proof}
\begin{corollary}
\label{cor:no-sing-vectors-during-iteration}
Assume that all singular values of $A\in\RR^{m\times d}$ have multiplicities less than $d-1$. Then it holds that if $v^{k}$ is not a right singular vector, then $v^{k+1}$ is almost surely also not a right singular vector.
Consequently, we have $a_{k}\neq 0$ (almost surely) throughout the iteration of Algorithm~\ref{alg:mafno-orth-exact} if $v^{0}$ is not a right singular vector of $A$ and the sequence $\norm{Av^{k}}$ is increasing and convergent (a.s.).
\end{corollary}
\begin{proof}
Let $\sigma$ be a singular value of $A$ with multiplicity $r$ and $E$ the respective eigenspace.
If $v^{k}+\tau x^{k}$ is in $E$, then (by Lemma~\ref{lem:dimension-directions}) $\hat{x} = \tau x^{k}$ is from an affine subspace $L$ of dimension at most $r$.
Hence, the set of directions $x^{k} \in T_{\sphere^{d-1}}(v^k) \cap \sphere^{d-1}$ that may result in $v^{k+1}\in E$ is a subset of the set $\left\{\sqrt{2} x/\norm{x} \mid x\in L \right\}$. Since $v^{k}$ and $x^{k}$ are not parallel, this is a set of probability zero (with respect to the Haar measure on $T_{\sphere^{d-1}}(v^k) \cap \sphere^{d-1}$, according to which $x^{k}$ is sampled due to remark~\ref{rem:distribution-x}), if $r<d-1$ (which then also implies $r-1<d$). The latter (restrictive case from Lemma~\ref{lem:dimension-directions}) is fulfilled by assumption. Since this is true for all (finitely many) eigenspaces of $A^{*}A$, the claim follows.
The monotonicity and the convergence of the sequence $\norm{Av^{k}}$
is a consequence of Remark~\ref{rem:conv-normAv} under the assumption.
\end{proof}
Corollary~\ref{cor:no-sing-vectors-during-iteration} guarantees that Algorithm~\ref{alg:mafno-orth-exact} will not run into division by $a_{k}=0$ almost surely
if all singular values of $A$ have multiplicity at most $d-2$,
(which can be guaranteed, for example, if $A$ has three different singular values).
Hence,
we make the following assumption and assume that it is fulfilled from hereon as long as nothing else is said.
\begin{assumption}
\label{ass:ass_sing_rang}
The operator $A$ from $\RR^{d}$ to $\RR^{m}$ has no singular value with multiplicity $d-1$ or $d$.
\end{assumption}
If we sample $v^{0}$ uniformly on $\sphere^{d-1}$, Assumption~\ref{ass:ass_sing_rang} guarantees that $a_{0}\neq 0$ almost surely and by Corollary~\ref{cor:no-sing-vectors-during-iteration} this is preserved throughout the iteration.
Later in Remarks~\ref{rem:multiplicity-d-1} and~\ref{rem:second_assump_gone} we will see that we can even drop Assumption~\ref{ass:ass_sing_rang};
hence, Algorithm~\ref{alg:mafno-orth-exact} can be applied to any linear map $A$ mapping from $\RR^d$ to $\RR^m$.
\subsection{Almost sure convergence}
\label{sec:convergence}
Now we set out to prove that Algorithm~\ref{alg:mafno-orth-exact} will almost surely give a sequence $(v^{k})_{k \in \NN}$ such that $\norm{Av^{k}}\to\norm{A}$.
The first step is to calculate the ascent in each iteration.
Secondly,
we discuss the convergence of $(a_k)_{k \in \NN}$.
Afterwards,
we can prove the convergence
of $(A v^k)_{k \in \NN}$.
Finally,
the almost sure convergence $\norm{A v^k} \to \norm{A}, k \to \infty$
is proven.
\begin{lemma}
\label{lem:ascent-formula}
For $v^{k}$, $a_{k}$ and $\tau_{k}$ from Algorithm~\ref{alg:mafno-orth-exact} it holds
\begin{align*}
\norm{Av^{k+1}}^{2} - \norm{Av^{k}}^{2}
=
\tau_{k}a_{k}.% = \tfrac12 \left( b_{k} + \sqrt{b_{k}^{2} + 4a_{k}^{2}} \right).
\end{align*}
\end{lemma}
\begin{proof}
We calculate the ascent in the objective:
\begin{align*}
\norm{A v^{k+1}}^2 - \norm{A v^{k}}^2 & = \tfrac1{1+\tau_k^2}\norm{Av^{k} + \tau_k Ax^{k}}^2 - \norm{Av^{k}}^2\\
& = \tfrac1{1+\tau_k^2}\left[\norm{Av^{k}}^{2} + 2\tau_k\scp{Av^{k}}{Ax^{k}} + \tau_k^2\norm{Ax^{k}}^2\right]- \norm{Av^{k}}^2\\
& = \tfrac{2\tau_k}{1+\tau_k^{2}}\scp{Av^{k}}{Ax^{k}} + \tfrac{\tau_k^{2}}{1+\tau_k^{2}}\norm{Ax^{k}}^{2} + \left( \tfrac1{1+\tau_k^{2}} -1\right)\norm{Av^{k}}^{2}\\
& = \tfrac{2\tau_k}{1+\tau_k^{2}}\scp{Av^{k}}{Ax^{k}} + \tfrac{\tau_k^{2}}{1+\tau_k^{2}}\left( \norm{Ax^{k}}^{2} - \norm{Av^{k}}^{2} \right)\\
& = \tfrac{2\tau_k}{1+\tau_k^{2}}a_{k} + \tfrac{\tau_k^{2}}{1+\tau_k^{2}} b_{k}\\
& = \tfrac{\tau_k}{1+\tau_k^{2}}\left( a_{k} + (a_{k} + b_{k}\tau_k) \right).
\end{align*}
Since we know that $\tau_k$ solves $-a_{k}\tau_k^{2} + b_{k}\tau_k + a_{k} = 0$ we replace $a_{k} + b_{k}\tau_k = a_{k}\tau_k^{2}$ and get
\begin{equation*}
\norm{Av^{k+1}}^{2} - \norm{Av^{k}}^{2} = \tfrac{\tau_k}{1+\tau_k^{2}}(a_{k} + \tau_k^{2}a_{k}) = \tau_k a_{k}.
\qedhere
\end{equation*}
\end{proof}
Now we show that the sequence $(a_{k})_{k \in \NN}$ converges to zero almost surely.
\begin{lemma}
\label{lem:a-to-0}
For $v^{k},x^{k}$ and $a_{k}$ generated by Algorithm~\ref{alg:mafno-orth-exact}
it holds that $a_{k}=\scp{Av^{k}}{Ax^{k}} \to 0$ almost surely.
\end{lemma}
\begin{proof}
First observe that since $\norm{Av^{k}}^{2}$ is convergent, Lemma~\ref{lem:ascent-formula} proves that $\tau_{k}a_{k}\to 0$.
Second, we have by definition that
$a_{k} = a_{k}\tau_k^{2} - b_{k}\tau_{k} = \tau_{k}(a_{k}\tau_{k} - b_{k})$.
We multiply by $a_{k}$
and get
\begin{align}
\label{eq:ab_tau}
a_k\tau_k(a_{k}\tau_{k}- b_k) = a_k^{2}.
\end{align}
Since $\abs{b_{k}} \leq \max\{\norm{A v^k}, \norm{A x^k}\} \leq \norm{A}$ is bounded,
the left-hand side vanishes which proves the claim.
\end{proof}
\begin{lemma}
\label{lem:E_a_k_sqr}
It holds that
\begin{align*}
\EE\big[ a_k^2 \mid v^k \big] = \frac{1}{d-1}\big\| \big(I_d-v^k(v^k)^*\big) A^*Av^k \big\|^2.
\end{align*}
\end{lemma}
\begin{proof}
By symmetry arguments (see also Remark~\ref{rem:distribution-x}) we have that
\begin{align*}
\EE[x^k(x^k)^*] = \frac{1}{d-1}\big(I_d-v^k(v^k)^*\big).
\end{align*}
It follows that
\begin{align*}
\EE\big[ a_k^2 \mid v^k \big]
&= \EE\big[ \langle Av^k,Ax^k\rangle^2 \mid v^k \big] \\
&= \EE\big[ (v^k)^*A^*Ax^k(x^k)^*A^*Av^k \mid v^k \big] \\
&= (v^k)^* A^*A \cdot \EE[x^k(x^k)^*] \cdot A^*Av^k
\end{align*}
and the assertion follows,
since $I_d - v^k(v^k)^*$ is idempotent.
\end{proof}
\begin{proposition}
\label{prop:acc-point-sing-vec}
Every accumulation point of the sequence $v^k$ generated by Algorithm~\ref{alg:mafno-orth-exact}
is a right singular vector of $A$ almost surely.
\end{proposition}
\begin{proof}
By Lemma~\ref{lem:a-to-0} we have $a_k^2 \to 0$ for $k \to \infty$ almost surely. As $a_k^2 \leq \|A\|^4$, Lebesgue's dominated convergence theorem shows that $\EE[a_k^2]\to 0$ for $k \to \infty$. Hence, by Lemma~\ref{lem:E_a_k_sqr} we obtain
\begin{align}
\label{eq:e_a_k_P_v}
\frac{1}{d-1} \EE\Big[ \|\big(I_d-v^k(v^k)^*\big)A^*Av^k\|_2^2 \Big] =
\EE\Big[\EE\big[ a_k^2 \mid v^k \big]\Big] =
\EE\big[ a_k^2\big] \to 0,
\end{align}
for $k \to \infty$,
which in turn shows that
$\|\big(I_d-v^k(v^k)^*\big)A^*Av^k\|_2^2 \to 0$ for $k \to \infty$ almost surely due to nonnegativity.
The event that the latter sequence converges is a subset of the event that accumulation points $v$ of $v^k$
fulfill that $\big(I_d-vv^*\big)A^*Av = 0$,
which implies that $A^*Av \in \ker(I_d - v v^*) = \langle v\rangle$,
i.e. $v$ is a right singular value of $A$.
\end{proof}
Since we now know that the iterates can only accumulate at right singular vectors and that $\norm{Av^{k}}$ is strictly ascending, we can conclude that we will reach the global maximum $\norm{A}$, once we are past the second largest singular value:
\begin{lemma}
\label{lem:B_Lemma}
Let
\begin{align*}
B \coloneqq \{v\in\sphere^{d-1}: \norm{Av}^2 > \sigma_2^2\}.
\end{align*}
Then, the following holds:
\begin{enumerate}
\item[(i)] If $v^{k}\in B$, then $v^{k+1}\in B$.
\item[(ii)] If $v^{k_0}\in B$ holds for some $k_0$, then $\lim_{k\to\infty} \norm{Av^{k}} = \sigma_1 = \norm{A}$ almost surely.
\end{enumerate}
\end{lemma}
\begin{proof}\mbox{}
\begin{itemize}
\item[(i)]
This holds,
since $\norm{Av^{k}}^2$ is increasing by construction.
\item[(ii)]
By Proposition~\ref{prop:acc-point-sing-vec} each accumulation point of $(v^{k})$ is a right singular vector,
so the accumulation points of $\norm{Av^{k}}$ are the corresponding singular values.
Since $\norm{Av^{k}}$ is increasing
and by assumption $\norm{Av^{k_0}}>\sigma_{2}$, the only possible accumulation point of $\norm{Av^{k}}$ is $\sigma_{1}$, and hence,
the whole sequence $\norm{Av^{k}}$ must converge to this singular value.
\end{itemize}
\end{proof}
Now comes a technical lemma that will help us to prove that in each step there is a non-zero probability, which is bounded from below by an absolute constant, that the next iterate will fulfill $\norm{Av^{k+1}}^{2}>\sigma_{2}^{2}$.
Roughly speaking, the lemma says that the probability to reach an $\varepsilon$-neighborhood of two antipodal points by a step in a direction taken uniformly at random with appropriate stepsize has an absolute lower bound that does depend neither on the point where we start nor on the point at which we aim.
\begin{lemma} \label{lem:inf_probability}
Let $\varepsilon>0$ and $v,v^*\in\sphere^{d-1}$ and define
\begin{align*}
D_v := \left\{x\in \{v\}^{\bot}\mid \frac{v+\tau x}{\norm{v+\tau x}} \in B_\varepsilon(v^*)\cup B_\varepsilon(-v^*) \text{ for some } \tau\in\RR\right\}
\end{align*}
and denote by \;$\mathcal{U}_{v}$ the uniform distribution on the unit sphere in $\{v\}^{\bot}$.
Then there exists a positive constant $p_{\varepsilon,d}$ such that
\begin{align}\label{eq:def_p_varepsilon}
\inf_{v\in\sphere^{d-1}} \mathcal{U}_v\big( D_v \big) \geq p_{\varepsilon,d}.
\end{align}
\end{lemma}
\begin{proof}
If $v+\tau x$ gets scaled inside the ball $B_{\varepsilon}(v^{*})$, then $v$ and $v^{*}$ lie in the same hemisphere, i.e. we can limit ourselves to this case. % $\scp{v}{v^{*}}\geq 0$.
The value $\mathcal{U}_{v}(D_{v})$ is the fraction of the part of the unit sphere of directions $x$ that point from $v$ towards the ball $B_{\varepsilon}(v^{*})$.
We can describe this set as follows: Consider the cone spanned by $B_{\varepsilon}(v^{*})$ with apex in the origin and intersect it with the tangent space $v + \{v\}^{\bot}$.
The set $D_{v}$ is the projection of that conic section onto the unit sphere of directions centered in $v$ and within $v+\{v\}^{\bot}$.
This projection is (due to symmetry reasons) a spherical cap in the unit sphere in the tangent space and hence, described by an opening angle (see Figure~\ref{fig:conic-section} for an illustration).
It remains to show that this opening angle is strictly positive on the full closed hemisphere.
For points $v$ with $\scp{v}{v^{*}}>0$ this opening angle is always positive.
Especially for $v$ on the equator, i.e. for $\scp{v}{v^*}=0$ where the conic section degenerates, the opening angle is still positive and equal to the opening angle of the cone.
Overall, the opening angle is a positive function on the compact closed hemisphere and hence, has a positive minimum.
\end{proof}
One can obtain explicit estimates for $p_{\varepsilon,d}$ above, but since we are not interested in the precise value, we omit the derivation.
\begin{figure}[htb]
\centering
\includegraphics[width=10cm]{simax-figures/conic-section}
\caption{Illustration of the situation in the proof of Lemma~\ref{lem:inf_probability}.}
\label{fig:conic-section}
\end{figure}
Now we can prove the main theorem about almost sure convergence of the algorithm,
i.e. $\norm{A v^k} \to \norm{A}$ for $k \to \infty$ a.s.
\begin{theorem}
\label{thm:convergence_almost_surely}
For the iterates of Algorithm~\ref{alg:mafno-orth-exact} it holds $\lim_{k\to\infty}\norm{A v^{k}} = \sigma_1$ almost surely. Moreover, if the singular space to $\sigma_1$ is one-dimensional, then $v^{k}$ converges almost surely to a right singular vector of $\sigma_1$.
\end{theorem}
\begin{proof}
We consider the set $B$ from Lemma~\ref{lem:B_Lemma} and define the events
\begin{align*}
A_k = \{ v^{k}\not\in B \}.
\end{align*}
Lemma~\ref{lem:B_Lemma}(i) shows $A_{k+1}\subset A_k$ and by Lemma~\ref{lem:B_Lemma}(ii) we have
\begin{align*}
\big\{ \norm{Av^{k}} \not\to \sigma_1 \big\} = \bigcap_{k=0}^\infty A_k.
\end{align*}
Now we show that there is some $p$ independent of $k$ such that we have
\begin{align}\label{eq:prob-geometric-decrease}
\PP(A_{k+1}) \leq (1-p)\cdot\PP(A_{k})
\end{align}
Then the assertion follows, as from $A_{k+1}\subset A_k$ we obtain that
\begin{align*}
\PP\Big( \bigcap_{k=0}^\infty A_k \Big) = \lim_{k\to\infty} \PP(A_k) = 0.
\end{align*}
We prove~\eqref{eq:prob-geometric-decrease} as follows:
By $U(v,x)$ we denote the update in Algorithm~\ref{alg:mafno-orth-exact} of~$v$
with a direction $x\in \sphere^{d-1}\cap v^\perp$ and by $\PP_{v}$ and $\PP_{x}$ we denote the probabilities for $v$ and $x$ respectively.
Further, we denote the update of the algorithm with the normal distributed direction $y$ by $S$, i.e. $v^{k+1} = S(v^{k},y^{k})$. Since $v^{k}$ and $y^{k}$ are independent it holds that
\begin{align*}
\PP(A_{k+1}) &= \PP(\{S(v^{k},y^{k})\notin B\})\\
& = \int\limits_{\sphere^{d-1}} \int\limits_{\RR^{d}} \mathbbm{1}_{S(v,y) \notin B}\ \dd \PP_{y^{k}}(y)\ \dd \PP_{v^{k}}(v)
\end{align*}
Furthermore $S(v^{k},y^{k}) = v^{k+1}\notin B$ implies $v^k\notin B$ (by Lemma~\ref{lem:B_Lemma} (ii)) and thus we can write
\begin{align}
\PP(A_{k+1}) &=\int\limits_{\sphere^{d-1}\setminus B} \quad \int\limits_{\RR^{d}}
\quad \mathbbm{1}_{S(v,y) \notin B}\ \dd \PP_{y^{k}}(y)\ \dd \PP_{v^{k}}(v)\nonumber \\
&= \int\limits_{\sphere^{d-1}\setminus B} \quad \int\limits_{\sphere^{d-1}\cap{v}^{\bot}} \mathbbm{1}_{U(v,x) \notin B}\ \dd \PP_{x^{k}}(x)\ \dd \PP_{v^{k}}(v).\label{eq:prob-Ak+1}
\end{align}
Now we use Lemma~\ref{lem:inf_probability} as follows:
As $B$ is open and contains a right singular vector $v_{1}$ for the singular value $\sigma_{1} = \norm{A}$,
there exists $\varepsilon>0$ with
\begin{align}
\label{eqn:relation_B_Bepsilon}
\Big( B_\varepsilon(v_1) \cup B_\varepsilon(-v_1)\Big)\cap \sphere^{d-1}\subset B.
\end{align}
Lemma~\ref{lem:inf_probability}
with this $\varepsilon$ and $v^*=v_1$ gives the lower bound $p_{\varepsilon,d}>0$ on the probability of sampling a direction $x$ for which there is a stepsize that will lead to a new point in $B_{\varepsilon}(v_{1})$.
The update of the algorithm maximizes $\norm{Av^{k+1}}$ over the stepsize and effectively maximizes the value of $\norm{Av}$ for $v$ on a semicircle with midpoint $v^{k}$ in direction $x^{k}$.
Thus, for each direction $x$ for which a step into~$B$ exists,
the algorithm will choose a stepsize that will land in $B$ as well.
Hence, we get from the lower bound from Lemma~\ref{lem:inf_probability}, \eqref{eqn:relation_B_Bepsilon} and~\eqref{eq:prob-Ak+1} (using Fubini) that
\begin{align*}
\PP(A_{k+1}) & \leq \int\limits_{\sphere^{d-1}\setminus B}^{} (1-p_{\varepsilon,d})\ \dd\PP_{v^{k}}(v) = (1-p_{\varepsilon,d})\PP(A_{k}).
\end{align*}
Now assume that the singular space to $\sigma_{1}$ is one-dimensional and spanned by $v_{1}\in \sphere^{d-1}$. Then we have that
\begin{align*}
\big\{ \norm{Av^{k}} \not\to \sigma_1 \big\} = \{ v^{k}\not\to v_{1}\}\cap\{v^{k}\not\to -v_{1}\}.
\end{align*}
However, $v^{k}$ cannot accumulate at both $v_{1}$ and $-v_{1}$: By construction, the sequence $\norm{Av^{k}}$ is increasing and, moreover, two consecutive iterates $v^{k+1}$ and $v^{k}$ always lie on the same hemisphere. Hence, one we have $v^{k}\in B_{\varepsilon}(v_{1})$, e.g., the next iterates cannot be in $B_{\varepsilon}(-v_{1})$ since, for $\varepsilon$ small enough, this set is too far apart to get there without decreasing the objective value.
\end{proof}
\begin{remark}[Stopping criteria]
\label{rem:stop_criteria} In Proposition~\ref{lem:E_a_k_sqr} we have seen that
$\EE(a_k^{2}) = \tfrac1{d-1}\norm{A^{*}Av^{k} - \norm{Av^{k}}^{2} v^{k}}^{2}$, i.e. a small expectation of $a_{k}^{2}$ shows that $v^{k}$ is close to a singular vector.
Furthermore,
Lemma~\ref{lem:a=0-at-sing-vecs} tells us
that $a_{k}=0$ only happens at right singular vectors.
This motivates to try to use the expectation of $a_{k}^{2}$ as stopping criterion,
i.e. stop if $(d-1)\EE(a_{k}^{2})\leq \varepsilon^{2}$ as a stopping criterion.
It turns out to be practical to start approximating the expectation when we observe that $a_{k}^{2}$ is small:
If $(d-1)a_{k}^{2}\leq\varepsilon^{2}$,
resample $x^{k}$ several times and stop if the observed empirical mean obeys the same bound.
\end{remark}
\begin{remark}[Handling the case of singular values of multiplicity $d-1$]
\label{rem:multiplicity-d-1}
The assertion of Lemma~\ref{lem:B_Lemma}(ii) also holds in the case of a singular value of multiplicity $d-1$, such that Algorithm~\ref{alg:mafno-orth-exact} will also converge in this case:
Under this assumption, there are only two distinct singular values, and we distinguish two cases: If the smaller singular value has multiplicity $d-1$, then a random initialization $v^{0}\in\sphere^{d-1}$ will almost surely give $\norm{Av^{0}}>\sigma_{2}$. Hence, we have $a^{0}\neq 0$ and can perform a step. Then either we are done (which is almost surely not the case) or we have $\sigma_{2}<\norm{Av^{k}}<\sigma_{1}$ and we obtain $\norm{Av^{k}}\to\norm{A}$ from Lemma~\ref{lem:B_Lemma}.
If the larger singular value has multiplicity $d-1$ we argue as follows:
We initialize $v^{0}$ which is almost surely not a right singular vector of $A$ and also it holds $\norm{Av^{0}}>\sigma_{2}$ almost surely.
Moreover, by Lemma~\ref{lem:a=0-at-sing-vecs} we obtain that $a_0 \neq 0$ holds (a.s.). By optimality of the stepsize $\tau_0 \in \RR$ we obtain a direct projection onto the space of normalized maximal singular vectors, which is in this case an equator of the $(d-1)$-sphere. We end up with convergence after one iteration, i.e. $\norm{A v^1} = \sigma_1$.
\end{remark}
\begin{remark}[Handling the case of singular values of multiplicity $d$---detecting orthonormal columns]
\label{rem:second_assump_gone}
A curious fact about Algorithm~\ref{alg:mafno-orth-exact} is as follows:
If we sample $v^{0}$ uniformly on the unit sphere and observe $a_0 = 0$, then we almost surely have that the operator~$A$ has orthogonal columns and all have the same norm, i.e. that
\begin{align}
\label{eqn:A_with_ortho_col_and_same_norm}
A^{*}A = cI_{d}
\end{align}
holds for some $c>0$.
To see this,
we argue by contradiction:
If \eqref{eqn:A_with_ortho_col_and_same_norm} does not hold,
then $A^*A$ has at least two distinct eigenvalues
and therewith at least two distinct proper spaces of eigenvectors.
Therefore $v^0$ is,
as uniformly sampled vector from the $(d-1)$-sphere,
almost surely not a right singular vector of $A$.
Hence,
we obtain from Lemma~\ref{lem:a=0-at-sing-vecs}
that $a_0=0$ happens with probability zero.
Equivalently,
we have
\begin{align*}
\PP[a_0 = 0] > 0
\quad \Rightarrow \quad
A^*A = cI_{d},
\end{align*}
which yields the assertion.
\end{remark}
\begin{remark}[Assumption~\ref{ass:ass_sing_rang} can be dropped]
\label{rem:assumption-can-be-dropped}
We started the convergence analysis with Assumption~\ref{ass:ass_sing_rang}, i.e. that $A$ has no singular value of multiplicity $d-1$ or $d$. This was used to ensure that if we have $a^{0}\neq 0$, we will have $a^{k}\neq 0$ throughout the iteration.
However, now we see that this assumption can be dropped: Remark~\ref{rem:multiplicity-d-1} shows that the method also works if there is a singular value of multiplicity $d-1$ and Remark~\ref{rem:second_assump_gone} shows the same if we have just a single singular value.
\end{remark}
\begin{remark}[Calculating leading right singular vectors]
One can extend Algorithm~\ref{alg:mafno-orth-exact} to compute a number of leading singular values and corresponding right singular vectors (if they are all distinct). Once $\sigma_{1}$ and $v_{1}$ have been found, we can restrict Algorithm~\ref{alg:mafno-orth-exact} to the space $v_{1}^{\bot}$ using Gram-Schmidt orthogonalization for $x^{k}$ and find $\sigma_{2}$ and $v_{2}$ in this way. Although this procedure gets more computationally heavy for each new singular value, we may still be able to compute the first few leading right singular vectors.
\end{remark}
\subsection{Convergence rates}
\label{sec:conv_rates}
In this subsection
we derive two convergence results:
one for the corresponding singular vector and singular value equation,
i.e.
\[
\norm{A^*A v^k - \norm{A v^k}^2 v^k}^2, \quad k \in \NN,
\quad \text{where}\quad
v^k\;\text{is from Alg.~\ref{alg:mafno-orth-exact}}
\]
and the second for the stepsizes $(\tau_k)_{k \in \NN}$,
again, generated by Alg.~\ref{alg:mafno-orth-exact},
under the assumption that the multiplicity of $\sigma_1 = \norm{A}$ is one.
The following proposition
shows the convergence of the $a_k^2 = \scp{A v^k}{A x^k}^2$
from Algorithm~\ref{alg:mafno-orth-exact} towards zero
in a sublinear rate.
\begin{proposition}\label{prop:ak-conv-rate}
Let $v^k \in \sphere^{d-1}$ be generated by Algorithm~\ref{alg:mafno-orth-exact}.
It holds almost surely
\[
\min_{0\leq k \leq n} a_k^2 \leq \tfrac{2\norm{A}^{4}}{n+1}.
\]
\end{proposition}
\begin{proof}
From \eqref{eq:ab_tau} we have $a_k\tau_k(a_k\tau_k- b_{k}) = a_k^2$ and
from Lemma~\ref{lem:ascent-formula} we get that $a_{k}\tau_{k} = \norm{Av^{k+1}}^{2} - \norm{Av^{k}}^{2}\geq 0$ and $a_{k}\tau_{k}\leq\norm{A}^{2}$. We also have $\abs{b_{k}}\leq\norm{A}^{2}$, so we get
\begin{align*}
a_{k}^{2} = a_k\tau_k(a_k\tau_k- b_{k}) \leq 2\norm{A}^{2}a_{k}\tau_{k} = 2\norm{A^{2}}\left(\norm{A v^{k+1}}^2 -\norm{A v^k}^2\right)
\end{align*}
We sum this equation from $k=0,\dots,n$ and arrive at
\begin{align}
\sum\limits_{k=0}^{n}a_{k}^{2}\leq 2\norm{A}^2 \norm{Av^{n+1}}^{2}\leq 2\norm{A}^4.
\label{eq:ak_min_O}
\end{align}
Finally, estimating $\sum\limits_{k=0}^{n}a_{k}^{2} \geq (n+1)\min_{k=0,\dots,n} a_{k}^{2}$ proves the claim.
\end{proof}
\begin{remark}[Convergence rate for $a_k \tau_k$]
\label{rem:pre_ak_min_O}
Summing up the equation from Lemma~\ref{lem:ascent-formula}
for $k = 0,...,n$ we have $(n+1)\min_{0 \leq k \leq n}a_k \tau_k \leq \sum_{0 \leq k \leq n} a_k \tau_k = \norm{A v^{n+1}}^2 - \norm{A v^0}^2 \leq \norm{A}^2$.
Hence we also have $\min_{0 \leq k \leq n} a_k \tau_k \leq \tfrac{\norm{A}^2}{n+1}$ (a.s.).
\end{remark}
We can turn the above proposition into a convergence rate for the error in
the eigenvector equation for $A^*A$ with approximative eigenvalue $\norm{A v^k}$
in the $k$-th iteration.
\begin{theorem}
Let $v^k \in \sphere^{d-1}$ be generated by Algorithm~\ref{alg:mafno-orth-exact}.
It holds almost surely
\[
\min_{0 \leq k \leq n}
\norm{A^*A v^k - \norm{A v^k}^2 v^k}^2 \leq \tfrac{2(d+1)\norm{A}^{4}}{n+1}.
\]
\end{theorem}
\begin{proof}
First,
from Lemma~\ref{lem:E_a_k_sqr},
we get
\[
\EE[a_k^2 \mid v^k]
= \tfrac{1}{d-1}\norm{\norm{Av^{k}}^{2}v^{k} - A^{*}Av^{k}}^{2}.
\]
Now we use Proposition~\ref{prop:ak-conv-rate} and denote by $k^{*}(n)$ the index at which the minimum is attained.
For this it holds that
\begin{align*}
\min_{0\leq k\leq n} \norm{\norm{Av^{k}}^{2}v^{k} - A^{*}Av_{k}}^{2}
& \leq \norm{\norm{Av^{k^{*}(n)}}^{2}v^{k^{*}(n)} - A^{*}Av^{k^{*}(n)}}^{2}\\
& = (d+1)\EE[a^2_{k^{*}(n)}\mid v^{k^{*}(n)}]\\
& \leq (d+1)\EE[\min_{0\leq k\leq n}a^2_{k}\mid v^{k}] \leq \tfrac{2(d+1)\norm{A}^{4}}{n+1}.
\qedhere
\end{align*}
\end{proof}
Under mild additional assumptions we can also show that the stepsizes $\tau_{k}$ tend to zero.
\begin{proposition}
Under assumption that the multiplicity of $\sigma_1 = \norm{A}$ is one,
we almost surely have
\[
\min_{0 \leq k \leq n} |\tau_k| \in \mathcal{O}(n^{-1/2}),
\quad n \to \infty.
\]
\end{proposition}
\begin{proof}
From Proposition~\ref{prop:stepsize-formula} we have
$a_k + \tau_k b_k - \tau_k^2 a_k = 0$ for all $k \in \NN$ (a.s.).
Rewriting yields
(if $\tau_k \neq 1$, such that $b_k \neq 0$, otherwise $b_k = 0$ is necessary)
\[
\tau_k a_k = -\tfrac{\tau_k^2 b_k}{1 - \tau_k^2}
\qquad \forall k \in \NN
\quad \text{(a.s.)}.
\]
By the convergence from Theorem~\ref{thm:convergence_almost_surely},
we have for any $\varepsilon > 0$ an $N \in \NN$ such that
$\norm{v^* - v^k} < \varepsilon$ for all $k \geq N$,
where $\norm{A v^*} = \sigma_1$.
Let $\{v^k_j\}_{j = 1}^{d-1}$
an orthonormal basis of $(v^k)^\perp$.
We obtain $\norm{v^*-v^k_j}^2 = 2(1-\scp{v^*}{v^k_j}) = 2(1-\scp{v^*-v^k}{v^k_j}) \geq 2(1-\varepsilon)$
for all $j \in \{1,...,d-1\}$ and $k \geq N$.
Suppose that $\varepsilon < (d-1)^{-1}$.
Since $x^k \in (v^k)^\perp$
we have $\norm{v^* - x^k}^2 \geq 2(1-\tfrac{\varepsilon}{d-1})$
and hence $\abs{\scp{v^*}{x^k}} < \tfrac{\varepsilon}{d-1}$
for all $k \geq N$.
Hence $\norm{A v^k} \geq \sigma_1 - \norm{A}\varepsilon$ for all $k \geq N$
and $\limsup_{k \to \infty}\norm{A x^k} \leq \sigma_2 = \max_{v \in (v^*)^\perp \cap \sphere^{d-1}} \|A v\|$.
Hence we also have (by choosing $N$ larger if necessary)
$\sup_{k \geq n} \norm{A x^k} \leq \sigma_2 + \norm{A}\varepsilon$
for all $n \geq N$
and hence
$\norm{A x^k} \leq \sigma_2 + \norm{A}\varepsilon$ for all $k \geq N$.
Now,
suppose that $\varepsilon < \min\{\tfrac{\sigma_1-\sigma_2}{4\norm{A}}, \tfrac{1}{d-1}\}$;
then,
together we have
\begin{align}
\label{eq:lower_bound_b_k}
\abs{b_k}
&= \abs{\norm{A v^k} - \norm{A x^k}} \cdot (\norm{A v^k} + \norm{A x^k}) \notag \\
&\geq \tfrac{\sigma_1 - \sigma_2}{2} \cdot \norm{A v^k}
\geq \tfrac{\sigma_1 - \sigma_2}{2}(\sigma_{1} - \norm{A}\varepsilon)
\geq \tfrac{\sigma_1 - \sigma_2}{2}\tfrac{d-2}{d-1}\sigma_{1} > 0
\end{align}
We finally bound
\[
|a_k \tau_k| = \abs{\tfrac{\tau_{k}^{2}b_{k}}{1-\tau_{k}^{2}}}
\geq \abs{\tfrac{\tau_k^2}{1 - \tau_k^2}}
\cdot \tfrac{\sigma_1 - \sigma_2}{2}\tfrac{d-2}{d-1}\sigma_1
\qquad \forall k \geq N
\quad \text{(a.s.)}.
\]
Utilizing Remark~\ref{rem:pre_ak_min_O}
yields $\min_{0 \leq k \leq n} \abs{\tfrac{\tau_k^2}{1 - \tau_k^2}} \in \bigO(n^{-1})$.
Let $k^*(n)$ be the sequence,
realizing the minimum for any $n \in \NN$.
Using the lower bound of $\abs{b_k}$ from \eqref{eq:lower_bound_b_k},
we obtain $\tau_{k^*(n)}^2 \to 0$ for $n \to \infty$ (a.s.),
otherwise one has $b_{k^*(n)} \to 0$.
Hence, we have for $n \in \NN$ large enough $\tau_{k^*(n)}^2 < 1$ such that
\begin{align*}
\min_{0 \leq k \leq n} \tau_{k}^2
\leq \tau_{k^*(n)}^2
\leq \tfrac{\tau_{k^*(n)}^2}{1 - \tau_{k^*(n)}^2}
\in \bigO(n^{-1})
\end{align*}
and the assertion follows.
\end{proof}
\section{Numerical experiments}
\label{sec:numerical-experiments}
We performed four experiments to illustrate the practical performance of Algorithm~\ref{alg:mafno-orth-exact}. The code (Jupyter notebooks with Python code) can be found at
\url{https://github.com/dirloren/matrix_free_norms}.
In the first experiment, we illustrate
that Algorithm~\ref{alg:mafno-orth-exact} may be fast in some cases.
We consider the matrix of the form
\begin{align}
\label{eq:matrix_fail_imp}
A
\coloneqq
\left[\begin{matrix}
1 & \varepsilon \\
0 & 1
\end{matrix}\right]
\end{align}
where $1 \gg \varepsilon > 0$ is small.
In this case,
the singular values are given by the eigenvalues of the matrix
\begin{align*}
B
\coloneqq
A^{*} A
=
\left[\begin{smallmatrix}
1 & \varepsilon \\
\varepsilon & 1 + \varepsilon^2
\end{smallmatrix}\right]
\quad \text{namely} \quad
\lambda_{\max,\min}=
\tfrac{\varepsilon^2 \pm \varepsilon\sqrt{\varepsilon^2 + 4} + 2}{2},
\end{align*}
such that
$\norm{A} = \sqrt{\lambda_{\max}} = \sqrt{1 + \tfrac{\varepsilon^2 + \varepsilon\sqrt{\varepsilon^2 + 4} }{2}}$.
We compare Algorithm~\ref{alg:mafno-orth-exact} and the power iteration.
Algorithm~\ref{alg:mafno-orth-exact} converges in one iteration up to machine precision
(the reason is that the method will explore a half circle by the exact line search and this half circle always contains one of the two solutions, cf. Figure~\ref{fig:circle-convergence-algorithm}).
The power iteration will converge to the solution but the convergence rate depends on the ratio of the largest and second largest eigenvalue of $A^{*}A$
in the magnitude \cite[Section 7.3]{golub2013}.
In this case,
we have
\begin{align*}
\tfrac{\lambda_{\max}}{\lambda_{\min}} = \tfrac{\varepsilon^2 + \varepsilon\sqrt{\varepsilon^2 + 4} + 2}{\varepsilon^2 - \varepsilon\sqrt{\varepsilon^2 + 4} + 2}
= (1 + \tfrac{\varepsilon^{2}}2 + \tfrac{\varepsilon}2\sqrt{\varepsilon^2+4})^{2} = 1 + \bigO(\varepsilon)
\end{align*}
and we see that we will need many iterations for small $\varepsilon$.
In our experiments we reached an error of $10^{-5}$ after about $100$ iterations (depending on the random initialization) for $\varepsilon=10^{-2}$ and already about $1.000$ iterations for $\varepsilon = 10^{-4}$.
\begin{figure}[htb]
\centering
\includegraphics[width=0.5\textwidth, clip=true, trim=0pt 0pt 50pt 50pt]{simax-figures/circle.pdf}
\caption{Illustration of Algorithm~\ref{alg:mafno-orth-exact} in two dimensions. Almost every initialization shares a half circle with one of the two global maxima $v^{*}, -v^{*}$. Since the line search is exact the method will arrive at $v^{*}$ after one step.}
\label{fig:circle-convergence-algorithm}
\end{figure}
As a second experiment,
we consider the matrix $A = \diag(1,1,0)$ where
the set of maximal right singular vectors is isomorphic to the 1-sphere.
Similarly to the first experiment,
our method converges theoretically in one iteration.
However, if we do not use the stopping criterion from Remark~\ref{rem:stop_criteria}
the iterates will wander around the set of optima, as
can be seen in Figure~\ref{fig:numerical_instabilities}, which results in an exploration
of the entire unit sphere in the space of right singular vectors.
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{simax-figures/experiment2}
\caption{First two components of the iterates $v^{k}$ (for $k=1,\dots,5000$)
of Algorithm~\ref{alg:mafno-orth-exact} for $A = \diag(1,1,0)$,
only plotted when $|v^{k}_3| < \varepsilon$.}
\label{fig:numerical_instabilities}
\end{figure}
In another experiment,
we consider the matrix $A = [1,0,...,0] \in \RR^{1\times d}$,
which has operator norm $\norm{A} = 1$.
We demonstrate the convergence of our method
by performing $50$ runs for $d \in \{100, 1000, 10000\}$
of Algorithm~\ref{alg:mafno-orth-exact} with $k = 10d$ iterations.
The resulting (relative) error $(\norm{A}-\norm{A v^k}/\norm{A}$
and the mean squared error for the last $d-1$ components
is visualized in Figure~\ref{fig:one_mat}.
We observe linear convergence in all cases and also that the convergence speed is independent of the dimension.
However,
the power iteration on the matrix $A^*A = \diag(1,0,...,0) \in \RR^{d \times d}$
would converge in exactly one iteration,
if the initial vector is uniformly sampled from the sphere $\sphere^{d-1}$.
\begin{figure}[htb]
\centering
\begin{tabular}{ccc}
$1\times100$ & $1\times1000$ & $1\times10000$\\
\includegraphics[width=0.30\textwidth]{simax-figures/one_vec_100.000000.pdf}
&\includegraphics[width=0.30\textwidth]{simax-figures/one_vec_1000.000000.pdf}
&\includegraphics[width=0.30\textwidth]{simax-figures/one_vec_10000.000000.pdf}
\end{tabular}
\caption{Results for 50 runs of Algorithm~\ref{alg:mafno-orth-exact} for matrices $A = [1,0,...,0] \in \RR^{1\times d}$. All plots are linear in the $x$-axis and logarithmic in the $x$-axis indicating linear convergence of the quantities in this case.}
\label{fig:one_mat}
\end{figure}
In a third experiment,
we investigate the convergence rate in dependence on the dimension
of the input and output space of the operator $A$, respectively.
To that end we generated random matrices of size $m\times d$ for different values of $m$ and $d$, performed $50$ runs of Algorithm~\ref{alg:mafno-orth-exact} with $k=20d$ iterations for each matrix and plot the relative error $(\norm{A}-\norm{Av^{k}})/\norm{A}$, the running minimum over the $a_{k}^{2}$ over iterations in Figure~\ref{fig:experiment3} and the upper bound from Proposition~\ref{prop:ak-conv-rate}.
We observe that the number $m$ of rows does not seem to have a large influence on the convergence speed, but for larger $d$ the method needs more iterations to reach a similar accuracy.
\begin{figure}[htb]
\centering
\begin{tabular}{cc}
$10\times50$ & $100\times500$\\
\includegraphics[width=0.47\textwidth]{simax-figures/experiment3_10x50}&
\includegraphics[width=0.47\textwidth]{simax-figures/experiment3_100x500}\\
$50\times50$ & $500\times500$\\
\includegraphics[width=0.47\textwidth]{simax-figures/experiment3_50x50}&
\includegraphics[width=0.47\textwidth]{simax-figures/experiment3_500x500}\\
$100\times50$ & $1000\times500$\\
\includegraphics[width=0.47\textwidth]{simax-figures/experiment3_100x50} &
\includegraphics[width=0.47\textwidth]{simax-figures/experiment3_1000x500}
\end{tabular}
\caption{Results for 50 runs of Algorithm~\ref{alg:mafno-orth-exact} for Gaussian matrices of different sizes.}
\label{fig:experiment3}
\end{figure}
In a fourth experiment,
we consider the Radon transform in different implementations. We start with the implementation in the Python package \texttt{scikit-\ image}~\cite{scikit-image}.
The function \texttt{skimage.transform.radon} implements a discretization of the linear operator given by the Radon transform, but \texttt{skimage} does not come with an adjoint of this operator.
However, it does provide a \emph{filtered} backprojection in \texttt{skimage.transform.iradon} and when the filter is set to \texttt{None}, an approximate adjoint is computed.
However,
this is not exact for at least two reasons:
One is the numerical accuracy of the implementation of the adjoint operation due to interpolation from polar to cartesian coordinates,
and the other is due to masking (\texttt{iradon} sets the values to zero that are outside of a circle).
This can be checked easily by running the following code:
\begin{lstlisting}[language=Python]
import numpy as np
from skimage.transform import radon, iradon
np.random.seed(4242)
N = 50
numAng = 70
theta = np.linspace(0.0, 180, numAng, endpoint=False)
v = np.random.randn(N,N)
w = np.random.randn(N,numAng)
Av = radon(v,theta=theta)
ATw = iradon(w, theta=theta, filter_name=None)
print(f'<v,A^Tw> = {np.sum(v*ATw)}, <Av,w> = {np.sum(Av*w)}')
\end{lstlisting}
which gives the output
\begin{lstlisting}[language=Python]
<v,A^Tw> = -17.458356056194354, <Av,w> = -946.1136269654232
\end{lstlisting}
The large difference of $\scp{v}{A^{*}w}$ and $\scp{Av}{w}$ indicates a problem with the implementation of the adjoint, may be an issue with scaling.
We explore how we can compute a reasonable approximation for the operator norm of the linear map given by \texttt{radon}.
In the same setting as in the above code,
the discretized Radon transform maps from $\RR^{n}$ with $n = 50\cdot50= 2500$ to $\RR^{m}$ with $m=50\cdot70 =3500$.
If we use $10\cdot n = 25000$ iterations of Algorithm~\ref{alg:mafno-orth-exact} initialized with the normalized all-ones vector
(which turns out to be a much better initialization than a random vector) we arrive at an estimate of the operator norm of $55.86$ (in fact, this has been achieved already at an iteration around $20650$).
Alternatively, we implemented the power method applied to $A^{*}A$ using the wrong adjoint from \texttt{skimage} and take $\norm{Av^{k}}$ as an approximation for the operator norm.
This converges much quicker (in less than $10$ iterations the value $\norm{Av^{k}}$ stabilized), but results in an estimate of $54.84$.
If we use $\sqrt{\norm{A^{*}Av^{k}}}$ as an approximation of the operator norm we only get an estimate of $8.22$ which is far off and again indicates a problem with scaling.
Both Algorithm~\ref{alg:mafno-orth-exact} and the power method,
produce a sequence $v^{k}$ which converges to a right singular vector
for the largest singular value
and in the case of the Radon transform
we can visualize these as $N\times N$ image.
We show the vectors in Figure~\ref{fig:singvec-radon}.
\begin{figure}[htb]
\centering
\includegraphics[width=0.46\textwidth]{simax-figures/experiment4-vAdjointFree}\
\includegraphics[width=0.46\textwidth]{simax-figures/experiment4-vPower}
\caption{Right singular vectors for the largest singular values of the Radon transform from \texttt{scikit-image} by Algorithm~\ref{alg:mafno-orth-exact} (left) and the power method (right).}
\label{fig:singvec-radon}
\end{figure}
{As an illustration where the adjoint is implemented more carefully
we use the \texttt{odl} Python package \cite{adler2017odl},
set the geometry on the $[-1,1]^2$ spaces with an $50\times50$ discretization grid
via \texttt{odl.tomo.parallel\_beam\_geometry}, where $70$ angles and $50$ detectors are used,
and define the Radon transform via \texttt{odl.tomo.Ray\ Transform}.
This package also provides directly the adjoint of the operator
and allows, again, the comparison between the power method
and Algorithm~\ref{alg:mafno-orth-exact}.
We observe that the power method converges after about 10 iterations
with an approximation of $1.9503840$ for the operator norm.
Our method hits this value after $9,800$ iterations
and converges after $40,000$ iterations
with an approximation of $1.9509808$ for the operator norm,
which gives us an error of about $6\cdot10^{-4}$.
The corresponding (approximate) right singular vectors are drawn in Fig.~\ref{fig:experiment6}
Differently to the implementation in \texttt{scikit-image},
the scaling error does not arise,
however, small approximation errors
by using the implemented adjoint can be observed.
Large differences can be observed for the corresponding right singular vectors,
see Fig.~\ref{fig:experiment6}.}
\begin{figure}[htb]
\centering
\includegraphics[width=0.46\textwidth]{./simax-figures/Radon_odl_large.pdf}\
\includegraphics[width=0.46\textwidth]{./simax-figures/Radon_odl_power_large.pdf}
\caption{Right singular vectors for the largest singular values of the Radon transform from \texttt{odl} by Algorithm~\ref{alg:mafno-orth-exact} (left) and the power method (right).}
\label{fig:experiment6}
\end{figure}
\section{Conclusion and outlook}
\label{sec:conclusion}
Algorithm~\ref{alg:mafno-orth-exact} is a stochastic method for computing the operator norm of a linear map $A$ between real Hilbert spaces and it only uses applications of the operator and computations of norms and inner products of vectors.
We show almost sure convergence, first under restricting the discussion to the case with no singular value of multiplicity $d$ or $d-1$ where $d$ is the dimension of the input space of $A$.
However, the case of multiplicity $d$ is also covered by Remark~\ref{rem:second_assump_gone} and also the case of multiplicity $d-1$ is handled by Remark~\ref{rem:multiplicity-d-1}.
Our examples show that the method works in practice but also show that it may be slow (while it is indeed faster than the power iteration in some special cases).
The question of the convergence speed regarding $\norm{A}-\norm{Av^k}$ is still open;
however, we answered the question of the convergence speed for the corresponding eigenvector and eigenvalue equation.
We only treated real Hilbert spaces here but the case of complex spaces can be treated similarly and only the definition of $a_{k}$ should be changed to $\operatorname{Re}(\scp{Av^{k}}{Ax^{k}})$.
Another question that we did not answer here is the following. How can we calculate $\norm{A-V}$ if we only can do evaluations of $A$ and $V^{*}$ (and no evaluations of $A^{*}$ and $V$ are possible)? This question is important in the case of adjoint mismatch as it happens, for example, in computerized tomography~\cite{lorenz2023chambolle,lorenz2018randomized,chouzenoux2023convergence,Savanier2021ProximalGA,Chouzenout2021pgm-adjoint,zeng2000unmatched}. The idea we presented here can be generalized to this case if we start from the formulation
\begin{align*}
\norm{A-V} = \max_{\norm{u}=1,\norm{v}=1} \scp{u}{(A-V)v} = \max_{\norm{u}=1,\norm{v}=1}\left( \scp{u}{Av} - \scp{V^{*}u}{v}\right)
\end{align*}
and will be subject of future research.
Finally, it is worth mentioning that Algorithm~\ref{alg:mafno-orth-exact} can be adapted to calculate the \emph{smallest} singular value of $A$.
To that end, one only has to replace the $\argmax$ in the calculation of the stepsize with an $\argmin$ which can also be computed exactly. The practicality of this method shall be investigated in upcoming work.
\bibliographystyle{plain}
\bibliography{./literature}
\end{document}
%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
\begin{thebibliography}{10}
\bibitem{adler2017odl}
Jonas Adler, Holger Kohr, et~al.
\newblock {Operator Discretization Library (ODL)}.
\newblock \url{https://github.com/odlgroup/odl}, 2017.
\newblock Software and documentation accessed \today.
\bibitem{boulle2024operator}
Nicolas Boull\'{e}, Diana Halikias, Samuel~E. Otto, and Alex Townsend.
\newblock Operator learning without the adjoint.
\newblock {\em J. Mach. Learn. Res.}, 25(1), 2024.
\bibitem{bredies2022degenerate}
Kristian Bredies, Enis Chenchene, Dirk~A Lorenz, and Emanuele Naldi.
\newblock Degenerate preconditioned proximal point algorithms.
\newblock {\em SIAM Journal on Optimization}, 32(3):2376--2401, 2022.
\bibitem{bredies2021convergence}
Kristian Bredies and Richard Huber.
\newblock Convergence analysis of pixel-driven radon and fanbeam transforms.
\newblock {\em SIAM Journal on Numerical Analysis}, 59(3):1399--1432, 2021.
\bibitem{brezinski1998transpose}
Claude Brezinski and Michela Redivo-Zaglia.
\newblock Transpose-free {L}anczos-type algorithms for nonsymmetric linear
systems.
\newblock {\em Numer. Algorithms}, 17(1):67--103, 1998.
\bibitem{chambolle2011first}
Antonin Chambolle and Thomas Pock.
\newblock A first-order primal-dual algorithm for convex problems with
applications to imaging.
\newblock {\em Journal of mathematical imaging and vision}, 40:120--145, 2011.
\bibitem{chan1998transpose}
Tony~F. Chan, Lisette de~Pillis, and Henk van~der Vorst.
\newblock Transpose-free formulations of {L}anczos-type methods for
nonsymmetric linear systems.
\newblock {\em Numer. Algorithms}, 17(1-2):51--66, 1998.
\bibitem{chouzenoux2023convergence}
Emilie Chouzenoux, Andr{\'e}s Contreras, Jean-Christophe Pesquet, and Marion
Savanier.
\newblock Convergence results for primal-dual algorithms in the presence of
adjoint mismatch.
\newblock {\em SIAM J. Imaging Sci.}, 16(1):1--34, 2023.
\bibitem{Chouzenout2021pgm-adjoint}
Emilie Chouzenoux, Jean-Christophe Pesquet, Cyril Riddell, Marion Savanier, and
Yves Trousset.
\newblock Convergence of proximal gradient algorithm in the presence of adjoint
mismatch.
\newblock {\em Inverse Probl.}, 37(6):No. 065009, 2021.
\bibitem{dudley2022}
Ethan Dudley, Arvind~K. Saibaba, and Alen Alexanderian.
\newblock Monte {C}arlo estimators for the {S}chatten {$p$}-norm of symmetric
positive semidefinite matrices.
\newblock {\em Electron. Trans. Numer. Anal.}, 55:213--241, 2022.
\bibitem{fikl2016comprehensive}
Alexandru Fikl, Vincent Le~Chenadec, Taraneh Sayadi, and Peter Schmid.
\newblock A comprehensive study of adjoint-based optimization of non-linear
systems with application to burgers’ equation.
\newblock In {\em 46th AIAA Fluid Dynamics Conference}, page 3805, 2016.
\bibitem{freund1993tfqmr}
Roland~W. Freund.
\newblock A transpose-free quasi-minimal residual algorithm for non-{H}ermitian
linear systems.
\newblock {\em SIAM J. Sci. Comput.}, 14(2):470--482, 1993.
\bibitem{golub2013}
Gene~H. Golub and Charles~F. Van~Loan.
\newblock {\em Matrix computations}.
\newblock Johns Hopkins Studies in the Mathematical Sciences. JHU Press,
Baltimore, MD, fourth edition edition, 2013.
\bibitem{gunzburger2002perspectives}
Max~D Gunzburger.
\newblock {\em Perspectives in flow control and optimization}.
\newblock SIAM, 2002.
\bibitem{huber2025convergence}
Richard Huber.
\newblock Convergence of ray-and pixel-driven discretization frameworks in the
strong operator topology.
\newblock {\em arXiv preprint arXiv:2503.03069}, 2025.
\bibitem{kenney1998}
C.~S. Kenney, A.~J. Laub, and M.~S. Reese.
\newblock Statistical condition estimation for linear systems.
\newblock {\em SIAM J. Sci. Comput.}, 19(2):566--583, 1998.
\bibitem{kong2017spectralestimation}
Weihao Kong and Gregory Valiant.
\newblock Spectrum estimation from samples.
\newblock {\em Ann. Statist.}, 45(5):2218--2247, 2017.
\bibitem{kuczynski1992estimating}
Jacek Kuczy{\'n}ski and Henryk Wo{\'z}niakowski.
\newblock Estimating the largest eigenvalue by the power and {L}anczos
algorithms with a random start.
\newblock {\em SIAM J. Matrix Anal. Appl.}, 13(4):1094--1122, 1992.
\bibitem{li2014sketching}
Yi~Li, Huy~L Nguyen, and David~P Woodruff.
\newblock On sketching matrix norms and the top singular vector.
\newblock In {\em Proceedings of the twenty-fifth annual ACM-SIAM symposium on
Discrete algorithms}, pages 1562--1581. SIAM, 2014.
\bibitem{lorenz2018randomized}
Dirk~A Lorenz, Sean Rose, and Frank Sch{\"o}pfer.
\newblock The randomized {K}aczmarz method with mismatched adjoint.
\newblock {\em BIT Numer. Math.}, 58(4):1079--1098, 2018.
\bibitem{lorenz2023chambolle}
Dirk~A. Lorenz and Felix Schneppe.
\newblock Chambolle-{P}ock’s primal-dual method with mismatched adjoint.
\newblock {\em Appl. Math. Opt.}, 87(2):22, 2023.
\bibitem{lorenz2023randomdescent}
Dirk~A. Lorenz, Felix Schneppe, and Lionel Tondji.
\newblock Linearly convergent adjoint free solution of least squares problems
by random descent.
\newblock {\em Inverse Probl.}, 39(12):No. 125019, 2023.
\bibitem{magdon2011note}
Malik Magdon-Ismail.
\newblock A note on estimating the spectral norm of a matrix efficiently.
\newblock {\em arXiv preprint}, 2011.
\bibitem{martinsson2002randomized}
Per-Gunnar Martinsson and Joel~A. Tropp.
\newblock Randomized numerical linear algebra: Foundations and algorithms.
\newblock {\em Acta Numer.}, 29:403–572, 2020.
\bibitem{ryu2022large}
Ernest~K Ryu and Wotao Yin.
\newblock {\em Large-scale convex optimization: algorithms \& analyses via
monotone operators}.
\newblock Cambridge University Press, 2022.
\bibitem{Savanier2021ProximalGA}
Marion Savanier, {\'E}.~Chouzenoux, J.~Pesquet, C.~Riddell, and Y.~Trousset.
\newblock Proximal gradient algorithm in the presence of adjoint mismatch.
\newblock {\em 2020 28th European Signal Processing Conference (EUSIPCO)},
pages 2140--2144, 2021.
\bibitem{sonneveld1989cgs}
Peter Sonneveld.
\newblock C{GS}, a fast {L}anczos-type solver for nonsymmetric linear systems.
\newblock {\em SIAM J. Sci. Statist. Comput.}, 10(1):36--52, 1989.
\bibitem{scikit-image}
{S}t\'efan van~der Walt, {J}ohannes~{L}. {S}ch\"onberger, {J}uan
{Nunez-Iglesias}, {F}ran\c{c}ois {B}oulogne, {J}oshua~{D}. {W}arner, {N}eil
{Y}ager, {E}mmanuelle {G}ouillart, {T}ony {Y}u, and the scikit-image
contributors.
\newblock scikit-image: image processing in {P}ython.
\newblock {\em PeerJ}, 2:e453, 6 2014.
\bibitem{zeng2000unmatched}
Gengsheng~L Zeng and Grant~T Gullberg.
\newblock Unmatched projector/backprojector pairs in an iterative
reconstruction algorithm.
\newblock {\em IEEE Trans. Med. Imaging.}, 19(5):548--555, 2000.
\end{thebibliography}