A Novel and Optimal Spectral Method for Permutation Synchronization
Extracted source text ยท arXiv 2303.12051
\begin{abstract}
% Permutation synchronization is an important task in different fields. To recover latent permutations from their noisy and incomplete pairwise measurements, spectral methods have gained increasing popularity in recent years for its simplicity and computational efficiency.
Permutation synchronization is an important problem in computer science that constitutes the key step of many computer vision tasks. The goal is to recover $n$ latent permutations from their noisy and incomplete pairwise measurements. In recent years, spectral methods have gained increasing popularity thanks to their simplicity and computational efficiency.
%. Spectral methods are simple and powerful approaches that have been widely used in permutation synchronization and have obtained growing popularity in recent years.
%They typically utilize the leading eigenspace of the data matrix and its block submatrices $U_1,U_2,\ldots, U_n$ and estimate latent permutations by projections of $U_j U_1^\top$ for all $j\geq 2$. However, the use of the block $U_1$ across different estimations amplify its error
% It utilizes the leading eigenspace of the data matrix and its block submatrices $U_1,U_2,\ldots, U_n$. In this paper, we propose a novel and statistically optimal spectral method for permutation synchronization. Different from existing spectral methods that are based on $\{U_jU_1^\top\}_{j\geq 2}$, we carefully construct an anchor matrix $M$ by aggregating useful information from the data and estimate latent permutations through $\{U_jM^\top\}_{j\geq 1}$. This overcomes a crucial limitation of existing methods caused by the repetitive use of $U_1$ and leads to a significantly improved numerical performance.
Spectral methods utilize the leading eigenspace $U$ of the data matrix and its block submatrices $U_1,U_2,\ldots, U_n$ to recover the permutations. In this paper, we propose a novel and statistically optimal spectral algorithm. Unlike the existing methods which use $\{U_jU_1^\top\}_{j\geq 2}$, ours constructs an anchor matrix $M$ by aggregating useful information from all the block submatrices and estimates the latent permutations through $\{U_jM^\top\}_{j\geq 1}$. This modification overcomes a crucial limitation of the existing methods caused by the repetitive use of $U_1$ and leads to an improved numerical performance.
% that overcomes a crucial limitation of existing ones in literature and outperforms them numerically.
%Our method utilizes the leading eigenspace of the data matrix and its block submatrices $U_1,U_2,\ldots, U_n$. While existing methods estimate latent permutations by projections of $U_j U_1^\top$
%
To establish the optimality of the proposed method, we carry out a fine-grained spectral analysis and obtain a sharp exponential error bound that matches the minimax rate.
\end{abstract}
%In this work, we study the phase synchronization problem where the latent objects belong to the permutation group and the pairwise measurements among them are noisy and missing at random.
%This stylized problem, known as the permutation group synchronization problem, has wide-ranging applications in computer vision.
% Among the various algorithms proposed in the literature, the spectral algorithm of Pachauri et al. (2013) is well known for its speed and good empirical performance. Despite these findings, the theoretical guarantee of the spectral algorithm has not been fully characterized. In this work, we propose a new spectral algorithm that overcomes a subtle yet crucial limitation of the vanilla spectral algorithm of Pachauri et al.. We prove that the proposed spectral algorithm has an error bound of $\ebr{-(1-o(1)) \frac{\text{SNR}}{2}}$. Thus, it achieves the asymptotic minimax error rate of $\ebr{-\frac{\text{SNR}}{2}}$. Our analysis produces a $\ell_{\infty}$-type bound on the blockwise deviation of the leading eigenspaces and a novel tail bound on this deviation.
%\keywords{Spectral Method, Permutation Synchronization, Leave-one-out}
In this section, we first give a detailed implementation of the proposed method. In Section \ref{sec:heuristic}, we provide intuitions to explain why it works and how it improves upon the existing algorithm. In Section \ref{sec:numerical}, we compare the numerical performances of the proposed method with the vanilla spectral method on synthetic datasets.
Algorithm \ref{alg:main-algo} describes our proposed spectral algorithm. The first step computes the top $d$ eigenvectors of the observation matrix $X$, which can be done using any off-the-shelf numerical eigendecomposition routine. The output of Step 1 is the empirical eigenspace $U \in \R^{nd\times d}$ corresponding to the top $d$ eigenvectors. Step 2 constructs the anchor $M$ used to recover the permutations by performing $d$-means clustering on the $nd$ rows of $U$ and extracting the estimated cluster centers. Step 3 recovers the underlying permutations via the Kuhn-Munkres algorithm. Note that (\ref{eqn:alg2}) in Step 3 can be interpreted as a projection of $U_j \hatH^\top$ onto $\Pi_d$. This is because all permutation matrices in $\Pi_d$ have the same Frobenius norm and (\ref{eqn:alg2}) can be equivalently written as
\begin{align}\label{eqn:alg3}
\hat Z_j = \argmin_{P\in\Pi_d}\fnorm{P - U_j M^\top}, \forall j\in[n].
\end{align}
\begin{algorithm}[ht]
\SetAlgoLined
\KwIn{Data matrix $X \in \mathr^{nd\times nd}$}
\KwOut{$n$ permutation matrices $\hat Z_1, \hat Z_2, \ldots, \hat Z_n\in\Pi_d$}
\nl Obtain the top $d$ eigenvectors of $X$ as $U \in \mathr^{nd \times d}$\;
\nl Denote the rows of $U$ as $v_1,\ldots,v_{nd}\in\mathr^{d\times 1}$ such that $U=(v_1^\top,\ldots, v_{nd}^\top)^\top$. Run $d$-means on $v_1,\ldots,v_{nd}$ and denote $\hat \mu_1,\ldots,\hat \mu_d\in\mathr^{d\times 1}$ to be cluster centers:
\begin{align}
(\hat \mu_1,\ldots,\hat \mu_d)\define\argmin_{\mu_1,\ldots,\mu_d\in\mathr^{d\times 1}} \min_{z\in[d]^{nd}} \sum_{j=1}^{nd}\norm{v_j - \mu_{z_j}}^2.\label{eqn:alg1}
\end{align}
Define $\hatH \define (\sqrt{n}\hat \mu_1^\top,\ldots,\sqrt{n}\hat \mu_d^\top)^\top\in\mathr^{d\times d}$ such that the rows of $\hatH$ are the $d$ centers multiplied by a $\sqrt{n}$ scaling\;
%are $\hat \mu_1,\ldots,\hat \mu_d$\;
\nl Compute
\begin{align}
\hat Z_j \define \argmax_{P\in\Pi_d} \inner{P}{U_j \hatH^\top} \label{eqn:alg2}
\end{align}
for each $i\in[n]$.
\caption{A new spectral method for permutation synchronization. \label{alg:main-algo}}
\end{algorithm}
\subsection{Intuitions}\label{sec:heuristic}
To understand Algorithm \ref{alg:main-algo}, we need to study $U$ through the lens of spectral perturbation theory. Denote
\begin{align}\label{eqn:U_star_def}
U^*\define (U^{*\top}_1,\ldots U^{*\top}_n)^\top \in\mathr^{nd\times n},\text{ where }U_j^* \define Z^*_j/\sqrt{n},\forall j\in[n].
\end{align}
Note that the expected value of the data matrix $X$ is equal to $\E X = p ((J_n - I_n) \otimes J_d) \circ Z^*(Z^*)^\top$. One could verify that $(\E X) U^* = (n-1)p U^*$. As a result, $U^*$ is the leading eigenspace of $\E X$ that includes its top $d$ eigenvectors. Since $X$ is a perturbed version of $\E X$, $U$ can also be seen as a perturbed version of $U^*$. However, this correspondence only holds with respect to an orthogonal transformation, as the leading eigenvalue of $\E X$, $(n-1)p $, has a multiplicity of $d$. That is, $UO\approx U^*$ for some $O\in\mathcal{O}_d$. As a result, the blocks $\{U_j\}$ are close to $\{U^*_j\}$ only up to a global orthogonal transformation, i.e., $U_jO\approx U^*_j$ for all $j\in[n]$.
To estimate the latent permutation $Z_j^*$, the vanilla spectral method uses the product $U_jU_1^\top$ as $U_jU_1^\top = (U_jO)(U_1O)^\top \approx U^*_j(U_1^*)^\top = Z^*_j(Z^*_1)^\top/n$. Intuitively, if $U_jU_1^\top$ is very close to $Z^*_jZ^*_1/n$ then by (\ref{eqn:old-spectral}), we have $\tilde Z_j = Z^*_j(Z^*_1)^\top$. As a result, when the perturbation between $UO$ and $U^*$ is sufficiently small, the vanilla spectral method is able to recover $Z^*$ up to a global permutation $Z^*_1$. However, as hinted at before, the use of the product $\{U_jU_1^\top\}$ in the vanilla spectral method inevitably leads to a crucial and fundamental limitation. For each $j\in[n]$, write $U_jO= U_j^*+\xi_j$ such that $\xi_j$ can be interpreted as the approximation `noise' of $U_jO$ with respect to $U_j^*$. Then
\begin{align*}
U_jU_1^\top = (U_jO)(U_1O)^\top = (U^*_j + \xi_j)(U^*_1 + \xi_1)^\top= U_j^*(U_1^*)^\top + \xi_j(U^*_1)^\top + U^*_j \xi_1^\top + \xi_j \xi_1^\top.
\end{align*}
That is, $ U_jU_1^\top$ is an approximation of $U_j^*(U_1^*)^\top$ with an error $\xi_j(U^*_1)^\top + U^*_j \xi_1^\top$ (the higher-order term $\xi_j \xi_1^\top$ is ignored). As a function of $ U_jU_1^\top$, the estimation accuracy of $\tilde Z_j$ is determined by both $\xi_j$ and $\xi_1$. As a result, the error caused by $\xi_1$ is carried forward in $\{\tilde Z_j\}_{j\geq 2}$ and impairs the numerical performance of the overall algorithm (see Figure \ref{fig:err}).
Additionally, using $U_1$ as the anchor makes the performance of the vanilla spectral algorithm less stable because it highly depends on the accuracy of $U_1$.
Algorithm \ref{alg:main-algo} overcomes this crucial limitation of the vanilla spectral method by constructing an anchor that `averages' information across all rows of $U$ instead of just using its first block $U_1$. The key insight into our construction is to recognize the special structure of the permutation synchronization problem where the true eigenspace $U^*$ consists of $d$ unique rows $e_1^\top/\sqrt n,\ldots, e_d^\top/\sqrt n$, each of cardinality exactly $n$.
% In addition, the euclidean distance between any two unique rows is the same and is rotationally invariant.
%Since $UO\approx U^*$, we have $U\approx U^*O^\top$.
The empirical eigenspace $U$, being a noisy estimate of $U^* O^\top$, thus exhibits clustering structures where the cluster centers are the transformed rows $\{e^\top_i O^\top/\sqrt n \}_{i=1}^d$
% This means rows of $U$ concentrates around $k$ rows of $O\top$ and consequently has a cluster structure.
Clustering algorithms such as the $d$-means algorithm can be used to approximate the cluster centers accurately. With $M$ being an accurate approximation of $O^\top$ (up to a permutation), we have
\begin{align*}
U_jM^\top \approx U_jO = U^*_j + \xi_j,
\end{align*}
and consequently the estimation accuracy of $\hat Z_j$ is only related to $\xi_j$. After proper scaling, the level of noise in $U_jM^\top $ is approximately half of that in $ U_jU_1^\top$ (i.e., $\xi_j$ only vs. $\xi_j$ and $\xi_1$). The reduced noise in $U_jM^\top$ leads to an improved and more stable numerical performance.
%This explains why
%\paragraph{A Novel and Optimal Spectral Algorithm.} In this work, we introduce a new spectral algorithm (Algorithm \ref{alg:main-algo}) that improves upon a subtle yet crucial limitation of the vanilla spectral algorithm in \cite{pachauri2013solving}. The limitation of the previous spectral method lies in the fact that it uses the first block $U_1$ as the anchor to perform the permutation recovery step in (\ref{eqn:old-spectral}). Its performance depends on how accurate $U_jU_1^\top$ is as an approximation of $U_j^*(U_1^*)^\top$. Note that each block $U_j$ itself is a noisy approximation of $U_j^*$. If the approximation error of $U_1$ is large, this amplifies the error of $ U_jU_1^\top$ for all subsequent block $j > 1$.
%
%On the other hand, one would think that using an anchor which `averages' information across all $n$ blocks of $U$ would lead to a more accurate final estimation because the approximation errors of many blocks tend to `cancel' one another. Building on this intuition, Algorithm \ref{alg:main-algo} carefully constructs an anchor $M$ by performing $d$-means clustering on the rows of the matrix $U$ and extracting the estimated cluster centers. Instead of using $U_1$ in (\ref{eqn:old-spectral}), the new algorithm uses $M$.
%\begin{align*}
%\hat Z_i = \argmin_{P\in\Pi(d)} \fnorm{P- U_i \hatH^\top}
%\end{align*}
%This crucial difference has important practical consequences. Figure \ref{fig:err} clearly shows that our spectral algorithm outperforms the vanilla algorithm in terms of the Hamming loss.
%
%The key insight into our construction of the anchor is to recognize the special structure of the permutation synchronization problem where the true eigenspace $U^*$ consist of $d$ clusters of rows, each of cardinality exactly $n$. Additionally, the euclidean distance between any two cluster centers is rotationally invariant. This means that the empirical eigenspace $U$, which approximate $U^*$ up to a global rotation, preserve this clustering structure. Global clustering algorithms such as the k-means algorithm can be used to approximate the cluster centers accurately. We then apply state-of-the-art clustering analysis to show that the new anchor enjoys a significantly better error guarantee than using single block $U_1$ as the anchor (cf. Proposition \ref{prop:hat_H_new}).
%This allows us to obtain a sharp error bound for our spectral algorithm, summarized by the following theorem.
%
%
%\paragraph{A Toy Model and Heuristics.} To develop an intuition for the inherent limitation of the vanilla spectral algorithm and the novel insight in the design of Algorithm \ref{alg:main-algo}, let us consider the setting where $Z_i^* = I_d \,\forall i \in [n]$ and $U_i^* = \frac{I_d}{\sqrt n} \,\forall i \in [n]$. Suppose \emph{just for the sake of building intuition} that each $d\times d$ block of the empirical eigenspace is an independent and noisy estimate of the corresponding true eigenspace modulo an appropriate rotation.
%\begin{align*}
% U_i = U_i^*O^\top + \xi_i \,\numberthis\label{eqn:heuristic}
%\end{align*}
%for some $O \in \O_d$ and matrix variate random variable $\xi_i$. In this setting, perfect recovery means to predict $\hat Z_1 = \hat Z_2 = \ldots = \hat Z_n = P$ for some $P \in \Pi_d$.
%
%One naive approach is to ignore the existence of $O$ and directly estimate the permutation matrix.
%\begin{align*}
% \hat Z_j = \argmax_{P \in \Pi_d} \inner{P}{U_j}\,. \numberthis\label{eqn:naive}
%\end{align*}
%The naive approach would fail if (\ref{eqn:naive}) produces different permutations for different blocks. Intuitively, this is the case when the output of the maximization step is sensitive to the perturbation $\xi_j$. Consider $Z \neq Z' \in \Pi_d$. We have
%\begin{align*}
% \inner{Z}{U_j} - \inner{Z'}{U_j} &= \inner{Z}{U_j^*O^\top + \xi_j} - \inner{Z'}{U_j^*O^\top + \xi_j}\\
% &= \inner{Z-Z'}{\frac{O^\top}{\sqrt n} + \xi_j} \\
% &= \inner{Z-Z'}{\frac{O^\top}{\sqrt n}} + \inner{Z-Z'}{\xi_j}\numberthis\label{eqn:naive-decomp}\,.
%\end{align*}
%When $O$ is a `diffused' matrix -- $\max_{i, j}\abs{O_{ij}}$ is small -- as opposed to a `concentrated' matrix such as a permutation matrix, $\inner{Z-Z'}{\frac{O^\top}{\sqrt n}}$ is small. On the other hand, any dense perturbation matrix $\xi_j$, such as the matrix variate normal distribution, would lead to a high variance of (\ref{eqn:naive-decomp}). This means that the optimal permutation of (\ref{eqn:naive}) can vary significantly between $j=1,\ldots, n$.
%
%The vanilla spectral algorithm of \cite{pachauri2013solving} overcomes the limitation of the naive algorithm by removing the effect of $O$. We have
%\begin{align*}
% U_jU_1^\top &= (U^*_jO^\top + \xi_j)(U^*_1O^\top + \xi_1)\\
% &= {U_j^*(U_1^*)^\top} + \underbrace{\xi_j U^*_1O^\top + U^*_jO^\top \xi_1 + \xi_j \xi_1}_{\text{perturbation}}\,.
%\end{align*}
%However, note that there are now two additive error terms ${\xi_jO^\top}$ and ${O^\top \xi_1}$. While seemingly innocuous, the presence of one additional error term actually leads to the performance difference between the naive spectral algorithm and ours.
%
%Now, suppose that we knew $O$, then a good solution would be to estimate $\hat Z_i$ using $O$ as follows.
%\begin{align*}
%\hat Z_j = \argmax_{P\in \Pi_d} \inner{P}{U_jO}\,. \numberthis\label{eqn:known-O}
%\end{align*}
%Note that ${U_jO} = {U_j^* + \xi_j O}$. Compared to the vanilla spectral approach, the estimate here only has one additive error term. Therefore, the estimate (\ref{eqn:known-O}) should have a lower error than (\ref{eqn:old-spectral}). Of course in practice, we do not observe $O$ exactly.
%
%However, we found the surprising fact that we can still estimate $O$ well up to a permutation using the anchor $M$ of Algorithm \ref{alg:main-algo}. In our setting where $U_j^* = \frac{I_d}{\sqrt n} \,\forall j \in [n]$, the true eigenspace $U^*$ contain $d$ types of rows $\{e_1^\top, \ldots, e_d^\top\}$ each of cardinality exactly $n$. Furthermore, the euclidean distance between any two types of rows is preserved under the rotation transformation $O^\top$. The empirical eigenspace $U$, being noisy estimates of $U^* O^\top$, thus exhibit clustering structures where the cluster centers are the transformed rows $\{e^\top_i O^\top \}_{i=1}^d$. This allows us to prove that the $d$-means clustering algorithm recovers, with high accuracy, the transformed cluster centers -- $\min_{P\in \Pi_d}\fnorm{M - PO^\top}$ is small. This means that the performance of the proposed spectral algorithm is close to that of (\ref{eqn:known-O}) and better than the vanilla approach.
\subsection{Numerical Analysis}\label{sec:numerical}
To verify our intuitions and showcase the improved performance of our spectral method over the vanilla spectral method, we perform experiments on synthetic data following the model in (\ref{eqn:Xjk-def}). We fix $d = 2, p = 0.5, n = 2048$ and vary $\sigma$ to control the signal-to-noise ratio. Figure \ref{fig:hamming} shows the normalized Hamming loss $\ell$ against decreasing $\sigma$ for the two algorithms. The two lines are the average over 100 independent trials. One can see that our method outperforms the vanilla spectral method with a significantly smaller error. %Furthermore, our algorithm is significantly more robust to input perturbations than the vanilla algorithm.
The box plot in Figure \ref{fig:box-plot} shows the distribution of the losses across 100 trials for both methods. The boxes extend from the first quartile to the third quartile of the observed losses with colored bolded lines at the medians and dotted points at the outliers. It is clear that the performance of the vanilla algorithm is highly variable across trials. On the contrary, the performance of our method is more concentrated, indicating that our method has a smaller variance and is more precise. Figure \ref{fig:err} reflects our intuition that because the vanilla algorithm repeatedly uses $U_1$ to estimate the latent permutations, its performance is highly dependent on the approximation error of a single block $U_1$. On the other hand, because our algorithm constructs an anchor that averages information across all $n$ blocks, it consistently outperforms the vanilla algorithm with a much tighter error spread.
\begin{figure}[ht]
\begin{subfigure}[t]{0.45\textwidth}
\centering
\includegraphics[width=1\linewidth]{figs/hamming.png}
\caption{
%Normalized Hamming loss against $\sigma$.}
Average over 100 trials: our spectral algorithm outperforms the vanilla algorithm with a smaller error.
%The lines show the average over 100 trials.
\label{fig:hamming}}
\end{subfigure}
% \hfill
\hspace{0.5cm}
\begin{subfigure}[t]{0.45\textwidth}
\centering
\includegraphics[width=1\linewidth]{figs/box_plot_err.png}
\caption{
%{Box plot of normalized Hamming losses.}
Box plot: our algorithm also has a smaller variance compared to the vanilla method. \label{fig:box-plot}}
\end{subfigure}
% ~\\
% \begin{subfigure}[b]{0.45\textwidth}
% \includegraphics[width=1\linewidth]{figs/pairwise_loss.png}
% \caption{\textbf{$\frac{1}{n(n-1)} \fnorm{ZZ^\top - Z^*(Z^*)^\top}^2$ against $n$.} \label{fig:pairwise}}
% \end{subfigure}
% \begin{subfigure}[b]{0.6\textwidth}
% \includegraphics[width=1\linewidth]{figs/phase_transition.png}
% \caption{\textbf{Phase transition for exact recovery}. \duc{Thinking about adding this kind of plot similar to Figure 2 in Shuyang Ling's paper. However, need to do more experiments with larger $n$ to cleanly see the phase transition. x-axis is $n$ while y-axis is signal to noise ratio and the color denotes the frequency of exact recovery. The goal is to have two figures for vanilla (left) and ours (right) where the phase transition threshold is clearly different.}\label{fig:exact}}
% \end{subfigure}
% \hspace{0.5cm}
\caption{{Comparisons between our method and the vanilla spectra method on synthetic data.} \label{fig:err}}
\end{figure}
% Group synchronization is a class of problems in various field where the goal is to identify $n$ objects based on pairwise measurements among them. In this work, we study an important subclass of the group synchronization known as the permutation synchronization problem.
% with additive Gaussian noises and incomplete observations.
% Since being popularized in \cite{pachauri2013solving}, permutation synchronization has become a fundamental problem in computer vision. The goal is to find a bijection between sets of features across multiple images. This is a key procedure in image registration \cite{shen2002hammer}, shape matching \cite{berg2005shape}, detecting structures from motion \cite{agarwal2011building}, among many other tasks in image processing.
In permutation synchronization, the objective is to estimate latent permutations using noisy and potentially incomplete pairwise measurements among them.
% Permutation synchronization is to estimate latent permutations from pairwise measurements among them.
It is an important task in computer vision and graphics where finding correspondence between sets of features across multiple images is a fundamental problem with wide-ranging applications
including image registration \cite{shen2002hammer}, shape matching \cite{berg2005shape}, multi-view matching \cite{gao2021isometric, maset2017practical}, detecting structures from motion \cite{agarwal2011building}, and so on. Various methods have been proposed for permutation synchronization including iterative algorithms \cite{gao2022iterative, zhou2015multi,chen2018projected}, semi-definite programming (SDP) \cite{huang2013consistent, chen2014near}, and spectral methods \cite{pachauri2013solving,ling2022near,maset2017practical, shen2016normalized}.
Compared to other approaches, spectral methods have gained increasing popularity and have been widely used in permutation synchronization thanks to their simplicity, fast computation speed, and impressive numerical performance. Despite the popularity, it remains unclear how well spectral methods perform theoretically and whether they achieve statistical optimality or not. In this paper, we address these questions by proposing a new and provably optimal spectral algorithm.
%SDP or iterative algorithms, the spectral algorithm is quite fast as eigendecomposition can be accomplished using highly optimized, off-the-shelf solvers. In fact, the spectral algorithm has been leveraged by previous authors in designing iterative algorithms that use the spectral estimate as initialization \cite{shen2016normalized,gao2022iterative}. However, the practical performance of \emph{the spectral algorithm alone is impressive} and the performance gain of the subsequent refinement procedure is often marginal. These empirical findings raise the following questions: Is it possible to obtain a sharp error guarantee for the spectral method and whether just the spectral method alone is optimal? The answer is yes but with a twist: we are not using the vanilla spectral algorithm of \cite{pachauri2013solving}. Our affirmative answers to these questions come with a \emph{new and minimax optimal} spectral algorithm.
%\paragraph{Problem Formulation and the Vanilla Spectral Algorithm.}
\paragraph{Problem Formulation.}
The permutation synchronization problem is formulated as follows.
Let $Z_1^*, \ldots, Z_n^* \in \Pi_d$ where $\Pi_d$ is the permutation group in $d$ dimension defined as:
\begin{equation}\label{eqn:Pi-def}
\Pi_d := \left\{ P \in \{0,1\}^{d\times d} \,:\, P^\top P = PP^\top = I_d \right\}\,.
\end{equation}
We introduce missing and noisy data by assuming that, for each pair $1\leq i < j \leq n$, the observation $X_{jk} \in \R^{d\times d}$ satisfies
\begin{equation}\label{eqn:Xjk-def}
X_{jk} \define \begin{cases} Z_i^*(Z_j^*)^\top + \sigma W_{jk}, &\text{if } A_{jk} = 1,\\
0_{d\times d}, &\text{otherwise},
\end{cases}
\end{equation}
where $A_{jk} \in \{0,1\}$ follows $ \text{Bernoulli}(p)$ for some $p \in \left(0, 1\right]$; $\sigma \in \R^+$ controls the amount of noise; $W_{jk} \in \R^{d\times d}$ is a random matrix with each entry following the standard normal distribution independently; and $0_{d\times d}$ is the $d\times d$ matrix of all zeros. Roughly speaking, each $X_{jk}$ block, if not missing, is a noisy measurement of $Z_i^*(Z_j^*)^\top$, the `difference' between two permutation matrices. We assume that all random variables $\{ A_{jk} \}_{1\leq j < k \leq n}, \{W_{jk}\}_{1\leq j < k\leq n}$ are independent. Denote $Z^*\define (Z_1^{*\top},\ldots, Z_n^{*\top})^\top\in \Pi_d^n$. The goal is to estimate $Z^*$ given $\{ A_{jk} \}_{1\leq j < k \leq n}$ and $ \{X_{jk}\}_{1\leq j < k\leq n}$.
Note that $Z_1^*,\ldots, Z^*_n$ are identifiable only up to a global permutation. For any estimator $\hat Z =(\hat Z_1^\top, \ldots, \hat Z_j^\top)^\top\in \Pi_d^n$, its performance can be measured by the following normalized Hamming loss (modulo a global permutation transformation):
\begin{equation}\label{eqn:hamming}
\ell(\hat Z, Z^*) \define \min_{P \in \Pi_d} \,\frac{1}{n} \sum_{j=1}^n \indic{\hat Z_j \neq Z_j^* P^\top} \,.
\end{equation}
Note that the model (\ref{eqn:Xjk-def}) has a matrix representation. % In matrix notations, one can write
The observation matrix $X \in \R^{nd\times nd}$ can be written as
\begin{equation}\label{eqn:X-def}
X = (A \otimes J_d) \circ Z^*(Z^*)^\top + \sigma (A\otimes J_d) \circ W,
\end{equation}
where $W \in \R^{nd\times nd}$ is a block-symmetric matrix with $W_{kj} = W_{jk}^\top, \forall 1\leq j < k \leq n$; $A \in\{0,1\}^{n\times n}$ is a symmetric matrix with $A_{kj}=A_{jk},A_{jj} = 0,\forall 1\leq j < k \leq n$; $J_d$ is the $d\times d$ matrix of all ones; $\otimes$ is the Kronecker product and $\circ$ is the Hadamard product.
\paragraph{Spectral Methods.}
Existing spectral methods \cite{pachauri2013solving,ling2022near,maset2017practical, shen2016normalized} use the eigendecomposition of $X$ followed by a rounding step to estimate the latent permutations. Let $U=(U_1^\top,\ldots, U_n^\top)^\top \in \R^{nd\times d}$ be the matrix composing the top $d$ eigenvectors of $X$. That is, the columns of $U$ are the eigenvectors corresponding to the $d$ largest eigenvalues of $X$. Its $d\times d$ blocks are denoted as $U_1,\ldots, U_n$. The first block $U_1 \in \R^{d\times d}$ is then used as an `anchor' to obtain an estimator $\tilde Z_1, \ldots, \tilde Z_n \in \Pi_d$:
\begin{equation}\label{eqn:old-spectral}
\tilde Z_1 \define I_d, \quad \tilde Z_j \define \argmax_{P\in \Pi_d} \inner{P}{U_jU_1^\top} ,\forall j = 2,\ldots , n,
\end{equation}
where the optimization subproblem serves to round $U_jU_1^\top$ into a permutation matrix and can be efficiently solved using the Kuhn-Munkres algorithm \cite{kuhn1955hungarian} (see Section \ref{sec:heuristic} for an intuitive description of the innerworking of this algorithm). To distinguish the existing algorithm from the one proposed in this paper, we refer to $\tilde Z:=(\tilde Z_1^\top,\ldots ,\tilde Z_n^\top)^\top$ as the `vanilla spectral estimator'.
%The vanilla spectral method $\tilde Z$
It is computationally efficient and has decent numerical performance.
Despite of all the aforementioned advantages, the vanilla spectral method suffers
%a crucial limitation as $U_1$ is used
from the repeated use of $U_1$ in constructing the estimator $\tilde Z_j$ for all $j\geq 2$. From a perturbation theoretical point of view, $U$ (along with its blocks $U_1,\ldots, U_n$) are approximations of their population counterparts. By using $U_jU_1^\top$, the estimation accuracy of $\tilde Z_j$ is determined by the approximation errors of both $U_j$ and $U_1$. As a result, the approximation error of $U_1$ is carried forward in $\{\tilde Z_j\}_{j\geq 2}$ and deteriorates the overall numerical performance.
To overcome this crucial limitation of $\tilde Z$, we propose a new spectral method that avoids the use of $U_1$ as the anchor. Instead, we construct an anchor matrix $M\in\mathr^{d\times d}$ by carefully aggregating useful information from all of $U$ and estimate the latent permutations by $\hat Z \define (\hat Z_1^\top, \ldots, \hat Z_n^\top)^\top$ where
\begin{align*}
\hat Z_j \define \argmax_{P\in \Pi_d} \inner{P}{U_jM^\top} ,\forall j\in[n].
\end{align*}
The construction of $M$ is built on an intuition that `averaging' information across all $n$ blocks of $U$ leads to a much more accurate anchor than $U_1$ because the approximation errors of many blocks tend to `cancel' one another. As a result, the estimation accuracy of $\hat Z_j$ is largely determined by the approximation error of $U_j$ only, which leads to an improved numerical performance. See Algorithm \ref{alg:main-algo} for the detailed implementation of the proposed method and Figure \ref{fig:err} for comparisons of numerical performances between the vanilla spectral estimator and ours.
\paragraph{Statistical Optimality.} By carrying out fine-grained spectral analysis, we establish a sharp upper bound for the theoretical performance of the proposed method, summarized below in Theorem \ref{thm:intro}. See Theorem \ref{thm:main_new} for its non-asymptotic version.
\begin{theorem}\label{thm:intro}
Assume $\frac{np}{\sigma^2}\rightarrow\infty$, $\frac{np}{\log^3 n} \rightarrow\infty$, and $2\leq d = O(1)$. Then the proposed spectral method $\hat Z$ satisfies
\begin{align*}
\E \ell(\hat Z, Z^*) \leq \ebr{-(1-o(1)) \frac{np}{2\sigma^2}} + n^{-8}\,.
\end{align*}
\end{theorem}
The upper bound in Theorem \ref{thm:intro} consists of an exponential error term and a polynomial error term $n^{-8}$. Note that by properties of the normalized Hamming loss $\ell$, the polynomial error term is negligible. Considering this, our spectral algorithm achieves the minimax lower bound \cite{gao2022iterative} which states that if $\frac{np}{\sigma^2} \rightarrow \infty$, then $\inf_{Z'} \sup_{Z^*\in\Pi_d^n} \,\E \ell(Z', Z^*) \geq \ebr{-(1+o(1))\frac{np}{2\sigma^2}}$. This establishes the statistical optimality of the proposed method for the partial recovery of the latent permutations. Theorem \ref{thm:intro} immediately implies the threshold for exact recovery. When $np/(2 \sigma^2) >(1+\epsilon)\log n$ for some constant $\epsilon>0$, we have $\ell(\hat Z,Z^*)=0$ holds with high probability. According to the minimax lower bound, no estimator is able to recover $Z^*$ exactly with vanishing error if $np/(2 \sigma^2) <(1-\epsilon)\log n$. As a result, simple but powerful, the proposed spectral method $\hat Z$ is an optimal procedure.
Theorem \ref{thm:intro} allows the observations $\{X_{jk}\}$ to be missing at random as long as the probability $p$ satisfies $np\gg \log^3 n$. Note that in order to have a connected comparison graph $A$, $np$ needs to be at least of order $\log n$. Compared to this condition, our assumption $np\gg \log^3 n$ requires some additional logarithm factor. The assumption $np \gg \sigma^2$ is the necessary and sufficient condition to achieve estimation consistency according to the minimax lower bound. Theorem \ref{thm:intro} assumes that $d$, the dimension of each permutation matrix, is a constant.
To establish Theorem \ref{thm:intro}, we first provide a block-wise $\ell_\infty$ perturbation analysis for all block submatrices $U_1,\ldots, U_n$, quantifying the maximum deviation between them and their population counterparts (see Theorem \ref{thm:l_infty_new}). In addition, we give a theoretical justification for the usage of the anchor $M$ by showing that it achieves a negligible error (see Proposition \ref{prop:hat_H_new}). With both results, we investigate the tail behavior of each $U_jM^\top$ and eventually obtain the upper bound in Theorem \ref{thm:intro}. We leverage the leave-one-out technique \cite{bean2013optimal, abbe2020entrywise} in our proofs.
%To solve the permutation synchronization problem is to estimate the underlying permutation matrices $Z_1^*,\ldots, Z_n^*$. The reader may immediately recognize an inherent identification problem. That is, for any permutation matrix $P' \in \Pi_d$, the two distributions over $X$ parametrized by $\{Z^*_1(P')^\top, \ldots, Z^*_n(P')^\top\}$ and $\{Z_1^*, \ldots, Z_n^* \}$ are statistically indistinguishable. Hence, we can only hope to identify the matrices $\{Z_1^*, \ldots, Z_n^*\}$ up to a global permutation and the metric of interest is the Hamming loss (modulo an optimal permutation transformation).
%The vanilla spectral algorithm \cite{pachauri2013solving,ling2022near} performs an eigendecomposition of the observation matrix $X \approx U\Lambda U^\top$ where $U \in \R^{nd\times d}$ is the top $d$ eigenvectors of $X$. Define $U_j$ to be the $j$-th $d\times d$ block of $U$. The first block $U_1 \in \R^{d\times d}$ of $U$ is then used as the `anchor' to obtain the estimate $\tilde Z_1, \ldots, \tilde Z_n \in \Pi_d$. Specifically,
%\begin{equation}\label{eqn:old-spectral}
%\tilde Z_1 \define I_d, \quad \tilde Z_j \define \argmax_{P\in \Pi_d} \inner{P}{U_jU_1^\top} \,\,\text{for }\, j = 2,\ldots , n
%\end{equation}
%where $U_j \in \R^{d\times d}$ is the $j$-th block of $U$ and the optimization subproblem can be efficiently solved using the Kuhn-Munkres algorithm \cite{kuhn1955hungarian}. To see intuitively why this spectral algorithm works, note that $\E[X] = (\E[A] \otimes J_d) \circ Z(Z^*)^\top = p ((J_n - I_n) \otimes J_d) \circ Z(Z^*)^\top$. One could verify that $U^* \in \R^{nd\times d}$, of which the $j$-th $d\times d$ block is $U_j^* \define \frac{Z^*_j}{\sqrt n}$, is the top-$d$ eigenspaces of $\E[X]$. Since $X$ is a perturbed version of $\E[X]$, $U$ is a perturbed version of $U^*$ up to a global rotation. That is, $U \approx U^* O^\top$ for some $O \in \O_d$. We have $U_jU_1^\top \approx (U_j^*O^\top)(U_1^*O^\top)^\top \approx \frac{Z_j^*(Z_1^*)^\top}{n}$. When the amount of noise is low, the approximation is accurate and the maximization step in (\ref{eqn:old-spectral}) returns the correct permutation. Naturally, the performance of this approach degrades when we introduce missing data and a higher amount of noise in (\ref{eqn:X-def}).
\paragraph{Related Literature.}
%The work \cite{ling2022near} studies the exact recovery of the vanilla spectral method under a different permutation synchronization model where there is no missing data and each observation is corrupted by a random permutation matrix with probability $q$. It shows that the vanilla spectral method achieves the exact recovery when $q$ satisfies certain condition. As a comparison, our analysis allows the signal-to-noise ratio to be small when the exact recovery is impossible and only the partial recover is possible. We further establish the statistical optimality of our method in the presence of missing data and Gaussian noise.
%As a comparison, our model allows missing data and Gaussian noise. More importantly, we
Permutation synchronization belongs to a broader class of group synchronization problems where the goal is to identify $n$ group objects based on pairwise measurements among them. In recent years, spectral methods have been widely used and studied in group synchronization problems. In \cite{zhang2022exact}, spectral methods are proved to be optimal for phase synchronization and orthogonal group synchronization in terms of squared $\ell_2$ losses. To obtain this result, \cite{zhang2022exact} develops perturbation analysis toolkits to show that the leading eigenstructures can be well-approximated by its first-order approximation with a small $\ell_2$ error. However, the difference that permutations are discrete-valued while phases and orthogonal matrices are continuous is critical. For permutation synchronization, instead of $\ell_2$ perturbation analysis, we need to develop block-wise analysis in order to obtain sharp exponential rates. \cite{abbe2020entrywise} considers a $\mathbb{Z}_2$ synchronization problem where each object is $\pm 1$ and assumes that there is no missing data. It proves that a simple spectral procedure using signs of coordinates of the first eigenvector of the data matrix achieves the optimal threshold for the exact recovery of objects by $\ell_\infty$ analysis of the leading eigenvector. \cite{ling2022near} extends \cite{abbe2020entrywise}'s $\ell_\infty$ analysis to a permutation synchronization setting where there is no missing data and each observation is corrupted with probability $q$ by a random permutation matrix. It shows that the vanilla spectral method achieves exact recovery when $q$ satisfies certain conditions. Our $\ell_\infty$ analysis is different from those in \cite{abbe2020entrywise, ling2022near} as we need to consider the low signal-to-noise ratio regime where exact recovery is impossible but partial recovery is possible. We go beyond $\ell_\infty$ analysis and study the tail behavior of each block of $U$ in order to obtain exponential error bounds for partial recovery. In addition, the presence of missing data in our model complicates the theoretical analysis as the magnitude and tail behavior of each block $U_j$ is not only related to the additive Gaussian noises but also the randomness of the Bernoulli random variables $\{A_{jk}\}_{k\neq j}$.
% For permutation synchronization, since the leading eigenspace $U$ is used, we need to carry out a block-wise $\ell_\infty$ analysis for $U$. Our $\ell_\infty$ analysis is also different from that in \cite{abbe2020entrywise} as we allow the signal-to-noise ratio to be small such that the exact recovery is impossible but partial recovery is possible.
%Built upon the block-wise $\ell_\infty$ analysis, we further go beyond study tail behaviors for each block of $U$
%spectral methods need to use with a more complicated rounding steps. As a result,
%\paragraph{Related Literature.} The permutation synchronization problem is a subclass of the phase synchronization problem. Other subclasses of the phase synchronization problem and their applications include, but are not limited to, $\Z_2$ synchronization in community detection \cite{abbe2014decoding,bandeira2016low}; $\Z_k$ synchronization in joint alignment \cite{chen2018projected}; orthogonal group synchronization in angular estimation \cite{arie2012global}. There are a variety of proposed algorithms for solving the phase synchronization problem including maximum likelihood estimate (MLE) \cite{boumal2013robust,gao2021exact,zhong2018near}, generalized power method (GPM) \cite{gao2021optimal,ling2022improved,perry2018message,liu2017estimation,shen2016normalized,gao2022iterative}, semi-definite programming (SDP) \cite{arie2012global,fan2021joint,gao2022sdp,ling2022solving,javanmard2016phase,singer2011three,wang2013exact} and spectral methods \cite{arie2012global,romanov2020noise,singer2011three}.
%
%For permutation synchronization, it has been shown that an instance of Loyld's iterative minimization algorithm achieves the minimax optimal rate with respect to the Hamming loss \cite{gao2022iterative}. Recently, \cite{ling2022near} studied the \emph{exact recovery guarantee} of the vanilla spectral algorithm for permutation synchronization and showed that its error guarantee is near-optimal up to a logarithmic and a constant factor. As mentioned briefly, the spectral method is often used as an initialization routine to more complicated procedures based on GPM or Loyld's algorithm. However, in practice, the performance of the spectral method is already good and the performance gain from GPM or Loyld's algorithm is often marginal \cite{zhang2022exact}. Our work fills an important gap in the literature by showing that the spectral estimates alone are in fact optimal up to the correct constant factor.
%
%Our paper also contributes to the theoretical analysis of the spectral algorithm. Fine-grained analysis of eigenspace perturbation has received substantial attention in recent years. For example, $\ell_{\infty}$ analysis of the leading eigenvector and $\ell_{2,\infty}$ analysis of the leading eigenspace allow statisticians to obtain improved guarantees in a variety of problems such as community detection \cite{zhang2023fundamental}, phase synchronization \cite{zhang2022exact} and joint alignment \cite{chen2018projected}. In the permutation synchronization problem, the generalization of the $\ell_{\infty}$ analysis is the blockwise error analysis, i.e., bounding $\min_{O\in \O_d} \max_{j\in [n]}\opnorm{U_j - U_j^* O}$ \cite{ling2022near,zhang2022exact}. However, to obtain the exponentially decaying error guarantee in Theorem \ref{thm:intro}, we require more than just a high probability blockwise deviation bound which incurs an additional logarithmic factor. We go further than previous works by furnishing a tail bound on the blockwise deviation.
% \paragraph{Organization.} We describe the new spectral algorithm in Section \ref{sect:algo}. In Section \ref{sect:analysis} we will present the theoretical analyses of the spectral algorithm. \duc{Finish after full proofs}
\paragraph{Notation.} For any positive integer $n$, define $[n] := \{1,2,\ldots, n\}$. Let $I_d$ denote the $d\times d$ identity matrix and $J_d$ denote the $d\times d$ matrix of all ones. Define $\mathcal{O}_d:=\{O\in\mathr^{d\times d}:OO^\top=O^\top O=I_d\}$ to be the set of all $d\times d$ orthogonal matrices.
Given $a, b\in \R$, let $a\lor b \define \max\{a, b\}$ and $a\land b \define \min\{a, b\}$. For a matrix $B \in \R^{d_1\times d_2}$, the Frobenius norm and the operator norm of $B$ are defined as $\fnorm{B} \define \left(\sum_{i=1}^{d_1}\sum_{j=1}^{d_2} B_{ij}^2 \right)^{1/2} $ and $\opnorm{B} \define \max_{u \in \S_{d_1}, v\in \S_{d_2}} u^\top B v $ where $\S_d \define \{v \in \R^d \,:\, \norm{v} = 1\}$ is the unit sphere in $d$ dimension and $\norm{.}$ is the Euclidean norm. For two matrices $A, B \in \R^{d_1\times d_2}$, let $\inner{A}{B} := \sum_{i\in [d_1]}\sum_{j\in [d_2]} A_{ij}B_{ij}$ denote the matrix inner product. For some $d_1, d_2 \in \mathbb{N}$, we use $\calN(\mu, \Sigma)$ to denote the normal distribution with mean $\mu \in \R^{d_1}$ and covariance $\Sigma \in \R^{d_1 \times d_1}$ and $\MN(S, \Sigma_1, \Sigma_2)$ to denote the matrix normal distribution \cite{de2004matrix} with mean parameter $S\in \R^{d_1\times d_1}$ and covariance parameters $\Sigma_1 \in \R^{d_1\times d_1}$, $\Sigma_2 \in \R^{d_2\times d_2}$. Define $(a)_+ \define \max\{a, 0\}$. For the rest of the paper, we will use $e_i \in \{0,1\}^d$ to denote the vector with $1$ in the $i$-th entry and zero everywhere else. For any matrix $Y$ and any positive integer $i$, we denote $\lambda_i(Y)$ to be the $i$th largest eigenvalue of $Y$.
%Unless specified otherwise, we use $\Lambda$ to denote the $d\times d$ diagonal matrix whose entries are the top eigenvalues of $X$.
For any positive sequences $a_n,b_n$, we use $a_n\lesssim b_n$ and $b_n\gtrsim a_n$ if $a_n\leq C b_n$ for some constant $C>0$.
Lastly, we use $\indic{.}$ to denote the indicator function.
\documentclass[11pt]{article}
\usepackage{amsmath, amsfonts, amssymb, amsthm, graphicx, enumerate}
\usepackage[letterpaper, left=1truein, right=1truein, top = 1truein, bottom = 1truein]{geometry}
\usepackage[numbers, square]{natbib}
\usepackage[%dvips,
CJKbookmarks=true,
bookmarksnumbered=true,
bookmarksopen=true,
colorlinks=true,
citecolor=red,
linkcolor=blue,
anchorcolor=red,
urlcolor=blue,
]{hyperref}
\usepackage[usenames,dvipsnames]{color}
\newcommand{\red}{\color{red}}
\newcommand{\blue}{\color{blue}}
\newcommand{\violet}{\color{violet}}
\newcommand{\brown}{\color{brown}}
\newcommand{\magenta}{\color{magenta}}
\usepackage[ruled, vlined, lined, commentsnumbered]{algorithm2e}
\usepackage{prettyref,soul,xspace}
\usepackage{tikz}
\usepackage{pgfplots,makecell}
\newcommand{\nb}[1]{{\sf\magenta[#1]}}
\newcommand{\nbr}[1]{{\sf\red[#1]}}
\newcommand{\rev}[1]{{#1}}
% My packages and new commands
\usepackage{bm}
\usepackage{fancyhdr}
\pagestyle{fancy}
\theoremstyle{remark}
\newtheorem{remark}{Remark}
\theoremstyle{plain}
\newtheorem{problem}{Problem}
\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}
\newtheorem{proposition}{Proposition}
\newtheorem{corollary}{Corollary}
\newcommand{\p}{\mathbb{P}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\En}{\mathbb{E}_n}
\newcommand{\Var}[1]{\text{Var} \left(#1\right)}
\newcommand{\Ber}{\text{Ber}}
\newcommand{\Poi}{\text{Poisson}}
\newcommand{\iid}{\stackrel{iid}{\sim}}
\newcommand{\dequal}{\stackrel{d}{=}}
\newcommand{\br}[1]{\left( #1 \right)}
\newcommand{\sbr}[1]{\left[ #1 \right]}
\newcommand{\cbr}[1]{\left\{ #1 \right\}}
\newcommand{\pbr}[1]{\p\left( #1 \right)}
\newcommand{\ebr}[1]{\exp\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\sign}[1]{\text{ sign}\left( #1 \right)}
\newcommand{\trace}[1]{\text{tr}\left( #1 \right)}
\newcommand{\tr}[1]{\text{tr}\br{#1}}
\newcommand{\rank}[1]{\text{rank}\br{#1}}
\newcommand{\sspan}[1]{\text{span}\br{#1}}
\newcommand{\kl}{\text{KL}}
\newcommand{\Binom}{\text{Binom}}
\newcommand{\mathr}{\mathbb{R}}
\newcommand{\mathn}{\mathcal{N}}
\newcommand{\mathf}{\mathcal{F}}
\newcommand{\mathh}{\mathcal{H}}
\newcommand{\diff}{{\rm d}}
\newcommand{\addbox}[1]{\noindent\fbox{\begin{minipage}{\textwidth} #1 \end{minipage}}}
\newcommand{\indic}[1]{{\mathbb{I}\left\{{#1}\right\}}}
\newcommand{\iprod}[2]{\left \langle #1, #2 \right\rangle}
\newcommand{\norm}[1]{\left\|{#1} \right\|}
\newcommand{\normt}[1]{\|{#1} \|}
\newcommand{\opnorm}[1]{\norm{#1}_{\rm op}}
\newcommand{\opnormt}[1]{\normt{#1}_{\rm op}}
\newcommand{\opnorminf}[1]{\norm{#1}_{\rm op,\infty}}
\newcommand{\lonenorm}[1]{\norm{#1}_{\rm 1}}
\newcommand{\fnorm}[1]{\norm{#1}_{\rm F}}
\newcommand{\argmin}{\mathop{\rm argmin}}
\newcommand{\argmax}{\mathop{\rm argmax}}
\newcommand{\define}{:=}
\newcommand{\definei}{=:}
\newcommand{\hatH}{M}
\newcommand{\Z}{\mathbb{Z}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%
% EXTRA DEFINITION AND PACKAGES
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{subcaption}
% \usepackage{subfigure}
\usepackage{bbm}
\newcommand{\duc}[1]{\textcolor{red}{Duc: {#1}}}
\newcommand{\loo}[1]{{{#1}^{(j)}}}
\newcommand{\calE}{\mathcal{E}}
\newcommand{\MN}{\mathcal{MN}}
\renewcommand{\v}{\mathbf{v}}
\newcommand{\calN}{\mathcal{N}}
\renewcommand{\vec}{\text{vec}}
\newcommand{\zero}{0}
\newtheorem{coro}{Corollary}
\newcommand{\R}{\mathbb{R}}
\usepackage{import}
\bibliographystyle{apalike}
\newcommand{\inner}[2]{\left\langle {#1}, {#2}\right\rangle}
\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\newcommand{\maxblocknorm}{\max_{k\in [n]} \opnorm{U_k H - U_k^*} }
\newcommand{\diag}{\text{diag}}
\newcommand{\Tr}{\text{tr}}
\renewcommand{\O}{\mathcal{O}}
\newcommand{\one}{\mathbbm{1}}
\renewcommand{\S}{\mathbb{S}}
% For repeating lemmas and theorems
\makeatletter
\newtheorem*{rep@theorem}{\rep@title}
\newcommand{\newreptheorem}[2]{%
\newenvironment{rep#1}[1]{%
\def\rep@title{#2 \ref{##1}}%
\begin{rep@theorem}}%
{\end{rep@theorem}}}
\makeatother
\newreptheorem{theorem}{Theorem}
\newreptheorem{lemma}{Lemma}
\newreptheorem{corollary}{Corollary}
\newreptheorem{proposition}{Proposition}
\newcommand{\deapprox}{\stackrel{d}{\approx}}
% Keywords command
\providecommand{\keywords}[1]
{
\small
\textbf{\textit{Keywords---}} #1
}
\title{A Novel and Optimal Spectral Method for Permutation Synchronization}
\lhead{}
\rhead{}
\author{Duc Nguyen and Anderson Ye Zhang\\
~\\
University of Pennsylvania
}
\begin{document}
\maketitle
\import{./}{abstract}
\section{Introduction}\label{sect:intro}
\import{./}{introduction}
\section{A Novel Spectral Methods}\label{sect:algo}
\import{./}{algorithm}
\section{Theoretical Guarantees}\label{sect:analysis}
\import{./}{theory}
% \subsection{Blockwise $\ell_\infty$ Analysis for Eigenspaces}
% \subsection{Main Results}
\section{Proofs}
\import{./}{proofs}
\bibliography{reference}
\end{document}
%{\color{red} to do: 1. replace $O(d)$ by $\mathcal{O}(d)$ or $\mathcal{O}_d$, 2. there exist constants $C,C'$ ``$>0$'', 3. use $\fnorm{\cdot}$, 4. use $\define$ and $\definei$, 5. use $\p$ instead of Pr}
%\duc{Small bugs in the proof}.
%
%In helper Lemma \ref{lem:event-E-prime_new}, to show $\opnorm{X - X^{(j)}} \leq 2\opnorm{X_j} \leq \frac{\lambda_d(X)- \lambda_{d+1}(X)}{2}$. In proof of Lemma \ref{lem:simplification} to derive upper bound on $\opnorm{B_j}$. In these cases, we have absorbed the $p\sqrt n$ term into the ${C_0}\sqrt{np}$ term. However, to be explicit, sufficient condition is $C_0 > 7$. In Lemma \ref{ref:event-E-prime_new}, if we assume $C_0 > 2$
%$$ \opnorm{X_j} \leq p\sqrt n + C_0(1+\sigma \sqrt d)\sqrt{np} \leq \frac{1}{2} C_0\sqrt{np} + C_0(1+\sigma \sqrt d)\sqrt{np} \leq \frac{3}{2} C_0(1+\sigma \sqrt d)\sqrt{np} \leq \frac{3np}{16} \leq \frac{\lambda_d-\lambda_{d+1}}{4} $$
%In Lemma \ref{lem:simplification}, if we assume $C_0 > 3$,
%\begin{align*}
% \opnorm{B_j} &\leq \opnorm{U_j } \opnorm{H\Lambda - \Lambda H} \opnorm{\Lambda^{-1}}+ \opnorm{X_j}\opnorm{UU^\top - U^{(j)}U^{(j)\top}} \opnorm{\Lambda^{-1}} \\
% &\leq {\frac{4}{3}\br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}}} \br{2C_0\br{1+\sigma\sqrt{d}}\sqrt{np}} \frac{8}{7np} \\
% &\quad + \br{p\sqrt{n} + C_0\br{1+\sigma\sqrt{d}}\sqrt{np}} \br{6\br{\opnorm{U_jH -U^*_j}+\frac{1}{\sqrt{n}}}} \frac{8}{7np} \\
% &\leq \br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}} \br{\frac{48}{7\sqrt n} + \frac{8\cdot 26}{21}\frac{C_0}{\sqrt{np}}\br{1+\sigma \sqrt{d}}}\\
% &\leq \br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}} 11C_0 \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} ,
%\end{align*}
%where last inequality comes from $\frac{48}{7\sqrt n} + \frac{8\cdot 26}{3\sqrt np}C_0 \leq \frac{7}{\sqrt n} + \frac{10C_0}{\sqrt{np}}\leq \frac{11C_0}{\sqrt{np}}$ if $\frac{7}{\sqrt n} \leq \frac{C_0}{\sqrt{np}}$ opr $C_0 \geq 7$.
%
%
%In the proof of the Theorem \ref{thm:main_new}, to derive the upper bound on,
%\begin{align*}
% &\E\indic{ \fnorm{U_j \hatH^\top - U_j^*\hat P^\top} \geq \fnorm{U_j \hatH^\top - \frac{R \hat P^\top}{\sqrt n} } } \indic{\mathf} \\
% &\leq \ebr{- \br{1- 3\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4}} - C_2\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}}_+^2 \br{1+C_3\sqrt{\frac{\log n}{np}}}^{-1} \frac{np}{2\sigma^2}} + 3n^{-10} \\
% &\quad + 4\ebr{-\frac{np}{\sigma^2}} + 2d \ebr{-\frac{3 \log n}{256} \frac{C_3 (\log n)^{1.25}}{\sqrt{d}(C''\vee 1)\br{ \frac{1}{\sqrt{\log n}} \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} + \frac{\sigma}{\sqrt{np}}}}} \\
% &\quad + 6d \ebr{ -\frac{3c \log n}{16^3} \br{\frac{\sqrt{C_3} (\log n)^{0.25}}{\sqrt{d}(C''\wedge1) \br{ \frac{1}{\sqrt{\log n}} \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} + \frac{\sigma}{\sqrt{np}}}}} ^2}\\
% &\leq \ebr{- \br{1- 3\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4}} - C_2\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}}_+^2 \br{1+C_3\sqrt{\frac{\log n}{np}}}^{-1} \frac{np}{2\sigma^2}} + 3n^{-10} \\
% &\quad + 4\ebr{-\frac{np}{\sigma^2}} + 9dn^{-10}.
% \end{align*}
%We have combined the two extra exponent terms into $9dn^{-10}$ but didn't say when this happens. Both happen when $\frac{\log n}{np}, \frac{\sigma}{\sqrt np}$ are smaller than some constant.
%
%\duc{End of bugs}
\subsection{Proof of Proposition \ref{prop:hat_H_new}}
We first state and prove a deterministic version of Proposition \ref{prop:hat_H_new}. The proof of Proposition \ref{prop:hat_H_new} follows from a simple probabilistic argument.
\begin{lemma}\label{lem:hat_H}
Assume that (\ref{eqn:5}) holds. If $\frac{np}{(\sqrt{d} + \sigma d)^2} \geq 32 C_0^2$, we have
\begin{align}\label{eqn:11}
\min_{P\in\Pi_d}\fnorm{\hatH - PH^\top} \leq \frac{{4C_0\br{\sqrt{d} + \sigma d}}}{\sqrt{np}}.
\end{align}
\end{lemma}
\begin{proof}
Note that
\begin{align*}
\opnorm{U-U^*H^\top} &= \opnorm{UU^\top U - U^*U^{*\top}U} \leq \opnorm{UU^\top - U^*U^{*\top}}\opnorm{U}\leq \opnorm{UU^\top - U^*U^{*\top}}.
\end{align*}
Since (\ref{eqn:5}) holds, we have
\begin{align*}
\fnorm{U - U^*H^\top} &\leq \sqrt{d} \opnorm{U-U^*H^\top} \leq \sqrt{d}\frac{8C_0\br{1+\sigma\sqrt{d}}}{7\sqrt{np}}.
\end{align*}
% and (\ref{eqn:6}), we have
%\begin{align*}
%\fnorm{U - U^*H^{-1}}&\leq \sqrt{d} \opnorm{U - U^*H^{-1}} \leq \sqrt{d}\opnorm{U H - U^*}\opnorm{H^{-1}} \leq 2\sqrt{d} \frac{1+\sigma \sqrt{d}}{\sqrt{np}}.
%\end{align*}
Note that
%according to Lemma \ref{lem:population},
$\sqrt{n}U^*$ has only $k$ unique rows $e_1^\top,\ldots,e_d^\top$ and each is of size $n$. Then $\sqrt{n}U^* H^\top$ also has $k$ unique rows $e_1^\top H^\top,\ldots,e_d^\top H^\top$ and each is of size $n$. From (\ref{eqn:5}), there exists a matrix $O\in \mathcal{O}_d$ such that $\opnorm{H-O}\leq \frac{8C_0\br{1+\sigma\sqrt{d}}}{7\sqrt{np}} \leq \frac{1}{7}$. Then
\begin{align*}
\min_{a,b\in[d]} \norm{(e_a - e_b)^\top H^\top} &\geq \min_{a,b\in[d]} \norm{(e_a - e_b)^\top \br{O^\top+ (H-O)^\top}}\\
&\geq \min_{a,b\in[d]} \norm{(e_a - e_b)^\top O^\top } - \opnorm{H-O}\\
&\geq \sqrt{2} -\frac{1}{7}\\
&\geq \frac{6}{5}.
\end{align*}
That is, $\Delta$, defined as the minimum distance among the unique rows of $U^*H^\top$, is at least $\frac{6}{5\sqrt{n}}$.
Let $\hat z\in[d]^{nd}$ be the minimizer of (\ref{eqn:alg1}) with $\{\hat \mu_1,\ldots,\hat \mu_d\}$. Denote $\hat \Theta \define (\hat \mu_{\hat z_1}^\top, \ldots, \hat \mu_{\hat z_{nd}}^\top)^\top\in\mathr^{nd\times d}$. According to (\ref{eqn:alg1}), we have $\fnorm{\hat \Theta - U}\leq \fnorm{U^*H^\top - U}$. Hence,
\begin{align*}
\fnorm{\hat \Theta - U^*H^\top} &\leq \fnorm{\hat \Theta -U } + \fnorm{U^*H^\top - U}\leq 2 \fnorm{U - U^*H^\top }\leq \sqrt{d}\frac{16C_0\br{1+\sigma\sqrt{d}}}{7\sqrt{np}}.
\end{align*}
Define $z^*\in[d]^{nd}$ such that the $i$th row of $U^* H^\top$ is equal to $e_{z^*_i}^\top H^T/\sqrt{n}$ for each $i\in[n]$.
Define the set $S$ as
\begin{align*}
S\define \cbr{i\in[nd]: \norm{\hat \mu_{\hat z_i} - e_{z^*_i}^\top H^T/\sqrt{n}} > \frac{\Delta}{2}}.
\end{align*}
Then we have
\begin{align*}
\abs{S} \leq \frac{\fnorm{\hat \Theta - U^*H^\top}^2}{(\Delta/2)^2}\leq \frac{\br{4C_0\br{\sqrt{d} + \sigma d}}^2}{p}
\end{align*}
Under the assumption $\frac{np}{(\sqrt{d} + \sigma d)^2} \geq 32 C_0^2,$
%\begin{align*}
%\frac{np}{(\sqrt{d} + \sigma d)^2} \geq 32 C_0^2,
%\end{align*}
we have $\abs{S}\leq n/2$. Then
by the same argument as in the proof of Proposition 3.1 of \cite{zhang2022leave}, there exists a bijection $\phi:[d]\rightarrow[d]$ such that $\hat z_i = \phi(z^*_i)$ for all $i\notin S$. Hence, for each $a\in[d]$, we have
\begin{align*}
\norm{\hat \mu_{\phi(a)} - e_a^\top H^\top/\sqrt{n} }^2 & = \frac{\sum_{i\in [nd]:\hat z_i = \phi(a),z^*_i =a} \norm{\hat \mu_{\hat z_i} - e_{z^*_i}^\top H^\top }^2}{\abs{i\in[nd]:\hat z_i = \phi(a),z^*_i =a}} \leq \frac{\sum_{i\in[nd]:\hat z_i = \phi(a)} \norm{\hat \mu_{\hat z_i} - e_{z^*_i}^\top H^\top }^2}{n-\abs{S}}\\
&\leq \frac{\sum_{i\in[nd]:\hat z_i = \phi(a)} \norm{\hat \mu_{\hat z_i} - e_{z^*_i}^\top H^\top }^2}{n/2}.
\end{align*}
Hence,
\begin{align*}
\sum_{a\in[d]}\norm{\hat \mu_{\phi(a)} - e_a^\top H^\top/\sqrt{n}}^2 \leq \frac{\sum_{i\in[nd]} \norm{\hat \mu_{\hat z_i} - e_{z^*_i}^\top H^\top }^2 }{n/2} = \frac{\fnorm{\hat \Theta -U^* H^\top}^2}{n/2} \leq \frac{\br{4C_0\br{\sqrt{d} + \sigma d}}^2}{n^2p}.
\end{align*}
That is, there exists a permutation matrix $P$ such that
\begin{align*}
\fnorm{\hatH - PH^\top}^2 \leq \frac{\br{4C_0\br{\sqrt{d} + \sigma d}}^2}{np},
\end{align*}
since rows of $M$ are $\sqrt{n}\hat \mu_1,\ldots,\sqrt{n}\hat \mu_d $.
\end{proof}
\begin{proof}[\textbf{Proof of Proposition \ref{prop:hat_H_new}}]
According to Lemma \ref{lem:event-E-prime_new}, there exist constants $C,C_0>0$ such that if $\frac{np}{\log n}>C$ and $\frac{np}{(1+\sigma \sqrt{d})^2 }\geq 64C_0^2$, we have (\ref{eqn:5}) hold with probability at least $1-n^{-10}$. The proof is complete by Lemma \ref{lem:hat_H}.
\end{proof}
%Throughout the proof and without loss of generality, we let $Z^*_j = I_d$ for all $j\in[n]$. Hence, (\ref{eqn:X-def}) becomes
%\begin{align*}
%X = (A \otimes I_d) + \sigma (A \otimes J_d) \circ W.
%\end{align*}
%For each $j\in[n]$, define $X_j\define (X_{j1},\ldots, X_{jn})\in\mathr^{d\times nd}$. Then we have
%\begin{align}\label{eqn:1}
%X_j = (A_j \otimes I_d) + \sigma (A_j \otimes J_d) \circ W_j.
%\end{align}
%As a result, $X =(X_1^\top,\ldots, X_n^\top)^\top$. We introduce a few leave-one-out matrices that will be used in the proofs. For each $j\in[n]$, define $X^{(j)}\in\mathr^{nd\times nd}$ such that
%\begin{align*}
%X^{(j)}_{ik} \define\quad \begin{cases}
%X_{ik}, \forall i,k\neq j,\\
%0_{d\times d},\quad \text{o.w..}
%\end{cases}
%\end{align*}
%In addition, let $U^{(j)}\in\mathr^{nd\times d}$ be the matrix including the leading $d$ eigenvectors of $X^{(j)}$. As a consequence, $X^{(j)}, U^{(j)}$ are independent of $\{A_{jk}\}_{k\neq j}$ and $\{W_{jk}\}_{k\neq j}$. The following decomposition of $UH-U^*$ is the starting point to establishing our main theorems.
%
%\paragraph{Decomposition of $UH-U^*$.}
%Define $\Lambda\in\mathr^{d\times d}$ to be the diagonal matrix of the leading eigenvalues of $X$. That is,
%\begin{align}\label{eqn:Lambda_def}
%\Lambda_{ii}\define \lambda_i(X) \text{ and }\Lambda_{ik}\define 0, \forall 1\leq i\neq k\leq d.
%\end{align}
%% $\Lambda_{ii}\define \lambda_i(X)$ and $\Lambda_{ik}\define 0$ for all $1\leq i\neq k\leq d$.
%Note that
%\begin{align*}
%UH\Lambda - XU^* & = U(H\Lambda - \Lambda H) + U\Lambda H -XU^*\\
%& = U(H\Lambda - \Lambda H) + XU H -XU^*\\
%&= U(H\Lambda - \Lambda H) + X(U H - U^*).
%\end{align*}
%Multiplying both sides by $\Lambda^{-1}$ and rearranging the terms, we have
%\begin{align}\label{eqn:decomposition1}
%UH - U^* = U(H\Lambda - \Lambda H) \Lambda^{-1} + X(U H - U^*) \Lambda^{-1} + XU^*\Lambda^{-1} - U^* .
%\end{align}
%The above display involves $ X(U H - U^*) $ where $X$ and $UH-U^*$ are dependent on each other. To decouple the dependence, we approximate $UH-U^*$ by its leave-one-out counterparts.
%Consider any $j\in[n]$. Define
%\begin{align*}
%H^{(j)}\define U^{(j)\top}U^*\in\mathr^{d\times d}.
%\end{align*}
%Then
%\begin{align*}
%UH-U^* &= UH - U^{(j)}H^{(j)} + U^{(j)}H^{(j)} - U^* \\
%& = (UU^\top - U^{(j)}U^{(j)\top})U^*+ U^{(j)}H^{(j)} - U^* .
%\end{align*}
%After plugging it into the right-hand side of (\ref{eqn:decomposition1}), we have
%\begin{align*}
%UH - U^* & = U(H\Lambda - \Lambda H) \Lambda^{-1} + X(UU^\top - U^{(j)}U^{(j)\top})U^* \Lambda^{-1} \\
%&\quad + X(U^{(j)}H^{(j)} - U^*) \Lambda^{-1} + XU^*\Lambda^{-1} - U^*.
%\end{align*}
%Then, the $j$th block matrix of $UH - U^*$ satisfies
%\begin{align}
%U_jH - U^*_j & = \underbrace{U_j(H\Lambda - \Lambda H) \Lambda^{-1} + X_j(UU^\top - U^{(j)}U^{(j)\top})U^* \Lambda^{-1}}_{\definei B_j} \nonumber\\
%&\quad + X_j(U^{(j)}H^{(j)} - U^*) \Lambda^{-1} + X_jU^*\Lambda^{-1} - U^*_j.\label{eqn:decomposition2}
%\end{align}
%The last two terms in (\ref{eqn:decomposition2}) can be further decomposed.
%Using (\ref{eqn:1}), we have
%\begin{align*}
%X_j(U^{(j)}H^{(j)} - U^*) \Lambda^{-1}&= \br{(A_j \otimes I_d) + \sigma (A_j \otimes J_d) \circ W_j}(U^{(j)}H^{(j)} - U^*) \Lambda^{-1}\\
%&= \underbrace{\sum_{k\neq j} A_{jk} (U^{(j)}_kH^{(j)} - U^*_k)\Lambda^{-1}}_{\definei F_{j1}} + \underbrace{\sigma \sum_{k\neq j} A_{jk} W_{jk}(U^{(j)}_kH^{(j)} - U^*_k)\Lambda^{-1}}_{\definei F_{j2}} ,
%\end{align*}
%and
%\begin{align*}
%X_jU^*\Lambda^{-1} - U^*_j &= \br{(A_j \otimes I_d) + \sigma (A_j \otimes J_d) \circ W_j}U^*\Lambda^{-1} - U^*_j \\
%&= \left(\sum_{k\neq j} A_{jk} U^*_k\right)\Lambda^{-1} - U_j^*+ \sigma \,\sum_{k\neq j} A_{jk} W_{jk} U^*_k\Lambda^{-1} \\
%&= \underbrace{\frac{1}{\sqrt{n}}\br{\left(\sum_{k\neq j} A_{jk} \right)\Lambda^{-1} - I_d}}_{\definei G_{j1}} + \underbrace{ \frac{\sigma}{\sqrt{n}} \,\sum_{k\neq j} A_{jk} W_{jk} \Lambda^{-1} }_{\definei G_{j2}} ,
%\end{align*}
%where the last equation is due to Lemma \ref{lem:population}.
%As a result,
%\begin{align}\label{eqn:decomposition3}
%U_jH - U^*_j & =
%%U_j(H\Lambda - \Lambda H) \Lambda^{-1} + X_j(UU^\top - U^{(j)}U^{(j)\top})U^* \Lambda^{-1}
%B_j + F_{j1} + F_{j2} + G_{j1} + G_{j2}.
%\end{align}
%Note we have a mutual independence among $\{A_{jk}\}_{k\neq i}$, $\{W_{jk}\}_{k\neq i}$, and $U^{(j)}H^{(j)} -U^*$ in the definitions of $F_{j1}$ and $F_{j2}$, which is crucial to obtaining sharp bounds for them.
%The decomposition (\ref{eqn:decomposition3}) holds for all $j\in[n]$.
\subsection{Proof of Theorem \ref{thm:l_infty_new}}
We first give a deterministic upper bound for $\opnormt{U_jH - U^*_j}$, using the decomposition (\ref{eqn:decomposition3}).
\begin{lemma}\label{lem:simplification}
Assume (\ref{eqn:2})-(\ref{eqn:12}) hold. Under the assumption $\frac{np}{(1+\sigma \sqrt{d})^2} \geq 22^2 C_0^2$, for each $j\in[n]$, we have
\begin{align}\label{eqn:13}
\opnorm{U_jH - U^*_j } &\leq \frac{22C_0}{\sqrt{n}}\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}+2\opnorm{F_{j1}} + 2\opnorm{F_{j2}} + 2\opnorm{G_{j1}} + 2\opnorm{G_{j2}},
\end{align}
and
\begin{align}\label{eqn:Bj-bound}
\opnorm{B_j} &\leq 22C_0 \br{\frac{1}{\sqrt{n}} +\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + \opnorm{G_{j2}}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}.
\end{align}
\end{lemma}
\begin{proof}
Consider any $j\in[n]$. We have
\begin{align*}
\opnorm{B_j} &\leq \opnorm{U_j } \opnorm{H\Lambda - \Lambda H} \opnorm{\Lambda^{-1}}+ \opnorm{X_j}\opnorm{UU^\top - U^{(j)}U^{(j)\top}} \opnorm{\Lambda^{-1}} \\
&\leq {\frac{4}{3}\br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}}} \br{2C_0\br{1+\sigma\sqrt{d}}\sqrt{np}} \frac{8}{7np} \\
&\quad + \br{p\sqrt{n} + C_0\br{1+\sigma\sqrt{d}}\sqrt{np}} \br{6\br{\opnorm{U_jH -U^*_j}+\frac{1}{\sqrt{n}}}} \frac{8}{7np} \\
&\leq \frac{8}{7np}\br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}} \br{6p\sqrt{n} + \frac{26}{3}C_0\br{1+\sigma \sqrt{d}}\sqrt{np}}\\
&\leq 11C_0 \br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} ,
\end{align*}
where the second inequality is by (\ref{eqn:2})-(\ref{eqn:12}) and the last inequality is by $C_0 > 7$. From (\ref{eqn:decomposition3}), we also have
\begin{align*}
\opnorm{U_jH - U^*_j } &\leq \opnorm{B_j}+\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + \opnorm{G_{j2}}.
\end{align*}
Plug the upper bound (\ref{eqn:Bj-bound}) of $\opnorm{B_j}$ into the above display. After rearrangement, we have
\begin{align*}
\br{1- 11C_0 \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} }\opnorm{U_jH - U^*_j }&\leq \frac{11C_0}{\sqrt{n}}\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}+\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + \opnorm{G_{j2}}.
\end{align*}
Under the assumption $\frac{np}{(1+\sigma \sqrt{d})^2} \geq 22^2 C_0^2$, we have
\begin{align*}
\opnorm{U_jH - U^*_j } &\leq \frac{22C_0}{\sqrt{n}}\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}+2\opnorm{F_{j1}} + 2\opnorm{F_{j2}} + 2\opnorm{G_{j1}} + 2\opnorm{G_{j2}}.
\end{align*}
Plugging it into the upper bound of $\opnorm{B_j}$, we have
\begin{align*}
\opnorm{B_j} &\leq 11C_0 \br{\br{\frac{22C_0}{\sqrt n}\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}+2\opnorm{F_{j1}} + 2\opnorm{F_{j2}} + 2\opnorm{G_{j1}} + 2\opnorm{G_{j2}}}+\frac{1}{\sqrt{n}}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \\
&\leq 22C_0 \br{\frac{1}{\sqrt{n}} +\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + \opnorm{G_{j2}}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}.
\end{align*}
\end{proof}
As a preview, the terms $\opnorm{F_{j1}}$, $\opnorm{F_{j2}}$, $\opnorm{G_{j1}}$ and $\opnorm{G_{j2}}$ in (\ref{eqn:13}) and (\ref{eqn:Bj-bound}) can be bounded using the helper Lemma \ref{lem:Fj1}, Lemma \ref{lem:Fj2_denominator}, Lemma \ref{lem:Fj2} and Lemma \ref{lem:vershynin_new} respectively. We defer the proof of these helper lemmas to a later section. To shorten the notation, let us denote
\begin{align}
\Delta \define UH-U^*\in\mathr^{nd\times d},\quad \Delta^{(j)} \define U^{(j)}H^{(j)} - U^* \in\mathr^{nd\times d},\forall j\in[n], \label{eqn:Delta_def}
\end{align}
such that block submatrices $\Delta_k = U_kH-U^*_k$ and $ \Delta^{(j)}_k = U^{(j)}_kH^{(j)} - U^*_k$ for each $j,k\in[n]$.
%For each $j\in[n]$, we analogously define $\Delta^{(j)} \define U^{(j)}H^{(j)} - U^* \in\mathr^{nd\times d}$.
We further introduce $\opnorminf{\cdot}$ norm such that
\begin{align}\label{eqn:Delta_norm_def}
\opnorminf{\Delta} &\define \max_{k\in[n]} \opnorm{\Delta_k},\quad\opnorminf{\Delta^{(j)}}\define \max_{k\in[n]} \opnorm{\Delta^{(j)}_k}.
\end{align}
They are related as shown below in Lemma \ref{lem:Delta_Delta_j}.
\begin{lemma}\label{lem:Delta_Delta_j}
Assume (\ref{eqn:14}) holds. Then for any $j\in[n]$, we have
\begin{align}
\opnorminf{\Delta^{(j)}} &\leq 7 \br{ \opnorminf{\Delta}+\frac{1}{\sqrt{n}}}. \label{eqn:15}
\end{align}
\end{lemma}
\begin{proof}
Consider any $j\in[n]$, we have
\begin{align*}
\opnorminf{\Delta^{(j)}} &\leq \opnorminf{\Delta} + \opnorminf{\Delta - \Delta^{(j)}} \leq \opnorminf{\Delta} + \opnorm{\Delta - \Delta^{(j)}} \\
&= \opnorminf{\Delta} + \opnorm{UH-U^{(j)}H^{(j)}} = \opnorminf{\Delta} + \opnorm{UU^\top U^*-U^{(j)}U^{(j)\top} U^*}\\
&\leq \opnorminf{\Delta} + \opnorm{UU^\top -U^{(j)}U^{(j)\top}} \leq \opnorminf{\Delta} + 6\br{\opnorm{U_jH -U^*_j}+\frac{1}{\sqrt{n}}}\\
&\leq 7 \br{ \opnorminf{\Delta}+\frac{1}{\sqrt{n}}},
\end{align*}
where the second to last inequality is due to (\ref{eqn:14}).
\end{proof}
We are now ready to prove Theorem \ref{thm:l_infty_new}.
\begin{proof}[\textbf{Proof of Theorem \ref{thm:l_infty_new}}]
From Lemma \ref{lem:event-E-prime_new}, there exist constants $C_0', C_0 > 0$ such that if $\frac{np}{\log n} > C_0'$, then the event $\calE$ holds with probability at least $1-n^{-10}$. Assume $\calE$ holds and $\frac{np}{(\sqrt{d} + \sigma d)^2} \geq 22^2 C_0^2$. Then according to Lemma \ref{lem:event-E-prime_new}, we have (\ref{eqn:2})-(\ref{eqn:12}) hold. As a consequence, by Lemma \ref{lem:simplification} and Lemma \ref{lem:Delta_Delta_j}, we have (\ref{eqn:13}) and (\ref{eqn:15}) all hold as well.
Consider any $j\in[n]$. Note that (\ref{eqn:13}) can be written as
\begin{align}
\opnorm{\Delta_j} &\leq \frac{22C_0}{\sqrt{n}}\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}+2\opnorm{F_{j1}} + 2\opnorm{F_{j2}} + 2\opnorm{G_{j1}} + 2\opnorm{G_{j2}}.\label{eqn:17}
\end{align}
Hence, to upper bound $\opnorm{\Delta_j}$, we need to study $\opnorm{F_{j1}}$, $\opnorm{F_{j2}}$, $\opnorm{G_{j1}}$, and $\opnorm{G_{j2}}$. By Lemma \ref{lem:Fj1}, we have
\begin{align*}
\opnorm{\sum_{k\neq j} A_{jk} \Delta_k^{(j)}} &\leq p\sqrt{n} \opnorm{\Delta^{(j)}} + \sqrt{40np\opnorminf{\Delta^{(j)}}^2 \log n} + \frac{40}{3} \log n\opnorminf{\Delta^{(j)}}\\
&\leq p\sqrt{n} \opnorm{\Delta^{(j)}} + \br{ \sqrt{40np \log n} + \frac{40}{3} \log n} \opnorminf{\Delta^{(j)}},
\end{align*}
holds with probability at least $1-2dn^{-10}$. Then with (\ref{eqn:3}), we have
\begin{align*}
\opnorm{F_{j1}} &\leq \opnorm{\sum_{k\neq j} A_{jk} \Delta_k^{(j)}} \opnorm{\Lambda^{-1}}\\
&\leq \frac{8}{7np} \br{ p\sqrt{n} \opnorm{\Delta^{(j)}} + \br{ \sqrt{40np \log n} + \frac{40}{3} \log n} \opnorminf{\Delta^{(j)}}}\\
&\leq \frac{C_1}{\sqrt{n}} \opnorm{\Delta^{(j)}} +C_1 \sqrt{\frac{\log n}{np}} \opnorminf{\Delta^{(j)}},\numberthis \label{eqn:18}
\end{align*}
for some constant $C_1>0$. By Lemma \ref{lem:Fj2}, we have
\begin{align*}
\opnorm{\sum_{k\neq j}A_{jk}W_{jk}\Delta^{(j)}_k} &\leq C_2 \sqrt{\log n} \opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2}
\end{align*}
holds with probability at least $1-n^{-10}$ for some constant $C_2>0$. In addition, by Lemma \ref{lem:Fj2_denominator}, we have
\begin{align*}
\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} &\leq p \opnorm{\Delta^{(j)}}^2 +\sqrt{40p\opnorm{\Delta^{(j)}}^2\opnorminf{\Delta^{(j)}}^2 \log n} + \frac{40}{3} \log n\opnorminf{\Delta^{(j)}}^2\\
&\leq p \opnorm{\Delta^{(j)}}^2 +\sqrt{40np\opnorminf{\Delta^{(j)}}^4 \log n} + \frac{40}{3} \log n\opnorminf{\Delta^{(j)}}^2\\
&\leq p \opnorm{\Delta^{(j)}}^2 +\br{ \sqrt{40np \log n} + \frac{40}{3} \log n} \opnorminf{\Delta^{(j)}}^2
\end{align*}
holds with probability at least $1-2dn^{-10}$. Then with (\ref{eqn:3}), we have
\begin{align*}
\opnorm{F_{j2}} &\leq \sigma \opnorm{\sum_{k\neq j}A_{jk}W_{jk}\Delta^{(j)}_k} \opnorm{\Lambda^{-1}}\\
&\leq \frac{8\sigma}{7np} C_2 \sqrt{\log n} \sqrt{p \opnorm{\Delta^{(j)}}^2 +\br{ \sqrt{40np \log n} + \frac{40}{3} \log n} \opnorminf{\Delta^{(j)}}^2}\\
&\leq \frac{8\sigma}{7np} C_2 \sqrt{\log n} \br{\sqrt{p} \opnorm{\Delta^{(j)}} + \sqrt{ \sqrt{40np \log n} + \frac{40}{3} \log n} \opnorminf{\Delta^{(j)}}}\\
&\leq \frac{C_3\sigma}{\sqrt{n}}\sqrt{\frac{\log n}{np}} \opnorm{\Delta^{(j)}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} \opnorminf{\Delta^{(j)}}, \numberthis \label{eqn:19}
\end{align*}
for some constant $C_3>0$. For $\norm{G_{j1}}$, its upper bound is given in (\ref{eqn:16}). By Lemma \ref{lem:vershynin_new}, using the fact that $\{A_{ij}\}_{k\neq j}$ and $\{W_{jk}\}_{k\neq j}$ are independent, we have
\begin{align*}
\opnorm{\sum_{k\neq j} A_{jk} W_{jk}} &\leq \sqrt{\sum_{k\neq j} A_{jk}} \br{2\sqrt{d} + C_4\sqrt{\log n}},
\end{align*}
with probability at least $1-n^{-10}$,
for some constant $C_4>0$. Then
\begin{align*}
\opnorm{G_{j2}} &\leq \frac{\sigma}{\sqrt{n}} \opnorm{\sum_{k\neq j} A_{jk} W_{jk}} \opnorm{\Lambda^{-1}}\\
&\leq \frac{\sigma}{\sqrt{n}} \sqrt{\sum_{k\neq j} A_{jk}} \br{2\sqrt{d} + C_4\sqrt{\log n}}\opnorm{\Lambda^{-1}}\\
&\leq C_5\frac{\sigma}{\sqrt{n}}\br{\sqrt{\frac{d}{np}} + \sqrt{\frac{\log n }{np}}}, \numberthis \label{eqn:20}
\end{align*}
for some constant $C_5>0$, where the last inequality is by $\calE$ and (\ref{eqn:3}).
Plugging (\ref{eqn:18}), (\ref{eqn:19}), (\ref{eqn:16}), and (\ref{eqn:20}) into (\ref{eqn:17}), we have
\begin{align*}
\opnorm{\Delta_j} &\leq \frac{22C_0}{\sqrt{n}}\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}+2\br{ \frac{C_1}{\sqrt{n}} \opnorm{\Delta^{(j)}} +C_1 \sqrt{\frac{\log n}{np}} \opnorm{\Delta^{(j)}}} \\
&\quad + 2\br{ \frac{C_3\sigma}{\sqrt{n}}\sqrt{\frac{\log n}{np}} \opnorm{\Delta^{(j)}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} \opnorminf{\Delta^{(j)}}} + \frac{4C_0}{\sqrt{n}}\br{\sqrt{\frac{\log n}{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} \\
&\quad + 2C_5\frac{\sigma}{\sqrt{n}}\br{\sqrt{\frac{d}{np}} + \sqrt{\frac{\log n }{np}}}\\
&\leq \frac{2}{\sqrt{n}} \br{C_1 + C_3\sigma\sqrt{\frac{\log n}{np}}} \opnorm{\Delta^{(j)}} +2\br{C_1\sqrt{\frac{\log n}{np}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} } \opnorminf{\Delta^{(j)}} \\
&\quad + \frac{26C_0 + 2C_5}{\sqrt{n}} \frac{\sqrt{\log n} + \sigma \sqrt{d} + \sigma\sqrt{\log n}}{\sqrt{np}}.
\end{align*}
By (\ref{eqn:15}) and (\ref{eqn:12}), we can replace the terms $\opnorm{\Delta^{(j)}}$, $\opnorminf{\Delta^{(j)}}$ with their respective upper bounds and have
\begin{align*}
\opnorm{\Delta_j} &\leq \frac{2}{\sqrt{n}} \br{C_1 + C_3\sigma\sqrt{\frac{\log n}{np}}} \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}} +14\br{C_1\sqrt{\frac{\log n}{np}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} } \br{ \opnorminf{\Delta}+\frac{1}{\sqrt{n}}} \\
&\quad + \frac{26C_0 + 2C_5}{\sqrt{n}} \frac{\sqrt{\log n} + \sigma \sqrt{d} + \sigma\sqrt{\log n}}{\sqrt{np}}.
\end{align*}
By a union bound, with probability at least $1-6dn^{-9}$, the above display holds for all $j\in[n]$. Then
\begin{align*}
\opnorminf{\Delta} &\leq \frac{2}{\sqrt{n}} \br{C_1 + C_3\sigma\sqrt{\frac{\log n}{np}}} \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}} +14\br{C_1\sqrt{\frac{\log n}{np}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} } \br{ \opnorminf{\Delta}+\frac{1}{\sqrt{n}}} \\
&\quad + \frac{26C_0 + 2C_5}{\sqrt{n}} \frac{\sqrt{\log n} + \sigma \sqrt{d} + \sigma\sqrt{\log n}}{\sqrt{np}}.
\end{align*}
After a rearrangement, we have
\begin{align*}
&\br{1- 14\br{C_1\sqrt{\frac{\log n}{np}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} }} \opnorminf{\Delta} \\
&\leq \frac{2}{\sqrt{n}} \br{C_1 + C_3\sigma\sqrt{\frac{\log n}{np}}} \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}} + \frac{14}{\sqrt{n}}\br{C_1\sqrt{\frac{\log n}{np}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} }\\
&\quad + \frac{26C_0 + 2C_5}{\sqrt{n}} \frac{\sqrt{\log n} + \sigma \sqrt{d} + \sigma\sqrt{\log n}}{\sqrt{np}}.
\end{align*}
Then, there exists a constant $C>0$ such that if $\frac{np}{(\sigma^{4/3} \vee 1)\log n} >C$, we have $14\br{C_1\sqrt{\frac{\log n}{np}} + C_3\sigma \br{\frac{\log n}{np}}^\frac{3}{4} }\leq \frac{1}{2}$. Consequently, we have
\begin{align*}
\opnorminf{\Delta} &\leq \frac{4}{\sqrt{n}} \br{C_1 + C_3\sigma\sqrt{\frac{\log n}{np}}} \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}} + \frac{28}{\sqrt{n}}\br{C_1\sqrt{\frac{\log n}{np}} + C_3 \sigma \br{\frac{\log n}{np}}^\frac{3}{4} }\\
&\quad + \frac{52C_0 + 4C_5}{\sqrt{n}} \frac{\sqrt{\log n} + \sigma \sqrt{d} + \sigma\sqrt{\log n}}{\sqrt{np}}\\
&\leq \frac{C_6}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}},
\end{align*}
for some constant $C_6>0$.
\end{proof}
%
%
%
%From (\ref{eqn:decomposition3}), we have
%\begin{align*}
%\opnorm{U_jH - U^*_j } &\leq \opnorm{U_j } \opnorm{H\Lambda - \Lambda H} \opnorm{\Lambda^{-1}}+ \opnorm{X_j}\opnorm{UU^\top - U^{(j)}U^{(j)\top}} \opnorm{\Lambda^{-1}} \\
%&\quad +\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + \opnorm{G_{j2}}.
%\end{align*}
%Assume
%\begin{align*}
%&\leq \br{ \opnorm{U_jH-U^*_j} + \frac{1}{\sqrt{n}}}\opnorm{H^{-1}} \opnorm{H\Lambda - \Lambda H} \opnorm{\Lambda^{-1}} \\
%&\quad + \opnorm{X_j}\opnorm{UU^\top - U^{(j)}U^{(j)\top}} \opnorm{\Lambda^{-1}} \\
%&\quad +\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + \opnorm{G_{j2}}.
%\end{align*}
%According to Lemma \ref{lem:event-E-prime}, there exist constants $C,C_0>0$ such that if $\frac{np}{\log n}>C$ and $\frac{np}{(1+\sigma \sqrt{d})^2 }\geq 64C_0^2$, we have (\ref{eqn:2})-(\ref{eqn:6}) hold with probability at least $1-n^{-10}$. With these results, we have
%\begin{align*}
%\opnorm{U_jH - U^*_j } &\leq C_1\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \br{ \opnorm{U_jH-U^*_j} + \frac{1}{\sqrt{n}}} \\
%&\quad +\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + \opnorm{G_{j2}},
%\end{align*}
%for some constant $C_1>0$. If $\frac{np}{(1+\sigma\sqrt{d})^2}\geq 4C
%_1^2$ is further assumed, after rearrangement, we have
%\begin{align*}
%\opnorm{U_jH - U^*_j } &\leq \frac{2C_1}{\sqrt{n}}\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} +2\opnorm{F_{j1}} + 2\opnorm{F_{j2}} + 2\opnorm{G_{j1}} + 2\opnorm{G_{j2}}.
%\end{align*}
\subsection{Proof of Theorem \ref{thm:main_new}}
%We first state and prove a result that is analogous to Theorem \ref{thm:l_infty_new} but is for $ \max_{i\neq j} \opnormt{U_i^{(j)} H^{(j)} - U^*_i}$ across all $j\in[n]$.
%\begin{lemma}\label{lem:l_infty_leave_one}
%There exist constants $C,C',C''>0$ such that if $\frac{np}{(\sigma^{4/3} \vee 1)\log n} >C$ and $\frac{np}{(\sqrt{d} + \sigma d)^2}\geq C'$, we have
%\begin{align}\label{eqn:22}
%\max_{j\in[n]} \max_{i\neq j} \opnorm{U_i^{(j)} H^{(j)} - U^*_i}\leq \frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}
%\end{align}
%with probability at least $1-7dn^{-8}$.
%\end{lemma}
%\begin{proof}
%Consider any $j\in[n]$. Then $X^{(j)}$ is essentially a data matrix of synchronization over $(n-1)$ permutations.
%Despite some abuse of notation, define $U^{*(j)}\in\mathr^{nd\times d}$ such that $U^{*(j)}_j\define 0_{d\times d}$ and $U^{*(j)}_i \define I_d/\sqrt{n-1}$ for all $i\neq j$. Apply Theorem \ref{thm:l_infty_new} to $X^{(i)}$. There exist constants $C,C',C''>0$ such that if $\frac{(n-1)p}{(\sigma^{4/3} \vee 1)\log (n-1)} >C$ and $\frac{(n-1)p}{(\sqrt{d} + \sigma d)^2}\geq C'$, we have
%\begin{align*}
%\max_{i\neq j} \opnorm{U^{(j)}_i (U^{(j)\top} U^{*{(j)}}) - U^{*{(j)}}} \leq \frac{ C''}{\sqrt{n-1}}\br{\sqrt{\frac{\log (n-1) }{(n-1)p}} + \frac{\sigma \sqrt{d}}{\sqrt{(n-1)p}} + \frac{\sigma \sqrt{\log (n-1)}}{\sqrt{(n-1)p}}}
%\end{align*}
%with probability at least $1-6d(n-1)^{-9}$. Note that $U^{(j)}_j =0_{d\times d}$ and $U^{*(j)}_i = \sqrt{n/(n-1)}U^*_i$ for each $i\neq j$.
%Then
%\begin{align*}
%U^{(j)\top} U^{*{(j)}} & = \sum_{k\neq j} U^{(j)\top}_k U^{*{(j)}}_k = \sqrt{\frac{n}{n-1}}\sum_{k\neq j} U^{(j)\top}_kU^{*}_k = \sqrt{\frac{n}{n-1}}U^{(j)\top} U^* = \sqrt{\frac{n}{n-1}}H^{(j)}.
%\end{align*}
%Consequently,
%\begin{align*}
% \max_{i\neq j} \opnorm{U_i^{(j)}H^{(j)} - U^*_i} &= \max_{i\neq j} \opnorm{U_i^{(j)} \sqrt{\frac{n-1}{n}} U^{(j)\top} U^{*{(j)}} - \sqrt{\frac{n-1}{n}} U^{*(j)}_i} \\
% & = \sqrt{\frac{n-1}{n}} \max_{i\neq j} \opnorm{U^{(j)}_i (U^{(j)\top} U^{*{(j)}}) - U^{*{(j)}}} \\
% &\leq \frac{2 C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}.
%\end{align*}
%A union bound over all $j\in[n]$ completes the proof.
%\end{proof}
\textbf{Decomposition of the Hamming loss.} From Lemma \ref{lem:event-E-prime_new}, there exist constants $C_0', C_0 > 0$ such that if $\frac{np}{\log n} > C_0'$, then the event $\calE$ holds with probability at least $1-n^{-10}$. By Theorem \ref{thm:l_infty_new}, there exist some constants $C,C',C''>0$ such that if $\frac{np}{(\sigma^{4/3} \vee 1)\log n} >C$ and $\frac{np}{(\sqrt{d} + \sigma d)^2}\geq C'$, we have (\ref{eqn:22}) holds with probability at least $1-6dn^{-9}$. Denote $\mathf$ to be the event that both $\calE$ and (\ref{eqn:22}) hold. By union bound, we have
\begin{align}\label{eqn:36}
\pbr{\mathf} \geq 1-7dn^{-9}.
\end{align}
Assume $\mathf$ holds and $\frac{np}{(\sqrt{d} + \sigma d)^2} \geq 22^2 C_0^2$. Then according to Lemma \ref{lem:event-E-prime_new}, we have (\ref{eqn:2})-(\ref{eqn:12}) hold. As a consequence, by Lemma \ref{lem:simplification}, Lemma \ref{lem:Delta_Delta_j}, and Lemma \ref{lem:hat_H}, we have (\ref{eqn:13}), (\ref{eqn:15}), and (\ref{eqn:11}) all hold as well. Since
\begin{align}
\E\ell(\hat Z,Z^*) \leq \E\ell(\hat Z,Z^*) \indic{\mathf} + \pbr{\mathf^\mathrm{c}},\label{eqn:35}
\end{align}
in the following we focus on analyzing $\E\ell(\hat Z,Z^*) \indic{\mathf} $.
Define
\begin{align*}
\hat P \define \argmin_{P\in \Pi_d} \, \fnorm{\hatH - P H^\top}.
\end{align*}
Since we let $Z^*_j=I_d$ for all $j\in[n]$, we have
\begin{align*}
\E \ell(\hat Z,Z^*) \indic{\mathf} &\leq \E \br{ \frac{1}{n} \sum_{j\in[n]} \indic{\hat Z_j \neq Z^*_j \hat P^\top} \indic{\mathf} } = \frac{1}{n}\sum_{j\in[n]} \E \indic{\hat Z_j \neq Z^*_j \hat P^\top} \indic{\mathf} \\
&\leq \frac{1}{n}\sum_{j\in[n]} \E \indic{\exists R\in\Pi_d \text{ s.t. }R\neq I_d \text{ and }\fnorm{ \sqrt{n} U_jM^\top - R \hat P^\top }^2 \leq \fnorm{ \sqrt{n} U_jM^\top - Z^*_j \hat P^\top}^2} \indic{\mathf} \\
& = \frac{1}{n}\sum_{j\in[n]} \E \indic{\exists R\in\Pi_d \text{ s.t. }R\neq I_d \text{ and }\fnorm{ U_jM^\top - \frac{R \hat P^\top}{\sqrt{n}} }^2 \leq \fnorm{ U_jM^\top - U^*_j \hat P^\top}^2} \indic{\mathf}\\
&\leq \frac{1}{n}\sum_{j\in[n]} \sum_{R\in\Pi_d: R\neq I_d} \E \indic{\fnorm{ U_jM^\top - \frac{R \hat P^\top}{\sqrt{n}} }^2 \leq \fnorm{ U_jM^\top - U^*_j \hat P^\top}^2} \indic{\mathf}, \numberthis \label{eqn:26}
\end{align*}
where the second inequity is by the definition of $\hat Z_j$ in (\ref{eqn:alg3}) and the second equation is by that we let $Z^*_j=I_d$.
Consider any $R\in \Pi_d$ such that $R\neq I_d$. Define
\begin{align*}
h_R \define \fnorm{R - I_d}^2/2.
\end{align*}
Then $2\leq h_R\leq d$. Consider any $j\in[n]$. In the following, we are going to establish an upper bound for $ \E \indic{\fnorm{ U_jM^\top - \frac{R \hat P^\top}{\sqrt{n}} }^2 \leq \fnorm{ U_jM^\top - U^*_j \hat P^\top}^2} \indic{\mathf}$, the key quantity in (\ref{eqn:26}).
We have
\begin{align*}
&\indic{ \fnorm{U_j \hatH^\top - U_j^*\hat P^\top} \geq \fnorm{U_j \hatH^\top - \frac{R \hat P^\top}{\sqrt n} } } \indic{\mathf} \\
&= \indic{\iprod{U_j \hatH^\top - U_j^*\hat P^\top}{ \frac{R \hat P^\top}{\sqrt n} - U_j^*\hat P^\top}\geq \frac{1}{2}\fnorm{ \frac{R \hat P^\top}{\sqrt n} - U_j^*\hat P^\top}^2 } \indic{\mathf} \\
&=\indic{\iprod{U_j \hatH^\top \hat P - U_j^*}{ \frac{R}{\sqrt n} - U_j^*}\geq \frac{1}{2}\fnorm{ \frac{R }{\sqrt n} - U_j^*}^2 } \indic{\mathf} \\
& = \indic{\iprod{U_j \hatH^\top \hat P - U_j^*}{R-I_d} \geq \frac{h_R}{\sqrt{n}}} \indic{\mathf} \\
& = \indic{\iprod{U_j H - U_j^*}{R-I_d} + \iprod{U_j\br{M - \hat PH^\top}^\top \hat P}{ R-I_d} \geq \frac{h_R}{\sqrt{n}}} \indic{\mathf}. \numberthis \label{eqn:9}
\end{align*}
\textbf{Bounding the Hamming loss.} A significant remaining portion of the proof is dedicated to bounding (\ref{eqn:9}). The two inner products involved in (\ref{eqn:9}) can be further decomposed. For the second inner product, we have
\begin{align*}
&\iprod{U_j\br{M - PH^\top}^\top \hat P}{ R-I_d} \\
&\leq \fnorm{U_j \br{M -\hat P \hat H^\top}^\top}\fnorm{ R-I_d}\\
&\leq \opnorm{U_j} \fnorm{M -\hat P \hat H^\top}\fnorm{ R-I_d}\\
&\leq \frac{4}{3}\br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}}\br{ \frac{{4C_0\br{\sqrt{d} + \sigma d}}}{\sqrt{np}}}\sqrt{2h_R}\\
&\leq \frac{4}{3}\br{\opnorm{B_j}+\opnorm{F_{j1}}+\opnorm{F_{j2}}+\opnorm{G_{j1}}+ \opnorm{G_{j2}}+\frac{1}{\sqrt{n}}}\br{ \frac{{4C_0\br{\sqrt{d} + \sigma d}}}{\sqrt{np}}}\sqrt{2h_R},
%&\leq \frac{4}{3}\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}+\frac{1}{\sqrt{n}}}\br{ \frac{1}{\sqrt{n}}\frac{{4C_0\br{\sqrt{d} + \sigma d}}}{\sqrt{np}}}\sqrt{2h_R}\\
%&\leq \frac{ C_1}{n} \br{1+\frac{\sigma \sqrt{\log n}}{\sqrt{np}}} \sqrt{h_R}, \numberthis \label{eqn:23}
%&\leq \frac{C_1}{\sqrt{n}} \br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} \frac{\sqrt{d} + \sigma d}{\sqrt{np}} \frac{\sqrt{h_R}}{\sqrt{n}}
\end{align*}
%for some constant $C_1>0$,
where the third inequality is by (\ref{eqn:10}) and (\ref{eqn:11}) and the four inequality is by (\ref{eqn:decomposition3}).
For the first inner product of (\ref{eqn:9}), by (\ref{eqn:decomposition3}), we have
\begin{align*}
\iprod{U_j H - U_j^*}{R-I_d} &= \iprod{B_j+ F_{j1} + F_{j2} + G_{j1} }{R-I_d} + \iprod{G_{j2}}{R-I_d}\\
&\leq \fnorm{B_j+ F_{j1} + F_{j2} + G_{j1}} \fnorm{R-I_d} + \iprod{G_{j2}}{R-I_d}\\
&\leq \sqrt{d}\opnorm{B_j+ F_{j1} + F_{j2} + G_{j1}} \sqrt{2h_R} + \iprod{G_{j2}}{R-I_d}\\
&\leq \br{\opnorm{B_j} + \opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} } \sqrt{2dh_R} + \iprod{G_{j2}}{R-I_d}.
\end{align*}
From (\ref{eqn:Bj-bound}) in Lemma \ref{lem:simplification}, since $\frac{np}{(1+\sigma \sqrt{d})^2} \geq 22^2 C_0^2$, we have
\begin{align*}
\opnorm{B_j} &\leq \frac{22C_0}{\sqrt{n}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} + \opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} + 22C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \opnorm{G_{j2}}.
\end{align*}
From the above three displays, we have
\begin{align*}
&\iprod{U_j H - U_j^*}{R-I_d} + \iprod{U_j\br{M - \hat PH^\top}^\top \hat P}{ R-I_d} \\
&\leq \br{1+ \frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }\br{\opnorm{B_j} + \opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} } \sqrt{2dh_R} \\
&\quad +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} \sqrt{2dh_R}\opnorm{G_{j2}}+ \iprod{G_{j2}}{R-I_d} + \frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} \frac{\sqrt{2dh_R}}{\sqrt{n}}\\
&\leq 2\br{1+ \frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }\br{ \opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} } \sqrt{2dh_R} \\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }22C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \opnorm{G_{j2}} \sqrt{2dh_R} + \iprod{G_{j2}}{R-I_d}\\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }22C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \frac{\sqrt{2dh_R} }{\sqrt{n}}\\
&\leq 2\br{1+ \frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }\br{ \opnorm{F_{j1}} + \opnorm{F_{j2}} } \sqrt{2dh_R} \\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }22C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \opnorm{G_{j2}} \sqrt{2dh_R} + \iprod{G_{j2}}{R-I_d}\\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }22C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \frac{\sqrt{2dh_R} }{\sqrt{n}}\\
&\quad + 4\br{1+ \frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} } C_0\br{\frac{\sqrt{\log n} + \sigma\sqrt{d}}{\sqrt{np}}} \frac{\sqrt{2dh_R}}{\sqrt{n}}\\
&\leq 2\br{1+ \frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }\br{ \opnorm{F_{j1}} + \opnorm{F_{j2}} } \sqrt{2dh_R} \\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }22C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \opnorm{G_{j2}} \sqrt{2dh_R} + \iprod{G_{j2}}{R-I_d}\\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }26C_0\br{\frac{\sqrt{\log n}+\sigma \sqrt{d}}{\sqrt{np}}} \frac{\sqrt{2dh_R} }{\sqrt{n}}\,,
\end{align*}
where the second last inequality is by the upper bound on $\opnorm{G_{j1}}$ in (\ref{eqn:16}) and the last inequality is obtained by combining the two terms containing $\frac{\sqrt{2dh_R}}{\sqrt n}$.
%\begin{align*}
%\iprod{U_j H - U_j^*}{R-I_d}&\leq \frac{22C_0\sqrt{2dh_R}}{\sqrt{n}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} + 2\sqrt{2dh_R} \br{\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} }\\
%&\quad + 22C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}}+ \iprod{G_{j2}}{R-I_d}. \numberthis \label{eqn:24}
%\end{align*}
The above display involves $ \iprod{G_{j2}}{R-I_d}$ which can be decomposed:
\begin{align*}
\iprod{G_{j2}}{R-I_d} & = \frac{\sigma}{\sqrt{n}} \iprod{\sum_{k\neq j} A_{jk} W_{jk} \Lambda^{-1}}{R-I_d} \\
&= \frac{\sigma}{\sqrt{n}} \iprod{\sum_{k\neq j} A_{jk} W_{jk} \frac{1}{np} }{R-I_d} + \frac{\sigma}{\sqrt{n}} \iprod{\sum_{k\neq j} A_{jk} W_{jk} \br{\Lambda^{-1} - \frac{1}{np}I_d} }{R-I_d} \\
&= \frac{\sigma}{\sqrt{n}} \frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} + \frac{\sigma}{\sqrt{n}} \iprod{\sum_{k\neq j} A_{jk} W_{jk} \Lambda^{-1}(npI_d-\Lambda) \frac{1}{np} }{R-I_d} \\
&= \frac{\sigma}{\sqrt{n}} \frac{1}{np}\iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} + \frac{1}{np}\iprod{G_{j2}(npI_d-\Lambda) }{R-I_d} \\
&\leq \frac{\sigma}{\sqrt{n}} \frac{1}{np}\iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} + \frac{\sqrt{d}}{np}\opnorm{G_{j2} (npI_d -\Lambda)} \fnorm{R-I_d}\\
&\leq \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} + \frac{\sqrt{d}}{np}\opnorm{G_{j2} }\max_{i\in[n]} \abs{\lambda_i(X)-np} \sqrt{2h_R}\\
&\leq \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} + \frac{C_0\sqrt{2dh_R}(1+\sigma\sqrt{d})}{\sqrt{np}} \opnorm{G_{j2} }.
\end{align*}
Hence,
\begin{align*}
&\iprod{U_j H - U_j^*}{R-I_d} + \iprod{U_j\br{M - \hat PH^\top}^\top \hat P}{ R-I_d} \\
&\leq 2\br{1+ \frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }\br{ \opnorm{F_{j1}} + \opnorm{F_{j2}} } \sqrt{2dh_R} \\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }23C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \opnorm{G_{j2}} \sqrt{2dh_R} \\
&\quad +\br{\frac{41}{33} +\frac{16C_0(1+\sigma\sqrt{d})}{3\sqrt{np}} }26C_0\br{\frac{\sqrt{\log n}+\sigma \sqrt{d}}{\sqrt{np}}} \frac{\sqrt{2dh_R} }{\sqrt{n}}\\
&\quad + \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d}\\
&\leq 4\br{ \opnorm{F_{j1}} + \opnorm{F_{j2}} } \sqrt{2dh_R} + 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \opnorm{G_{j2}} \sqrt{2dh_R} \\
&\quad + 52C_0\br{\frac{\sqrt{\log n}+\sigma \sqrt{d}}{\sqrt{np}}} \frac{\sqrt{2dh_R} }{\sqrt{n}} + \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d}, \numberthis \label{eqn:25}
\end{align*}
where the last inequality holds when $\frac{\sqrt{np}}{1+\sigma\sqrt{d}} >\frac{176}{25}C_0$.
%With this and (\ref{eqn:16}), (\ref{eqn:24}) becomes
%\begin{align*}
%&\iprod{U_j H - U_j^*}{R-I_d}\\
%&\leq \frac{22C_0\sqrt{2dh_R}}{\sqrt{n}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} + 2\sqrt{2dh_R} \br{\opnorm{F_{j1}} + \opnorm{F_{j2}} + \opnorm{G_{j1}} }\\
%&\quad + 23C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}}+\frac{\sigma}{\sqrt{n}} \frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \\
%&\leq \frac{22C_0\sqrt{2dh_R}}{\sqrt{n}} \br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} + \frac{4C_0\sqrt{2dh_R}}{\sqrt{n}} \br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}} + 2\sqrt{2dh_R} \br{\opnorm{F_{j1}} + \opnorm{F_{j2}} }\\
%&\quad + 23C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}}+\frac{\sigma}{\sqrt{n}} \frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \\
%&\leq \frac{26C_0\sqrt{2dh_R}}{\sqrt{n}} \br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}} + 2\sqrt{2dh_R} \br{\opnorm{F_{j1}} + \opnorm{F_{j2}} }\\
%&\quad + 23C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}}+\frac{\sigma}{\sqrt{n}} \frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d}. \numberthis \label{eqn:25}
%\end{align*}
As a result, by (\ref{eqn:25}), (\ref{eqn:9}) becomes
\begin{align*}
&\indic{ \fnorm{U_j \hatH^\top - U_j^*\hat P^\top} \geq \fnorm{U_j \hatH^\top - \frac{R \hat P^\top}{\sqrt n} } } \indic{\mathf} \\
% &\leq \mathbb{I}\Bigg\{ \frac{C_1}{n} \br{1+\frac{\sigma \sqrt{\log n}}{\sqrt{np}}} \sqrt{h_R}+ \frac{26C_0\sqrt{2dh_R}}{\sqrt{n}} \br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}} + 2\sqrt{2dh_R} \br{\opnorm{F_{j1}} + \opnorm{F_{j2}} }\\
%&\quad + 23C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}}+\frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \geq \frac{h_R}{\sqrt{n}} \Bigg\} \indic{\mathf}\\
&\leq \mathbb{I}\Bigg\{ 4\sqrt{2dh_R} \br{\opnorm{F_{j1}} + \opnorm{F_{j2}} } + 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}}\\
&\quad + \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \geq \br{1- C_2 \sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} \frac{h_R}{\sqrt{n}} \Bigg\} \indic{\mathf},
\end{align*}
for some constant $C_2>0$. Let $\rho>0$ be some quantity whose value will be determined later. We have
\begin{align*}
&\E\indic{ \fnorm{U_j \hatH^\top - U_j^*\hat P^\top} \geq \fnorm{U_j \hatH^\top - \frac{R \hat P^\top}{\sqrt n} } } \indic{\mathf} \\
&\leq \E \Bigg( \indic{4\sqrt{2dh_R} \opnorm{F_{j1}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} + \indic{4\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\quad + \indic{ 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\quad + \indic{ \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \geq \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} \frac{h_R}{\sqrt{n}} }\indic{\mathf} \Bigg)\\
&\leq \E\indic{4\sqrt{2dh_R} \opnorm{F_{j1}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} + \E\indic{4\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf}\\
&\quad + \E \indic{ 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\quad +\E \indic{ \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \geq \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} \frac{h_R}{\sqrt{n}} }\indic{\mathf}. \numberthis \label{eqn:27}
\end{align*}
In the following, we are going to establish upper bounds for each of the four terms in (\ref{eqn:27}). Recall the definitions of $\Delta,\Delta^{(j)}$ in (\ref{eqn:Delta_def}) and those of $\opnorminf{\Delta},\opnorminf{\Delta^{(j)}}$ in (\ref{eqn:Delta_norm_def}).
\textbf{For the first term in (\ref{eqn:27})}, by (\ref{eqn:3}), we have
\begin{align*}
\E\indic{4\sqrt{2dh_R} \opnorm{F_{j1}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} &\leq \E\indic{4\sqrt{2dh_R}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \opnorm{\Lambda^{-1}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\leq \E\indic{\frac{32\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf}.
\end{align*}
Define an event
\begin{align*}
\mathf_{j}\define \mathbb{I} \Bigg\{ & \opnorm{\Delta^{(j)}} \leq \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}},\\
&\opnorminf{\Delta^{(j)}}\leq 7\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}
\Bigg\}.
\end{align*}
Note that the upper bound for $\opnorminf{\Delta^{(j)}}$ in $\mathf_j$ is a direct consequence of (\ref{eqn:15}) and (\ref{eqn:22}). Together with (\ref{eqn:12}), we have $\mathf\subset \mathf_j$. The event $\mathf_j$ has an equivalent expression. Define a set
\begin{align*}
\mathcal{Y}\define \Bigg\{ Y&=(Y_1^\top,Y_2^\top,\ldots,Y_n^\top)^\top\in\mathr^{nd\times d}: \opnorm{Y} \leq \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}},\\
&\max_{i\in[n]}\opnorm{Y_i}\leq 7\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}
\Bigg\}.
\end{align*}
Then $\mathf_j = \indic{\Delta^{(j)} \in \mathcal{Y}}$. Using that $\Delta^{(j)}$ is independent of $\{A_{jk}\}_{k\neq j}$,
we have
\begin{align*}
\E\indic{4\sqrt{2dh_R} \opnorm{F_{j1}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} &\leq \E\indic{\frac{32\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf_j}\\
& = \pbr{\frac{32\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}}, \mathf_j}\\
& = \E \br{ \pbr{\frac{32\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}}, \mathf_j \Bigg| \Delta^{(j)}}}\\
& = \E\br{ \pbr{\frac{32\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \Delta^{(j)}} \indic{ \Delta^{(j)} \in \mathf_j}}\\
&\leq \sup_{\Delta^{(j)}\in\mathcal{Y}} \pbr{\frac{32\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \Delta^{(j)}}\\
%& = \pbr{\frac{16\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \mathf_j} \pbr{\mathf_j}\\
%&\leq \pbr{\frac{16\sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \mathf_j} \\
%&\leq \pbr{\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{7\sqrt{n}p}{16\sqrt{d}} \Bigg| \mathf_j}\\
&\leq \sup_{\Delta^{(j)}\in\mathcal{Y}} \pbr{\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq \rho \frac{7\sqrt{n}p}{32\sqrt{d}} \Bigg| \Delta^{(j)}}, \numberthis \label{eqn:28}
\end{align*}
where in the last inequality, we use the fact that $h_R\geq 2$. Assume $\rho$ satisfies
\begin{align}
% \rho\frac{7\sqrt{n}p}{32\sqrt{d}} &\geq 9C_0(1+\sigma\sqrt{d}) \sqrt{p}, \label{eqn:rho_1} \\
% \rho\frac{7\sqrt{n}p}{32\sqrt{d}} &\geq7\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}} np.\label{eqn:rho_2} \\
\rho\frac{7\sqrt{n}p}{64\sqrt{d}} \geq 9C_0(1+\sigma\sqrt{d}) \sqrt{dp} \label{eqn:rho_2}\,.
% {\color{red} \text{(38) replaced by } } &{\color{red} \rho\frac{7\sqrt{n}p}{32\sqrt{d}} \geq 9C_0(1+\sigma\sqrt{d}) \sqrt{dp}.} \nonumber
\end{align}
Then by Lemma \ref{lem:Fj1}, we have
\begin{align*}
\E\indic{4\sqrt{2dh_R} \opnorm{F_{j1}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} &\leq \sup_{\Delta^{(j)}\in\mathcal{Y}} \pbr{\opnorm{\sum_{k\neq j} A_{jk} \Delta^{(j)}_k} \geq p\sqrt{n}\opnorm{\Delta^{(j)}}+ \rho \frac{7\sqrt{n}p}{64\sqrt{d}} \Bigg| \Delta^{(j)}}\\
&\leq 2d\sup_{\Delta^{(j)}\in\mathcal{Y}} \ebr{-\frac{\br{ \rho \frac{7\sqrt{n}p}{64\sqrt{d}} }^2/2}{p\sqrt{nd} \opnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}} + \rho \frac{7\sqrt{n}p}{32\sqrt{d}} \opnorminf{\Delta^{(j)}} /3}}\\
% &\leq 2d\sup_{\Delta^{(j)}\in\mathcal{Y}} \ebr{-\frac{\br{ \rho \frac{7\sqrt{n}p}{32\sqrt{d}} }^2/2}{np \opnorminf{\Delta^{(j)}}^2 + \rho \frac{7\sqrt{n}p}{32\sqrt{d}} \opnorminf{\Delta^{(j)}} /3}}\\
% {\color{red} \text{above inequaltiy replaced by} }&{\color{red} \leq 2d\sup_{\Delta^{(j)}\in\mathcal{Y}} \ebr{-\frac{\br{ \rho \frac{7\sqrt{n}p}{32\sqrt{d}} }^2/2}{p\sqrt{nd} \opnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}} + \rho \frac{7\sqrt{n}p}{32\sqrt{d}} \opnorminf{\Delta^{(j)}} /3}} }\\
&\leq 2d\sup_{\Delta^{(j)}\in\mathcal{Y}} \ebr{ - \frac{3}{8} \frac{ \rho \frac{7\sqrt{n}p}{64\sqrt{d}}}{ \opnorminf{\Delta^{(j)}}}}\\
&\leq 2d \ebr{-\frac{3}{512} \frac{\rho\sqrt{n}p}{\sqrt{d}\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}},\numberthis \label{eqn:31}
\end{align*}
where in the first and third inequality, we have used (\ref{eqn:rho_2}) and the upper bound on $\opnorm{\Delta^{(j)}}$ per $\mathcal{Y}$, and in the last inequality, we have used the upper bound on $\opnorminf{\Delta^{(j)}}$ per $\mathcal{Y}$.
\textbf{For the second term in (\ref{eqn:27})}, by (\ref{eqn:3}), we have
\begin{align*}
&\E\indic{4\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\leq \E\indic{ 4 \sigma \sqrt{2dh_R} \opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \opnorm{\Lambda^{-1}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf}\\
&\leq \E\indic{\frac{32 \sigma \sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf}\\
&\leq \E\indic{\frac{32 \sigma \sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf_j}.
\end{align*}
By the independence among $\{A_{jk}\}_{k\neq j}$, $\{W_{jk}\}_{k\neq j}$, and $\Delta^{(j)}$, following the same argument as used in (\ref{eqn:28}), we have
\begin{align*}
& \E\indic{4\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
& = \E \br{\pbr{\frac{32 \sigma \sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}},\mathf_j \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)}}} \\
& = \E \br{ \pbr{\frac{32 \sigma \sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)}} \indic{\Delta^{(j)}\in\mathf_j}}\\
&\leq \E\br{ \pbr{\frac{32 \sigma \sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)}} \indic{\Delta^{(j)}\in\mathf_j} }\\
&\leq \E\br{ \pbr{ \opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{7 \sqrt{n}p}{32\sigma\sqrt{d}} \Bigg| \{A_{jk}\}_{k\neq j},\Delta^{(j)} } \indic{\Delta^{(j)}\in\mathf_j} }\\
&\leq \E \left( \br{ 2\ebr{- \frac{c\br{\rho \frac{7 \sqrt{n}p}{32\sigma\sqrt{d}} }^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}}} + \indic{\frac{\rho \frac{7 \sqrt{n}p}{32\sigma\sqrt{d}}}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2}} <4\sqrt{d}} }\indic{\Delta^{(j)}\in\mathf_j} \right),
\end{align*}
for some constant $c>0$,
where the second to last inequality is by the fact that ${h_R}\geq 2$ and the last inequality is by Lemma \ref{lem:Fj2}. Since
\begin{align*}
\ebr{- \frac{c\br{\rho \frac{7 \sqrt{n}p}{32\sigma\sqrt{d}} }^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}}} &\leq \ebr{-\frac{np}{\sigma^2}} + \indic{\frac{c\br{\rho \frac{7 \sqrt{n}p}{32\sigma\sqrt{d}} }^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}} \leq \frac{np}{\sigma^2}},
%&\leq \ebr{-\frac{np}{\sigma^2}} + \indic{ \opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} \geq \frac{c^2 \rho^2p }{16 d} },
\end{align*}
we have
\begin{align*}
& \E\indic{2\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\leq \E \left( \br{2\ebr{-\frac{np}{\sigma^2}} + 3\indic{\frac{\br{\rho \frac{7 \sqrt{n}p}{32\sigma\sqrt{d}} }^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}} \leq \frac{np}{c\sigma^2} \wedge 16d} }\indic{\Delta^{(j)}\in\mathf_j} \right)\\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 3 \E \indic{\frac{\br{\rho \frac{7 \sqrt{n}p}{32\sigma\sqrt{d}} }^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}} \leq \frac{np}{c\sigma^2} }\indic{\Delta^{(j)}\in\mathf_j} ,
\end{align*}
where the last inequality holds as long as $\frac{np}{\sigma^2d}\geq 16c$. Then characterizing the above display is about controlling the tail probabilities of $\opnormt{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}$. Similar to the establishment of (\ref{eqn:28}), we have
\begin{align*}
& \E\indic{4\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 3 \sup_{\Delta^{(j)}\in \mathcal{Y}} \pbr{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} \geq \frac{7^2\rho^2 cp}{32^2d}\Bigg| \Delta^{(j)}}.
\end{align*}
Assume $\rho$ satisfies
\begin{align}
& \frac{1}{2}\frac{7^2\rho^2 cp}{32^2d} \geq p\br{ \frac{9C_0(1+\sigma\sqrt{d})}{\sqrt{np}}}^2. \label{eqn:rho_3}
\end{align}
By Lemma \ref{lem:Fj2_denominator}, we have
\begin{align*}
& \E\indic{4\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 6d \sup_{\Delta^{(j)}\in \mathcal{Y}} \ebr{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} \geq p \opnorm{\Delta^{(j)}}^2 + \frac{1}{2}\frac{7^2\rho^2 cp}{32^2d} }\\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 6d \sup_{\Delta^{(j)}\in \mathcal{Y}} \ebr{ - \frac{\br{ \frac{1}{2}\frac{7^2\rho^2 cp}{32^2d} }^2/2}{ p \opnorminf{\Delta^{(j)}}^2\opnorm{\Delta^{(j)}}^2 + \opnorminf{\Delta^{(j)}}^2 \frac{1}{2}\frac{7^2\rho^2 cp}{16^2d} /3}}\\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 6d \sup_{\Delta^{(j)}\in \mathcal{Y}} \ebr{-\frac{3}{8} \frac{ \frac{1}{2}\frac{7^2\rho^2 cp}{32^2d}}{\opnorminf{\Delta^{(j)}}^2} } \\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 6d \ebr{-\frac{3}{8} \frac{ \frac{1}{2}\frac{7^2\rho^2 cp}{32^2d}}{7^2\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}^2} }\\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 6d \ebr{ -\frac{3c }{16\times 32^2} \br{ \frac{\rho \sqrt{p}}{\sqrt{d} \br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}}^2}.\numberthis \label{eqn:32}
\end{align*}
\textbf{For the third term in (\ref{eqn:27})}, using (\ref{eqn:3}) and the fact that $h_R\geq 2$, we have
\begin{align*}
& \E \indic{ 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\leq \E \indic{ 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \frac{\sigma}{\sqrt{n}} \opnorm{\Lambda^{-1}}\opnorm{\sum_{k\neq j} A_{jk}W_{jk}} \geq \rho \frac{h_R}{\sqrt{n}}}\\
& \leq \E \indic{\opnorm{\sum_{k\neq j} A_{jk}W_{jk}} \geq \frac{7}{8\times 46C_0\sqrt d\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} } \frac{\rho np}{\sigma} }\\
& = \E \br{\pbr{\opnorm{\sum_{k\neq j} A_{jk}W_{jk}} \geq \frac{7}{8\times 46C_0\sqrt d\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} }\frac{\rho np}{\sigma} \Bigg| \{A_{jk}\}_{k\neq j}} }.
\end{align*}
%Since $\frac{np}{(\sqrt{d} + \sigma d)^2} \geq 22^2 C_0^2$, we have $\frac{7}{8\times 23C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} } \geq 0.8\sqrt{d}$. Assume $\frac{np}{\sigma}$
By the independence between $\{A_{jk}\}_{k\neq j}$ and $\{W_{jk}\}_{k\neq j}$, we have $\sum_{k\neq j} A_{jk}W_{jk} | \{A_{jk}\}_{k\neq j} \dequal \sqrt{\{A_{jk}\}_{k\neq j}} \xi$ for some $\xi \sim \MN(\zero,I_d,I_d)$ that is independent of $\{A_{jk}\}_{k\neq j}$
Using Lemma \ref{lem:vershynin_new}, we have
\begin{align*}
& \E \indic{ 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
&\leq \E \Bigg( 2\ebr{-c \br{\frac{1}{2} \frac{7}{8\times 46C_0\sqrt d\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}}}\frac{\rho np}{\sigma } }^2 \frac{1}{{\sum_{k\neq j}A_{jk}}} } \\
&\quad + \indic{\frac{1}{2} \frac{7}{8\times 46C_0\sqrt d\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} }\frac{\rho np}{\sigma} \frac{1}{\sqrt{\sum_{k\neq j}A_{jk}}} < 2\sqrt{d} }\Bigg).
\end{align*}
%$\sum_{k\neq j} A_{jk}W_{jk} \dequal \br{\sum_{k\neq j} A_{jk}} W_{j1}$
Note that by Bernstein's inequality, there exists some constant $C_3>0$ such that
\begin{align}\label{eqn:29}
\pbr{\sum_{k\neq j} A_{jk} \geq np + C_3\sqrt{np\log n}}\leq n^{-10}.
\end{align}
We further assume $\frac{np}{\log n}\geq C_3^2$ such that $2np\geq np + C_3\sqrt{np\log n}$ and consequently $\pbr{\sum_{k\neq j} A_{jk} \geq 2np}\leq n^{-10}$.
Assume $\rho$ satisfies
\begin{align}
\frac{1}{ \sqrt{2np}}\br{\frac{1}{2} \frac{7}{8\times 46C_0\sqrt d\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} }\frac{\rho np}{\sigma}} \geq c^{-\frac{1}{2}} \vee \br{2\sqrt{d}}.\label{eqn:rho_4}
\end{align}
We have
\begin{align*}
\E \indic{ 46C_0\br{\frac{1+\sigma \sqrt{d}}{\sqrt{np}}} \sqrt{2dh_R} \opnorm{G_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} &\leq 2\ebr{-\frac{np}{\sigma^2}} + 2\pbr{\sum_{k\neq j} A_{jk} \geq 2np}\\
&\leq 2\ebr{-\frac{np}{\sigma^2}} + 2n^{-10}.\numberthis \label{eqn:33}
\end{align*}
\textbf{For the fourth and last term in (\ref{eqn:27})}, by properties of the Gaussian distribution, one can verify that $\iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \Big | \{A_{jk}\}_{k\neq j} \sim \mathn(0,4\sum_{k\neq j} A_{jk}) $. Assume $\rho$ satisfies
\begin{align}\label{eqn:rho_5}
3\rho + C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}} \leq \frac{1}{2}.
\end{align}
By the fact $h_R\geq 2$, we then have
\begin{align*}
& \E \indic{ \frac{\sigma}{\sqrt{n}}\frac{1}{np} \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \geq \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} \frac{h_R}{\sqrt{n}} }\indic{\mathf} \\
&\leq \E \indic{ \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \geq \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} \frac{2np}{\sigma} }\\
& =\E \br{\pbr{ \iprod{\sum_{k\neq j} A_{jk} W_{jk} }{R-I_d} \geq \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} \frac{2np}{\sigma} \Bigg |\{A_{jk}\}_{k\neq j}}}\\
&\leq \E \ebr{- \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}}_+^2 \frac{(np)^2}{2\sigma^2 \sum_{k\neq j} A_{jk}}}\\
&\leq \ebr{- \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} _+^2 \frac{(np)^2}{2\sigma^2 \br{np+C_3\sqrt{np\log n} }}} \\
&\quad + \pbr{\sum_{k\neq j}A_{jk}\geq np+C_0\sqrt{np\log n} }\\
&\leq \ebr{- \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}} _+^2 \br{1+C_3\sqrt{\frac{\log n}{np}}}^{-1} \frac{np}{2\sigma^2}} + n^{-10}, \numberthis \label{eqn:34}
\end{align*}
where the last inequality is by (\ref{eqn:29}).
\textbf{Obtaining an upper bound on (\ref{eqn:27}).} Plugging (\ref{eqn:31}), (\ref{eqn:32}), (\ref{eqn:33}), and (\ref{eqn:34}) into (\ref{eqn:27}), we have
\begin{align*}
&\E\indic{ \fnorm{U_j \hatH^\top - U_j^*\hat P^\top} \geq \fnorm{U_j \hatH^\top - \frac{R \hat P^\top}{\sqrt n} } } \indic{\mathf} \\
&\leq 2d \ebr{-\frac{3}{512} \frac{\rho\sqrt{n}p}{\sqrt{d}\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}} \\
&\quad + 2\ebr{-\frac{np}{\sigma^2}} + 6d \ebr{ -\frac{3c }{16\times 32^2} \br{ \frac{\rho \sqrt{p}}{\sqrt{d} \br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}}^2}\\
&\quad+ 2\ebr{-\frac{np}{\sigma^2}} + 2n^{-10} \\
&\quad + \ebr{- \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}}_+^2 \br{1+C_3\sqrt{\frac{\log n}{np}}}^{-1} \frac{np}{2\sigma^2}} + n^{-10}\\
&\leq \ebr{- \br{1- 3\rho - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}}_+^2 \br{1+C_3\sqrt{\frac{\log n}{np}}}^{-1} \frac{np}{2\sigma^2}} + 3n^{-10} \\
&\quad + 4\ebr{-\frac{np}{\sigma^2}} + 2d \ebr{-\frac{3}{512} \frac{\rho\sqrt{n}p}{\sqrt{d}\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}} \\
&\quad + 6d \ebr{ -\frac{3c }{16\times 32^2} \br{ \frac{\rho \sqrt{p}}{\sqrt{d} \br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}}^2}.
\end{align*}
We let $\rho = \br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4}$. Then, under the assumption that $d$ is a constant, $\frac{np}{\log^3 n},\frac{np}{\sigma^2}\geq C_3$ for some large enough constant $C_3>0$, we can make sure conditions (\ref{eqn:rho_2}), (\ref{eqn:rho_3}), (\ref{eqn:rho_4}), and (\ref{eqn:rho_5}) are all satisfied. In addition, we have
\begin{align*}
&\frac{1}{\log n} \frac{\rho\sqrt{n}p}{\sqrt{d}\br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}} \\
&\geq \frac{\rho np}{\sqrt{d}(C''\vee 1) (\log n) \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}} \\
&\geq \frac{C_3\br{\frac{1}{\log n}}^\frac{1}{4} \log^3 n}{\sqrt{d}(C''\vee 1) (\log n) \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}} \\
& =\frac{C_3 (\log n)^{1.25}}{\sqrt{d}(C''\vee 1)\br{ \frac{1}{\sqrt{\log n}} \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} + \frac{\sigma}{\sqrt{np}}}}.
\end{align*}
Similarly, we have
\begin{align*}
&\frac{1}{\sqrt{\log n}}\frac{\rho \sqrt{p}}{\sqrt{d} \br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}\\
&\geq \frac{\sqrt{C_3} (\log n)^{0.25}}{\sqrt{d}(C''\wedge1) \br{ \frac{1}{\sqrt{\log n}} \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} + \frac{\sigma}{\sqrt{np}}}}.
\end{align*}
\textbf{Putting things together.} We have
\begin{align*}
&\E\indic{ \fnorm{U_j \hatH^\top - U_j^*\hat P^\top} \geq \fnorm{U_j \hatH^\top - \frac{R \hat P^\top}{\sqrt n} } } \indic{\mathf} \\
&\leq \ebr{- \br{1- 3\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4}} - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}}_+^2 \br{1+C_3\sqrt{\frac{\log n}{np}}}^{-1} \frac{np}{2\sigma^2}} + 3n^{-10} \\
&\quad + 4\ebr{-\frac{np}{\sigma^2}} + 2d \ebr{-\frac{3 \log n}{512} \frac{C_3 (\log n)^{1.25}}{\sqrt{d}(C''\vee 1)\br{ \frac{1}{\sqrt{\log n}} \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} + \frac{\sigma}{\sqrt{np}}}}} \\
&\quad + 6d \ebr{ -\frac{3c \log n}{16\times 32^2} \br{\frac{\sqrt{C_3} (\log n)^{0.25}}{\sqrt{d}(C''\wedge1) \br{ \frac{1}{\sqrt{\log n}} \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} + \frac{\sigma}{\sqrt{np}}}}} ^2}\\
&\leq \ebr{- \br{1- 3\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4}} - C_2\sqrt d\br{\frac{\sqrt{\log n}+ \sigma\sqrt{d}}{\sqrt{np}}}}_+^2 \br{1+C_3\sqrt{\frac{\log n}{np}}}^{-1} \frac{np}{2\sigma^2}} + 3n^{-10} \\
&\quad + 4\ebr{-\frac{np}{\sigma^2}} + 9dn^{-10},
\end{align*}
where the last inequality holds when $\frac{np}{\log n}, \frac{np}{\sigma^2}$ are greater than a sufficiently large constant factor. Note that the term $9dn^{-10}$ comes from combining the two $d\ebr{.}$ terms and assuming that $\log n$ is greater than a sufficiently large constant factor. Then there exists a constant $C_4>0$ such that
\begin{align*}
&\E\indic{ \fnorm{U_j \hatH^\top - U_j^*\hat P^\top} \geq \fnorm{U_j \hatH^\top - \frac{R \hat P^\top}{\sqrt n} } } \indic{\mathf} \\
&\leq \ebr{-\br{1-C_4\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4} }}_+\frac{np}{2\sigma^2}} + 12dn^{-10}. \numberthis\label{eqn:tail-bound}
%&\leq \ebr{-\br{1-C_4\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4} }}\frac{np}{2\sigma^2}} + 12dn^{-10},
\end{align*}
Combining (\ref{eqn:36}), (\ref{eqn:35}), (\ref{eqn:26}) and (\ref{eqn:tail-bound}) we have
\begin{align*}
\E \ell(\hat Z,Z^*) &\leq \frac{1}{n}\sum_{j\in[n]}\sum_{R\in \Pi_d:R\neq I_d} \br{\ebr{-\br{1-C_4\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4} }}_+\frac{np}{2\sigma^2}} + 12dn^{-10}} + \pbr{\mathf^\mathrm{c}}\\
&\leq d^d \br{\ebr{-\br{1-C_4\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4} }}_+\frac{np}{2\sigma^2}} + 12dn^{-10}} + 7dn^{-9}\\
&\leq \ebr{-\br{1-C_5\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4} }}_+\frac{np}{2\sigma^2}} + 8dn^{-9}\\
&\leq \ebr{-\br{1-C_5\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4} }}\frac{np}{2\sigma^2}} + 8dn^{-9},
\end{align*}
for some constant $C_5>0$, where the last inequality holds when $\frac{np}{\sigma^2}$ is large enough.
%\begin{align*}
%&\frac{1}{\log n}\frac{\rho \sqrt{p}}{\sqrt{d} \br{\frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}} + \frac{1}{\sqrt{n}}}}\\
%& \geq \frac{\rho \sqrt{np}}{\sqrt{d}(\log n) (C''\wedge 1)\br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}}\\
%&\geq \frac{\sqrt{C_3}(\log n)^{\frac{3}{4}}}{\sqrt{d}(C''\wedge1) \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}} \\
%&\geq \frac{\sqrt{C_3}(\log n)^{\frac{1}{4}}}{\sqrt{d}(C''\wedge1) \br{ \frac{1}{\sqrt{\log n}} \br{1+ \sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}} + \frac{\sigma}{\sqrt{np}}}}.
%\end{align*}
%\begin{align*}
%& \E\indic{2\sqrt{2dh_R} \opnorm{F_{j2}} \geq \rho \frac{h_R}{\sqrt{n}}}\indic{\mathf} \\
%&\leq \E \br{ \pbr{\frac{16 \sigma \sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \{A_{jk}\}_{k\neq j}, \mathf_j}\pbr{\mathf_j}}\\
%&\leq \E \br{ \sup_{\Delta^{(j)}\in\mathcal{Y}} \pbr{\frac{16 \sigma \sqrt{2dh_R} }{7np}\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{h_R}{\sqrt{n}} \Bigg| \{A_{jk}\}_{k\neq j},\Delta^{(j)}}}\\
%&\leq \E \br{ \sup_{\Delta^{(j)}\in\mathcal{Y}} \pbr{\opnorm{\sum_{k\neq j} A_{jk} W_{jk}\Delta^{(j)}_k} \geq \rho \frac{7 \sqrt{n}p}{16\sigma\sqrt{d}} \Bigg| \{A_{jk}\}_{k\neq j},\Delta^{(j)}}}\\
%&\leq \E \br{ \sup_{\Delta^{(j)}\in\mathcal{Y}} \br{ 2\ebr{- \frac{c\br{\rho \frac{7 \sqrt{n}p}{16\sigma\sqrt{d}} }^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}}} + \indic{\rho \frac{7 \sqrt{n}p}{16\sigma\sqrt{d}} <4\sqrt{d}\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2}}}}
%\end{align*}
%for some constant $c>0$,
%where the last inequality is due to Lemma \ref{lem:Fj2}.
%\begin{align*}
% &\leq \indic{\iprod{U_j H - U_j^*}{R-I_d} + \fnorm{U_j \br{M - PH^\top}^\top}\fnorm{ R-I_d} \geq \frac{h_R}{\sqrt{n}}}\\
% &\leq \indic{\iprod{U_j H - U_j^*}{R-I_d} + \opnorm{U_j} \fnorm{M - PH^\top}\fnorm{ R-I_d} \geq \frac{h_R}{\sqrt{n}}}\\
%\end{align*}
\subsection{Lemmas}
\begin{lemma}\label{lem:population}
We have $\lambda_i(\E A\otimes I_d)=(n-1)p$ for all $i\leq d$ and $\lambda_i(\E A\otimes I_d)=-p$ for all $d+1\leq i\leq nd$.
%For any $j\in[n]$, we have $U^*_j =I_d/\sqrt{n}$.
%In addition, there exists some $O\in O(d)$ such that $U^*_j = O/\sqrt{n}$ for all $j\in[n]$. As a result, $\opnorm{U^*_j}=1$ for all $j\in[n]$.
\end{lemma}
\begin{proof} Recall that $\E A = pJ_n - pI_n$. The eigenvalues of the matrix $\E A$ are characterized as follows.
$$ \lambda_1(\E A) = (n-1)p \,,$$
$$\lambda_2(\E A) = \ldots = \lambda_n(\E A) = -p\,. $$
Therefore the eigenvalues of $\E A \otimes I_d$ are as follows.
$$ \lambda_1(\E A\otimes I_d) = \ldots = \lambda_d(\E A \otimes I_d) = (n-1)p \,,$$
$$\lambda_{d+1}(\E A\otimes I_d) = \ldots = \lambda_{dn}(\E A \otimes I_d) = -p\,. $$
%The conclusion about $\{U^*_j\}$ can be easily verified and its proof is omitted.
\end{proof}
%\begin{lemma}\label{lem:degree}
%Assume $p\geq 1/n$. Then
%\begin{align*}
%\max_{j\in[n]}\abs{\sum_{k\neq j}A_{jk}-np}\leq \sqrt{np\log n}
%\end{align*}
%with probability at least $1-n^{-10}$.
%\end{lemma}
%\begin{proof}
%By Bernstein's inequality, we have
%\end{proof}
\begin{lemma}\label{lem:event-E-prime_new}
There exist constants $C_0'>0, C_0 > 7$ such that if $\frac{np}{\log n} > C_0'$, then the following event holds
\begin{align*}
\calE &= \left\{ \opnorm{A - \E A} \leq C_0\sqrt{np} ,\; \opnorm{(A \otimes J_d)\circ W} \leq C_0\sqrt{npd},\; \max_{j\in[n]}\abs{\sum_{k\neq j}A_{jk}-np}\leq C_0\sqrt{np\log n} \right\}\,
\end{align*}
with probability at least $1- n^{-10}$. Under the event $\calE$, we have
\begin{align}
\opnorm{X - (\E A \otimes J_d)} ,\max_{i\in[d]} \abs{\lambda_i(X) - (n-1)p} ,\max_{d+1\leq i \leq nd} \abs{\lambda_i(X)+p} &\leq C_0\br{1+\sigma\sqrt{d}}\sqrt{np},\label{eqn:2}\\
\max_{j\in[n]} \opnorm{X_j} \leq p\sqrt{n} + C_0\br{1+\sigma\sqrt{d}}\sqrt{np}.\label{eqn:7}
\end{align}
If $\frac{np}{(1+\sigma \sqrt{d})^2 }\geq 64C_0^2$ is further assumed, under the event $\calE$, the following hold for $\Lambda, U$, and $H$:
\begin{align}
\opnorm{\Lambda} =\lambda_1(X) &\leq \frac{9np}{8},\label{eqn:8}\\
\opnorm{\Lambda^{-1}}^{-1} =\lambda_d(X) &\geq \frac{7np}{8},\label{eqn:3}\\
%\opnorm{\Lambda^{-1}} &\leq \frac{4}{3np},\\
\lambda_d(X)- \lambda_{d+1}(X) &\geq \frac{3np}{4},\label{eqn:4}\\
\min_{O\in \mathcal{O}_d}\opnorm{H - O}, \opnorm{UH-U^*} , \opnorm{UU^\top - U^* U^{*\top} } &\leq \frac{8C_0\br{1+\sigma\sqrt{d}}}{7\sqrt{np}},\label{eqn:5}\\
\opnorm{H\Lambda - \Lambda H} &\leq 2C_0\br{1+\sigma\sqrt{d}}\sqrt{np},\\
\opnorm{H^{-1}} &\leq \frac{4}{3},\label{eqn:6}
\end{align}
the following hold any $j\in[n]$:
\begin{align}
\opnorm{G_{j1}} &\leq \frac{2C_0}{\sqrt{n}}\br{\sqrt{\frac{\log n}{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}},\label{eqn:16}\\
\opnorm{U_j} &\leq \frac{4}{3}\br{\opnorm{U_jH-U^*_j}+\frac{1}{\sqrt{n}}},\label{eqn:10} \\
\opnorm{\loo{U}(\loo{U})^\top -UU^\top} &\leq 6\br{\opnorm{U_jH -U^*_j}+\frac{1}{\sqrt{n}}}, \label{eqn:14}\\
\opnorm{\loo{U}\loo{H} -U^*}&\leq \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}}. \label{eqn:12}
\end{align}
%\begin{align*}
%\calE' = \calE \cap \Big\{&\lambda_d(X) \geq \frac{3np}{4},\quad \lambda_d(X)- \lambda_{d+1}(X) \geq \frac{np}{2}, \\
%& \min_{O\in O(d)}\opnorm{H - O} \leq \opnorm{UU^\top - U^* U^{*\top} }\leq \frac{8C_0\br{1+\sigma\sqrt{d}}}{7\sqrt{np}}, \\
%&\opnorm{H\Lambda - \Lambda H}\leq 2C_0\br{1+\sigma\sqrt{d}}\sqrt{np},\\
%&\Big\}
%\end{align*}
%happens with probability at least $1- n^{-10}$.
\end{lemma}
\begin{proof}
The probability of $\calE$ is from
Lemma 9 of \cite{zhang2022exact} (for $\opnorm{A - \E A} $), Theorem 5.2 of \cite{lei2015consistency} (for $ \opnorm{(A \otimes J_d)\circ W} $), and Bernstein's inequality (for $\max_{j\in[n]}|\sum_{k\neq j}A_{jk}-np|$). Note that
\begin{align*}
\opnorm{X - (\E A \otimes J_d)} & = \opnorm{ (A \otimes I_d) + \sigma (A \otimes J_d) \circ W - (\E A \otimes J_d)} \\
&\leq \opnorm{(A-\E A) \otimes I_d} +\sigma\opnorm{(A \otimes J_d) \circ W }\\
& = \opnorm{A -\E A } +\sigma\opnorm{(A \otimes J_d) \circ W }.
\end{align*}
By Weyl's inequality, we have $\max_{i\in[nd]}\abs{\lambda_i(X) - \lambda_i(\E A\otimes I_d)}\leq \opnorm{X - (\E A \otimes J_d)} $. Together with Lemma \ref{lem:population}, (\ref{eqn:2}) holds under the event $\calE$. For (\ref{eqn:7}), we have
\begin{align*}
\max_j\opnorm{X_j} &= \max_j \opnorm{ (\E A_j \otimes J_d)+ \br{X_j - (\E A_j \otimes J_d)}} \\
&\leq \max_j \opnorm{ (\E A_j \otimes J_d)} + \opnorm{X - (\E A \otimes J_d)}\\
&\leq p\sqrt{n} + C_0\br{1+\sigma\sqrt{d}}\sqrt{np}.
\end{align*}
If $\frac{np}{(1+\sigma \sqrt{d})^2 }\geq 64C_0^2$ is further assumed, we have $C_0({1+\sigma\sqrt{d}})\sqrt{np} \leq np/8$. Then (\ref{eqn:2}), together with the fact $\opnorm{\Lambda^{-1}} = 1/\lambda_d(X)$ and $\opnorm{\Lambda} = \lambda_1(X)$, leads to (\ref{eqn:8})-(\ref{eqn:4}).
%
%Assume $\calE$ holds. Then
%\begin{align*}
%\opnorm{X - (\E A \otimes J_d)} & = \opnorm{ (A \otimes I_d) + \sigma (A \otimes J_d) \circ W - (\E A \otimes J_d)} \\
%&\leq \opnorm{(A-\E A) \otimes I_d} +\sigma\opnorm{(A \otimes J_d) \circ W }\\
%& = \opnorm{A -\E A } +\sigma\opnorm{(A \otimes J_d) \circ W }\\
%&\leq C_0\br{1+\sigma\sqrt{d}}\sqrt{np}.
%\end{align*}
%Under the assumption $\frac{np}{(1+\sigma \sqrt{d})^2 }\geq 64C_0^2$, we have $\opnorm{X - (\E A \otimes J_d)} \leq np/8$.
Since $\lambda_d(\E A \otimes J_d) - \lambda_{d+1}(\E A \otimes J_d) = np$ according to Lemma \ref{lem:population}, we have $\opnorm{X - (\E A \otimes J_d)} \leq (\lambda_d(\E A \otimes J_d) - \lambda_{d+1}(\E A \otimes J_d))/8$.
By Lemma 2 of \cite{abbe2020entrywise}, we have
\begin{align*}
\min_{O\in \mathcal{O}_d}\opnorm{H - O} \leq \opnorm{UU^\top - U^* U^{*\top} }\leq \frac{\opnorm{X - (\E A \otimes J_d)} }{\br{1-\frac{1}{8}}np}\leq \frac{8C_0\br{1+\sigma\sqrt{d}}}{7\sqrt{np}},
\end{align*}
and
\begin{align*}
\opnorm{H\Lambda - \Lambda H}\leq 2\opnorm{X - (\E A \otimes J_d)} \leq 2C_0\br{1+\sigma\sqrt{d}}\sqrt{np},
\end{align*}
and $\opnorm{H^{-1}}\leq 4/3$. Note that $\opnorm{UH-U^*} = \opnorm{UU^\top U^* - U^*U^{*\top}U^*} \leq \opnorm{UU^\top - U^*U^{*\top}}\opnorm{U^*}\leq \opnorm{UU^\top - U^*U^{*\top}}$. Hence, (\ref{eqn:5})-(\ref{eqn:6}) hold.
%\begin{align*}
%\opnorm{H^{-1}}\leq
%\end{align*}
%\begin{align*}
%\opnorm{U_j\br{H\Lambda - \Lambda H}} \leq \frac{8}{3}C_0\br{1+\sigma\sqrt{d}}\sqrt{np} \opnorm{U_j H}.
%\end{align*}
Consider any $j\in[n]$. By (\ref{eqn:6}), we have
\begin{align*}
\opnorm{U_j} &\leq \opnorm{U_j H}\opnorm{H^{-1}} \leq \br{\opnorm{U_jH-U^*_j} + \opnorm{U^*_j}}\opnorm{H^{-1}} = \frac{4}{3}\br{\opnorm{U_jH-U^*_j} +\frac{1}{\sqrt{n}} }.
\end{align*}
From (\ref{eqn:7}), (\ref{eqn:4}) and that $C_0 > 7$, we have $\opnorm{X-\loo{X}} \leq 2\opnorm{X_j} \leq (\lambda_d(X)- \lambda_{d+1}(X))/2$.
By Davis-Kahan theorem, we have
\begin{align*}
\opnorm{\loo{U}(\loo{U})^\top -UU^\top} &\leq \frac{2\opnorm{\br{X-\loo{X}}U}}{\lambda_d(X) - \lambda_{d+1}(X)} \\
&\leq \frac{2\br{\opnorm{X_j U} + \opnorm{X_j} \opnorm{U_j}}}{\lambda_d(X) - \lambda_{d+1}(X)}\\
& = \frac{2\br{\opnorm{U_j\Lambda} + \opnorm{X_j} \opnorm{U_j}}}{\lambda_d(X) - \lambda_{d+1}(X)}\\
& \leq \frac{2\br{\opnorm{U_j}\opnorm{\Lambda} + \opnorm{X_j} \opnorm{U_j}}}{\lambda_d(X) - \lambda_{d+1}(X)}\\
&\leq 4\opnorm{U_j}\\
&\leq 6\br{\opnorm{U_jH -U^*_j}+\frac{1}{\sqrt{n}}},
\end{align*}
where the second to last inequality is due to (\ref{eqn:7}), (\ref{eqn:8}), and (\ref{eqn:4}), and the last inequality is due to (\ref{eqn:10}).
%Then by (\ref{eqn:6}) and Lemma \ref{lem:population}, we have
%\begin{align*}
%\opnorm{\loo{U}(\loo{U})^\top -UU^\top} &\leq 4\opnorm{U_jH}\opnorm{H^{-1}}\\
%&\leq 6\br{\opnorm{U_jH -U^*_j}+\norm{U^*_j}}\\
%&= 6\br{\opnorm{U_jH -U^*_j}+\frac{1}{\sqrt{n}}}.
%\end{align*}
Together with (\ref{eqn:5}), we have
\begin{align*}
\opnorm{\loo{U}(\loo{U})^\top -UU^\top}&\leq 6\br{\opnorm{UH -U^*}+\frac{1}{\sqrt{n}}}\leq \frac{7C_0(1+\sigma \sqrt{d})}{\sqrt{np}}.
\end{align*}
Then,
\begin{align*}
\opnorm{\loo{U}\loo{H} - U^*} &= \opnorm{\br{\loo{U}\loo{U}^\top - UU^\top}U^* + UH- U^*} \\
&\leq \opnorm{\loo{U}\loo{U}^\top - UU^\top} + \opnorm{UH- U^*}\\
&\leq \frac{9C_0(1+\sigma \sqrt{d})}{\sqrt{np}}.
\end{align*}
For $\opnorm{G_{j1}}$, we have
\begin{align*}
\sqrt{n}\opnorm{G_{j1}} &= \opnorm{\sum_{k\neq j} A_{jk} \Lambda^{-1}}\\
&\leq \opnorm{np \Lambda^{-1}-I_d} + \opnorm{ \br{\sum_{k\neq j} A_{jk} - np}\Lambda^{-1}}\\
&\leq \opnorm{np -\Lambda} \opnorm{\Lambda^{-1}} + \abs{\sum_{k\neq j} A_{jk} - np}\opnorm{\Lambda^{-1}}\\
&\leq \max_{i\in[d]} \abs{np - \lambda_i(X)}\opnorm{\Lambda^{-1}} + \abs{\sum_{k\neq j} A_{jk} - np} \opnorm{\Lambda^{-1}}\\
&\leq 2C_0\br{\sqrt{\frac{\log n}{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}}},
\end{align*}
where the last inequality is due to $\calE$, (\ref{eqn:2}), and (\ref{eqn:3}).
\end{proof}
%To shorten notations, for each $j\in[n]$, denote
%\begin{align*}
%\Delta^{(j)} \define U^{(j)}H^{(j)} - U^* \in\mathr^{nd\times d}
%\end{align*}
%such that its block submatrices $\Delta^{(j)}_k = U^{(j)}_kH^{(j)} - U^*_k$ for each $k\in[n]$. We further define
%\begin{align*}
%\opnorminf{\Delta^{(j)}}\define \max_{k\in[n]} \opnorm{\Delta^{(j)}_k}.
%\end{align*}
\begin{lemma}\label{lem:Fj1}
For any $j\in[n]$, we have
\begin{align*}
\pbr{\opnorm{\sum_{k\neq j} A_{jk} \Delta_k^{(j)}} \geq p\sqrt{n} \opnorm{\Delta^{(j)}} + t\Bigg| \Delta^{(j)}} &\leq 2d\ebr{- \frac{t^2/2}{np\opnorminf{\Delta^{(j)}}^2 + \opnorminf{\Delta^{(j)}}t/3}}
\end{align*}
and
\begin{align*}
\pbr{\opnorm{\sum_{k\neq j} A_{jk} \Delta_k^{(j)}} \geq p\sqrt{n} \opnorm{\Delta^{(j)}} + t\Bigg| \Delta^{(j)}} &\leq 2d\ebr{- \frac{t^2/2}{p\sqrt{nd}\opnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}} + \opnorminf{\Delta^{(j)}}t/3}},
\end{align*}
for any $t>0$.
\end{lemma}
\begin{proof}
Consider any $j\in[n]$.
Then
\begin{align*}
%\opnorm{\sum_{k\neq j} A_{jk} (U^{(j)}_kH^{(j)} - U^*_k)} &=
\opnorm{\sum_{k\neq j} A_{jk}\Delta_k^{(j)} } &\leq \opnorm{\sum_{k\neq j} (A_{jk}-p)\Delta_k^{(j)}} + p\opnorm{\sum_{k\neq j} \Delta_k^{(j)}}\\
&\leq \opnorm{\sum_{k\neq j} (A_{jk}-p)\Delta_k^{(j)}} + p\sqrt{n} \opnorm{\Delta^{(j)}}.
\end{align*}
Since $\{A_{jk}-p\}_{k\neq j}$ is independent of $\Delta^{(j)}$, we use the matrix Bernstein's inequality (Lemma \ref{lem:matrix-bernstein-inequality_new}) for the operator norm of $\sum_{k\neq j} (A_{jk}-p)\Delta_k^{(j)}$. For each $k\in[n]$, note that $\E \br{(A_{jk}-p)\Delta_k^{(j)} | \Delta_k^{(j)} }=0$ and $\opnorm{(A_{jk}-p)\Delta_k^{(j)}}\leq \opnorm{\Delta_k^{(j)}}\leq \opnorminf{\Delta^{(j)}}$. For the matrix variance term, we have
\begin{align*}
&\max\left\{ \opnorm{\E \left[ \sum_{k\neq j} (A_{jk}-p)^2 {\Delta_k^{(j)}}^\top \Delta_k^{(j)} \Bigg| \Delta^{(j)} \right]} , \opnorm{\E \left[ \sum_{k\neq j} (A_{jk}-p)^2 \Delta_k^{(j)} {\Delta_k^{(j)}}^\top \Bigg| \Delta^{(j)} \right] } \right\}\\
&= p(1-p) \cdot \max\left\{ \opnorm{\sum_{k\neq j} {\Delta_k^{(j)}} ^\top \Delta_k^{(j)} }, \opnorm{\sum_{k\neq j} \Delta_k^{(j)}{\Delta_k^{(j)} }^\top } \right\} \\
&= p(1-p) \cdot \max\left\{ \opnorm{\Delta^{(j)}}^2, \opnorm{\br{\Delta_1^{(j)},\ldots, \Delta_n^{(j)}}^\top}^2 \right\} \\ \numberthis \label{eqn:lem_Fj1_1}
&\leq np(1-p) \opnorminf{\Delta^{(j)}}^2 \leq np \opnorminf{\Delta^{(j)}}^2.
\end{align*}
%In addition, we have $\opnorm{(A_{jk}-p)}$
Then we have
\begin{align*}
\pbr{\opnorm{\sum_{k\neq j} (A_{jk}-p)\Delta_k^{(j)}} \geq t\Bigg| \Delta^{(j)}} &\leq 2d\ebr{- \frac{t^2/2}{np\opnorminf{\Delta^{(j)}}^2 + \opnorminf{\Delta^{(j)}}t/3}}.
\end{align*}
Hence,
\begin{align}
\pbr{\opnorm{\sum_{k\neq j} A_{jk} \Delta_k^{(j)}} \geq p\sqrt{n} \opnorm{\Delta^{(j)}} + t\Bigg| \Delta^{(j)}} &\leq 2d\ebr{- \frac{t^2/2}{np\opnorminf{\Delta^{(j)}}^2 + \opnorminf{\Delta^{(j)}}t/3}}. \label{eqn:lem_Fj1_2}
\end{align}
In addition, note that $ \opnorm{\Delta^{(j)}}^2\leq \sqrt{n}\opnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}}$ and
\begin{align*}
\opnorm{\br{\Delta_1^{(j)},\ldots, \Delta_n^{(j)}}^\top}^2&\leq \sqrt{n}\opnorm{\br{\Delta_1^{(j)},\ldots, \Delta_n^{(j)}}^\top}\opnorminf{\Delta^{(j)}}\\
&\leq \sqrt{n}\fnorm{\br{\Delta_1^{(j)},\ldots, \Delta_n^{(j)}}^\top}\opnorminf{\Delta^{(j)}}\\\\
&= \sqrt{n}\fnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}}\\
&\leq \sqrt{nd}\opnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}}.
\end{align*}
Then (\ref{eqn:lem_Fj1_1}) can also be upper bounded by $p\sqrt{nd}\opnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}}$. Following a similar argument that leads to (\ref{eqn:lem_Fj1_2}), we also have
\begin{align*}
\pbr{\opnorm{\sum_{k\neq j} A_{jk} \Delta_k^{(j)}} \geq p\sqrt{n} \opnorm{\Delta^{(j)}} + t\Bigg| \Delta^{(j)}} &\leq 2d\ebr{- \frac{t^2/2}{p\sqrt{nd}\opnorm{\Delta^{(j)}}\opnorminf{\Delta^{(j)}} + \opnorminf{\Delta^{(j)}}t/3}}.
\end{align*}
\end{proof}
\begin{lemma}\label{lem:Fj2_denominator}
For any $j\in[n]$, we have
\begin{align*}
\pbr{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} \geq p \opnorm{\Delta^{(j)}}^2 + t \Bigg| \Delta^{(j)} } \leq 2d\ebr{ - \frac{t^2/2}{p \opnorminf{\Delta^{(j)}}^2 \opnorm{ \Delta^{(j)} }^2 + \opnorminf{\Delta^{(j)}}^2t/3}},
\end{align*}
for any $t\geq 0$.
\end{lemma}
\begin{proof}
We follow the proof of Lemma \ref{lem:Fj1}. We first have
\begin{align*}
\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} &\leq \opnorm{ \sum_{k\neq j}(A_{jk}-p)(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} + p\opnorm{ \sum_{k\neq j}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} \\
&\leq \opnorm{ \sum_{k\neq j}(A_{jk}-p)(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} + p \opnorm{\Delta^{(j)}}^2.
\end{align*}
Note that for each $k\in[n]$, $\E \br{(A_{jk}-p)(\Delta^{(j)}_k)^\top \Delta^{(j)}_k |\Delta^{(j)}} =0$ and $\opnorm{(A_{jk}-p)(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}\leq \opnorm{ \Delta^{(j)}_k}^2 \leq \opnorminf{\Delta^{(j)}}^2$. In addition,
\begin{align*}
&\opnorm{\E \left[ \sum_{k\neq j} (A_{jk}-p)^2 (\Delta^{(j)}_k)^\top \Delta^{(j)}_k(\Delta^{(j)}_k)^\top \Delta^{(j)}_k \Bigg| \Delta^{(j)} \right]} \\
& =p(1-p)\opnorm{\sum_{k\neq j} (\Delta^{(j)}_k)^\top \Delta^{(j)}_k(\Delta^{(j)}_k)^\top \Delta^{(j)}_k } \\
&\leq p(1-p) \opnorm{ \,\left[{\Delta^{(j)}_1}^\top \,\cdots\, {\Delta^{(j)}_n}^\top \right] \begin{bmatrix} {\Delta^{(j)}_1}{\Delta^{(j)}_1}^\top & &\\ & \cdots &\\ & & {\Delta^{(j)}_n}{\Delta^{(j)}_n}^\top \end{bmatrix} \begin{bmatrix} \Delta^{(j)}_1\\ \vdots \\ \Delta^{(j)}_n \end{bmatrix} } \\
&\leq p(1-p) \opnorm{ \begin{bmatrix} {\Delta^{(j)}_1}{\Delta^{(j)}_1}^\top & &\\ & \cdots &\\ & & {\Delta^{(j)}_n}{\Delta^{(j)}_n}^\top \end{bmatrix} } \opnorm{ \Delta^{(j)} }^2\\
&=p(1-p) \br{\max_{k\in[n]} \opnorm{\Delta^{(j)}_k{\Delta^{(j)}_k}^\top}} \opnorm{ \Delta^{(j)} }^2\\
&\leq p(1-p) \opnorminf{\Delta^{(j)}}^2 \opnorm{ \Delta^{(j)} }^2\,.
\end{align*}
Hence,
\begin{align*}
\pbr{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k} \geq p \opnorm{\Delta^{(j)}}^2 + t \Bigg| \Delta^{(j)} } \leq 2d\ebr{ - \frac{t^2/2}{p \opnorminf{\Delta^{(j)}}^2 \opnorm{ \Delta^{(j)} }^2 + \opnorminf{\Delta^{(j)}}^2t/3}}.
\end{align*}
\end{proof}
\begin{lemma}\label{lem:Fj2}
There exists some constant $c>0$, such that for any $t\geq 4\sqrt{d}\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2}$ and for any $j\in[n]$, we have
\begin{align*}
\pbr{\opnorm{\sum_{k\neq j}A_{jk}W_{jk}\Delta^{(j)}_k}\geq t\Bigg| \{A_{jk}\}_{k\neq j},\Delta^{(j)}}\leq 2\ebr{- \frac{ct^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}}}.
\end{align*}
\end{lemma}
\begin{proof}
Consider any $j\in[n]$. Note that $\{A_{jk}\}_{k\neq j}$, $\{W_{jk}\}_{k\neq j}$, and $\{\Delta^{(j)}_k\}_{k\neq j}$ are mutually independent of each other. By Lemma \ref{lem:useful-MN-identities_new}, we have
\begin{align*}
\sum_{k\neq j}A_{jk}W_{jk}\Delta^{(j)}_k \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)} \dequal E \br{\sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2} \Bigg| \{A_{jk}\}_{k\neq j},\Delta^{(j)}
\end{align*}
where $E \sim \MN(\zero, I_d, I_d)$ and is independent of $\{A_{jk}\}_{k\neq j}$ and $\{\Delta^{(j)}_k\}_{k\neq j}$. Since $\sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k$ is symmetric and positive semi-definite, its square root is well-defined. By Lemma \ref{lem:vershynin_new}, there exists some constant $c>0$, such that for any $t\geq 4\sqrt{d}\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2} $, we have
\begin{align*}
& \pbr{\opnorm{\sum_{k\neq j}A_{jk}W_{jk}\Delta^{(j)}_k } \geq t \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)}} \\
&= \pbr{\opnorm{E \br{\sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2}} \geq t \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)}}\\
&\leq \pbr{\opnorm{E}\opnorm{ \br{\sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2}} \geq t \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)}}\\
& = \pbr{\opnorm{E}\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}^\frac{1}{2} \geq t \Bigg| \{A_{jk}\}_{k\neq j}, \Delta^{(j)}}\\
&\leq 2\ebr{- \frac{ct^2}{\opnorm{ \sum_{k\neq j}A_{jk}(\Delta^{(j)}_k)^\top \Delta^{(j)}_k}}}.
\end{align*}
\end{proof}
\subsection{Auxiliary Lemmas}
\begin{lemma}\label{lem:matrix-bernstein-inequality_new} [Theorem 1.6 of \cite{tropp2012user}] Consider a finite sequence $\{S_k\}_{k=1}^n$ of independent random square matrices with dimension $d\times d$. Assume that each random matrix satisfies
$$\E S_k =0 \text{ and } \opnorm{S_k - \E S_k} \leq L, \,\forall k \in [n] \,.$$
Define
\begin{align*}
V &\define \max\left\{ \opnorm{\sum_k \E\left[ S_k S_k ^\top \right]}, \opnorm{ \sum_k\E\left[ S_k^{\top}S_k \right]} \right\} \,.
\end{align*}
Then
$$ \pbr{\opnorm{\sum_k S_k} \geq t} \leq 2d \cdot \ebr{-\frac{t^2/2}{V + Lt/3}} \,,$$
for any $t\geq 0$.
\end{lemma}
\begin{lemma}\label{lem:vershynin_new} [Corollary 7.3.3 of \cite{vershynin2018high}] Let $W$ be a $d\times d$ matrix with independent $\calN(0, 1)$ entries. Then there exists some constant $c>0$ such that for every $t \geq 0$, we have
$$ \Pr\left( \opnorm{W} \geq 2\sqrt{d} + t \right) \leq 2\,\ebr{-ct^2} \,. $$
\end{lemma}
\begin{lemma}\label{lem:useful-MN-identities_new}
%\textbf{Linear transformation of matrix-variate normal variables.}
Consider any $W,W',\Delta \in\mathr^{d\times d}$ such that $W,W'\iid \MN(\zero, I_d, I_d)$ and $\Delta$ is a fixed matrix.
%For two $d\times d$ matrices $W,\Delta$ where $ W \sim \MN(\zero, I_d, I_d)$,
We have
$$ W\Delta \sim \MN(\zero, I_d, \Delta^\top \Delta)\, $$
%\textbf{Sums of matrix-variate normal variables.}
%For two $d\times d$ matrices $Z_1, Z_2$ such that $Z_1\sim \MN(\zero, I_d, \Sigma_1)$ and $Z_2 \sim \MN(\zero, I_d, \Sigma_2)$, we have
and
$$ W + W' \sim \MN(\zero, I_d, \Sigma_1 + \Sigma_2) \,.$$
\end{lemma}
\begin{proof}
The first result in the lemma is a property of the matrix normal distribution. To prove the second statement, note that $Z \sim \MN(\zero, \Sigma_0, \Sigma_1)$ is equivalent to $\vec(Z) \sim \calN(0, \Sigma_1 \otimes \Sigma_0)$. Since $ \vec(W) \sim \calN(\zero, \Sigma_1 \otimes I_d) $ and $ \vec(W') \sim \calN(\zero, \Sigma_2 \otimes I_d) ,$ we have
%By definition, $Z \sim \MN(\zero, \Sigma_0, \Sigma_1)$ is equivalent to $\vec(Z) \sim \calN(0, \Sigma_1 \otimes \Sigma_0)$. Another way to view the first result in the lemma is to realize that
%$$ (W\Delta)_i = W_i^\top \Delta\,,$$
%where $W_i$ is the $i$-th row of $W$. Since each row $W_i$ is an \emph{independently} multivariate normal random variable with mean 0 and identity covariance, $W_i^\top \Delta \sim \calN(\zero, \Delta^\top \Delta)$. One can also verify that $\{W_i^\top \Delta\}_{i=1}^d$ are independently distributed. As such, $\vec(W\Delta)$ is a zero-mean multivariate normal random variable with block-structured covariance $I_d\otimes \Delta^\top \Delta$.
%
%To prove the second statement, recognize that
%$$ \vec(Z_1) \sim \calN(\zero, \Sigma_1 \otimes I_d)\,, $$
%$$ \vec(Z_2) \sim \calN(\zero, \Sigma_2 \otimes I_d) \,.$$
%Hence,
$ \vec(W) + \vec(W') \sim \calN(\zero, (\Sigma_1 +\Sigma_2) \otimes I_d).$
\end{proof}
%\section{Conclusion}\label{sect:conclusion}
%In this paper, we propose a new spectral algorithm that overcomes a subtle yet important limitation of the vanilla spectral algorithm in \cite{pachauri2013solving}. With the new algorithm, we are able to achieve essentially the minimax optimal error rate for the permutation synchronization problem, up to the correct $\frac{1}{2}$ constant. For future works, it would be very interesting to consider alternative noise models to (\ref{eqn:X-def}) such as adversarially constructed noise, discrete-valued noise or heavy-tailed noise. We hope that our insight towards designing the new spectral algorithm will inspire future works in designing more advanced algorithms to solve other instances of the phase synchronization problem.
%
%
%\newpage
%\section{Proofs of Theorems}
%
%\subsection{Preliminaries}
%\paragraph{Decomposition of $U$.}
%\begin{align*}
%UH\Lambda - XU^* & = U(H\Lambda - \Lambda H) + U\Lambda H -XU^*\\
%& = U(H\Lambda - \Lambda H) + XU H -XU^*\\
%&= U(H\Lambda - \Lambda H) + X(U H - U^*).
%\end{align*}
%Multiplied by $\Lambda^{-1}$ and after arrangements, it leads to
%\begin{align}\label{eqn:decomposition1}
%UH - U^* = U(H\Lambda - \Lambda H) \Lambda^{-1} + X(U H - U^*) \Lambda^{-1} + XU^*\Lambda^{-1} - U^* .
%\end{align}
%Consider any $j\in[n]$. Define
%\begin{align*}
%H^{(j)}= U^{(j)\top}U^*\in\mathr^{d\times d}.
%\end{align*}
%Then
%\begin{align*}
%UH-U^* &= UH - U^{(j)}H^{(j)} + U^{(j)}H^{(j)} - U^* \\
%& = (UU^\top - U^{(j)}U^{(j)\top})U^*+ U^{(j)}H^{(j)} - U^* .
%\end{align*}
%Plug it into the right-hand side of (\ref{eqn:decomposition1}), we have
%\begin{align*}
%UH - U^* & = U(H\Lambda - \Lambda H) \Lambda^{-1} + X(UU^\top - U^{(j)}U^{(j)\top})U^* \Lambda^{-1} \\
%&\quad + X(U^{(j)}H^{(j)} - U^*) \Lambda^{-1} + XU^*\Lambda^{-1} - U^*.
%\end{align*}
%Then, the $j$th block matrix of $UH - U^*$ satisfies
%\begin{align}
%U_jH - U^*_j & = U_j(H\Lambda - \Lambda H) \Lambda^{-1} + X_j(UU^\top - U^{(j)}U^{(j)\top})U^* \Lambda^{-1} \nonumber\\
%&\quad + X_j(U^{(j)}H^{(j)} - U^*) \Lambda^{-1} + X_jU^*\Lambda^{-1} - U^*_j.\label{eqn:decomposition2}
%\end{align}
%The decomposition (\ref{eqn:decomposition2}) holds for all $j\in[n]$ is the starting point to establish our main theorems.
%
%\subsection{Proof of Theorem \ref{thm:l_infty}}
%
%\import{./}{proof_of_ell_infty_main}
%
%\subsection{Proof of Theorem \ref{thm:tail-bound}}
%
%\import{./}{proof_of_tail_bound_main}
%
%\subsection{Proof of Theorem \ref{thm:main}}
%
%\import{./}{proof_of_partial_recovery_main}
%
%\newpage
%\section{Lemmas and Their Proofs}
%
%\import{./}{proof_of_ell_infty_misc}
%
%\newpage
%\import{./}{proof_of_tail_bound_misc}
%
%\newpage
%\import{./}{proof_of_partial_recovery_misc}
%
%
%
%\newpage
%\section{Additional Useful Results}
%
%\import{./}{preliminaries}
In this section, we establish theoretical guarantees for Algorithm \ref{alg:main-algo}. Without loss of generality, starting from this part of the paper, we let $Z^*_j = I_d$ for all $j\in[n]$. That is, all the latent permutations are the identity matrix. In this way, the population eigenspace $U^*$ has a simpler expression with
\begin{align}\label{eqn:U_star_simplified}
U^*_j =I_d/\sqrt{n},\forall j\in[n].
\end{align}
To establish the statistical optimality of the proposed method, we first give a justification for the choice of the anchor $M$, followed by a fine-grained perturbation analysis for $U$.
\subsection{Theoretical Justification of $M$}
We present a justification for the anchor $M$. Note that the empirical eigenspace $U$ and the population eigenspace $U^*$ are close modulo a suitable transformation. Readers who are familiar with the literature on spectral methods \cite{abbe2020entrywise} may recognize that a natural choice of transformation is
\begin{equation}
H \define U^\top U^*\,.
\end{equation}
Classical perturbation theory such as Davis-Kahan theorem can be applied to quantify the distance between $UH$ and $U^*$ (see (\ref{eqn:5}) of Lemma \ref{lem:event-E-prime_new} for its proof):
\begin{align*}
\opnorm{UH-U^*} \lesssim \frac{{1+\sigma\sqrt{d}}}{\sqrt{np}}.
\end{align*}
Though $H$ is not orthonormal,
it can be shown to be close to some orthogonal matrix (see Lemma \ref{lem:event-E-prime_new}).
Then following the intuition given in Section \ref{sec:heuristic} and applying state-of-the-art clustering analysis, we have the following proposition.
\begin{proposition}\label{prop:hat_H_new}
There exist constants $C_1,C_2,C_3>0$ such that if $\frac{np}{\log n}>C_1$ and $\frac{np}{(\sqrt{d} + \sigma d)^2}>C_2$ then the anchor $M$ constructed in Algorithm \ref{alg:main-algo} satisfies
\begin{align}\label{eqn:21}
\min_{P\in\Pi_d}\fnorm{\hatH - PH^\top}\leq \frac{C_3(\sqrt{d} + \sigma d)}{\sqrt{np}},
\end{align}
with probability at least $1- n^{-10}$.
\end{proposition}
Proposition \ref{prop:hat_H_new} shows that $ M$ is an approximation of $H^\top$ up to some permutation matrix $\hat P\define \argmin_{P\in\Pi_d} \|\hatH - PH^\top\|_{\rm F}$. The existence of $\hat P$ is due to the fact that the clusters are identifiable only up to a permutation in cluster analysis. (\ref{eqn:21}) shows the estimation error goes to 0 when $\frac{np}{(\sqrt{d}+\sigma d)^2}$ grows. As a comparison, the operator norm of the matrix $H$ is `of a constant order' as it is close to an orthogonal matrix. Hence, the error incurred by $M$ as an estimate of $H^\top$ is of diminishing proportion. It is also worth mentioning that Proposition \ref{prop:hat_H_new} holds under weaker assumptions compared to Theorem \ref{thm:intro}. It only requires $np \gtrsim \log n$ and allows $d$ to grow as long as $np \gtrsim (\sqrt{d} + \sigma d)^2$. With Proposition \ref{prop:hat_H_new}, we can have a decomposition of the quantity $U_jM^\top$, the key quantity in the proposed method (\ref{eqn:alg2}):
\begin{align*}
U_jM^\top -U_j^* \hat P^\top &= U_j(\hat PH^\top + M - \hat PH^\top)^\top -U_j^* \hat P^\top \\
&= \br{U_j H - U_j^*}\hat P^\top + U_j(M - \hat PH^\top)^\top\\
&\approx \br{U_j H - U_j^*}\hat P^\top. \numberthis \label{eqn:approx1}
\end{align*}
The approximation (\ref{eqn:approx1}) is due to (\ref{eqn:21}) as the term above $ U_j(M - \hat PH^\top)^\top$ turns out to be negligible. Hence, the difference $U_jM^\top -U_j^* \hat P^\top $ is primarily about the perturbation $U_j H - U_j^*$. To analyze our spectral method, we need to have a deep understanding on the behaviors of the blockwise perturbation $U_j H - U_j^*$.
%To its behavior, we need to have a uniformly
%$U_jM^\top$, the key quantity in the proposed method (\ref{eqn:alg2}), and its deviation from $U^*_j$ up to a global permutation $\hat P$, we need to
%\subsection{Block-wise $\ell_\infty$ Perturbation Analysis}
%Our first contribution is a $\ell_{\infty}$ type bound for the blockwise deviation between the empirical eigenspaces $U$ and $U^*$ modulo a suitable transformation. Readers who are familiar with the literature on spectral methods may recognize that the optimal \emph{rotational transformation} from $U$ to $U^*$ is theoretically well defined. Let
%\begin{equation}
%H \define U^\top U^*\,.
%\end{equation}
%The matrix sign function \cite{gross2011recovering} $\sign{H} \define QV^\top$ where $QSV^\top$ is the singular value decomposition of $H$ satisfies $\sign{H} = \argmin_{O \in \O_d} \fnorm{UO - U^*}$.
%To simplify the overall analysis, we focus on bounding $\opnorm{UH - U^*}$ instead of $\opnorm{U\sign{H} - U^*}$. However, one can show that $\opnorm{H -\sign{H}}$ is small and a bound on $\opnorm{U\sign{H} - U^*}$ can be easily derived from Theorem \ref{thm:l_infty_new}.
\subsection{Statistical Optimality}\label{sec:optimality}
In Theorem \ref{thm:l_infty_new}, we first derive a block-wise $\ell_\infty$ upper bound for $UH-U^*$, i.e., an upper bound for $\opnormt{U_j H - U^*_j}$ that holds uniformly across all $j\in[n]$.
\begin{theorem}\label{thm:l_infty_new}
There exist constants $C,C',C''>0$ such that if $\frac{np}{(\sigma^{4/3} \vee 1)\log n} >C$ and $\frac{np}{(\sqrt{d} + \sigma d)^2}\geq C'$, we have
\begin{align}\label{eqn:22}
\max_{j\in[n]} \opnorm{U_j H - U^*_j}\leq \frac{ C''}{\sqrt{n}}\br{\sqrt{\frac{\log n }{np}} + \frac{\sigma \sqrt{d}}{\sqrt{np}} + \frac{\sigma \sqrt{\log n}}{\sqrt{np}}}
\end{align}
with probability at least $1-6dn^{-9}$.
\end{theorem}
The upper bound in (\ref{eqn:22}) is equal to the upper bound on $\opnorm{UH - U^*}$
%established in \cite{zhang2022exact}
(see Equation (\ref{eqn:5})) multiplied by a $\sqrt{(\log n)/n}$ factor. This is because $UH-U^*$ consists of $n$ blocks $\{U_jH-U^*\}_{j\in[n]}$ and all blocks behave similarly. The magnitude of each block is then on average $1/\sqrt{n}$ of that of the whole matrix, and the $\sqrt{\log n}$ factor is due to the use of union bound to control the supreme. Theorem \ref{thm:l_infty_new} assumes $\frac{np}{(\sigma^{4/3} \vee 1)\log n} \gtrsim 1$. A sufficient condition is ${np}/{\log^3 n}\gtrsim 1$ as $np/\sigma^2\gtrsim 1$ is implied by the other assumption ${np}/{(\sqrt{d} + \sigma d)^2}\gtrsim 1$.
% Note that $U^*$ is not observed and neither is $H$. The vanilla spectral algorithm uses the fact that $U_j \approx U_j^* \sign{H}^\top$ and $
% U_jU_1^\top \approx U_j^*\sign{H}^\top \sign{H} (U_1^*)^\top = U_j^*(U_1^*)^\top = \frac{1}{n} Z_j^*(Z_1^*)^\top$ to recover the permutations without explicit knowledge of $H$.
%As hinted at before, we can estimate $H$ well \emph{up to a global permutation}. Since $U_j^* = \frac{Z_j^*}{\sqrt n} \,\forall j \in [n]$, the true eigenspaces $U^*$ contain $d$ types of rows $\{e_1^\top, \ldots e_d^\top\}$ each of cardinality exactly $n$. The euclidean distance between any two types of rows is largely preserved under a linear transformation $H^\top$. The empirical eigenspaces $U$, being noisy estimates of $U^* H^\top$, thus exhibit clustering structures where the cluster centers are close, modulo a permutation of the labels, to the transformed rows $\{e^\top_i H^\top\}_{i=1}^d$.
%
%Define $\hat P \define \argmin_{P\in \Pi_d} \, \fnorm{\hatH - P H^\top}$. As noted before, the key to the improved performance of our spectral algorithm is the construction of the anchor $M$ in Algorithm \ref{alg:main-algo}.
%One can see that the bound on $\fnorm{M - \hat P H^\top}$ is smaller than that on $\fnorm{U_1 H - U_1^*}$ by a $O(\sqrt{\log n})$ factor. This crucial difference allows us to overcome the limitation of the previous works which analyze the vanilla algorithm and obtain a sharp bound on the Hamming loss.
%\subsection{Statistical Optimality}
With the upper bound for each $\opnormt{U_j H - U^*_j}$ derived, we further study the tail behavior of each difference $U_j H - U^*_j$. This leads to a sharp theoretical analysis of the performance of the proposed spectral method. The main theoretical result of this paper is stated below in Theorem \ref{thm:main_new}.
\begin{theorem}\label{thm:main_new}
Assume $2\leq d\leq \bar C$ for some constant $\bar C>0$.
There exist constants $\bar C',\bar C'',\bar C'''>0$ such that if $\frac{np}{\log^3 n}>\bar C'$ , $\frac{np}{\sigma^2}>\bar C''$ then the estimates $\hat Z_1,\ldots, \hat Z_n$ of Algorithm \ref{alg:main-algo} satisfy
\begin{align*}
\E \ell(\hat Z,Z^*)\leq \ebr{-\br{1-\bar C'''\br{\br{\frac{\sigma^2}{np}}^\frac{1}{4} + \br{\frac{1}{\log n}}^\frac{1}{4} }}\frac{np}{2\sigma^2}} + n^{-8}.
\end{align*}
\end{theorem}
Theorem \ref{thm:main_new} is a non-asymptotic version of Theorem \ref{thm:intro} stated in the Introduction. By letting $np/\sigma^2$ go to infinity, the exponential term in Theorem \ref{thm:main_new} takes an asymptotic form of $\ebr{-(1-o(1))\frac{np}{2\sigma^2}}$ and matches with the minimax lower bound. In this way, we establish the statistical optimality of the proposed spectral method.
While the proof of Theorem \ref{thm:main_new} is complicated, one can still develop an intuition as to why we can achieve the error bound $\ebr{-(1-o(1))\frac{np}{2\sigma^2}}$. %, particularly with the sharp constant $\frac{1}{2}$ in the exponent.
At a high level, we will prove and use a variant of the linear approximation \cite{abbe2020entrywise} $UH - U^* \approx XU^*\Lambda^{-1} - U^*$ where $\Lambda$ is the diagonal matrix of the leading eigenvalues of $X$ (see (\ref{eqn:decomposition1})). A further decomposition using the structure of $X$ reveals that
\begin{align}\label{eqn:approx2}
U_jH-U_j^*\approx \frac{\sigma}{\sqrt{n}} \,\sum_{k\neq j} A_{jk} W_{jk} \Lambda^{-1} \approx \frac{1}{\sqrt{n}} \frac{\sigma}{np} \sum_{k\neq j}A_{jk}W_{jk}.
\end{align}
Recall that $U_j^*= I_d/\sqrt{n}$ according to (\ref{eqn:U_star_simplified}). Then $\sqrt{n }U_jH - I_d \approx \frac{\sigma}{np} \sum_{k\neq j}A_{jk}W_{jk}$ which follows a Gaussian distribution conditioned on $\{A_{jk}\}_{k\neq j}$. Since $\sum_{k\neq j}A_{jk}$ concentrates around $(n-1)p$, roughly speaking, $\sqrt{n }U_jH - I_d$ is a random matrix with each entry i.i.d. following $\mathcal{N}(0,\frac{\sigma^2}{np})$. The Gaussian tail is used to characterize the probability of the event when $\sqrt{n }U_jH$ considerably deviates away from $I_d$ and eventually leads to a probability bound of $\ebr{-(1-o(1))\frac{np}{2\sigma^2}}$.
\subsection{Block-wise Decomposition of $UH-U^*$}
In this section, we provide a block-wise decomposition of $UH-U^*$. The decomposition is the key towards the block-wise analysis in Section \ref{sec:optimality} and provides insights on how Theorem \ref{thm:l_infty_new} and Theorem \ref{thm:main_new} are established. Recall that we let $Z^*_j = I_d$ for all $j\in[n]$. Then, (\ref{eqn:X-def}) becomes
\begin{align*}
X = (A \otimes I_d) + \sigma (A \otimes J_d) \circ W.
\end{align*}
For each $j\in[n]$, define $X_j\define (X_{j1},\ldots, X_{jn})\in\mathr^{d\times nd}$ to be the $j$-th block row of $X$ and define $W_j \in \mathr^{d\times nd}$ analogously for $W$. Then we have
\begin{align}\label{eqn:1}
X_j = (A_j \otimes I_d) + \sigma (A_j \otimes J_d) \circ W_j.
\end{align}
As a result, $X =(X_1^\top,\ldots, X_n^\top)^\top$. Define $\Lambda\in\mathr^{d\times d}$ to be the diagonal matrix of the leading eigenvalues of $X$. That is,
%\begin{align}\label{eqn:Lambda_def}
%\Lambda_{ii}\define \lambda_i(X) \text{ and }\Lambda_{ik}\define 0, \forall 1\leq i\neq k\leq d.
%\end{align}
%%
$\Lambda_{ii}\define \lambda_i(X)$ and $\Lambda_{ik}\define 0$ for all $1\leq i\neq k\leq d$.
Then we have
\begin{align*}
UH\Lambda - XU^* & = U(H\Lambda - \Lambda H) + U\Lambda H -XU^*\\
& = U(H\Lambda - \Lambda H) + XU H -XU^*\\
&= U(H\Lambda - \Lambda H) + X(U H - U^*).
\end{align*}
Multiplying both sides by $\Lambda^{-1}$ and rearranging the terms, we have
\begin{align}\label{eqn:decomposition1}
UH - U^* = U(H\Lambda - \Lambda H) \Lambda^{-1} + X(U H - U^*) \Lambda^{-1} + XU^*\Lambda^{-1} - U^* .
\end{align}
The above display involves $ X(U H - U^*) $ where $X$ and $UH-U^*$ are dependent on each other. To decouple the dependence, we approximate $UH-U^*$ by its leave-one-out counterparts.
Consider any $j\in[n]$. Define $X^{(j)}\in\mathr^{nd\times nd}$ such that
\begin{align*}
X^{(j)}_{ik} \define\quad \begin{cases}
X_{ik}, &\forall i,k\neq j,\\
0_{d\times d}, &\text{otherwise.}
\end{cases}
\end{align*}
In addition, let $U^{(j)}\in\mathr^{nd\times d}$ be the matrix including the leading $d$ eigenvectors of $X^{(j)}$. As a consequence, $X^{(j)}, U^{(j)}$ are independent of $\{A_{jk}\}_{k\neq j}$ and $\{W_{jk}\}_{k\neq j}$.
Define
$
H^{(j)}\define U^{(j)\top}U^*\in\mathr^{d\times d}.
$
Then
\begin{align*}
UH-U^* &= UH - U^{(j)}H^{(j)} + U^{(j)}H^{(j)} - U^* = (UU^\top - U^{(j)}U^{(j)\top})U^*+ U^{(j)}H^{(j)} - U^* .
\end{align*}
After plugging it into the right-hand side of (\ref{eqn:decomposition1}), we have
\begin{align*}
UH - U^* & = U(H\Lambda - \Lambda H) \Lambda^{-1} + X(UU^\top - U^{(j)}U^{(j)\top})U^* \Lambda^{-1} \\
&\quad + X(U^{(j)}H^{(j)} - U^*) \Lambda^{-1} + XU^*\Lambda^{-1} - U^*.
\end{align*}
Then, the $j$th block matrix of $UH - U^*$ satisfies
\begin{align}
U_jH - U^*_j & = \underbrace{U_j(H\Lambda - \Lambda H) \Lambda^{-1} + X_j(UU^\top - U^{(j)}U^{(j)\top})U^* \Lambda^{-1}}_{\definei B_j} \nonumber\\
&\quad + X_j(U^{(j)}H^{(j)} - U^*) \Lambda^{-1} + X_jU^*\Lambda^{-1} - U^*_j.\label{eqn:decomposition2}
\end{align}
The last two terms in (\ref{eqn:decomposition2}) can be further decomposed.
Using (\ref{eqn:1}), we have
\begin{align*}
X_j(U^{(j)}H^{(j)} - U^*) \Lambda^{-1}&= \br{(A_j \otimes I_d) + \sigma (A_j \otimes J_d) \circ W_j}(U^{(j)}H^{(j)} - U^*) \Lambda^{-1}\\
&= \underbrace{\sum_{k\neq j} A_{jk} (U^{(j)}_kH^{(j)} - U^*_k)\Lambda^{-1}}_{\definei F_{j1}} + \underbrace{\sigma \sum_{k\neq j} A_{jk} W_{jk}(U^{(j)}_kH^{(j)} - U^*_k)\Lambda^{-1}}_{\definei F_{j2}} ,
\end{align*}
and
\begin{align*}
X_jU^*\Lambda^{-1} - U^*_j &= \br{(A_j \otimes I_d) + \sigma (A_j \otimes J_d) \circ W_j}U^*\Lambda^{-1} - U^*_j \\
&= \left(\sum_{k\neq j} A_{jk} U^*_k\right)\Lambda^{-1} - U_j^*+ \sigma \,\sum_{k\neq j} A_{jk} W_{jk} U^*_k\Lambda^{-1} \\
&= \underbrace{\frac{1}{\sqrt{n}}\br{\left(\sum_{k\neq j} A_{jk} \right)\Lambda^{-1} - I_d}}_{\definei G_{j1}} + \underbrace{ \frac{\sigma}{\sqrt{n}} \,\sum_{k\neq j} A_{jk} W_{jk} \Lambda^{-1} }_{\definei G_{j2}} ,
\end{align*}
where the last equation is due to Lemma \ref{lem:population}.
As a result, we have a decomposition of $U_jH - U^*_j$
\begin{align}\label{eqn:decomposition3}
U_jH - U^*_j & =
B_j + F_{j1} + F_{j2} + G_{j1} + G_{j2},
\end{align}
holds for all $j\in[n]$.
The block-wise decomposition (\ref{eqn:decomposition3}) of $UH-U^*$ is the starting point to establishing both Theorem \ref{thm:l_infty_new} and Theorem \ref{thm:main_new}.
Note that we have a mutual independence among $\{A_{jk}\}_{k\neq i}$, $\{W_{jk}\}_{k\neq i}$, and $U^{(j)}H^{(j)} -U^*$ in the definitions of $F_{j1}$ and $F_{j2}$, which is crucial to obtaining sharp bounds and tail probabilities for them. Theorem \ref{thm:l_infty_new} is proved by establishing upper bounds for the operator norm of $B_j, F_{j1}, F_{j2}, G_{j1} $, and $G_{j2}$. By further analyzing their tail bounds, we establish Theorem \ref{thm:main_new}. Among these terms, $G_{j2}$ is the one contributing to the exponential error bound in Theorem \ref{thm:main_new}, as we illustrate in (\ref{eqn:approx2}).
%For any other permutation matrix $R\in \Pi_d$ that is different from $$
%. We have $UM^\top - U^*\hat P^\top \approx (UH - U^*)\hat P^\top \approx \left( XU^*\Lambda^{-1} - U^* \right)\hat P^\top $. For each block $j\in [n]$,
%\begin{align*}
% U_jM^\top - U_j^*\hat P^\top &\approx (U_jH - U_j^*)\hat P^\top \approx (X_j U^*\Lambda^{-1} - U_j^*)\hat P^\top \\
% &= \left(\sum_{k\neq j} A_{jk}U_k^*\Lambda^{-1} - U_j^*\right)\hat P^\top + \left(\sum_{k\neq j}\sigma A_{jk} W_{jk} U_k^*\Lambda^{-1} \right) \hat P^\top \numberthis\label{eqn:lin-approx} \,.
%\end{align*}
%We see that both terms in (\ref{eqn:lin-approx}) contribute to the errors of the approximation $U_jM^\top \approx U_j^*\hat P^\top$. However, we will show that the deviation associated with the first term is small while the second term is dominant and behaves like a sum of Gaussian random variables. Due to the maximization problem (\ref{eqn:alg2}) in Algorithm \ref{alg:main-algo}, the performance of Algorithm \ref{alg:main-algo} crucially depends on the tail property of the second term. Bounding this second term essentially gives us the desired rate. Roughly speaking, we will show the following chain of argument.
%\begin{align*}
%&\p\left( Z_j^*\hat P^\top \neq \argmax_{P\in \Pi_d} \inner{P}{U_jM^\top} \right) = \p\left( \exists R \in \Pi_d \backslash \{ Z_j^* \} \,:\, \inner{U_jM^\top}{Z_j^*\hat P^\top - R\hat P^\top} < 0 \right)\\
%&= \p\left( \exists R \in \Pi_d \backslash \{ Z_j^* \} \,:\, \inner{U_jM^\top - \frac{Z_j^*\hat P^\top}{\sqrt n}}{Z_j^*\hat P^\top - R\hat P^\top} < -\frac{\fnorm{Z_j^* - R}^2}{2\sqrt n} \right) \\
%&\approx \p\left(\exists R \in \Pi_d \backslash \{ Z_j^* \} \,:\, \inner{\sum_{k\neq j}\sigma A_{jk} W_{jk} U_k^*\Lambda^{-1} }{Z_j^* - R} <- \frac{\fnorm{Z_j^* - R}^2}{2\sqrt n} \right) \numberthis\label{eqn:rough-lin} \\
%&\approx \p\left( \exists R \in \Pi_d \backslash \{ Z_j^* \} \,:\,\frac{\fnorm{Z_j^* - R}^2}{2\sqrt n} \cdot \left(\sum_{k\neq j} \frac{\sigma}{np} A_{jk} w_{jk} \right) <- \frac{\fnorm{Z_j^* - R}^2}{2\sqrt n} \right) \numberthis\label{eqn:rough-1} \\
%&\approx \p\left( \calN\left(0, \frac{\sum_{k\neq j} A_{jk}}{np} \, \frac{\sigma^2}{np}\right) < -1 \right) = \ebr{-(1-o(1))\frac{np}{2\sigma^2}} \numberthis\label{eqn:rough-2}\,,
%\end{align*}
%where $w_{jk} \iid \calN(0, 1)$ captures the Gaussian like behaviour of the second term of (\ref{eqn:lin-approx}). To derive (\ref{eqn:rough-lin}), we use the approximation (\ref{eqn:lin-approx}). To derive (\ref{eqn:rough-1}), we prove $\opnorm{\Lambda^{-1}} \approx \frac{1}{np}$ and $\opnorm{U_j^*} = \frac{1}{\sqrt n}$. To derive (\ref{eqn:rough-2}), we will prove $\sum_{k\neq j} A_{jk} \approx np$ and apply union bounding over all $R \in \Pi_d \backslash \{Z_j^*\}$.
\begin{thebibliography}{}
\bibitem[Abbe et~al., 2020]{abbe2020entrywise}
Abbe, E., Fan, J., Wang, K., and Zhong, Y. (2020).
\newblock Entrywise eigenvector analysis of random matrices with low expected
rank.
\newblock {\em Annals of statistics}, 48(3):1452.
\bibitem[Agarwal et~al., 2011]{agarwal2011building}
Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.~M.,
and Szeliski, R. (2011).
\newblock Building rome in a day.
\newblock {\em Communications of the ACM}, 54(10):105--112.
\bibitem[Bean et~al., 2013]{bean2013optimal}
Bean, D., Bickel, P.~J., El~Karoui, N., and Yu, B. (2013).
\newblock Optimal m-estimation in high-dimensional regression.
\newblock {\em Proceedings of the National Academy of Sciences},
110(36):14563--14568.
\bibitem[Berg et~al., 2005]{berg2005shape}
Berg, A.~C., Berg, T.~L., and Malik, J. (2005).
\newblock Shape matching and object recognition using low distortion
correspondences.
\newblock In {\em 2005 IEEE computer society conference on computer vision and
pattern recognition (CVPR'05)}, volume~1, pages 26--33. IEEE.
\bibitem[Chen and Cand{\`e}s, 2018]{chen2018projected}
Chen, Y. and Cand{\`e}s, E.~J. (2018).
\newblock The projected power method: An efficient algorithm for joint
alignment from pairwise differences.
\newblock {\em Communications on Pure and Applied Mathematics},
71(8):1648--1714.
\bibitem[Chen et~al., 2014]{chen2014near}
Chen, Y., Guibas, L., and Huang, Q. (2014).
\newblock Near-optimal joint object matching via convex relaxation.
\newblock In {\em International Conference on Machine Learning}, pages
100--108. PMLR.
\bibitem[De~Waal, 1985]{de2004matrix}
De~Waal, D. (1985).
\newblock Matrix-valued distributions.
\newblock {\em Encyclopedia of statistical sciences}.
\bibitem[Gao and Zhang, 2022]{gao2022iterative}
Gao, C. and Zhang, A.~Y. (2022).
\newblock Iterative algorithm for discrete structure recovery.
\newblock {\em The Annals of Statistics}, 50(2):1066--1094.
\bibitem[Gao et~al., 2021]{gao2021isometric}
Gao, M., Lahner, Z., Thunberg, J., Cremers, D., and Bernard, F. (2021).
\newblock Isometric multi-shape matching.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 14183--14193.
\bibitem[Gross, 2011]{gross2011recovering}
Gross, D. (2011).
\newblock Recovering low-rank matrices from few coefficients in any basis.
\newblock {\em IEEE Transactions on Information Theory}, 57(3):1548--1566.
\bibitem[Huang and Guibas, 2013]{huang2013consistent}
Huang, Q.-X. and Guibas, L. (2013).
\newblock Consistent shape maps via semidefinite programming.
\newblock In {\em Computer graphics forum}, volume~32, pages 177--186. Wiley
Online Library.
\bibitem[Kuhn, 1955]{kuhn1955hungarian}
Kuhn, H.~W. (1955).
\newblock The hungarian method for the assignment problem.
\newblock {\em Naval research logistics quarterly}, 2(1-2):83--97.
\bibitem[Lei and Rinaldo, 2015]{lei2015consistency}
Lei, J. and Rinaldo, A. (2015).
\newblock Consistency of spectral clustering in stochastic block models.
\newblock {\em The Annals of Statistics}, 43(1):215--237.
\bibitem[Ling, 2022]{ling2022near}
Ling, S. (2022).
\newblock Near-optimal performance bounds for orthogonal and permutation group
synchronization via spectral methods.
\newblock {\em Applied and Computational Harmonic Analysis}, 60:20--52.
\bibitem[Maset et~al., 2017]{maset2017practical}
Maset, E., Arrigoni, F., and Fusiello, A. (2017).
\newblock Practical and efficient multi-view matching.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 4568--4576.
\bibitem[Pachauri et~al., 2013]{pachauri2013solving}
Pachauri, D., Kondor, R., and Singh, V. (2013).
\newblock Solving the multi-way matching problem by permutation
synchronization.
\newblock {\em Advances in neural information processing systems}, 26.
\bibitem[Shen and Davatzikos, 2002]{shen2002hammer}
Shen, D. and Davatzikos, C. (2002).
\newblock Hammer: hierarchical attribute matching mechanism for elastic
registration.
\newblock {\em IEEE transactions on medical imaging}, 21(11):1421--1439.
\bibitem[Shen et~al., 2016]{shen2016normalized}
Shen, Y., Huang, Q., Srebro, N., and Sanghavi, S. (2016).
\newblock Normalized spectral map synchronization.
\newblock {\em Advances in neural information processing systems}, 29.
\bibitem[Tropp, 2012]{tropp2012user}
Tropp, J.~A. (2012).
\newblock User-friendly tail bounds for sums of random matrices.
\newblock {\em Foundations of computational mathematics}, 12(4):389--434.
\bibitem[Vershynin, 2018]{vershynin2018high}
Vershynin, R. (2018).
\newblock {\em High-dimensional probability: An introduction with applications
in data science}, volume~47.
\newblock Cambridge university press.
\bibitem[Zhang, 2022]{zhang2022exact}
Zhang, A.~Y. (2022).
\newblock Exact minimax optimality of spectral methods in phase synchronization
and orthogonal group synchronization.
\newblock {\em arXiv preprint arXiv:2209.04962}.
\bibitem[Zhang and Zhou, 2022]{zhang2022leave}
Zhang, A.~Y. and Zhou, H.~H. (2022).
\newblock Leave-one-out singular subspace perturbation analysis for spectral
clustering.
\newblock {\em arXiv preprint arXiv:2205.14855}.
\bibitem[Zhou et~al., 2015]{zhou2015multi}
Zhou, X., Zhu, M., and Daniilidis, K. (2015).
\newblock Multi-image matching via fast alternating minimization.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision}, pages 4032--4040.
\end{thebibliography}