Subspace Perspective on Canonical Correlation Analysis: Dimension Reduction and Minimax Rates
Extracted source text ยท arXiv 1605.03662
\documentclass[11pt]{article}
\usepackage{graphicx, subfigure}
\usepackage[top=1.25in,bottom=1.25in,left=1.25in,right=1.25in]{geometry}
% \usepackage[sort&compress]{natbib} \setlength{\bibsep}{0.7pt}
\usepackage{amsmath,amsfonts,amscd,amssymb,bm,epsfig,epsf,bbm,constants,amssymb, prettyref, mathabx}
\usepackage{algorithm,algorithmic}
\usepackage[usenames]{color}
\usepackage[authoryear, square]{natbib}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% colors
\definecolor{darkred}{RGB}{100,0,0}
\definecolor{darkgreen}{RGB}{0,100,0}
\definecolor{darkblue}{RGB}{0,0,150}
\definecolor{red}{RGB}{255,0,0}
\definecolor{xl}{RGB}{200,50,120}
\newcommand{\XL}[1]{\textcolor{xl}{XL: #1}}
% links
\usepackage{hyperref}
\hypersetup{colorlinks=true, linkcolor=darkred, citecolor=darkgreen, urlcolor=darkblue}
\usepackage{url}
%--------------
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{assumption}[theorem]{Assumption}
\newtheorem{remark}[theorem]{Remark}
\usepackage{booktabs} % \toprule, \midrule, \bottomrule
\usepackage{multirow}
\usepackage{longtable}
\newcommand{\vct}[1]{\bm{#1}}
\newcommand{\mtx}[1]{\bm{#1}}
%\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}
\newcommand{\transp}{T}
\newcommand{\adj}{*}
\newcommand{\psinv}{\dagger}
%\newcommand{\lspan}[1]{\operatorname{span}{#1}}
\newcommand{\lspan}{\operatorname{span}}
\newcommand{\range}{\operatorname{range}}
\newcommand{\colspan}{\operatorname{colspan}}
\newcommand{\rank}{\operatorname{rank}}
\newcommand{\diag}{\operatorname{diag}}
\newcommand{\tr}{\operatorname{tr}}
\newcommand{\supp}[1]{\operatorname{supp}(#1)}
\newcommand{\lin}[1]{\operatorname{\text{Lin}}(#1)}
\newcommand{\smax}{\sigma_{\max}}
\newcommand{\smin}{\sigma_{\min}}
\newcommand{\restrict}[1]{\big\vert_{#1}}
\newcommand{\Id}{\text{\em I}}
\newcommand{\OpId}{\mathcal{I}}
\newcommand{\U}{{\bf U}}
\newcommand{\I}{{\bf I}}
\newcommand{\D}{{\bf D}}
\newcommand{\V}{{\bf V}}
\newcommand{\Uh}{{\widehat{\bf U}}}
\newcommand{\Dh}{{\widehat{\bf D}}}
\newcommand{\Vh}{{\widehat{\bf V}}}
\newcommand{\A}{{\bf A}}
\newcommand{\B}{{\bf B}}
\newcommand{\h}{{\bf H}}
\newcommand{\E}{{\bf \mathbb{E}}}
\newcommand{\Pro}{{\bf P}}
\newcommand{\F}{{\bf F}}
\newcommand{\R}{{\bf R}}
\newcommand{\K}{{\bf K}}
\newcommand{\Q}{{\bf Q}}
\newcommand{\X}{{\bf X}}
\newcommand{\Y}{{\bf Y}}
\newcommand{\W}{{\bf W}}
\newcommand{\px}{\mathbold{\Phi}}
\newcommand{\py}{\mathbold{\Psi}}
\newcommand{\pxk}{\mathbold{\Phi}_{1:k}}
\newcommand{\pyk}{\mathbold{\Psi}_{1:k}}
\newcommand{\pxh}{\widehat{\bf \Phi}}
\newcommand{\pyh}{\widehat{\bf \Psi}}
\newcommand{\pxhk}{\widehat{\bf \Phi}_{1:k}}
\newcommand{\pyhk}{\widehat{\bf \Psi}_{1:k}}
\newcommand{\s}{\mtx{\Sigma}}
\newcommand{\sx}{\mtx{\Sigma}_x}
\newcommand{\sy}{\mtx{\Sigma}_y}
\newcommand{\sxy}{\mtx{\Sigma}_{xy}}
\newcommand{\syx}{\mtx{\Sigma}_{yx}}
\newcommand{\sxx}{\mtx{\Sigma}_x^{1/2}}
\newcommand{\syy}{\mtx{\Sigma}_y^{1/2}}
\newcommand{\sxxx}{\mtx{\Sigma}_x^{-1/2}}
\newcommand{\syyy}{\mtx{\Sigma}_y^{-1/2}}
\newcommand{\sh}{\widehat{\mtx{\Sigma}}}
\newcommand{\sxh}{\widehat{\mtx{\Sigma}}_x}
\newcommand{\syh}{\widehat{\mtx{\Sigma}}_y}
\newcommand{\sxyh}{\widehat{\mtx{\Sigma}}_{xy}}
\newcommand{\syxh}{\widehat{\mtx{\Sigma}}_{yx}}
\newcommand{\one}{{\bf 1}}
%\newcommand{\sxxx}{{\bf \Sigma_x^{-\frac{1}{2}}}}
%\newcommand{\syyy}{{\bf \Sigma_y^{-\frac{1}{2}}}}
\newcommand{\svd}{{\bf \Sigma_x^{-1/2}\Sigma_{xy}\Sigma_y^{-1/2}}}
\newcommand{\lm}{{\bf \Lambda}}
\newcommand{\cor}{{ \bLambda}}
\newcommand{\wt}[1]{\widetilde{#1}}
\newcommand{\wh}[1]{\widehat{#1}}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\mb}[1]{\mathbb{#1}}
\newcommand{\hada}[1]{\|#1\|}
\newcommand{\norm}[1]{{\left\vert\kern-0.25ex\left\vert\kern-0.25ex\left\vert #1
\right\vert\kern-0.25ex\right\vert\kern-0.25ex\right\vert}}
\newcommand{\half}{1/2}
\newcommand{\os}{\bU_{\star}}
\numberwithin{equation}{section}
\parskip = 0 pt
\def \endprf{\hfill {\vrule height6pt width6pt depth0pt}\medskip}
\newenvironment{proof}{\noindent {\bf Proof} }{\endprf\par}
\newcommand{\qed}{{\unskip\nobreak\hfil\penalty50\hskip2em\vadjust{}
\nobreak\hfil$\Box$\parfillskip=0pt\finalhyphendemerits=0\par}}
% For algorithms
%\usepackage{algorithm}
%\usepackage{algorithmic}
\usepackage{authblk}
\input{04-pre-am-short.tex}
\title{Subspace Perspective on Canonical Correlation Analysis: Dimension Reduction and Minimax Rates}
\author[1]{Zhuang Ma\thanks{Z. Ma is with Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA. \href{mailto:zhuangma@wharton.upenn.edu}{zhuangma@wharton.upenn.edu}.} \qquad Xiaodong Li\thanks{X. Li is with the Statistics Department at the University of California, Davis, CA. \href{mailto:xdgli@ucdavis.edu}{xdgli@ucdavis.edu}.}}
%\author[2]{}
%\affil[1]{University of Pennsylvania}
%\affil[2]{University of California, Davis}
%\renewcommand\Authands{ and }
%\date{}
\begin{document}
\maketitle
\begin{abstract}
Canonical correlation analysis (CCA) is a fundamental statistical tool for exploring the correlation structure between two sets of random variables. In this paper, motivated by recent success of applying CCA to learn low dimensional representations of high dimensional objects, we propose to quantify the estimation loss of CCA by the excess prediction loss defined through a prediction-after-dimension-reduction framework. Such framework suggests viewing CCA estimation as estimating the subspaces spanned by the canonical variates. Interestedly, the proposed error metrics derived from the excess prediction loss turn out to be closely related to the principal angles between the subspaces spanned by the population and sample canonical variates respectively.
We characterize the non-asymptotic minimax rates under the proposed metrics, especially the dependency of the minimax rates on the key quantities including the dimensions, the condition number of the covariance matrices, the canonical correlations and the eigen-gap, with minimal assumptions on the joint covariance matrix. To the best of our knowledge, this is the first finite sample result that captures the effect of the canonical correlations on the minimax rates.
%Specifically, we argue that the discrepancy between population and sample canonical variates could be interpreted as the discrepancy of the subspaces
% In this paper, motivated by the empirical success of CCA as a tool for dimension reduction, we propose a new framework to interpret CCA as subspace estimation and corresponding error metrics to quantify the estimation error of CCA by excess prediction loss. Surprisingly, the proposed error metrics turn out to be closely related to the principal angles between the subspaces spanned by the population and sample canonical variates respectively. We fully characterize the non-asymptotic minimax estimation rates under the proposed metrics, especially the dependency of the minimax rates on the key quantities including dimensions, the condition number of the covariance matrices, the canonical correlations and the eigen-gap.
\end{abstract}
\section{Introduction}
Canonical correlation analysis (CCA), first introduced by \cite{hotelling36}, is a fundamental statistical tool to characterize the relationship between two groups of random variables and finds a wide range of applications across many different fields. For example, in genome-wide association study (GWAS), CCA is used to discover the genetic associations between the genotype data of single nucleotide polymorphisms (SNPs) and the phenotype data of gene expression leves \citep{witten2009penalized, chen2012structured}. In information retrieval, CCA is used to embed both the search space (e.g. images) and the query space (e.g. text) into a shared low dimensional latent space such that the similarity between the queries and the candidates can be quantified \citep{rasiwasia2010new, gong2014multi}. In natural language processing, CCA is applied to the word co-occurrence matrix, which learns a vector representation of the words that is able to capture the semantics \citep{dhillon11, faruqui2014improving}. Other applications, to name a few, include fMRI data analysis \citep{friman2003adaptive}, computer vision \citep{kim2007tensor} and speech recognition \citep{arora2013multi, wang2015deep}.
%The applications of canonical correlation analysis mainly falls into two categories. One of them focuses on identifying the specific subset of variables in either group that are most correlated with variables in the other group. An typical example is genome-wide association study (GWAS) where CCA is a powerful hammer to discover the genetic associations between the genotype data of single nucleotide polymorphisms (SNPs) and the phenotype data of gene expression levels \citep{witten2009penalized, chen2012structured}. The other range of applications mostly lie in the machine learning and engineering community where CCA is used for dimension reduction across many fields like, to name a few, information retrieval \citep{hardoon07}, speech recognition \citep{arora2013multi}, computer vision \citep{kim2007tensor}, and natural language processing \citep{dhillon11, dhillon12}.
%In classical statistics, people tend to treat one group of the variables in CCA as response and the other group as predictors. This point of view naturally links CCA with reduced rank regression \citep{izenman1975reduced, anderson1999asymptotic}.
The enormous empirical success motivates us to revisit the estimation problem of canonical correlation analysis. From a decision-theoretic point of view, two questions are naturally posed: What is the proper error metric to quantify the discrepancy between population CCA and its sample estimates? And under such metric, what are the quantities that characterize the fundamental statistical limits?
The justification of loss functions, in the context of CCA, has seldom appeared in the literature. Start from the first principle that the proper metric to quantify the estimation loss should depend on the specific purpose of using CCA, we find that the applications discussed at the beginning mainly fall into two categories: identify variables of interest and dimension reduction. The first category, mostly in genomic research \citep{witten2009penalized, chen2012structured}, treats one group of variables as responses and the other group as covariates. The goal is to discover the specific subset of the covariates that are most correlated with the responses. Such applications are featured by low signal-to-noise ratio and the interpretability of the results is the major concern. The other category is investigated extensively in statistical machine learning and engineering community where CCA is used to learn low dimensional latent representations of complex objects such as images \citep{rasiwasia2010new}, text \citep{dhillon11} and speeches \citep{arora2013multi}. These scenarios are usually accompanied with relatively high signal-to-noise ratio and the prediction accuracy, using the learned low dimensional embeddings as the new set of predictors, is of primary interest. In recent years, there has been a series of publications establishing fundamental theoretical guarantees for CCA to achieve sufficient dimension reduction \citep{kakade07, foster08, SridharanK08, fukumizu2009kernel, chaudhuri2009multi}, especially under a multi-view setup as will be discussed more detailedly in Section~\ref{section-multi-view}.
In this paper, we aim to address the problems raised above by treating CCA as a tool for dimension reduction.
\subsection{Canonical Correlation Analysis and Linear Invariance}
On the population level, CCA is designed to extract the most correlated directions between two sets of random variables: $\bx\in\mathbb{R}^{p_1}$ and $\by\in\mathbb{R}^{p_2}$. Specifically, CCA recursively finds the pairs of vectors $\bphi_i\in\mathbb{R}^{p_1}, \bpsi_i\in\mathbb{R}^{p_2}, 1\leq i\leq p:=\min\{p_1, p_2\}$ such that
\begin{equation}
\begin{gathered}
(\bphi_i, \bpsi_i)=\arg\max_{\bphi^\top\sx\bphi=1, \bpsi^\top\sy\bpsi=1} \bphi^\top\sxy\bpsi\\
\text{subject to }~\bphi^\top\sx \bphi_j=0, ~\bpsi^\top\sy\bpsi_j=0,~ \forall~1\leq j\leq i-1.
\end{gathered}
\label{eq-def-cca}
\end{equation}
For $1\leq i\leq p$, $(\bphi_i , \bpsi_i)$ are the canonical coefficients (or loadings), $(\bphi_i^\top\bx , \bpsi_i^\top\by)$ are the canonical variates and $\lambda_i:=\bphi_i^\top\sx\bpsi_i$ are the canonical correlations. Let $\px=[\bphi_1, \cdots, \bphi_p]$ and $\py=[\bpsi_1, \cdots, \bpsi_p]$ be the loading matrices. With these notations, the first type of applications discussed above can be understood as identifying the support of the top-$k$ canonical vectors: $\pxk$ and $\pyk$, where $\pxk\in\mathbb{R}^{p_1\times k}$ and $\pyk\in\mathbb{R}^{p_2\times k}$ consist of the first $k$ columns of $\px$ and $\py$ respectively. While for dimension reduction, which motivates the paper, is concerned with the leading $k$ canonical variates: $\pxk^\top\bx$ and $\pyk^\top\by$ ($k$ is assumed to be known \textit{a priori}, or a pre-specified budget).
What distinguishes CCA from other dimension reduction methods like principal component analysis or partial least squares is its linear invariance. As highlighted in \cite{hotelling36} when canonical correlation analysis was first developed:
\medskip
``The relations between two sets of variates with which we shall be concerned are those that remain invariant under internal linear transformations of each sets separately."
\medskip
Among all the population parameters, \cite{hotelling36} noticed that the canonical correlations $\lambda_1, \cdots, \lambda_p$ and the functions of these quantities are the only linear invariants of the system. On the contrary, the loadings $\px$ and $\py$ will change accordingly either with rotation of axes or scaling of the variables, which diminishes the rationale for using an error metric built directly upon the loadings. If extending Hotelling's notion of invariants to include random vectors, the canonical variates are actually invariant under linear transformations of each sets separately. To illustrate, let $\bT_1, \bT_2$ be any two nonsingular matrices and define the new random vectors $\ba=\bT_1^\top\bx,\bb=\bT_2^\top\by$. As will be shown in Section~\ref{section-reduction}, $\bT_1^{-1}\pxk, \bT_2^{-1}\pyk$ are the top-$k$ canonical coefficients of $(\ba, \bb)$. Therefore, the top-$k$ canonical variates of $(\ba, \bb)$ will be $(\bT_1^{-1}\pxk)^\top \ba=\pxk^\top\bx$ and $(\bT_2^{-1}\pyk)^\top \bb=\pyk^\top\by$, which are the same as those of $(\bx, \by)$. This fact substantiates our interest in the canonical variates instead of the loadings. Let $(\pxhk, \pyhk)$ be any generic estimator of the loadings. Then the two questions that we aim to answer can be recast as:
\medskip
What is the proper error metric to quantify the discrepancy between $(\pxk^\top\bx, \pyk^\top\by)$ and its sample estimates $(\pxhk^\top\bx, \pyhk^\top\by)$? And under such metric, what are the quantities that characterize the fundamental statistical limits?
\medskip
For the rest of the paper, we will only focus on the relationship between $\pxhk^\top\bx$ and $\pxk^\top\bx$ since similar results can be obtained for the other pair by symmetry.
%To sum up, with linear invariance as the golden standard, canonical variates are the proper (population level) targets to aim for and sample CCA variates are natural legitimate estimators. In this way, the two problems can be recast as:
%\subsection{Sample Canonical Correlation Analysis}
%The sample canonical coefficients, denoted by $\pxh$ and $\pyh$, can be obtained in the same manner as \eqref{eq-def-cca} by replacing the population covariance matrices with corresponding sample estimates. This duality between population and sample CCA suggests parallel invariant properties can also be obtained for sample CCA. Suppose $\bX\in\mathbb{R}^{n\times p_1}, \bY\in\mathbb{R}^{n\times p_2}$ are the original sample matrices. Under the transformation $\ba=\bT_1^\top\bx,\bb=\bT_2^\top\by$, the data matrices will be transformed as well, that is $\bA=\bX\bT_1, \bB=\bY\bT_2$. By the same argument, $\bT_1^{-1}\pxhk, \bT_2^{-1}\pyhk$ will be the top-$k$ sample canonical coefficients of $(\bA, \bB)$ and it implies that the top-$k$ sample canonical variates are $(\bT_1^{-1}\pxhk)^\top \ba=\pxhk^\top\bx$ and $(\bT_2^{-1}\pyhk)^\top \bb=\pyhk^\top\by$, same as those of $(\bx, \by)$.
%This can be regarded as certain justification for using sample CCA as the estimator.
%Then the two questions we aim to answer can be reformulated as:
%\medskip
%
%What is the proper error metric to quantify the discrepancy between $(\pxk^\top\bx, \pyk^\top\by)$ and its sample estimates $(\pxhk^\top\bx, \pyhk^\top\by)$? And under such metric, what are the quantities that characterize the fundamental statistical limits?
%\subsection{Linear Invariants of CCA and Sample CCA}
%What distinguishes CCA from other dimension reduction methods like Principal component analysis and Partial least squares is its linear invariance. As highlighted in \cite{hotelling36} when canonical correlation analysis was first developed:
%\medskip
%
%``The relations between two sets of variates with which we shall be concerned are those that remain invariant under internal linear transformations of each sets separately."
%\medskip
%
%Among all the population parameters, \cite{hotelling36} argued that the canonical correlations $\lambda_1, \cdots, \lambda_p$ and the functions of these quantities are the only linear invariants of the system. However, the loadings will change with rotation of axes and scaling of the variables, which diminishes the rationale of error metrics built directly upon the loadings.
%%It is obvious that the loadings are not invariant, which is one of the reasons that we do not advocate error metrics built directly upon the loadings.
% If extending Hotelling's notion of invariants to include random vectors, then the canonical variates $(\pxk^\top\bx, \pyk^\top\by)$ are also invariant under linear transformations of each sets separately. To illustrate, let $\bT_1\in\mathbb{R}^{p_1\times p_1}, \bT_2\in\mathbb{R}^{p_2\times p_2}$ be any two nonsingular matrices and define the new random vectors $\ba=\bT_1^\top\bx,\bb=\bT_2^\top\by$. As we will show in Section~\ref{section-reduction}, $\bT_1^{-1}\pxk, \bT_2^{-1}\pyk$ are top-$k$ CCA coefficients of $(\ba, \bb)$. Therefore, the top-$k$ CCA variates of $(\ba, \bb)$ are $(\bT_1^{-1}\pxk)^\top \ba=\pxk^\top\bx$ and $(\bT_2^{-1}\pyk)^\top \bb=\pyk^\top\by$, which are the same as those of $(\bx, \by)$. This fact justifies our interest in the canonical variates instead of the loadings.
%especially when CCA is used as dimension reduction.
%\subsection{Linear Invariants of Sample CCA}
%The duality between population and sample CCA suggests parallel results can be obtained for sample CCA as well. Suppose $\bX\in\mathbb{R}^{n\times p_1}, \bY\in\mathbb{R}^{n\times p_2}$ are the sample matrices. Under the transformation $\ba=\bT_1^\top\bx,\bb=\bT_2^\top\by$, the data matrices will be transformed accordingly, that is $\bA=\bX\bT_1, \bB=\bY\bT_2$. By the same argument, $\bT_1^{-1}\pxhk, \bT_2^{-1}\pyhk$ will be the top-$k$ sample CCA coefficients of $(\bA, \bB)$ and it implies that the top-$k$ sample CCA variates are $(\bT_1^{-1}\pxhk)^\top \ba=\pxhk^\top\bx$ and $(\bT_2^{-1}\pyhk)^\top \bb=\pyhk^\top\by$. This can be regarded as certain justification for using sample CCA as the estimator.
\subsection{Subspace Estimation and Subspace Loss}
%Suppose $\mathcal{L}(\cdot, \cdot)$ is the proper metric to quantify the discrepancy between $\pxk^\top\bx$ and $\pxhk^\top\bx$. Then the error metric should be
%
%On the one hand, the metric $\mathcal{L}(\cdot, \cdot)$ should take care of the identifiability issues. The canonical coefficient $\pxk$ is only identifiable up to orthogonal transformations without assuming eigen-gaps between each pair of adjacent canonical correlations. Since we do not want to make such strong assumptions, the error metric should not distinguish the top $k$ canonical variates: $\bphi_1^\top\bx, \cdots, \bphi_k^\top\bx$. Mathematically, the loss $\mathcal{L}(\cdot, \cdot)$ should treat $\{(\pxk\bQ)^\top\bx: \bQ\in\mathcal{O}(k)\}$ as equivalent.
%
%One the other hand, it is crucial to notice that the constraint that $\text{Var}(\bphi_i^\top\bx)=1$ in the definition of CCA is only for the sake of algebraic identifiability. Scaling $\bphi_i$ arbitrarily will not change the correlation between $\bphi_i^\top\bx$ and $\bphi_j^\top\bx, \bpsi_j^\top\by$. Therefore, an ideal error metric, should also treat $\{(\pxhk\bD)^\top\bx: \bD\in\mathbb{R}^{k\times k} \text{ is diagonal matrix}\}$ as equivalent.
%We propose to quantify the discrepancy between $\pxk^\top\bx$ and $\pxhk^\top\bx$ as the distance between the linear subspaces spanned by these two random vectors which we denote as $\lspan(\bx^\top\pxk)$ and $\lspan(\bx^\top\pxhk)$. This perspective is natural when CCA is used for dimension reduction, where the canonical variates $\pxhk^\top\bx$ will be further treated as predictors for supervised learning tasks. From this point of view, $\lspan(\bx^\top\pxk)$ and $\lspan(\bx^\top\pxhk)$ exactly characterize the
%In Section~\ref{section-prediction}, we propose two error metrics for quantifying the discrepancy between $\pxk^\top\bx$ and $\pxhk^\top\bx$ through a prediction-after-dimension-reduction framework. In this section, we illustrate
%One quick solution to measure the discrepancy between $\pxhk^\top\bx$ and $\pxk^\top\bx$ is to lump together their coordinate-wise difference. However, $\pxk$ is only identifiable up to orthogonal transformations without assuming eigen-gaps between each pair of adjacent canonical correlations.
%It was proposed in \cite{gao2015minimax} to consider
%\[
%\inf_{\bQ\in\mathcal{O}(k)}\E \left\|\pxk^\top\bx - \bQ\pxhk^\top\bx\right\|^2
%\]
In Section~\ref{section-prediction}, we show that when CCA is used for dimension reduction, it is the difference between the predictive power of $\pxk^\top\bx$ and $\pxhk^\top\bx$ that matters, instead of the distance by simply treating $\pxk^\top\bx$ and $\pxhk^\top\bx$ as vectors. We propose to characterize the discrepancy between the predictive power by the excess prediction loss induced by replacing the population canonical variates $\pxk^\top\bx$ with the sample estimates $\pxhk^\top\bx$ in terms of predicting a generic response. Specifically, when linear prediction is concerned, such discrepancy is reduced to the difference between the subspaces spanned by the population and sample canonical variates, denoted by $\lspan(\bx^\top\pxhk)$ and $\lspan(\bx^\top\pxk)$. This suggests that CCA estimation can be viewed as subspace estimation, that is, estimating the subspace spanned by the leading-$k$ canonical variates: $\lspan(\bx^\top\pxk)$. From this perspective, the error metric $\mc{L}(\cdot, \cdot)$ we pursue should be rewritten as
\begin{equation}
\mathcal{L}(\pxk^\top\bx, \pxhk^\top\bx):=\mathcal{L}(\lspan(\bx^\top\pxk), \lspan(\bx^\top\pxhk)),
\end{equation}
Interestedly, the error metrics derived through the excess prediction loss turn out to be closely related to the principal angles (defined in Section~\ref{section-angle}) between $\lspan(\pxhk^\top\bx)$ and $\lspan(\pxk^\top\bx)$. Suppose $\btheta=(\theta_1, \cdots, \theta_k)$ is the vector of such principal angles. It is proved in Theorem~\ref{thm-metric} that,
\begin{equation}
\begin{gathered}
\text{Worst case excess prediction loss}\simeq\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|^2=\left\|\sin(\btheta)\right\|_{\infty}^2\\
\begin{aligned}
\text{Bayesian excess prediction loss}\simeq\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|_F^2/2k=\left\|\sin(\btheta)\right\|_{2}^2/k
\end{aligned}
\end{gathered}
\label{eq-loss-intro}
\end{equation}
where $\simeq$ means `equal up to constant' and $\bP_{(\cdot)}$ denotes the projection matrix.
%One of the key messages we want to deliver in this paper is: when CCA is used for dimension reduction, the discrepancy between the population and sample canonical variates $\pxk^\top\bx$ and $\pxhk^\top\bx$ could be naturally interpreted as the discrepancy between the corresponding linear subspaces, that is, $\lspan(\bx^\top\pxhk)$ and $\lspan(\bx^\top\pxk)$. In other words, CCA estimation could be understood as subspace estimation and $\lspan(\bx^\top\pxk)$ might be a more sensible target than $\pxk^\top\bx$ when dimension reduction is concerned. In terms of loss functions, we try to advocate
%\begin{equation}
%\mathcal{L}(\pxk^\top\bx, \pxhk^\top\bx):=\mathcal{L}(\lspan(\bx^\top\pxk), \lspan(\bx^\top\pxhk)),
%\end{equation}
%which implies that the error metric $\mathcal{L}(\cdot, \cdot)$ we pursue is linear invariant. This subspace point of view is substantiated in Section~\ref{section-prediction} through a prediction-after-dimension-reduction framework. Specifically, we propose two error metrics to quantify the excess prediction loss induced by replacing the population canonical variates $\pxk^\top\bx$ with the sample estimates $\pxhk^\top\bx$, for predicting a generic response. Surprisingly, these error metrics turn out to be closely related to the principal (sometimes also called canonical) angles (defined in Section~\ref{section-angle}) between the subspaces: $\lspan(\pxhk^\top\bx)$ and $\lspan(\pxk^\top\bx)$. Define $\btheta=(\theta_1, \cdots, \theta_k)$ as the vector of such principal angles. It is proved in Theorem~\ref{thm-metric} that,
%\begin{equation}
%\begin{gathered}
%\text{Worst case excess prediction loss}\simeq\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|^2=\left\|\sin(\btheta)\right\|_{\infty}^2\\
%\begin{aligned}
%\text{Bayesian excess prediction loss}\simeq\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|_F^2/2k=\left\|\sin(\btheta)\right\|_{2}^2/k
%\end{aligned}
%\end{gathered}
%\label{eq-loss-intro}
%\end{equation}
%These two error metrics are linear invariant, which characterize both the excess prediction loss and the subspace distance in terms of principal angles, while the statistical meaning of most loss functions in literature are not very clear.
%where $\pxhk$ is treated as fixed and the expectation is taken with respect to an independent sample $\bx$. The notation $\mathcal{O}(k)$ stands for the set of all $k\times k$ orthogonal matrices and the factor $\bQ$ exists because $\pxk$ is only identifiable up to orthogonal transformations without assuming eigen-gaps between each pair of adjacent canonical correlations. Taking infimum over all orthogonal matrices essentially treats bb as equivalent.
%We further notice that the constraint that $\text{Var}(\bphi_i^\top\bx)=\text{Var}(\bpsi_i^\top\by)=1$ in the definition of CCA is only for the sake of identifiability but not statistically meaningful. Scaling the canonical covariates $\bphi_1^\top\bx, \cdots, \bphi_k^\top\bx, \bpsi_1^\top\by, \cdots, \bpsi_k^\top\by$ separately will not change their pairwise correlations. Therefore, the metric we pursue, should also treat $\{\bx^\top(\pxhk\bD): \bD\in\mathbb{R}^{k\times k} \text{ is diagonal matrix}\}$ as equivalent.
%
%To summarize, the proper error metric, denoted by $\mathcal{L}(\cdot, \cdot)$, should be simultaneously rotation invariant and dilation invariant for both of its arguments, or equivalently, linear invariant. Interestedly, what characterizes such linear invariant relationship between two random vectors is actually CCA itself.
%
%Throughout the paper, without loss of generality, we assume the random vectors $\bx$ and $\by$ are centered unless otherwise specified. Define the Hilbert space $\mathcal{H}$ by $\mathcal{H}=\lspan(\bx)=\{X|X=\vct{x}^\top\vct{w}, \vct{w} \in \mathbb{R}^{p}\}$ with covariance operator as the inner product. Then $\lspan(\pxhk^\top\bx)$ and $\lspan(\pxk^\top\bx)$ are two linear subspaces of $\mathcal{H}$. The linear invariance of the error metric implies that $\mathcal{L}(\pxk^\top\bx, \pxhk^\top\bx)$ can be rewritten as $\mathcal{L}(\lspan(\pxk^\top\bx), \lspan(\pxhk^\top\bx))$. Consider the canonical correlation analysis between these two subspaces and assume $\theta_1\geq \cdots\geq \theta_k$ are the canonical angles. In Section~\ref{section-angle}, we propose the following two types of loss functions:
%\begin{equation}
%\begin{gathered}
%\mathcal{L}_1(\lspan(\pxk^\top\bx), \lspan(\pxhk^\top\bx)) :=\sin^2(\theta_1)=\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|^2,\\
%\begin{aligned}
%\mathcal{L}_2(\lspan(\pxk^\top\bx), \lspan(\pxhk^\top\bx)) :&=\sum_{i=1}^k \sin^2(\theta_i)=\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|_F^2
%%\\
%%&=\inf_{\bQ\in\mathbb{R}^{k\times k}}\E \left\|\pxk^\top\bx - \bQ\pxhk^\top\bx\right\|_F^2
%\end{aligned}
%\end{gathered}
%\label{eq-loss-intro}
%\end{equation}
%The first metric $\mathcal{L}_1$ is the sine square of the largest canonical angle which measures the discrepancy of the two subspaces along the most discriminative direction while the other metric $\mathcal{L}_2$ measures the average distance between the two subspaces. As laid out in Section~\ref{section-angle}, these geometric quantities can be expressed algebraically as the difference between the projection matrices of the covariance-adjusted loadings under operator norm and Frobenius norm respectively.
%
%In Section~\ref{section-prediction}, this subspace point of view will be reiterated under a linear prediction framework and the same loss functions will be derived again from a different perspective.
%This subspace point of view is natural when CCA is used for dimension reduction as will be shown in the Section~\ref{section-prediction}, where the same loss functions are derived from a different perspective.
\subsection{Minimax Rates}
In section~\ref{section-theory}, we characterize the non-asymptotic minimax estimation rates for CCA under the proposed error metrics in \eqref{eq-loss-intro}, especially the dependency of the minimax rates on the key quantities, including the dimensions, the condition number of the covariance matrices, the canonical correlations and the eigen-gap. Informally, with operator norm error as an example, in Theorem~\ref{thm:upper} and Theorem~\ref{thm:lower}, we show that under certain sample size condition ($n\geq C_{\lambda_k, \lambda_{k+1}}(p_1+p_2)$), the minimax rate is characterized by
%for all joint covariance matrices with $\sx, \sy$ nonsingular,
\[
\inf\limits_{\pxhk}\sup_{\s\in \mathcal{F}} \E\left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|^2\right] \asymp \frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{(\lambda_k-\lambda_{k+1})^2}\frac{p_1}{n}.
\]
To the best of our knowledge, it is the first finite sample result to capture the factor $(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)$. This term is not negligible because $\lambda_k, \lambda_{k+1}$ are parameters depending on the dimensions and should not be treated as constants. In practice, as the number of variables increases, one should expect the canonical correlations to increase as well. The other important feature is the independence of the dimension $p_2$. If one is only interested in the 'estimation' of the canonical variates of $\bx$, then even in the regime $p_2>>p_1$, as long as the sample size is large enough, the minimax rate of `estimating' $\pxk^\top\bx$ does not depend on $p_2$. This phenomenon was also revealed in \cite{gao2015minimax} and the recent work of \cite{cai2016rate} with extra assumption that all the residual canonical correlations are zero: $\lambda_{k+1}=\cdots=\lambda_{p_1}=0$. Finally, the minimax rates are independent of the condition number of the covariance matrices: $\kappa(\sx), \kappa(\sy)$. This is due to the linear invariance of the canonical variates as illustrated in Section~\ref{section-reduction}. We hope our theoretical findings could provide some guidance for the practical use of CCA because in real applications, all these factors do matter both computationally and statistically \citep{ma2015finding}.
Specifically, the upper bound is achieved by sample CCA which is defined in the same manner as \eqref{eq-def-cca} by replacing the population covariance matrices with the corresponding sample estimates. The sample canonical variates are also linear invariant, which is crucial to reduce the estimation error of sample CCA to the ``standard form'', as spelled out in Section~\ref{section-reduction}, that the separate covariance matrices are identity and the cross covariance matrix is diagonal.
%The geometric property of the proposed error metric is crucial for obtaining the results.
%Specifically, we show that sample CCA achieves nearly optimal estimation rates in terms of all the quantities above.
% This duality between population and sample CCA suggests parallel invariant properties can also be obtained for sample CCA. Suppose $\bX\in\mathbb{R}^{n\times p_1}, \bY\in\mathbb{R}^{n\times p_2}$ are the original sample matrices. Under the transformation $\ba=\bT_1^\top\bx,\bb=\bT_2^\top\by$, the data matrices will be transformed as well, that is $\bA=\bX\bT_1, \bB=\bY\bT_2$. By the same argument, $\bT_1^{-1}\pxhk, \bT_2^{-1}\pyhk$ will be the top-$k$ sample canonical coefficients of $(\bA, \bB)$ and it implies that the top-$k$ sample canonical variates are $(\bT_1^{-1}\pxhk)^\top \ba=\pxhk^\top\bx$ and $(\bT_2^{-1}\pyhk)^\top \bb=\pyhk^\top\by$, same as those of $(\bx, \by)$.
Theoretical understanding for the estimation of CCA dates back to the study of the asymptotic distribution of sample CCA, in the low dimensional regime with fixed dimensions and sample size going to infinity, for both sample canonical coefficients and sample canonical correlations \citep{hotelling36, hsu1941limiting, izenman1975reduced, tAND84a, anderson1999asymptotic} (and many others). More recently, \cite{chen2013sparse} and \cite{gao2014sparse, gao2015minimax} have studied the non-asymptotic minimax rates of sparse CCA in a high dimensional setup. We defer the detailed comparison between these results and ours to Section~\ref{section-minimax-rates}.
%Our major contribution is to show sample CCA achieves nearly optimal minimax rates for all these quantities.
%Nonetheless, it was not well understood how condition numbers, canonical correlations and eigen-gaps affect the minimax rates. Even for dimensions, the minimax rates are not completely clear for general covariance structure.
\subsection{Notations}
Throughout the paper, we use lower-case and upper-case bold letters to represent vectors and matrices. For any matrix $\bU\in\mathbb{R}^{n\times p}$ and vector $\bu\in\mathbb{R}^p$, $\|\bU\|, \|\bU\|_F$ denotes operator norm and Frobenius norm respectively, $\|\bu\|$ denotes the vector $l_2$ norm, $\bU_{1:k}$ denotes the submatrix consisting of the first $k$ columns of $\bU$, and $\bP_{\bU}$ stands for the projection matrix onto the column space of $\bU$. Moreover, we use $\sigma_{\max}(\bU)$ and $\sigma_{\min}(\bU)$ to represent the largest and smallest singular value of $\bU$ respectively, and $\kappa(\bU)=\sigma_{\max}(\bU)/\sigma_{\min}(\bU)$ to denote the condition number of the matrix. We use $\bI_{p}$ for the identity matrix of dimension $p$ and $\bI_{p, k}$ for the submatrix composed of the first $k$ columns of $\bI_p$. Further, $\mathcal{O}(m, n)$ stands for the set of $m\times n$ matrices with orthonormal columns and $\mathbb{S}_+^p$ denotes the set of $p\times p$ strictly positive definite matrices. For a random vector $\bx\in\mathbb{R}^p$, $\lspan(\bx^\top)=\{\bx^\top\bw, \bw\in\mathbb{R}^p\}$ denotes the subspace of all the linear combinations of $\bx$. Other notations will be specified within the corresponding context.
\section{Prediction Loss for Dimension Reduction}
\label{section-prediction}
\input{multiview}
%\section{Reduction}
\section{Theory}
\label{section-theory}
\input{estimation}
\section{Upper Bound: Proof of Theorem~\ref{thm:upper}}
\label{sec::upper-proof}
\input{asymptotic}
\section{Proof of the Main Results}
\label{sec::proof}
\input{proof}
\section{Appendix}
\input{appendix}
\bibliographystyle{chicago}
\bibliography{cca}
%\bibliographystyle{plain}
\end{document}
\def\argmax{\mathop{\rm arg\, max}}
\def\argmin{\mathop{\rm arg\, min}}
\sloppy \hyphenpenalty=10000
\hfuzz=20truept
%%%% special math stuff %%%%
\newcommand{\bel}{\begin{eqnarray}\label}
\newcommand{\eel}{\end{eqnarray}}
\newcommand{\bes}{\begin{eqnarray*}}
\newcommand{\ees}{\end{eqnarray*}}
\newcommand{\bei}{\begin{itemize}}
\newcommand{\beiftnt}{\begin{itemize}\footnotesize}
\newcommand{\eei}{\end{itemize}}
\def\bess{\bes\small }
\newcommand{\convas}{\stackrel{{\rm a.s}}{\longrightarrow}}
\newcommand{\convp}{\stackrel{{\rm P}}{\longrightarrow}}
\newcommand{\convd}{\stackrel{{\rm D}}{\longrightarrow}}
\def\toD{\convd}
\def\towLik{\buildrel{\hbox{\rm w-lik}}\over\longrightarrow}
\def\toLik{\buildrel{\hbox{\rm lik}}\over\longrightarrow}
%%% steph stuff
\def\benu{\begin{enumerate}}
\def\eenu{\end{enumerate}}
\def\argmax{\mathop{\rm arg\, max}}
\def\argmin{\mathop{\rm arg\, min}}
%\def\real{\mathop{{\rm I}\kern-.2em\hbox{\rm R}}\nolimits}
\def\real{{\mathbb{R}}}
\def\R{{\real}}
\def\S{{\mathbb{S}}}
\def\E{{\mathbb{E}}}
\def\P{{\mathbb{P}}}
\def\integer{{\mathbb{Z}}}
\def\breal{\overline{\real}}
\def\complex{\mathop{{\rm I}\kern-.58em\hbox{\rm C}}\nolimits}
\def\pa{\partial}
\def\f{\frac}
\def\vsp{\vspace{5mm}}
\def\dsp{\displaystyle}
\def\diag{\hbox{\rm diag}}
\def\new{\hbox{\rm new}}
\def\old{\hbox{\rm old}}
\def\pnew{(\hbox{\rm new})}
\def\pold{(\hbox{\rm old})}
\def\rank{\hbox{\rm rank}}
\def\Rem{\hbox{\rm Rem}}
\def\sgn{\hbox{\rm sgn}}
\def\trace{\hbox{\rm trace}}
\def\Cov{\hbox{\rm Cov}}
\def\Corr{\hbox{\rm Corr}}
\def\Var{\hbox{\rm Var}}
\def\as{\hbox{\rm a.s.}}
\def\supp{\hbox{\rm supp}}
\newcommand{\Lone}{\ensuremath{{L_1}}}
\newcommand{\Lpar}{\ensuremath{{\mathbf{L}}}}
\newcommand{\Bfbeta}{{\mbox{\boldmath $\beta$}}}
\newcommand{\Bf}[1]{{\ensuremath{\mathbf{#1}}}}
\newcommand{\BfZ}{\mathbf{Z}}
\newcommand{\RK}[1]{\ensuremath{\mathbb{R}^{#1}}}
%%% letters with "\def" are sorted
\def\mathbold{\boldsymbol} %\def\mathbold{\mathbf}
%%% English, example
%\def\bc{\mathbold{c}}\def\chat{\widehat{c}}\def\ctil{\widetilde{c}}\def\cbar{{\overline c}}
%\def\hbc{{\widehat{\bc}}}\def\tbc{{\widetilde{\bc}}}
%\def\bC{\mathbold{C}}\def\hbC{{\widehat{\bC}}}\def\tbC{{\widetilde{\bC}}}
%\def\calC{{\cal C}}\def\scrC{{\mathscr C}}\def\Chat{\widehat{C}}\def\Ctil{{\widetilde C}}\def\Cbar{{\overline C}}
%\def\bfc{\mathbold{c}}\def\chat{\widehat{c}}\def\ctil{\widetilde{c}}\def\cbar{{\overline c}}
%\def\hbc{{\widehat{\bfc}}}\def\tbc{{\widetilde{\bfc}}}
%\def\bC{\mathbold{C}}\def\hbC{{\widehat{\bC}}}\def\tbC{{\widetilde{\bC}}}
%\def\calC{{\cal C}}\def\scrC{{\mathscr C}}\def\Chat{\widehat{C}}\def\Ctil{{\widetilde C}}\def\Cbar{{\overline C}}
\def\ba{\mathbold{a}}\def\bfa{\mathbold{a}}\def\ahat{\widehat{a}}\def\atil{\widetilde{a}}
\def\hba{{\widehat{\ba}}}\def\tba{{\widetilde{\ba}}}
\def\bA{\mathbold{A}}\def\calA{{\cal A}}\def\scrA{{\mathscr A}}\def\Ahat{\widehat{A}}\def\Atil{{\widetilde A}}
\def\hbA{{\widehat{\bA}}}\def\tbA{{\widetilde{\bA}}}
\def\Ahati{{{\widehat A}_i}}\def\AhatJ{{{\widehat A}_{J,n}}}\def\Atili{{{\widetilde A}_i}}\def\AtilJ{{{\widetilde A}_{J,n}}}
\def\bb{\mathbold{b}}\def\bhat{\widehat{b}}\def\btil{\widetilde{b}}
\def\hbb{{\widehat{\bb}}}\def\tbb{{\widetilde{\bb}}}
\def\bB{\mathbold{B}}\def\calB{{\cal B}}\def\scrB{{\mathscr B}}\def\Bhat{\widehat{B}}\def\Btil{{\widetilde B}}
\def\hbB{{\widehat{\bB}}}\def\tbB{{\widetilde{\bB}}}
\def\bfc{\mathbold{c}}\def\chat{\widehat{c}}\def\ctil{\widetilde{c}}
\def\hbc{{\widehat{\bfc}}}\def\tbc{{\widetilde{\bfc}}}
\def\bfC{\mathbold{C}}\def\calC{{\cal C}}\def\scrC{{\mathscr C}}\def\Chat{\widehat{C}}\def\Ctil{{\widetilde C}}
\def\hbC{{\widehat{\bfC}}}\def\tbC{{\widetilde{\bfC}}}
\def\bc{\mathbold{c}}\def\chat{\widehat{c}}\def\ctil{\widetilde{c}}
\def\hbc{{\widehat{\bc}}}\def\tbc{{\widetilde{\bc}}}
\def\bC{\mathbold{C}}\def\calC{{\cal C}}\def\scrC{{\mathscr C}}\def\Chat{\widehat{C}}\def\Ctil{{\widetilde C}}
\def\hbC{{\widehat{\bC}}}\def\tbC{{\widetilde{\bC}}}
\def\bfd{\mathbold{d}}\def\dhat{\widehat{d}}\def\dtil{\widetilde{d}}
\def\hbd{{\widehat{\bd}}}\def\tbd{{\widetilde{\bd}}}
\def\bD{\mathbold{D}}\def\calD{{\cal D}}\def\scrD{{\mathscr D}}\def\Dhat{\widehat{D}}\def\Dtil{{\widetilde D}}
\def\hbD{{\widehat{\bD}}}\def\tbD{{\widetilde{\bD}}}
\def\bfe{\mathbold{e}}\def\ehat{\widehat{e}}\def\etil{\widetilde{e}}
\def\hbe{{\widehat{\bfe}}}\def\tbe{{\widetilde{\bfe}}}
\def\bE{\mathbold{E}}\def\calE{{\cal E}}\def\scrE{{\mathscr E}}\def\Ehat{\widehat{E}}\def\Etil{{\widetilde E}}
\def\hbE{{\widehat{\bE}}}\def\tbE{{\widetilde{\bE}}}
\def\bff{\mathbold{f}}\def\fhat{\widehat{f}}\def\hf{\widehat{f}}\def\ftil{\widetilde{f}}
\def\hbf{{\widehat{\bff}}}\def\tbf{{\widetilde{\bff}}}
\def\bF{\mathbold{F}}\def\calF{{\cal F}}\def\scrF{{\mathscr F}}\def\Fhat{\widehat{F}}\def\Ftil{{\widetilde F}}
\def\hbF{{\widehat{\bF}}}\def\tbF{{\widetilde{\bF}}}
\def\fphat{\widehat{f'_i}}\def\fbara{\bar{f}^{(1)}}\def\fbarb{\bar{f}^{(2)}}\def\fbarc{\bar{f}^{(3)}}
\def\bg{\mathbold{g}}\def\ghat{\widehat{g}}\def\gtil{\widetilde{g}}
\def\hbg{{\widehat{\bg}}}\def\tbg{{\widetilde{\bg}}}
\def\bG{\mathbold{G}}\def\calG{{\cal G}}\def\scrG{{\mathscr G}}\def\Ghat{\widehat{G}}\def\Gtil{{\widetilde G}}
\def\hbG{{\widehat{\bG}}}\def\tbG{{\widetilde{\bG}}}
\def\bh{\mathbold{h}}\def\hhat{\widehat{h}}\def\htil{\widetilde{h}}
\def\hbh{{\widehat{\bh}}}\def\tbh{{\widetilde{\bh}}}
\def\bH{\mathbold{H}}\def\calH{{\cal H}}\def\scrH{{\mathscr H}}\def\Hhat{\widehat{H}}\def\Htil{{\widetilde H}}
\def\hbH{{\widehat{\bH}}}\def\tbH{{\widetilde{\bH}}}
\def\bi{\mathbold{i}}\def\ihat{\widehat{i}}\def\itil{\widetilde{i}}\def\ibar{{\overline i}}
\def\hbi{{\widehat{\bi}}}\def\tbi{{\widetilde{\bi}}}
\def\bI{\mathbold{I}}\def\hbI{{\widehat{\bI}}}\def\tbI{{\widetilde{\bI}}}
\def\calI{{\cal I}}\def\scrI{{\mathscr I}}\def\Ihat{\widehat{I}}\def\Itil{{\widetilde I}}\def\Ibar{{\overline I}}
\def\bj{\mathbold{j}}\def\jhat{\widehat{j}}\def\jtil{\widetilde{j}}\def\jbar{{\overline j}}
\def\hbj{{\widehat{\bj}}}\def\tbj{{\widetilde{\bj}}}
\def\bJ{\mathbold{J}}\def\hbJ{{\widehat{\bJ}}}\def\tbJ{{\widetilde{\bJ}}}
\def\calJ{{\cal J}}\def\scrJ{{\mathscr J}}\def\Jhat{\widehat{J}}\def\Jtil{{\widetilde J}}\def\Jbar{{\overline J}}
\def\bk{\mathbold{k}}\def\khat{\widehat{k}}\def\ktil{\widetilde{k}}\def\kbar{{\overline k}}
\def\hbk{{\widehat{\bk}}}\def\tbk{{\widetilde{\bk}}}
\def\bK{\mathbold{K}}\def\hbK{{\widehat{\bK}}}\def\tbK{{\widetilde{\bK}}}
\def\calK{{\cal K}}\def\scrK{{\mathscr K}}\def\Khat{\widehat{K}}\def\Ktil{{\widetilde K}}\def\Kbar{{\overline K}}
\def\bell{\mathbold{\ell}}\def\ellhat{\widehat{\ell}}\def\elltil{\widetilde{\ell}}\def\ellbar{{\overline \ell}}
\def\hbell{{\widehat{\bell}}}\def\tbell{{\widetilde{\bell}}}
\def\bL{\mathbold{L}}\def\hbL{{\widehat{\bL}}}\def\tbL{{\widetilde{\bL}}}
\def\calL{{\cal L}}\def\scrL{{\mathscr L}}\def\Lhat{\widehat{L}}\def\Ltil{{\widetilde L}}\def\Lbar{{\overline L}}
%\def\bm{\mathbold{m}}\def\mhat{\widehat{m}}\def\mtil{\widetilde{m}}\def\mbar{{\overline m}}
\def\hbm{{\widehat{\bm}}}\def\tbm{{\widetilde{\bm}}}
\def\bM{\mathbold{M}}\def\hbM{{\widehat{\bM}}}\def\tbM{{\widetilde{\bM}}}
\def\calM{{\cal M}}\def\scrM{{\mathscr M}}\def\Mhat{\widehat{M}}\def\Mtil{{\widetilde M}}\def\Mbar{{\overline M}}
\def\bn{\mathbold{n}}\def\nhat{\widehat{n}}\def\ntil{\widetilde{n}}\def\nbar{{\overline n}}
\def\hbn{{\widehat{\bn}}}\def\tbn{{\widetilde{\bn}}}
\def\bN{\mathbold{N}}\def\hbN{{\widehat{\bN}}}\def\tbN{{\widetilde{\bN}}}
\def\calN{{\cal N}}\def\scrN{{\mathscr N}}\def\Nhat{\widehat{N}}\def\Ntil{{\widetilde N}}\def\Nbar{{\overline N}}
\def\bo{\mathbold{o}}\def\ohat{\widehat{o}}\def\otil{\widetilde{o}}\def\obar{{\overline o}}
\def\hbo{{\widehat{\bo}}}\def\tbo{{\widetilde{\bo}}}
\def\bO{\mathbold{O}}\def\hbO{{\widehat{\bO}}}\def\tbO{{\widetilde{\bO}}}
\def\calO{{\cal O}}\def\scrO{{\mathscr O}}\def\Ohat{\widehat{O}}\def\Otil{{\widetilde O}}\def\Obar{{\overline O}}
\def\bp{\mathbold{p}}\def\phat{\widehat{p}}\def\ptil{\widetilde{p}}\def\pbar{{\overline p}}
\def\hbp{{\widehat{\bp}}}\def\tbp{{\widetilde{\bp}}}
\def\bP{\mathbold{P}}\def\hbP{{\widehat{\bP}}}\def\tbP{{\widetilde{\bP}}}
\def\calP{{\cal P}}\def\scrP{{\mathscr P}}\def\Phat{\widehat{P}}\def\Ptil{{\widetilde P}}\def\Pbar{{\overline P}}
\def\bq{\mathbold{q}}\def\qhat{\widehat{q}}\def\qtil{\widetilde{q}}\def\qbar{{\overline q}}
\def\hbq{{\widehat{\bq}}}\def\tbq{{\widetilde{\bq}}}
\def\bQ{\mathbold{Q}}\def\hbQ{{\widehat{\bQ}}}\def\tbQ{{\widetilde{\bQ}}}
\def\calQ{{\cal Q}}\def\scrQ{{\mathscr Q}}\def\Qhat{\widehat{Q}}\def\Qtil{{\widetilde Q}}\def\Qbar{{\overline Q}}
\def\br{\mathbold{r}}\def\rhat{\widehat{r}}\def\rtil{\widetilde{r}}\def\rbar{{\overline r}}
\def\hbr{{\widehat{\br}}}\def\tbr{{\widetilde{\br}}}
\def\bR{\mathbold{R}}\def\hbR{{\widehat{\bR}}}\def\tbR{{\widetilde{\bR}}}
\def\calR{{\cal R}}\def\scrR{{\mathscr R}}\def\Rhat{\widehat{R}}\def\Rtil{{\widetilde R}}\def\Rbar{{\overline R}}
\def\bs{\mathbold{s}}\def\shat{\widehat{s}}\def\stil{\widetilde{s}}\def\sbar{{\overline s}}
\def\hbs{{\widehat{\bs}}}\def\tbs{{\widetilde{\bs}}}
\def\bS{\mathbold{S}}\def\hbS{{\widehat{\bS}}}\def\tbS{{\widetilde{\bS}}}
\def\calS{{\cal S}}\def\scrS{{\mathscr S}}\def\Shat{\widehat{S}}\def\Stil{{\widetilde S}}\def\Sbar{{\overline S}}
\def\bt{\mathbold{t}}\def\that{\widehat{t}}\def\ttil{\widetilde{t}}\def\tbar{{\overline t}}
\def\hbt{{\widehat{\bt}}}\def\tbt{{\widetilde{\bt}}}
\def\bT{\mathbold{T}}\def\hbT{{\widehat{\bT}}}\def\tbT{{\widetilde{\bT}}}
\def\calT{{\cal T}}\def\scrT{{\mathscr T}}\def\That{\widehat{T}}\def\Ttil{{\widetilde T}}\def\Tbar{{\overline T}}
\def\bu{\mathbold{u}}\def\uhat{\widehat{u}}\def\util{\widetilde{u}}\def\ubar{{\overline u}}
\def\hbu{{\widehat{\bu}}}\def\tbu{{\widetilde{\bu}}}
\def\bU{\mathbold{U}}\def\hbU{{\widehat{\bU}}}\def\tbU{{\widetilde{\bU}}}
\def\calU{{\cal U}}\def\scrU{{\mathscr U}}\def\Uhat{\widehat{U}}\def\Util{{\widetilde U}}\def\Ubar{{\overline U}}
\def\bv{\mathbold{v}}\def\vhat{\widehat{v}}\def\vtil{\widetilde{v}}\def\vbar{{\overline v}}
\def\hbv{{\widehat{\bv}}}\def\tbv{{\widetilde{\bv}}}
\def\bV{\mathbold{V}}\def\hbV{{\widehat{\bV}}}\def\tbV{{\widetilde{\bV}}}
\def\calV{{\cal V}}\def\scrV{{\mathscr V}}\def\Vhat{\widehat{V}}\def\Vtil{{\widetilde V}}\def\Vbar{{\overline V}}
\def\bw{\mathbold{w}}\def\what{\widehat{w}}\def\wtil{\widetilde{w}}\def\wbar{{\overline w}}
\def\hbw{{\widehat{\bw}}}\def\tbw{{\widetilde{\bw}}}
\def\bW{\mathbold{W}}\def\hbW{{\widehat{\bW}}}\def\tbW{{\widetilde{\bW}}}
\def\calW{{\cal W}}\def\scrW{{\mathscr W}}\def\What{\widehat{W}}\def\Wtil{{\widetilde W}}\def\Wbar{{\overline W}}
\def\bx{\mathbold{x}}\def\xhat{\widehat{x}}\def\xtil{\widetilde{x}}\def\xbar{{\overline x}}\def\vbx{{\bf x}}
\def\hbx{{\widehat{\bx}}}\def\tbx{{\widetilde{\bx}}}\def\bxbar{{\overline \bx}}
\def\bX{\mathbold{X}}\def\hbX{{\widehat{\bX}}}\def\tbX{{\widetilde{\bX}}}\def\bXbar{{\overline \bX}}
\def\calX{{\cal X}}\def\scrX{{\mathscr X}}\def\Xhat{\widehat{X}}\def\Xtil{{\widetilde X}}\def\Xbar{{\overline X}}
\def\by{\mathbold{y}}\def\yhat{\widehat{y}}\def\ytil{\widetilde{y}}\def\ybar{{\overline y}}\def\vby{{\bf y}}
\def\hby{{\widehat{\by}}}\def\tby{{\widetilde{\by}}}
\def\bY{\mathbold{Y}}\def\hbY{{\widehat{\bY}}}\def\tbY{{\widetilde{\bY}}}
\def\calY{{\cal Y}}\def\scrY{{\mathscr Y}}\def\Yhat{\widehat{Y}}\def\Ytil{{\widetilde Y}}\def\Ybar{{\overline Y}}
\def\bz{\mathbold{z}}\def\zhat{\widehat{z}}\def\ztil{\widetilde{z}}\def\zbar{{\overline z}}
\def\hbz{{\widehat{\bz}}}\def\tbz{{\widetilde{\bz}}}
\def\bZ{\mathbold{Z}}\def\hbZ{{\widehat{\bZ}}}\def\tbZ{{\widetilde{\bZ}}}
\def\calZ{{\cal Z}}\def\scrZ{{\mathscr Z}}\def\Zhat{\widehat{Z}}\def\Ztil{{\widetilde Z}}\def\Zbar{{\overline Z}}
%%% Greek
\def\balpha{\mathbold{\alpha}}\def\alphahat{\widehat{\alpha}}\def\halpha{\widehat{\alpha}}
\def\alphatil{\widetilde{\alpha}}\def\talpha{\widetilde{\alpha}}\def\alphabar{{\overline{\alpha}}}
\def\hbalpha{{\widehat{\balpha}}}\def\tbalpha{{\widetilde{\balpha}}}
\def\bbeta{\mathbold{\beta}}\def\betahat{\widehat{\beta}}\def\hbeta{\widehat{\beta}}
\def\betatil{\widetilde{\beta}}\def\tbeta{\widetilde{\beta}}\def\betabar{{\overline{\beta}}}
\def\hbbeta{{\widehat{\bbeta}}}\def\tbbeta{{\widetilde{\bbeta}}}
\def\bbetabar{{\overline\bbeta}}
\def\bgamma{\mathbold{\gamma}}\def\gammahat{\widehat{\gamma}}\def\hgamma{\widehat{\gamma}}
\def\gammatil{\widetilde{\gamma}}\def\tgamma{\widetilde{\gamma}}\def\gammabar{{\overline{\gamma}}}
\def\hbgamma{{\widehat{\bgamma}}}\def\tbgamma{{\widetilde{\bgamma}}}
\def\bGamma{\mathbold{\Gamma}}\def\hbGamma{{\widehat{\bGamma}}}
\def\tbGamma{{\widetilde{\bGamma}}}\def\calGamma{{\cal \Gamma}}
\def\scrGamma{{\mathscr \Gamma}}\def\Gammahat{\widehat{\Gamma}}
\def\Gammatil{{\widetilde \Gamma}}\def\Gammabar{{\overline \Gamma}}
\def\bdelta{\mathbold{\delta}}\def\deltahat{\widehat{\delta}}\def\hdelta{\widehat{\delta}}
\def\deltatil{\widetilde{\delta}}\def\tdelta{\widetilde{\delta}}\def\deltabar{{\overline{\delta}}}
\def\hbdelta{{\widehat{\bdelta}}}\def\tbdelta{{\widetilde{\bdelta}}}
\def\bDelta{\mathbold{\Delta}}\def\hbDelta{{\widehat{\bDelta}}}
\def\tbDelta{{\widetilde{\bDelta}}}\def\calDelta{{\cal \Delta}}
\def\scrDelta{{\mathscr \Delta}}\def\Deltahat{\widehat{\Delta}}
\def\Deltatil{{\widetilde \Delta}}\def\Deltabar{{\overline \Delta}}
\def\ep{\varepsilon}\def\eps{\epsilon}\def\veps{\varepsilon}
\def\bep{\mathbold{\ep}}\def\ephat{\widehat{\ep}}\def\hep{\widehat{\ep}}
\def\eptil{\widetilde{\ep}}\def\tep{\widetilde{\ep}}\def\epbar{{\overline{\ep}}}
\def\hbep{{\widehat{\bep}}}\def\tbep{{\widetilde{\bep}}}
\def\bzeta{\mathbold{\zeta}}\def\zetahat{\widehat{\zeta}}\def\hzeta{\widehat{\zeta}}
\def\zetatil{\widetilde{\zeta}}\def\tzeta{\widetilde{\zeta}}\def\zetabar{{\overline{\zeta}}}
\def\hbzeta{{\widehat{\bzeta}}}\def\tbzeta{{\widetilde{\bzeta}}}
\def\bfeta{\mathbold{\eta}}\def\etahat{\widehat{\eta}}\def\heta{\widehat{\eta}}
\def\etatil{\widetilde{\eta}}\def\teta{\widetilde{\eta}}\def\etabar{{\overline{\eta}}}
\def\hbfeta{{\widehat{\bfeta}}}\def\tbfeta{{\widetilde{\bfeta}}}
\def\btheta{\mathbold{\theta}}\def\thetahat{\widehat{\theta}}\def\htheta{\widehat{\theta}}
\def\thetatil{\widetilde{\theta}}\def\ttheta{\widetilde{\theta}}\def\thetabar{{\overline{\theta}}}
\def\hbtheta{{\widehat{\btheta}}}\def\tbtheta{{\widetilde{\btheta}}}
\def\bTheta{\mathbold{\Theta}}\def\hbTheta{{\widehat{\bTheta}}}
\def\tbTheta{{\widetilde{\bTheta}}}\def\calTheta{{\cal \Theta}}
\def\scrTheta{{\mathscr \Theta}}\def\Thetahat{\widehat{\Theta}}
\def\Thetatil{{\widetilde \Theta}}\def\Thetabar{{\overline \Theta}}
\def\bkappa{\mathbold{\kappa}}\def\kappahat{\widehat{\kappa}}\def\hkappa{\widehat{\kappa}}
\def\kappatil{\widetilde{\kappa}}\def\tkappa{\widetilde{\kappa}}\def\kappabar{{\overline{\kappa}}}
\def\hbkappa{{\widehat{\bkappa}}}\def\tbkappa{{\widetilde{\bkappa}}}
\def\lam{\lambda}
\def\blam{\mathbold{\lam}}\def\lamhat{\widehat{\lam}}\def\hlam{\widehat{\lam}}
\def\lamtil{\widetilde{\lam}}\def\tlam{\widetilde{\lam}}\def\lambar{{\overline{\lam}}}
\def\hblam{{\widehat{\blam}}}\def\tblam{{\widetilde{\blam}}}
\def\blambda{\mathbold{\lambda}}\def\lambdahat{\widehat{\lambda}}\def\hlambda{\widehat{\lambda}}
\def\lambdatil{\widetilde{\lambda}}\def\tlambda{\widetilde{\lambda}}\def\lambdabar{{\overline{\lambda}}}
\def\hblambda{{\widehat{\blambda}}}\def\tblambda{{\widetilde{\blambda}}}
\def\bLambda{\mathbold{\Lambda}}\def\hbLambda{{\widehat{\bLambda}}}
\def\tbLambda{{\widetilde{\bLambda}}}\def\calLambda{{\cal \Lambda}}
\def\scrLambda{{\mathscr \Lambda}}\def\Lambdahat{\widehat{\Lambda}}
\def\Lambdatil{{\widetilde \Lambda}}\def\Lambdabar{{\overline \Lambda}}
\def\bmu{\mathbold{\mu}}\def\muhat{\widehat{\mu}}\def\hmu{\widehat{\mu}}
\def\mutil{\widetilde{\mu}}\def\tmu{\widetilde{\mu}}\def\mubar{{\overline{\mu}}}
\def\hbmu{{\widehat{\bmu}}}\def\tbmu{{\widetilde{\bmu}}}
\def\bnu{\mathbold{\nu}}\def\nuhat{\widehat{\nu}}\def\hnu{\widehat{\nu}}
\def\nutil{\widetilde{\nu}}\def\tnu{\widetilde{\nu}}\def\nubar{{\overline{\nu}}}
\def\hbnu{{\widehat{\bnu}}}\def\tbnu{{\widetilde{\bnu}}}
\def\bxi{\mathbold{\xi}}\def\xihat{\widehat{\xi}}\def\hxi{\widehat{\xi}}
\def\xitil{\widetilde{\xi}}\def\txi{\widetilde{\xi}}\def\xibar{{\overline{\xi}}}
\def\hbxi{{\widehat{\bxi}}}\def\tbxi{{\widetilde{\bxi}}}
\def\bXi{\mathbold{\Xi}}\def\hbXi{{\widehat{\bXi}}}
\def\tbXi{{\widetilde{\bXi}}}\def\calXi{{\cal \Xi}}
\def\scrXi{{\mathscr \Xi}}\def\Xihat{\widehat{\Xi}}
\def\Xitil{{\widetilde \Xi}}\def\Xibar{{\overline \Xi}}
\def\bpi{\mathbold{\pi}}\def\pihat{\widehat{\pi}}\def\hpi{\widehat{\pi}}
\def\pitil{\widetilde{\pi}}\def\tpi{\widetilde{\pi}}\def\pibar{{\overline{\pi}}}
\def\hbpi{{\widehat{\bpi}}}\def\tbpi{{\widetilde{\bpi}}}
\def\bPi{\mathbold{\Pi}}\def\hbPi{{\widehat{\bPi}}}
\def\tbPi{{\widetilde{\bPi}}}\def\calPi{{\cal \Pi}}
\def\scrPi{{\mathscr \Pi}}\def\Pihat{\widehat{\Pi}}
\def\Pitil{{\widetilde \Pi}}\def\Pibar{{\overline \Pi}}
\def\brho{\mathbold{\rho}}\def\rhohat{\widehat{\rho}}\def\hrho{\widehat{\rho}}
\def\rhotil{\widetilde{\rho}}\def\trho{\widetilde{\rho}}\def\rhobar{{\overline{\rho}}}
\def\hbrho{{\widehat{\brho}}}\def\tbrho{{\widetilde{\brho}}}
\def\bsigma{\mathbold{\sigma}}\def\sigmahat{\widehat{\sigma}}\def\hsigma{\widehat{\sigma}}
\def\sigmatil{\widetilde{\sigma}}\def\tsigma{\widetilde{\sigma}}\def\sigmabar{{\overline{\sigma}}}
\def\hbsigma{{\widehat{\bsigma}}}\def\tbsigma{{\widetilde{\bsigma}}}
\def\bsigmabar{{\overline\bsigma}}
\def\bSigma{\mathbold{\Sigma}}\def\hbSigma{{\widehat{\bSigma}}}
\def\tbSigma{{\widetilde{\bSigma}}}\def\calSigma{{\cal \Sigma}}
\def\scrSigma{{\mathscr \Sigma}}\def\Sigmahat{\widehat{\Sigma}}
\def\Sigmatil{{\widetilde \Sigma}}\def\Sigmabar{{\overline \Sigma}}
\def\bSigmabar{{\overline\bSigma}}
\def\btau{\mathbold{\tau}}\def\tauhat{\widehat{\tau}}\def\htau{\widehat{\tau}}
\def\tautil{\widetilde{\tau}}\def\ttau{\widetilde{\tau}}\def\taubar{{\overline{\tau}}}
\def\hbtau{{\widehat{\btau}}}\def\tbtau{{\widetilde{\btau}}}
\def\bphi{\mathbold{\phi}}\def\phihat{\widehat{\phi}}\def\hphi{\widehat{\phi}}
\def\phitil{\widetilde{\phi}}\def\tphi{\widetilde{\phi}}\def\phibar{{\overline{\phi}}}
\def\hbphi{{\widehat{\bphi}}}\def\tbphi{{\widetilde{\bphi}}}
\def\bPhi{\mathbold{\Phi}}\def\hbPhi{{\widehat{\bPhi}}}
\def\tbPhi{{\widetilde{\bPhi}}}\def\calPhi{{\cal \Phi}}
\def\scrPhi{{\mathscr \Phi}}\def\Phihat{\widehat{\Phi}}
\def\Phitil{{\widetilde \Phi}}\def\Phibar{{\overline \Phi}}
\def\bchi{\mathbold{\chi}}\def\chihat{\widehat{\chi}}\def\hchi{\widehat{\chi}}
\def\chitil{\widetilde{\chi}}\def\tchi{\widetilde{\chi}}\def\chibar{{\overline{\chi}}}
\def\hbchi{{\widehat{\bchi}}}\def\tbchi{{\widetilde{\bchi}}}
\def\bpsi{\mathbold{\psi}}\def\psihat{\widehat{\psi}}\def\hpsi{\widehat{\psi}}
\def\psitil{\widetilde{\psi}}\def\tpsi{\widetilde{\psi}}\def\psibar{{\overline{\psi}}}
\def\hbpsi{{\widehat{\bpsi}}}\def\tbpsi{{\widetilde{\bpsi}}}
\def\bPsi{\mathbold{\Psi}}\def\hbPsi{{\widehat{\bPsi}}}
\def\tbPsi{{\widetilde{\bPsi}}}\def\calPsi{{\cal \Psi}}
\def\scrPsi{{\mathscr \Psi}}\def\Psihat{\widehat{\Psi}}
\def\Psitil{{\widetilde \Psi}}\def\Psibar{{\overline \Psi}}
\def\bomega{\mathbold{\omega}}\def\omegahat{\widehat{\omega}}\def\homega{\widehat{\omega}}
\def\omegatil{\widetilde{\omega}}\def\tomega{\widetilde{\omega}}\def\omegabar{{\overline{\omega}}}
\def\hbomega{{\widehat{\bomega}}}\def\tbomega{{\widetilde{\bomega}}}
\def\bOmega{\mathbold{\Omega}}\def\hbOmega{{\widehat{\bOmega}}}
\def\tbOmega{{\widetilde{\bOmega}}}\def\calOmega{{\cal \Omega}}
\def\scrOmega{{\mathscr \Omega}}\def\Omegahat{\widehat{\Omega}}
\def\Omegatil{{\widetilde \Omega}}\def\Omegabar{{\overline \Omega}}
%% additional Asaf stuff
\def\0{\mathbold{0}}
%!TEX root = SpikeCCA.tex
In this section, we propose two natural loss functions to quantify the discrepancy between the population and sample canonical variates through a prediction-after-dimension-reduction framework.
%$\pxhk^\top\bx$ and $\pxk^\top\bx$ .
%further justify the subspace point of view and the proposed loss functions under a prediction-after-dimension-reduction framework. Specifically, we quantify the linear prediction loss for the output of any dimension reduction algorithm.
\subsection{Linear Prediction Revisited}
First of all, we review the basics of linear model theory under the random design setup. Given
\[
x_1, \ldots, x_p, z \in L^2(\Omega, \mathcal{F}, \mathcal{P}),
\]
where $ L^2(\Omega, \mathcal{F}, \mathcal{P})$ is the set of random variables with mean zero and finite second moment.
%with mean zero and $\vct{x} = (x_1, \ldots, x_p)^\top$.
Suppose the goal is to predict the response $z$ with the random vector $\bx = (x_1, \ldots, x_p)^\top$. We measure the prediction loss by
\[
loss(z|\vct{x}) = \min_{\vct{\beta} \in \mathbb{R}^p} \E [(z - \vct{x}^\top \vct{\beta})^2].
\]
We further assume that $(\bx, z)$ has joint covariance matrix:
\[
\text{Cov}\left(\begin{bmatrix} \vct{x} \\ z \end{bmatrix}\right) = \begin{bmatrix} \mtx{\Sigma}_x & \vct{\Sigma}_{xz} \\ \vct{\Sigma}_{xz}^\top & \sigma_z^2 \end{bmatrix}.
\]
By classical linear model theory
\begin{gather*}
\beta^*=\arg \min_{\vct{\beta} \in \mathbb{R}^p} \E [(z - \vct{x}^\top \vct{\beta})^2]=\sx^{-1}\s_{xz}\\
loss(z|\vct{x})=\sigma_z^2-\s_{zx}\sx^{-1}\s_{xz}=\sigma_z^2(1-\|\br_{xz}\|^2)
\end{gather*}
where $\br_{xz}=\sx^{-1}\s_{xz}\sigma_z^{-1}$ is the correlation vector between $\bx$ and $z$, and $\|\br_{xz}\|^2$ is the population $R^2$ which characterizes the proportion of the variability in the response $z$ explained by the predictor $\bx$. One important feature of such prediction loss is its linear invariance. Define the linear subspace spanned by the coordinates of $\bx$ as
\[
\lspan (\vct{x}^\top) = \{ \vct{x}^\top\bw: \vct{w} \in \mathbb{R}^p\} \subset L^2(\Omega, \mathcal{F}, \mathcal{P}),
\]
For another set of random variables $\{v_1, \ldots, v_q\}\in L^2(\Omega, \mathcal{F}, \mathcal{P})$, if $\lspan(\bv)=\lspan(\bx)$ by denoting $\vct{v} = (v_1, \ldots, v_q)^\top$, then $loss(z|\vct{x}) = loss(z|\vct{v})$ and this implies that we can rewrite
\[
loss(z|\vct{x}) = loss(z|\lspan(\vct{x}^\top)).
\]
These two notations will be used interchangeably throughout the paper. The linear invariance property can be revealed by noticing that $\|\br_{xz}\|^2=\E[(\bP_{\lspan(\bx)} z)^2]/\E [z^2]$ where $\bP_{(\cdot)}$ is the projection operator defined in the Hilbert space $ L^2(\Omega, \mathcal{F}, \mathcal{P})$ with inner product defined as the covariance between two random variables.
%\subsection{On the Bayesian Setup}
%If $\text{Cov}(\pi)=\bI_p/p$, then
%\begin{align*}
%&\underset{ \br_{xz}\sim \pi}{\E}\left[ loss(z|\lspan(\mtx{U}_1 \vct{x})) - loss(z|\lspan(\mtx{U}_2 \vct{x}))\right ]\\
%&=\sigma_z^2~ \E\left[\br_{xz}^\top \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\br_{xz}\right)\right]\\
%& =\sigma_z^2\tr\left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)/p\\
%&\leq \sigma_z^2\tr\left(\bP_{\sx^{\half}\bU_1}+\bP_{\sx^{\half}\bU_2}-2\bP_{\sx^{\half}\bU_1}\bP_{\sx^{\half}\bU_2}\right)/2p\\
%&=\sigma_z^2\left\|\bP_{\sx^{\half}\bU_1}-\bP_{\sx^{\half}\bU_2}\right\|_F^2/2p
%\end{align*}
%The factor $p$ appears instead of $k$ is due to the fact that $r_{xz}$ is drawn independent of $\bU_1$ and $\bU_2$. Maybe it is more sensible to consider the case where $\lspan(\bU_2\bx)$ is the oracle reduction space that fully captures the correlation, namely $\bP_{\sx^{\half}\bU_2}\br_{xz}=\br_{xz}$. Assume $\bP_{\sx^{\half}\bU_2}=\bQ\bQ^\top$ and there exists $\wt{\br}\in\mathbb{R}^k$ such that $\br_{xz}=Q\wt{\br}$. Suppose $\wt{\br}\sim \wt{\pi}$ and $\text{Cov}(\wt{\pi})=\bI_k/k$, then
%\begin{align*}
%&\underset{ \br_{xz}\sim \pi}{\E}\left[ loss(z|\lspan(\mtx{U}_1 \vct{x})) - loss(z|\lspan(\mtx{U}_2 \vct{x}))\right ]\\
%&=\sigma_z^2~ \E\left[\br_{xz}^\top \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\br_{xz}\right]\\
%& =\sigma_z^2~ \E\left[\wt{\br}^\top\bQ^\top \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\bQ\wt{\br} \right]\\
%&= \sigma_z^2\tr\left(\bI_k-\bQ^\top\bP_{\sx^{\half}\bU_1}\bQ \right)/k\\
%&= \sigma_z^2\tr\left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_2}\bP_{\sx^{\half}\bU_1} \right)/k\\
%&=\sigma_z^2\left\|\bP_{\sx^{\half}\bU_1}-\bP_{\sx^{\half}\bU_2}\right\|_F^2/2k
%\end{align*}
%With $\bU_2$ being the oracle reduction space, the worst case excess loss bound can be strengthened as well.
%\begin{align*}
%&loss(z|\lspan(\mtx{U}_1 \vct{x})) - loss(z|\lspan(\mtx{U}_2 \vct{x}))\\
%&=\sigma_z^2 \left(\br_{xz}^\top \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\br_{xz}\right)\\
%&=\sigma_z^2 \br_{xz}^\top \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\br_{xz}\\
%&=\sigma_z^2 \br_{xz}^\top\bP_{\sx^{\half}\bU_2} \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\bP_{\sx^{\half}\bU_2}\br_{xz}
%\end{align*}
%Therefore,
%\begin{align*}
%&\max\limits_{\br_{xz}\in\mathbb{R}^p, \|\br_{xz}\|\leq 1} loss(z|\lspan(\mtx{U}_1 \vct{x})) - loss(z|\lspan(\mtx{U}_2 \vct{x})) \\
%&= \sigma_z^2 \left\|\bP_{\sx^{\half}\bU_2} \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\bP_{\sx^{\half}\bU_2}\right\|\\
%&=\sigma_z^2 \left\|\bP_{\sx^{\half}\bU_2} \left(\bI_p-\bP_{\sx^{\half}\bU_1}\right)\bP_{\sx^{\half}\bU_2}\right\|\\
%&=\sigma_z^2 \left\|\bP_{\sx^{\half}\bU_2} \left(\bI_p-\bP_{\sx^{\half}\bU_1}\right)\right\|^2\\
%&=\sigma_z^2 \left\| \bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right\|^2
%\end{align*}
%With $bU_2$ being the oracle space, the algebra turns out be much nicer than the generic case.
\subsection{Competing with Oracles}
Consider the scenario where the predictor $\bx$ is in a high dimensional space and many directions in $\lspan(\bx)$ might be redundant in terms of predicting the response $z$. Practitioners usually perform certain kind of dimension reduction on $\bx$ before applying supervised learning algorithms. Suppose $\bU \in \mathbb{R}^{p \times k}$ is the reduction matrix obtained by some generic dimension reduction method. The subspace perspective of the prediction loss discussed in previous section suggests $loss(z|\lspan(\bx^\top\bU))-loss(z|\lspan(\bx))$, or simply $loss(z|\lspan(\bx^\top\bU))$ as the measure of goodness for dimension reduction algorithms.
For any given reduction matrices $\bU_1, \bU_2\in\mathbb{R}^{p\times k}$, the discrepancy between their prediction loss can be quantified by
\begin{align}
loss(z|\lspan(\bx^\top\bU_1)) - loss(z|\lspan(\bU_2^\top \vct{x})) &= \E[(\bP_{\lspan(\bx^\top\bU_2)} z)^2- (\bP_{\lspan(\bx^\top\bU_1)} z)^2]\\
&=\sigma_z^2 \left(\br_{xz}^\top \left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\br_{xz}\right)
\label{eq-excess-risk}
\end{align}
as will be proved in Section~\ref{section-proof-loss}. The first equality is geometrically straightforward, measuring the proportion of the variability in response $z$ explained by the two subspaces $\lspan(\mtx{U}_1^\top \vct{x})$ and $\lspan(\bU_2^\top \vct{x})$. The algebraic expression in the second equality is less obvious but decouples the loss into an interaction between a supervised learning factor $\br_{xz}$ and an unsupervised learning factor $\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}$. To shed more light on this excess risk, we parametrize the joint covariance matrix of $(\bx, z)$ in terms of separate covariance matrices $\sx, \sigma_z^2$ and the correlation vector $\br_{xz}=\text{correlation}(\bx, z)$, that is
\begin{align}
\text{Cov}\left(\begin{bmatrix} \vct{x} \\ z \end{bmatrix}\right) = \begin{bmatrix} \mtx{\Sigma}_x & \vct{\Sigma}_{xz} \\ \vct{\Sigma}_{xz}^\top & \sigma_z^2 \end{bmatrix}=\begin{bmatrix} \mtx{\Sigma}_x^{\half} & 0 \\ 0 & \sigma_z \end{bmatrix} \begin{bmatrix} \bI_{p} & \vct{r}_{xz} \\ \vct{r}_{zx} & 1 \end{bmatrix}\begin{bmatrix} \mtx{\Sigma}_x^{\half} & 0 \\ 0 & \sigma_z \end{bmatrix} .
\label{eq-cov}
\end{align}
%Notice that $\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}$ is a symmetric matrix. By the variational characterization of eigenvalues, if we fix $\|\br_{xz}\|^2= R^2$ and
Considering the worst case discrepancy across all possible correlation structures,
\begin{align}
\sup \limits_{\|\br_{xz}\|^2= R^2} \left\{ loss(z|\lspan(\bx^\top\bU_1)) - loss(z|\lspan(\bx^\top\bU_2))\right\} = \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU_1}-\bP_{\sx^{\half}\bU_2}\right\|,
\end{align}
which suggests the right hand side of the equation as a sensible metric to quantify the difference between the two reduction matrices. Sometimes, it might be more informative to replace the competitor $\bU_2$ with an oracle reduction matrix, denoted by $\os$. As suggested by \eqref{eq-excess-risk}, we say a reduction matrix $\os$ is an oracle reduction matrix if $\bP_{\sx^{\half}\os}\br_{xz}=\br_{xz}$.
%Under a frequentist's setup, we can assume the oracle reduction matrix $\os$ fully captures the correlation vector $\br_{xz}$, that is, $\bP_{\sx^{\half}\os}\br_{xz}=\br_{xz}$ as suggested by \eqref{eq-excess-risk}.
Define $\mc{A}=\{\br: \|\br\|^2=R^2, \bP_{\sx^{\half}\os}\br=\br\}$ as the set of correlation vectors with fixed population $R^2$ such that $\os$ is an oracle reduction matrix. If considering the worst case excess prediction loss within $\mc{A}$, as proved in Section~\ref{section-proof-loss},
\begin{align}
\sup\limits_{\br_{xz}\in\mc{A}}\left\{ loss(z|\lspan(\bx^\top\bU)) - loss(z|\lspan(\vct{x}^\top\os))\right\} = \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2.
\label{eq-oracle1}
\end{align}
%\begin{align}
%\sup\limits_{\substack{\|\br_{xz}\|^2= R^2,\\\bP_{\sx^{\half}\os}\br_{xz}=\br_{xz}}}\left\{ loss(z|\lspan(\mtx{U}^\top \vct{x})) - loss(z|\lspan(\os^\top \vct{x}))\right\} = \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2
%\label{eq-adversary1}
%\end{align}
%\begin{align}
%\underset{\|\br_{xz}\|^2= R^2,\bP_{\sx^{\half}\os}\br_{xz}=\br_{xz}}{\sup}\left\{ loss(z|\lspan(\mtx{U}^\top \vct{x})) - loss(z|\lspan(\os^\top \vct{x}))\right\} = \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2
%\label{eq-adversary1}
%\end{align}
Interestedly, the operator norm is replaced by its square when the competitor is an oracle reduction matrix. On the other hand, from a Bayesian perspective, considering the prior that the correlation vector $\br_{xz}$ is sampled according to the uniform measure (Haar measure) on $\mc{A}$, then it is proved in Section~\ref{section-proof-loss} that the average excess prediction loss satisfies:
%let $\bP_{\sx^{\half}\os}=\bQ_{\star}\bQ_{\star}^\top$, one might want to consider the prior that the correlation direction $\br_{xz}$ is `uniform' in the $k$ dimensional subspace spanned by the columns of $\bQ_{\star}$. To be comparable with the frequentist's setup, we assume $\br_{xz}=R\bQ_{\star}\wt{\br}$ where $\wt{\br}\sim \pi$ and $\pi$ is the uniform measure (Haar measure) on the sphere $\mathbb{S}^{k-1}$. It is proved in Section~\ref{section-proof-loss} that
% $\wt{\br}\in\mathbb{R}^k\sim \pi$ where $\pi$ is non-informative, supported on the unit ball with $\text{Cov}(\pi)=R^2\bI_k/k$. As shown later,
\begin{align}
\E_{\br_{xz}\sim\pi}\left\{ loss(z|\lspan(\bx^\top\mtx{U} )) - loss(z|\lspan( \vct{x}^\top\os ))\right\} =\frac{\sigma_z^2 R^2}{2k} \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|_F^2.
\label{eq-oracle2}
\end{align}
%To link such measure of goodness with the estimation of the reduction matrix, suppose there exists an oracle reduction matrix $\os\in\mathbb{R}^{p \times k}$ and we aim for an estimate $\bU$ to mimic its performance.
% The discrepancy between the prediction loss of $\bU$ and the oracle matrix $\os$ can be quantified by
%\begin{align*}
%loss(z|\lspan(\mtx{U}^\top \vct{x})) - loss(z|\lspan(\os^\top \vct{x})) &= \E[(\bP_{\lspan(\mtx{U}^\top \vct{x})} z)^2- (\bP_{\lspan(\os^\top \vct{x})} z)^2]\\
%&=\sigma_z^2 \left(\br_{xz}^\top \left(\bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right)\br_{xz}\right)
%\end{align*}
%The first equality is geometrically intuitive, measuring the proportion of the response $z$ explained by the two subspaces $\lspan(\mtx{U}^\top \vct{x})$ and $\lspan(\os^\top \vct{x})$. The algebraic expression in the second equality (proved later) is less obvious but decouples the loss into an interaction between a supervised learning factor $\br_{xz}$ and an unsupervised learning factor $\bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}$. To shed more light on this excess risk, we parametrize the joint covariance matrix of $(\bx, z)$ in terms of separate covariance matrices $\sx, \sigma_z^2$ and the correlation vector $\br_{xz}=\text{correlation}(\bx, z)$, that is
%\begin{align}
%\text{Cov}\left(\begin{bmatrix} \vct{x} \\ z \end{bmatrix}\right) = \begin{bmatrix} \mtx{\Sigma}_x & \vct{\Sigma}_{xz} \\ \vct{\Sigma}_{xz}^\top & \sigma_z^2 \end{bmatrix}=\begin{bmatrix} \mtx{\Sigma}_x^{\half} & 0 \\ 0 & \sigma_z \end{bmatrix} \begin{bmatrix} \bI_{p} & \vct{r}_{xz} \\ \vct{r}_{zx} & 1 \end{bmatrix}\begin{bmatrix} \mtx{\Sigma}_x^{\half} & 0 \\ 0 & \sigma_z \end{bmatrix} .
%\label{eq-cov}
%\end{align}
%From decision-theoretic point of view, as we will show later, the worst case excess risk over all possible correlation structures with fixed $R^2$ statistic is reduced to
%\begin{align}
%\sup \limits_{\|\br_{xz}\|^2= R^2} \left\{ loss(z|\lspan(\mtx{U}^\top \vct{x})) - loss(z|\lspan(\os^\top \vct{x}))\right\} = \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2
%\label{eq-adversary1}
%\end{align}
%Notice that we do not impose any kind of optimality assumptions on $\os$ and actually the notation of oracle is already taken into account by `sup' over all possible correlation structures and the direction that achieves the maximum will favor $\os$. On the other hand, from a Bayesian perspective, consider the prior that the correlation direction $\br_{xz}$ lies in a $k$ dimensional subspace with orthonormal basis matrix $\bQ\in\mathbb{O}(p, k)$, that is $\br_{xz}=\bQ\wt{\br}$ for some $\wt{\br}\in\mathbb{R}^k\sim \pi$ where $\pi$ is non-informative, supported on the unit ball with $\text{Cov}(\pi)=R^2\bI_k/k$, then
%\begin{align*}
%\sup\limits_{\bQ}~\E_{\pi}\left\{ loss(z|\lspan(\mtx{U}^\top \vct{x})) - loss(z|\lspan(\os^\top \vct{x}))\right\} = \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|_F^2/k
%\end{align*}
%Under this Bayesian framework, the ability of the adversary is restricted because the correlation direction is forced to be randomized over a subspace, while in \eqref{eq-adversary1}, the adversary is allowed to choose a specific direction for $\br_{zx}$. In this way, it is not surprising that on the left hand side, the operator norm changes to Frobenius norm.
The analysis in this section essentially connects the prediction loss for the response $z$ with the estimation loss for the oracle reduction matrix $\os$
%under the metric proposed in \eqref{eq-loss-intro}.
under the metrics derived in \eqref{eq-oracle1} and \eqref{eq-oracle2}. Therefore, when CCA is used for dimension reduction, it is natural to quantify the discrepancy between $\pxhk^\top\bx$ and $\pxk^\top\bx$ by the excess prediction loss:
\[
\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|^2, ~\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|_F^2.
\]
%\eqref{eq-loss-intro} exactly characterizes the prediction loss.
% under the metric proposed in \eqref{eq-loss-intro}.
%when the estimation error is quantified
%\[
%\left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2, ~~~ \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|_F^2
%\]
%which are exactly the loss functions we proposed in \eqref{eq-loss-intro}.
% in terms of $\left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2$ and $\left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|_F^2$. As spelled out in the next section, these two quantities turn out to be closely related to the subspace angles between $\lspan(\mtx{U}^\top \vct{x})$ and $\lspan(\os^\top \vct{x})$.
\subsection{Measuring Subspace Distance by Principal Angles}
\label{section-angle}
In this section, we show that the excess prediction loss derived in \eqref{eq-oracle1} and \eqref{eq-oracle2} are closely related to the principal angles between the two subspaces spanned by the predictors.
%In this section, we spell out the algebra that connects the algebraic expression of the loss functions with the geometric quantities in terms of subspace angles.
%reveal the geometric meaning of $\left\|\bP_{\sx^{\half}\bU_1}-\bP_{\sx^{\half}\bU_2}\right\|^2$ and $\left\|\bP_{\sx^{\half}\bU_1}-\bP_{\sx^{\half}\bU_2}\right\|_F^2$
For any $p$ dimensional random vector $\bx$ with mean zero and bounded second moments, define the Hilbert space
\[
\mathcal{H}=\lspan(\bx)=\{X|X=\vct{x}^\top\vct{w}, \vct{w} \in \mathbb{R}^{p}\}
\]
with covariance operator as the inner product, that is, for any $X_1, X_2\in \mathcal{H}$, $\langle X_1, X_2 \rangle=\text{Cov}(X_1, X_2)=\E(X_1X_2)$. For any pair of full column rank matrices $\bU_1, \bU_2\in\mathbb{R}^{p\times k}$, consider the canonical correlation analysis between the two subspaces: $\lspan(\vct{x}^\top\bU_1)$ and $\lspan(\vct{x}^\top\bU_2)$.
% Define the canonical angles between the two subspaces $\lspan(\mtx{U}_1^\top \vct{x})$ and $\lspan(\mtx{U}_2 ^\top\vct{x})$.Consider the canonical correlation analysis between two subspaces: $\lspan(\vct{x}^\top\bU_1)$ and $\lspan(\vct{x}^\top\bU_2)$ of the Hilbert space $\mathcal{H}$.
Suppose $(W_1, \widehat{W}_1), (W_2, \widehat{W}_2), \ldots, (W_k, \widehat{W}_k)$ are the first, second, ..., $k_{\text{th}}$ pair of canonical variates between $\lspan(\vct{x}^\top\bU_1)$ and $\lspan(\vct{x}^\top\bU_2)$, that is $\lspan(W_1, \ldots, W_k)=\lspan(\vct{x}^\top\bU_1)$, $\lspan(\widehat{W}_1, \ldots, \widehat{W}_k)=\lspan(\vct{x}^\top\bU_2)$ and $\langle W_i, W_j\rangle=\langle W_i, \widehat{W}_j\rangle=\langle \widehat{W}_i, \widehat{W}_j \rangle=0$, for any $i \neq j$ and $\text{Var}(W_i)=\text{Var}(\widehat{W}_i)=1$, for $i=1, \ldots, k$. Then $\{W_1, \ldots, W_k\}$ and $\{\widehat{W}_1, \ldots, \widehat{W}_k\}$ are orthonormal bases of $\lspan(\vct{x}^\top\bU_1)$ and $\lspan(\vct{x}^\top\bU_2)$, respectively. The $i_{\text{th}}$ principal angle is defined as $\theta_i = \angle (W_i, \widehat{W}_i)$. Without loss of generality we assume $\theta_1\geq \cdots\geq \theta_k$ and define the `distance' between the two subspaces as:
\[
\mathcal{L}_2(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2)):=\sum_{i=1}^k \sin^2 \theta_i = \sum_{i=1}^k \left(1 - \left|\left\langle W_i, \widehat{W}_i\right\rangle\right|^2\right).
\]
This is a valid `metric' because the principal angles are uniquely defined while the canonical variates need not be. Since $\vct{x}^\top\mtx{\Sigma}_x^{-\half}$ is an orthonormal basis of $\mathcal{H}$ under the covariance inner product, it is convenient to represent the elements in $\mathcal{H}$ by this basis. Let
\[
(W_1, \ldots, W_k)=\vct{x}^\top\mtx{\Sigma}_x^{-\half}\mtx{B}, \text{~and~} (\widehat{W}_1, \ldots, \widehat{W}_k)=\vct{x}^\top\mtx{\Sigma}_x^{-\half}\widehat{\mtx{B}},
\]
where $\mtx{B}:=[\vct{b}_1, \ldots, \vct{b}_k], \widehat{\mtx{B}}:=[\widehat{\vct{b}}_1, \ldots, \widehat{\vct{b}}_k] \in \mathbb{R}^{p \times k}$ are the coordinate representations under $\vct{x}^\top\mtx{\Sigma}_x^{-\half}$. Then $\mtx{B}, \widehat{\mtx{B}}$ are $p \times k$ basis matrices. Moreover, we have $\vct{b}_i^\top\widehat{\vct{b}}_j=\langle W_i, \widehat{W}_j \rangle=0$, for all $i \neq j$.
Let's now see how to represent $\mc{L}_2(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2))$ by $\mtx{B}$ and $\widehat{\bB}$. In fact, since
\[
1 - \left|\left\langle W_i, \widehat{W}_i\right\rangle\right|^2=1 - \left|\vct{b}_i^\top\widehat{\vct{b}}_i\right|^2 = \frac{1}{2} \left\|\vct{b}_i\vct{b}_i^\top - \widehat{\vct{b}}_i \widehat{\vct{b}}_i^\top\right\|_F^2,
\]
we have
\begin{align*}
\mathcal{L}_2(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2)) &= \frac{1}{2} \sum_{i=1}^k \left\|\vct{b}_i\vct{b}_i^\top - \widehat{\vct{b}}_i \widehat{\vct{b}}_i^\top\right\|_F^2
\\
&=\frac{1}{2}\left\|\sum_{i=1}^k \left(\vct{b}_i\vct{b}_i^\top - \widehat{\vct{b}}_i \widehat{\vct{b}}_i^\top\right) \right\|_F^2
\\
&= \frac{1}{2}\left\|\mtx{B}\mtx{B}^\top - \widehat{\mtx{B}}\widehat{\mtx{B}}^\top\right\|_F^2.
\end{align*}
Here the second equality is due to $\vct{b}_i^\top\vct{b}_j=\widehat{\vct{b}}_i^\top\vct{b}_j=\widehat{\vct{b}}_i^\top\widehat{\vct{b}}_j=0$, for all $i \neq j$.
Finally, notice that $\lspan(\vct{x}^\top\bU_1)=\lspan(W_1, \ldots, W_k)$, $\vct{x}^\top\bU_1= (\vct{x}^\top\mtx{\Sigma}_x^{-\half})(\mtx{\Sigma}_x^{\half}\bU_1)$, and $(W_1, \ldots, W_k)=\vct{x}^\top\mtx{\Sigma}_x^{-\half}\mtx{B}$. Then $\mtx{B}$ and $\mtx{\Sigma}_x^{\half}\bU_1$ have the same column space. Since $\mtx{B} \in \mathbb{R}^{p \times k}$ is a basis matrix, we have $\mtx{B}\mtx{B}^\top = \mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_1}$, which is the orthogonal projector to the column space of $\mtx{\Sigma}_x^{\half}\bU_1$. Similarly, we have $\widehat{\mtx{B}}\widehat{\mtx{B}}^\top = \mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_2}$, which implies
\[
\mathcal{L}_2(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2)) = \frac{1}{2}\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_1} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_2}\right\|_F^2.
\]
Similarly, we can also define the closeness through the leading principal angle:
\[
\mathcal{L}_1(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2)):=\sin^2 \theta_1 = 1 - \left|\left\langle W_1, \widehat{W}_1\right\rangle\right|^2.
\]
Let $\btheta=(\theta_1, \cdots, \theta_k)$, then $\bB\bB^\top\hbB\hbB^\top=\bB\text{diag}(\cos(\btheta))\hbB^\top$, and by Lemma~\ref{lemma-projection},
%$\bTheta(\bU_1, \bU_2)=\text{diag}(\theta_1, \cdots, \theta_k)\in\mathbb{R}^{k\times k}$,
%then $\bB\bB^\top\hbB\hbB^\top=\bB\cos \left(\bTheta(\bU_1, \bU_2)\right)\hbB^\top$, and by Lemma~\ref{lemma-projection},
\begin{align*}
\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_1} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_2}\right\|^2&=\left\|\bB\bB^\top-\hbB\hbB^\top\right\|^2=1-\sigma_{\min}^2\left(\bB\bB^\top\hbB\hbB^\top\right)\\
&=\sin^2(\theta_1)=\mathcal{L}_1(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2))
\end{align*}
%\[
%\bB\bB^\top\hbB\hbB^\top=\bB\cos (\bTheta)\hbB^\top
%, ~~\hbB\hbB^\top\bB\bB^\top=\hbB\cos (\bTheta)\bB^\top
%\]
%Notice that
%\[
%\bB\bB^\top-\hbB\hbB^\top=\bB\bB^\top(\bI_p-\hbB\hbB^\top)+(\bI_p-\bB\bB^\top)\hbB\hbB^\top
%\]
%and the two matrices on the right hand side has orthogonal row and column space, then
%Then by a similar argument, one can show that
%\[
%\wt{L}(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2))=\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_1} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\bU_2}\right\|^2.
%\]
%{\color{blue} Should we include the following, which is not very relevant?\\
%Should be put in the discussion of related work.\\
%
%Another interesting connection to mention is that the loss in terms of Frobenius norm can be interpreted as the prediction loss between two random vectors $\vct{x}^\top\bU_1$ and $\vct{x}^\top\bU_2$ if we assume $\text{Var}(\vct{x}^\top\bU_2)=\bI_k$ (which is the case for CCA as we will discuss later). This restriction does not lose any generality because the subspace $\lspan(\vct{x}^\top\bU_2)$ is rotation invariant. It is not hard to show
%\begin{gather*}
%L(\lspan(\vct{x}^\top\bU_1), \lspan(\vct{x}^\top\bU_2))=\inf\limits_{\mtx{Q} \in \mathbb{R}^{k \times k}} \E\left\|\vct{x}^\top\bU_1 - \vct{x}^\top\bU_2\mtx{Q}\right\|^2
%\end{gather*}
%where the expectation is over $\bx\sim \P_{\bx}$. Simple algebra yields
%\begin{align*}
% \E\left\|{\bx}^\top \bU_1-{\bx}^\top\bU_2\mtx{Q}\right\|_2^2&=\trace \left((\bU_1-\bU_2\mtx{Q})^\top\mtx{\Sigma}_x(\bU_1- \bU_2\mtx{Q})\right)=\left\|{\sx}^{\half}\bU_1- {\sx}^{\half}\bU_2\mtx{Q}\right\|_F^2.
%\end{align*}
%which is reduced to a least square problem and $\bQ^*=(\bU_2^\top\sx\bU_2)^{-1}\bU_2^\top\sx\bU_1$ achieves the minimum. Substitute $\bQ^*$ into the objective will show that
%\begin{equation*}
%\begin{aligned}
%\inf\limits_{\mtx{Q} \in \mathbb{R}^{k \times k}} \left\|\sx^{\half}\bU_1-\sx^{\half}\bU_2\Q \right\|_F^2 = \frac{1}{2} \left\|\mtx{P}_{\sx^{\half}\bU_1} - \mtx{P}_{\sx^{\half}\bU_2} \right\|_F^2.
%\end{aligned}
%\end{equation*}
%}
We summarize the results into the following theorem.
\begin{theorem}
Suppose $(\bx, z)\sim \P$ for some unknown distribution $\P$ with covariance structure specified in \eqref{eq-cov} and subspace angles defined above. For any reduction matrices $\bU, \os \in\mathbb{R}^{p\times k}$,
\begin{gather*}
loss(z|\lspan(\bx^\top\mtx{U})) - loss(z|\lspan(\bx^\top\os))=\sigma_z^2 \left(\br_{xz}^\top \left(\bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right)\br_{xz}\right)\\
\begin{aligned}
&\sup_{\|\br_{xz}\|^2= R^2} loss(z|\lspan(\bx^\top\mtx{U})) - loss(z|\lspan(\bx^\top\os))\\
& \qquad\qquad= \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\| =\sigma_z^2 R^2 \left\|\sin(\btheta)\right\|_{\infty}.
\end{aligned}
\end{gather*}
By treating $\os$ as an oracle reduction matrix, let $\mc{A}=\{\br: \|\br\|^2=R^2, \bP_{\sx^{\half}\os}\br=\br\}$
\begin{align*}
&\sup\limits_{\br_{xz}\in\mc{A}}\left\{ loss(z|\lspan(\vct{x}^\top\mtx{U})) - loss(z|\lspan(\bx^\top\os))\right\} \\
& \qquad\qquad= \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2=\sigma_z^2 R^2 \left\|\sin(\btheta)\right\|_{\infty}^2.
\end{align*}
By treating $\os$ as a Bayes oracle, that is $\br_{xz}\sim \pi$ where $\pi$ is the uniform measure (Haar measure) on $\mc{A}$, then
\begin{align*}
&\E_{\br_{xz}\sim\pi}\left\{ loss(z|\lspan(\bx^\top\mtx{U})) - loss(z|\lspan(\bx^\top\os))\right\}\\
& \qquad\qquad=\frac{ \sigma_z^2 R^2}{2k} \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|_F^2=\frac{\sigma_z^2 R^2}{k}\left\|\sin(\btheta)\right\|_{2}^2
\end{align*}
\label{thm-metric}
\end{theorem}
%Under a Bayesian setup, assuming $\br_{xz}=\bQ\wt{\br}$ for some $\bQ\in\mathbb{O}(p, k), \wt{\br}\in\mathbb{R}^k\sim \pi$ where $\pi$ is non-informative, supported on the unit ball with $\text{Cov}(\pi)=R^2\bI_k/k$, then
%\begin{align*}
%&\sup\limits_{\bQ}~\E_{\pi}\left\{ loss(z|\lspan(\mtx{U}^\top \vct{x})) - loss(z|\lspan(\os^\top \vct{x}))\right\} = \\
%&= \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|_F^2/k=\left\|\sin\left(\bTheta(\bU, \os)\right)\right\|_{F}^2/k
%\end{align*}
%\label{thm-metric}
%\end{theorem}
\subsection{CCA for Multi-view Dimension Reduction}
\label{section-multi-view}
In the research and applications of multi-media analytics, data of the same object, is collected from multiple sources and exhibit heterogeneous properties. Features obtained from different domains are referred to as different `views'. Usually each view summarizes a specific aspect of the studied object and different views are complementary to one another. For example, in web-page classification, the hyperlink structure and the words on the page are two different views \citep{chaudhuri2009multi}. In video surveillance, images of cameras from different angles constitute different views \citep{loy2009multi}. For more recent results, see the survey paper of \cite{xu2013survey} and references therein.
Although multiple views provide more potential discriminative information to distinguish the patterns of different classes, the feature vector of each view usually lies in a high dimensional space. It is critical, both statistically and computationally, to perform dimension reduction before applying any supervised learning algorithm. It has been shown by many researchers that canonical correlation analysis can achieve sufficient dimension reduction under certain multi-view assumptions \citep{kakade07, foster08, SridharanK08, fukumizu2009kernel, chaudhuri2009multi}.
Suppose the input variable $\bx$ can be split into two views $\bx^{(1)}, \bx^{(2)}$ and the goal is to predict the response $z$ based on the two views. Let $(\pxk, \pyk)$ be the top-$k$ population canonical coefficients between $\bx^{(1)}$ and $\bx^{(2)}$. With notations in this paper, \cite{foster08} essentially proved the following theorem.
%Define
%\[
%\bx=\begin{bmatrix}\bx^{(1)}\\ \bx^{(2)}\end{bmatrix}\in\mathbb{R}^{p_1+p_2}, ~~~\Pi_{CCA}=\begin{bmatrix} \pxk\\ \pyk\end{bmatrix}
%\]
%as the concatenation of the two views and the concatenation of the top $k$ CCA pairs between $\bx_i^{(1)}$ and $\bx_i^{(2)}$ respectively.
%\footnote{\cite{foster08} actually prove Theorem~\ref{thm-foster} under much milder assumptions (gaussianity and conditional independence are not necessary) but we state in this way for simplicity.}
%\begin{assumption}
%There exists a $k$ dimensional hidden state $\bh\sim N(0, \s_{h})$ such that the following conditional distributions are independent.
%\begin{equation}
%\begin{gathered}
%\bx^{(1)} | \bh \sim N(\bW_1^\top \bh, \s_1), ~\bx^{(2)} | \bh \sim N(\bW_2^\top \bh, \s_2), ~z|\bh\sim N(\bw_z^\top\bh, \sigma_z^2)
%\end{gathered}
%\nonumber
%\end{equation}
%\label{assump-gaussian}
%\end{assumption}
%\vspace{-1cm}
\begin{theorem}(Sufficient Multi-view Dimension Reduction by CCA)
%Under Assumption~\ref{assump-independence} and \ref{assump-linear},
%Under assumption~\ref{assump-gaussian},
Under certain multi-view assumptions\footnote{See \cite{foster08} for details},
$$loss(z|\bx^{(1)})=loss(z|\lspan(( \bx^{(1)})^\top\pxk)), ~~loss(z|\bx^{(2)})=loss(z|\lspan((\bx^{(2)})^\top\pyk))$$
%\begin{equation}
%\begin{gathered}
%loss(z|\lspan(\bx^{(1)}))=loss(z|\lspan(\pxk^\top \bx^{(1)})),\\
%loss(z|\lspan(\bx^{(2)}))=loss(z|\lspan(\pyk^\top \bx^{(2)})).
%\end{gathered}
%\nonumber
%\end{equation}
\label{thm-foster}
\end{theorem}
\vspace{-0.6cm}
The theorem shows that all the predictive power of the original high-dimensional predictors $\bx^{(1)}$ and $\bx^{(2)}$ is fully captured by the top $k$ canonical variates. However, the theorem focuses on the population level and does not take into account the estimation error induced by replacing the population canonical coefficients with the sample estimates. Such sample-population discrepancy can be quantified by
\begin{equation}
\begin{gathered}
loss(z|\lspan(( \bx^{(1)})^\top\pxhk)) - loss(z|\lspan(( \bx^{(1)})^\top\pxk)),\\
loss(z|\lspan((\bx^{(2)})^\top\pyhk)) - loss(z|\lspan((\bx^{(2)})^\top\pyk)).
\end{gathered}
\nonumber
\end{equation}
which are exactly characterized by the proposed loss functions according to Theorem~\ref{thm-metric}.
%As discussed earlier, by Theorem~\ref{thm-metric}, in terms of worst case error and Bayes error respectively, it is equivalent to consider the proposed loss functions in \eqref{e}.
%\[
%\left\|\mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|^2~~~\text{and}~~~ \left\| \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{\half}\pxk}\right\|_F^2.
%\]
%which suggests the following recipe for semi-supervised multi-view learning.
%\begin{enumerate}
%\item Compute top-$k$ CCA directions $(\pxhk, \pyhk)$ with the unlabeled data $(\bx_i^{(1)}, \bx_i^{(2)})_{i=(n+1)}^m$
%\item Project the labeled data to the top-$k$ CCA directions and obtain low dimensional predictors $(\pxhk^\top \bx^{(1)}_i, \pyhk^\top \bx^{(2)}_i)_{i=1}^n$
%\item Run supervised learning algorithms on $(\pxhk^\top \bx^{(1)}_i, \pyhk^\top \bx^{(2)}_i, z_i)_{i=1}^n$
%\end{enumerate}
%
%
%with the training data $(\bx_i^{(1)}, \bx_i^{(2)}, z_i)_{i=1}^n$.
%A notable feature for many multi-view learning applications is the availability of an enormous amount of unlabeled data $(\bx_i^{(1)}, \bx_i^{(2)})_{i=(n+1)}^m$, usually $m>> n$. How to utilize the unlabeled data to improve the supervised learning accuracy has been a long standing topic in this area. In recent years, under certain conditions, CCA has been shown to be powerful, both theoretically and empirically, to achieve sufficient dimension reduction by using unlabeled data alone. We first briefly review the results in \cite{foster08}, which relies on the following two hidden state assumption.
%\begin{assumption}
%There exists a $k$ dimensional hidden state $\bh$ such that condition on $\bh$, $\bx_i^{(1)}, \bx_i^{(2)}, z$ are uncorrelated.
%\label{assump-independence}
%\end{assumption}
%\begin{assumption}
%The hidden state $\bh$ is linear in the sense that $\E[\bx^{(1)}|\bh], \E[\bx^{(2)}|\bh], \E[z|\bh]$ are linear functions of $\bh$.
%\label{assump-linear}
%\end{assumption}
%These two assumptions are satisfied under the probabilistic interpretation (generative model) of CCA proposed by \cite{bach05}.
% Define
%\[
%\bx=\begin{bmatrix}\bx^{(1)}\\ \bx^{(2)}\end{bmatrix}\in\mathbb{R}^{p_1+p_2}, ~~~\Pi_{CCA}=\begin{bmatrix} \pxk\\ \pyk\end{bmatrix}
%\]
%as the concatenation of the two views and the concatenation of the top $k$ CCA pairs respectively. Suppose there exists a $k$ dimensional hidden state $\bh\sim N(0, \s_{h})$ such that the conditional distributions
%\begin{equation}
%\begin{gathered}
%\bx^{(1)} | \bh \sim N(\bW_1^\top \bh, \s_1), \bx^{(2)} | \bh \sim N(\bW_2^\top \bh, \s_2), z|\bh\sim N(\bw_z^\top\bh, \sigma_z)
%\end{gathered}
%\end{equation}
%are independent.
%With notations in this paper, \cite{foster08} essentially proved the following theorem.
%\begin{theorem}(Sufficient Dimension Reduction by CCA)
%Under Assumption~\ref{assump-independence} and \ref{assump-linear}, $loss(z|\bx_i^{(1)})=loss(z|\pxk^\top \bx^{(1)})$, $loss(z|\bx^{(2)})=loss(z|\pyk^\top \bx^{(2)})$, and $\,loss(z|\bx)=loss(z|\Pi_{CCA}^\top \bx)$
%\label{thm-foster}
%\end{theorem}
%!TEX root = SpikeCCA.tex
In this section, we introduce our main results on non-asymptotic upper and lower bounds for estimating CCA under the proposed loss functions. Specifically, the upper bound is achieved by sample CCA.
\subsection{Reduction for Sample CCA}
\label{section-reduction}
The linear invariance of both population canonical variates and sample canonical variates enables us to reduce the estimation error of sample CCA to the special case where $\sx=\bI_{p_1}, \sy=\bI_{p_2}$ and $\sxy=\begin{bmatrix} \bLambda ~\vct{0} \end{bmatrix}$ (we assume $p_1\leq p_2$ without loss of generality).
Let $\bX=(\bx_1, \cdots, \bx_n)^\top$ be the data matrix where $\bx_1, \cdots, \bx_n$ are $i.i.d.$ observations from some distribution with covariance matrix $\sx$ and similarly we define $\bY$. It is well known that the sample CCA can be formulated as the solution to the following optimization problem
\begin{equation}
\label{eq:CCA_standard}
\begin{gathered}
(\pxhk, \pyhk)=\arg\max_{\bW_x,\bW_y} \tr(\bW_x^\top \widehat{\mtx{\Sigma}}_{xy}\bW_y)\\
\text{subject to} ~~~ \bW_x^\top \widehat{\mtx{\Sigma}}_x \bW_x=\I_k, ~~\bW_y^\top \widehat{\mtx{\Sigma}}_y \bW_y=\I_k.
\end{gathered}
\end{equation}
where $\sxh, \syh, \sxyh$ are sample covariance matrices defined as
\[
\widehat{\mtx{\Sigma}}_{xy} = \frac{1}{n}\mtx{X}^\top \mtx{Y},~~\widehat{\mtx{\Sigma}}_{x} = \frac{1}{n}\mtx{X}^\top \mtx{X}, ~~\text{~and~}\widehat{\mtx{\Sigma}}_{y} = \frac{1}{n}\mtx{Y}^\top \mtx{Y}.
\]
%Although this is a non-convex optimization, it is well-known that any solution to \prettyref{eq:CCA_standard} can be written as $\pxhk = \widehat{\mtx{\Sigma}}_x^{-\frac{1}{2}}\widehat{\bU}_{1:k}$ and $\pyhk = \widehat{\mtx{\Sigma}}_y^{-\frac{1}{2}}\widehat{\mtx{V}}_{1:k}$, where
%\[
%\widehat{\mtx{\Sigma}}_x^{-\frac{1}{2}} \widehat{\mtx{\Sigma}}_{xy}\widehat{\mtx{\Sigma}}_y^{-\frac{1}{2}} \approx \widehat{\mtx{U}}_{1:k} \widehat{\mtx{\Lambda}}_{1:k} \widehat{\mtx{V}}_{1:k}^\top
%\]
%is the $k$-partial SVD. \\
In the original definition, the loadings $\py$ is a $p_2\times p_1$ matrix. In this subsection, we abuse notation and redefine $\py$ as a $p_2\times p_2$ matrix by adding $p_2-p_1$ columns such that $\sy^{\half}\py\in\mc{O}(p_2)$. Let $\vct{a}_i = \px^\top\vct{x}_i$ and $\vct{b}_i = \py^\top\vct{y}_i$. Notice that $\px^\top\sx\px=\bI_{p_1}$ and $\py^\top\sy\py=\bI_{p_2}$. Then we have $\vct{a}_i \overset{i.i.d}{\sim} \vct{a}$ with distribution $\mathcal{N}(\vct{0}, \mtx{I}_{p_1})$ and $\vct{b}_i \overset{i.i.d}{\sim} \vct{b}$ with distribution $\mathcal{N}(\vct{0}, \mtx{I}_{p_2})$. Moreover, we have
\[
\mtx{\Sigma}_{ab}:= \E \vct{a}_i \vct{b}_i^\top = \px^\top\sxy\py=\px^\top\sx\px\bLambda\py^\top\sy\py= \begin{bmatrix} \bLambda ~\vct{0} \end{bmatrix}.
\]
This implies that
\[
\begin{bmatrix} \vct{a} \\ \vct{b} \end{bmatrix} \overset{i.i.d}{\sim} \mathcal{N}\left(\vct{0}, \begin{bmatrix} \mtx{I}_{p_1} & \mtx{\Sigma}_{ab} \\ \mtx{\Sigma}_{ba} & \mtx{I}_{p_2} \end{bmatrix}\right),
\]
and $(\pxk^a, \pyk^a)=(\bI_{p_1, k}, \bI_{p_2, k})$ is the population top-$k$ CCA pairs for $(\ba, \bb)$. Let $\mtx{A}=(\vct{a}_1, \cdots, \vct{a}_n)^\top = \bX\px $ and $\mtx{B}=(\vct{b}_1, \cdots, \vct{b}_n)^\top = \bY \py$. Then
\begin{align*}
\widehat{\mtx{\Sigma}}_{a} &= \frac{1}{n} \mtx{A}^\top \mtx{A} = \px^\top
\widehat{\mtx{\Sigma}}_{x}\px,
\\
\widehat{\mtx{\Sigma}}_{b} &= \frac{1}{n} \mtx{B}^\top \mtx{B} =\py^\top \widehat{\mtx{\Sigma}}_{y} \py,\\
\widehat{\mtx{\Sigma}}_{ab} & = \frac{1}{n} \mtx{A}^\top \mtx{B} = \px^\top \widehat{\mtx{\Sigma}}_{xy} \py.
\end{align*}
Since $(\pxhk, \pyhk)$ is a solution to the sample CCA \prettyref{eq:CCA_standard}, we know that $(\px^{-1}\pxhk,\py^{-1}\pyhk)$ must be a solution to
\begin{equation*}
\begin{gathered}
\max_{\bW_x,\bW_y} \tr(\bW_x^\top \px^\top \widehat{\mtx{\Sigma}}_{xy} \py\bW_y)\\
\text{subject to} ~~~ \bW_x^\top \px^\top
\widehat{\mtx{\Sigma}}_{x}\px \bW_x=\bI_k, ~~\bW_y^\top \py^\top \widehat{\mtx{\Sigma}}_{y} \py\bW_y=\bI_k,
\end{gathered}
\end{equation*}
which is equivalent to
\begin{equation*}
\begin{gathered}
\max_{\bW_x,\bW_y} \tr(\bW_x^\top \widehat{\mtx{\Sigma}}_{ab} \bW_y)\\
\text{subject to} ~~~\bW_x^\top \widehat{\mtx{\Sigma}}_{a} \bW_x=\bI_k, ~~\bW_y^\top \widehat{\mtx{\Sigma}}_{b} \bW_y=\bI_k.
\end{gathered}
\end{equation*}
Therefore, $(\px^{-1}\pxhk,\py^{-1}\pyhk)$ is indeed the sample canonical vectors for $(\vct{a}, \vct{b})$, which we denote by $(\pxhk^a, \pyhk^b)$. By the discussion in Section~\ref{section-angle},
\begin{align*}
\left\|\mtx{P}_{\sx^{\half}\pxk} - \mtx{P}_{\sx^{\half}\pxhk} \right\|^2&=\mc{L}_1(\lspan(\vct{x}^\top\pxhk), \lspan(\vct{x}^\top\pxk))\\
&=\mc{L}_1(\lspan(\vct{a}^\top\pxhk^a), \lspan(\vct{a}^\top\pxk^a))\\
&=\left\|\mtx{P}_{\pxk^a} - \mtx{P}_{\pxhk^a} \right\|^2
\end{align*}
By the same argument,
\begin{align*}
\left\|\mtx{P}_{\sx^{\half}\pxk} - \mtx{P}_{\sx^{\half}\pxhk} \right\|_F^2=\left\|\mtx{P}_{\pxk^a} - \mtx{P}_{\pxhk^a} \right\|_F^2
\end{align*}
To sum up, it suffices to consider the special covariance structure $\sx=\bI_{p_1}, \sy=\bI_{p_2}, \sxy=\begin{bmatrix} \bLambda ~\vct{0} \end{bmatrix}$ for the estimation error of sample CCA.
\begin{remark}
The reduction argument essentially reveals the fact that the estimation error of sample CCA is independent of the condition numbers of the covariance matrices: $\kappa(\sx)$ and $\kappa(\sy)$. It is worthwhile to mention that such phenomenon is due to the linear invariance of sample canonical variates and might not exist for other estimators.
\end{remark}
%{\color{blue} original version: \\
%Notice that $\sx^{\half}\px\in\mathbb{O}(p_1, p_1)$, then for any rotation invariant norm $\|\cdot\|$ (e.g. spectral norm and Frobenius norm),
%\begin{align*}
% \left\|\mtx{P}_{\sx^{\half}\pxk} - \mtx{P}_{\sx^{\half}\pxhk} \right\|& = \left\|\mtx{P}_{\px^\top\sx\pxk} - \mtx{P}_{\px^\top\sx\pxhk} \right\|\\
% &=\left\|\mtx{P}_{\bI_{p_1,k}} - \mtx{P}_{\px^{-1}\pxhk} \right\|\\
% &=\left\|\mtx{P}_{\pxk^a} - \mtx{P}_{\pxhk^a} \right\|
%\end{align*}
%}
% under the loss function
%\[
% \left\|\mtx{P}_{\sx^{\half}\pxk} - \mtx{P}_{\sx^{\half}\pxhk} \right\|
% \]
\subsection{Upper and Lower Bounds}
\label{section-minimax-rates}
In this section, we assume $\bx\in\mathbb{R}^{p_1},\by\in\mathbb{R}^{p_2}$ are jointly normal with mean zero and joint covariance matrix $\s$ specified by
%(without loss of generality, we assume $p_1\leq p_2$ throughout this section)
\begin{align*}
\s= \begin{bmatrix} \mtx{\Sigma}_x & \mtx{\Sigma}_{xy} \\ \mtx{\Sigma}_{xy}^\top & \mtx{\Sigma}_y \end{bmatrix}
\end{align*}
%\[
%\begin{bmatrix}\vct{x} \\ \vct{y} \end{bmatrix} \sim \mathcal{N}\left(\vct{0}, \s=\begin{bmatrix} \mtx{\Sigma}_x & \mtx{\Sigma}_{xy} \\ \mtx{\Sigma}_{xy}^\top & \mtx{\Sigma}_y \end{bmatrix}\right).
%\]
where $\sx$ and $\sy$ are nonsingular. Let $\lambda_1, \cdots, \lambda_{p_1\wedge p_2}$ be the canonical correlations and $(\px, \py)$ be the canonical coefficient matrices (loadings) as defined in \eqref{eq-def-cca}. For any $1\leq k<p_1\wedge p_2$, define the $k_{th}$ eigen-gap as $\Delta=\lambda_k-\lambda_{k+1}$.
%and the top-$k$ minimum eigen-gap $\Delta_{\min, k}=\min\limits_{1\leq i\leq k, ~\lambda_i\neq \lambda_{i+1}}\lambda_i-\lambda_{i+1}$.
\begin{theorem}(Upper bound)
\label{thm:upper}
There exists universal positive constants $C, C_1, C_2$ independent of $n, p_1, p_2$ and $\s$ such that if $n\geq C(p_1+p_2)$, the sample CCA coefficients $\pxhk$ satisfies
\begin{gather*}
\E\left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|^2\right] \leq C_1\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2}\frac{p_1}{n}+C_2\left(\frac{p_1+p_2}{n\Delta^2}\right)^2\\
\E\left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|_F^2/k \right] \leq C_1\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2}\frac{p_1-k}{n}+C_2\left(\frac{p_1+p_2}{n\Delta^2}\right)^2
\end{gather*}
Symmetric upper bounds hold for $\pyhk$ by switching $p_1$ and $p_2$.
\end{theorem}
This theorem exhibits several notable features:
\begin{enumerate}
%\item No artificial assumptions are imposed on $\sx, \sy$ and $\sxy$. This is in contrast with the classical asymptotic theory of \cite{anderson1999asymptotic}, where the top-$k$ canonical correlations are required to be different because their argument is based on the asymptotic convergence of each individual canonical coefficient.
%Further, when loadings are concerned, the condition number of
%For sparse CCA of \cite{gao2014sparse, gao2015minimax}, $\lambda$
%most of the existing loss functions in the literature directly focus on the loadings with error bound inevitably proportional to the condition numbers.
% geometric property of the proposed loss function This result reflects the linear invariant nature of CCA variates which is the property that distinguishes CCA from other dimension reduction methods and in return justifies the proposed loss functions.
\item The multiplicative factor $(1-\lambda_k^2)(1-\lambda_{k+1}^2)/\Delta^2$ appears in the principal term. The inverse dependence on the eigen-gap $\Delta^2$ is inherent for spectral estimations. The factor $(1-\lambda_k^2)$ reveals that the estimation error decreases with increasing correlations between the two sets of random variables. When there is perfect correlation, that is $\lambda_k=1$, we can recover the CCA directions errorlessly because the observed data along those directions are perfectly co-linear. The factor $(1-\lambda_{k+1}^2)$ comes as surprise and appears in our lower bound as well. When $\lambda_k$ is close to 1 and the eigen-gap $\Delta$ is close to 0, for example, in the regime $\lambda_k=1-C\Delta$ and $\Delta\rightarrow 0$, then
\begin{gather*}
(1-\lambda_k^2)(1-\lambda_{k+1}^2)/\Delta^2\asymp \text{ constant}
\end{gather*}
which shows consistency can still be achieved. On the contrast, without the factor $(1-\lambda_{k+1}^2)$, the error will explode since $(1-\lambda_k^2)/\Delta^2\asymp 1/\Delta$. We remark that $\lambda_k, \lambda_{k+1}$ are parameters depending on the dimensions and should not be treated as constant. As the number of variables increases, one should expect the canonical correlations to increase as well.
To the best of our knowledge, this is the first time for $(1-\lambda_k^2)$ and $(1-\lambda_{k+1}^2)$ to be obtained as a finite sample result. This is achieved by Taylor expanding the estimating equations for $\pxhk$ and $\pyhk$, inspired by the classical multivariate theory of \cite{anderson1963asymptotic, tAND84a, anderson1999asymptotic}, while the analysis of \cite{gao2014sparse, gao2015minimax} and the recent work by \cite{cai2016rate} does not yield this factor.
% while a direct application of Wedin's $\sin\theta$ law on $\sxh^{-\half}\sxyh\syh^{-\half}$ does not suffice as in \cite{gao2014sparse, gao2015minimax} who considered sparse CCA in a high dimensional setup.
%Current analysis in the literature based on Wedin's $\sin\theta$ law \cite{gao2014sparse, gao2015minimax}fail to
%A direct application of Wedin's $\sin\theta$ law on $\sxh^{-\half}\sxyh\syh^{-\half}$ will not suffice as in \cite{gao2014sparse, gao2015minimax} who considered sparse CCA in a more sophisticated high dimensional setup.
\item The dimension parameter $p_2$ only appears in the high order term, which implies that even in the regime $p_2>>p_1$, as long as the sample size is large enough (see Corollary~\ref{thm::samplesize}), the estimation error of $\pxk$ does not depend on $p_2$. This phenomenon was first revealed in \cite{gao2015minimax} through multi-stage estimation and sample splitting.
%(they dealt with the sparse case but the analysis also applies without sparsity assumptions).
The recent work of \cite{cai2016rate} directly proved such kind of result for sample CCA without splitting the samples. The results of both \cite{gao2015minimax} and \cite{cai2016rate} are based on the extra assumption that all the residual canonical correlations are zero: $\lambda_{k+1}=\cdots=\lambda_{p_1}=0$ (or equivalently, the rank of $\sxy$ is $k$).
%The sample size condition for the higher order term in the theorem to be dominated is
%\begin{gather*}
%\frac{p_1+p_2}{n\Delta^2}<c\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{1+p_2/p_1}
%\end{gather*}
%for some constant $c$, while consistency only requires the left hand side converging to zero. Under this sample side condition, the upper bound will match the lower bound in Theorem~\ref{thm:upper}.
\item The upper bound does not depend on the condition number of the covariance matrices: $\kappa(\sx)$ and $\kappa(\sy)$. It is directly implied by the reduction argument but not obvious because the separate estimation errors, $\|\sx-\sxh\|, \|\sy-\syh\|, \|\sxy-\sxyh\|$, are in fact proportional to these condition numbers. The success of the reduction argument relies on the linear invariance of both population and sample canonical variates. For loss functions directly based on the loadings \citep{chen2013sparse, gao2014sparse}, the upper bounds will be proportional to these condition numbers.
\item The only assumption made in Theorem~\ref{thm:upper} is that $\sx, \sy$ are invertible. \cite{anderson1999asymptotic} assumes the canonical correlations are all distinct because the argument requires the asymptotic convergence of each individual canonical coefficient. Moreover, the result is asymptotic without finite sample guarantee. \cite{gao2015minimax, cai2016rate} assume $\lambda_{k+1}=\cdots=\lambda_{p_1}=0$ and \cite{gao2014sparse, gao2015minimax} assume the condition number $\kappa(\sx)$ and $\kappa(\sy)$ are bounded.
%\item The only assumption made in Theorem~\ref{thm:upper} is that $\sx, \sy$ are invertible.
%\item Similar phenomena were observed in \cite{anderson1999asymptotic} when studying the asymptotic distribution of the loadings. They need to assume the canonical correlations are different because their argument is based on the asymptotic convergence of each individual canonical coefficient.
%Our results are not implied by \cite{anderson1963asymptotic, anderson1999asymptotic} neither who focus on the asymptotic distribution of the loadings while we pursue a non-asymptotic bound for the subspace distance.
%\item The estimation error in terms of operator norm, motivated naturally from prediction loss, is to the best of our knowledge, the first time being considered in the literature. The analysis is more challenging than the Frobenius norm which requires careful covering arguments and dealing with the operator norm of the Hadamard product of two matrices.
\end{enumerate}
To establish the minimax lower bound, we define the parameter space $\mathcal{F}(p_1, p_2, k, \lambda_{k}, \lambda_{k+1}, \kappa_1, \kappa_2)$ as the collection of joint covariance matrices $\s$ satisfying
\[
\sx, \sy \text{~are nonsingular}, ~ \kappa(\sx)=\kappa_1, \kappa(\sy)=\kappa_2
\]
\[
0\leq \lambda_{p_1\wedge p_2}\leq \cdots\leq \lambda_{k+1}<\lambda_k\leq \cdots\leq \lambda_1\leq 1
\]
%where $\lambda_1, \cdots, \lambda_{p_1\wedge p_2}$ are the canonical correlations.
%The notion $\lambda_{up}, \lambda_{low}$ will play the role of $\lambda_k$ and $\lambda_{k+1}$ respectively.
%where $\lambda_{up}-\lambda_{low}$ quantifies the separation of the canonical correlations.
We deliberately specify $\kappa(\sx)=\kappa_1, \kappa(\sy)=\kappa_2$ to show the lower bound is independent of the condition number as well. For the rest of the paper, we will use the shorthand $\mathcal{F}$ to represent this parameter space for simplicity.
%$\mathcal{F}(p_1, p_2, k, \lambda_{k}, \lambda_{k+1}, \kappa_1, \kappa_2)$ for simplicity.
\begin{theorem}(Lower bound)
\label{thm:lower}
There exists a universal constant $c$ independent of $n, p_1, p_2$ and $\s$ such that
\begin{gather*}
\inf\limits_{\pxhk}\sup_{\s\in \mathcal{F}} \E\left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|^2\right]\geq c^2\left\{\left(\frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{\Delta^2}\frac{p_1-k}{n}\right)\wedge 1\wedge \frac{p_1-k}{k}\right\}\\
\inf\limits_{\pxhk}\sup_{\s\in \mathcal{F}} \E\left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|_F^2/k\right]\geq c^2\left\{\left(\frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{\Delta^2}\frac{p_1-k}{n}\right)\wedge 1\wedge \frac{p_1-k}{k}\right\}
\end{gather*}
where $\Delta=\lambda_{k}-\lambda_{k+1}$. The lower bounds for $\pyhk$ can be obtained by switching $p_1$ and $p_2$.
\end{theorem}
The proof of the lower bound can be found in Section~\ref{sec::lower-proof}, which relies on Lemma~\ref{KL-distance} and constructing a hypothesis class based on this lemma.
\begin{remark}
The upper and lower bound together shows that the condition number of $\sx$ and $\sy$ is neither cursing nor blessing.
% This should be anticipated considering the linear invariant nature of both the canonical variates and the proposed loss functions.
\end{remark}
\cite{gao2015minimax} obtained minimax lower bounds for sparse CCA in high dimensional regime. Rephrasing their results without sparsity assumptions, they essentially proved
%the following minimax lower bound for the projection matrix of the loadings under the Frobenius norm error,
\[
\E\left\{\inf\limits_{\bQ\in\mc{O}(p_1)}\E \left[\left\| \bx^\top\pxk-\bx^\top\pxhk\bQ \right\|_F^2/k\right]\right\} \geq c^2\left\{ \left(\frac{1-\lambda_k^2}{\kappa_1\lambda_k^2}\frac{p_1-k}{n}\right)\wedge 1\wedge \frac{p_1-k}{k}\right\},
\]
with the parameter space $\lambda_{p_1\wedge p_2}= \cdots= \lambda_{k+1}=0< \lambda_k\leq \cdots\leq \lambda_1\leq 1$. The inner expectation is with respect to an independent sample $\bx$ and the outer expectation is with respect to the data from which $\pxhk$ is constructed.
%Their proof is inverse dependence on the condition number is inevitable when projection matrix of the loadings is concerned.
The inverse dependency on the condition number in their results can be removed by noticing that
\[
\inf\limits_{\bQ\in\mc{O}(p_1)}\E \left[\left\| \bx^\top\pxk-\bx^\top\pxhk\bQ \right\|_F^2 \right]\geq \frac{1}{2}\left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|_F^2.
\]
and apply Theorem~\ref{thm:lower} with $\lambda_{k+1}=0$.
\begin{corollary}
\label{thm::samplesize}
When $p_1\geq 2k$, there exists a universal positive constant $c$ such that when
\begin{equation}
\frac{p_1+p_2}{n\Delta^2}\leq c\frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{(1+p_2/p_1)},
\label{eq:sample-size}
\end{equation}
the minimax rates are characterized by
\begin{gather*}
\inf\limits_{\pxhk}\sup_{\s\in \mathcal{F}} \E\left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|^2\right] \asymp \frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{n\Delta^2}p_1,\\
\inf\limits_{\pxhk}\sup_{\s\in \mathcal{F}} \E\left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|_F^2/k\right] \asymp \frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{n\Delta^2}p_1.
\end{gather*}
\end{corollary}
\begin{remark}
For consistency, we only need the left hand side of \eqref{eq:sample-size} to converge to zero while in order for the high order term to be dominated by the principal term, the left hand side of \eqref{eq:sample-size} is required to converge to zero faster than the right hand size.
\end{remark}
%\subsection{Connection with Other Loss Functions}
%Their results can be obtained from our lower bound by setting $\lambda_{k+1}=0$ and noticing that
%\[
%\left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|_F^2\geq \frac{1}{\kappa_1}\left\| \bP_{\pxhk}-\bP_{\pxk} \right\|_F^2
%\]
%Adding an eigen-gap $\Delta$ to the parameter space complicates the lower bound proof. The hypothesis class we construct is different from that of \cite{gao2014sparse, gao2015minimax}.
%!TEX root = SpikeCCA.tex
Throughout the proof, we assume $p_1=p_2$ for the ease of presentation and the same argument works for any $p_1$ and $p_2$ at the cost of heavier notations (when $p_1\neq p_2$, in the definition \eqref{eq:p1p2}, $\bLambda_2, \hbLambda_2$ will be rectangular instead of square matrices. As a result, the subsequent places where $\bLambda_2$ appears in the current proof should be understood as either $\bLambda_2$ or $\bLambda_2^\top$ according to the dimensions in the specific context).
We will still use $p_1, p_2~(p_1\leq p_2)$ to denote the dimension of $\bx$ and $\by$ separately such that the results will be interpretable when $p_1\neq p_2$.
% since one of the interesting cases is that $p_1$ and $p_2$ are not of the same order as indicated by our upper and lower bounds in Section~\ref{section-minimax-rates}.
% We will mention at the end how the notations in the proof need to be changed (slightly but not friendly) for the case: $p_1\neq p_2$.
%Without loss of generality, we assume $p_1\leq p_2$.
By the reduction argument in Section~\ref{section-reduction}, it suffices to consider
\[
\sx=\bI_{p_1}, ~\sy=\bI_{p_2}, ~\sxy=\mtx{\Lambda}
\]
where $\mtx{\Lambda}=\text{Diag}(\lambda_1, \lambda_2, \ldots, \lambda_{p_1})\in\mathbb{R}^{p_1\times p_1}$ is the diagonal matrix with $1\geq \lambda_1 \geq \ldots \geq \lambda_{p_1}\geq 0$.
%, and $\bm{0}\in\mathbb{R}^{p_1\times {(p_2-p_1)}}$ is a $p_1\times (p_2-p_1)$ matrix of zeros.
Under this setup
\[
\px_{1:p_1}=\bI_{p_1}, ~\py_{1:p_1}=\bI_{p_2, p_1}.
\]
and $\lambda_1, \lambda_2, \cdots, \lambda_{p_1}$ are the canonical correlations. Then the error metric is reduced to
\[
\norm{\mtx{P}_{\mtx{\Sigma}_x^{1/2}\pxhk} - \mtx{P}_{\mtx{\Sigma}_x^{1/2}\pxk}}=\norm{\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}}
\]
where $\norm{\cdot}$ denotes either operator or Frobenius norm. Divide $\pxhk$ into two blocks such that
$\pxhk=\begin{bmatrix}
\pxhk^u\\ \pxhk^l \end{bmatrix}$ where $\pxhk^u$ and $\pxhk^l$ are the upper $k\times k$ and lower $(p_1-k)\times k$ sub-matrices of $\pxhk$ respectively. Let $\bU=\begin{bmatrix}
\pxhk^u\\ \bm{0}
\end{bmatrix}\in\mathbb{R}^{p_1\times k}$. Then $\mtx{P}_{\pxk}= \mtx{P}_{\bU}$ and
by Wedin's $\sin\theta$ law \citep{wedin1972perturbation}, there exists universal constant $C$ such that
\begin{equation}
\norm{\mtx{P}_{\pxhk} - \mtx{P}_{\bU}}^2\leq \frac{C\norm{\pxhk-\bU}^2}{\left(\sigma_k(\pxhk)-\sigma_{k+1}(\bU)\right)^2}=\frac{C\norm{\pxhk^l}^2}{\sigma_k^2(\pxhk)}
\label{eq-target}
\end{equation}
The denominator in \eqref{eq-target} is close to 1 with high probability when $n\geq C(p_1+p_2)$ for some constant $C$. The remaining proof will focus on obtaining upper bounds for the numerator. The upper bound in terms of operator norm is involved which we will present detailed proof. The Frobenius norm bound can be obtained in a very similar (but simpler) manner of which the proof is only sketched. The proof mainly contains the following steps:
%We begin will the following outline for proving the operator norm bound.
% Moreover, deriving the operator norm bound is far more challenging than the Frobenius norm bound. We will present detailed proof for operator norm while only sketch the proof for Frobenius norm. We begin will the following outline to prove the operator norm bound.
\begin{enumerate}
\item Express explicitly (principal term + high order term) each cell of the matrix $\pxhk^l$ by Taylor expanding the estimating equations of sample CCA.
\item Derive two separate deterministic upper bounds for the principal part of $\|\pxhk^l\|$. One of the bounds is tight when $\lambda_{k+1}$ is bounded away from 1 and the other one is tight when $\lambda_{k+1}$ is bounded away from 0.
%\item Derive tight deterministic bound for $\|\pxhk^l\|$ when $\lambda_{k+1}$ is bounded away from 1 (without the multiplicative factor $1-\lambda_{k+1}^2$).
%\item Derive finer deterministic bound for $\|\pxhk^l\|$ when $\lambda_{k+1}$ is large (with the multiplicative factor $1-\lambda_{k+1}^2$).
\item Derive deterministic bound for the high order terms.
\item Put pieces together and obtain the desired upper bounds in Theorem~\ref{thm:upper}.
\end{enumerate}
Throughout the proof, the constants $c, C, C_1, \cdots$ might change from line to line.
\subsection{Taylor Expansion for $\pxhk^l $}
\label{analytic-expression}
Recall that $\pxh\in\mathbb{R}^{p_1\times p_1}, \pyh\in\mathbb{R}^{p_2\times p_1}$ are the sample canonical coefficients. By definition, the sample canonical coefficients can be written as the solution to the following two estimating equations,
\begin{align}
\sxyh\pyh=\sxh\pxh\hbLambda \label{eq:pillar1}
\\
\syxh\pxh = \syh \pyh \hbLambda \label{eq:pillar2}
\end{align}
Dive the matrices into blocks,
\begin{align*}
\sxh=\begin{bmatrix}
\sxh^{11} &\sxh^{12}\\
\sxh^{21}&\sxh^{22}
\end{bmatrix}, ~
\syh=\begin{bmatrix}
\syh^{11} &\syh^{12}\\
\syh^{21}&\syh^{22}
\end{bmatrix}, ~
\sxyh=\begin{bmatrix}
\sxyh^{11} &\sxyh^{12}\\
\sxyh^{21}&\sxyh^{22}
\end{bmatrix},~
\syxh=\begin{bmatrix}
\syxh^{11} &\syxh^{12}\\
\syxh^{21}&\syxh^{22}
\end{bmatrix}
\end{align*}
where $\sxh^{11}, \syh^{11}, \sxyh^{11}, \syxh^{11}$ are $k\times k$ matrices. Similarly, we define
\begin{align}
\bLambda=\begin{bmatrix}
\bLambda_1 & \\
& \bLambda_2
\end{bmatrix},~~
\hbLambda=\begin{bmatrix}
\hbLambda_1 & \\
& \hbLambda_2
\end{bmatrix},
\label{eq:p1p2}
\end{align}
where $\bLambda_1,\hbLambda_1$ are also $k\times k$ matrices. Finally, we define $\pyhk^u\in\mathbb{R}^{k\times k}, \pyhk^l\in\mathbb{R}^{(p_2-k)\times k}$ in the same way as $\pxhk^u, \pxhk^l$. With these notations, we can write down the lower left $(p_1-k)\times k$ sub-matrix of \eqref{eq:pillar1} and \eqref{eq:pillar2} explicitly as
\begin{align}
\sxyh^{21}\pyhk^u+\sxyh^{22}\pyhk^l=\sxh^{21}\pxhk^u\hbLambda_1+\sxh^{22}\pxhk^l \hbLambda_1, \label{eq:pillar3}\\
\syxh^{21}\pxhk^u+\syxh^{22}\pxhk^l=\syh^{21}\pyhk^u\hbLambda_1+\syh^{22}\pyhk^l \hbLambda_1. \label{eq:pillar4}
\end{align}
Similarly, the upper left $k\times k$ sub-matrix of \eqref{eq:pillar1} and \eqref{eq:pillar2} can be written explicitly as
\begin{align}
\sxyh^{11}\pyhk^u+\sxyh^{12}\pyhk^l=\sxh^{11}\pxhk^u\hbLambda_1+\sxh^{12}\pxhk^l \hbLambda_1, \label{eq:pillar5}\\
\syxh^{11}\pxhk^u+\syxh^{12}\pxhk^l=\syh^{11}\pyhk^u\hbLambda_1+\syh^{12}\pyhk^l \hbLambda_1. \label{eq:pillar6}
\end{align}
Manipulate the terms in \eqref{eq:pillar5},
\begin{align*}
\bLambda_1\pyhk^u+(\sxyh^{11}-\bLambda_1)\pyhk^u+\sxyh^{12}\pyhk^l = \pxhk^u\hbLambda_1+(\sxh^{11}-\bI_k)\pxhk^u\hbLambda_1+\sxh^{12}\pxhk^l \hbLambda_1.
\end{align*}
Therefore,
\begin{align}
\bLambda_1\pyhk^u- \pxhk^u\bLambda_1&= \pxhk^u(\hbLambda_1-\bLambda_1)+(\sxh^{11}-\bI_k)\pxhk^u\hbLambda_1+\sxh^{12}\pxhk^l \hbLambda_1\nonumber\\
&~~~-(\sxyh^{11}-\bLambda_1)\pyhk^u-\sxyh^{12}\pyhk^l:=\bdelta_1. \label{eq:delta1}
\end{align}
The equation essentially implies $\bLambda_1\pyhk^u\approx \pxhk^u\bLambda_1$ because $\bdelta_1$ will be proved to be a higher order term. Apply the same argument to \eqref{eq:pillar6}, and we will obtain
\begin{align}
\bLambda_1\pxhk^u- \pyhk^u\bLambda_1&= \pyhk^u(\hbLambda_1-\bLambda_1)+(\syh^{11}-\bI_k)\pyhk^u\hbLambda_1+\syh^{12}\pyhk^l \hbLambda_1 \nonumber\\
&~~~-(\syxh^{11}-\bLambda_1)\pxhk^u-\syxh^{12}\pxhk^l:=\bdelta_2 .\label{eq:delta2}
\end{align}
Similarly, massage the terms in \eqref{eq:pillar3},
\begin{align*}
\sxyh^{21}\pyhk^u+\bLambda_2\pyhk^l+(\sxyh^{22}-\bLambda_2)\pyhk^l&=\sxh^{21}\pxhk^u\bLambda_1+\sxh^{21}\pxhk^u(\hbLambda_1-\bLambda_1)\\
&+\pxhk^l \bLambda_1 +(\sxh^{22}\pxhk^l \hbLambda_1-\pxhk^l \bLambda_1),
\end{align*}
which can be equivalently written as
\begin{align}
&\sxyh^{21}\pyhk^u+\bLambda_2\pyhk^l-\sxh^{21}\pxhk^u\bLambda_1-\pxhk^l \bLambda_1 \nonumber\\
&~~ =\sxh^{21}\pxhk^u(\hbLambda_1-\bLambda_1)+(\sxh^{22}\pxhk^l \hbLambda_1-\pxhk^l \bLambda_1)-(\sxyh^{22}-\bLambda_2)\pyhk^l:=\bdelta_3. \label{eq:pillar7}
\end{align}
Apply the same argument to \eqref{eq:pillar4},
\begin{align}
&\syxh^{21}\pxhk^u+\bLambda_2\pxhk^l-\syh^{21}\pyhk^u\bLambda_1-\pyhk^l \bLambda_1 \nonumber\\
&~~ =\syh^{21}\pyhk^u(\hbLambda_1-\bLambda_1)+(\syh^{22}\pyhk^l \hbLambda_1-\pyhk^l \bLambda_1)-(\syxh^{22}-\bLambda_2)\pxhk^l:=\bdelta_4. \label{eq:pillar8}
\end{align}
Consider $\eqref{eq:pillar7}\times (-\bLambda_1)-\bLambda_2\times \eqref{eq:pillar8}$, then
\begin{align*}
\pxhk^l \bLambda_1^2-\bLambda_2^2\pxhk^l +\sxh^{21}\pxhk^u\bLambda_1^2&-\sxyh^{21}\pyhk^u\bLambda_1-\bLambda_2\syxh^{21}\pxhk^u+\bLambda_2\syh^{21}\pyhk^u\bLambda_1\\
&=-(\bdelta_3\bLambda_1+\bLambda_2\bdelta_4):=\bdelta_5,
\end{align*}
that is
\begin{align}
\pxhk^l \bLambda_1^2-\bLambda_2^2\pxhk^l =\sxyh^{21}\pyhk^u\bLambda_1+\bLambda_2\syxh^{21}\pxhk^u-\sxh^{21}\pxhk^u\bLambda_1^2-\bLambda_2\syh^{21}\pyhk^u\bLambda_1+\bdelta_5
\label{eq-upper-target}
\end{align}
The equation above indicates that, ignoring $\pxhk^u, \pyhk^u$ and the high order term $\bdelta_5$, the target is expressed as a linear function of the sample covariance matrices. Later we will show that the sample covariance matrices and $\pxhk^u, \pyhk^u$ can be decoupled.
%Notice that $\bLambda_1\times\eqref{eq:delta1}+\bLambda_1\times \eqref{eq:delta2}$ will yield
%\begin{align}
%\bLambda_1^2\pyhk^u-\pyhk^u \bLambda_1^2=\bLambda_1\bdelta_1+\bdelta_2\bLambda_1
%\label{eq-commute}
%\end{align}
\subsection{Upper Bounds for $\|\pxhk^l\|$}
The proof in this part heavily replies on the following lemma about the Hadamard operator norm for some structured matrices.
\begin{lemma}(Hadamard Operator Norm)
For $\bA\in\mathbb{R}^{m\times n}$, define the Hadamard operator norm as
\[
\norm{\bA}=\sup\left\{\|\bA\circ \bB\|: \|\bB\|\leq 1, \bB\in\mathbb{R}^{m\times n}\right\}
\]
Let $\alpha_1, \cdots, \alpha_m$ and $\beta_1, \cdots, \beta_n$ be arbitrary positive numbers lower bounded by a positive constant $\delta$. Define $\bA_1, \bA_2, \bA_3\in\mathbb{R}^{m\times n}$ by
\[
[\bA_1]_{ij}=\frac{1}{\alpha_i+\beta_j}, ~[\bA_2]_{ij}=\frac{\min\{\alpha_i, \beta_j\}}{\alpha_i+\beta_j}, ~[\bA_3]_{ij}=\frac{\max\{\alpha_i, \beta_j\}}{\alpha_i+\beta_j}
\]
Then
\[
\norm{\bA_1}\leq \frac{1}{2\delta},~ \norm{\bA_2}\leq \frac{1}{2}, ~\norm{\bA_3}\leq \frac{3}{2}
\]
\label{lemma-hada-norm}
\end{lemma}
See Section~\ref{sec::hada-norm} for the proof of this lemma.
%\subsubsection{Case 1: upper bound without the factor \texorpdfstring{$1-\lambda_{k+1}^2$}{TEXT}}
\subsubsection{Upper Bound I: tight for \texorpdfstring{$\lambda_{k+1}\leq 1/2$}{TEXT}}
%The upper bound derived in this section is universally true but only tight when $\lambda_{k+1}$ is bounded away from 1.
Multiply both sides of \eqref{eq:delta1} by $\bLambda_1$ on the right will yield
\begin{align}
\bLambda_1\pyhk^u \bLambda_1- \pxhk^u\bLambda_1^2=\bdelta_1\bLambda_1.
\label{eq:commute2}
\end{align}
Substitute \eqref{eq:delta1}, \eqref{eq:delta2} and \eqref{eq:commute2} into \eqref{eq-upper-target},
\begin{align}
\pxhk^l \bLambda_1^2-\bLambda_2^2\pxhk^l &=\sxyh^{21}\pyhk^u\bLambda_1+\bLambda_2\syxh^{21}\pxhk^u-\sxh^{21}\bLambda_1\pyhk^u\bLambda_1+\sxh^{21}\bdelta_1\bLambda_1\nonumber\\
&~~~ -\bLambda_2\syh^{21}\bLambda_1\pxhk^u-\bLambda_2\syh^{21}\bdelta_1+\bdelta_5 \nonumber\\
&=(\sxyh^{21}-\sxh^{21}\bLambda_1)\pyhk^u\bLambda_1+\bLambda_2(\syxh^{21}-\syh^{21}\bLambda_1)\pxhk^u \nonumber\\
&~~~+\sxh^{21}\bdelta_1\bLambda_1-\bLambda_2\syh^{21}\bdelta_1+\bdelta_5 \nonumber\\
&:=\bB_1\pyhk^u\bLambda_1+\bLambda_2\bB_2\pxhk^u+\bdelta_6.
\label{eq:target:upper1}
\end{align}
where $\bB_1=\sxyh^{21}-\sxh^{21}\bLambda_1$, $\bB_2=\syxh^{21}-\syh^{21}\bLambda_1$ and
\begin{align*}
\begin{gathered}
\bdelta_6=\sxh^{21}\bdelta_1\bLambda_1-\bLambda_2\syh^{21}\bdelta_1+\bdelta_5.
\end{gathered}
\end{align*}
Further, define the matrix $(p_1-k)\times k$ matrices $\bA_1, \bA_2$ by
\[
[\bA_1]_{ij}=\frac{1}{\lambda_j+\lambda_{k+i}}, ~[\bA_2]_{ij}=\frac{1}{\lambda_j-\lambda_{k+i}},~ 1\leq i\leq p_1-k, 1\leq j\leq k
\]
Then we can rewrite \eqref{eq:target:upper1} as
\begin{align*}
\pxhk^l &= \bA_1\circ \bA_2 \circ (\bB_1\pyhk^u\bLambda_1)+\bA_1\circ \bA_2 \circ (\bLambda_2\bB_2\pxhk^u)+\bA_1\circ \bA_2 \circ \bdelta_6\\
&=(\bA_1 \bLambda_1)\circ \bA_2 \circ (\bB_1\pyhk^u)+(\bLambda_2 \bA_1)\circ \bA_2 \circ (\bB_2\pxhk^u)+\bA_1\circ \bA_2 \circ \bdelta_6,
\end{align*}
where $\circ$ denotes the matrix Hadamard (element-wise) product. Define $\alpha_j=\lambda_j, 1\leq j\leq k$ and $\beta_i=\lambda_{k+i}, 1\leq i\leq p_1-k$, then
\begin{align}
[\bA_1\bLambda_1]_{ij}=\frac{\lambda_j}{\lambda_j+\lambda_{k+i}}=\frac{\max\{\alpha_j, \beta_i\}}{\alpha_j+ \beta_i}, ~[\bLambda_2\bA_1]_{ij}=\frac{\lambda_{k+i}}{\lambda_j+\lambda_{k+i}}=\frac{\min\{\alpha_j, \beta_i\}}{\alpha_j+ \beta_i}
\label{eq:A1-hada}
\end{align}
Therefore, by Lemma~\ref{lemma-hada-norm},
\begin{align*}
\begin{gathered}
\|(\bA_1 \bLambda_1)\circ \bA_2 \circ (\bB_1\pyhk^u)\|\leq \frac{3}{2} \| \bA_2 \circ (\bB_1\pyhk^u)\|, \\
\|(\bLambda_2 \bA_1)\circ \bA_2 \circ (\bB_2\pxhk^u)\|\leq \frac{1}{2}\|\bA_2 \circ (\bB_2\pxhk^u)\|.
\end{gathered}
\end{align*}
Observe that
\begin{align}
[\bA_2]_{ij}=\frac{1}{\lambda_j-\lambda_{k+i}}=\frac{1}{(\lambda_j-\lambda_k+\Delta/2)+(\lambda_k-\Delta/2-\lambda_{k+i})}=\frac{1}{a_j+b_i},
\label{eq:A2-hada}
\end{align}
where $a_j=\lambda_j-(\lambda_k-\Delta/2), 1\leq j\leq k$ and $b_i=(\lambda_k-\Delta/2)-\lambda_{i+k}, 1\leq i\leq p_1-k$. Then $a_j, b_i\geq \Delta/2$ and again apply Lemma~\ref{lemma-hada-norm},
\begin{align*}
\begin{gathered}
\left\|\bA_2 \circ (\bB_1\pyhk^u)\right\|\leq \frac{1}{\Delta} \left\|\bB_1\pyhk^u\right\|\leq \frac{1}{\Delta} \left\|\bB_1\right\| \left\|\pyhk^u\right\|,\\
\left\|\bA_2 \circ (\bB_2\pxhk^u)\right\|\leq \frac{1}{\Delta} \left\|\bB_2\pxhk^u\right\|\leq \frac{1}{\Delta} \left\|\bB_2\right\| \left\|\pxhk^u\right\|.
\end{gathered}
\end{align*}
which implies that
\begin{align}
\left\|\pxhk^l \right\|\leq \frac{1}{2\Delta}\left(3\left\|\bB_1\right\| \left\|\pyhk^u\right\|+ \left\|\bB_2\right\| \left\|\pxhk^u\right\|\right)+\left\|\bA_1\circ \bA_2 \circ \bdelta_6\right\|
\label{eq:upper1}
\end{align}
In Section~\ref{sec::final-bound}, we will show that this bound is tight and matches Theorem~\ref{thm:upper} when $\lambda_{k+1}$ is away from 1. Now we switch to deriving the other upper bound which will be tight when $\lambda_{k+1}$ is close to 1.
%\subsubsection{Case 2: refined upper bound with the factor \texorpdfstring{$1-\lambda_{k+1}^2$}{TEXT}}
\subsubsection{Upper Bound II: tight for \texorpdfstring{$\lambda_{k+1}\geq 1/2$}{TEXT}}
%Specifically, we consider the case where $\lambda_{k+1}\geq 1/2$. In such scenario, the factor $1-\lambda_{k+1}^2$ in the upper bound is significant since $\lambda_{k+1}$ can be arbitrarily close to 1. First
Notice that $\bLambda_1\times\eqref{eq:delta1}+\bLambda_1\times \eqref{eq:delta2}$ yields
\begin{align}
\bLambda_1^2\pyhk^u-\pyhk^u \bLambda_1^2=\bLambda_1\bdelta_1+\bdelta_2\bLambda_1.
\label{eq-commute}
\end{align}
Substitute \eqref{eq:delta1}, \eqref{eq:delta2} and \eqref{eq-commute} into \eqref{eq-upper-target},
\begin{align}
\pxhk^l \bLambda_1^2-\bLambda_2^2\pxhk^l &=\sxyh^{21}\bLambda_1\pxhk^u+\sxyh^{21}\bdelta_1+ \bLambda_2\syxh^{21}\pxhk^u-\sxh^{21}\bLambda_1^2\pxhk^u \nonumber\\
&+\sxh^{21}(\bLambda_1\bdelta_1+\bdelta_2\bLambda_1)-\bLambda_2\syh^{21}\bLambda_1\pxhk^u+\bLambda_2\syh^{21}\bdelta_2+\bdelta_5 \nonumber\\
&=\bB\pxhk^u+\bdelta_7,
\end{align}
where we define
\begin{align*}
\begin{gathered}
\bB=\sxyh^{21}\bLambda_1+\bLambda_2\syxh^{21}-\sxh^{21}\bLambda_1^2-\bLambda_2\syh^{21}\bLambda_1, \\
\bdelta_7=\sxyh^{21}\bdelta_1+\sxh^{21}(\bLambda_1\bdelta_1+\bdelta_2\bLambda_1)+\bLambda_2\syh^{21}\bdelta_2+\bdelta_5.
\end{gathered}
\end{align*}
Again with the definition of $\bA_1$ and $\bA_2$,
\begin{align*}
\pxhk^l = \bA_1\circ \bA_2 \circ (\bB\pxhk^u)+\bA_1\circ \bA_2 \circ \bdelta_7.
\end{align*}
Notice that we can rewrite $\bA_1$ as
\[
[\bA_1]_{ij}=\frac{1}{\lambda_j+\lambda_{k+i}}=\frac{1}{(\lambda_j-\lambda_k/2)+(\lambda_{k+i}+\lambda_k/2)}=\frac{1}{\alpha_j+\beta_{i}},
\]
where $\alpha_j=\lambda_j-\lambda_k/2, 1\leq j\leq k$ and $\beta_i=\lambda_{k+i}+\lambda_k/2, 1\leq i\leq p_1-k$. Hence $\alpha_j, \beta_i\geq \lambda_k/2$, and apply Lemma~\ref{lemma-hada-norm},
\begin{align}
\|\pxhk^l \|\leq \frac{1}{\lambda_k}\left(\left\|\bA_2 \circ (\bB\pxhk^u)\right\|+\left\|\bA_2 \circ \bdelta_7\right\|\right).
\label{eq:target-refined}
\end{align}
Define $k^*$ as the largest index $i$ such that
\[
k+1\leq i\leq p_1,~~ \lambda_i\geq 2\lambda_k-1-\Delta.
\]
Divide the indexes $1, \cdots, p_1-k$ into two sets:
\[
\mc{I}_1=\{i-k: k+1\leq i\leq k^*\}, ~~\mc{I}_2=\{i-k: k^*+1\leq i\leq p_1\},
\]
and accordingly divide $\bA_2$ and $\bB\pxhk^u$ into two blocks:
\[
\bA_2=\begin{bmatrix}
\bA_2^{(1)} \\
\bA_2^{(2)}
\end{bmatrix}, ~~
\bB\pxhk^u=\begin{bmatrix}
\bB^{(1)}\pxhk^u\\
\bB^{(2)}\pxhk^u
\end{bmatrix}.
\]
where $\bA_2^{(1)}, \bB^{(1)}$ corresponds to the rows indexed by $\mc{I}_1$ and $\bA_2^{(2)}, \bB^{(2)}$ corresponds to the rows indexed by $\mc{I}_2$. Then
\[
\bA_2 \circ (\bB\pxhk^u)=\begin{bmatrix}
\bA_2^{(1)}\circ (\bB^{(1)}\pxhk^u) \\
\bA_2^{(2)}\circ (\bB^{(2)}\pxhk^u)
\end{bmatrix},
\]
and by triangle inequality,
\begin{align*}
\left\|\bA_2 \circ (\bB\pxhk^u)\right\|\leq \left\|\bA_2^{(1)}\circ (\bB^{(1)}\pxhk^u)\right\|+\left\|\bA_2^{(2)}\circ (\bB^{(2)}\pxhk^u)\right\|.
\end{align*}
For the first part, by the same argument as in \eqref{eq:A2-hada}
\begin{align}
\|\bA_2^{(1)}\circ (\bB^{(1)}\pxhk^u)\|\leq \frac{1}{\Delta}\|\bB^{(1)}\|\|\pxhk^u\|.
\label{eq:Bu}
\end{align}
For the second part, let $\bD=\text{diag}(\lambda_k-\Delta/2-\lambda_{k^*+1}, \cdots, \lambda_k-\Delta/2-\lambda_{p_1})\in\mathbb{R}^{(p_1-k^*)\times (p_1-k^*)}$. Then
\begin{align*}
\bA_2^{(2)}\circ (\bB^{(2)}\pxhk^u)&=(\bD\bA_2^{(2)})\circ (\bD^{-1}\bB^{(2)}\pxhk^u).
\end{align*}
Notice that for $1\leq i\leq p_1-k^*$, $1\leq j\leq k$
\begin{align*}
[\bD\bA_2^{(2)}]_{ij}&=\frac{(\lambda_k-\Delta/2-\lambda_{k^*+i})}{\lambda_j-\lambda_{i+k}}\\
&=\frac{(\lambda_k-\Delta/2-\lambda_{k^*+i})}{\lambda_j-(\lambda_{k}-\Delta/2)+(\lambda_k-\Delta/2-\lambda_{k^*+i})}
\\
&=\frac{b_i}{a_j+b_i},
\end{align*}
where $a_j=\lambda_j-(\lambda_{k}-\Delta/2), 1\leq j\leq k$ and $b_i=(\lambda_k-\Delta/2)-\lambda_{k^*+i}$ for $1\leq i\leq p_1-k^*$. Further observe that by definition of $k^*$,
\[
b_i-a_j=(\lambda_k-\Delta/2)-\lambda_{k^*+i}-\lambda_j+(\lambda_{k}-\Delta/2)\geq 2\lambda_k-1-\Delta-\lambda_{k^*+i}\geq 0,
\]
that is
\[
[\bD\bA_2^{(2)}]_{ij}=\frac{\max\{a_j, b_i\}}{a_j+b_i}.
\]
Again, by Lemma~\ref{lemma-hada-norm},
\begin{align}
\|\bA_2^{(2)}\circ (\bB^{(2)}\pxhk^u)\|\leq\frac{3}{2}\|\bD^{-1}\bB^{(2)}\pxhk^u\|\leq \frac{3}{2}\|\bD^{-1}\bB^{(2)}\| \|\pxhk^u\|.
\label{eq:Bl}
\end{align}
Substitute \eqref{eq:Bu} and \eqref{eq:Bl} into \eqref{eq:target-refined},
\begin{align}
\|\pxhk^l \|&\leq \frac{1}{\lambda_k}\left(\left\|\bA_2 \circ (\bB\pxhk^u)\right\|+\left\|\bA_2 \circ \bdelta_7\right\|\right)\\
&\leq \frac{1}{2\lambda_k\Delta}\|\bB^{(1)}\|\|\pxhk^u\|+\frac{3}{2\lambda_k}\|\bD^{-1}\bB^{(2)}\| \|\pxhk^u\|+\frac{1}{\lambda_k}\left\|\bA_2 \circ \bdelta_7\right\| .
\label{eq:upper2}
\end{align}
\subsection{Upper Bounds in Expectation for the Principal Terms}
\label{section:upper-lemma}
We state upper bounds for key quantities in \eqref{eq:upper1} and \eqref{eq:upper2} with proofs deferred to Section~\ref{sec::proof}.
\begin{lemma}
There exists universal constant $C$ such that
\begin{equation}
\E[\|\bB_1\|^2],~ \E[\|\bB_2\|^2] \leq C\frac{p_1}{n}(1-\lambda_k^2).
\end{equation}
\label{lemma-B1B2}
\end{lemma}
See Section~\ref{sec::B1B2} for the proof of this lemma.
\begin{lemma}
There exists universal constants $c, C_1$ such that the following inequality holds with probability at least $1-2e^{-ct^2}$,
\begin{equation}
\|\bD^{-1}\bB^{(2)}\|\leq \sqrt{\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2}}\max\{\delta, \delta^2\} \quad\quad \delta=C_1\left(\sqrt{\frac{p_1}{n}}+\frac{t}{\sqrt{n}}\right).
\end{equation}
As a corollary, there exists constant $C_2$,
\begin{equation}
\E \|\bD^{-1}\bB^{(2)}\|^2\leq C_2\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)p_1}{n\Delta^2}.
\end{equation}
\label{lemma-Bl}
\end{lemma}
See Section~\ref{sec::Bl} for the proof of this lemma.
\begin{lemma}
There exists universal constants $c, C_1$ such that the following inequality holds with probability at least $1-2e^{-ct^2}$,
\begin{equation}
\|\bB^{(1)}\|\leq \sqrt{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}\max\{\delta, \delta^2\} \quad\quad \delta=C_1\left(\sqrt{\frac{p_1}{n}}+\frac{t}{\sqrt{n}}\right).
\end{equation}
As a corollary, there exists constant $C_2$,
\begin{equation}
\E \|\bB^{(1)}\|^2\leq C_2\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)p_1}{n}.
\end{equation}
\label{lemma-Bu}
\end{lemma}
See Section~\ref{sec::Bu} for the proof of this lemma.
%\subsubsection{On Frobenius norm of \texorpdfstring{$\bB$}{TEXT}}
%Notice that $(\bx_{\alpha i}, \by_{\alpha i})$ are mutually uncorrelated pairs for any $1\leq \alpha\leq n, 1\leq i\leq p_1$. It is easy to compute
%\begin{align*}
%\E b_{ij}^2&=\frac{1-\lambda_j^2}{(\lambda_j^2-\lambda_i^2)^2} (\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2)\\
%&=\frac{1-\lambda_j^2}{(\lambda_j-\lambda_i)^2} \frac{\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2}{\lambda_i^2+\lambda_j^2+2\lambda_i^2\lambda_j^2}
%\end{align*}
%and thus
%\begin{align*}
%\E \|\bB\|_F^2 &= \sum_{1\leq j\leq k<i\leq p_1} \frac{1-\lambda_j^2}{(\lambda_j-\lambda_i)^2} \frac{\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2}{\lambda_i^2+\lambda_j^2+2\lambda_i^2\lambda_j^2}\\
%&\leq (1-\lambda_k^2)\sum_{1\leq j\leq k<i\leq p_1} \frac{1}{(\lambda_j-\lambda_i)^2} \frac{\lambda_i^2(1-\lambda_j^2)+\lambda_j^2(1-\lambda_i^2)}{\lambda_i^2+\lambda_j^2+2\lambda_i^2\lambda_j^2}\\
%&\leq (1-\lambda_k^2)\sum_{1\leq j\leq k<i\leq p_1} \frac{1}{(\lambda_j-\lambda_i)^2} \frac{(\lambda_i^2+\lambda_j^2)(1-\lambda_i^2)}{\lambda_i^2+\lambda_j^2+2\lambda_i^2\lambda_j^2}\\
%&\leq (1-\lambda_k^2)\sum_{1\leq j\leq k<i\leq p_1} \frac{1-\lambda_i^2}{(\lambda_j-\lambda_i)^2} \\
%&\leq (1-\lambda_k^2)\sum_{1\leq j\leq k<i\leq p_1} \frac{1-\lambda_{k+1}^2}{(\lambda_j-\lambda_{k+1})^2} \\
%&\leq \frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2} (p_1-k)k
%\end{align*}
%where the second last inequality is due to the fact that for $\lambda>\lambda_{k+1}$, $f(x)=\frac{1-x^2}{(\lambda-x)^2}$ is increasing in the interval $[0, \lambda_{k+1}]$.
%\subsubsection{On the higher order term \texorpdfstring{$\wt{\mtx{\mathcal{E}}}$}{TEXT}}
%We derived the following deterministic Frobenius norm bound on the higher order term $\wt{\mtx{\mathcal{E}}}$ and we defer the proof to the appendix.
%\begin{lemma} Let $\bz\in\mathbb{R}^{p_1+p_2}$ be the concatenation of $\bx$ and $\by$. Define the covariance matrix $\s_z$ and sample covariance matrix $\sh_z$ accordingly. It holds deterministically that
%\label{lemma-high-order}
%\begin{align*}
%\|\wt{\mtx{\mathcal{E}}}\|_F^2\leq \frac{2\cdot 36^2nk}{\Delta_{\min, k}^2\Delta^2} \left(C_1(\bX, \bY)^4+C_2(\bX, \bY)^4\right)\|\s_z-\sh_z\|^4
%\end{align*}
%where $\Delta_{\min, k}=\min\limits_{2\leq j\leq k}\min\{\lambda_{j-1}-\lambda_j, \lambda_j-\lambda_{j+1}\}$ and
%\begin{equation}
%\begin{gathered}
%C_1(\bX, \bY)=\max\Big\{3+\|\sxh^{-1/2}\| , \|\sxh\|+\|\syh^{-1/2}\|\Big\}\\
%C_2(\bX, \bY)=\max\Big\{3+\|\syh^{-1/2}\| , \|\syh\|+\|\sxh^{-1/2}\|\Big\}
%\end{gathered}
%\nonumber
%\end{equation}
%\end{lemma}
\subsection{Upper Bound for Operator Norm}
\label{sec::final-bound}
For the ease of presentation, we introduce $\bz=(\bx^\top, \by^\top)^\top$ as the concatenation of $\bx$ and $\by$. Then the population and sample covariances of $\bz$ can be written as,
\[
\s_z=\begin{bmatrix}
\bI_{p_1} & \sxy\\
\syx&\bI_{p_2}
\end{bmatrix}, ~~\sh_z=\begin{bmatrix}
\sxh & \sxyh\\
\syxh&\syh
\end{bmatrix}.
\]
The advantage of introducing $\bz$ is that the sample-population discrepancy for $\bx$ and $\by$ can be simultaneously bounded by that of $\bz$.
\begin{lemma}
There exists universal constant $C$ such that the following inequality holds deterministically,
\begin{align*}
\|\bA_1\circ\bA_2\circ\bdelta_6\|, ~\|\bA_2\circ\bdelta_7\|\leq \frac{C\|\s_z-\sh_z\|^2}{\Delta^2} (2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|),
\end{align*}
where $\Delta=\lambda_k-\lambda_{k+1}$ is the eigen-gap.
\label{lemma-high-order}
\end{lemma}
See Section~\ref{sec::high-order} for the proof.
\begin{lemma}
There exists universal constant $c, C_1, C_2$ such that when $n\geq C_1(p_1+p_2)$, the following inequality holds
\begin{align*}
\begin{gathered}
\sigma_k(\pxhk)^2\geq 1/2, \\
(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|)\leq C_2,
\end{gathered}
\end{align*}
with probability at least $1-e^{-cn}$.
\label{lemma-high-prob}
\end{lemma}
See Section~\ref{sec::high-prob} for the proof.
%\begin{lemma}
%There exists universal constant $c, C$ such that when $n\geq C(p_1+p_2)/\Delta^2$, with probability at least $1-exp(-c\Delta n)$, the following inequalities hold simultaneously
%\begin{align*}
%\begin{gathered}
%|\hlambda_k-\lambda_{k+1}|\leq \Delta/2, \\
%\sigma_k^2(\pxhk)\geq 1/2, ~\|\pxhk\|\leq 2,~ \|\pyhk\|\leq 2, \\
%(1+\|\sh_z\|+\|\sh_z^{-1}\|)^2(1+ \|\s_z-\sh_z\|)\leq 20.
%\end{gathered}
%\end{align*}
%\label{lemma-high-prob}
%\end{lemma}
%By definition, $\pxh^\top\sxh\pxh=\bI_{p_1}$, then
%\[
%\pxh^\top\pxh-\bI_{p_1}=-\pxh^\top(\sxh-\bI_{p_1})\pxh.
%\]
%Notice that $\sxh^{\half}\pxh\in\mc{O}(p_1)$,
%\begin{align*}
%\|\pxh^\top\pxh-\bI_{p_1}\|&\leq \|\pxh^\top(\sxh-\bI_{p_1})\pxh\|\leq \|\pxh^\top\sxh^{1/2}\|\|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|\|\sxh^{1/2}\pxh\|\\
%&=\|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|.
%\end{align*}
%As a submatrix,
%\[
%\|\pxhk^\top\pxhk-\bI_k\|\leq \|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|,
%\]
%which implies that
%\begin{align*}
%\begin{gathered}
%\sigma_k^2(\pxhk)\geq 1- \|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|\geq 1- \|\sxh^{-1}\|\|\sxh-\bI_{p_1}\|\geq 1-\|\sh_z^{-1}\|\|\sh_z-\s_z\|\\
%\|\pxhk\|^2 \leq 1 + \|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|\leq 1+ \|\sxh^{-1}\|\|\sxh-\bI_{p_1}\|\leq 1+\|\sh_z^{-1}\|\|\sh_z-\s_z\|
%\end{gathered}
%\end{align*}
%For sufficiently small constant $\tau\in (0, 1/2)$, define the event $A_{\tau}$ as
%\[
%A_{\tau}=\left\{ \|\s_z-\sh_z\|_2 \leq \tau\Delta\right\}
%\]
%On event $A_{\tau}$,
%\[
%\sigma_k^2(\pxhk)\geq 1- \|\sxh^{-1}\|\|\sxh-\bI_{p_1}\|\geq 1-\frac{\tau}{1-\tau}\geq 1-2\tau
%\]
%and
%\[
%C_1(\bX, \bY), C_2(\bX, \bY)\leq 3+\frac{1}{\sqrt{1-\tau}}\leq 5
%\]
\medskip
Let $G$ be the event that the inequalities in Lemma~\ref{lemma-high-prob} hold. Notice that $\|\bP_1-\bP_2\|\leq 1$ for any pair of projection matrices with equal rank. Substitute into equation \eqref{eq-target},
\begin{align*}
\E \left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|^2&\leq \P(G^c)+\E \left[\left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|^2 I_{G}\right]\\
&\leq \P(G^c)+ \E\left[ \frac{\|\pxhk^l\|^2_F}{\sigma_k^2(\pxh_{k,k})} I_{G}\right]\\
&\leq exp(-cn)+ 2\E\left[\|\pxhk^l\|^2 I_{G}\right].
\end{align*}
%\begin{align*}
%\E \left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|^2&\leq \P(G^c)+ \E\left[ \frac{\|\pxhk^l\|^2_F}{\sigma_k^2(\pxh_{k,k})} I_{G}\right]\\
%&\leq exp(-cn)+ 2\E\left[\|\pxhk^l\|^2 I_{G}\right]
%\end{align*}
Now we plug in the upper bounds of $\|\pxhk^l\|$ obtained in \eqref{eq:upper1} and \eqref{eq:upper2} respectively. On event $G$, \eqref{eq:upper1} can be reduced to
\begin{align*}
\|\pxhk^l\|^2\leq \frac{C}{\Delta^2}\left(\left\|\bB_1\right\|^2+ \left\|\bB_2\right\|^2\right)+C\left\|\bA_1\circ \bA_2 \circ \bdelta_6\right\|^2.
\end{align*}
Further, by Lemma~\ref{lemma-high-order} and Lemma~\ref{lemma-high-prob}, on event $G$,
\begin{align*}
\|\pxhk^l\|^2\leq \frac{C}{\Delta^2}\left(\left\|\bB_1\right\|^2+ \left\|\bB_2\right\|^2\right)+\frac{C}{\Delta^4}\|\s_z-\sh_z\|^4.
\end{align*}
Therefore,
\begin{align*}
\E\left[\|\pxhk^l\|^2 I_{G}\right]&\leq \frac{C}{\Delta^2}\E\left(\left\|\bB_1\right\|^2+ \left\|\bB_2\right\|^2\right)+\frac{C}{\Delta^4}\E\|\s_z-\sh_z\|^4.
\end{align*}
By Lemma~\ref{lemma-B1B2} and Lemma~\ref{lemma-expectation-covariance} (notice that $\|\s_z\|\leq 2$),
\begin{align}
\E\left[\|\pxhk^l\|^2 I_{G}\right]&\leq \frac{C(1-\lambda_k^2)p_1}{\Delta^2n}+C\left(\frac{p_1+p_2}{n\Delta^2}\right)^2.
\label{eq:result1}
\end{align}
This upper bound implies the result in Theorem~\ref{thm:upper} when $\lambda_{k+1}$ is bounded away from 1, for instance, when $\lambda_{k+1}\leq 1/2$. Now, we use \eqref{eq:upper2} for the case when $\lambda_{k+1}\geq 1/2$. On event $G$, with $\lambda_{k+1}\geq 1/2$, \eqref{eq:upper2} can be reduced to
\begin{align*}
\|\pxhk^l\|^2\leq C\left(\frac{1}{\Delta^2}\|\bB^{(1)}\|^2+\|\bD^{-1}\bB^{(2)}\|^2+\left\|\bA_2 \circ \bdelta_7\right\|^2\right).
\end{align*}
Further apply Lemma~\ref{lemma-high-order} and Lemma~\ref{lemma-high-prob}, on event $G$,
\begin{align*}
\|\pxhk^l\|^2\leq C\left(\frac{1}{\Delta^2}\|\bB^{(1)}\|^2+\|\bD^{-1}\bB^{(2)}\|^2+\frac{1}{\Delta^4}\|\s_z-\sh_z\|^4\right).
\end{align*}
By Lemma~\ref{lemma-Bl}, Lemma~\ref{lemma-Bu} and Lemma~\ref{lemma-high-order},
\begin{align}
\E\left[\|\pxhk^l\|^2 I_{G}\right]&\leq \frac{C(1-\lambda_k^2)(1-\lambda_{k+1}^2)p_1}{\Delta^2n}+C\left(\frac{p_1+p_2}{n\Delta^2}\right)^2.
\label{eq:result2}
\end{align}
Combine the results of \eqref{eq:result1} and \eqref{eq:result2},
\begin{align*}
\E \left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|^2
&\leq exp(-c n)+ \frac{C(1-\lambda_k^2)(1-\lambda_{k+1}^2)p_1}{\Delta^2n}+C\left(\frac{p_1+p_2}{n\Delta^2}\right)^2\\
&\leq C\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)p_1}{\Delta^2n}+C_2\left(\frac{p_1+p_2}{n\Delta^2}\right)^2
\end{align*}
\subsection{Upper Bound for Frobenius Norm}
A quick upper bound in terms of Frobenius norm can be obtained by noticing that $\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}$ has rank at most $2k$ and therefore,
\begin{align*}
\frac{1}{k}\E \left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|_F^2&\leq 2\E \left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|^2\\
&\leq C\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)p_1}{\Delta^2n}+C\left(\frac{p_1+p_2}{n\Delta^2}\right)^2.
\end{align*}
In fact, the factor $p_1$ in the main term can be reduced to $p_1-k$ by similar (but much simpler) arguments as done for the operator norm. We state the corresponding results in this section without proof. Specifically, the following Frobenius norm counterparts for \eqref{eq:upper1} and \eqref{eq:upper2} can be obtained.
\begin{lemma}
Let $\wt{\bD}=\text{diag}(\lambda_k-\lambda_{k+1}, \cdots, \lambda_{k}-\lambda_{p_1})\in\mathbb{R}^{(p_1-k)\times (p_1-k)}$, then
\begin{gather}
\left\|\pxhk^l \right\|_F\leq \frac{1}{2\Delta}\left(3\left\|\bB_1\right\|_F \left\|\pyhk^u\right\|+ \left\|\bB_2\right\|_F \left\|\pxhk^u\right\|\right)+\left\|\bA_1\circ \bA_2 \circ \bdelta_6\right\|_F,\\
\left\|\pxhk^l \right\|_F \leq \frac{1}{\lambda_k}\|\wt{\bD}^{-1}\bB\|_F\|\pxhk^u\|+\frac{1}{\lambda_k}\left\|\bA_2 \circ \bdelta_7\right\|_F.
\end{gather}
\end{lemma}
Notice that for the second bound, divide-and-conquer type of analysis in the proof for operator norm is no longer necessary due to the observation that
\[
\|\bA\circ \bM\|_F\leq \|\bM\|_F,
\]
when $\max_{i, j}|\bA_{ij}|\leq 1$ while the inequality is not true for operator norm. Similarly, parallel results to the lemmas in Section~\ref{section:upper-lemma} can be derived as follows (see Section~\ref{sec::B-Fnorm} for the proof of the second inequality in the lemma as illustration).
\begin{lemma}
There exists universal constant $C$ such that
\begin{equation}
\begin{gathered}
\E[\|\bB_1\|_F^2],~ \E[\|\bB_2\|_F^2] \leq C\frac{(1-\lambda_k^2)(p_1-k)k}{n}, \\
\E \left[\left\|\wt{\bD}^{-1}\bB\right\|_F^2\right] \leq 2\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)(p_1-k)k}{n\Delta^2}, \\
\|\bA_1\circ\bA_2\circ\bdelta_6\|_F, ~\|\bA_2\circ\bdelta_7\|_F\leq \frac{C\sqrt{k}\|\s_z-\sh_z\|^2}{\Delta^2}(2+\|\sh_z\|)^2
(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|).
\end{gathered}
\nonumber
\end{equation}
\label{lemma-B-Fnorm}
\end{lemma}
Substituting these lemmas into the procedure of Section~\ref{sec::final-bound} will obtain
\begin{align*}
\frac{1}{k}\E \left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|_F^2\leq \frac{C(1-\lambda_k^2)(1-\lambda_{k+1}^2)(p_1-k)}{\Delta^2n}+C\left(\frac{p_1+p_2}{n\Delta^2}\right)^2.
\end{align*}
%\begin{align*}
%\E &\left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|^2\leq \P(A)+\E \left[\left\|\mtx{P}_{\pxhk} - \mtx{P}_{\pxk}\right\|^2 I_{A}\right]\\
%&\leq 4k\P(A_{\tau})+ \E\left[ \frac{\|\pxhk^l\|^2_F}{\sigma_k^2(\pxh_{k,k})} I_{A}\right]\\
%&\leq 4k\P(A_{\tau})+ \frac{1}{1-2\tau}\E\left[\|\pxhk^l\|^2 I_{A}\right]\\
%&\leq 4k\P(A_{\tau})+ \frac{2}{(1-2\tau)n} \left( \E \|\bB\|^2 + \E\left[\|\wt{\mtx{\mathcal{E}}}\|_F^2 I_A\right]\right)\\
%&\leq 4k\P(A_{\tau})+ \frac{2}{1-2\tau}\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2} \frac{(p_1-k)k}{n} + \frac{2\cdot 72^2\cdot 5^4 k}{\Delta_{\min, k}^2\Delta^2} \E\|\s_z-\sh_z\|^4\\
%&\leq 4k\P(A_{\tau})+\frac{2}{1-2\tau}\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2} \frac{(p_1-k)k}{n} +\frac{C}{(1-\tau)^4}\frac{(p_1+p_2)^2k}{n^2\Delta^2\Delta_{\min, k}^2}
%\end{align*}
%where the last inequality is due to Lemma~\ref{lemma-expectation-covariance}. Finally we need to bound the probability of $A_{\tau}$. Notice that $\sxh-\bI_{p_1}, \syh-\bI_{p_2}, \sxyh-\sxy$ are all sub-matrices of $\s_z-\sh_z$. Therefore, their spectral norms are uniformly upper bounded by $\|\s_z-\sh_z\|$. Apply Lemma~\ref{lemma-vershynin} with $\delta=\tau/2$ and $\|\s_z\|\leq 2$, there exists positive constants $c_1, c_2$,
%\[
%\P(\|\s_z-\sh_z\|\geq \tau)\leq e^{-c_1(\sqrt{n}\tau/2-c_2\sqrt{p_1+p_2})_+^2}
%\]
%If $n\geq 16c_2^2(p_1+p_2)/\tau^2$, then
%\[
%\P(A_{\tau})\leq P(\|\s_z-\sh_z\|\geq \tau)\leq e^{-c_1n\tau/16}
%\]
%%Further by Lemma~\ref{lemma-expectation-covariance}, there exists positive constant $C_0$ such that $\E\|\s_z-\sh_z\|^4\leq C_0(p_1+p_2)^2/n^2$. Therefore,
%Substitute into the inequality above,
%\begin{align*}
%\E \left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|_F^2\right] &\leq 4ke^{-\frac{c_1\tau n}{16}}+\frac{2}{1-2\tau}\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2} \frac{(p_1-k)k}{n} +\frac{C}{(1-\tau)^4}\frac{(p_1+p_2)^2k}{n^2\Delta^2\Delta_{\min, k}^2}
%\end{align*}
%For the spectral norm, by the same argument,
%\begin{align*}
%\E \left[ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk} \right\|^2\right] &\leq 4ke^{-\frac{c_1\tau n}{16}}+\frac{C}{1-2\tau}\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)p_1}{\Delta^2}+\frac{C}{(1-\tau)^4}\frac{(p_1+p_2)^2k}{n^2\Delta^2\Delta_{\min, k}^2}
%\end{align*}
%!TEX root = SpikeCCA.tex
\subsection{Proof of Theorem~\ref{thm-metric}}
\label{section-proof-loss}
For any $\bU_1, \bU_2\in\mathbb{R}^{p\times k}$, from classical linear model theory,
\[
\bbeta_i :=\arg\min_{\bbeta}\E\left[\left|z-\bbeta^{\top} (\bU_i^\top\bx_i)\right|^2\right]=(\bU_i^\top\sx\bU_i)^{-1}\bU_i^\top\s_{xz}, ~~i=1, 2,
\]
and
\begin{align*}
loss(z|\lspan(\bx^\top\bU_i))&=\E[z^2]-\E[(\bbeta_i^{\top} (\bU_i^\top\bx_i))^2]\\
&=\sigma_z^2-\s_{zx}\bU_i(\bU_i^\top\sx\bU_i)^{-1}\bU_i^\top\sx\bU_i(\bU_i^\top\sx\bU_i)^{-1}\bU_i^\top\s_{xz}\\
&=\sigma_z^2\left(1-\br_{zx}(\sx^{\half}\bU_i)(\bU_i^\top\sx\bU_i)^{-1}(\sx^{\half}\bU_i)^\top\br_{xz}\right).
\end{align*}
Notice that $(\sx^{\half}\bU_i)(\bU_i^\top\sx\bU_i)^{-\half}$ has orthonormal columns with column space $\lspan(\sx^{\half}\bU_i)$, then
\begin{align*}
loss(z|\lspan(\bx^\top\bU_i))=\sigma_z^2\left(1-\br_{zx}\bP_{\sx^{\half}\bU_i}\br_{xz}\right).
\end{align*}
Therefore,
\begin{align*}
loss(z|\lspan(\bx^\top\bU_1))-loss(z|\lspan(\bx^\top\bU_2))=\sigma_z^2\br_{zx}\left(\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right)\br_{xz}.
\end{align*}
By the variational definition of the leading eigenvalue,
\begin{align*}
\sup\limits_{\|\br_{xz}\|^2=R^2} loss(z|\lspan(\bx^\top\bU_1))-loss(z|\lspan(\bx^\top\bU_2))&=\sigma_z^2R^2 \lambda_{\max} (\bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1})\\
&=\sigma_z^2R^2 \left\| \bP_{\sx^{\half}\bU_2}-\bP_{\sx^{\half}\bU_1}\right\|,
\end{align*}
where the second inequality is by the characterization of the difference between two projection matrices described in \cite{wedin1983angles}. This proves the first claim of the theorem. For the second part, since $\br_{xz}\in\mc{A}$,
\begin{align*}
&loss(z|\lspan(\bx^\top\mtx{U})) - loss(z|\lspan(\bx^\top\os))\\
&=\sigma_z^2 \br_{xz}^\top \left(\bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right)\br_{xz}\\
&=\sigma_z^2 \br_{xz}^\top\bP_{\sx^{\half}\os} \left(\bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right)\bP_{\sx^{\half}\bU_2}\br_{xz}\\
&= \sigma_z^2 \br_{xz}^\top\bP_{\sx^{\half}\os} \left(\bI_{p}-\bP_{\sx^{\half}\bU}\right)\bP_{\sx^{\half}\os}\br_{xz}\\
&\leq \sigma_z^2 R^2\left\|\bP_{\sx^{\half}\os} \left(\bI_{p}-\bP_{\sx^{\half}\bU}\right)\bP_{\sx^{\half}\os}\right\|\\
&=\sigma_z^2R^2 \left\|\bP_{\sx^{\half}\os} \left(\bI_{p}-\bP_{\sx^{\half}\bU}\right)\right\|^2\\
&=\sigma_z^2R^2 \left\| \bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right\|^2.
\end{align*}
Notice that $\bP_{\sx^{\half}\os} \left(\bI_p-\bP_{\sx^{\half}\bU}\right)\bP_{\sx^{\half}\os}$ is positive definite. Let $\bu_1$ be the leading singular vector of this matrix and define $\br_{xz}^*=R\bu_1$. Then $\br_{xz}^*\in\mc{A}$ and with such choice of $\br_{xz}^*$, the inequality above will become equality and this implies that
\begin{align*}
\sup \limits_{\br_{xz}^*\in\mc{A}} \left\{ loss(z|\lspan(\bx^\top\mtx{U})) - loss(z|\lspan(\bx^\top\os))\right\} = \sigma_z^2 R^2 \left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|^2.
\end{align*}
For the last part of the theorem, let $\bP_{\sx^{\half}\os}=\bQ\bQ^\top$. Then the set $\mc{A}$ can be rewritten as
\[
\mc{A}=\{\br: \br=R^2\bQ\wt{\br}, \wt{\br}\in\mathbb{S}^{k-1}\}.
\]
The uniform measure (Haar measure) $\pi$ on $\mc{A}$ can be expressed through the uniform measure (Haar measure) $\wt{\pi}$ on the sphere $\mathbb{S}^{k-1}$. Because $\wt{\pi}$ is uniform on the sphere $\mathbb{S}^{k-1}$, then $\text{Var}(\wt{\pi})=\bI_{k}/k$ and
\begin{align*}
&\underset{ \br_{xz}\sim \pi}{\E}\left[ loss(z|\lspan(\bx^\top\mtx{U})) - loss(z|\lspan(\bx^\top\os))\right ]\\
&=\sigma_z^2R^2~ \E_{\wt{r}\sim\wt{\pi}}\left[\wt{\br}^\top\bQ^\top \left(\bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right)\bQ\wt{\br} \right]\\
&= \sigma_z^2R^2\tr\left(\bQ^\top\left( \bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right)\bQ \right)/k\\
&= \sigma_z^2R^2\tr\left(\left( \bP_{\sx^{\half}\os}-\bP_{\sx^{\half}\bU}\right)\bQ \bQ^\top\right)/k\\
&= \sigma_z^2R^2\tr\left(\left( \bI_{p}-\bP_{\sx^{\half}\bU}\right)\bP_{\sx^{\half}\os}\right)/k\\
&=\sigma_z^2R^2 \tr\left(\bP_{\sx^{\half}\os} \left(\bI_p-\bP_{\sx^{\half}\bU}\right)\bP_{\sx^{\half}\os} \right)/k\\
&= \sigma_z^2R^2\left\|\bP_{\sx^{\half}\os} \left(\bI_p-\bP_{\sx^{\half}\bU}\right)\right\|_F^2/k\\
&=\sigma_z^2R^2\left\|\bP_{\sx^{\half}\bU}-\bP_{\sx^{\half}\os}\right\|_F^2/2k,
\end{align*}
where the last equality is due to Lemma~\ref{lemma-projection}.
\subsection{Proof of Lemma~\ref{lemma-hada-norm}}
\label{sec::hada-norm}
The proof of Lemma~\ref{lemma-hada-norm} relies on the following two results.
\begin{lemma}(\cite{hom1991topics}, Theorem 5.5.18)
If $\bA, \bB\in\mathbb{R}^{n\times n}$ and $\bA$ is positive semidefinite. Then,
\begin{equation}
\|\bA\circ \bB\|\leq \left(\max_{1\leq i\leq n} \bA_{ii}\right)\|\bB\|,
\nonumber
\end{equation}
where $\|\cdot\|$ is the operator norm.
\label{HadaNorm}
\end{lemma}
\begin{lemma}(\cite{fiedler2010notes}, Theorem A)
A symmetric Cauthy matrix
$$\bC=\Big(\frac{1}{a_i+a_j}\Big)_{1\leq i, j\leq n}$$
is positive semidefinite if $a_i>0, 1\leq i\leq n$.
\label{cauthy}
\end{lemma}
Define $\gamma_i=\beta_i, 1\leq i\leq n$ and $\gamma_{i}=\alpha_{i-n}, n+1\leq i\leq m+n$. Consider the matrix $\bM_1, \bM_2\in\mathbb{R}^{(m+n)\times (m+n)}$ define by
\[
[\bM_1]_{ij}=\frac{1}{\gamma_i+\gamma_j}, ~[\bM_2]_{ij}=\frac{\min\{\gamma_i, \gamma_j\}}{\gamma_i+\gamma_j}.
\]
By Lemma~\ref{cauthy}, $\bM_1$ is positive semidefinite and by Lemma~\ref{HadaNorm},
\[
\norm{\bM_1}\leq \frac{1}{2\min\limits_{1\leq i\leq m+n}\{\gamma_i\}}\leq \frac{1}{2\delta}.
\]
Notice that $\bA_1$ is the lower left sub-matrix of $\bM_1$, therefore,
\[
\norm{\bA_1}\leq \norm{\bM_1}\leq \frac{1}{2\delta}.
\]
By Theorem 3.2 of \cite{mathias1993hadamard}, $\bM_2$ is also positive semidefinite. Again, apply Lemma~\ref{HadaNorm} and notice that $\bA_2$ is the lower left sub-matrix of $\bM_2$,
\[
\norm{\bA_2}\leq \norm{\bM_2}\leq \frac{1}{2}.
\]
Finally, observe that, by definition, $\bA_3\circ \bB=\bB-\bA_2\circ \bB$, hence
\[
\|\bA_3\circ \bB\|\leq\|\bB\|+\|\bA_2\circ \bB\|,
\]
which implies,
\[
\norm{\bA_3}\leq 1+\norm{\bA_2}\leq \frac{3}{2}.
\]
\subsection{Proof of Lemma~\ref{lemma-B1B2}}
\label{sec::B1B2}
Divide $\bx, \by$ into two parts,
\[
\bx=\begin{pmatrix}
\bx_1 \\ \bx_2
\end{pmatrix}, ~
\by=\begin{pmatrix}
\by_1\\ \by_2
\end{pmatrix},
\]
where $\bx_1, \by_1\in\mathbb{R}^k$, $\bx_2\in\mathbb{R}^{p_1-k}$ and $\by_2\in\mathbb{R}^{p_2-k}$. By definition,
\[
\bB_1=\widehat{\text{Cov}}(\bx_2, \by_1-\bLambda_1\bx_1),
\]
where $\widehat{\text{Cov}}(\cdot, \cdot)$ denotes the sample covariance operator. When $\lambda_k=1$, by definition of CCA, $\by_1=\bx_1$ almost surely, which implies that $\bB_1=\mtx{0}$ almost surely. When $\lambda_k<1$,
\[
\bB_1=\widehat{\text{Cov}}(\bx_2, \by_1-\bLambda_1\bx_1)=\sqrt{1-\lambda_k^2}\widehat{\text{Cov}}\left(\bx_2, \frac{\by_1-\bLambda_1\bx_1}{\sqrt{1-\lambda_k^2}}\right)=\sqrt{1-\lambda_k^2}\widehat{\text{Cov}}(\bw_1, \bw_2),
\]
where we define $\bw_1=\frac{\by_1-\bLambda_1\bx_1}{\sqrt{1-\lambda_k^2}}$ and $\bw_2=\bx_2$. Let $\bw=(\bw_1^\top, \bw_2^\top)^\top$ be the concatenation of $\bw_1$ and $\bw_2$. Then
\[
\s_w=\text{Var}(w)=\text{diag}\left(\frac{1-\lambda_1^2}{1-\lambda_k^2},\frac{1-\lambda_2^2}{1-\lambda_k^2}, \cdots, \frac{1-\lambda_k^2}{1-\lambda_k^2}, 1, \cdots, 1\right).
\]
Notice that $\|\s_{\bw}\|\leq 1$ and $\E[\widehat{\text{Cov}}(\bw_1, \bw_2)]=\mtx{0}$. Therefore,
\[
\|\bB_1\|^2\leq (1-\lambda_k^2)\|\widehat{\text{Var}}(\bw)-\s_w\|^2.
\]
By Lemma~\ref{lemma-expectation-covariance},
\[
\E[\|\bB_1\|^2]\leq C\frac{(1-\lambda_k^2)p_1}{n}.
\]
Similarly, we can prove the result for $\bB_2$ in the same manner.
\subsection{Proof of Lemma~\ref{lemma-Bl}}
\label{sec::Bl}
\textbf{Step 1. Reduction. } For $\epsilon>0$ and any pair of vectors $\bu\in\mathbb{R}^{p_1-k^*}, \bv\in\mathbb{R}^k$, we can choose $\bu_{\epsilon}\in \mathcal{N}(\mathbb{S}^{p_1-k^*-1}, \epsilon), \bv_{\epsilon}\in \mathcal{N}(\mathbb{S}^{k-1}, \epsilon)$ such that $\|\bu-\bu_{\epsilon}\|, \|\bv-\bv_{\epsilon}\|\leq \epsilon$. Then
\begin{align*}
\bu^\top \bD^{-1}\bB^{(2)}\bv&=\bu^\top \bD^{-1}\bB^{(2)}\bv-\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv+\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv-\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv_\epsilon+\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv_\epsilon\\
&\leq \|\bu-\bu_{\epsilon}\|\| \bD^{-1}\bB^{(2)}\bv\|+\|\bu_\epsilon^\top\bD^{-1}\bB^{(2)}\|\|\bv-\bv_{\epsilon}\|+\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv_\epsilon\\
&\leq 2\epsilon\|\bD^{-1}\bB^{(2)}\|+\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv_\epsilon\\
&\leq 2\epsilon\|\bD^{-1}\bB^{(2)}\|+\max\limits_{\bu_{\epsilon}, \bv_{\epsilon}}\bu_\epsilon^\top\bD^{-1} \bB^{(2)}\bv_\epsilon.
\end{align*}
Maximize over $\bu$ and $\bv$, we obtain
\begin{equation}
\|\bD^{-1}\bB^{(2)}\|\leq 2\epsilon\|\bD^{-1}\bB^{(2)}\|+\max\limits_{\bu_\epsilon, \bv_\epsilon}\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv_\epsilon.
\end{equation}
Therefore, $\|\bD^{-1}\bB^{(2)}\|\leq (1-2\epsilon)^{-1}\max\limits_{\bu_\epsilon, \bv_\epsilon}\bu_\epsilon^\top \bD^{-1}\bB^{(2)}\bv_\epsilon$. Let $\epsilon=1/4$. Then it suffices to prove with required probability,
\begin{equation}
\max\limits_{\bu_\epsilon, \bv_\epsilon}\bu_\epsilon^\top\bD^{-1} \bB^{(2)}\bv_\epsilon\leq \frac{1}{2}\max\{\delta, \delta^2\} .
\end{equation}
\textbf{Step 2. Concentration. }
Notice that for $1\leq j\leq k<k^*+1\leq i\leq p_1$,
\begin{align*}
[\bD^{-1}\bB^{(2)}]_{i-k^*, j}&=\frac{1}{\lambda_k-\Delta/2-\lambda_i} \frac{1}{n}\sum\limits_{\alpha=1}^n (\lambda_j \bx_{\alpha i} \by_{\alpha j}- \lambda_j^2 \bx_{\alpha i} \bx_{\alpha j}+ \lambda_i \bx_{\alpha j} \by_{\alpha i} - \lambda_i\lambda_j \by_{\alpha i}\by_{\alpha j})\\
&=\frac{1}{\lambda_k-\Delta/2-\lambda_i} \frac{1}{n}\sum\limits_{\alpha=1}^n \Big\{(1-\lambda_j^2)\lambda_i\lambda_j \bx_{\alpha i}\bx_{\alpha j} -\lambda_j^2(\by_{\alpha i}-\lambda_i\bx_{\alpha i})(\by_{\alpha j}-\lambda_j\bx_{\alpha j})\\
&~~~+(1-\lambda_j^2)\lambda_j(\by_{\alpha i}-\lambda_i\bx_{\alpha i})\bx_{\alpha j}+(1-\lambda_j^2)\lambda_i(\by_{\alpha j}-\lambda_j\bx_{\alpha j})\bx_{\alpha i}\Big\}.
\end{align*}
Let $\bz_l=(\by_l-\lambda_i\bx_l)/\sqrt{1-\lambda_i^2}, 1\leq l\leq p_1$. Then
\begin{align*}
[\bD^{-1}\bB^{(2)}]_{i-k^*, j}&=\frac{1}{\lambda_k-\Delta/2-\lambda_i} \frac{1}{n}\sum\limits_{\alpha=1}^n \Big\{(1-\lambda_j^2)\lambda_i\lambda_j \bx_{\alpha i}\bx_{\alpha j} -\lambda_j^2\sqrt{1-\lambda_i^2}\sqrt{1-\lambda_j^2}\bz_{\alpha i}\bz_{\alpha j}\\
&~~~+(1-\lambda_j^2)\lambda_j\sqrt{1-\lambda_i^2}\bz_{\alpha i}\bx_{\alpha j}+(1-\lambda_j^2)\lambda_i\sqrt{1-\lambda_i^2}\bz_{\alpha j}\bx_{\alpha i}\Big\}.
\end{align*}
In this way, $\{\bx_{\alpha i}, \bz_{\alpha i}, 1\leq i\leq p_1, 1\leq \alpha\leq n\}$ are mutually independent standard gaussian random variables. For any given pair of vectors $\bu\in\mathbb{R}^{p_1-k^*}, \bv\in\mathbb{R}^k$,
\begin{align*}
\bu^\top\bD^{-1}\bB^{(2)}\bv&=\frac{1}{n}\sum_{\alpha=1}^n \sum_{i=k^*+1}^{p_1}\sum_{j=1}^k \frac{\bu_{i-k}\bv_j}{\lambda_k-\Delta/2-\lambda_i}\Big\{(1-\lambda_j^2)\lambda_i\lambda_j \bx_{\alpha i}\bx_{\alpha j} -\lambda_j^2\sqrt{1-\lambda_i^2}\sqrt{1-\lambda_j^2}\bz_{\alpha i}\bz_{\alpha j}\\
&~~~+(1-\lambda_j^2)\lambda_j\sqrt{1-\lambda_i^2}\bz_{\alpha i}\bx_{\alpha j}+(1-\lambda_j^2)\lambda_i\sqrt{1-\lambda_i^2}\bz_{\alpha j}\bx_{\alpha i} \Big\}\\
&\doteq \frac{1}{n}\sum_{\alpha=1}^n \bw_{\alpha},
\end{align*}
where $\bw_1, \cdots, \bw_n$ are $i.i.d.$ quadratic forms of the concatenated vector $(\bx^\top, \bz^\top)$ and the quadratic form can be represented by a matrix $\bA$. In order to apply Lemma~\ref{HW-inequality}, we first compute,
\begin{align*}
\|\bA\|_F^2&=\sum_{i=k^*+1}^{p_1}\sum_{j=1}^k \frac{\bu_{i-k}^2\bv_j^2}{(\lambda_k-\Delta/2-\lambda_i)^2}\Big\{(1-\lambda_j^2)^2\lambda_i^2\lambda_j^2 +\lambda_j^4(1-\lambda_i^2)(1-\lambda_j^2)\\
&~~~+(1-\lambda_j^2)^2\lambda_j^2(1-\lambda_i^2)+(1-\lambda_j^2)^2\lambda_i^2(1-\lambda_i^2) \Big\}\\
&=\sum_{i=k^*+1}^{p_1}\sum_{j=1}^k \frac{\bu_{i-k}^2\bv_j^2}{(\lambda_k-\Delta/2-\lambda_i)^2} \left(1-\lambda_j^2\right)\left(\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2\right).
\end{align*}
By definition,
\[
\sum_{i=k^*+1}^{p_1}\sum_{j=1}^k \bu_{i-k}^2\bv_j^2=1.
\]
Then $\|\bA\|_2^2, \|\bA\|_F^2$ can be upper bounded by
\begin{align*}
\|\bA\|_2^2\leq \|\bA\|_F^2&\leq \max_{1\leq j\leq k< k^*+1\leq i\leq p_1}\frac{(1-\lambda_j^2)(\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2)}{(\lambda_k-\Delta/2-\lambda_i)^2} \\
&\leq \max_{1\leq j\leq k<k^*+1\leq i\leq p_1}\frac{(1-\lambda_k^2)(\lambda_i^2(1-\lambda_j^2)+\lambda_j^2(1-\lambda_i^2))}{(\lambda_k-\Delta/2-\lambda_i)^2} \\
&\leq (1-\lambda_k^2) \max_{1\leq j\leq k<k^*+1\leq i\leq p_1} \frac{2(1-\lambda_i^2)}{(\lambda_k-\Delta/2-\lambda_i)^2} \\
&\leq (1-\lambda_k^2) \max_{1\leq j\leq k<k^*+1\leq i\leq p_1} \frac{2(1-\lambda_{k+1}^2)}{(\lambda_k-\Delta/2-\lambda_{k+1})^2} \\
&\leq 8\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{\Delta^2}\doteq K^2,
\end{align*}
where the second last inequality is due to the fact that for $\lambda>\lambda_{k+1}$, $f(x)=\frac{1-x^2}{(\lambda-x)^2}$ is increasing in the interval $[0, \lambda_{k+1}]$.
Therefore, Lemma~\ref{HW-inequality} implies that
\begin{equation}
P\left\{|\bw_\alpha|\geq t \right\}\leq 2exp\left\{-c_0\min\left(\frac{t^2}{K^2}, \frac{t}{K}\right)\right\}.
\end{equation}
Observe that $\forall t\geq 0, \min\left(1, 2exp\left\{-c_0\min\left(\frac{t^2}{K^2}, \frac{t}{K}\right)\right\}\right)\leq exp\left\{1-c_0\frac{t}{K}\right\}$, then
\begin{equation}
P\left\{|\bw_\alpha|\geq t \right\}\leq 2exp\left\{-c_0\left(\frac{t}{K}-1\right)\right\}.
\end{equation}
By Definition 5.13 in \cite{vershynin2010introduction} , $\bw_\alpha$ is sub-exponential random variable with $\|\bw_\alpha\|_{\psi_1}\leq c_1K$ for some universal constant $c_1$. Let $\widetilde{\delta}=\max\{\delta, \delta^2\}$. Apply Bernstein inequality (Lemma~\ref{Bernstein}) to $\bw_{\alpha}/K$ with $\ba_i=1/\sqrt{n}$,
\begin{align*}
P\left\{|\frac{1}{\sqrt{n}}\sum_{\alpha=1}^n \bw_\alpha/K |\geq \widetilde{\delta}/2 \right\}&\leq 2exp\left\{-c_2n\min\left(\frac{\widetilde{\delta}^2}{4c_1^2}, \frac{\widetilde{\delta}}{2c_1}\right)\right\}\\
&\leq 2exp\left\{-\frac{c_2}{1+4c_1^2}n\delta^2\right\}\\
&\leq 2exp\left\{-\frac{c_2}{1+4c_1^2}(C^2p_1+t^2)\right\}.
\end{align*}
\textbf{Step 3. Union Bound. } By Lemma~\ref{CoverNum}, we can choose $1/4$-net such that
%$|\mathcal{N}(\mathbb{S}^{p-1}, \epsilon)|\leq 9^p$
\begin{align*}
P\left\{\max\limits_{\bu_\epsilon, \bv_\epsilon}\bu_\epsilon^\top \bB^{(2)}\bv_\epsilon \geq K\widetilde{\delta}/2 \right\}&\leq 9^{p_1-k^*}9^{k}\times 2exp\left\{-\frac{c_2}{1+4c_1^2}(C^2p_1+t^2)\right\}\\
&\leq 2exp\left\{-\frac{c_2}{1+4c_1^2}t^2\right\},
\end{align*}
where the second inequality follows if we choose $C\geq \sqrt{\frac{(1+4c_1^2)\text{log}9}{c_2}}$. We finish the proof by choosing $c=\frac{c_2}{1+4c_1^2}$. The expectation bound can be obtained using the formula
\[
E[X]=\int_0^{+\infty} P(X\geq t) dt
\]
where $X$ is nonnegative random variable. The calculation is essentially the same as the proof in Section~\ref{sec::integration} and we leave out the details.
\subsection{Proof of Lemma~\ref{lemma-Bu}}
\label{sec::Bu}
The proof is essentially the same as the proof for Lemma~\ref{lemma-Bl} except that the term $\|\bA\|_F^2$ will be different (but simpler) and is sketched as follows,
\begin{align*}
\|\bA\|_F^2&=\sum_{i=k+1}^{k^*}\sum_{j=1}^k \bu_{i-k}^2\bv_j^2\Big\{(1-\lambda_j^2)^2\lambda_i^2\lambda_j^2 +\lambda_j^4(1-\lambda_i^2)(1-\lambda_j^2)\\
&~~~+(1-\lambda_j^2)^2\lambda_j^2(1-\lambda_i^2)+(1-\lambda_j^2)^2\lambda_i^2(1-\lambda_i^2) \Big\}\\
&=\sum_{i=k+1}^{k^*}\sum_{j=1}^k \bu_{i-k}^2\bv_j^2\left(1-\lambda_j^2\right)\left(\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2\right)\\
&\leq \max_{1\leq j\leq k<i\leq k^*}(1-\lambda_k^2)(\lambda_i^2(1-\lambda_j^2)+\lambda_j^2(1-\lambda_i^2))\\
&\leq 2(1-\lambda_k^2)\max_{1\leq j\leq k<i\leq k^*} (1-\lambda_i^2).
\end{align*}
Notice that by definition, for $k+1\leq i\leq k^*$, $\lambda_i\geq 2\lambda_k-1-\Delta$, then
\begin{align*}
\|\bA\|_F^2&\leq 2(1-\lambda_k^2)(1+\lambda_i)(2-2\lambda_k+\Delta)\\
&\leq 2(1-\lambda_k^2)(1+\lambda_{k+1})2(1-\lambda_{k+1})\\
&\leq 4(1-\lambda_k^2)(1-\lambda_{k+1}^2).
\end{align*}
The rest of the argument proceeds in the same way as in the proof of Lemma~\ref{lemma-Bl}.
\subsection{Proof of Lemma~\ref{lemma-high-prob}}
\label{sec::high-prob}
By definition, $\pxh^\top\sxh\pxh=\bI_{p_1}$, then
\[
\pxh^\top\pxh-\bI_{p_1}=-\pxh^\top(\sxh-\bI_{p_1})\pxh.
\]
Notice that $\sxh^{\half}\pxh\in\mc{O}(p_1)$,
\begin{align*}
\|\pxh^\top\pxh-\bI_{p_1}\|&\leq \|\pxh^\top(\sxh-\bI_{p_1})\pxh\|\leq \|\pxh^\top\sxh^{1/2}\|\|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|\|\sxh^{1/2}\pxh\|\\
&=\|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|.
\end{align*}
As a submatrix,
\begin{align*}
\|\pxhk^\top\pxhk-\bI_k\|&\leq \|\sxh^{-1/2}(\sxh-\bI_{p_1})\sxh^{-1/2}\|\\
&\leq \|\sxh^{-1}\| \|\sxh-\bI_{p_1}\|\\
&\leq \frac{1}{1-\|\sxh-\bI_{p_1}\|}\|\sxh-\bI_{p_1}\|\\
&\leq \frac{\|\sh_z-\s_z\|}{1-\|\sh_z-\s_z\|},
\end{align*}
which implies that
\begin{align*}
\sigma_k^2(\pxhk)\geq 1- \frac{\|\sh_z-\s_z\|}{1-\|\sh_z-\s_z\|}, ~ \|\pxhk\|^2\leq 1+\frac{\|\sh_z-\s_z\|}{1-\|\sh_z-\s_z\|}.
\end{align*}
Notice that $\|\s_z\|\leq 2$. By Lemma~\ref{lemma-vershynin}, for any given positive constant $\tau$, there exists constants $c, C$ depending on $\tau$ such that when $n\geq C(p_1+p_2)$,
\[
\|\sh_z-\s_z\|\leq \tau
\]
holds with probability at least $1-e^{-cn}$. Choose $\tau$ small enough such that $\sigma_k^2(\pxhk)\geq 1/2$ and $\|\pxhk\|^2\leq 3/2$. By the same argument,
\[
\sigma_k^2(\pyhk)\geq 1/2, \|\pyhk\|^2\leq 3/2
\]
will hold as well and therefore
\begin{align*}
(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|)\leq (2+\|\s_z\|+\tau)^2\times 3\times (1+\tau)\leq 150
\end{align*}
\subsection{Proof of Lemma~\ref{lemma-B-Fnorm}}
\label{sec::B-Fnorm}
We can write down explicitly, for $1\leq j\leq k< i\leq p_1$
\[
[\bB]_{i-k, j}=\frac{1}{n}\sum\limits_{\alpha=1}^n \left(\lambda_j \bx_{\alpha i} \by_{\alpha j}- \lambda_j^2 \bx_{\alpha i} \bx_{\alpha j}+ \lambda_i \bx_{\alpha j} \by_{\alpha i} - \lambda_i\lambda_j \by_{\alpha i}\by_{\alpha j}\right).
\]
Notice that $(\bx_{\alpha i}, \by_{\alpha i})$ are mutually uncorrelated pairs for any $1\leq \alpha\leq n, 1\leq i\leq p_1$. It is easy to compute
\begin{align*}
\E [\bB]_{i-k, j}^2&=(1-\lambda_j^2) (\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2)/n,
\end{align*}
and thus
\begin{align*}
\E \|\wt{\bD}^{-1}\bB\|_F^2 &=\frac{1}{n} \sum_{1\leq j\leq k<i\leq p_1} (1-\lambda_j^2) \frac{\lambda_i^2+\lambda_j^2-2\lambda_i^2\lambda_j^2}{(\lambda_k-\lambda_i)^2}\\
&\leq \frac{1-\lambda_k^2}{n}\sum_{1\leq j\leq k<i\leq p_1} \frac{\lambda_i^2(1-\lambda_j^2)+\lambda_j^2(1-\lambda_i^2)}{(\lambda_k-\lambda_i)^2}\\
&\leq \frac{2(1-\lambda_k^2)}{n}\sum_{1\leq j\leq k<i\leq p_1} \frac{1-\lambda_i^2}{(\lambda_k-\lambda_i)^2}\\
&\leq \frac{2(1-\lambda_k^2)}{n}\sum_{1\leq j\leq k<i\leq p_1} \frac{1-\lambda_{k+1}^2}{(\lambda_k-\lambda_{k+1})^2} \\
&\leq \frac{2(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{n\Delta^2} (p_1-k)k,
\end{align*}
where the second last inequality is due to the fact that for $\lambda>\lambda_{k+1}$, $f(x)=\frac{1-x^2}{(\lambda-x)^2}$ is increasing in the interval $[0, \lambda_{k+1}]$.
\subsection{Lower Bound: Proof of Theorem~\ref{thm:lower}}
\input{lower}
\subsection{Technique Lemmas}
\begin{lemma}
For any matrices $\U_1, \U_2\in \mc{O}(p, k)$,
\begin{equation}
\inf\limits_{\Q\in \mc{O}(k, k)}\|\U_1-\U_2\Q\|_F\leq \|\bP_{\U_1}-\bP_{\U_2}\|_F
\nonumber
\end{equation}
\label{subspace-metric}
\end{lemma}
See Section~\ref{sec::subspace-metric} for the proof.
\begin{lemma}(\cite{wedin1983angles})
For $\U_i\in O(p, k)$ and $\bP_i=\bU_i\bU_i^\top, ~i=1, 2$,
\begin{equation}
\begin{gathered}
\|\bP_1-\bP_2\|_F^2=2\|(\bI_p-\bP_1)\bP_2\|_F^2=2\|(\bI_p-\bP_2)\bP_1\|_F^2,\\
\|\bP_1-\bP_2\|^2=\|(\bI_p-\bP_1)\bP_2\|^2=\|(\bI_p-\bP_2)\bP_1\|^2=1-\sigma_{\min}^2\left(\bP_1\bP_2\right)
\nonumber
\end{gathered}
\end{equation}
\label{lemma-projection}
\end{lemma}
%\begin{lemma}(Wedin's $sin\theta$ Theorem, \cite{wedin1972perturbation})
%Let $\bA, \bE\in \mathbb{R}^{n\times m}$ and $\bB=\bA+\bE$. Suppose
%$$\bA=\bA_1+\bA_2,\, \bB=\bB_1+\bB_2$$
%where $\bA_1^\top\bA_2=\bA_2^\top\bA_1=\bB_1^\top\bB_2=\bB_2^\top\bB_1=\mtx{0}$ and $\sigma_{\min}(\bA_1)\geq \alpha+\delta, \sigma_{\min}(\bB_2)\leq \alpha$. Then for every unitary invariant norm,
%\begin{equation}
%\|\sin \theta (R(\bA_1), R(\bB_1))\|\,,\,\| \sin \theta (R(\bA_1^\top), R(\bB_1^\top))\|\leq \frac{\max\big\{\| \bP_{R(\A_1)}\bE \|, \| \bE \bP_{R(\A_1^\top)}\|\big \}}{\delta}
%\nonumber
%\end{equation}
%\label{wedin}
%where $\|\sin\theta (R(\bA_1), R(\bB_1))\|\doteq\|(\bI-\bP_{R(\B_1)})\bP_{R(\A_1)}\|$
%\end{lemma}
The following variant of Wedin's $\sin\theta$ law \citep{wedin1972perturbation} is proved in Proposition 1 of \cite{cai2015optimal}
% and Theorem 2 of \cite{yu2014useful}.
\begin{lemma}
For $\bA, \bE\in \mathbb{R}^{m\times n}$ and $\hbA=\bA+\bE$, define the singular value decomposition of $\bA$ and $\hbA$ as
\[
\bA=\bU\bD\bV^\top,~ \hbA=\hbU\hbD\hbV^\top.
\]
Then the following perturbation bound holds,
\[
\left\|(\bI-\bP_{\bU_{1:k}})\bP_{\hbU_{1:k}}\right\|=\left\|\bP_{\bU_{1:k}}-\bP_{\hbU_{1:k}}\right\|\leq \frac{2\|\bE\|}{\sigma_{k}(\bA)-\sigma_{k+1}(\bA)},
\]
where $\sigma_k(\bA), \sigma_{k+1}(\bA)$ are the $k_{th}$ and $(k+1)_{th}$ singular values of $\bA$ respectively.
\label{lemma-wedin}
\end{lemma}
\begin{lemma}(Covariance Matrix Estimation, \cite{vershynin2010introduction}, Remark 5.40)
Assume $\bA\in \mathbb{R}^{n\times p}$ has independent sub-gaussian random rows with second moment matrix $\bSigma$. Then there exists universal constant $C$ such that for every $t\geq 0$, the following inequality holds with probability at least $1-e^{-ct^2}$,
\begin{equation}
\|\frac{1}{n}\bA^\top\bA-\bSigma\|\leq \max\{\delta, \delta^2\}\|\bSigma\| \quad\quad \delta=C\sqrt{\frac{p}{n}}+\frac{t}{\sqrt{n}}.
\end{equation}
\label{lemma-vershynin}
\end{lemma}
\begin{lemma}
Assume $\bA\in \mathbb{R}^{n\times p}, n\geq p$ has independent sub-gaussian random rows with second moment matrix $\bSigma$. Then there exists universal constant $C$ such that
\begin{equation}
\E \|\frac{1}{n}\bA^\top\bA-\bSigma\|^4 \leq C\frac{p^2}{n^2}\|\s\|^4.
\end{equation}
\label{lemma-expectation-covariance}
\end{lemma}
See Section~\ref{sec::integration} for the proof.
\begin{lemma}(Bernstein inequality, \cite{vershynin2010introduction}, Proposition 5.16)
Let $X_1, \cdots, X_n$ be independent centered sub-exponential random variables and $K=\max_{i}\|X_i\|_{\psi_1}$. Then for every $\ba=(\ba_1, \cdots, \ba_n)\in\mathbb{R}^n$ and every $t\geq 0$, we have
\begin{equation}
P\left\{|\sum_{i=1}^n \ba_i X_i|\geq t \right\}\leq 2exp\left\{-c\min\left(\frac{t^2}{K^2\|\ba\|_2^2}, \frac{t}{K\|\ba\|_{\infty}}\right)\right\}.
\end{equation}
\label{Bernstein}
\end{lemma}
\begin{lemma}(Hanson-Wright inequality, \cite{hw2013})
Let $\bx=(x_1, \cdots, x_p)$ be a random vectors with independent components $x_i$ which satisfy $\E x_i=0$ and $\|x_i\|_{\psi_2}\leq K$, Let $\bA\in\mathbb{R}^{p\times p}$. Then there exists universal constant $c$ such that for every $t\geq 0$,
\begin{equation}
P\left\{|\bx^\top \bA\bx-\E\bx^\top \bA\bx|\geq t \right\}\leq 2exp\left\{-c\min\left(\frac{t^2}{K^4\|\A\|_F^2}, \frac{t}{K^2\|\A\|_2}\right)\right\}.
\end{equation}
\label{HW-inequality}
\end{lemma}
\begin{lemma}(Covering Number of the Sphere, \cite{vershynin2010introduction}, Lemma 5.2). The unit Euclidean sphere $\mathbb{S}^{n-1}$ equipped with the Euclidean metric satisfies for every $\epsilon> 0$ that
\begin{equation}
|\mathcal{N}(\mathbb{S}^{n-1}, \epsilon)|\leq (1+\frac{2}{\epsilon})^n,
\end{equation}
where $\mathcal{N}(\mathbb{S}^{n-1}, \epsilon)$ is the $\epsilon$-net of $\,\mathbb{S}^{n-1}$ with minimal cardinality.
\label{CoverNum}
\end{lemma}
%\begin{lemma}(von Neumann's Trace Inequality)
%Let $\bA, \bB$ be two complex $n\times n$ matrices, then
%\begin{equation}
%|tr(\bA\bB)|\leq \sum_{i=1}^n \sigma_i(\bA)\sigma_i(\bB)
%\end{equation}
%where $\sigma_i(\cdot)$ is the $i_{th}$ singular value of a matrix.
%\label{lemma-Neumann}
%\end{lemma}
%!TEX root = SpikeCCA.tex
\label{sec::lower-proof}
\subsubsection{On Kullback-Leibler Divergence}
\begin{lemma}
For $i=1, 2$ and $p_2\geq p_1\geq k$, let $\begin{bmatrix}
\bU_{(i)}, ~\bW_{(i)}
\end{bmatrix}\in \mc{O}(p_1, p_1), \begin{bmatrix}
\bV_{(i)}, ~\bZ_{(i)}
\end{bmatrix}\in \mc{O}(p_2, p_1)$ where $\bU_{(i)}\in\mathbb{R}^{p_1\times k}, \bV_{(i)}\in\mathbb{R}^{p_2\times k}$. For $0\leq \lambda_2<\lambda_1<1$, let $\Delta=\lambda_1-\lambda_2$ and define
\[
\s_{(i)}=\begin{bmatrix} \sx & \sx^{1/2}(\lambda_1\bU_{(i)}\bV_{(i)}^\top+\lambda_2\bW_{(i)}\bZ_{(i)}^\top)\sy^{1/2}\\ \sy^{1/2}(\lambda_1\V_{(i)}\U_{(i)}^\top+\lambda_2\bZ_{(i)}\bW_{(i)}^\top) \sx^{1/2}& \sy \end{bmatrix}~i=1,2,
\]
Let $\P_{(i)}$ denote the distribution of a random $i. i. d. $ sample of size $n$ from $N(0, \s_{(i)})$. If we further assume
\begin{align}
[\bU_{(1)}, \bW_{(1)}]\begin{bmatrix}
\smallskip
\bV_{(1)}^\top\\
\bZ_{(1)}^\top
\end{bmatrix}=[\bU_{(2)}, \bW_{(2)}]\begin{bmatrix}
\smallskip
\bV_{(2)}^\top\\
\bZ_{(2)}^\top
\end{bmatrix},
\label{eq::kl-construction}
\end{align}
Then one can show that
\[
D(\P_{(1)}||\P_{(2)})=\frac{n\Delta^2(1+\lambda_1\lambda_2)}{2(1-\lambda_1^2)(1-\lambda_2^2)}\|\U_{(1)}\V_{(1)}^\top-\U_{(2)}\V_{(2)}^\top\|_F^2.
\]
\label{KL-distance}
\end{lemma}
The construction in \eqref{eq::kl-construction} is crucial to prove the lower bound. The proof of this lemma can be found in Section~\ref{sec::KL}.
\subsubsection{Packing Number and Fano's Lemma}
The following result on the packing number is based on the metric entropy of the Grassmannian manifold $G(k,r)$ due to \cite{szarek1982nets}. We use the version adapted from Lemma 1 of \cite{cai2013sparse} which is also used in \cite{gao2015minimax}.
\begin{lemma}
For any fixed $\U_0\in \mc{O}(p, k)$ and $\mc{B}_{\epsilon_0}=\{\U\in \mc{O}(p, k): \|\bU\bU^\top-\U_0\U_0^\top\|_F\leq \epsilon_0 \}$ with $\epsilon_0\in (0, \sqrt{2[k\wedge (p-k)]}\,)$. Define the semi-metric $\rho(\cdot, \cdot)$ on $\mc{B}_{\epsilon_0}$ by
\[
\rho(\bU_1, \bU_2)=\|\bU_1\bU_1^\top - \bU_2\bU_2^\top\|_F.
\]
Then there exists universal constant $C$ such that for any $\alpha\in (0, 1)$, the packing number $\mathcal{M}(\mc{B}_{\epsilon_0}, \rho, \alpha\epsilon_0)$ satisfies
\[
\mathcal{M}(\mc{B}_{\epsilon_0}, \rho, \alpha\epsilon_0)\geq \left(\frac{1}{C\alpha}\right)^{k(p-k)}.
\]
\label{lemma-packing-szarek}
\end{lemma}
The following corollary is used to prove the lower bound.
\begin{corollary}
If we change the set in Lemma~\ref{lemma-packing-szarek} to $\wt{\mc{B}}_{\epsilon_0}=\{\U\in \mc{O}(p, k): \|\bU-\bU_0\|_F\leq \epsilon_0 \}$, then we still have
\[
\mathcal{M}(\wt{\mc{B}}_{\epsilon_0}, \rho, \alpha\epsilon_0)\geq \left(\frac{1}{C\alpha}\right)^{k(p-k)}.
\]
\label{corollary-packing}
\end{corollary}
\begin{proof}
Apply Lemma~\ref{lemma-packing-szarek} to $\mc{B}_{\epsilon_0}$, there exists $\bU_1, \cdots, \bU_{n}$ with $n\geq \left(1/C\alpha\right)^{k(p-k)}$ such that
\[
\|\bU_{i}\bU_{i}^\top-\bU_0\bU_0^\top\|_F\leq \epsilon_0,~ 1\leq i\leq n, ~
\|\bU_{i}\bU_{i}^\top-\bU_j\bU_j^\top\|_F\geq \alpha\epsilon_0, 1\leq i\leq j\leq n.
\]
Define $\wt{\bU}_i=\arg\min\limits_{\bU\in\{\bU_i\bQ, ~\bQ\in\mc{O}(k)\}} \|\bU-\bU_0\|_F$, by Lemma~\ref{subspace-metric},
\[
\|\wt{\bU}_i-\bU_0\|_F\leq \|\wt{\bU}_i\wt{\bU}_i^\top-\bU_0\bU_0^\top\|_F\leq \epsilon_0.
\]
Therefore, $\wt{\bU}_1, \cdots, \wt{\bU}_{n}\in\wt{\mc{B}}_{\epsilon_0}$ and
\[
\|\wt{\bU}_i\wt{\bU}_i^\top-\wt{\bU}_j\wt{\bU}_j^\top\|_F=\|\bU_{i}\bU_{i}^\top-\bU_j\bU_j^\top\|_F\geq \alpha\epsilon_0.
\]
which implies,
\[
\mathcal{M}(\wt{\mc{B}}_{\epsilon_0}, \rho, \alpha\epsilon_0)\geq n\geq \left(\frac{1}{C\alpha}\right)^{k(p-k)}.
\]
\end{proof}
\begin{lemma}[Fano's Lemma \cite{yu1997assouad}] Let $(\Theta, \rho)$ be a (semi)metric space and $\{\P_{\theta}: \theta\in\Theta\}$ a collection of probability measures. For any totally bounded $T\subset\Theta$, denote $\mathcal{M}(T, \rho, \epsilon)$ the $\epsilon$-packing number of $T$ with respect to the metric $\rho$, i.e. , the maximal number of points in $T$ whoese pairwise minimum distance in $\rho$ is at least $\epsilon$. Define the Kullback-Leibler diameter of $T$ by
\[
d_{\text{KL}}(T)=\sup\limits_{\theta, \theta^\prime\in T} D(\P_{\theta}||\P_{\theta^\prime}).
\]
Then,
\[
\inf\limits_{\htheta}\sup\limits_{\theta\in\Theta} \E_{\theta} \left[\rho^2 (\htheta, \theta) \right]\geq \sup\limits_{T\subset\Theta}\sup\limits_{\epsilon>0} \frac{\epsilon^2}{4}\Big(1-\frac{d_{\text{KL}}(T)+\text{log }2}{\text{log } \mathcal{M}(T, \rho, \epsilon)}\Big)
\]
\label{Lemma-exp-Fano}
\end{lemma}
\subsubsection{Proof of Lower Bound}
%Here we only prove the lower bound for the Frobenius norm loss. The proof can be easily generalized to the spectral norm loss by only considering rank one perturbation and it can be found in the appendix. \\
%\medskip
For any fixed $\begin{bmatrix}
\bU_{(0)}, ~\bW_{(0)}
\end{bmatrix}\in\mathcal{O}(p_1, p_1)$ and $ \begin{bmatrix}
\bV_{(0)}, ~\bZ_{(0)}
\end{bmatrix}\in \mathcal{O}(p_2, p_1)$ where $\bU_{(0)}\in\mathbb{R}^{p_1\times k}, \bV_{(0)}\in\mathbb{R}^{p_2\times k}, \bW_{(0)}\in\mathbb{R}^{p_1\times (p_1-k)}, \bV_{(0)}\in\mathbb{R}^{p_2\times (p_2-k)}$, define
\begin{align*}
\mathcal{H}_{\epsilon_0}=\Big\{\big(\bU, \bW, \bV, \bZ\big): ~ \begin{bmatrix}
\bU, ~\bW
\end{bmatrix}\in \mc{O}(p_1, p_1)~\text{with}~ \bU\in \mathbb{R}^{p_1\times k},~ \begin{bmatrix}
\bV, ~\bZ
\end{bmatrix}\in \mc{O}(p_2, p_1)\\~\text{with}~\bV\in \mathbb{R}^{p_2\times k}
,\|\bU-\bU_{(0)}\|_F\leq \epsilon_0, ~
[\bU, \bW]\begin{bmatrix}
\smallskip
\bV^\top\\
\bZ^\top
\end{bmatrix}=[\bU_{(0)}, \bW_{(0)}]\begin{bmatrix}
\smallskip
\bV_{(0)}^\top\\
\bZ_{(0)}^\top
\end{bmatrix}
\Big\}.
\end{align*}
%\begin{align*}
%\mathcal{H}_{\epsilon_0}=\Big\{\big(\bU, \bW, \bV, \bZ\big): ~ \begin{bmatrix}
%\bU, ~\bW
%\end{bmatrix}\in \mc{O}(p_1, p_1), \bU\in \mathbb{R}^{p_1\times k}, \bW\in\mathbb{R}^{p_1\times (p_1-k)}, \bV\in \mathbb{R}^{p_2\times k}\\ \bZ\in \mathbb{R}^{p_2\times (p_2-k)}
%\begin{bmatrix}
%\bV, ~\bZ
%\end{bmatrix}\in \mc{O}(p_2, p_1),\|\bU-\bU_{(0)}\|_F\leq \epsilon_0, ~
%[\bU, \bW]\begin{bmatrix}
%\smallskip
%\bV^\top\\
% \bZ^\top
%\end{bmatrix}=[\bU_{(0)}, \bW_{(0)}]\begin{bmatrix}
%\smallskip
%\bV_{(0)}^\top\\
% \bZ_{(0)}^\top
%\end{bmatrix}
% \Big\}.
%\end{align*}
%For any fixed $\sx\in\mathbb{S}^{p_1}_+, \sy\in\mathbb{S}^{p_2}_+$, choose $\px_{(0)}, \py_{(0)}$ such that $\U_{(0)}=\sx^{\half}\px_{(0)}=\begin{bmatrix}\bI_k, 0\end{bmatrix}^\top\in O(p_1, k), \V_{(0)}=\sy^{\half}\py_{(0)}=\begin{bmatrix}\bI_k, 0\end{bmatrix}^\top\in O(p_2, k)$. For some $\epsilon_0>0$ to be specified later, define
For any fixed $\sx\in\mathbb{S}^{p_1}_+, \sy\in\mathbb{S}^{p_2}_+$ with $\kappa(\sx)=\kappa_x, \kappa(\sy)=\kappa_y$, consider the parametrization $\s_{xy}=\sx\px\Lambda\py^\top\sy$, for $0\leq \lambda_{k+1}<\lambda_k<1$, define
\begin{align*}
\mathcal{T}_{\epsilon_0}=\Big\{
\s=\begin{bmatrix} \sx & \sx^{1/2}(\lambda_k\bU\bV^\top+\lambda_{k+1}\bW\bZ^\top)\sy^{1/2}\\ \sy^{1/2}(\lambda_k\V\U^\top+\lambda_{k+1}\bZ\bW^\top) \sx^{1/2}& \sy \end{bmatrix},\\
\px=\sx^{-1/2}[\bU,\bW], \py=\sy^{-1/2}[\bV, \bZ], \big(\bU, \bW, \bV, \bZ\big)\in\mathcal{H}_{\epsilon_0} \Big\}.
\end{align*}
It is straightforward to verify that $\mathcal{T}_{\epsilon_0}\subset \mathcal{F}(p_1, p_2, k, \lambda_k,\lambda_{k+1},\kappa_x, \kappa_y)$. For any $\s_{(i)}\in\mathcal{T}_{\epsilon_0}, ~i=1, 2$, they yield to the parametrization,
\begin{align*}
\s_{(i)}=\begin{bmatrix} \sx & \sx^{1/2}(\lambda_k\bU_{(i)}\bV_{(i)}^\top+\lambda_{k+1}\bW_{(i)}\bZ_{(i)}^\top)\sy^{1/2}\\ \sy^{1/2}(\lambda_k\V_{(i)}\U_{(i)}^\top+\lambda_{k+1}\bZ_{(i)}\bW_{(i)}^\top) \sx^{1/2}& \sy \end{bmatrix},
\end{align*}
where $\big(\bU_{(i)}, \bW_{(i)}, \bV_{(i)}, \bZ_{(i)}\big)\in\mathcal{H}_{\epsilon_0}$ and the leading-$k$ canonical vectors are $\pxk^{(i)}=\sx^{-\half}\bU_{(i)}, \pyk^{(i)}=\sy^{-\half}\bV_{(i)}$. We define a semi-metric on $\mathcal{T}_{\epsilon_0}$ as
\[
\rho(\s_{(1)}, \s_{(2)})=\left\|\bP_{\sx^{\half}\pxk^{(1)}}-\bP_{\sx^{\half}\pxk^{(2)}}\right\|_F=\left\| \bP_{\bU_{(1)}}-\bP_{\bU_{(2)}}\right\|_F.
\]
By Lemma~\ref{KL-distance},
\begin{align*}
D(\P_{\s_1}||\P_{\s_2})=\frac{n\Delta^2(1+\lambda_k\lambda_{k+1})}{2(1-\lambda_k^2)(1-\lambda_{k+1}^2)}\|\U_{(1)}\V_{(1)}^\top-\U_{(2)}\V_{(2)}^\top\|_F^2.
\end{align*}
Further by the definition of $d_{KL} (T)$,
\begin{align}
d_{KL} (T)&=\frac{n\Delta^2(1+\lambda_k\lambda_{k+1})}{2(1-\lambda_k^2)(1-\lambda_{k+1}^2)}\sup\limits_{\s_{(1)}, \s_{(2)}\in \mathcal{T}_{\epsilon_0}}\|\U_{(1)}\V_{(1)}^\top-\U_{(2)}\V_{(2)}^\top\|_F^2.
\label{eq-KL-obj}
\end{align}
%\begin{lemma}[Kullback-Leibler Diameter]
%\begin{align*}
%d_{KL} (\mc{T}_{\epsilon_0})&=\frac{n\Delta^2(1+\lambda_1\lambda_2)}{2(1-\lambda_1^2)(1-\lambda_2^2)}\sup\limits_{\s_{(1)}, \s_{(2)}\in \mathcal{H}_{\epsilon_0}}\|\U_{(1)}\V_{(1)}^\top-\U_{(2)}\V_{(2)}^\top\|_F^2\\
%&\leq \frac{8n\Delta^2(1+\lambda_1\lambda_2)}{(1-\lambda_1^2)(1-\lambda_2^2)}\epsilon_0^2
%\end{align*}
%\label{lemma-KL}
%\end{lemma}
To bound the Kullback-Leibler diameter, for any $\s_{(1)}, \s_{(2)}\in \mathcal{T}_{\epsilon_0}$, by definition,
\[
[\bU_{(1)}, \bW_{(1)}]\begin{bmatrix}
\smallskip
\bV_{(1)}^\top\\
\bZ_{(1)}^\top
\end{bmatrix}=[\bU_{(2)}, \bW_{(2)}]\begin{bmatrix}
\smallskip
\bV_{(2)}^\top\\
\bZ_{(2)}^\top
\end{bmatrix},
\]
which implies that they are singular value decompositions of the same matrix. Therefore, there exists $\bQ\in\mathcal{O}(p_1, p_1)$ such that
\begin{align}
[\bU_{(2)}, \bW_{(2)}]=[\bU_{(1)}, \bW_{(1)}]\bQ~,~[\bV_{(2)}, \bZ_{(2)}]=[\bV_{(1)}, \bZ_{(1)}]\bQ.
\label{eq-UV}
\end{align}
Decompose $\bQ$ into four blocks such that
\[
\bQ=\begin{bmatrix}
\bQ_{11}& \bQ_{12}\\
\bQ_{21} & \bQ_{22}
\end{bmatrix}.
\]
Substitute into \eqref{eq-UV},
\[
\bU_{(2)}=\bU_{(1)}\bQ_{11}+\bW_{(1)}\bQ_{21},~ \bV_{(2)}=\bV_{(1)}\bQ_{11}+\bZ_{(1)}\bQ_{21}.
\]
Then,
\begin{align*}
\|\bU_{(2)}-\bU_{(1)}\|_F^2&=\|\bU_{(1)}(\bQ_{11}-\bI_{k})+\bW_{(1)}\bQ_{21}\|_F^2\\
&=\|\bU_{(1)}(\bQ_{11}-\bI_{k})\|_F^2+\|\bW_{(1)}\bQ_{21}\|_F^2\\
&=\|\bQ_{11}-\bI_{k}\|_F^2+\|\bQ_{21}\|_F^2.
\end{align*}
The second equality is due to the fact that $\bU_{(1)}$ and $\bW_{(1)}$ have orthogonal column space and the third equality is valid because $\bU_{(1)}, \bW_{(1)}\in\mathcal{O}(p_1, k)$. By the same argument, we will have
\[
\|\bV_{(2)}-\bV_{(1)}\|_F^2=\|\bQ_{11}-\bI_{k}\|_F^2+\|\bQ_{21}\|_F^2.
\]
Notice that
\begin{align*}
\|\U_{(1)}\V_{(1)}^\top-\U_{(2)}\V_{(2)}^\top\|_F^2&=\|(\bU_{(1)}-\bU_{(2)})\bV_{(1)}+\bU_{(2)}(\bV_{(1)}-\bV_{(2)})\|_F^2\\
&\leq 2\|\bU_{(1)}-\bU_{(2)}\|_F^2+2\|\bV_{(1)}-\bV_{(2)}\|_F^2\\
&=4\|(\bU_{(1)}-\bU_{(2)})\|_F^2\\
&\leq 8\left(\|(\bU_{(1)}-\bU_{(0)})\|_F^2+\|(\bU_{(0)}-\bU_{(2)})\|_F^2\right)\\
&\leq 16\epsilon_0^2.
\end{align*}
Then, substitute into \eqref{eq-KL-obj}
\begin{align}
d_{KL} (T)\leq \frac{8n\Delta^2(1+\lambda_k\lambda_{k+1})}{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}\epsilon_0^2.
\label{eq-KL-bound}
\end{align}
Let $\mathcal{B}_{\epsilon_0}=\{\U\in O(p_1, k): \|\U-\U_{(0)}\|_F\leq \epsilon_0 \}$. Under the semi-metric $\wt{\rho}(\bU_{(1)}, \bU_{(2)})=\|\bU_{(1)}\bU_{(1)}^\top - \bU_{(2)}\bU_{(2)}^\top\|_F$, we claim that the packing number of $\mathcal{H}_{\epsilon_0}$ is lower bounded by the packing number of $\mathcal{B}_{\epsilon_0}$. To prove this claim, it suffices to show that for any $\bU\in\mathcal{B}_{\epsilon_0}$, there exists corresponding $\bW, \bV, \bZ$ such that $\left(\bU, \bW, \bV, \bZ\right)\in \mathcal{H}_{\epsilon_0}$. First of all, by definition, $\|\bU-\bU_{0}\|_F\leq \epsilon_0$. Let $\bW\in\mathcal{O}(p_1, p_1-k)$ be the orthogonal complement of $\bU$. Then $[\bU, \bW]\in\mathcal{O}(p_1, p_1)$ and therefore there exists $\bQ\in \mathcal{O}(p_1, p_1)$ such that
\[
[\bU, \bW]=[\bU_{(0)}, \bW_{0}]\bQ.
\]
Set $[\bV, \bZ]=[\bV_{(0)}, \bZ_{0}]\bQ\in \mathcal{O}(p_2, p_1)$, then
\[
[\bU, \bW]\begin{bmatrix}
\smallskip
\bV^\top\\
\bZ^\top
\end{bmatrix}=[\bU_{(0)}, \bW_{(0)}]\begin{bmatrix}
\smallskip
\bV_{(0)}^\top\\
\bZ_{(0)}^\top
\end{bmatrix},
\]
which implies $\left(\bU, \bW, \bV, \bZ\right)\in \mathcal{H}_{\epsilon_0}$. Let
\[
\epsilon=\alpha\epsilon_0=c\left(\sqrt{k\wedge(p_1-k)}\wedge\sqrt{\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{n\Delta^2(1+\lambda_k\lambda_{k+1})}k(p_1-k)}\right),
\]
where $c\in (0, 1)$ depends on $\alpha$ and is chosen small enough such that $\epsilon_0=\epsilon/\alpha\in (0, \sqrt{2[k\wedge (p_1-k)]}\,)$. By Corollary~\ref{corollary-packing},
\[
\mathcal{M}(\mathcal{T}_{\epsilon_0}, \rho, \alpha\epsilon_0)=\mathcal{M}(\mathcal{H}_{\epsilon_0}, \wt{\rho}, \alpha\epsilon_0)\geq \mathcal{M}(\mathcal{B}_{\epsilon_0}, \wt{\rho}, \alpha\epsilon_0)\geq \left(\frac{1}{C\alpha}\right)^{k(p_1-k)}.
\]
Apply Lemma~\ref{Lemma-exp-Fano} with $\mathcal{T}_{\epsilon_0}, \rho, \epsilon$,
\[
\inf\limits_{\pxhk}\sup_{\s\in \mathcal{F}} \E\left[~\left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk}\right\|_F^2 \right]\geq \sup\limits_{T\subset\Theta}\sup\limits_{\epsilon>0} \frac{\epsilon^2}{4}\left(1- \frac{8c^2k(p_1-k)+\text{log} 2}{k(p_1-k)\text{log}\frac{1}{C\alpha}}\right).
\]
Choose $\alpha$ small enough such that
\[
1- \frac{8c^2k(p_1-k)+\text{log} 2}{k(p_1-k)\text{log}\frac{1}{C\alpha}}\geq \frac{1}{2}.
\]
%If $n>[k\vee (p_1 - k)]\frac{(1-\lambda_1^2)(1-\lambda_2^2)}{\Delta^2(1+\lambda_1\lambda_2)}$,
Then the lower bound is reduced to
\begin{align*}
\inf\limits_{\pxhk}\sup_{\s\in \mathcal{F}} \E\left[~ \left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk}\right\|_F^2 \right]&\geq \frac{c^2}{8} \left\{\frac{(1-\lambda_k^2)(1-\lambda_{k+1}^2)}{n\Delta^2(1+\lambda_k\lambda_{k+1})}k(p_1-k)\wedge k\wedge(p_1-k)\right\}\\
&\geq C^2k\left\{\left(\frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{\Delta^2}\frac{p_1-k}{n}\right)\wedge 1\wedge \frac{p_1-k}{k}\right\}
\end{align*}
By symmetry,
%if $n>[k\vee (p_2 - k)]\frac{(1-\lambda_1^2)(1-\lambda_2^2)}{\Delta^2(1+\lambda_1\lambda_2)}$,
\[
\inf\limits_{\pyhk}\sup_{\s\in \mathcal{F}} \E\left[~ \left\| \bP_{\sy^{\half}\pyhk}-\bP_{\sy^{\half}\pyk}\right\|_F^2 \right]\geq C^2k\left\{\left(\frac{(1-\lambda_{k}^2)(1-\lambda_{k+1}^2)}{\Delta^2}\frac{p_1-k}{n}\right)\wedge 1\wedge \frac{p_1-k}{k}\right\}
\]
The lower bound for operator norm error can be immediately obtained by noticing that $\bP_{\sy^{\half}\pyhk}-\bP_{\sy^{\half}\pyk}$ has at most rank $2k$ and
\[
\left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk}\right\|^2\geq \frac{1}{2k}\left\| \bP_{\sx^{\half}\pxhk}-\bP_{\sx^{\half}\pxk}\right\|_F^2
\]
%!TEX root = SpikeCCA.tex
\subsection{Proof of Lemma~\ref{lemma-high-order}}
\label{sec::high-order}
%\input{highOrder}
In this section, we show how to control the higher order terms $\bdelta_6$ and $\bdelta_7$. The universal constants $C, C_1, c, \cdots$ might change from line to line. To facilitate presentation, we again introduce $\bz=(\bx^\top, \by^\top)^\top$ as the concatenation of $\bx$ and $\by$. Then
\[
\s_z=\begin{bmatrix}
\sx & \sxy\\
\syx&\sy
\end{bmatrix}=\begin{bmatrix}
\bI_{p_1} & \sxy\\
\syx&\bI_{p_2}
\end{bmatrix}, ~~\sh_z=\begin{bmatrix}
\sxh & \sxyh\\
\syxh&\syh
\end{bmatrix}.
\]
\begin{lemma}
There exists universal constant $C$ such that the following inequalities hold deterministically
\begin{gather*}
\|\sxh\|, \|\syh\|, \|\sxyh\|\leq \|\sh_z\|,\\
\|\sx-\sxh\|, \|\sy-\syh\|, \|\sxy-\sxyh\| \leq \|\s_z-\sh_z\|,\\
\|\hbLambda -\bLambda\|\leq \|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|\leq \|\s_z-\sh_z\|\left(2+\|\sh_z\|\right),\\
\begin{aligned}
\|\pxhk^l\|, \|\pyhk^l\|
%&\leq C\|\s_z-\sh_z\|\left(2+\|\sh_z\|\right)/\Delta+\|\s_z-\sh_z\|\left(\|\pxhk\|+ \|\pyhk\|\right)\\
&\leq C\|\s_z-\sh_z\|\left( \frac{2+\|\sh_z\|}{\Delta}+\|\pxhk\|+ \|\pyhk\|\right),
\end{aligned}
\end{gather*}
where $\Delta=\lambda_k-\lambda_{k+1}$ is the eigen-gap.
\label{lemma-matrix-inequality}
\end{lemma}
Lemma~\ref{lemma-matrix-inequality} and triangle inequality are frequently employed in this proof. Notice that $\sx^{ij}, \sy^{ij}, \sxy^{ij}, \syx^{ij}, 1\leq i, j\leq 2$ are sub-matrices of $\sx, \sy, \sxy, \syx$. We will also repeatedly use the fact that the operator norm of a matrix is no less than that of its sub-matrices.
\begin{align*}
\|\bdelta_1\|&\leq \|\pxhk^u(\hbLambda_1-\bLambda_1)+(\sxh^{11}-\bI_k)\pxhk^u\hbLambda_1-(\sxyh^{11}-\bLambda_1)\pyhk^u+\sxh^{12}\pxhk^l\hbLambda_1-\sxyh^{12}\pyhk^l\|\\
&\leq \|\pxhk\| \|\s_z-\sh_z\|(2+\|\sh_z\|)+\|\s_z-\sh_z\|\|\pxhk\|+\|\s_z-\sh_z\|\|\pyhk\|\\
&~~~~~+2C\|\s_z-\sh_z\|^2\left( \frac{2+\|\sh_z\|}{\Delta}+\|\pxhk\|+ \|\pyhk\|\right)\\
&\leq C_1\|\s_z-\sh_z\|(2+\|\sh_z\|)(\|\pxhk\|+ \|\pyhk\|)+C_2\|\s_z-\sh_z\|^2\left( \frac{2+\|\sh_z\|}{\Delta}+\|\pxhk\|+ \|\pyhk\|\right)\\
&\leq C_3 \|\s_z-\sh_z\|(2+\|\sh_z\|)(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|/\Delta).
\end{align*}
where in the last inequality we use $\|\s_z\|\leq 2$ and
\[
\|\s_z-\sh_z\|^2\left(\|\pxhk\|+ \|\pyhk\|\right)\leq \|\s_z-\sh_z\| (\|\s_z\|+\|\sh_z\|)\left(\|\pxhk\|+ \|\pyhk\|\right)
\]
By the same argument,
\begin{align*}
\|\bdelta_2\|\leq C_3 \|\s_z-\sh_z\|(2+\|\sh_z\|)(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|/\Delta).
\end{align*}
We can also bound $\bdelta_3, \bdelta_4$ in the same manner and will obtain,
\begin{align*}
\|\bdelta_3\|, \|\bdelta_4\|\leq C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)/\Delta
\end{align*}
Recall that $\bdelta_6=\sxh^{21}\bdelta_1\bLambda_1-\bLambda_2\syh^{21}\bdelta_1+\bdelta_5 $ and $\bdelta_5=-\bdelta_3\bLambda_1+\bLambda_2\bdelta_4$, then
\begin{align*}
\|\bA_1\circ\bdelta_6\|&\leq \|\bA_1\circ (\sxh^{21}\bdelta_1\bLambda_1-\bLambda_2\syh^{21}\bdelta_1+\bdelta_5)\|\\
&\leq\|\bA_1\circ (\sxh^{21}\bdelta_1\bLambda_1)\|+\|\bA_1\circ (\bLambda_2\syh^{21}\bdelta_1)\|+\|\bA_1\circ (\bdelta_3\bLambda_1)\|+\|\bA_1\circ(\bLambda_2\bdelta_4)\|\\
&=\|(\bA_1\bLambda_1)\circ (\sxh^{21}\bdelta_1)\|+\|(\bLambda_2\bA_1)\circ (\syh^{21}\bdelta_1)\|+\|(\bA_1\bLambda_1)\circ \bdelta_3\|+\|(\bLambda_2\bA_1)\circ\bdelta_4\|.
\end{align*}
By the same argument as in \eqref{eq:A1-hada},
\begin{align*}
\|\bA_1\circ\bdelta_6\|&\leq \frac{3}{2}\|\sxh^{21}\bdelta_1\|+\frac{1}{2}\| \syh^{21}\bdelta_1\|+\frac{3}{2}\| \bdelta_3\|+\frac{1}{2}\|\bdelta_4\|\\
&\leq 2\|\s_z-\sh_z\|\|\bdelta_1\|+\frac{3}{2}\| \bdelta_3\|+\frac{1}{2}\|\bdelta_4\|\\
&\leq C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|/\Delta)\\
&~~~~~+C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)/\Delta\\
&\leq C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|)/\Delta.
\end{align*}
By the same argument as in \eqref{eq:A2-hada},
\begin{align*}
\|\bA_1\circ\bA_2\circ\bdelta_6\|&\leq \frac{1}{\Delta}\|\bA_1\circ\bdelta_6\| \\
&\leq C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|)/\Delta^2.
\end{align*}
Recall that $\bdelta_7=\sxyh^{21}\bdelta_1+\sxh^{21}(\bLambda_1\bdelta_1+\bdelta_2\bLambda_1)+\bLambda_2\syh^{21}\bdelta_2+\bdelta_5$, then
\begin{align*}
\|\bdelta_7\|&\leq \|\s_z-\sh_z\|(2\|\bdelta_1\|+2\|\bdelta_2\|)+\|\bdelta_3\|+\|\bdelta_4\|\\
&\leq C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|/\Delta)\\
&~~~~~+C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)/\Delta\\
&\leq C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|)/\Delta.
\end{align*}
Again, by the same argument in \eqref{eq:A2-hada},
\begin{align*}
\|\bA_2\circ \bdelta_7\|\leq C\|\s_z-\sh_z\|^2(2+\|\sh_z\|)^2(\|\pxhk\|+ \|\pyhk\|)(1+ \|\s_z-\sh_z\|)/\Delta^2.
\end{align*}
\subsection{Proof of Lemma~\ref{lemma-matrix-inequality}}
The first two inequalities are trivial because the operator norm of a matrix is not less than that of its sub-matrices. Notice that $\hbLambda$ and $\bLambda$ are singular values of $\sxh^{-1/2} \sxyh\syh^{-1/2}$ and $\sxy$ respectively. Hence by Weyl's inequality,
\[
\|\hbLambda -\bLambda\|\leq \|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|.
\]
Further observe that
\begin{align*}
\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy&=(\bI_{p_1}-\sxh^{1/2})\sxh^{-1/2}\sxyh\syh^{-1/2}\\
&+ \sxh^{1/2}\sxh^{-1/2}\sxyh\syh^{-1/2}(\bI_{p_2}-\syh^{1/2})+(\sxyh-\sxy).
\end{align*}
%Then by triangle inequality and Lemma~\ref{lemma-matrix-root}
and $\|\sxh^{-1/2}\sxyh\syh^{-1/2}\|=\hlambda_1\leq 1$. Then
\begin{align*}
\|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|&\leq \|\bI_{p_1}-\sxh^{1/2}\|+ \|\sxh\|\|\bI_{p_2}-\syh^{1/2}\|+\|\sxyh-\sxy\|.
%\\
%&\leq \|\bI_{p_1}-\sxh\|+ \|\sxh\|\|\bI_{p_2}-\syh\|+\|\sxyh-\sxy\|\\
%&\leq \|\s_z-\sh_z\|\left(2+\|\sh_z\|\right).
\end{align*}
Also notice that
\[
\|\bI-\syh\|=\|(\bI-\sy^{\half})(\bI+\sy^{\half})\|\geq \sigma_{\min}(\bI+\sy^{\half})\|\bI-\sy^{\half}\|\geq \|\bI-\sy^{\half}\|.
\]
Therefore,
\begin{align*}
\|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|&\leq \|\bI_{p_1}-\sxh\|+ \|\sxh\|\|\bI_{p_2}-\syh\|+\|\sxyh-\sxy\|\\
&\leq \|\s_z-\sh_z\|\left(2+\|\sh_z\|\right).
\end{align*}
The last inequality in the lemma relies on the fact that $\sxh^{\half}\pxhk$ and $\bI_{p_1, k}$ are leading $k$ singular vectors of $\sxh^{-1/2} \sxyh\syh^{-1/2}$ and $\sxy$ respectively. By a variant of Wedin's $\sin\theta$ law as stated in Lemma~\ref{lemma-wedin},
\begin{align*}
\left\|\bP_{\sxh^{\half}\pxhk}(\bI_{p_1}-\bP_{\bI_{p_1, k}})\right\|\leq \frac{C\|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|}{\Delta}.
\end{align*}
On the other hand,
\begin{align*}
\left\|\bP_{\sxh^{\half}\pxhk}(\bI_{p_1}-\bP_{\bI_{p_1, k}})\right\|&=\left\|\sxh^{\half}\pxhk(\sxh^{\half}\pxhk)^\top(\bI_{p_1}-\bP_{\bI_{p_1, k}})\right\|\\
&=\left\|(\sxh^{\half}\pxhk)^\top(\bI_{p_1}-\bP_{\bI_{p_1, k}})\right\|\\
&=\left\|(\sxh^{\half}\pxhk)^l\right\|,
\end{align*}
where the second equality is due to the fact that $\sxh^{\half}\pxhk$ has orthonormal columns and $(\sxh^{\half}\pxhk)^l$ denotes the lower $(p_1-k)\times k$ sub-matrix of $\sxh^{\half}\pxhk$. Again, by triangle inequality,
\begin{align*}
\left\|\pxhk^l\right\|&=\left\|(\sxh^{\half}\pxhk)^l-\left((\sxh^{\half}-\bI_{p_1})\pxhk\right)^l\right\|\\
&\leq \left\|(\sxh^{\half}\pxhk)^l\right\|+\left\|(\sxh^{\half}-\bI_{p_1})\pxhk\right\|\\
&\leq \frac{C\|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|}{\Delta}+\left\|\sh_z-\sh\right\|\left\|\pxhk\right\|.
\end{align*}
The last inequality is obtained by substituting the upper bound for $\|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|$ obtained above.
\subsection{Proof of Lemma~\ref{KL-distance}}
\label{sec::KL}
By simple algebra, the Kullback-Leibler divergence between two multivariate gaussian distributions satisfies
\[
D(\P_{\s_{(1)}}||\P_{\s_{(2)}})=\frac{n}{2}\Big\{\text{Tr}\left(\s_{(2)}^{-1}(\s_{(1)}-\s_{(2)})\right)-\text{log det}(\s_{(2)}^{-1}\s_{(1)})\Big\}.
\]
Notice that
\[
\s_{(i)}=\begin{bmatrix}
\sx^{\half} & \\
&\sy^{\half}
\end{bmatrix}\bOmega_{(i)}\begin{bmatrix}
\sx^{\half} & \\
&\sy^{\half}
\end{bmatrix},
\]
where
\[
\bOmega_{(i)}=\begin{bmatrix} \bI_{p_1} & \lambda_1\bU_{(i)}\bV_{(i)}^\top+\lambda_2\bW_{(i)}\bZ_{(i)}^\top\\ \lambda_1\V_{(i)}\U_{(i)}^\top+\lambda_2\bZ_{(i)}\bW_{(i)}^\top& \bI_{p_2}\end{bmatrix}.
\]
Then,
\[
D(\P_{\s_{(1)}}||\P_{\s_{(2)}})=\frac{n}{2}\Big\{\text{Tr}(\bOmega_{(2)}^{-1}\bOmega_{(1)})-(p_1+p_2)-\text{log det}(\bOmega_{(2)}^{-1}\bOmega_{(1)})\Big\}.
\]
Also notice that
\begin{align*}
\bOmega_{(i)}=\begin{bmatrix} \bI_{p_1} & \\ & \bI_{p_2}\end{bmatrix}+\frac{\lambda_1}{2}\begin{bmatrix}
\bU_{(i)}\\
\bV_{(i)}
\end{bmatrix}\begin{bmatrix}
\bU_{(i)}^\top&\bV_{(i)}^\top
\end{bmatrix}-\frac{\lambda_1}{2}\begin{bmatrix}
\bU_{(i)}\\
-\bV_{(i)}
\end{bmatrix}\begin{bmatrix}
\bU_{(i)}^\top&-\bV_{(i)}^\top
\end{bmatrix}\\
+\frac{\lambda_2}{2}\begin{bmatrix}
\bW_{(i)}\\
\bZ_{(i)}
\end{bmatrix}\begin{bmatrix}
\bW_{(i)}^\top&\bZ_{(i)}^\top
\end{bmatrix}-\frac{\lambda_2}{2}\begin{bmatrix}
\bW_{(i)}\\
-\bZ_{(i)}
\end{bmatrix}\begin{bmatrix}
\bW_{(i)}^\top&-\bZ_{(i)}^\top
\end{bmatrix}.
\end{align*}
Therefore $\bOmega_{(1)}, \bOmega_{(2)}$ share the same set of eigenvalues: $1+\lambda_1$ with multiplicity $k$, $1-\lambda_1$ with multiplicity $k$, $1+\lambda_2$ with multiplicity $p_1-k$, $1-\lambda_2$ with multiplicity $p_1-k$ and $1$ with multiplicity $2(p_2-p_1)$. This implies $\text{log det}(\bOmega_{(2)}^{-1}\bOmega_{(1)}))=0$. On the other hand,
by block inversion formula, we can compute
\[
\bOmega_{(2)}^{-1}=\begin{bmatrix}
\bI_{p_1}+ \frac{\lambda_1^2}{1-\lambda_1^2}\bU_{(2)}\bU_{(2)}^\top+\frac{\lambda_2^2}{1-\lambda_2}\bW_{(2)}\bW_{(2)}^\top& -\frac{\lambda_1}{1-\lambda_1^2}\bU_{(2)}\bV_{(2)}^\top-\frac{\lambda_2}{1-\lambda_2}\bW_{(2)}\bZ_{(2)}^\top\\
-\frac{\lambda_1}{1-\lambda_1^2}\bV_{(2)}\bU_{(2)}^\top-\frac{\lambda_2}{1-\lambda_2}\bZ_{(2)}\bW_{(2)}^\top&\bI_{p_2}+ \frac{\lambda_1^2}{1-\lambda_1^2}\bV_{(2)}\bV_{(2)}^\top+\frac{\lambda_2^2}{1-\lambda_2}\bZ_{(2)}\bZ_{(2)}^\top
\end{bmatrix}.
\]
%and
%\[
%\bOmega_{(1)}-\bOmega_{(2)}=\begin{bmatrix}
%0 & \lambda_1(\bU_{(1)}\bV_{(1)}^\top- \bU_{(2)}\bV_{(2)}^\top)+\lambda_2(\bW_{(1)}\bZ_{(1)}^\top-\bW_{(2)}\bZ_{(2)}^\top)\\
%\lambda_1(\bV_{(1)}\bU_{(1)}^\top- \bV_{(2)}\bU_{(2)}^\top)+\lambda_2(\bZ_{(1)}\bW_{(1)}^\top-\bZ_{(2)}\bW_{(2)}^\top)& 0
%\end{bmatrix}
%\]
Divide $\bOmega_{(2)}^{-1}\bOmega_{(1)}$ into blocks such that
\begin{align*}
\bOmega_{(2)}^{-1}\bOmega_{(1)}=\begin{bmatrix}
\bJ_{11} & \bJ_{12} \\
\bJ_{21}&\bJ_{22}
\end{bmatrix}~~where ~~\bJ_{11}\in\mathbb{R}^{p_1\times p_1},~\bJ_{22}\in\mathbb{R}^{p_2\times p_2},
\end{align*}
and
\begin{align*}
\bJ_{11}&=\frac{\lambda_1^2}{1-\lambda_1^2}(\bU_{(2)}\bU_{(2)}^\top-\bU_{(2)}\bV_{(2)}^\top\bV_{(1)}\bU_{(1)}^\top)+\frac{\lambda_2^2}{1-\lambda_2^2}(\bW_{(2)}\bW_{(2)}-\bW_{(2)}\bZ_{(2)}^\top\bZ_{(1)}\bW_{(1)}^\top)\\
&~~~-\frac{\lambda_1\lambda_2}{1-\lambda_1^2}(\bU_{(2)}\bV_{(2)}^\top\bZ_{(1)}\bW_{(1)}^\top)-\frac{\lambda_1\lambda_2}{1-\lambda_2^2}(\bW_{(2)}\bZ_{(2)}^\top\bV_{(1)}\bU_{(1)}^\top)\\
\bJ_{22}&=\frac{\lambda_1^2}{1-\lambda_1^2}(\bV_{(2)}\bV_{(2)}^\top-\bV_{(2)}\bU_{(2)}^\top\bU_{(1)}\bV_{(1)}^\top)+\frac{\lambda_2^2}{1-\lambda_2^2}(\bZ_{(2)}\bZ_{(2)}-\bZ_{(2)}\bW_{(2)}^\top\bW_{(1)}\bZ_{(1)}^\top)\\
&~~~-\frac{\lambda_1\lambda_2}{1-\lambda_1^2}(\bV_{(2)}\bU_{(2)}^\top\bW_{(1)}\bZ_{(1)}^\top)-\frac{\lambda_1\lambda_2}{1-\lambda_2^2}(\bZ_{(2)}\bW_{(2)}^\top\bU_{(1)}\bV_{(1)}^\top).
\end{align*}
We spell out the algebra for $tr(\bJ_{11})$, and $tr(\bJ_{22})$ can be computed in exactly the same fashion.
\begin{align*}
tr(\bU_{(2)}\bU_{(2)}^\top-\bU_{(2)}\bV_{(2)}^\top\bV_{(1)}\bU_{(1)}^\top)&=\frac{1}{2}tr(\bU_{(2)}\bV_{(2)}^\top\bV_{(2)}\bU_{(2)}^\top+\bU_{(1)}\bV_{(1)}^\top\bV_{(1)}\bU_{(1)}^\top-2\bU_{(2)}\bV_{(2)}^\top\bV_{(1)}\bU_{(1)}^\top)\\
&=\frac{1}{2}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2.
\end{align*}
Similarly,
\begin{align*}
tr(\bW_{(2)}\bW_{(2)}-\bW_{(2)}\bZ_{(2)}^\top\bZ_{(1)}\bW_{(1)}^\top)=\frac{1}{2}\|\bW_{(1)}\bZ_{(1)}^\top-\bW_{(2)}\bZ_{(2)}\|_F^2.
\end{align*}
By the assumption, $\bU_{(1)}\bV_{(1)}^\top+\bW_{(1)}\bZ_{(1)}^\top=\bU_{(2)}\bV_{(2)}^\top+\bW_{(2)}\bZ_{(2)}^\top$, which implies
\begin{align*}
tr(\bW_{(2)}\bW_{(2)}-\bW_{(2)}\bZ_{(2)}^\top\bZ_{(1)}\bW_{(1)}^\top)=\frac{1}{2}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2.
\end{align*}
Further,
\begin{align*}
tr(\bU_{(2)}\bV_{(2)}^\top\bZ_{(1)}\bW_{(1)}^\top)&=tr\left(\bU_{(2)}\bV_{(2)}^\top(\bU_{(2)}\bV_{(2)}^\top+\bW_{(2)}\bZ_{(2)}^\top-\bU_{(1)}\bV_{(1)}^\top)^\top\right)\\
&=tr\left(\bU_{(2)}\bV_{(2)}^\top(\bU_{(2)}\bV_{(2)}^\top-\bU_{(1)}\bV_{(1)}^\top)^\top\right)\\
&=\frac{1}{2}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2,
\end{align*}
and by the same argument,
\begin{align*}
tr(\bW_{(2)}\bZ_{(2)}^\top\bV_{(1)}\bU_{(1)}^\top)=\frac{1}{2}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2.
\end{align*}
Sum these equations,
\begin{align*}
\text{tr}(\bJ_{11})&=\frac{1}{2}\left\{ \frac{\lambda_1^2}{1-\lambda_1^2}+\frac{\lambda_2^2}{1-\lambda_2^2}-\frac{\lambda_1\lambda_2}{1-\lambda_1^2}-\frac{\lambda_1\lambda_2}{1-\lambda_2^2}
\right\}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2\\
&=\frac{\Delta^2(1+\lambda_1\lambda_2)}{2(1-\lambda_1^2)(1-\lambda_2^2)}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2.
\end{align*}
Repeat the argument for $\bJ_{22}$, one can show that
\begin{align*}
\text{tr}(\bJ_{22})=\text{tr}(\bJ_{11})=\frac{\Delta^2(1+\lambda_1\lambda_2)}{2(1-\lambda_1^2)(1-\lambda_2^2)}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2.
\end{align*}
Therefore,
\begin{align*}
D(\P_{\s_{(1)}}||\P_{\s_{(2)}})&=\frac{n}{2} tr(\bOmega_{(2)}^{-1}\bOmega_{(1)})=\frac{n}{2}\left(tr(\bJ_{11})+tr(\bJ_{22})\right)\\
&=\frac{n\Delta^2(1+\lambda_1\lambda_2)}{2(1-\lambda_1^2)(1-\lambda_2^2)}\|\bU_{(1)}\bV_{(1)}^\top-\bU_{(2)}\bV_{(2)}\|_F^2.
\end{align*}
%%%%%%%%%%%%%%%%Subspace-Metric
\subsection{Proof of Lemma~\ref{subspace-metric}}
\label{sec::subspace-metric}
\begin{align*}
\|\U_1-\U_2\Q\|_F^2=2k-2tr(\U_1^\top\U_2\Q)
\end{align*}
Let $\U_1^\top\U_2=\U\D\V^\top$ be the singular value decomposition. Then $\V\U^\top\in O(k, k)$ and
\begin{align*}
\inf\limits_{\Q\in O(k, k)}\|\U_1-\U_2\Q\|_F^2&\leq 2k-2tr(\U_1^\top\U_2\V\U^\top)\\\
&=2k-2tr(\U\D\U^\top)\\
&=2k-2tr(\D).
\end{align*}
On the other hand,
\begin{align*}
\|\bP_{\U_1}-\bP_{\U_2}\|_F^2&=\|\U_1\U_1^\top-\U_2\U_2^\top\|_F^2\\
&=2k-2tr(\U_1\U_1^\top\U_2\U_2^\top)\\
&=2k-2tr(\U_1^\top\U_2\U_2^\top\U_1)\\
&=2k-2tr(\D^2).
\end{align*}
Since $\U_1, \U_2\in O(p, k)$, $\|\U_1^\top\U_2\|\leq 1$ and therefore all the diagonal elements of $\D$ is less than 1, which implies that $tr(\D)\geq tr(\D^2)$ and
\begin{equation}
\inf\limits_{\Q\in O(k, k)}\|\U_1-\U_2\Q\|_F^2\leq \|\bP_{\U_1}-\bP_{\U_2}\|_F^2.
\nonumber
\end{equation}
%\subsection{Proof of Lemma~\ref{lemma-wedin}}
%The result for operator norm is proved in Proposition 1 of \cite{cai2015optimal}. The result in terms for Frobenius norm and symmetric matrices is proved in Theorem 2 of \cite{yu2014useful} (Frobenius norm). Now, we extend the results to asymmetric case.
%Let $p=m+n$ and define the symmetric matrices $\s, \sh \in\mathbb{R}^{p\times p}$ as
%\[
%\s=\begin{pmatrix}
%\mtx{0} & \bA\\
%\bA^\top & \mtx{0}
%\end{pmatrix}
%\quad
%\sh=\begin{pmatrix}
%\mtx{0} & \hbA\\
%\hbA^\top & \mtx{0}
%\end{pmatrix}
%\]
%Further, define matrices $\bW, \hbW\in\mathbb{R}^{p\times 2m}$ as
%\[
%\bW=\frac{1}{\sqrt{2}}\begin{pmatrix}
%\bU & \bU\\
%\bV & -\bV
%\end{pmatrix}
%\quad
%\hbW=\frac{1}{\sqrt{2}}\begin{pmatrix}
%\hbU & \hbU\\
%\hbV & -\hbV
%\end{pmatrix}
%\]
%It is easy to check that the columns of $\bW$ are the eigenvectors of $\s$ with eigenvalues $\lambda_1, \cdots, \lambda_{p_1}, -\lambda_1, \cdots, -\lambda_{p_1}$ and the columns of $\hbW$ are the eigenvectors of $\sh$ with eigenvalues $\hlambda_1, \cdots, \hlambda_{p_1}, -\hlambda_1, \cdots, -\hlambda_{p_1}$. By Theorem 2 of \cite{yu2014useful}, we will obtain
%\begin{align*}
%\|\text{sin}~\bTheta (\hbW_{1:2k}, \bW_{1:2k})\|_F&\leq \frac{2\|\s-\sh\|_F}{\sigma_{2k}(\s)-\sigma_{2k+1}(\s)}\\
%&=\frac{2 \sqrt{2}\|\bA-\hbA\|_F}{\sigma_k(\bA)-\sigma_{k+1}(\bA)}
%\end{align*}
%Also notice that
%\begin{align*}
%\|\text{sin}~\bTheta (\hbW_*, \bW_*)\|_F^2&=d-\|\hbW_*^\top\bW_*\|_F^2\\
%&=\frac{1}{2}\left(d-\|\hbU_*^\top\bU_*\|_F^2\right)+\frac{1}{2}\left( d-\|\hbV_*^\top\bV_*\|_F^2\right)\\
%&=\frac{1}{2} \left(\|\text{sin}~\bTheta (\hbU_*, \bU_*)\|_F^2+\|\text{sin}~\bTheta (\hbV_*, \bV_*)\|_F^2\right)
%\end{align*}
%which implies the first claim. The second claim is derived in the same manner.
%The case when $\bA$ and $\hbA$ are symmetric were proved by Proposition 1 of \cite{cai2015optimal} (operator norm) and Theorem 2 of \cite{yu2014useful} (Frobenius norm). We extend the results to the asymmetric case. For operator norm, the proof is essentially the same as \cite{cai2015optimal}. By Wedin's $\sin\theta$ law,
%\begin{align*}
%\left\|(\bI-\bP_{\bU_{1:k}})\bP_{\hbU_{1:k}}\right\|=\left\|\bP_{\bU_{1:k}}-\bP_{\hbU_{1:k}}\right\|\leq \frac{C\|\bE\|}{\sigma_{k}(\hbA)-\sigma_{k+1}(\bA)}
%\end{align*}
%when $\sigma_r$
\subsection{Proof of Lemma~\ref{lemma-expectation-covariance}}
\label{sec::integration}
Without loss of generality, we can assume $\|\s\|=1$ or else we can scale $\bA$ by $1/\sqrt{\|\s\|}$. Let $\bJ=\|\frac{1}{n}\bA^\top\bA-\bSigma\|$ and by Lemma~\ref{lemma-vershynin}, there exists positive constants $c_1, C_1$ such that
\[
P(\bJ\geq \max\{\delta, \delta^2\})\leq e^{-c_1 t^2}, ~~\delta=C_1\sqrt{\frac{p}{n}}+\frac{t}{\sqrt{n}}.
\]
Notice that $\bJ\geq 0$, then
\begin{align*}
\E[\bJ^4]&=\int_{0}^{+\infty} P(\bJ\geq x^{1/4}) dx\\
&=\int_{0}^{C_1^2p^2/n^2} P(\bJ\geq x^{1/4}) dx+\int_{C_1^2p^2/n^2}^{1} P(\bJ\geq x^{1/4}) dx+\int_{1}^{+\infty} P(\bJ\geq x^{1/4}) dx\\
&\leq C_1^2p^2/n^2+\int_{C_1^2p^2/n^2}^{1}e^{-(\sqrt{n}x^{1/4}-C_1\sqrt{p})^2} dx+\int_1^{+\infty}e^{-(\sqrt{n}x^{1/8}-C_1\sqrt{p})^2} dx\\
&=C_1^2p^2/n^2+\int_{C_1^2p^2/n^2}^{1} 4e^{-y^2}\left(\frac{y+C_1\sqrt{p}}{\sqrt{n}}\right)^3\frac{1}{\sqrt{n}} dy+\int_{1}^{+\infty} 8e^{-y^2}\left(\frac{y+C_1\sqrt{p}}{\sqrt{n}}\right)^7\frac{1}{\sqrt{n}} dy .
\end{align*}
There exists a large constant $C_2$ such that
\begin{align*}
\E[\bJ^4]&\leq C_1^2p^2/n^2+\frac{4}{n^2} \int_{C_1^2p^2/n^2}^{1} 4e^{-y^2}C_2(y^3+p^{2/3})dy+\frac{8}{n^4} \int_{1}^{+\infty} C_2e^{-y^2} (y^7+p^{7/2})dy\\
&\leq C_1^2p^2/n^2+\frac{4}{n^2} \int_{0}^{+\infty} 4e^{-y^2}C_2(y^3+p^{2/3})dy+\frac{8}{n^4} \int_{0}^{+\infty} C_2e^{-y^2} (y^7+p^{7/2})dy.
\end{align*}
Notice that $\int_{0}^{+\infty} e^{-y^2} y^k dy$ is bounded for any $k\in\mathbb{Z}_+$ and $n\geq p$. There exists a large constant $C_3$ such that
\begin{align*}
\E[\bJ^4]&\leq C_3\frac{p^2}{n^2}.
\end{align*}
%\subsection{Supporting lemmas}
%\begin{lemma}[Square Root Perturbation \cite{schmitt1992perturbation} (Lemma 2.1)]
%For positive definite matrices $\bA_1, \bA_2$ with $\lambda_{\min}(\bA_1), \lambda_{\min}(\bA_2)\geq 0$ being their smallest eigenvalue respectively,
%\begin{equation}
%\|\bA_1^{\half}-\bA_2^{\half}\|\leq \frac{\|\bA_1-\bA_2\|}{\lambda_{\min}(\bA_1)^{\half}+\lambda_{\min}(\bA_2)^\half}
%\end{equation}
%\label{lemma-matrix-root}
%\end{lemma}
%\begin{lemma}
%\label{lemma-matrix-inequality}
%The following inequalities hold deterministically
%\begin{gather*}
%\|\sxh^{-1/2}-\bI_{p_1}\|_2\leq \|\sxh^{-1/2}\|_2\|\bI_{p_1}-\sxh\|_2\\
%\|\syh^{-1/2}-\bI_{p_2}\|_2\leq \|\syh^{-1/2}\|_2\|\bI_{p_2}-\syh\|_2\\
%\|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|_2\leq \|\bI_{p_1}-\sxh\|+ \|\sxh\|\|\bI_{p_2}-\syh\|+\|\sxyh-\sxy\|
%\end{gather*}
%\end{lemma}
%\begin{proof}[Proof of Lemma~\ref{lemma-matrix-inequality}]
%\begin{align*}
%\|\sxh^{-1/2}-\bI_{p_1}\|_2=\|\sxh^{-1/2}(\bI_{p_1}-\sxh^{1/2})\|_2\leq \|\sxh^{-1/2}\|_2\|\bI_{p_1}-\sxh^{1/2}\|_2
%\end{align*}
%Apply Lemma~\ref{lemma-matrix-root}, $\|\bI_{p_1}-\sxh^{1/2}\|_2\leq \|\bI_{p_1}-\sxh\|$ and we obtain the first inequality. The second inequality is proved in the same manner. As for the third inequality, notice that
%\begin{align*}
%\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy&=(\bI_{p_1}-\sxh^{1/2})\sxh^{-1/2}\sxyh\syh^{-1/2}\\
%&+ \sxh^{1/2}\sxh^{-1/2}\sxyh\syh^{-1/2}(\bI_{p_2}-\syh^{1/2})+(\sxyh-\sxy)
%\end{align*}
%Then by triangle inequality and Lemma~\ref{lemma-matrix-root}
%\begin{align*}
%\|\sxh^{-1/2} \sxyh\syh^{-1/2} -\sxy\|_2&\leq \|\bI_{p_1}-\sxh^{1/2}\|+ \|\sxh\|\|\bI_{p_2}-\syh^{1/2}\|+\|\sxyh-\sxy\|\\
%&\leq \|\bI_{p_1}-\sxh\|+ \|\sxh\|\|\bI_{p_2}-\syh\|+\|\sxyh-\sxy\|
%\end{align*}
%\end{proof}
%\begin{lemma}[Perturbation of Eigenvectors \citep{yu2014useful}]
%Let $\s, \sh\in\mathbb{R}^{p\times p}$ be symmetric, with eigenvalues $\lambda_1\geq \cdots\geq \lambda_p$ and $\hlambda_1\geq \cdots\geq \hlambda_p$. Fix $1\leq r\leq s\leq p$ and assume that $\min \{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}>0$, where we define $\lambda_0=\infty, \lambda_{p+1}=-\infty$. Let $d=s-r+1$, and let $\bV=(\bv_r, \bv_{r+1}, \cdots, \bv_s)\in\mathbb{R}^{p\times d}$ and $\hbV=(\hbv_r, \hbv_{r+1}, \cdots, \hbv_s)\in\mathbb{R}^{p\times d}$ have orthonormal columns satisfying $\s\bv_j=\lambda_j\bv_j$ and $\sh\hbv_j=\lambda_j\hbv_j$ for $j=r, r+1, \cdots, s$. Then
%\[
%\|\text{sin}~\bTheta (\hbV, \bV)\|_F\leq \frac{2\min\{d^{1/2}\|\s-\sh\|_2, \|\s-\sh\|_F\}}{\min\{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}}
%\]
%and there exists an orthogonal matrix $\hbO\in\mathbb{R}^{d\times d}$ such that
%\[
%\|\bV-\hbV\hbO\|_F\leq \frac{2\sqrt{2}\min\{d^{1/2}\|\s-\sh\|_2, \|\s-\sh\|_F\}}{\min\{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}}
%\]
%\label{lemma-eigen-vector}
%\end{lemma}
%This lemma can be easily generalized to bound the singular vector perturbation.
%\begin{lemma}[Perturbation of Singular Vectors]
%For $\bA, \hbA\in\mathbb{R}^{p_1\times p_1}$ and $p_1\leq p_2$, let $\bA=\bU\lm\bV^\top$ and $\hbA=\hbU\wh{\lm}\hbV^\top$ be the singular value decompositions with $\bU=(\bu_1, \cdots, \bu_{p_1}), \bV=(\bv_1, \cdots, \bv_{p_1}), \lm=diag(\lambda_1, \cdots, \lambda_{p_1})$ and $\hbU=(\hbu_1, \cdots, \hbu_{p_1}), \hbV=(\hbv_1, \cdots, \hbv_{p_1}), \wh{\lm}=diag(\hlambda_1, \cdots, \hlambda_{p_1})$. Fix $1\leq r\leq s\leq p_1$, assume that $\min \{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}>0$ and let $d=s-r+1$. Define $\bU_*=(\bu_r, \bu_{r+1}, \cdots, \bu_s)\in\mathbb{R}^{p_1\times d}, \bV_*=(\bv_r, \bv_{r+1}, \cdots, \bv_s)\in\mathbb{R}^{p_2\times d}$ and correspondingly define $\hbU_*, \hbV_*$. Then
%\[
%\sqrt{\|\sin~\bTheta (\hbU_*, \bU_*)\|_F^2+\|\sin~\bTheta (\hbV_*, \bV_*)\|_F^2}\leq \frac{2\sqrt{2}\min\{d^{1/2}\|\bA-\hbA\|_2, \sqrt{2}\|\bA-\hbA\|_F\}}{\min\{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}}
%\]
%and there exists an orthogonal matrix $\hbO\in\mathbb{R}^{d\times d}$ such that
%\[
%\sqrt{\|\bU_*-\hbU_*\hbO\|_F^2+\|\bV_*-\hbV_*\hbO\|_F^2}\leq \frac{4\min\{d^{1/2}\|\bA-\hbA\|_2, \sqrt{2}\|\bA-\hbA\|_F\}}{\min\{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}}
%\]
%\label{lemma-singular-vector}
%\end{lemma}
%
%\begin{proof}
%Let $p=p_1+p_2$. Define the symmetric matrices $\s, \sh \in\mathbb{R}^{p\times p}$ as
%\[
%\s=\begin{pmatrix}
%\mtx{0} & \bA\\
%\bA^\top & \mtx{0}
%\end{pmatrix}
%\quad
%\sh=\begin{pmatrix}
%\mtx{0} & \hbA\\
%\hbA^\top & \mtx{0}
%\end{pmatrix}
%\]
%Further, define matrices $\bW, \hbW\in\mathbb{R}^{p\times 2p_1}$ as
%\[
%\bW=\frac{1}{\sqrt{2}}\begin{pmatrix}
%\bU & \bU\\
%\bV & -\bV
%\end{pmatrix}
%\quad
%\hbW=\frac{1}{\sqrt{2}}\begin{pmatrix}
%\hbU & \hbU\\
%\hbV & -\hbV
%\end{pmatrix}
%\]
%Then the columns of $\bW$ are the eigenvectors of $\s$ with eigenvalues $\lambda_1, \cdots, \lambda_{p_1}, -\lambda_1, \cdots, -\lambda_{p_1}$ and the columns of $\hbW$ are the eigenvectors of $\sh$ with eigenvalues $\hlambda_1, \cdots, \hlambda_{p_1}, -\hlambda_1, \cdots, -\hlambda_{p_1}$
%%Then $\s$ has eigenvectors $(\bu_i^\top, \bv_i^\top)^\top/\sqrt{2}$ with corresponding eigenvalue $\lambda_i$ and eigenvectors $(\bu_i^\top, -\bv_i^\top)^\top/\sqrt{2}$ with corresponding eigenvalue $-\lambda_i$. All other eigenvalues are 0. Similar observations hold for $\sh$.
%Finally define the target matrices $\bW_*, \hbW_*\in\mathbb{R}^{p\times d}$ by
%\[
%\bW=\frac{1}{\sqrt{2}}\begin{pmatrix}
%\bU_* \\
%\bV_*
%\end{pmatrix}
%\quad
%\hbW=\frac{1}{\sqrt{2}}\begin{pmatrix}
%\hbU_*\\
%\hbV_*
%\end{pmatrix}
%\]
%By Theorem 2 of \cite{yu2014useful}, we will obtain
%\begin{align*}
%\|\text{sin}~\bTheta (\hbW_*, \bW_*)\|_F&\leq \frac{2\min\{d^{1/2}\|\s-\sh\|_2, \|\s-\sh\|_F\}}{\min\{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}}\\
%&=\frac{2\min\{d^{1/2}\|\bA-\hbA\|_2, \sqrt{2}\|\bA-\hbA\|_F\}}{\min\{\lambda_{r-1}-\lambda_r, \lambda_s-\lambda_{s+1}\}}
%\end{align*}
%Also notice that
%\begin{align*}
%\|\text{sin}~\bTheta (\hbW_*, \bW_*)\|_F^2&=d-\|\hbW_*^\top\bW_*\|_F^2\\
%&=\frac{1}{2}\left(d-\|\hbU_*^\top\bU_*\|_F^2\right)+\frac{1}{2}\left( d-\|\hbV_*^\top\bV_*\|_F^2\right)\\
%&=\frac{1}{2} \left(\|\text{sin}~\bTheta (\hbU_*, \bU_*)\|_F^2+\|\text{sin}~\bTheta (\hbV_*, \bV_*)\|_F^2\right)
%\end{align*}
%which implies the first claim. The second claim is derived in the same manner.
%\end{proof}
%
%
%
%
%
\begin{thebibliography}{}
\bibitem[\protect\citeauthoryear{Anderson}{Anderson}{1963}]{anderson1963asymptotic}
Anderson, T.~W. (1963).
\newblock Asymptotic theory for principal component analysis.
\newblock {\em The Annals of Mathematical Statistics\/}~{\em 34\/}(1),
122--148.
\bibitem[\protect\citeauthoryear{Anderson}{Anderson}{1984}]{tAND84a}
Anderson, T.~W. (1984).
\newblock {\em An Introduction to Multivariate Statistical Analysis\/} (second
ed.).
\newblock New York, NY: Wiley.
\bibitem[\protect\citeauthoryear{Anderson}{Anderson}{1999}]{anderson1999asymptotic}
Anderson, T.~W. (1999).
\newblock Asymptotic theory for canonical correlation analysis.
\newblock {\em Journal of Multivariate Analysis\/}~{\em 70\/}(1), 1--29.
\bibitem[\protect\citeauthoryear{Arora and Livescu}{Arora and
Livescu}{2013}]{arora2013multi}
Arora, R. and K.~Livescu (2013).
\newblock Multi-view cca-based acoustic features for phonetic recognition
across speakers and domains.
\newblock In {\em Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE
International Conference on}, pp.\ 7135--7139. IEEE.
\bibitem[\protect\citeauthoryear{Cai, Ma, and Wu}{Cai
et~al.}{2015}]{cai2015optimal}
Cai, T., Z.~Ma, and Y.~Wu (2015).
\newblock Optimal estimation and rank detection for sparse spiked covariance
matrices.
\newblock {\em Probability theory and related fields\/}~{\em 161\/}(3-4),
781--815.
\bibitem[\protect\citeauthoryear{Cai, Ma, Wu, et~al.}{Cai
et~al.}{2013}]{cai2013sparse}
Cai, T.~T., Z.~Ma, Y.~Wu, et~al. (2013).
\newblock Sparse pca: Optimal rates and adaptive estimation.
\newblock {\em The Annals of Statistics\/}~{\em 41\/}(6), 3074--3110.
\bibitem[\protect\citeauthoryear{Cai and Zhang}{Cai and
Zhang}{2016}]{cai2016rate}
Cai, T.~T. and A.~Zhang (2016).
\newblock Rate-optimal perturbation bounds for singular subspaces with
applications to high-dimensional statistics.
\newblock {\em arXiv preprint arXiv:1605.00353\/}.
\bibitem[\protect\citeauthoryear{Chaudhuri, Kakade, Livescu, and
Sridharan}{Chaudhuri et~al.}{2009}]{chaudhuri2009multi}
Chaudhuri, K., S.~M. Kakade, K.~Livescu, and K.~Sridharan (2009).
\newblock Multi-view clustering via canonical correlation analysis.
\newblock In {\em Proceedings of the 26th annual international conference on
machine learning}, pp.\ 129--136. ACM.
\bibitem[\protect\citeauthoryear{Chen, Gao, Ren, and Zhou}{Chen
et~al.}{2013}]{chen2013sparse}
Chen, M., C.~Gao, Z.~Ren, and H.~H. Zhou (2013).
\newblock Sparse cca via precision adjusted iterative thresholding.
\newblock {\em arXiv preprint arXiv:1311.6186\/}.
\bibitem[\protect\citeauthoryear{Chen, Liu, and Carbonell}{Chen
et~al.}{2012}]{chen2012structured}
Chen, X., H.~Liu, and J.~G. Carbonell (2012).
\newblock Structured sparse canonical correlation analysis.
\newblock In {\em International Conference on Artificial Intelligence and
Statistics}, pp.\ 199--207.
\bibitem[\protect\citeauthoryear{Dhillon, Foster, and Ungar}{Dhillon
et~al.}{2011}]{dhillon11}
Dhillon, P.~S., D.~Foster, and L.~Ungar (2011).
\newblock Multi-view learning of word embeddings via cca.
\newblock In {\em Advances in Neural Information Processing Systems (NIPS)},
Volume~24.
\bibitem[\protect\citeauthoryear{Faruqui and Dyer}{Faruqui and
Dyer}{2014}]{faruqui2014improving}
Faruqui, M. and C.~Dyer (2014).
\newblock Improving vector space word representations using multilingual
correlation.
\newblock Association for Computational Linguistics.
\bibitem[\protect\citeauthoryear{Fiedler}{Fiedler}{2010}]{fiedler2010notes}
Fiedler, M. (2010).
\newblock Notes on hilbert and cauchy matrices.
\newblock {\em Linear Algebra and its Applications\/}~{\em 432\/}(1), 351--356.
\bibitem[\protect\citeauthoryear{Foster, Johnson, Kakade, and Zhang}{Foster
et~al.}{2008}]{foster08}
Foster, D.~P., R.~Johnson, S.~M. Kakade, and T.~Zhang (2008).
\newblock Multi-view dimensionality reduction via canonical correlation
analysis.
\newblock Technical report.
\bibitem[\protect\citeauthoryear{Friman, Borga, Lundberg, and Knutsson}{Friman
et~al.}{2003}]{friman2003adaptive}
Friman, O., M.~Borga, P.~Lundberg, and H.~Knutsson (2003).
\newblock Adaptive analysis of fmri data.
\newblock {\em NeuroImage\/}~{\em 19\/}(3), 837--845.
\bibitem[\protect\citeauthoryear{Fukumizu, Bach, and Jordan}{Fukumizu
et~al.}{2009}]{fukumizu2009kernel}
Fukumizu, K., F.~R. Bach, and M.~I. Jordan (2009).
\newblock Kernel dimension reduction in regression.
\newblock {\em The Annals of Statistics\/}, 1871--1905.
\bibitem[\protect\citeauthoryear{Gao, Ma, Ren, Zhou, et~al.}{Gao
et~al.}{2015}]{gao2015minimax}
Gao, C., Z.~Ma, Z.~Ren, H.~H. Zhou, et~al. (2015).
\newblock Minimax estimation in sparse canonical correlation analysis.
\newblock {\em The Annals of Statistics\/}~{\em 43\/}(5), 2168--2197.
\bibitem[\protect\citeauthoryear{Gao, Ma, and Zhou}{Gao
et~al.}{2014}]{gao2014sparse}
Gao, C., Z.~Ma, and H.~H. Zhou (2014).
\newblock Sparse cca: Adaptive estimation and computational barriers.
\newblock {\em arXiv preprint arXiv:1409.8565\/}.
\bibitem[\protect\citeauthoryear{Gong, Ke, Isard, and Lazebnik}{Gong
et~al.}{2014}]{gong2014multi}
Gong, Y., Q.~Ke, M.~Isard, and S.~Lazebnik (2014).
\newblock A multi-view embedding space for modeling internet images, tags, and
their semantics.
\newblock {\em International journal of computer vision\/}~{\em 106\/}(2),
210--233.
\bibitem[\protect\citeauthoryear{Hom and Johnson}{Hom and
Johnson}{1991}]{hom1991topics}
Hom, R.~A. and C.~R. Johnson (1991).
\newblock Topics in matrix analysis.
\newblock {\em Cambridge UP, New York\/}.
\bibitem[\protect\citeauthoryear{Hotelling}{Hotelling}{1936}]{hotelling36}
Hotelling, H. (1936).
\newblock Relations between two sets of variables.
\newblock {\em Biometrika\/}~{\em 28}, 312--377.
\bibitem[\protect\citeauthoryear{Hsu}{Hsu}{1941}]{hsu1941limiting}
Hsu, P. (1941).
\newblock On the limiting distribution of the canonical correlations.
\newblock {\em Biometrika\/}~{\em 32\/}(1), 38--45.
\bibitem[\protect\citeauthoryear{Izenman}{Izenman}{1975}]{izenman1975reduced}
Izenman, A.~J. (1975).
\newblock Reduced-rank regression for the multivariate linear model.
\newblock {\em Journal of multivariate analysis\/}~{\em 5\/}(2), 248--264.
\bibitem[\protect\citeauthoryear{Kakade and Foster}{Kakade and
Foster}{2007}]{kakade07}
Kakade, S.~M. and D.~P. Foster (2007).
\newblock Multi-view regression via canonical correlation analysis.
\newblock In {\em In Proc. of Conference on Learning Theory}.
\bibitem[\protect\citeauthoryear{Kim, Wong, and Cipolla}{Kim
et~al.}{2007}]{kim2007tensor}
Kim, T.-K., S.-F. Wong, and R.~Cipolla (2007).
\newblock Tensor canonical correlation analysis for action classification.
\newblock In {\em Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE
Conference on}, pp.\ 1--8. IEEE.
\bibitem[\protect\citeauthoryear{Loy, Xiang, and Gong}{Loy
et~al.}{2009}]{loy2009multi}
Loy, C.~C., T.~Xiang, and S.~Gong (2009).
\newblock Multi-camera activity correlation analysis.
\newblock In {\em Computer Vision and Pattern Recognition, 2009. CVPR 2009.
IEEE Conference on}, pp.\ 1988--1995. IEEE.
\bibitem[\protect\citeauthoryear{Ma, Lu, and Foster}{Ma
et~al.}{2015}]{ma2015finding}
Ma, Z., Y.~Lu, and D.~Foster (2015).
\newblock Finding linear structure in large datasets with scalable canonical
correlation analysis.
\newblock In {\em Proceedings of The 32nd International Conference on Machine
Learning}, pp.\ 169--178.
\bibitem[\protect\citeauthoryear{Mathias}{Mathias}{1993}]{mathias1993hadamard}
Mathias, R. (1993).
\newblock The hadamard operator norm of a circulant and applications.
\newblock {\em SIAM journal on matrix analysis and applications\/}~{\em
14\/}(4), 1152--1167.
\bibitem[\protect\citeauthoryear{Rasiwasia, Costa~Pereira, Coviello, Doyle,
Lanckriet, Levy, and Vasconcelos}{Rasiwasia et~al.}{2010}]{rasiwasia2010new}
Rasiwasia, N., J.~Costa~Pereira, E.~Coviello, G.~Doyle, G.~R. Lanckriet,
R.~Levy, and N.~Vasconcelos (2010).
\newblock A new approach to cross-modal multimedia retrieval.
\newblock In {\em Proceedings of the 18th ACM international conference on
Multimedia}, pp.\ 251--260. ACM.
\bibitem[\protect\citeauthoryear{Rudelson and Vershynin}{Rudelson and
Vershynin}{2013}]{hw2013}
Rudelson, M. and R.~Vershynin (2013).
\newblock Hanson-wright inequality and sub-gaussian concentration.
\newblock {\em Electron. Commun. Probab.\/}~{\em 18}, no. 82, 1--9.
\bibitem[\protect\citeauthoryear{Sridharan and Kakade}{Sridharan and
Kakade}{2008}]{SridharanK08}
Sridharan, K. and S.~M. Kakade (2008).
\newblock An information theoretic framework for multi-view learning.
\newblock In R.~A. Servedio and T.~Zhang (Eds.), {\em COLT}, pp.\ 403--414.
Omnipress.
\bibitem[\protect\citeauthoryear{Szarek}{Szarek}{1982}]{szarek1982nets}
Szarek, S.~J. (1982).
\newblock Nets of grassmann manifold and orthogonal group.
\newblock In {\em Proceedings of research workshop on Banach space theory (Iowa
City, Iowa, 1981)}, Volume 169, pp.\ 185.
\bibitem[\protect\citeauthoryear{Vershynin}{Vershynin}{2010}]{vershynin2010introduction}
Vershynin, R. (2010).
\newblock Introduction to the non-asymptotic analysis of random matrices.
\newblock {\em arXiv preprint arXiv:1011.3027\/}.
\bibitem[\protect\citeauthoryear{Wang, Arora, Livescu, and Bilmes}{Wang
et~al.}{2015}]{wang2015deep}
Wang, W., R.~Arora, K.~Livescu, and J.~Bilmes (2015).
\newblock On deep multi-view representation learning.
\newblock In {\em Proceedings of the 32nd International Conference on Machine
Learning (ICML-15)}, pp.\ 1083--1092.
\bibitem[\protect\citeauthoryear{Wedin}{Wedin}{1972}]{wedin1972perturbation}
Wedin, P.-{\AA}. (1972).
\newblock Perturbation bounds in connection with singular value decomposition.
\newblock {\em BIT Numerical Mathematics\/}~{\em 12\/}(1), 99--111.
\bibitem[\protect\citeauthoryear{Wedin}{Wedin}{1983}]{wedin1983angles}
Wedin, P.~{\AA}. (1983).
\newblock On angles between subspaces of a finite dimensional inner product
space.
\newblock In {\em Matrix Pencils}, pp.\ 263--285. Springer.
\bibitem[\protect\citeauthoryear{Witten, Tibshirani, and Hastie}{Witten
et~al.}{2009}]{witten2009penalized}
Witten, D.~M., R.~Tibshirani, and T.~Hastie (2009).
\newblock A penalized matrix decomposition, with applications to sparse
principal components and canonical correlation analysis.
\newblock {\em Biostatistics\/}, kxp008.
\bibitem[\protect\citeauthoryear{Xu, Tao, and Xu}{Xu
et~al.}{2013}]{xu2013survey}
Xu, C., D.~Tao, and C.~Xu (2013).
\newblock A survey on multi-view learning.
\newblock {\em arXiv preprint arXiv:1304.5634\/}.
\bibitem[\protect\citeauthoryear{Yu}{Yu}{1997}]{yu1997assouad}
Yu, B. (1997).
\newblock Assouad, fano, and le cam.
\newblock In {\em Festschrift for Lucien Le Cam}, pp.\ 423--435. Springer.
\end{thebibliography}