The Price of Linear Time: Error Analysis of Structured Kernel Interpolation
Extracted source text · arXiv 2502.00298
Gaussian Processes (GPs) \citep{kolmogorov1940wienersche,rasmussen2006gaussian} are an important class of stochastic processes used in machine learning and statistics, with use cases including spatial data analysis \citep{liu2021missing}, time series forecasting \citep{girard2002gaussian}, bioinformatics \citep{luo2023diseasegps} and Bayesian optimization \citep{frazier2018tutorial}. GPs offer a non-parametric framework for modeling distributions over functions, enabling both flexibility and uncertainty quantification. These capabilities, combined with the ability to incorporate prior knowledge and specify relationships by choice of kernel function, make Gaussian Processes effective for both regression and classification.
However, GPs have substantial computational and memory bottlenecks. Both training and inference require computing the action of the inverse kernel Gram matrix, while training requires computing its log-determinant: both are $O(n^3)$ operations with sample size $n$. Further, storing the full Gram matrix requires $O(n^2)$ memory. These bottlenecks require scalable approximations for larger datasets.
Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} helps scale Gaussian Processes (GPs) to large datasets by approximating the kernel matrix using interpolation on a set of inducing points. For stationary kernels, this requires $O(n+m \log m)$ computational complexity. The core idea is to express the original kernel as a combination of interpolation functions and a kernel matrix defined on a set of inducing points. However, despite its effectiveness, popularity (over $600$ citations, a large number for a GP paper) and high quality software availability (\cite{gardner2018gpytorch} has 3.5k stars on github), it currently lacks theoretical analysis. A key initial question is, given a fixed error bound for the SKI Gram matrix and use of cubic convolutional interpolation, how many inducing points are required to achieve that error bound? Given the required value of $m$ as a function of $n$, for what error tolerance is $O(n+m\log m)$ still linear? Following this, what do these errors imply for hyperparameter estimation and posterior inference?
\begin{table*}[h]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Quantity} & \textbf{Bound} \\
\hline
SKI kernel error & $O(\frac{c^{2d}}{m^{3/d}})$ \\
\hline
SKI Gram matrix error & $O(\frac{nc^{2d}}{m^{3/d}})$ \\
\hline
SKI cross-kernel matrix error & $O(\frac{\max(n,T)c^{2d}}{m^{3/d}})$ \\
\hline
SKI score function error & $O(\frac{\sqrt{p}n^{2}c^{4d}}{m^{3/d}})$ \\
\hline
SKI posterior mean error & $O(c^{2d}\frac{\max(T,n)+\sqrt{Tn}n}{m^{3/d}})$ \\
\hline
SKI posterior covariance error & $O(\frac{Tn^{2}mc^{4d}+\sqrt{Tn}mc^{4d}\max(T,n)}{m^{3/d}})$ \\
\hline
\end{tabular}
\caption{Summary of Theoretical Results when using SKI with convolutional cubic interpolation. This shows the rate at which the error of using SKI (vs the exact kernel) grows as a function of important variables. Here $n$ and $T$ are the train/test sample sizes, $d$ is the dimensionality, $m$ the number of inducing points, $p$ is the number of hyperparameters and $c>0$ is a constant. Most importantly, the Gram matrix error grows linearly with the sample size, exponentially with the dimension while decaying at an $m^{3/d}$ rate in the inducing points.}
\label{table:theoretical_results}
\end{table*}
In this paper, we begin to bridge the gap between practice and a theoretical understanding of SKI. We have three primary contributions: 1) The first error analysis for the SKI kernel and relevant quantities, including the SKI gram matrix's spectral norm error. Based on this we provide \textit{a practical guide to select the number of inducing points}: they should grow as $n^{d/3}$ to control error. 2) SKI hyperparameter estimation analysis. 3) SKI inference analysis: the error of the GP posterior means and variances at test points. We find two interesting results: 1) we identify two dimensionality regimes relating SKI Gram matrix error to computational complexity. For $d\leq 3$, for \textit{any} fixed spectral norm error, we can achieve it in linear time using SKI with a sufficient sample size. For $d>3$, the error must \textit{increase} with the sample size to maintain our guarantee of linear time. 2) For a $\mu$-smooth log-likelihood, gradient ascent on the SKI log-likelihood will approach a neighborhood of a stationary point of the true log-likelihood at a $O\left(\frac{1}{K}\right)$ rate, with the neighborhood size determined by the SKI score function's error, which aside from the response variables grows \textit{linearly} with the sample size when increasing inducing points as we suggested. To obtain this, we leverage a recent result \cite{stonyakin2023stopping} from the inexact gradient descent \cite{daspremont2008smooth,devolder2014first} literature.
% By our downstream analysis, this implies that for small dimensionality and kernels exhibiting desirable regularity conditions, we can have arbitrarily small error in the SKI Gram matrix, in our estimated parameters and in posterior inference all in linear time as long as the sample size is sufficiently large.
In section \ref{sec:related} we describe related work. In section \ref{sec:ski-background} we give a brief background on SKI. In section \ref{sec:important-quantities} we bound the error of important quantities: specifically the SKI kernel, Gram matrix and cross-kernel matrix errors. In section \ref{sec:gp-applications} we use these to analyze the error of the SKI MLE and posteriors. We conclude in section \ref{sec:discussion} by summarizing our results and discussing limitations and future work.
We can divide related works into three groups: those theoretically analyzing Gaussian process regression or kernel methods when using approximate kernels, SKI and its extensions, and papers developing techniques we use to obtain our guarantees. In the first group, the most relevant works are \cite{burt2019rates,burt2020convergence}, where they analyzed the sparse variational GP framework \citep{titsias2009variational,hensman2013gaussian} and derived bounds on the Kullback-Leibler divergence between the true posterior and the variational approximate posterior. \cite{moreno2023ski} gave bounds on the approximation error of the SKI Gram matrix. However, they only handled the case of univariate features and only bounded how much worse the SKI Gram matrix can be than the Nystr{\"o}m one. Further, they did not analyze the downstream effects on the approximate MLE or GP posterior. Also relevant are \cite{wynne2022variational,wild2021connections}, who gave a Banach space view of sparse variational GPs and connected them to the Nystr{\"o}m method, respectively. Finally, \cite{modell2024entrywise} provide entry-wise error bounds for low-rank approximations of kernel matrices: our approach also relies on entry-wise error bounds, but theirs are for the \textit{best} low-rank approximation to a given gram matrix, while ours are for the SKI gram matrix. Only one of these papers \cite{moreno2023ski} treated SKI specifically, and it only covered a very special case setting.
In the second group, the foundational work by \cite{wilson2015kernel} that we analyze introduced SKI as a scalable method for large-scale GP inference. \cite{kapoor2021skiing} extended SKI to high-dimensional settings using the permutohedral lattice. \cite{yadav2022kernel} developed a sparse grid approach to kernel interpolation that also helps address the curse of dimensionality. Most recently, \cite{ban2024malleable} proposed a flexible adaptation of SKI with a hyperparameter that adjusts the number of grid points based on kernel hyperparameters. We focus our analysis on the original technique of \cite{wilson2015kernel} in this paper, but future work could extend to the settings of the latter papers.
Also relevant are papers where we leverage or extend their results and proof techniques. We require a multivariate extension to the error analysis of \cite{keys1981cubic} for convolutional cubic interpolation, which we derive. We also use a recent result from the inexact gradient descent literature \cite{stonyakin2023stopping}, which allows us to analyze the effect of doing gradient ascent on the SKI log-likelihood instead of the true log-likelihood. Finally, we use a proof technique \cite{bach2013sharp,musco2017recursive} commonly used to bound the in-sample error of approximate kernel ridge regression to bound the test SKI mean function error.
% Further, when analyzing the hyperparameter MLE error of gradient ascent on the SKI log-likelihood, we use a strategy that is commonly used in the expectation maximization theory literature \cite{balakrishnan2017statistical,wu2016convergence} where they decompose the EM algorithm error into a population error, which decays geometrically, and a sample error, which involves a geometric sum. We instead analyze the error of gradient ascent on the SKI log-likelihood. We decompose into gradient ascent on the true log-likelihood, which under mild regularity conditions decays geometrically and the error due to using the SKI log-likelihood, which also involves a geometric sum.
This section provides background on Gaussian Processes (GPs) and two key techniques for enabling scalable inference: Structured Kernel Interpolation (SKI) and Convolutional Cubic Interpolation. SKI \cite{wilson2015kernel} addresses GPs scalability issue by approximating the kernel matrix through interpolation on a set of inducing points, leveraging the efficiency of convolutional kernels. In particular, cubic convolutional kernels, as detailed in \cite{keys1981cubic}, provide a smooth and accurate interpolation scheme that forms the foundation of the SKI framework. In this paper, we focus on this cubic case as it is used by SKI. Future work may extend this to study higher-order interpolation methods. Here, we formally define these concepts and lay the groundwork for the subsequent error analysis.
\subsection{Gaussian Processes}\label{sec:gp-background}
A Gaussian process $\xi\sim \textrm{GP}(\nu,k_\theta)$ is a stochastic process $\{\xi(\textbf{x})\}_{\textbf{x}\in \mathcal{X}}$ such that any finite subcollection $\{\xi(\textbf{x}_i)\}_{i=1}^n$ is multivariate Gaussian distributed. We assume that we have index locations $\textbf{x}_i\in \mathbb{R}^d$ and observations $y_i\in \mathbb{R}$ for a set of training points $i=1,\ldots,n$ such that
\begin{align*}
y_i&=\xi(\textbf{x}_i)+\epsilon_i,\epsilon_i\sim \mathcal{N}(0,\sigma^2).
\end{align*}
where $\nu:\mathcal{X}\rightarrow\mathbb{R}$, $k_\theta:\mathcal{X}\times \mathcal{X}\rightarrow\mathbb{R}$ are the prior mean and covariance functions, respectively, with $k$ an SPD kernel with hyperparameters $\theta$. Given $\{\textbf{x}_i,y_i\}_{i=1}^n$ we are primariliy interested in two tasks: 1) estimate hyperparameters $\boldsymbol{\theta}\in \Theta\subseteq \mathbb{R}^p$ of kernel $k_\theta$ (e.g. RBF kernel) 2) do Bayesian inference for the posterior mean $\boldsymbol{\mu}(\cdot)\in \mathbb{R}^{T}$ and covariance $\boldsymbol{\Sigma}(\cdot)\in \mathbb{R}^{T\times T}$ at a set of test points $\{\textbf{x}_t\}_{t=1}^T$. Assuming $\nu\equiv 0$ (a mean-zero GP prior), for 1), one maximizes the log-likelihood
\begin{align}
\mathcal{L}(\boldsymbol{\theta};X) &= -\frac{1}{2}\textbf{y}^\top (\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}\nonumber\\
&\qquad-\frac{1}{2}\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert -\frac{n}{2}\log (2\pi)\label{eqn:log-likelihood}
\end{align}
to find $\boldsymbol{\theta}\in \mathcal{D}\subseteq \Theta$ where $\textbf{K}\in \mathbb{R}^{n\times n}$ with entries $\textbf{K}_{ij}=k_\theta(\textbf{x}_i,\textbf{x}_j)$ is the Gram matrix for the training dataset. For 2), given the kernel function and known observation variance $\sigma^2$, the posterior mean and covariance are given by
\begin{align}
\boldsymbol{\mu}(\cdot) &= \mathbf{K}_{\cdot, \mathbf{X}}\left(\mathbf{K}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\label{eqn:posterior-mean}\\
\boldsymbol{\Sigma}(\cdot) &= \textbf{K}_{\cdot,\cdot}+\sigma^2I-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}\label{eqn:posterior-covariance}
\end{align}
where $\textbf{K}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ is the matrix of kernel evaluations between test and training points. Intuitively, the GP prior represents our belief about all possible functions before seeing any data. When we observe data points, the posterior represents our updated belief - it gives higher probability to functions that fit our observations while maintaining the smoothness properties encoded in the kernel. The posterior mean can be viewed as a weighted average of these functions, where the weights depend on how well each function fits the data and satisfies the prior assumptions. The posterior variance indicates our remaining uncertainty - it is smaller near observed points where we have more confidence, and larger in regions far from our data.
A challenge is that, between the log-likelihood and the posteriors, one first needs to compute the action of the inverse of the regularized Gram matrix, $(\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}$. Second, one needs to compute the log-determinant $\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert$. These are both $O(n^3)$ computationally and $O(n^2)$ memory.
\subsection{Structured Kernel Interpolation}\label{subsec:ski}
Structured kernel interpolation \citep{wilson2015kernel} or (SKI) addresses these computational and memory bottlenecks by approximating the original kernel function \(k_\theta:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R},\mathcal{X}\subseteq\mathbb{R}^d\) by interpolating kernel values at a chosen set of inducing points \(\mathbf{U}=\left(\begin{array}{c}
\textbf{u}_1^\top\\
\vdots\\
\textbf{u}_m^\top
\end{array}\right)\in \mathbb{R}^{m\times d}\). The approximate kernel function \(\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow\mathbb{R}\) can be expressed as:
\[
\tilde{k}(\mathbf{x}, \mathbf{x}') = \mathbf{w}(\mathbf{x})^{\top} \mathbf{K}_{\mathbf{U}} \mathbf{w}(\mathbf{x}')
\]
where \(\mathbf{K}_{\mathbf{U}}\in \mathbb{R}^{m\times m}\) is the kernel matrix computed on the inducing points, and \(\mathbf{w}(\mathbf{x}),\mathbf{w}(\mathbf{x}')\in \mathbb{R}^m\) are vectors of interpolation weights using (usually cubic) convolutional kernel $u:\mathbb{R}\rightarrow \mathbb{R}$ for the points $\textbf{x}$ and $\textbf{x}'$, respectively. One can then form the SKI Gram matrix $\tilde{\textbf{K}}=\textbf{W}\textbf{K}_\textbf{U}\textbf{W}^\top$ with $\textbf{W}$ a \textit{sparse} matrix of $L$ interpolation weights per row for a polynomial of degree $L-1$. By exploiting the sparsity of each row, for stationary kernels this leads to a computational complexity of $O(nL+m\log m)$ and a memory complexity of $O(nL+m)$.
In order to learn kernel hyperparameters, one can maximize the SKI approximation to the log-likelihood (henceforth the SKI log-likelihood)
\begin{align*}
\tilde{\mathcal{L}}(\boldsymbol{\theta};X)
&=-\frac{1}{2}\textbf{y}^\top (\tilde{\textbf{K}}+\sigma^2 \textbf{I})^{-1}\textbf{y}\\
&\qquad-\frac{1}{2}\log \vert \tilde{\textbf{K}}+\sigma^2 \textbf{I}\vert -\frac{n}{2}\log (2\pi)
\end{align*}
Given the SKI kernel $\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ with learned hyperparameters, one can do posterior inference of the SKI approximations to the mean $\tilde{\boldsymbol{\mu}}(\cdot)$ and covariance $\tilde{\boldsymbol{\Sigma}}(\cdot)$ at a set of $T$ test points $\cdot$ as
\begin{align*}
\tilde{\boldsymbol{\mu}}(\cdot)&=\tilde{\mathbf{K}}_{\cdot, \mathbf{X}}\left(\tilde{\mathbf{K}}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\\
\tilde{\boldsymbol{\Sigma}}(\cdot)&=\tilde{\textbf{K}}_{\cdot,\cdot}+\sigma^2I-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}
\end{align*}
where $\tilde{\textbf{K}}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ is the matrix of SKI kernels between test points and training points and $\tilde{\textbf{K}}_{\cdot,\cdot}\in \mathbb{R}^{T\times T}$ is the SKI Gram matrix for the test points. Going forward, we may write $\mathcal{L}(\boldsymbol{\theta})$ and $\tilde{\mathcal{L}}(\boldsymbol{\theta})$, dropping the explicit dependence on the data but implying it.
\subsection{Convolutional Cubic Interpolation}\label{subsec:convolutional-cubic-interpolation}
Convolutional cubic interpolation \citep{keys1981cubic} gives a continuously differentiable interpolation of a function given its values on a regular grid, where its cubic convolutional kernel is a piecewise polynomial function designed to ensure continuous differentiability. We formalize this using the definitions of the cubic convolutional interpolation kernel and the tensor-product cubic convolutional function below. We also define an upper bound for the sum of weights for each dimension, which will be a useful constant going forward. Such a bound will exist for all continuous stationary kernels vanishing at infinity.
\begin{definition}\label{def:cubic-interpolation-kernel}
The cubic convolutional interpolation kernel $u:\mathbb{R}\rightarrow\mathbb{R}$ is given by
$$
u(s)\equiv\begin{cases}
1, & s=0\\
\frac{3}{2}\vert s\vert^3-\frac{5}{2}\vert s\vert^2+1, & 0<\vert s\vert <1\\
-\frac{1}{2}\vert s\vert^3+\frac{5}{2}\vert s\vert^2-4\vert s\vert+2, & 1<\vert s\vert<2\\
0, & \text{otherwise}
\end{cases}
$$
\end{definition}
\begin{definition}\label{def:tensor-product-cubic-interpolation-alt1}
Let $\mathbf{x} = (x_1, x_2, ..., x_d) \in \mathbb{R}^d$ be a d-dimensional point. Let $f:\mathbb{R}^d \rightarrow \mathbb{R}$ be a function defined on a regular grid with spacing $h$ in each dimension. Let $\mathbf{c_x}$ denote the grid point closest to $\mathbf{x}$. The tensor-product cubic convolutional interpolation function $g:\mathbb{R}^d\rightarrow \mathbb{R}$ is defined as:
{\footnotesize
\begin{align*}
g(\mathbf{x}) \equiv \sum_{\mathbf{k} \in \{-1, 0, 1, 2\}^d} f(\mathbf{c_x} + h\mathbf{k}) \prod_{j=1}^d u\left(\frac{x_j - (\mathbf{c_x})_j - h k_j}{h}\right)
\end{align*}
}
where $u$ is the cubic convolutional interpolation kernel and $\mathbf{k} = (k_1, \ldots, k_d)$ is a vector of integer indices.
\end{definition}
\begin{definition}\label{def:sum-weight-upper-bound}
Given an interpolation kernel $u:\mathbb{R}\rightarrow\mathbb{R}$ and a fixed $n\in \mathbb{N}$, let $c>0$ be an upper bound such that, for any $x\in \mathbb{R}$ and a set of data points $\{x_i\}_{i=1}^n \subset \mathbb{R}$,
$$
\sum_{i=1}^n \left\vert u\left(\frac{x - x_i}{h}\right) \right\vert\leq c,
$$
\end{definition}
Going forward, we always assume that we use convolutional cubic polynomial interpolation, so that $L=4$ as in \cite{wilson2015kernel}, but that we may vary the number of inducing points $m$. In particular, we will analyze how the number of inducing points affects error for different terms of interest, and how to choose the number of inducing points.
This section derives bounds for key quantities in Structured Kernel Interpolation (SKI). Section \ref{subsubsec:elementwise-error} provides a bound on the elementwise error between the true kernel and its SKI approximation. In Section \ref{subsubsec:spectral-norm-error}, we extend this to the spectral norm error of the SKI approximation for the training Gram matrix and train-test kernel matrix. Finally, in section \ref{subsec:linear-time} we present conditions on the number of inducing points for achieving specific error tolerance $\epsilon>0$ and error needed to guarantee linear time complexity, noting linear time always holds for $d\leq 3$ with sufficiently large samples.
\subsection{Error Bounds for the Ski Kernel}\label{subsec:ski-kernel-error-bounds}
This subsection analyzes the error introduced by the SKI approximation of the kernel function. We start by extending the analysis of \cite{keys1981cubic} to the multivariate setting, deriving error bounds for multivariate cubic convolutional polynomial interpolation. We then use these to derive the elementwise error for the SKI approximation \(\tilde{k}(\textbf{x},\textbf{x}')\). We next apply these elementwise bounds to derive spectral norm error bounds for SKI kernel matrices, which will be crucial for understanding the downstream effects of the SKI approximation on Gaussian process hyperparameter estimation and posterior inference.
\subsubsection{Elementwise}\label{subsubsec:elementwise-error}
Our first lemma shows that multivariate tensor-product cubic convolutional interpolation retains error cubic in the grid spacing of \cite{keys1981cubic}, which is equivalent to $m^{-3/d}$ decay with the number of inducing points $m$, but exhibits exponential error growth with increasing dimensions. The proof uses induction on dimensions, starting with the 1D case from Keys.
\begin{restatable}{lemma}{tensorproductinterpolationerror}\label{lemma:tensor-product-interpolation-error}
The error of tensor-product cubic convolutional interpolation is $O(c^d h^3)$, or equivalently $O\left(\frac{c^d}{m^{3/d}}\right)$.
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proof-tensor-product-interpolation}.
\end{proof}
The following Lemma allows us to bound the absolute difference between the true and SKI kernels \textit{uniformly} with the same big-$O$ error as for the underlying interpolation itself. The proof uses the the triangle inequality to decompose the error into two parts: the first is the error from a single interpolation, while the second is the error of the nested interpolations.
\begin{restatable}{lemma}{skikernelelementwiseerror}\label{lemma:ski-kernel-elementwise-error}
Let $\delta_{m,L}$ be the interpolation error for $m$ inducing points and interpolation degree $L-1$. The SKI kernel $\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ with grid spacing $h$ in each dimension has error
\begin{align*}
\vert k(\textbf{x},\textbf{x}')-\tilde{k}(\textbf{x},\textbf{x}')\vert&= \delta_{m,L}+\sqrt{L}c^d\delta_{m,L}\\
&=O\left(\frac{c^{2d}}{m^{3/d}}\right).
\end{align*}
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proof-ski-kernel-elementwise-error}
\end{proof}
\subsubsection{Spectral Norm Error}\label{subsubsec:spectral-norm-error}
We now transition from elementwise error bounds to spectral norm bounds for the SKI gram matrix's approximation error, finding that it grows linearly with the sample size and exponentially with the dimension and decays as $m^{-3/d}$ with the number of inducing points. This is both of independent interest but will also be important to nearly all downstream analysis for estimation and inference. We also provide a bound on the spectral norms of the SKI train/test kernel matrix's approximation error. This is useful when analyzing the GP posterior parameter error.% Finally, we bound the error of the SKI gram matrix's regularized inverse, which is useful for analyzing the SKI MLE's error and the GP posterior parameter error.
For this next lemma we will express it both in the general interpolation setting and again give the specific big-$O$ for convolutional cubic interpolation, but going forward we sometimes only show the latter setting in the main paper and derive the general settings in the proof. In particular, \emph{\textbf{whenever we use big $O$-notation}} we are assuming convolutional cubic interpolation.
\begin{restatable}{proposition}{spectralnorm}\label{prop:spectral-norm}
For the SKI approximation $\tilde{\textbf{K}}$ of the true Gram matrix $\textbf{K}$, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&=n \left(\delta_{m,L}+\sqrt{L} c^d\delta_{m,L}\right)\\
&\equiv \gamma_{n,m,L}\\
&= O\left(\frac{nc^{2d}}{m^{3/d}}\right)
\end{align*}
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proof-spectral-norm}
\end{proof}
\begin{restatable}{lemma}{testtrainkernelmatrixerror}\label{lemma:test-train-kernel-matrix-error}
Let \(\textbf{K}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}\) be the matrix of kernel evaluations between \(T\) test points and \(n\) training points, and let \(\tilde{\textbf{K}}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}\) be the corresponding SKI approximation. Then
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2=O\left(\frac{\max(n,T)c^{2d}}{m^{3/d}}\right)
\end{align*}
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proof-test-train-kernel-matrix-error}.
\end{proof}
\subsection{Achieving Errors in Linear Time}\label{subsec:linear-time}
Here, we show how many inducing points $m$ are sufficient to achieve a desired error tolerance $\epsilon > 0$ for the SKI Gram matrix when using cubic convolutional interpolation. Based on the Theorem, we should grow the number of inducing points at an $n^{d/3}$ rate. We then show corollaries describing 1) how $\epsilon$ and $m$ must grow to maintain linear time 2) how the dimension affects whether the error must grow with the sample size to ensure linear time SKI.
The following theorem shows the number of inducing points that will guarantee a Gram matrix error tolerance. It says that the number of inducing points should grow as $n^{d/3}$ to achieve a fixed error. The proof starts by lower bounding the desired spectral norm error with the upper bound on the actual spectral norm error derived in Proposition \ref{prop:spectral-norm}: this is a sufficient condition for the desired spectral norm error to hold. It then relates the number of inducing points to the grid spacing in the SKI approximation, assuming a regular grid with equal spacing in each dimension. By substituting this relationship into the sufficient condition, the proof derives the sufficient number of inducing points to control error.
\begin{restatable}{theorem}{inducingpointscountalt}\label{thm:inducing-points-count-alt}
If the domain is $[-D, D]^d$, then to achieve a spectral norm error of $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$, it is sufficient to choose the number of inducing points $m$ such that:
$$
m = \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
$$
for some constant $K'$ that depends only on the kernel function and the interpolation scheme.
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proof-inducing-points-count-alt}.
\end{proof}
This result shows that the number of inducing points should grow
\begin{itemize}
\item Sub-linearly with the sample size and decrease in error for $d<3$, linearly for $d=2$ and super-linearly for $d>3$. Thus, as we want a tighter error tolerance or have more observations we need more inducing points, but at very different rates depending on the dimensionality.
\item Linearly with the volume of the domain $(2D)^d$. Thus, if our observations are concentrated in a small region and we select an appropriately sized domain to cover it we need fewer inducing points.
\item Exponentially with the dimension $d$, as we have a $c^{2d}$ term.
\end{itemize}
The next Corollary establishes a condition on the spectral norm error, $\epsilon$, that ensures linear-time $O(n)$ computational complexity for SKI. The core idea is that $\epsilon$ should be such that if we choose $m$ based on the previous Theorem, $m=O(n/\log n)$ and thus $m\log m=O(n)$.
\begin{restatable}{corollary}{corlineartime}\label{cor:linear-time}
If
\begin{equation} \label{eq:epsilon_condition}
\epsilon \geq \frac{(1 + 2c^d) K' 8 c^{2d} D^3}{C^{3/d}} \cdot \frac{n (\log n)^{3/d}}{n^{3/d}}
\end{equation}
for some constants $K,C>0$ that depend on the kernel function and the interpolation scheme and we choose $m>0$ based on the previous theorem, then we have both $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2\leq \epsilon$ and
SKI computational complexity of $O(n)$.
\end{restatable}
\begin{proof}
See Appendix \ref{proof:cor-linear-time}.
\end{proof}
Interestingly, the previous Theorem and Corollary implies a fundamental difference between two dimensionality regimes. For $d \leq 3$, the choice of $m$ required for a fixed $\epsilon$ grows more slowly than $n/\log n$. This means that for any fixed $\epsilon > 0$, SKI with cubic interpolation is guaranteed to be a linear-time algorithm for sufficiently large $n$. In contrast, for $d > 3$, the choice of $m$ required for a fixed $\epsilon>0$ eventually grows faster than $n/\log n$. Thus, to maintain linear-time complexity for $d > 3$ and the guarantees from Theorem \ref{thm:inducing-points-count-alt}, we must allow the error $\epsilon$ to increase with $n$. This demonstrates that the curse of dimensionality impacts the scalability of SKI, making it challenging to achieve both high accuracy and linear-time complexity in higher dimensions. The next corollary formalizes this.
\begin{corollary}
For $d\leq 3$, for any $\epsilon>0$, Corollary \ref{cor:linear-time} holds for any $n$ sufficiently large, so that choosing $m$ based on Theorem \ref{thm:inducing-points-count-alt} is sufficient to achieve linear complexity. For $d>3$, $\epsilon$ must grow with the sample size to maintain linear complexity.
\end{corollary}
\begin{proof}
For $d\leq3 $, the RHS of Eqn. \ref{eq:epsilon_condition} decreases with $n$ with limit $0$ and thus for sufficiently large sample size will be $\leq \epsilon$, satisfying the conditions to guarantee small error and linear time. For $d>3$, the RHS of Eqn. \ref{eq:epsilon_condition} grows with $n$, so that $\epsilon$ must grow to satisfy the conditions for the guarantee.
\end{proof}
% \subsubsection{Log Determinant}\label{subsubsec:log-determinant}
% \begin{lemma}\label{lemma:log-determinant-error}
% Let \(\mathbf{\tilde{K}}\) be the SKI approximation of the kernel matrix \(\mathbf{K}\), and let \(\sigma^2\) be the regularization parameter. The error in the log-determinant of the regularized SKI matrix can be bounded as follows:
% \[
% \left|\log \det(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - \log \det(\mathbf{K} + \sigma^2 \mathbf{I})\right| \leq \frac{\gamma_{n, m, L}}{\sigma^2}.
% \]
% \end{lemma}
% \begin{proof}
% We begin by applying the functional Mean Value Theorem for matrices, which implies that for some matrix \(Z = (\mathbf{\tilde{K}} + \alpha (\mathbf{K} - \mathbf{\tilde{K}})) + \sigma^2 \mathbf{I}\) with \(\alpha \in [0, 1]\),
% \[
% |f(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - f(\mathbf{K} + \sigma^2 \mathbf{I})| \leq \|f'(Z)\|_2 \|\mathbf{K} - \mathbf{\tilde{K}}\|_2
% \]
% where \(f(X) = \log |X|\) and thus \(f'(X) = X^{-\top}\).
% Thus,
% \[
% |\log \det(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - \log \det(\mathbf{K} + \sigma^2 \mathbf{I})| \leq \left\| (\mathbf{\tilde{K}} + \alpha (\mathbf{K} - \mathbf{\tilde{K}}) + \sigma^2 I)^{-1} \right\|_2 ||\mathbf {K} - {\tilde {\mathbf {K}}}||_2\label{eqn:initial-log-det-bound}
% \]
% Using Claim 1, which bounds the eigenvalues of a convex combination of matrices, we have:
% \[
% \lambda_{\min}(Z) = (\lambda_{\min} (\mathbf{\tilde{K}})+ (1-\alpha)\lambda_{\min} (\mathbf {K}))+\sigma^2 > 0
% \]
% Therefore since $\mathbf{\tilde{K}}$ and $(\mathbf {K}$ are SPD and thus have positive eigenvalues,
% \[
% \left\| (\mathbf{\tilde{K}} + (1-\alpha)(\mathbf {K}- {\tilde {\mathbf {K}}})+ {\sigma^2 I})^{-1} )^{-1}\right\|_2 < {\frac {1}{ {\sigma^2}}}
% \]
% Substituting these bounds into Eqn. \ref{eqn:initial-log-det-bound} gives:
% \[
% |\log \det (\tilde{\mathbf {K}}+ {\sigma^2 I})- \log \det ({\mathbf {K}}+ {\sigma^2 I})| < {\frac {1}{ {\sigma^2}}} ||{\mathbf {K}- {\tilde {\mathbf {K}}}}||_2
% \leq {\frac {1}{ {\sigma^2}}} {\gamma_{n,m,L}}
% \]
% \end{proof}
In this section, we address how SKI affects Gaussian Processes Applications. In Section \ref{sec:kernel-hyperparameter-estimation} we address how using the SKI kernel and log-likelihood affect hyperparameter estimation. In Section and posterior inference.
\subsection{Kernel Hyperparameter Estimation}\label{sec:kernel-hyperparameter-estimation}
\subsubsection{Difference in MLEs}
We formalize the intuition that a good initialization for gradient ascent on the SKI log-likelihood, combined with a sufficiently accurate SKI approximation, leads to a SKI MLE that is close to the true MLE. We achieve this by first bounding the difference in partial derivatives between the true and SKI log-likelihoods, assuming that the partial derivatives themselves are SPD kernels. Then, under the assumption of one point strong concavity of the true log-likelihood at the true parameter, we show that if the initialization is close to the true MLE, the resulting SKI MLE will also be close to the true MLE. Finally, we show that for the Gaussian RBF kernel, for the signal variance parameter 1) the derivative of the kernel with respect to it is itself an SPD kernel 2) local strong concavity holds when learning it while fixing the lengthscale parameter. We leave the highly challenging case of the lengthscale to future work. To start with, we define our notation. Let $\mathcal{X} \subset \mathbb{R}^d$ be the input domain and let $\Theta \subset \mathbb{R}^p$ be the domain of the kernel hyperparameters. Further, let $k_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be a kernel function parameterized by $\theta \in \Theta$, and let $\tilde{k}_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be its SKI approximation using $m$ inducing points and interpolation degree $L-1$.
We now state our assumptions: 1) one point strong concavity and gradient smoothness of the population log-likelihood 2) That the kernel's partial derivatives are themselves SPD kernels. 3) That the kernel is continuous differentiable. We then mention an equivalence between strong concavity and the Hessian having an upper bound on its smallest eigenvalue.
\begin{assumption}[One Point Strong Concavity of the Population Log-Likelihood]\label{assumption:strong-concavity}
The population log-likelihood $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave at $\theta^*$ with parameter $\mu > 0$ in a neighborhood $\mathcal{D}$ of the true hyperparameters $\theta^*$. That is, for all $\theta\in \mathcal{D}$,
$$
\bar{\mathcal{L}}(\theta) \leq \bar{\mathcal{L}}(\theta^*) + \nabla \bar{\mathcal{L}}(\theta^*)^\top (\theta - \theta^*) - \frac{\mu}{2} \| \theta - \theta^* \|_2^2.
$$
\end{assumption}
\begin{assumption}
(One Point Gradient Smoothness of the Population Log-Likelihood) $\| \nabla \bar{\mathcal{L}}(\theta) - \nabla \bar{\mathcal{L}}(\theta^*) \|_2 \leq L \| \theta - \theta^* \|_2$ for some $L>0$.
\end{assumption}
\begin{assumption}
(SPD Kernel Partials) For each $i \in \{1, ..., p\}$, the partial derivative of $k_{\theta}$ with respect to a hyperparameter $\theta_i$, denoted as $k'_{\theta_i}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$, is also a valid SPD kernel.
\end{assumption}
\begin{assumption}
(Kernel Smoothness) $k_\theta(x,x')$ is $C^1$ in $\theta$. That is, for each $i \in \{1, ..., p\}$, $k'_{\theta_i}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$ exists and is Lipschitz for $\theta\in \mathcal{D}$.
\end{assumption}
\begin{proposition}\label{prop:strong_concavity_hessian}
A function is strongly concave if and only if the Hessian matrix $H_f(\theta)$ satisfies $H_f(\theta) \preceq -\mu I$ for all $\theta \in \mathcal{D}$, where $\preceq$ denotes the negative semidefinite ordering.
\end{proposition}
\begin{proof}
See \cite{boyd2004convex}.
\end{proof}
Here we argue that we can apply the same elementwise error we derived previously to the SKI partial derivatives.
\begin{restatable}{lemma}{skikernelderivativeerrorkernel}[Bound on Derivative of SKI Kernel Error using Kernel Property of Derivative]
\label{lemma:ski_kernel_derivative_error_kernel}
Let $\tilde{k}'_{\theta}(x,x')$ be the SKI approximation of $k'_{\theta_i}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$. Then, for all $x, x' \in \mathcal{X}$, all $\theta \in \Theta$, the following inequality holds:
\begin{equation*}
\left\vert \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}-\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i}\right\vert = \left\vert k'_{\theta_i}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \leq\delta_{m,L}'+\sqrt{L}c^d\delta_{m,L}',
\end{equation*}
where $\delta_{m,L}'$ is an upper bound on the error of the SKI approximation of the kernel $k'_{\theta_i}(x,x')$ with $m$ inducing points and interpolation degree $L-1$, as defined in Lemma \ref{lemma:ski-kernel-elementwise-error}.
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proofski_kernel_derivative_error_kernelz}
\end{proof}
We then use the elementwise bound to bound the spectral norm of the SKI kernel's gradient error.
\begin{restatable}{lemma}{gradientspectralnormbound}[Bound on Spectral Norm of Gradient Difference]
\label{lemma:gradient_spectral_norm_bound}
Assume that $k_{\theta}$ with respect to $\theta_i$, denoted as $k'_{\theta,i}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$, is a valid SPD kernel. Let $\tilde{k}'_{\theta,i}(x,x')$ be the SKI approximation of $k'_{\theta,i}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$.
Then, the spectral norm of the difference between the gradient of the true kernel matrix and the gradient of the SKI kernel matrix is bounded by:
\begin{equation}
\| \nabla_\theta K - \nabla_\theta \tilde{K} \|_2 \leq \sqrt{p} \max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i},
\end{equation}
where $\nabla_\theta K$ and $\nabla_\theta \tilde{K}$ are represented as $p \times n^2$ matrices using the vec-notation (denominator layout), $p$ is the number of hyperparameters, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ (analogous to Proposition \ref{prop:spectral-norm}, but for the kernel $k'_{\theta,i}$).
\end{restatable}
\begin{proof}
See Section \ref{sec:proof_gradient_spectral_norm_bound}
\end{proof}
Next, we extend the previous result on the SKI kernel's spectral norm gradient error to the SKI log-likelihood's spectral norm gradient error.
\begin{lemma}[Corrected Bound on Gradient Difference]\label{lemma:gradient-difference-bound-corrected}
Let $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ be the true log-likelihood and $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$ be the SKI approximation of the log-likelihood. Let $\nabla \mathcal{L}(\theta)$ and $\nabla \tilde{\mathcal{L}}(\theta)$ denote their respective gradients with respect to $\theta$. Then, for any $\theta\in \mathcal{D}$,
$$
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 \leq \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_n\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right],
$$
where $\gamma_{n,m,L}$ bounds the elementwise difference between $\mathbf{K}$ and $\tilde{\mathbf{K}}$, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ and $C_n$ is a constant depending on the derivatives of the kernel function and the sample size.
\end{lemma}
\begin{proof}
We start with the expressions for the gradients:
$$
\nabla \mathcal{L}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
$$
\nabla \tilde{\mathcal{L}}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
Thus, the difference is:
\begin{align*}
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 &= \left\| \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| \right) \right. \\
&\quad \left. - \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&\leq \underbrace{\left\| \nabla \left( \frac{1}{2} \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2}_{T_1} \\
&\quad + \underbrace{\left\| \frac{1}{2} \nabla \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2}_{T_2}.
\end{align*}
We will bound $T_1$ and $T_2$ separately.
**Bounding $T_1$:**
Using the chain rule and the matrix derivative identity $\frac{\partial}{\partial \theta} \mathbf{X}^{-1} = -\mathbf{X}^{-1} (\frac{\partial \mathbf{X}}{\partial \theta}) \mathbf{X}^{-1}$, we have
\begin{align*}
T_1 &= \frac{1}{2} \left\| \nabla_\theta \left( \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2 \\
&= \frac{1}{2} \left\| \sum_{i=1}^n \sum_{j=1}^n y_i y_j \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2 \\
&\leq \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n |y_i y_j| \left\| \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2.
\end{align*}
Now, applying the matrix derivative identity, we get:
\begin{align*}
&\left\| \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} + \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\right.\\
&\quad \left.\left. + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \nabla_\theta \mathbf{K} (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\right.\\
&\quad \left.\left. - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&\leq \underbrace{\left\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(a)} \\
&\quad + \underbrace{\left\| \left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right)(\nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(b)} \\
&\quad + \underbrace{\left\|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}(\nabla_\theta \mathbf{K})\left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right) \right\|_{2, }}_{(c)}\\
\end{align*}
We now explicitly bound (a), (b), and (c).
\begin{align*}
(a) &\leq \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2 \| \nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} \|_2 \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2\\
&\leq \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2^2 \| \nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} \|_2 \\
&\leq \frac{1}{\sigma^4} \| \nabla_\theta \mathbf{K} - \nabla_\theta \tilde{\mathbf{K}} \|_2\text{ Proposition }\ref{lemma:gradient_spectral_norm_bound}\\
&\leq \frac{1}{\sigma^4}\sqrt{p} \max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}.
\end{align*}
\begin{align*}
(b) &\leq \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \|\nabla_\theta \mathbf{K}\|_2\text{ proof of Lemma \ref{lemma:action-inverse-error}}.
\end{align*}
\begin{align*}
(c) &\leq \|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \|\nabla_\theta \mathbf{K}\|_2\text{ proof of Lemma \ref{lemma:action-inverse-error}}.
\end{align*}
As the kernel is $C^1$ with respect to $\theta$, $\|\nabla_\theta \mathbf{K}\|$ can be bounded by a constant over $\mathcal{D}$, say $C_n$. Thus, we can bound $T_1$ as:
\begin{align*}
T_1 &\leq \sum_{i,j} \vert y_iy_j\vert \frac{1}{\sigma^4} \left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_n\gamma_{n,m,L}\right)\\
&\leq \frac{1}{\sigma^4}\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_n\gamma_{n,m,L}\right)\sum_{i,j} \vert y_iy_j\vert \\
&=\frac{\Vert y\Vert_1^2}{\sigma^4}\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_n\gamma_{n,m,L}\right)
\end{align*}
**Bounding $T_2$:**
Using the identity $\nabla \log |\mathbf{X}| = (\mathbf{X}^{-1})^\top$, we have
\begin{align*}
T_2 &= \frac{1}{2} \left\| \nabla_\theta \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&= \frac{1}{2} \left\| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_2.
\end{align*}
We can rewrite the difference as:
$$
(\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} = (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\tilde{\mathbf{K}} - \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}
$$
Then, applying Lemma \ref{lemma:action-inverse-error} we obtain
\begin{align*}
T_2 &\leq \frac{1}{2} \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2 \| \tilde{\mathbf{K}} - \mathbf{K} \|_2 \| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \|_2 \\
&\leq \frac{\gamma_{n,m,L}}{2\sigma^4}
\end{align*}
where $C_2(\theta)$ is another constant depending on $\theta$ (from the derivatives of the kernel when applying Lemma \ref{lemma:action-inverse-error}).
**Combining the Bounds:**
Combining the bounds for $T_1$ and $T_2$, we have
$$
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 \leq \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_n\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right].
$$
\end{proof}
\begin{proposition}[One-Step Improvement on True Log-Likelihood]\label{prop:one-step-improvement-on-true-log-likelihood}
Let $\theta^*\in \mathcal{D}$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient in $\mathcal{D}$.
If the step size $\eta$ satisfies $\eta \leq \frac{1}{L}$ (where $L$ is the Lipschitz constant of the gradient of the true log-likelihood) and $\eta\mu < 1$,
then starting at $\theta_0\in \mathcal{D}$, after one step of projected gradient ascent on $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$, yielding $\theta_1$, we have:
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2
$$
\end{proposition}
\begin{proof}
By the optimality of $\theta^*$, we have $\nabla \mathcal{L}(\theta^*) = 0$.
By the $\mu$-strong concavity of $\mathcal{L}$, we have for any $\theta$:
$$
\mathcal{L}(\theta) \leq \mathcal{L}(\theta^*) + \nabla \mathcal{L}(\theta^*)^\top (\theta - \theta^*) - \frac{\mu}{2} \| \theta - \theta^* \|_2^2 = \mathcal{L}(\theta^*) - \frac{\mu}{2} \| \theta - \theta^* \|_2^2
$$
Rearranging, we get:
\begin{equation} \label{eq:strong_concavity_alt}
\| \theta - \theta^* \|_2^2 \leq \frac{2}{\mu} (\mathcal{L}(\theta^*) - \mathcal{L}(\theta))
\end{equation}
Since $\theta_1$ is obtained by one step of projected gradient ascent from $\theta_0$, we have:
$$
\theta_1 = \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \mathcal{L}(\theta_0))
$$
By the properties of projection onto a convex set, we have:
$$
\| \theta_1 - \theta^* \|_2^2 \leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2^2
$$
Expanding the right-hand side, we get:
\begin{align*}
\| \theta_1 - \theta^* \|_2^2 &\leq \| \theta_0 - \theta^* \|_2^2 + 2 \eta (\theta_0 - \theta^*)^\top \nabla \mathcal{L}(\theta_0) + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \\
&\leq \| \theta_0 - \theta^* \|_2^2 + 2 \eta (\mathcal{L}(\theta_0) - \mathcal{L}(\theta^*) + \frac{\mu}{2} \| \theta_0 - \theta^* \|_2^2) + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \\
&\text{(by strong concavity, with } \nabla \mathcal{L}(\theta^*) = 0 \text{)}\\
&\leq \| \theta_0 - \theta^* \|_2^2 - \eta \mu \| \theta_0 - \theta^* \|_2^2 + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \quad \text{(by \eqref{eq:strong_concavity_alt})} \\
&= (1 - \eta \mu) \| \theta_0 - \theta^* \|_2^2 + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2
\end{align*}
By the Lipschitz continuity of the gradient, we have:
$$
\| \nabla \mathcal{L}(\theta_0) \|_2 = \| \nabla \mathcal{L}(\theta_0) - \nabla \mathcal{L}(\theta^*) \|_2 \leq L \| \theta_0 - \theta^* \|_2
$$
Thus,
$$
\| \theta_1 - \theta^* \|_2^2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2^2 + \eta^2 L^2 \| \theta_0 - \theta^* \|_2^2 = (1 - \eta \mu + \eta^2 L^2) \| \theta_0 - \theta^* \|_2^2
$$
Taking the square root of both sides:
$$
\| \theta_1 - \theta^* \|_2 \leq \sqrt{1 - \eta \mu + \eta^2 L^2} \| \theta_0 - \theta^* \|_2
$$
If we choose $\eta \leq \frac{1}{L}$, we have:
$$
\sqrt{1 - \eta \mu + \eta^2 L^2} \leq \sqrt{1-2\eta \mu+\eta^2\mu^2} = 1 - \eta \mu
$$
Thus,
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2
$$
as long as $\eta \leq \frac{1}{L}$.
\end{proof}
\begin{proposition}[MLE Closeness via One-Step Improvement]\label{prop:mle-closeness-one-step}
Let $\theta^*$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient near $\theta^*$. Assume that the gradient difference is bounded:
$$
\| \nabla \tilde{\mathcal{L}}(\theta) - \nabla \mathcal{L}(\theta) \|_2 \leq \epsilon_g:= \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\frac{1}{\sqrt{p}}\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right]
$$
for all $\theta\in \mathcal{D}$.
If the step size $\eta>0$ satisfies $\eta \leq \frac{1}{\tilde{L}}$ (where $L$ is the Lipschitz constant of the gradient of the log-likelihood) $\mu\eta\leq 1$:
$$
\| \theta_0 - \theta^* \|_2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2+\eta \epsilon_g
$$
Then, after one step of projected gradient ascent on $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$, yielding $\theta_1$, we have:
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2 + \eta \epsilon_g
$$
\end{proposition}
\begin{proof}
Let $\theta_1$ be the result of one step of projected gradient ascent on $\tilde{\mathcal{L}}$ from $\theta_0$ with step size $\eta$:
$$
\theta_1 = \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0))
$$
where $\mathcal{D}$ is a convex set containing $\theta^*$.
By the triangle inequality, we have:
\begin{align*}
\| \theta_1 - \theta^* \|_2 &= \| \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0)) - \theta^* \|_2 \\
&\leq \| \theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0) - \theta^* \|_2 \\
&= \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* + \eta (\nabla \tilde{\mathcal{L}}(\theta_0) - \nabla \mathcal{L}(\theta_0)) \|_2 \\
&\leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 + \eta \| \nabla \tilde{\mathcal{L}}(\theta_0) - \nabla \mathcal{L}(\theta_0) \|_2 \\
&\leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 + \eta \epsilon_g \quad \text{(by the gradient difference bound)}
\end{align*}
Now, let's consider the term $\| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2$. This represents the distance to $\theta^*$ after taking one step of gradient ascent on the true log-likelihood $\mathcal{L}$. By Proposition \ref{prop:one-step-improvement-on-true-log-likelihood}, if $\| \theta_0 - \theta^* \|_2 \leq \frac{\eta(2\mu - \eta L)}{2}$ and $\eta \leq \frac{1}{L}$, we have:
$$
\| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2
$$
Therefore,
\begin{align*}
\| \theta_1 - \theta^* \|_2 &\leq (1-\eta\mu)\|\theta_0-\theta^*\|_2+\eta\epsilon_g
\end{align*}
\end{proof}
\begin{itemize}
\item \textbf{Not globally concave:} The GP log-likelihood with the RBF kernel is generally \textbf{not globally concave} with respect to the lengthscale and variance, even for large values of $l$.
\item \textbf{Locally concave in regions:} However, as you suggest, it \textbf{can be locally concave in certain regions of the hyperparameter space, including regions with large lengthscales}. This means that if your optimizer starts in such a region, it might converge to a local optimum, but not necessarily the global optimum. The reason why it is locally concave is due to the fact that the kernel matrix is close to a rank-1 matrix. When this happens, the log determinant term can be approximated as $\log \det(\sigma^2 \mathbf{11}^T) = n\log(\sigma^2) + C$, where $C$ is some constant, and the quadratic term can be approximated as $\mathbf{y}^T(\sigma^2 \mathbf{11}^T)^{-1}\mathbf{y} = c/\sigma^2$, where $c$ is some constant. When $l$ is very large, the log likelihood is approximately $-1/2*(c/\sigma^2) - 1/2*(n*\log(\sigma^2)) = -1/2*(c/\sigma^2 + n\log(\sigma^2))$. The second derivative of $-1/2*(c/\sigma^2 + n\log(\sigma^2))$ with respect to $\sigma^2$ is $(c - n\sigma^4)/(2\sigma^6)$, which can be positive, but it can also be negative when $\sigma^4 > c/n$. This implies that when $\sigma^4 > c/n$, the log likelihood is locally concave. But $\sigma^4 < c/n$ means the log likelihood is convex.
\item \textbf{Multiple local optima:} It's common to have multiple local optima in the GP log-likelihood landscape, especially when using flexible kernels like RBF. This is a well-known issue in GP hyperparameter optimization.
\end{itemize}
\begin{theorem}[Closeness of SKI MLE to True MLE]\label{thm:mle-closeness-gradient-ascent}
Let $\theta^*$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient in $\mathcal{D}$. Assume that the gradient difference between the true log-likelihood and the SKI log-likelihood is bounded by
$$
\| \nabla \tilde{\mathcal{L}}(\theta) - \nabla \mathcal{L}(\theta) \|_2 \leq \epsilon_g := \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\frac{1}{\sqrt{p}}\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right]
$$
for all $\theta$ in a neighborhood of $\theta^*$, where $\gamma_{n,m,L}$, $\gamma'_{n,m,L,i}$ are defined as in Proposition \ref{prop:spectral-norm} and Proposition \ref{lemma:gradient_spectral_norm_bound}, respectively, and $C$ is a bound on $C_n$ in this neighborhood. Let $\tilde{\theta}$ be the SKI MLE obtained by running projected gradient ascent on $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$ from initialization $\theta_0$ with a fixed step size $\eta$.
If the step size $\eta$ satisfies $0 < \eta \leq \frac{1}{L}$ and $\eta \mu < 1$, then, after $t$ steps of projected gradient ascent, the iterate $\theta_t$ satisfies
$$
\| \theta_t - \theta^* \|_2 \leq (1 - \eta\mu)^t \| \theta_0 - \theta^* \|_2 + \frac{\epsilon_g}{\eta\mu}.
$$
\end{theorem}
\begin{proof}
Consider that starting from $\theta_t$ and applying one step of gradient ascent, by Proposition \ref{prop:mle-closeness-one-step}
\begin{align*}
\Vert \theta_{t+1}-\theta^*\Vert_2 &\leq (1-\eta\mu)\Vert \theta_t-\theta^*\Vert_2+\eta \epsilon_g
\end{align*}
and iterating we have
\begin{align*}
\Vert \theta_t-\theta^*\Vert_2&\leq (1-\eta\mu)^t\Vert \theta_0-\theta^*\Vert_2+\sum_{s=0}^{t-1} (1-\eta\mu)^s \eta \epsilon_g\\
&\leq (1-\eta\mu)^t\Vert \theta_0-\theta^*\Vert_2+\frac{\eta\epsilon_G}{\eta\mu}\text{ geometric series}
\end{align*}
\end{proof}
% $$
% \frac{\partial^2 L}{\partial \ell^2} = \frac{1}{2}\alpha^T K'' \alpha - \alpha^T K' K_y^{-1} K' \alpha + \frac{1}{2} \text{tr}\left( K_y^{-1} K' K_y^{-1} K' - K_y^{-1} K''\right)
% $$
\begin{lemma}[Strong Concavity of the Log-Likelihood for Signal Variance with Fixed Lengthscale]
Let $\mathbf{X}_D = \{\mathbf{x}_1, \ldots, \mathbf{x}_N\}$ be a set of $N$ distinct training input points in $\mathbb{R}^d$. Consider a Gaussian Process with an RBF kernel (or an RQ kernel with fixed shape parameter $\alpha$) where the lengthscale $\ell$ is fixed at its true value $\ell^*$. The negative log-likelihood function of the Gaussian Process, as a function of the signal variance $\sigma^2$, is locally strongly convex in a neighborhood of the true signal variance $\sigma^{*2}$.
\end{lemma}
\begin{proof}
\textbf{(Fixed Lengthscale):}
\begin{enumerate}
\item \textbf{Kernel and Parameters:}
Assume an RBF kernel (the argument for RQ is analogous):
$$
k(\mathbf{x}_i, \mathbf{x}_j; \sigma^2, \ell) = \sigma^2 \exp\left(-\frac{||\mathbf{x}_i - \mathbf{x}_j||^2}{2\ell^2}\right)
$$
Fix the lengthscale at its true value, $\ell = \ell^*$.
The only parameter we are varying is $\theta = \sigma^2$.
\item \textbf{Log-Likelihood:}
The log-likelihood function for a GP with fixed hyperparameters is:
$$
\mathcal{L}(\sigma^2) = -\frac{N}{2} \log(2\pi) - \frac{1}{2} \log |K| - \frac{1}{2} \mathbf{y}^T K^{-1} \mathbf{y}
$$
where $K = K(\mathbf{X}_D, \mathbf{X}_D; \sigma^2, \ell^*)$ is the kernel matrix.
\item \textbf{Simplifying the Kernel Matrix:}
Since $\ell^*$ is fixed, we can write the kernel matrix as:
$$
K = \sigma^2 K_*
$$
where $K_* = K(\mathbf{X}_D, \mathbf{X}_D; 1, \ell^*)$ is a constant matrix that does not depend on $\sigma^2$.
\item \textbf{Log-Likelihood in Terms of $\sigma^2$:}
Substituting $K = \sigma^2 K_*$ into the log-likelihood, we get:
$$
\mathcal{L}(\sigma^2) = -\frac{N}{2} \log(2\pi) - \frac{1}{2} \log |\sigma^2 K_*| - \frac{1}{2} \mathbf{y}^T (\sigma^2 K_*)^{-1} \mathbf{y}
$$
$$
\mathcal{L}(\sigma^2) = -\frac{N}{2} \log(2\pi) - \frac{N}{2} \log(\sigma^2) - \frac{1}{2} \log |K_*| - \frac{1}{2\sigma^2} \mathbf{y}^T K_*^{-1} \mathbf{y}
$$
\item \textbf{Derivatives:}
Now, let's take the first and second derivatives of the log-likelihood with respect to $\sigma^2$:
\begin{itemize}
\item First derivative:
$$
\frac{d\mathcal{L}}{d\sigma^2} = -\frac{N}{2\sigma^2} + \frac{1}{2(\sigma^2)^2} \mathbf{y}^T K_*^{-1} \mathbf{y}
$$
\item Second derivative:
$$
\frac{d^2\mathcal{L}}{d(\sigma^2)^2} = \frac{N}{2(\sigma^2)^2} - \frac{1}{(\sigma^2)^3} \mathbf{y}^T K_*^{-1} \mathbf{y}
$$
\end{itemize}
\item \textbf{Strong Concavity of the Negative Log-Likelihood:}
To show strong concavity of the *negative* log-likelihood, we need to show that the second derivative of the log-likelihood is *negative*:
$$
\frac{d^2\mathcal{L}}{d(\sigma^2)^2} = \frac{N}{2(\sigma^2)^2} - \frac{1}{(\sigma^2)^3} \mathbf{y}^T K_*^{-1} \mathbf{y} < 0
$$
Rearranging, we get:
$$
\frac{N}{2} < \frac{\mathbf{y}^T K_*^{-1} \mathbf{y}}{\sigma^2}
$$
Since $K_*$ is positive definite, $\mathbf{y}^T K_*^{-1} \mathbf{y} > 0$. Thus, for a fixed dataset, we can always find a neighborhood of $\sigma^2$ values such that this inequality holds. This means that the negative log-likelihood is locally strongly convex/concave.
\end{enumerate}
\end{proof}
In this section, we address how SKI affects Gaussian Processes Applications. In Section \ref{sec:kernel-hyperparameter-estimation} we address how using the SKI kernel and log-likelihood affect hyperparameter estimation, showing that gradient ascent on the SKI log-likelihood approaches a ball around a stationary point of the true log-likelihood. In section \ref{sec:posterior-inference} we describe how using SKI affects the accuracy of posterior inference.
\subsection{Kernel Hyperparameter Estimation}\label{sec:kernel-hyperparameter-estimation}
Here we show that, for a $\mu$-smooth log-likelihood, an iterate of gradient ascent on the SKI log-likelihood approaches a neighborhood of a stationary point of the true log-likelihood at an $O\left(\frac{1}{K}\right)$ rate, with the neighborhood size determined by the SKI score function's error. To show this, we leverage a recent result for non-convex inexact gradient ascent \cite{stonyakin2023stopping}, which requires an upper bound on the SKI score function's error. This requires bounding the spectral norm error of the SKI Gram matrix's partial derivatives. In order to obtain this, we note that for many SPD kernels, under weak assumptions, the partial derivatives are \textit{also} SPD kernels, and thus we can reuse the previous results directly on the partial derivatives.
Note that \cite{stonyakin2023stopping} does not actually imply \textit{convergence} to a neighborhood of a critical point, only that at least one iterate will approach it. Given the challenges of non-concave optimization and the fact that we leverage a fairly recent result, we leave stronger results to future work.
%In order to obtain this, we need to bound the errors of the partial derivatives of the SKI gram matrix. To do so, we note that in many cases, the partial derivatives of an SPD kernel are themselves SPD kernels. This allows us to express the gram matrix's gradient error as a different gram matrix error and thus apply our previous results.
Let $\mathcal{D}\subseteq \Theta$ be a \textit{compact} subset that we wish to optimize over. In the most precise setting we would analyze projected gradient ascent, but for simplicity we analyze gradient ascent. Let let $\tilde{k}_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be the SKI approximation of $k_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ using $m$ inducing points and interpolation degree $L-1$. We are interested in the convergence properties of \textit{inexact gradient ascent} using the SKI log-likelihood, e.g.
\begin{align*}
\boldsymbol{\theta}_{k+1}&=\boldsymbol{\theta}_k+\eta \nabla \tilde{\mathcal{L}}(\boldsymbol{\theta}_k),
\end{align*}
where $\eta\in \mathbb{R}$ is the learning rate and $\nabla \tilde{\mathcal{L}}(\boldsymbol{\theta}_k)$ is the SKI score function (gradient of its log-likelihood). We assume: 1) a $\mu$-smooth log-likelihood. If we optimize on a bounded domain, then for infinitely differentiable kernels (e.g. RBF) this will immediately hold. 3) That the kernel's partial derivatives are themselves SPD kernels (this can be easily shown for the RBF kernel's lengthscale by noting that the product of SPD kernels are themselves SPD kernels).
%We then mention an equivalence between strong concavity and the Hessian having an upper bound on its smallest eigenvalue.
% 2) That the kernel is continuously differentiable. 3) That the response vectors squared $l^2$ norm grows linearly with the sample size.
\begin{assumption}[$\mu$-smooth-log-likelihood]\label{assumption:mu-smoothness}
The true log-likelihood is $\mu$-smooth over $\mathcal{D}$. That is, for all $\boldsymbol{\theta},\boldsymbol{\theta}'\in \mathcal{D}$,
\begin{align*}
\Vert \nabla \mathcal{L}(\boldsymbol{\theta})-\nabla \mathcal{L}(\boldsymbol{\theta}')\Vert &\leq \mu \Vert \boldsymbol{\theta}-\boldsymbol{\theta}'\Vert
\end{align*}
\end{assumption}
\begin{assumption}
(Kernel Smoothness) $k_\theta(x,x')$ is $C^1$ in $\boldsymbol{\theta}$ over $\mathcal{D}$. That is, for each $l \in \{1, ..., p\}$, $k'_{\theta_l}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_l}$ exists and is continuous for $\boldsymbol{\theta}\in \mathcal{D}$.
\end{assumption}
\begin{assumption}
(SPD Kernel Partials) For each $l \in \{1, ..., p\}$, the partial derivative of $k_{\theta}$ with respect to a hyperparameter $\theta_l\in \mathbb{R}$, denoted as $k'_{\theta_l}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_l}$, is also a valid SPD kernel.
\end{assumption}
% \begin{assumption}
% (Linear Squared $l^2$ Response) $\Vert \textbf{y}\Vert_2^2=O(n)$.
% \end{assumption}
We next state several results leading up to our bound on the SKI score function's error. Here we argue that we can apply the same elementwise error we derived previously to the SKI partial derivatives.
\begin{restatable}{lemma}{skikernelderivativeerrorkernel}[Bound on Derivative of SKI Kernel Error using Kernel Property of Derivative]
\label{lemma:ski_kernel_derivative_error_kernel}
Let $\tilde{k}'_{\theta_l}(x,x')$ be the SKI approximation of $k'_{\theta_l}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$. Then, for all $x, x' \in \mathcal{X}$ and all $\boldsymbol{\theta} \in \Theta$, the following inequality holds:
\begin{align*}
\left\vert \frac{\partial k_{\theta}(x,x')}{\partial \theta_l}-\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_l}\right\vert &= \left\vert k'_{\theta_l}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \\
&\leq\delta_{m,L}'+\sqrt{L}c^d\delta_{m,L}'\\
&=O\left(\frac{c^{2d}}{m^{3/d}}\right)
\end{align*}
where $\delta_{m,L}'$ is an upper bound on the error of the SKI approximation of the kernel $k'_{\theta_l}(x,x')$ with $m$ inducing points and interpolation degree $L-1$, as defined in Lemma \ref{lemma:ski-kernel-elementwise-error}.
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proofski_kernel_derivative_error_kernelz}
\end{proof}
We then use the elementwise bound to bound the spectral norm of the SKI gram matrix's partial derivative error. This again leverages Proposition \ref{prop:spectral-norm}, noting that these partial derivatives of the Gram matrices are themselves Gram matrices.
\begin{restatable}{lemma}{partialgradientspectralnormbound}[Partial Derivative Gram Matrix Difference Bound]
\label{lemma:partial_gradient_spectral_norm_bound}
For any $l \in \{1, \dots, p\}$,
\begin{align*}
\left\| \frac{\partial \mathbf{K}}{\partial \theta_l} - \frac{\partial \tilde{\mathbf{K}}}{\partial \theta_l} \right\|_2 &\leq \gamma'_{n,m,L,l} \\
&= O\left(\frac{nc^{2d}}{m^{3/d}}\right)
\end{align*}
where $\gamma'_{n,m,L,l}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta_l}$ and its SKI approximation $\tilde{k}'_{\theta_l}$ (analogous to Proposition \ref{prop:spectral-norm}, but for the kernel $k'_{\theta_l}$).
\end{restatable}
\begin{proof}
See Section \ref{section:proof_partial_gradient_spectral_norm_bound}.
\end{proof}
% \begin{restatable}{lemma}{gradientspectralnormbound}[Bound on Spectral Norm of Gradient Difference]
% \label{lemma:gradient_spectral_norm_bound}
% Assume that $k'_{\theta,i}(x, x')$, is a valid SPD kernel. Let $\tilde{k}'_{\theta,i}(x,x')$ be the SKI approximation of $k'_{\theta,i}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$. Then, the spectral norm of the difference between the gradient of the true kernel matrix and the gradient of the SKI kernel matrix is bounded by:
% \begin{align*}
% \| \nabla_\theta K - \nabla_\theta \tilde{K} \|_2 &\leq \sqrt{p} \max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}\\
% &=O\left(\sqrt{p}nc^{2d}h^3\right)\text{, convolutional cubic interpolation,}
% \end{align*}
% where $\nabla_\theta K$ and $\nabla_\theta \tilde{K}$ are represented as $p \times n^2$ matrices using the vec-notation (denominator layout), $p$ is the number of hyperparameters, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ (analogous to Proposition \ref{prop:spectral-norm}, but for the kernel $k'_{\theta,i}$).
% \end{restatable}
% \begin{proof}
% See Section \ref{sec:proof_gradient_spectral_norm_bound}
% \end{proof}
We now bound the SKI score function. The key insight to the proof is that the partial derivatives of the difference between regularized gram matrix inverses is in fact a difference between two quadratic forms. We can then use standard techniques \citep{horn2012matrix} for bounding the difference between quadratic forms to obtain our result. The result says that, aside from the response vector's norm, the error grows quadratically in the sample size, at a square root rate in the number of hyperparameters and exponentially in the dimensionality. It further decays at an $m^{\frac{3}{d}}$ rate in the number of inducing points. Noting that to maintain linear time, $m$ should grow at an $n^{d/3}$ rate, we have that aside from the response vector, the error in fact grows linearly with the sample size when choosing the number of inducing points based on Theorem \ref{thm:inducing-points-count-alt}.
%by an expression that is \textit{quadratic} in the sample size. The reason for the quadratic dependency is that the difference in gradients of the quadratic forms in the log-likelihoods can itself be expressed as a gradient of a quadratic form involving the response variables. The response variables contribute a linear dependence and the gradient of the difference between the regularized inverses multiplicatively contribute a linear dependence: combined this is quadratic.
%\amnote{In an attempt to improve the bound, I got something that doesn't decrease with the number of inducing points, which is bad.}
\begin{restatable}{lemma}{scorefunctionbound}[Score Function Bound]\label{lemma:score-function-bound}
Let $\mathcal{L}(\boldsymbol{\theta})$ be the true log-likelihood and $\tilde{\mathcal{L}}(\boldsymbol{\theta})$ be the SKI approximation of the log-likelihood at $\boldsymbol{\theta}$. Let $\nabla \mathcal{L}(\boldsymbol{\theta})$ and $\nabla \tilde{\mathcal{L}}(\boldsymbol{\theta})$ denote their respective gradients with respect to $\boldsymbol{\theta}$. Then, for any $\boldsymbol{\theta}\in \mathcal{D}$,
\begin{align*}
&\| \nabla \mathcal{L}(\boldsymbol{\theta}) - \nabla \tilde{\mathcal{L}}(\boldsymbol{\theta}) \|_2 \\
&\leq \frac{1}{2\sigma^4}\Vert \textbf{y}\Vert\sqrt{p}\max_{1\leq l\leq p} \left( \gamma'_{n,m,L,l}+Cn\gamma_{n,m,L}\right.\\
&\qquad\left.+\gamma_{n,m,L}\gamma'_{n,m,L,l} \right)+\frac{\gamma_{n,m,L}}{2\sigma^4}\\
&=\Vert \textbf{y}\Vert_2 O\left(\frac{\sqrt{p}n^2c^{4d}}{m^{3/d}}\right)\\
&\equiv \epsilon_G
\end{align*}
where $C$ is a constants depending on the upper bound of the derivatives of the kernel function over $\mathcal{D}$.
\end{restatable}
% \begin{restatable}{lemma}{scorefunctionbound}[Score Function Bound]\label{lemma:score-function-bound}
% Let $\mathcal{L}(\theta)$ be the true log-likelihood and $\tilde{\mathcal{L}}(\theta)$ be the SKI approximation of the log-likelihood at $\theta$. Let $\nabla \mathcal{L}(\theta)$ and $\nabla \tilde{\mathcal{L}}(\theta)$ denote their respective gradients with respect to $\theta$. Then, for any $\theta\in \mathcal{D}$,
% \begin{align*}
% \| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 &\leq \frac{1}{\sigma^4}\left[\Vert y\Vert_2^2\sqrt{p}\max_{1\leq l\leq p}\left( \gamma'_{n,m,L,l}+Cn\gamma_{n,m,L}+\frac{\gamma_{n,m,L}}{\sigma^4} \gamma'_{n,m,L,l}\right)+\frac{1}{2}\gamma_{n,m,L}\right]\\
% &=O(\sqrt{p}n^3c^{2d}h^3)\\
% &\equiv \epsilon_G
% \end{align*}
% where $\gamma_{n,m,L}$ bounds the elementwise difference between $\mathbf{K}$ and $\tilde{\mathbf{K}}$, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ and $C_n$ is a constant depending on the upper bound of the derivatives of the kernel function over $\mathcal{D}$ and the sample size.
% \end{restatable}
\begin{proof}
See Section \ref{sec:proof-score-function-bound}.
\end{proof}
We apply \cite{stonyakin2023stopping} below: the result is the same as in their paper (and assumes $\mu$-smoothness as we did on $\mathcal{L}$), but using gradient ascent instead of descent and using the score function error above. It says that at an $O\left(\frac{1}{K}\right)$ rate, at least one iterate of gradient ascent has its squared gradient norm approach a neighborhood proportional to the squared SKI score function's spectral norm error.
\begin{theorem} \citep{stonyakin2023stopping}
For inexact gradient ascent on $\mathcal{L}$ with additively inexact gradients satisfying $\|\nabla \mathcal{L}(\boldsymbol{\theta}) - \nabla \tilde{\mathcal{L}}(\boldsymbol{\theta})\| \leq \epsilon_g$, we have:
\begin{equation}
\max_{k=0,...,N-1} \|\nabla \mathcal{L}(\theta_k)\|^2 \leq \frac{2\mu(\mathcal{L}^* - \mathcal{L}(\boldsymbol{\theta}_0))}{K} + \frac{\epsilon_g^2}{2\mu}
\end{equation}
where $\mathcal{L}^*$ is the value at a stationary point, $\mathcal{L}(\boldsymbol{\theta}_0)$ is the initial, function value, $K$ is the number of iterations and $\epsilon_g$ is the gradient error bound in the previous Lemma.
\end{theorem}
\subsection{Posterior Inference}\label{sec:posterior-inference}
Finally, we treat posterior inference. As the current hyperparameter optimization results only say that \textit{some} iterate approaches a stationary point, we will focus on the error when the SKI and true kernel hyperparameter match. We first add an assumption
\begin{assumption}
(Bounded Kernel) Assume that the true kernel satisfies the condition that $|k(\mathbf{x}, \mathbf{x}')| \leq M$ for all $\mathbf{x}, \mathbf{x}'\in \mathcal{X}$.
\end{assumption}
% Consider on a compact domain,
% \begin{align*}
% \vert k_\theta(x,x')-k_{\theta'}(x,x')\vert &\leq \rho \Vert \theta-\theta'\Vert
% \end{align*}
% which implies that
% \begin{align*}
% \Vert \textbf{K}^{(\theta)}-\textbf{K}^{(\theta')}\Vert_2&\leq n\rho \Vert \theta-\theta'\Vert
% \end{align*}
% and we can then
Now we bound the spectral error for the SKI mean function evaluated at a set of test points. The proof follows a standard strategy commonly used for approximate kernel ridge regression. See \cite{bach2013sharp,musco2017recursive} for examples. The result says that the $l^2$ error (aside from the response vector) grows exponentially in the dimensionality, super-linearly but sub-quadratically in the training sample size and at worst linearly in the test sample size. It decays at an $m^{\frac{3}{d}}$ rate in the number of inducing points. Similarly to for the score function error, if we follow Theorem \ref{thm:inducing-points-count-alt} for selecting the number of inducing points, the error in fact grows \textit{sublinearly} with the training sample size.
\begin{restatable}{lemma}{meaninference}\label{lemma:mean-inference} (SKI Posterior Mean Error)
Let $\boldsymbol{\mu}(\cdot)$ be the GP posterior mean at a set of test points $\cdot\in \mathbb{R}^{T\times d}$ and $\tilde{\boldsymbol{\mu}}(\cdot)$ be the SKI posterior mean at those points. Then the SKI posterior mean $l^2$ error is bounded by:
{\footnotesize
\begin{align*}
&\Vert \tilde{\boldsymbol{\mu}}(\cdot)- \boldsymbol{\mu}(\cdot)\Vert_2\\
&\leq\left(\frac{\max(\gamma_{T,m,L},\gamma_{n,m,L})}{\sigma^2}+\frac{\sqrt{Tn}Mc^{2d}}{\sigma^4}\gamma_{n,m,L}\right)\Vert \textbf{y}\Vert_2\\
&=\Vert \textbf{y}\Vert_2O\left(c^{2d}\frac{\max(T,n)+\sqrt{Tn}n}{m^{3/d}}\right)
%\left(\frac{\max(\gamma_{T,m,L},\gamma_{n,m,L})}{\sigma^2}+\frac{\Vert \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \Vert_2}{\sigma^4}\gamma_{n,m,L}\right)\Vert \textbf{y}\Vert_2
\end{align*}
}
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proof-mean-inference}.
\end{proof}
We now derive the spectral error bound for the test SKI covariance matrix. The proof involves noticing that a key term is a difference between two quadratic forms, and using standard techniques for bounding such a difference. The result shows that the error grows at worst super-linearly but subquadratically in the number of test points, quadratically in the training sample size and exponentially in the dimension. Interestingly, due to the use of standard techniques for bounding the difference between quadratic forms, the error is only guaranteed to decay with the number of inducing points at an $m^{3/d-1}$ rate, so that it is only guaranteed to decay at all if $d<3$. If we select the number of inducing points to be proportional to $n^{d/3}$, then the error grows at rate $n^{1+d/3}$ for $d<3$. An interesting question is whether alternate techniques can improve the result for higher dimensional settings e.g. $d\geq 3$.
\begin{restatable}{lemma}{skiposteriorcovarianceerror}[SKI Posterior Covariance Error]\label{lemma:ski-posterior-covariance-error}
Let $\boldsymbol{\Sigma}(\cdot)$ be the GP posterior covariance matrix at a set of test points $\cdot\in \mathbb{R}^{T\times d}$ and $\tilde{\boldsymbol{\Sigma}}(\cdot)$ be its SKI approximation. Then
\begin{align*}
&\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2\\ &\leq \gamma_{T,m,L} + \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L})\\
&\quad+ \frac{\gamma_{n,m,L}}{\sigma^4}Tn m c^{2d} M^2 \\
&\quad+ \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}).\\
&=O\left(\frac{Tn^2mc^{4d}+\sqrt{Tn}mc^{4d}\max(T,n)}{m^{3/d}}\right).
\end{align*}
where $\gamma_{T,m,L}$ is defined as in Proposition \ref{prop:spectral-norm}.
\end{restatable}
\begin{proof}
See Appendix \ref{sec:proof-ski-posterior-covariance-error}
\end{proof}
In this paper, we provided the first rigorous theoretical analysis for structured kernel interpolation. A key practical takeaway is that to control the SKI Gram matrix's spectral norm error, the number of inducing points should grow as $n^{d/3}$. Additionally, we showed the spectral norm error of the SKI gram and cross-kernel matrices, and how this impacts achieving a specific error in linear time. We then analyzed kernel hyperparameter estimation, showing that gradient ascent has an iterate approach a ball around a stationary point, where the ball's radius depends on the spectral error of the SKI score function. We concluded with analysis of the error of the SKI posterior mean and variance.
This work could be extended by analyzing the error of SKI with other interpolation schemes such as Lagrange interpolation \citep{lagrange1795lecons}, using potentially higher order polynomials. This would allow us to not only analyze how to vary $m$ for fixed $L=4$, but how to vary them jointly. Additionally, one could analyze the error of SKI in more complex settings, such as when the inducing points are not placed on a regular grid \citep{snelson2006sparse} or for non-stationary kernel functions, in which case the computational complexity would no longer be $O(n+m\log m)$. Further, we analyze the optimization properties under gradient ascent: it would be interesting to analyze it under stochastic gradient ascent, analogous to \cite{lin2024stochastic}, but now using inexact noisy SKI gradients. Finally, one could analyze the methods for extending SKI to higher dimensions \cite{kapoor2021skiing,yadav2022kernel} and for faster SKI inference \cite{yadav2021faster}.
This paper heavily used LLMs: particularly reasoning models. The paper idea and early error analysis of the kernel error and the spectral norm error came from the authors. Beyond that LLMs were used to outline the statements to be made, turn initial rough descriptions into more formal language, and attempt to prove the results. In general, LLM attempts at proofs were \textit{wrong}, but could drive insights into a working proof strategy. We also used the versions with internet access to help bring up relevant papers. While the LLMs sometimes hallucinated papers, the rate at which it did so was quite low and the usefulness of the papers it found was often very high.
This work contributes to a deeper theoretical understanding Structured Kernel Interpolation (SKI) \citep{wilson2015kernel} for Gaussian Processes (GPs). By establishing error bounds and analyzing the impact of SKI on hyperparameter estimation and posterior inference, this research can lead to more confident use of approximate Gaussian Processes. These models have broad applications in various domains, including those mentioned in the introduction as well as robotics \citep{deisenroth2015gaussian}, environmental modeling \citep{desai2023deep}, and healthcar \citep{alaa2017bayesian}. Improved Gaussian Process models can enhance prediction accuracy and decision-making, potentially leading to advancements in robotics, more accurate environmental predictions, and better healthcare outcomes. It is important to acknowledge that the application of Gaussian Process models also carries potential risks. For instance, in healthcare, inaccurate predictions or biased models can lead to misdiagnosis or inappropriate treatment \citep{morley2020ethics}. Therefore, understanding potential sources of error when using approximations can be crucial to understanding how reliable we can expect them to be.
\begin{lemma}\label{lemma:switch-sum-product}
Given a function $f: \mathbb{R}^d \rightarrow \mathbb{R}$ of the form $f(x_1, x_2, ..., x_d) = \prod_{j=1}^d f_j(x_j)$, where each $f_j: \mathbb{R} \rightarrow \mathbb{R}$. Let $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$ be a fixed d-dimensional grid, where each $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is a finite set of $n_j$ grid points along the j-th dimension for $j = 1, 2, ..., d$. Then the following equality holds:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d f_j(p_{k_j}^{(j)}) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\end{lemma}
\begin{proof}
\textbf{By Induction on d (the number of dimensions):}
\textbf{Base Case (d = 1):}
When $d=1$, the statement becomes:
$$
\sum_{k_1=1}^{n_1} f_1(p_{k_1}^{(1)}) = \sum_{k_1=1}^{n_1} f_1(p_{k_1}^{(1)})
$$
This is trivially true.
\textbf{Inductive Hypothesis:}
Assume the statement holds for $d = m$, i.e.,
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \prod_{j=1}^m f_j(p_{k_j}^{(j)}) = \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\textbf{Inductive Step:}
We need to show that the statement holds for $d = m+1$. Consider the left-hand side for $d = m+1$:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_{m+1}=1}^{n_{m+1}} \prod_{j=1}^{m+1} f_j(p_{k_j}^{(j)})
$$
We can rewrite this as:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \left( \sum_{k_{m+1}=1}^{n_{m+1}} \left( \prod_{j=1}^m f_j(p_{k_j}^{(j)}) \right) f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right)
$$
Notice that the inner sum (over $k_{m+1}$) does not depend on $k_1, k_2, ..., k_m$. Thus, for any fixed values of $k_1, k_2, ..., k_m$, we can treat $\prod_{j=1}^m f_j(p_{k_j}^{(j)})$ as a constant. Let $C(k_1, ..., k_m) = \prod_{j=1}^m f_j(p_{k_j}^{(j)})$. Then we have:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \left( C(k_1, ..., k_m) \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right)
$$
Now, the inner sum $\sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)})$ is a constant with respect to $k_1, ..., k_m$. Let's call this constant $S_{m+1}$. So we have:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} C(k_1, ..., k_m) S_{m+1} = S_{m+1} \sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \prod_{j=1}^m f_j(p_{k_j}^{(j)})
$$
By the inductive hypothesis, we can replace the nested sums with a product:
$$
S_{m+1} \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right) = \left( \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right) \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
Rearranging the terms, we get:
$$
\prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right) \left( \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right) = \prod_{j=1}^{m+1} \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
This is the right-hand side of the statement for $d = m+1$. Thus, the statement holds for $d = m+1$.
\textbf{Conclusion:}
By induction, the statement holds for all $d \geq 1$. Therefore,
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d f_j(p_{k_j}^{(j)}) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\end{proof}
\begin{claim}\label{claim:convex-combo-eigenvalues}
Given a convex combination \(\mathbf{C} = \alpha \mathbf{A} + (1-\alpha) \mathbf{B}\), where \(\alpha \in [0,1]\), and \(\mathbf{A}\) and \(\mathbf{B}\) are symmetric matrices, the eigenvalues of \(\mathbf{C}\) lie in the interval \(\left[\min \left(\lambda_n(\mathbf{A}), \lambda_n(\mathbf{B})\right), \max \left(\lambda_1(\mathbf{A}), \lambda_1(\mathbf{B})\right)\right]\).
% Given a convex combination $C=\alpha A+(1-\alpha)B,\alpha \in [0,1]$, the eigenvalues of $C$ lie in the interval $[\min(\lambda(A)_n, \lambda(B)_n), \max(\lambda(A)_1,\lambda(B)_1)]$
\end{claim}
\begin{proof}
First, recall that for a symmetric matrix \(\mathbf{A}\), the Rayleigh quotient \(R(\mathbf{A}, \mathbf{x}) = \frac{\mathbf{x}^{\top} \mathbf{A} \mathbf{x}}{\mathbf{x}^{\top} \mathbf{x}}\) is bounded by the smallest and largest eigenvalues of \(\mathbf{A}\):
\[
\lambda_n(\mathbf{A}) \leq R(\mathbf{A}, \mathbf{x}) \leq \lambda_1(\mathbf{A})
\]
Consider the Rayleigh quotient for the matrix \(\mathbf{C}\):
\[
R(\mathbf{C}, \mathbf{x}) = \frac{\mathbf{x}^{\top} (\alpha \mathbf{A} + (1-\alpha) \mathbf{B}) \mathbf{x}}{\mathbf{x}^{\top} \mathbf{x}} = \alpha R(\mathbf{A}, \mathbf{x}) + (1-\alpha) R(\mathbf{B}, \mathbf{x})
\]
Since \(R(\mathbf{A}, \mathbf{x})\) and \(R(\mathbf{B}, \mathbf{x})\) are bounded by their respective eigenvalues, we have:
\[
R(\mathbf{C}, \mathbf{x}) \leq \alpha \lambda_1(\mathbf{A}) + (1-\alpha) \lambda_1(\mathbf{B})
\]
which implies:
\[
R(\mathbf{C}, x) \leq \max(\lambda_1(\textbf{A}), \lambda_1(\textbf{B}))
\]
Similarly,
\[
R(\textbf{C}, \textbf{x}) \geq \min(\lambda_n(\textbf{A}), \lambda_n(\textbf{B}))
\]
Thus, the eigenvalues of \(\textbf{C} = \alpha \textbf{A} + (1-\alpha)\textbf{B}\) are bounded by:
\[
\min(\lambda_n(\textbf{A}), \lambda_n(\textbf{B})) \leq \lambda(\textbf{C}) \leq \max(\lambda_1(\textbf{A}), \lambda_1(\textbf{B}))
\]
\end{proof}
\subsection{Proofs Related to Ski Kernel Error Bounds}\label{subsec:proofs-ski-kernel-error}
\input{body/appendices/important-quantities/polynomial-interpolation}
% \subsubsection{Elementwise}\label{subsubsec:proofs-elementwise}
\input{body/appendices/important-quantities/error-bounds-ski-kernel}
% \subsubsection{Spectral Norm}\label{subsubsec:spectral-norm}
\input{body/appendices/important-quantities/spectral-norm-error}
\subsection{Proofs Related to Linear Time Analysis}
\input{body/appendices/important-quantities/linear-time}
\subsection{Proofs Related to Hyperparameter Estimation}\label{proofs-hyperparameter-estimation}
\input{body/appendices/gp-applications/hyperparameter-estimation}
\subsection{Proofs Related to Posterior Inference}\label{proofs-posterior-inference}
\input{body/appendices/gp-applications/posterior-inference}
\subsubsection{Proof of Lemma \ref{lemma:ski_kernel_derivative_error_kernel}}\label{sec:proofski_kernel_derivative_error_kernelz}
\skikernelderivativeerrorkernel*
\begin{proof}
By assumption, $k'_{\theta_i}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$ is a valid SPD kernel. The SKI approximation of $k'_{\theta_i}(x, x')$ using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}(x, x')$ is given by $\tilde{k}'_{\theta}(x, x')$. For the kernel $k'_{\theta_i}(x, x')$, we have:
\begin{align*}
\left\vert k'_{\theta_i}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \leq \delta_{m,L}',
\end{align*}
where $\delta_{m,L}'$ is the upper bound on the error of the SKI approximation of $k'_{\theta_i}(x, x')$ as defined in Lemma \ref{lemma:ski-kernel-elementwise-error}.
Now, we need to show that $\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta}(x, x')$. Recall that the SKI approximation $\tilde{k}_{\theta}(x, x')$ is a linear combination of kernel evaluations at inducing points, with weights that depend on $x$ and $x'$:
\begin{align*}
\tilde{k}_{\theta}(x, x') = \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k_{\theta}(u_j, u_l)
\end{align*}
where $w_{jl}(x, x')$ are the interpolation weights. Taking the partial derivative with respect to $\theta_i$, we get:
\begin{align*}
\frac{\partial \tilde{k}_{\theta}(x, x')}{\partial \theta_i} &= \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') \frac{\partial k_{\theta}(u_j, u_l)}{\partial \theta_i} \\
&= \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k'_{\theta_i}(u_j, u_l).
\end{align*}
This is precisely the SKI approximation of the kernel $k'_{\theta_i}(x, x')$ using the same inducing points and weights:
\begin{align*}
\tilde{k}'_{\theta}(x, x') = \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k'_{\theta_i}(u_j, u_l).
\end{align*}
Therefore, $\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta}(x, x')$.
Substituting this into our inequality, we get:
\begin{align*}
\left\vert \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}-\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i}\right\vert &= \left\vert k'_{\theta_i}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \\
&\leq \delta_{m,L}'+\sqrt{L}c^d\delta_{m,L}'.
\end{align*}
\end{proof}
\subsubsection{Proof of Lemma \ref{lemma:partial_gradient_spectral_norm_bound}}\label{section:proof_partial_gradient_spectral_norm_bound}
\partialgradientspectralnormbound*
\begin{proof}
Let $K'_{\theta,l}$ be the kernel matrix corresponding to the kernel $k'_{\theta,l}(x,x') = \frac{\partial k_{\theta}(x,x')}{\partial \theta_l}$, and let $\tilde{K}'_{\theta,l}$ be the kernel matrix corresponding to its SKI approximation $\tilde{k}'_{\theta,l}(x,x')$.
From Lemma \ref{lemma:ski_kernel_derivative_error_kernel}, we have:
\begin{equation}
\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_l} = \tilde{k}'_{\theta,l}(x, x')
\end{equation}
Therefore:
\begin{equation}
\frac{\partial K}{\partial \theta_l} - \frac{\partial \tilde{K}}{\partial \theta_l} = K'_{\theta,l} - \tilde{K}'_{\theta,l}
\end{equation}
By Proposition \ref{prop:spectral-norm}, we have a bound on the spectral norm difference between a kernel matrix and its SKI approximation. Let $\gamma'_{n,m,L,l}$ be the corresponding bound for the kernel $k'_{\theta,l}$ and its SKI approximation $\tilde{k}'_{\theta,l}$. Then:
\begin{equation}
\| K'_{\theta,l} - \tilde{K}'_{\theta,l} \|_2 \leq \gamma'_{n,m,L,l}
\end{equation}
Thus,
\begin{align*}
\left\| \frac{\partial K}{\partial \theta_l} - \frac{\partial \tilde{K}}{\partial \theta_l} \right\|_2 = \| K'_{\theta,l} - \tilde{K}'_{\theta,l} \|_2 \leq \gamma'_{n,m,L,l}
\end{align*}
This completes the proof.
\end{proof}
% \subsubsection{Proof of Lemma \ref{lemma:gradient_spectral_norm_bound}}\label{sec:proof_gradient_spectral_norm_bound}
% \gradientspectralnormbound*
% \begin{proof}
% We represent the gradients $\nabla_\theta K$ and $\nabla_\theta \tilde{K}$ as $p \times n^2$ matrices using the vec-notation (denominator layout):
% \begin{align*}
% \nabla_\theta K = \begin{bmatrix} \frac{\partial \text{vec}(K)}{\partial \theta_1} & \frac{\partial \text{vec}(K)}{\partial \theta_2} & \cdots & \frac{\partial \text{vec}(K)}{\partial \theta_p} \end{bmatrix}^T, \\
% \nabla_\theta \tilde{K} = \begin{bmatrix} \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_1} & \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_2} & \cdots & \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_p} \end{bmatrix}^T.
% \end{align*}
% Let $K'_{\theta,i}$ be the kernel matrix corresponding to the kernel $k'_{\theta,i}(x,x') = \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}$, and let $\tilde{K}'_{\theta,i}$ be the kernel matrix corresponding to its SKI approximation $\tilde{k}'_{\theta,i}(x,x')$.
% From Lemma \ref{lemma:ski_kernel_derivative_error_kernel}, we have:
% \begin{equation}
% \frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta,i}(x, x')
% \end{equation}
% Thus, the $i$-th row of $\nabla_\theta K - \nabla_\theta \tilde{K}$ can be written as:
% \begin{equation}
% \frac{\partial \text{vec}(K)}{\partial \theta_i} - \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_i} = \text{vec}(K'_{\theta,i}) - \text{vec}(\tilde{K}'_{\theta,i}).
% \end{equation}
% Therefore, we can write:
% \begin{equation}
% \nabla_\theta K - \nabla_\theta \tilde{K} = \begin{bmatrix}
% \text{vec}(K'_{\theta,1}) - \text{vec}(\tilde{K}'_{\theta,1}) \\
% \vdots \\
% \text{vec}(K'_{\theta,p}) - \text{vec}(\tilde{K}'_{\theta,p})
% \end{bmatrix}^T
% \end{equation}
% Then, we have that:
% \begin{align}
% (\nabla_\theta K - \nabla_\theta \tilde{K})^T = \begin{bmatrix}
% \text{vec}(K'_{\theta,1}) - \text{vec}(\tilde{K}'_{\theta,1}) &
% \cdots &
% \text{vec}(K'_{\theta,p}) - \text{vec}(\tilde{K}'_{\theta,p})
% \end{bmatrix}
% \end{align}
% Now we bound the spectral norm of $(\nabla_\theta K - \nabla_\theta \tilde{K})^T$:
% \begin{align*}
% \| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_2 &\leq \| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_F \\
% &= \left( \sum_{i=1}^p \| \text{vec}(K'_{\theta,i}) - \text{vec}(\tilde{K}'_{\theta,i}) \|_2^2 \right)^{1/2} \\
% &= \left( \sum_{i=1}^p \| K'_{\theta,i} - \tilde{K}'_{\theta,i} \|_F^2 \right)^{1/2} \\
% &\leq \left( \sum_{i=1}^p \left( \max_{j \in \{1,\ldots,p\}} \| K'_{\theta,j} - \tilde{K}'_{\theta,j} \|_2 \right)^2 \right)^{1/2}
% \end{align*}
% \jmei{the term in the sum is already independent of $i$, so we could replace the sum by $p\times$ and then swap the norm for $\gamma$.}
% By Proposition \ref{prop:spectral-norm}, we have a bound on the spectral norm difference between a kernel matrix and its SKI approximation. Let $\gamma'_{n,m,L,i}$ be the corresponding bound for the kernel $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$. Then:
% \begin{equation}
% \| K'_{\theta,i} - \tilde{K}'_{\theta,i} \|_2 \leq \gamma'_{n,m,L,i}
% \end{equation}
% for each $i \in \{1, \ldots, p\}$. Thus,
% \begin{align*}
% \| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_2 &\leq \left( \sum_{i=1}^p (\gamma'_{n,m,L,i})^2 \right)^{1/2} \\
% &\leq \left( \sum_{i=1}^p \left(\max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}\right)^2 \right)^{1/2} \\
% &= \left( p \left(\max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}\right)^2 \right)^{1/2} \\
% &= \sqrt{p} \max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}
% \end{align*}
% This completes the proof.
% \end{proof}
\subsubsection{Proof of Lemma \ref{lemma:score-function-bound}}\label{sec:proof-score-function-bound}
\scorefunctionbound*
\begin{proof}
We start with the expressions for the gradients:
$$
\nabla \mathcal{L}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
$$
\nabla \tilde{\mathcal{L}}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
Thus, the difference is:
\begin{align*}
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 &= \left\| \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| \right) \right. \\
&\quad \left. - \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&\leq \underbrace{\left\| \nabla \left( \frac{1}{2} \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2}_{T_1} \\
&\quad + \underbrace{\left\| \frac{1}{2} \nabla \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2}_{T_2}.
\end{align*}
We will bound $T_1$ and $T_2$ separately.
\textbf{Bounding $T_1$:}
\begin{align*}
T_1 &= \frac{1}{2} \left\| \nabla_\theta \left( \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2\\
&=\frac{1}{2}\sqrt{\sum_{l=1}^p \left(\frac{\partial}{\partial \theta_l}\mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y}\right)^2}\\
&\leq \frac{1}{2}\sqrt{p}\max_{1\leq l\leq p}\sqrt{\left(\frac{\partial}{\partial \theta_l}\mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y}\right)^2}\\
&=\frac{1}{2}\sqrt{p}\max_{1\leq l\leq p} \left\vert \frac{\partial}{\partial \theta_l}\mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y}\right\vert
\end{align*}
We will then bound $\left\vert \frac{\partial}{\partial \theta_l}\mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y}\right\vert$. Using the following equality $\frac{\partial}{\partial \theta_l} \mathbf{X}^{-1} = -\mathbf{X}^{-1} (\frac{\partial \mathbf{X}}{\partial \theta_l}) \mathbf{X}^{-1}$, we can express this derivative as a quadratic form as a difference between two quadratic forms and apply standard techniques for bounding differences between quadratic forms.
% \begin{align*}
% &\left\vert \frac{\partial}{\partial \theta_l}\mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y}\right\vert\\
% &\leq\Vert \textbf{y}\Vert_2^2 \left\Vert \frac{\partial}{\partial \theta_l }\left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \right\Vert_2\text{ CS inequality}\\
% &=\Vert \textbf{y}\Vert_2^2 \left\Vert \frac{\partial}{\partial \theta_l }\left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \right\Vert_2\\
% &= \Vert \textbf{y}\Vert_2^2 \left\| - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}}\right) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right\|_2\text{ above equality}\\
% &\leq \frac{1}{\sigma^4}\Vert \textbf{y}\Vert_2^2\left(\left\Vert\frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}}\right\Vert+\left\Vert\frac{\partial}{\partial \theta_l} \mathbf{K}\right\Vert\right)\text{ CS inequality}
% \end{align*}
% Since the kernel is $C^1$ wrt $\theta$ and $\mathcal{D}$ is compact, we can bound the entries of $\frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}}$ and $\frac{\partial}{\partial \theta_l} \mathbf{K}$ uniformly over all $1\leq l\leq p$ with constants $\tilde{C},C>0$. Recalling the property $\Vert \cdot\Vert_2\leq \sqrt{\Vert \cdot\Vert_1\Vert \cdot\Vert_\infty}$, we can bound $T_1$ as:
\begin{align*}
&\left\vert \frac{\partial}{\partial \theta_l}\mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y}\right\vert\\
&\leq\Vert \textbf{y}\Vert_2^2 \left\Vert \frac{\partial}{\partial \theta_l }\left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \right\Vert_2\text{ CS inequality}\\
&=\Vert \textbf{y}\Vert_2^2 \left\Vert \frac{\partial}{\partial \theta_l }\left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \right\Vert_2\\
&= \Vert \textbf{y}\Vert_2^2 \left\| - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}}\right) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right\|_2 \\
&= \Vert \textbf{y}\Vert_2^2 \left\| - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}} - \frac{\partial}{\partial \theta_l} \mathbf{K} + \frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\\
&\quad \left. + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \frac{\partial}{\partial \theta_l} \mathbf{K} (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right\|_2 \\
&= \Vert \textbf{y}\Vert_2^2 \left\| - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}} - \frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\\
&\quad \left. - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right\|_2 \\
&\leq\Vert \textbf{y}\Vert_2^2 \left(\underbrace{\left\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}} - \frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(a)} \right.\\
&\quad \left.+ \underbrace{\left\| \left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right)\left(\frac{\partial}{\partial \theta_l} \mathbf{K}\right) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(b)} \right.\\
&\quad \left.+ \underbrace{\left\|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial}{\partial \theta_l} \mathbf{K}\right)\left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right) \right\|_{2}}_{(c)}\right).
\end{align*}
We now explicitly bound (a), (b), and (c).
\begin{align*}
(a) &\leq \left\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_2 \left\| \frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}} - \frac{\partial}{\partial \theta_l} \mathbf{K} \right\|_2 \left\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_2\\
&\leq \left\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_2^2 \left\| \frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}} - \frac{\partial}{\partial \theta_l} \mathbf{K} \right\|_2 \\
&\leq \frac{1}{\sigma^4} \left\| \frac{\partial}{\partial \theta_l} \mathbf{K} - \frac{\partial}{\partial \theta_l} \tilde{\mathbf{K}} \right\|_2\\
&\leq \frac{1}{\sigma^4} \gamma'_{n,m,L,l} \quad \text{ (Using Lemma \ref{lemma:partial_gradient_spectral_norm_bound})}
\end{align*}
\begin{align*}
(b) &\leq \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \left\|\frac{\partial}{\partial \theta_l} \mathbf{K}\right\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \left\|\frac{\partial}{\partial \theta_l} \mathbf{K}\right\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \left\|\frac{\partial}{\partial \theta_l} \mathbf{K}\right\|_2 \quad \text{(Using Lemma \ref{lemma:action-inverse-error})}
\end{align*}
Since the kernel is $C^1$ wrt $\theta$ and $\mathcal{D}$ is compact, we can bound the entries of $\frac{\partial}{\partial \theta_l} \mathbf{K}$ uniformly over $\mathcal{D}$ and $l$ with some constant, say $C>0$. Then by Lemma \ref{lemma:test-train-bound}, reusing the training points instead of using the test points,
\begin{align*}
(b)&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \left\|\frac{\partial}{\partial \theta_l} \mathbf{K}\right\|_2\\
% &\leq \frac{\gamma_{n,m,L}}{\sigma^4} \sqrt{\Vert \frac{\partial}{\partial \theta_l} \mathbf{K}\Vert_1\Vert \frac{\partial}{\partial \theta_l} \mathbf{K}\Vert_\infty}\\
&\leq Cn\frac{\gamma_{n,m,L}}{\sigma^4}
\end{align*}
and finally
\begin{align*}
(c) &\leq \|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \left\|\frac{\partial}{\partial \theta_l} \mathbf{K}\right\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \left\|\frac{\partial}{\partial \theta_l} \mathbf{K}\right\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \left\|\frac{\partial}{\partial \theta_l} \mathbf{K}\right\|_2 \quad \text{(Using Lemma \ref{lemma:action-inverse-error})} \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \gamma'_{n,m,L,l} \quad \text{(Using Lemma \ref{lemma:partial_gradient_spectral_norm_bound} )}
\end{align*}
Combining these, we obtain
\begin{align*}
T_1 &\leq \frac{1}{2\sigma^4}\Vert \textbf{y}\Vert\sqrt{p}\max_{1\leq l\leq p} \left( \gamma'_{n,m,L,l}+Cn\gamma_{n,m,L}+\gamma_{n,m,L}\gamma'_{n,m,L,l} \right)
\end{align*}
% \begin{align*}
% T_1&\leq \frac{\Vert \textbf{y}\Vert_2^2 n}{2\sigma^4}\sqrt{p}\left(\tilde{C}+C\right)
% \end{align*}
**Bounding $T_2$:**
Using the identity $\nabla \log |\mathbf{X}| = (\mathbf{X}^{-1})^\top$, we have
\begin{align*}
T_2 &= \frac{1}{2} \left\| \nabla_\theta \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&= \frac{1}{2} \left\| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_2.
\end{align*}
We can rewrite the difference as:
$$
(\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} = (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\tilde{\mathbf{K}} - \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}
$$
Then
\begin{align*}
T_2 &\leq \frac{1}{2} \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2 \| \tilde{\mathbf{K}} - \mathbf{K} \|_2 \| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \|_2 \\
&\leq \frac{\gamma_{n,m,L}}{2\sigma^4}
\end{align*}
**Combining the Bounds:**
Combining the bounds for $T_1$ and $T_2$, we have
\begin{align*}
\| \nabla \mathcal{L}(\boldsymbol{\theta}) - \nabla \tilde{\mathcal{L}}(\boldsymbol{\theta}) \|_2 &\leq \frac{1}{2\sigma^4}\Vert \textbf{y}\Vert\sqrt{p}\max_{1\leq l\leq p} \left( \gamma'_{n,m,L,l}+Cn\gamma_{n,m,L}+\gamma_{n,m,L}\gamma'_{n,m,L,l} \right)+\frac{\gamma_{n,m,L}}{2\sigma^4}
\end{align*}
\end{proof}
\subsubsection{Proof of Lemma \ref{lemma:mean-inference}}\label{sec:proof-mean-inference}
\meaninference*
\begin{proof}
We start by expressing the difference between the true and SKI posterior means:
% \begin{align*}
% & \left\|\mathbf{K}_{\cdot,\mathbf{X}} \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} \right\|_{2} \\
% &\leq \left\|\mathbf{K}_{\cdot,\mathbf{X}} \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y}\right\|_2 +\left\| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} \right\|_{2}\\
% &\leq \frac{\Vert \textbf{y}\Vert_2}{\sigma^2} \left(\left\Vert\mathbf{K}_{\cdot,\mathbf{X}}\right\Vert_2+\left\Vert \tilde{\mathbf{K}}_{\cdot, \mathbf{X}}\right\Vert_2\right)\\
% &\leq \frac{\Vert \textbf{y}\Vert_2}{\sigma^2} (\sqrt{Tn}M+\sqrt{Tn}mc^{2d}M)
% \end{align*}
% where the last line follows from Lemmas \ref{lemma:test-train-bound} and \ref{lemma:ski-test-train-bound}.
\begin{align*}
& \left\|\mathbf{K}_{\cdot,\mathbf{X}} \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} \right\|_{2} \\
&= \left\| \left( \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} -\mathbf{K}_{\cdot,\mathbf{X}}\right) \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} +\mathbf{K}_{\cdot,\mathbf{X}}\left[ \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} - \left(\mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} \right] \mathbf{y} \right\|_{2}\\
%&= \left\| \left(\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \right) \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} + \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left[ \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right] \mathbf{y} \right\|_{2}
\end{align*}
Applying the triangle inequality and submultiplicative property gives:
\begin{align*}
&\leq \frac{1}{\sigma^2}\Vert \textbf{y}\Vert_2\Vert \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} -\mathbf{K}_{\cdot,\mathbf{X}}\Vert_2+\Vert\mathbf{K}_{\cdot,\mathbf{X}}\Vert_2\left\Vert \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} - \left(\mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1}\right\Vert_2\Vert \textbf{y}\Vert_2\\
&\leq \frac{\max\left(\gamma_{T,m,L},\gamma_{n,m,L}\right)}{\sigma^2}\Vert \textbf{y}\Vert_2+\Vert\mathbf{K}_{\cdot,\mathbf{X}}\Vert_2\left\Vert \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} - \left(\mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1}\right\Vert_2\Vert \textbf{y}\Vert_2\text{ Lemma \ref{lemma:test-train-kernel-matrix-error}}\\
&\leq \frac{\max\left(\gamma_{T,m,L},\gamma_{n,m,L}\right)}{\sigma^2}\Vert \textbf{y}\Vert_2+\sqrt{Tn}M\left\Vert \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} - \left(\mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1}\right\Vert_2\Vert \textbf{y}\Vert_2\text{ Lemma \ref{lemma:test-train-bound}}\\
&\leq \frac{\max\left(\gamma_{T,m,L},\gamma_{n,m,L}\right)}{\sigma^2}\Vert \textbf{y}\Vert_2+\frac{\sqrt{Tn}M}{\sigma^4}\gamma_{n,m,L}\Vert \textbf{y}\Vert_2 \text{ Lemma \ref{lemma:action-inverse-error}}\\
&=\frac{1}{\sigma^2}\Vert \textbf{y}\Vert_2\left(\max\left(\gamma_{T,m,L},\gamma_{n,m,L}\right)+\frac{\sqrt{Tn}M}{\sigma^4}\gamma_{n,m,L}\right)\\
&=\frac{1}{\sigma^2}\Vert \textbf{y}\Vert_2O\left(c^{2d}\frac{\max(T,n)+\sqrt{Tn}Mn}{m^{3/d}}\right)
\end{align*}
% \begin{align*}
% & \leq \left\| \left(\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \right) \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \| \mathbf{y} \|_{2} + \left\| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right) \mathbf{y} \right\|_{2} \\
% &\leq \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} \left\| \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \| \mathbf{y} \|_{2} + \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} \left\| \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \cdot \| \mathbf{y} \|_{2} \\
% & \leq \frac{ \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{2} } \| \mathbf{y} \|_{2} + \gamma_{n, m, L}\frac{ \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{4} } \| \mathbf{y} \|_{2} \textrm{ by Lemma \ref{lemma:action-inverse-error}}\\
% & \leq \left( \frac{ \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{2} } + \frac{ \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{4} } \gamma_{n, m, L} \right) \| \mathbf{y} \|_{2}\\
% &\leq \left(\frac{\max(\gamma_{T,m,L},\gamma_{n,m,L})}{\sigma^2}+\frac{\Vert \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \Vert_2}{\sigma^4}\gamma_{n,m,L}\right)\Vert \textbf{y}\Vert_2\\
% &\leq \left(\frac{\max(\gamma_{T,m,L},\gamma_{n,m,L})}{\sigma^2}+\frac{\sqrt{Tn}mMc^{2d}}{\sigma^4}\gamma_{n,m,L}\right)\Vert \textbf{y}\Vert_2
% \end{align*}
\end{proof}
\subsubsection{Proof of Lemma \ref{lemma:ski-posterior-covariance-error}}\label{sec:proof-ski-posterior-covariance-error}
\begin{proof}
% Note that
% \begin{align*}
% \Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \Vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\Vert_2\\
% &+ \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
% &\leq \gamma_{T,m,L} + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
% &\leq \gamma_{T,m,L}+\frac{1}{\sigma^2}\left(\Vert \textbf{K}_{\cdot,X}\Vert^2_2+\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2^2\right)\\
% &\leq \gamma_{T,m,L}+\frac{1}{\sigma^2}\left(TnM^2+Tnm^2c^{4d}M^2\right)
% \end{align*}
% where the last line again follows from Lemmas \ref{lemma:test-train-bound} and \ref{lemma:ski-test-train-bound}.
First, note that
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \Vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\Vert_2\\
&+ \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \gamma_{T,m,L} + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2,
\end{align*}
where we used Proposition \ref{prop:spectral-norm} and the fact that $\Vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\Vert_2 \leq \gamma_{T,m,L}$.
% Noting that the second term is a difference between two quadratic forms and the inequality $\Vert \textbf{B}^T \textbf{A}_1 \textbf{B} - \textbf{C}^T \textbf{A}_2 \textbf{C}\Vert_2 \leq \Vert \textbf{B}\Vert_2^2 \Vert \textbf{A}_1 - \textbf{A}_2\Vert_2 + \Vert \textbf{A}_2\Vert_2 (\Vert \textbf{B}\Vert_2 + \Vert \textbf{C}\Vert_2) \Vert \textbf{B} - \textbf{C}\Vert_2$, we can set $\textbf{A}_1$ and $\textbf{A}_2$ to be the regularized inverse of the true and SKI gram matrices, respectively and $\textbf{B}$ and $\textbf{C}$ to be the true and SKI cross-kernel matrices. Then
% \begin{align*}
% &\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2,\\
% &\quad\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2^2\Vert (\textbf{K}+\sigma^2 I)^{-1}-(\tilde{\textbf{K}}+\sigma^2 I)^{-1}\Vert_2\\
% &\qquad+\Vert \tilde{\textbf{K}}\Vert_2\left(\Vert \textbf{K}_{\textbf{X},\cdot}\Vert_2+\Vert \tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\right)\Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2
% \end{align*}
% Now
Now, we bound the second term, which is a different between two quadratic forms:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&+ \Vert\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}(\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot})\Vert_2 + \Vert (\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1})\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert (\textbf{K}+\sigma^2 I)^{-1} \Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2\\
&\leq \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2,
\end{align*}
where we used the fact that $(\textbf{K}+\sigma^2 I)^{-1} \preceq \frac{1}{\sigma^2}I$.
Next, we bound the term $\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2$:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \\
&= \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1} - \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} + \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1} - \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&= \Vert \textbf{K}_{\cdot,\textbf{X}} [(\textbf{K}+\sigma^2 I)^{-1} - (\tilde{\textbf{K}}+\sigma^2 I)^{-1}] \Vert_2 + \Vert (\textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}}) (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert (\textbf{K}+\sigma^2 I)^{-1} - (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \Vert (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \frac{\gamma_{n,m,L}}{\sigma^4} + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \frac{1}{\sigma^2},
\end{align*}
where we used Lemma \ref{lemma:action-inverse-error} in the last inequality. Substituting this back into the main inequality, we get:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \left(\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \frac{\gamma_{n,m,L}}{\sigma^4} + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \frac{1}{\sigma^2}\right) \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2\\
&= \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \frac{\gamma_{n,m,L}}{\sigma^4}\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 + \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2.
\end{align*}
Using Lemma \ref{lemma:test-train-kernel-matrix-error} and the fact that $\Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \leq \max(\gamma_{T,m,L},\gamma_{n,m,L})$ and that $\textbf{K}_{\cdot, \textbf{X}} = \textbf{K}_{\textbf{X},\cdot}^\top$, we have $\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 = \Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2$. Also, by assumption, $\Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}M$. Using Lemma \ref{lemma:ski-test-train-bound}, we have $\Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}mc^{2d}M$. Substituting these bounds, we get:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) + \frac{\gamma_{n,m,L}}{\sigma^4}(\sqrt{Tn}M)(\sqrt{Tn} m c^{2d} M) + \frac{1}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) (\sqrt{Tn} m c^{2d} M) \\
&= \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) + \frac{\gamma_{n,m,L}}{\sigma^4}Tn m c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}).
\end{align*}
Finally, substituting this back into the original inequality, we obtain the desired bound:
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \gamma_{T,m,L} + \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L})\\
&+ \frac{\gamma_{n,m,L}}{\sigma^4}Tn m c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}).\\
&=O\left(\frac{Tn^2mc^{4d}M^2+\sqrt{Tn}mc^{4d}M\max(T,n)}{m^{3/d}}\right).
\end{align*}
\end{proof}
\subsubsection{Proof of Lemma \ref{lemma:ski-kernel-elementwise-error}}\label{sec:proof-ski-kernel-elementwise-error}
\skikernelelementwiseerror*
\begin{proof}
Recall that SKI approximates the kernel as
\begin{align*}
k(\mathbf{x}, \mathbf{x}') &\approx \tilde{k}(\mathbf{x}, \mathbf{x}')\\
&= \boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}'),
\end{align*}
Let $\textbf{K}_{\textbf{U},\textbf{x}'}\in \mathbb{R}^m$ be the vector of kernels between the inducing points and the vector $\textbf{x}'$
\begin{align}
\vert k(\mathbf{x}, \mathbf{x}') -\tilde{k}(\mathbf{x}, \mathbf{x}')\vert &= \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} +\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\nonumber\\
&\leq \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\nonumber\\
&\leq \delta_{m,L}+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert \textrm{ since $\vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert$ is a single polynomial interpolation}\label{eqn:applying-single-poly-interp}
\end{align}
Now note that $\textbf{w}(x)\in \mathbb{R}^m$ is a sparse matrix with at most $L$ non-zero entries. Thus, letting $\tilde{\textbf{w}}(x)\in \mathbb{R}^L$ be the non-zero entries of $\textbf{w}(x)$ and similarly $\tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}\in \mathbb{R}^L$ be the entries of $\textbf{K}_{\textbf{U},\textbf{x}'}$ in the dimensions corresponding to non-zero entries of $\textbf{w}(x)\in \mathbb{R}^m$, while $\tilde{\textbf{K}}_{\textbf{U}}\in \mathbb{R}^{L\times m}$ is the analogous matrix for $\textbf{K}_{\textbf{U}}$, we have
\begin{align}
\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert&=\vert \tilde{\textbf{w}}(\textbf{x})^\top \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\textbf{w}}(\textbf{x})^\top \tilde{\textbf{K}}_\textbf{U}\textbf{w}(\textbf{x}')\vert\nonumber\\
&\leq \Vert \tilde{\textbf{w}}(\textbf{x})\Vert_2\Vert \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\mathbf{K}}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2\nonumber\\
&\leq c^d\sqrt{L}\Vert \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\mathbf{K}}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_\infty\nonumber\text{ Lemma \ref{lemma:cubic-interpolation-weights-curse-dimensionality}}\\
&\leq \sqrt{L}c^d\delta_{m,L}\label{eqn:applying-sparsity-single-kernel-evaluation}
\end{align}
where the last line follows as each element of $\mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')$ is a polynomial interpolation approximating each element of $\textbf{K}_{\textbf{U},\textbf{x}'}$. Plugging Eqn. \ref{eqn:applying-sparsity-single-kernel-evaluation} into Eqn. \ref{eqn:applying-single-poly-interp} gives us the desired initial result of
\begin{align*}
\vert k(\textbf{x},\textbf{x}')-\tilde{k}(\textbf{x},\textbf{x}')\vert&\leq \delta_{m,L}+\sqrt{L}c^d\delta_{m,L}
\end{align*}
and Lemma \ref{lemma:tensor-product-interpolation-error} gives us the result when the convolutional kernel is cubic.
\end{proof}
\subsubsection{Proof of Theorem \ref{thm:inducing-points-count-alt}}\label{sec:proof-inducing-points-count-alt}
\inducingpointscountalt*
\begin{proof}
We want to choose $m$ such that the spectral norm error $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$. From Proposition \ref{prop:spectral-norm}, we have:
$$
\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq n(1 + \sqrt{L}c^d) \delta_{m,L}
$$
For cubic interpolation ($L=4$), Lemma \ref{lemma:tensor-product-interpolation-error}, combined with the analysis in Lemma \ref{lemma:tensor-product-interpolation-error}, gives us:
$$
\delta_{m,L} \leq K' c^{2d} h^3
$$
where $K'$ is a constant that depends only on the kernel function (through its derivatives) and the interpolation scheme, but not on $n$, $m$, $h$, or $d$.
Therefore, a sufficient condition to ensure $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$ is:
\begin{equation} \label{eq:sufficient_condition_final}
n(1 + 2c^d) K' c^{2d} h^3 \leq \epsilon
\end{equation}
Since the inducing points are placed on a regular grid with spacing $h$ in each dimension, and the domain is $[-D,D]^d$ and assuming that $2D\mod h\equiv 0$, the number of inducing points $m$ satisfies:
$$
m = \left(\frac{2D}{h}\right)^d
$$
We can rearrange this to get:
$$
h = \frac{2D}{m^{1/d}}
$$
Substituting this into the sufficient condition \eqref{eq:sufficient_condition_final}, we get:
$$
n (1 + 2c^d) K' c^{2d} \left(\frac{2D}{m^{1/d}}\right)^3 \leq \epsilon
$$
Rearranging to isolate $m$, we obtain:
$$
m^{3/d} \geq \frac{n}{\epsilon} (1 + 2c^d) K' c^{2d} (8D^3)
$$
$$
m \geq \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
$$
\end{proof}
\subsubsection{Proof of Corollary \ref{cor:linear-time}}\label{proof:cor-linear-time}
\corlineartime*
\begin{proof}
% The idea is that we want to have an error sufficiently large so that when we choose $m$ based on Corollary \ref{cor:inducing-points-count-alt}, we have $m=O\left(\frac{n}{\log n}\right)$.
Assume that
$$
\epsilon \geq \frac{(1 + 2c^d) K' 8 c^{2d} D^3}{C^{3/d}} \cdot \frac{n (\log n)^{3/d}}{n^{3/d}}.
$$
Rearranging this we obtain
\begin{align*}
\left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}&\leq C\frac{n}{\log n}.\\
&=O\left(\frac{n}{\log n}\right).
\end{align*}
Now taking
\begin{align*}
m &= \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
\end{align*}
we have that $m=O\left(\frac{n}{\log n}\right)$ and by Theorem \ref{thm:inducing-points-count-alt}, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2\leq \epsilon$. Now plugging in $\frac{n}{\log n}$ into $m\log m$ we obtain
\begin{align*}
O\left(m\log m\right)&=O\left(\frac{n}{\log n}\log \frac{n}{\log n}\right)\\
&=O\left(\frac{n}{\log n}\log n-\frac{n}{\log n}\log \log n\right)\\
&=O(n)
\end{align*}
as desired.
\end{proof}
% \subsubsection{Proof of Corollary \ref{cor:linear-time-low-dimensional}}
% \begin{proof}
% We need that $m=O\left(\frac{n}{\log n}\right)$ and $m=\left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}$ can jointly hold. That is, that
% \begin{align*}
% \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}&=O\left(\frac{n}{\log n}\right)
% \end{align*}
% Now let $C\equiv (1 + 2c^d) K' (8 c^{2d} D^3)$. Then a sufficient condition is that the following holds for $n$ sufficiently large:
% \begin{align*}
% \left( \frac{n}{\epsilon}C\right)^{d/3}&\leq \frac{n}{\log n}\\
% C^{d/3} \frac{n^{d/3-1}}{\log n}&\leq \epsilon^{d/3}\\
% \frac{n^{d/3-1}}{\log n}&\leq \epsilon^{d/3}C^{-d/3}.
% \end{align*}
% For $d\leq 3$, the lhs goes to $0$ as $n\rightarrow \infty$ and thus for sufficiently large $n$, the inequality will hold.
% \end{proof}
% \subsection{Additional Quantities}\label{subsec:additional-quantities}
% \subsubsection{Action of Regularized Inverse}\label{subsubsec:action-regularized-inverse}
\subsubsection{Proof of Lemma \ref{lemma:tensor-product-interpolation-error}}\label{sec:proof-tensor-product-interpolation}
\tensorproductinterpolationerror*
\begin{proof}
We define a sequence of intermediate interpolation functions. Let $g_0(\mathbf{x}) \equiv f(\mathbf{x})$ be the original function. For $i = 1, \ldots, d$, we recursively define $g_i(\mathbf{x})$ as the function obtained by interpolating $g_{i-1}$ along the $i$-th dimension using the cubic convolution kernel $u$:
$$
g_i(\mathbf{x}) \equiv \sum_{k=-1}^2 g_{i-1}\left(\mathbf{x} + \left( (\mathbf{c_x})_i - x_i + kh\right)\mathbf{e}_i \right) u\left(\frac{x_i - (\mathbf{c_x})_i - kh}{h}\right).
$$
Here, $\mathbf{c_x}$ is the grid point closest to $\mathbf{x}$, and $\mathbf{e}_i$ is the $i$-th standard basis vector. Thus, $g_1(\mathbf{x})$ interpolates $f$ along the first dimension, $g_2(\mathbf{x})$ interpolates $g_1$ along the second dimension, and so on, until $g_d(\mathbf{x}) = g(\mathbf{x})$ is the final tensor-product interpolated function.
We analyze the error accumulation across multiple dimensions using induction. Using \cite{keys1981cubic}, the error introduced by interpolating a thrice continuous differentiable function along a single dimension with the cubic convolution kernel is uniformly bounded over the interval domain by $Kh^3$ for some constant $K > 0$, provided the grid spacing $h$ is sufficiently small. This gives us the base case:
$$
|g_1(\mathbf{x}) - g_0(\mathbf{x})| \leq Kh^3.
$$
For the inductive step, assume that for some $i=k$ the error is uniformly bounded by
$$
|g_k(\mathbf{x}) - g_{k-1}(\mathbf{x})| \leq c^{k-1}Kh^3.
$$
We want to show that this bound also holds for $i=k+1$. We can express the difference $g_{k+1}(\mathbf{x}) - g_k(\mathbf{x})$ as follows:
\begin{align*}
g_{k+1}(\mathbf{x}) - g_k(\mathbf{x}) &= \sum_{k_{k+1}=-1}^2 g_k\left(\mathbf{x} + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) \\
&\quad - g_k(\mathbf{x}) \\
&= \sum_{k_{k+1}=-1}^2 \left[\sum_{k_k=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) \right. \\
&\quad \left. u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right)\right] u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) \\
&\quad - \sum_{k_k=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right) \\
&= \sum_{k_k=-1}^2 u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right) \left[\sum_{k_{k+1}=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) \right. \\
&\quad \left. u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k\right)\right].
\end{align*}
The inner term in the last expression represents the difference between interpolating $g_{k-1}$ along the $(k+1)$-th dimension and $g_{k-1}$ itself, evaluated at $\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k$. This can be written as:
\begin{align*}
&\sum_{k_{k+1}=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) \\
&\quad - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k\right) \\
&= g_k\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right).
\end{align*}
Therefore, we can bound the error as follows:
\begin{align*}
|g_{k+1}(\mathbf{x}) - g_k(\mathbf{x})| &\leq \left|\sum_{k_k=-1}^2 u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right)\right| \cdot \left|g_k\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right)\right|.
\end{align*}
Let $c>0$ be a uniform upper bound for $\sum_{k_k=-1}^2 \left|u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right)\right|$, which exists because $u$ is bounded. By the inductive hypothesis, we have $\left|g_k\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right)\right| \leq c^{k-1}Kh^3$. Thus,
$$
|g_{k+1}(\mathbf{x}) - g_k(\mathbf{x})| \leq c \cdot c^{k-1}Kh^3 = c^k Kh^3.
$$
This completes the inductive step.
Finally, we bound the total error $|g(\mathbf{x}) - f(\mathbf{x})| = |g_d(\mathbf{x}) - g_0(\mathbf{x})|$ by summing the errors introduced at each interpolation step:
$$
|g(\mathbf{x}) - f(\mathbf{x})| \leq \sum_{i=1}^d |g_i(\mathbf{x}) - g_{i-1}(\mathbf{x})| \leq \sum_{i=1}^d c^{i-1}Kh^3 = Kh^3 \sum_{i=0}^{d-1} c^i.
$$
The last sum is a geometric series, which evaluates to $Kh^3 \frac{1 - c^d}{1 - c}$. For a fixed $c>1$ (independent of $d$), this expression is $O(c^{d})$ when $d$ is large. Therefore, tensor-product cubic convolutional interpolation has $O(c^d h^3)$ error. Finally, noticing that $h=O\left(\frac{1}{m^{d/3}}\right)$ gives us the desired result.
\end{proof}
\subsubsection{Curse of Dimensionality for Kernel Regression}
The next lemma shows that when using a product kernel for $d$-dimensional kernel regression (where cubic convolutional interpolation is a special case), the sum of weights suffers from the curse of dimensionality. The proof strategy involves expressing the multi-dimensional sum as a product of sums over each individual dimension, leveraging the initial condition on the one-dimensional bound for each dimension, and taking advantage of the structure of the Cartesian grid.
\begin{restatable}{lemma}{cubicInterpolationWeights}
\label{lemma:cubic-interpolation-weights-curse-dimensionality}
Let $u: \mathbb{R} \rightarrow \mathbb{R}$ be a one-dimensional kernel function with constant $c>0$ defined as in \ref{def:sum-weight-upper-bound}. Let $u_d: \mathbb{R}^d \rightarrow \mathbb{R}$ be a d-dimensional product kernel defined as:
$$
u_d\left(\frac{x - x_i}{h}\right) = \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right),
$$
where $x = (x^{(1)}, x^{(2)}, ..., x^{(d)}) \in \mathbb{R}^d$ and $x_i = (x_i^{(1)}, x_i^{(2)}, ..., x_i^{(d)}) \in \mathbb{R}^d$ are d-dimensional points. Assume the data points $\{x_i\}_{i=1}^n$ ($n$ may differ from the univariate case) lie on a fixed d-dimensional grid $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$, where each $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is a finite set of $n_j$ grid points along the j-th dimension for $j = 1, 2, ..., d$. Then, for any $x \in \mathbb{R}^d$, the sum of weights in the d-dimensional kernel regression is bounded by $c^d$:
$$
\sum_{i=1}^n \left|u_d\left(\frac{x - x_i}{h}\right) \right|\leq c^d.
$$
\end{restatable}
\begin{proof}
Let the fixed d-dimensional grid be defined by the Cartesian product of d sets of 1-dimensional grid points: $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$, where $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is the set of grid points along the j-th dimension.
We start with the sum of weights in the d-dimensional case:
$$
\sum_{i=1}^n u_d\left(\frac{x - x_i}{h}\right) = \sum_{i=1}^n \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right)
$$
Since the data points lie on the fixed grid $G$, we can rewrite the outer sum as a nested sum over the grid points in each dimension:
$$
\sum_{i=1}^n \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right) = \sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right)
$$
Now we can change the order of summation and product, as proven in Lemma \ref{lemma:switch-sum-product}:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) \right)
$$
By the assumption of the lemma, we know that for each dimension $j$, the sum of weights is bounded by $c$. Note that $\{p_{k_j}^{(j)}\}_{k_j=1}^{n_j}$ is simply a set of points in $\mathbb{R}$, thus:
$$
\sum_{k_j=1}^{n_j} \left|u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right)\right| \leq c
$$
Therefore, we have:
$$
\prod_{j=1}^d \left(\left| \sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right)\right| \right) \leq \prod_{j=1}^d c = c^d
$$
Thus, we have shown that:
$$
\sum_{i=1}^n \left|u_d\left(\frac{x - x_i}{h}\right) \right|\leq c^d
$$
\end{proof}
\subsubsection{Proof of Proposition \ref{prop:spectral-norm}}\label{sec:proof-spectral-norm}
\spectralnorm*
\begin{proof}
Recall that for any matrix $\textbf{A}$, $\Vert \textbf{A} \Vert_2 \leq \sqrt{\Vert \textbf{A} \Vert_1 \Vert \textbf{A} \Vert_\infty}$. Since $\textbf{K}-\tilde{\textbf{K}}$ is symmetric, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq \sqrt{\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_1\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty} = \Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty
\end{align*}
Furthermore, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty$ is the maximum absolute row sum of $\textbf{K}-\tilde{\textbf{K}}$. Since there are $n$ rows and, by Lemma \ref{lemma:ski-kernel-elementwise-error}, each element of $\textbf{K} - \tilde{\textbf{K}}$ is bounded by $\delta_{m,L}+\sqrt{L}c^d\delta_{m,L}$ in absolute value, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty &\leq n \left(\delta_{m,L}+\sqrt{L} c^d\delta_{m,L}\right) = \gamma_{n,m,L}.
\end{align*}
Therefore, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2 \leq \gamma_{n,m,L}$.
\end{proof}
\subsubsection{Proof of Lemma \ref{lemma:test-train-kernel-matrix-error}}\label{sec:proof-test-train-kernel-matrix-error}
\testtrainkernelmatrixerror*
\begin{proof}
Using the same reasoning as in Proposition \ref{prop:spectral-norm}, we have
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq \sqrt{\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty} \\
&\leq \max \left(\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1, \Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty\right).
\end{align*}
Now, $\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1$ is the maximum absolute column sum, which is less than or equal to $T(\delta_{m,L} + \sqrt{L}c^d\delta_{m,L}) = \gamma_{T,m,L}$. Similarly, $\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty$ is the maximum absolute row sum, which is upper bounded by $n(\delta_{m,L} + \sqrt{L}c^d\delta_{m,L}) = \gamma_{n,m,L}$. Therefore,
$$
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq \max(\gamma_{T,m,L},\gamma_{n,m,L}).
$$
\end{proof}
\subsubsection{Additional Spectral Norm Bounds}
\begin{restatable}{lemma}{traintestbound}\label{lemma:test-train-bound}
Let $\textbf{K}_{\cdot,\textbf{X}} \in \mathbb{R}^{T \times n}$ be cross kernel matrix between $T$ test points and $n$ training points, where the SKI approximation uses $m$ inducing points. If the kernel function $k$ is bounded such that $|k(\textbf{x}, \textbf{x}')| \leq M$ for all $\textbf{x}, \textbf{x}'\in \mathcal{X}$, then:
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2&\leq \sqrt{Tn}M
\end{align*}
\end{restatable}
\begin{proof}
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2&\leq \sqrt{\Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_1\Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_\infty}\\
&\leq \sqrt{Tn}M
\end{align*}
\end{proof}
\begin{restatable}{lemma}{skitesttrainbound}\label{lemma:ski-test-train-bound}
Let $\tilde{\textbf{K}}_{\cdot,\textbf{X}} \in \mathbb{R}^{T \times n}$ be the matrix of SKI kernel evaluations between $T$ test points and $n$ training points, where the SKI approximation uses $m$ inducing points. Let $\textbf{W}(\cdot) \in \mathbb{R}^{T \times m}$ and $\textbf{W}(\textbf{X}) \in \mathbb{R}^{n \times m}$ be the matrices of interpolation weights for the test points and training points, respectively. Assume that the interpolation scheme is such that the sum of the absolute values of the interpolation weights for any point is bounded by $c^d$, where $c>0$ is a constant. Let $\textbf{K}_{\textbf{U}} \in \mathbb{R}^{m \times m}$ be the kernel matrix evaluated at the inducing points. If the kernel function $k$ is bounded such that $|k(\textbf{x}, \textbf{x}')| \leq M$ for all $\textbf{x}, \textbf{x}'\in \mathcal{X}$, then:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq \sqrt{Tn} m c^{2d} M
$$
\end{restatable}
\begin{proof}
By the definition of the SKI approximation and the submultiplicativity of the spectral norm, we have:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 = \Vert \textbf{W}(\cdot)\textbf{K}_{\textbf{U}}(\textbf{W}(\textbf{X}))^\top\Vert_2 \leq \Vert \textbf{W}(\cdot)\Vert_2 \Vert\textbf{K}_{\textbf{U}}\Vert_2 \Vert\textbf{W}(\textbf{X})\Vert_2
$$
We now bound each term.
1. **Bounding $\Vert \textbf{W}(\cdot)\Vert_2$ and $\Vert \textbf{W}(\textbf{X})\Vert_2$:**
Since the spectral norm is induced by the Euclidean norm, and using the assumption that the sum of absolute values of interpolation weights for any point is bounded by $c^d$, we have
$$\Vert \textbf{W}(\cdot)\Vert_2 \leq \sqrt{\Vert \textbf{W}(\cdot)\Vert_1 \Vert \textbf{W}(\cdot)\Vert_\infty} \leq \sqrt{T c^d \cdot c^d} = \sqrt{T} c^d.$$
Similarly, $\Vert\textbf{W}(\textbf{X})\Vert_2 \leq \sqrt{n}c^d$.
2. **Bounding $\Vert\textbf{K}_{\textbf{U}}\Vert_2$:**
Since $\textbf{K}_{\textbf{U}}$ is symmetric, $\Vert \textbf{K}_{\textbf{U}} \Vert_2 \leq \Vert \textbf{K}_{\textbf{U}} \Vert_\infty$. Each entry of $\textbf{K}_{\textbf{U}}$ is bounded by $M$ (by the boundedness of $k$), and each row has $m$ entries, so $\Vert \textbf{K}_{\textbf{U}} \Vert_\infty \leq mM$. Thus, $\Vert\textbf{K}_{\textbf{U}}\Vert_2 \leq mM$.
Combining these bounds, we get:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq (\sqrt{T} c^d) (mM) (\sqrt{n} c^d) = \sqrt{Tn} m c^{2d} M
$$
as required.
\end{proof}
% \amnote{might get rid of this}
\begin{lemma}\label{lemma:action-inverse-error}
Let \(\mathbf{\tilde{K}}\) be the SKI approximation of the kernel matrix \(\mathbf{K}\), and let \(\sigma^2\) be the regularization parameter. The spectral error of the regularized inverse can be bounded as follows:
\begin{align*}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} \right\|_2 &\leq \frac{\gamma_{n, m, L}}{\sigma^4}
% \\
% &=\frac{O(n\sqrt{L}\rho^{-L})}{\sigma^4}
\end{align*}
\end{lemma}
\begin{proof}
Note that
\[
\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} = \left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} (\mathbf{K} - \mathbf{\tilde{K}}) \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}
\]
Taking the spectral norm, we have
\[
\begin{aligned}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 &\leq \left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \|\mathbf{K} - \mathbf{\tilde{K}}\|_2 \left\|\left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \\
&\leq \gamma_{n, m, L}\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \left\|\left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2\textrm{ by Proposition \ref{prop:spectral-norm}}\\
&\leq \frac{\gamma_{n, m, L}}{\sigma^4}
\end{aligned}
\]
\end{proof}
%%%%%%%% ICML 2021 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{thmtools}
\usepackage{thm-restate}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2021} with \usepackage[nohyperref]{icml2021} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\newcommand{\jmei}[1]{\textcolor{magenta}{#1}}
\newcommand{\amnote}[1]{\textcolor{red}{#1}}
% Use the following line for the initial blind version submitted for review:
\newtheorem{claim}{Claim}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem{corollary}{Corollary}
\newtheorem{proposition}{Proposition}
\newtheorem{assumption}{Assumption}
\newtheorem{theorem}{Theorem}
\usepackage{icml2021}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{icml2021}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Error Bounds for Structured Kernel Interpolation}% via Multivariate Polynomial Interpolation Analysis}
\begin{document}
\twocolumn[
\icmltitle{Error Bounds for Structured Kernel Interpolation}% via Multivariate Polynomial Interpolation Analysis}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2021
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Aeiau Zzzz}{equal,to}
\icmlauthor{Bauiu C.~Yyyy}{equal,to,goo}
\icmlauthor{Cieua Vvvvv}{goo}
\icmlauthor{Iaesut Saoeu}{ed}
\icmlauthor{Fiuea Rrrr}{to}
\icmlauthor{Tateu H.~Yasehe}{ed,to,goo}
\icmlauthor{Aaoeu Iasoh}{goo}
\icmlauthor{Buiui Eueu}{ed}
\icmlauthor{Aeuia Zzzz}{ed}
\icmlauthor{Bieea C.~Yyyy}{to,goo}
\icmlauthor{Teoau Xxxx}{ed}
\icmlauthor{Eee Pppp}{ed}
\end{icmlauthorlist}
\icmlaffiliation{to}{Department of Computation, University of Torontoland, Torontoland, Canada}
\icmlaffiliation{goo}{Googol ShallowMind, New London, Michigan, USA}
\icmlaffiliation{ed}{School of Computation, University of Edenborrow, Edenborrow, United Kingdom}
\icmlcorrespondingauthor{Cieua Vvvvv}{c.vvvvv@googol.com}
\icmlcorrespondingauthor{Eee Pppp}{ep@eden.co.uk}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\onecolumn
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
%\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
\printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
\begin{abstract}
Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} helps scale Gaussian Processes (GPs) to large datasets by approximating the kernel matrix via interpolation at inducing points, achieving linear computational complexity. However, SKI lacks rigorous theoretical error analysis. This paper addresses this gap: we establish error bounds for the SKI kernel and its Gram matrix, examine their effect on hyperparameter estimation, and evaluate the resulting error in posterior means and variances. Crucially, using convolutional cubic interpolation, we identify two dimensionality regimes governing the trade-off between SKI Gram matrix spectral norm error and computational complexity. For $d \leq 3$, linear time is achievable for \textit{any} error tolerance, provided the sample size is sufficiently large. For $d > 3$, the error must \textit{increase} with sample size to maintain linear time. Our analysis provides key insights into SKI's scalability-accuracy trade-offs, establishing precise conditions for achieving linear-time GP inference with controlled approximation error for the Gram matrix, optimization and posterior inference.
% Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} helps scale Gaussian Processes (GPs) to large datasets by approximating the kernel matrix via interpolation at inducing points, leading to a computational complexity of $O(n+m\log m)$, where $n$ is the sample size and $m$ is the number of inducing points. Despite its widespread use, SKI lacks rigorous theoretical error analysis. In particular, for a given error tolerance for the approximate kernel matrix, how many inducing points $m$ do we need, and will $m\log m=O(n)$ hold so that the computational complexity is linear as claimed? This paper addresses this gap by deriving error bounds for the SKI kernel approximation and analyzing the impact on GP training and inference. We: (1) establish error bounds for the SKI kernel and its gram matrix; (2) examine how these errors affect kernel hyperparameter estimation; (3) evaluate the error of SKI GP posterior means and variances. Interestingly, when using convolutional cubic interpolation, we identify two non-obvious dimensionality regimes relating SKI gram matrix spectral norm error to computational complexity. For $d\leq 3$, as long as the sample size is sufficiently large, we maintain linear time for \textit{any} error tolerance. For $d>3$, the error must \textit{increase} with the sample size to maintain our guarantee of linear time. Our theoretical analysis thus provides crucial insights into the scalability-accuracy trade-offs of SKI and how they change with the feature dimensionality, establishing precise conditions under which linear-time GP inference can be achieved while maintaining controlled approximation error.
% \jmei{It looks like the regime boundary in general would end up being $d\leq L-1$, so maybe we could further explicitly link polynomial degree to the dimensionality, error, and complexity. Not sure if there's a generalization of Keys' result that leys us easily do this though.}
\end{abstract}
\section{Introduction}\label{sec:introduction}
\input{body/01intro}
\section{Related Work}\label{sec:related}
\input{body/02related}
\section{Gaussian Processes, Structured Kernel Interpolation and Convolutional Cubic Interpolation}\label{sec:ski-background}
\input{body/03background}
\section{Important Quantities}\label{sec:important-quantities}
\input{body/04important-quantities}
\section{Gaussian Processes Applications}\label{sec:gp-applications}
\input{body/05gp-applications}
\section{Discussion}\label{sec:discussion}
In this paper, we provided theoretical analysis for structured kernel interpolation. In particular, we analyzed the error of tensor-product cubic convolutional interpolation, showed the elementwise SKI kernel error and how that propagates to spectral norm error of the SKI gram and cross-kernel matrices, and showed how this impacts achieving a specific error in linear time. We then analyzed kernel hyperparameter estimation in Gaussian processes, showing that gradient ascent has an iterate approach a ball around a stationary point with size quadratic in the sample size.
\bibliography{main}
\bibliographystyle{icml2021}
\appendix
\section{Auxiliary Technical Results}
\input{body/appendices/01auxiliary}
\section{Proofs Related to Important Quantities}
\input{body/appendices/04important-quantities}
\section{Proofs Related to Gaussian Process Applications}
\input{body/appendices/05gp-applications}
\end{document}
%%%%%%%% ICML 2021 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2021} with \usepackage[nohyperref]{icml2021} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\newcommand{\jmei}[1]{\textcolor{magenta}{#1}}
\newcommand{\amnote}[1]{\textcolor{red}{#1}}
% Use the following line for the initial blind version submitted for review:
\newtheorem{claim}{Claim}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem{corollary}{Corollary}
\newtheorem{proposition}{Proposition}
\newtheorem{theorem}{Theorem}
\usepackage{icml2021}
\newtheorem{assumption}{Assumption}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{icml2021}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Error Bounds for Structured Kernel Interpolation via Multivariate Polynomial Interpolation Analysis}
\begin{document}
\twocolumn[
\icmltitle{Error Bounds for Structured Kernel Interpolation via Multivariate Polynomial Interpolation Analysis}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2021
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Aeiau Zzzz}{equal,to}
\icmlauthor{Bauiu C.~Yyyy}{equal,to,goo}
\icmlauthor{Cieua Vvvvv}{goo}
\icmlauthor{Iaesut Saoeu}{ed}
\icmlauthor{Fiuea Rrrr}{to}
\icmlauthor{Tateu H.~Yasehe}{ed,to,goo}
\icmlauthor{Aaoeu Iasoh}{goo}
\icmlauthor{Buiui Eueu}{ed}
\icmlauthor{Aeuia Zzzz}{ed}
\icmlauthor{Bieea C.~Yyyy}{to,goo}
\icmlauthor{Teoau Xxxx}{ed}
\icmlauthor{Eee Pppp}{ed}
\end{icmlauthorlist}
\icmlaffiliation{to}{Department of Computation, University of Torontoland, Torontoland, Canada}
\icmlaffiliation{goo}{Googol ShallowMind, New London, Michigan, USA}
\icmlaffiliation{ed}{School of Computation, University of Edenborrow, Edenborrow, United Kingdom}
\icmlcorrespondingauthor{Cieua Vvvvv}{c.vvvvv@googol.com}
\icmlcorrespondingauthor{Eee Pppp}{ep@eden.co.uk}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\onecolumn
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
%\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
\printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
\begin{abstract}
Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} is helps scale Gaussian Processes (GPs) to large datasets by approximating the kernel matrix via interpolation at inducing points, leading to a computational complexity of $O(n+m\log m)$, where $n$ is the sample size and $m$ is the number of inducing points. Despite its widespread use and effectiveness, SKI lacks rigorous theoretical error analysis. In particular, for a given error tolerance for the approximate kernel matrix, how many inducing points $m$ do we need, and will $O(m\log m)\leq O(n)$ hold so that the computational complexity is linear as claimed? This paper addresses this gap by deriving error bounds for the SKI kernel approximation and analyzing the impact on GP training and inference. We: (1) establish error bounds for the SKI kernel and its gram matrix; (2) examine how these errors affect maximum likelihood estimation of mean function and kernel hyperparameters; (3) evaluate the error of SKI GP posterior means and variances. Interestingly, we identify two dimensionality regimes relating SKI gram matrix error to computational complexity. For $d\leq 3$, as long as the sample size grows as we want to \textit{decrease} error, we maintain linear time. For $d>3$, the error must \textit{increase} with the sample size to maintain linear time. Our theoretical analysis thus provides crucial insights into the scalability-accuracy trade-offs of SKI and how they change with the feature dimensionality, establishing precise conditions under which linear-time GP inference can be achieved while maintaining controlled approximation error.
%find that for \textit{any} error tolerance for the SKI gram matrix, $O(m\log m)\leq O(n)$ as long as the observation dimension $d\leq 3$ and the sample size is sufficiently large, while for $d>3$ the error must grow with the sample size in order to maintain a linear time algorithm.
% We validate both our assumptions and finding in a set of simulation studies.
% Our findings aim to increase confidence in using SKI among practitioners by providing a theoretical foundation for its application in Gaussian process inference and hyperparameter estimation.
% % Structured kernel interpolation \citep{wilson2015kernel}, which approximates a gram matrix by interpolating the kernel values at a set of inducing points, has substantially advanced the scalability of Gaussian processes (GPs) and is a core algorithm in one of the primary GP libraries \cite{gardner2018gpytorch}. Despite its effectiveness and popularity, it currently lacks theoretical error analysis. In this paper, we derive error bounds (describe how). We find that the error decays as ... with inducing points on a fixed grid, which is an improvement over the Nystr{\"o}m method, which has .... We hope that these bounds can inspire confidence in the use of these methods among practitioners. (Maybe something about regularized empirical risk minimization using it instead of Nystr{\"o}m?)
\end{abstract}
% \begin{align*}
% \Tilde{k}(x_i,u_j)&=w_ik(u_a,u_j)+(1-w_i)k(u_b,u_j)\\
% &=\frac{u_b-x_i}{u_b-u_a}k(u_a,u_j)+\frac{x_i-u_a}{u_b-u_a}k(u_b,u_j)
% \end{align*}
\section{Introduction}\label{sec:introduction}
Gaussian Processes (GPs) are an important class of stochastic processes used in machine learning and statistics, with use cases including spatial data analysis \citep{liu2021missing}, time series forecasting \citep{girard2002gaussian} and bioinformatics \citep{luo2023diseasegps}. GPs offer a non-parametric framework for modeling distributions over functions, enabling both flexibility and uncertainty quantification. These capabilities, combined with the ability to incorporate prior knowledge and specify relationships by choice of kernel function, make Gaussian Processes effective for regression, classification, and optimization.
However, GPs have substantial computational and memory bottlenecks. Both training and inference require computing the action of the inverse kernel gram matrix, while training requires computing its log-determinant: both are $O(n^3)$ operations, where $n$ is the sample size. Further, storing the full gram matrix requires $O(n^2)$ memory. These bottlenecks require scalable approximations to extend Gaussian Processes to larger datasets.
Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} is an efficient method for scaling Gaussian Processes (GPs) to large datasets by approximating the kernel matrix using interpolation on a set of inducing points. For stationary kernels, this requires $O(n+m \log m)$ computational complexity. The core idea is to express the original kernel as a combination of interpolation functions and a kernel matrix defined on a set of inducing points. However, despite its effectiveness, popularity (over $600$ citations, a large number for a GP paper) and high quality software availability (\cite{gardner2018gpytorch} has 3.5k stars on github), it currently lacks theoretical analysis. A key initial question is, given a fixed error bound for the SKI gram matrix and their use of cubic interpolation, how many inducing points are required to achieve that error bound? Given the required value of $m$ as a function of $n$, for what error tolerance is $O(n+m\log m)$ still linear? Following this, how do these errors propagate to the SKI log-likelihood and posterior inference?
In this paper, we begin to bridge the gap between practice and a theoretical understanding of SKI. We have three primary contributions: 1) error bounds for the SKI kernel and relevant quantities, including the spectral norm of the difference between the true and SKI gram matrices. 2) SKI estimation analysis: maximum likelihood estimation (MLE) of kernel hyperparameters assuming a good initialization. 3) SKI inference analysis: the error of the GP posterior means and variances at test points. Interestingly, we identify two dimensionality regimes relating SKI gram matrix error to computational complexity. For $d\leq 3$, as long as the sample size grows as we want to \textit{decrease} error, we maintain linear time. For $d>3$, the error must \textit{increase} with the sample size to maintain linear time. By our downstream analysis, this implies that for small dimensionality and kernels exhibiting desirable regularity conditions, we can have arbitrarily small error in the SKI gram matrix, in our estimated parameters and in posterior inference all in linear time as long as the sample size is sufficiently large.
%Interestingly, we find that for \textit{any} error tolerance for the SKI gram matrix, $O(m\log m)\leq O(n)$ as long as the observation dimension $d\leq 3$ and the sample size is sufficiently large, while for $d>3$ the error must grow with the sample size in order to maintain a linear time algorithm.
% We do this both assuming the same kernel hyperparameters for both, and looking at the error propagation of incorrect MLEs for SKI.\amnote{One thing that may be worth looking at: for some specific case (e.g. RBF kernel), how do hyperparameters affect eigenvalues of kernel, how do eigenvalues of kernel affect eigenvalues of gram matrix?}
% This paper overcomes two key technical challenges. First, we extend Keys' univariate interpolation error bounds to the multivariate tensor-product setting, requiring a novel inductive proof to handle the compounding errors across dimensions, unlike standard polynomial analysis. Second, because theoretical work for SKI was lacking, in order to analyze the SKI MLE and posterior we must provide a comprehensive, end-to-end analysis that directly links the elementwise kernel approximation errors to their downstream impact on both the maximum likelihood estimate (MLE) and the GP posterior, even when using a potentially `wrong' MLE obtained under the SKI approximation. This involves a novel multi-stage analysis: sequentially bounding the difference in log-likelihoods, then the MLEs using strong concavity arguments, and finally the posterior mean and variance – to connect the elementwise kernel errors to the learned parameters and distributions. This contrasts with prior work on GP approximations\amnote{cite}, which often focuses on algorithm-specific bounds or kernel approximation accuracy in isolation, without a complete connection to the final posterior inference. Our work's novelty lies in the derived error bounds for multivariate interpolation and, more importantly, in the comprehensive, multi-stage analysis that reveals the full impact of SKI's approximation on GP inference.\amnote{there is one more thing, we're going to use optimization techniques (local strong concavity, ) to rigorously handle the MLE approximation error}
\amnote{Fix references}In section \ref{sec:related} we describe related work. In section \ref{sec:ski-background} we give a brief background on SKI. In section \ref{sec:important-quantities} we bound the error of important quantities: specifically the SKI kernel error, the SKI gram matrix error and the error of the SKI inverse's action and the SKI log-determinant. In section \ref{sec:gp-applications} we use these to analyze the error of the SKI MLE and the KL divergence, total variation and Wasserstein distances between the true and SKI predictive posteriors. We conclude in section \ref{sec:discussion} by summarizing our results and discussing limitations and future work.
\section{Related Work}\label{sec:related}
We can divide related works into three groups: those theoretically analyzing Gaussian process regression or kernel methods when using approximate kernels, SKI and its extensions, and papers developing techniques we use to obtain our guarantees. In the first group, the most relevant works are \cite{burt2019rates,burt2020convergence}, where they analyzed the sparse variational GP framework and derived bounds on the Kullback-Leibler divergence between the true posterior and the variational approximate posterior. We obtain analogous results when using the SKI kernel. \cite{moreno2023ski} gave bounds on the approximation error of the SKI Gram matrix. However, they only handled the case of univariate features and only bounded how much worse the SKI Gram matrix can be than the Nystr{\"o}m one. Further, they did not analyze the downstream effects on the approximate MLE or GP posterior. Also relevant are \cite{wynne2022variational,wild2021connections}, who gave a Banach space view of sparse variational GPs and connected them to the Nystr{\"o}m method, respectively. We briefly discuss and compare to the Nystr{\"o}m method based on their results, while leaving a full Banach space analysis to future work.
%\amnote{Might want to look at methods that improve dependence on $d$.}
%, while existing empirical results \cite{wilson2015kernel} show that the resulting approximations are often \textit{better} than the Nystr{\"o}m ones.
% \section{Strategy}
% The easiest approach is probably to do something just for GPs so that we don't have to go deep into RKHS theory. Maybe like \cite{burt2019rates} but for SKI. Also \cite{wild2021connections} looks at connections between sparse variational GPs and Nystr{\"o}m method. \cite{wynne2022variational} also looks interesting: looks at sparse variational GPs from the functional analysis perspective.
% \begin{enumerate}
% \item Univariate
% \begin{enumerate}
% \item Stationary, linear interpolation
% \item Stationary, polynomial interpolation
% \item Nonstationary, linear interpolation
% \item Nonstationary, polynomial interpolation% interpolation \jmei{[[doubled?]]}
% \end{enumerate}
% \item Multivariate: might be harder, use \cite{sauer1995multivariate} eqn 3.18.
% \item Chebyshev's
% \item Matrix version, we don't want just want at a single point, we want norm of difference between matrices.
% \item How does this compare to Nystr{\"o}m?
% \item GPs: actually interested not in the error of the kernel itself, but in
% \begin{enumerate}
% \item action of inverse
% \item log determinant of regularized kernel.
% \item Distance between mean/covariance matrix of GP?
% \item how far away predictive distribution is from true? e.g. Multivariate Gaussian under true vs SKI: bound KL divergence, TVD, etc.
% \item How far away are true MLE and SKI MLE?
% \item Error propagation: say we get SKI MLE, how does this affect mean/covariance matrix under 'wrong' MLE, then how does this affect predictive distribution?
% \item Error of mean (which is KRR) in RKHS norm?
% \end{enumerate}
% \item Stationary implies Toeplitz inducing point Gram matrix: how does that affect bound?
% \item (Sprinkled throughout the paper), special case results for given kernels like Gaussian kernel.
% \item Potentially:
% \begin{itemize}
% \item how does using this approximation affect regularized empirical risk minimization for kernel methods e.g. kernel ridge regression?
% \item How does it affect generalization error? E.g. look at \cite{chen2016error}.
% \item asymmetric setting?
% \end{itemize}
% \end{enumerate}
% \amnote{Something meaningful, not on a grid, on a manifold. High enough dimensions, certain regularity assumptions, distribution of data points w.h.p they all land on a lower dimensional manifold, doing a grid }
\section{Gaussian Processes, Structured Kernel Interpolation and Convolutional Cubic Interpolation}\label{sec:ski-background}
\subsection{Gaussian Processes}
A Gaussian process $\xi\sim \textrm{GP}(\nu,k_\theta)$ is a stochastic process $\{\xi(\textbf{x})\}_{\textbf{x}\in \mathcal{X}}$ such that any finite subcollection $\{\xi(\textbf{x}_i)\}_{i=1}^n$ is multivariate Gaussian distributed. In machine learning, we generally assume that we have index locations $\textbf{x}_i\in \mathbb{R}^d$ and observations $y_i\in \mathbb{R}$ for a set of training points $i=1,\ldots,n$ such that
\begin{align*}
y_i&=\xi(\textbf{x}_i)+\epsilon_i, \qquad \epsilon_i\sim \mathcal{N}(0,\sigma^2),
\end{align*}
where $\nu:\mathcal{X}\rightarrow\mathbb{R}$, $k_\theta:\mathcal{X}\times \mathcal{X}\rightarrow\mathbb{R}$ are the prior mean and covariance functions, respectively, with $\nu$ the mean function generally assumed to be $0$ everywhere and $k$ an SPD kernel with hyperparameters $\theta$. Given $\{\textbf{x}_i,y_i\}_{i=1}^n$ we are primariliy interested in two tasks: 1) estimate the kernel hyperparameters, given an assumed form for $k_\theta$ (e.g. RBF kernel) 2) do Bayesian inference for the posterior mean $\boldsymbol{\mu}(\cdot)\in \mathbb{R}^{T}$ and covariance $\boldsymbol{\Sigma}(\cdot)\in \mathbb{R}^{T\times T}$ at a set of test points $\{\textbf{x}_t\}_{t=1}^T$. Assuming $\nu\equiv 0$ (a mean-zero GP prior), for 1), one maximizes the log-likelihood
\begin{align}
\mathcal{L}(\theta;X) = -\frac{1}{2}\textbf{y}^\top (\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}-\frac{1}{2}\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert -\frac{N}{2}\log (2\pi)\label{eqn:log-likelihood}
\end{align}
where $\textbf{K}\in \mathbb{R}^{n\times n}$ with entries $\textbf{K}_{ij}=k_\theta(\textbf{x}_i,\textbf{x}_j)$ is the Gram matrix for the training dataset. For 2), given the kernel function and known observation variance $\sigma^2$, the posterior mean and covariance are given by
\begin{align}
\boldsymbol{\mu}(\cdot) &= \mathbf{K}_{\cdot, \mathbf{X}}\left(\mathbf{K}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\label{eqn:posterior-mean}\\
\boldsymbol{\Sigma}(\cdot) &= \textbf{K}_{\cdot,\cdot}+\sigma^2I-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}\label{eqn:posterior-covariance}
\end{align}
where $\textbf{K}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ is the matrix of kernel evaluations between test and training points. However, between the log-likelihood and the posteriors, one first needs to compute the action of the inverse of the regularized Gram matrix, $(\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}$. Second, one needs to compute the log-determinant $\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert$. These are both $O(n^3)$ computationally and $O(n^2)$ memory.
\subsection{Structured Kernel Interpolation}
Structured kernel interpolation \citep{wilson2015kernel} or (SKI) addresses these computational and memory bottlenecks by approximating the original kernel function \(k_\theta:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R},\mathcal{X}\subseteq\mathbb{R}^d\) by interpolating kernel values at a chosen set of inducing points \(\mathbf{U}=\left(\begin{array}{c}
\textbf{u}_1^\top\\
\vdots\\
\textbf{u}_m^\top
\end{array}\right)\in \mathbb{R}^{m\times d}\). The approximate kernel function \(\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow\mathbb{R}\) can be expressed as:
\[
\tilde{k}(\mathbf{x}, \mathbf{x}') = \mathbf{w}(\mathbf{x})^{\top} \mathbf{K}_{\mathbf{U}} \mathbf{w}(\mathbf{x}')
\]
where \(\mathbf{K}_{\mathbf{U}}\in \mathbb{R}^{m\times m}\) is the kernel matrix computed on the inducing points, and \(\mathbf{w}(\mathbf{x}),\mathbf{w}(\mathbf{x}')\in \mathbb{R}^m\) are vectors of interpolation weights using (usually cubic) convolutional kernel $u:\mathbb{R}\rightarrow \mathbb{R}$ for the points $\textbf{x}$ and $\textbf{x}'$, respectively. One can then form the SKI Gram matrix $\tilde{\textbf{K}}=\textbf{W}\textbf{K}_\textbf{U}\textbf{W}^\top$ with $\textbf{W}$ a \textit{sparse} matrix of $L$ interpolation weights per row for a polynomial of degree $L-1$. By exploiting the sparsity of each row, for stationary kernels this leads to a computational complexity of $O(nL+m\log m)$ and a memory complexity of $O(nL+m)$.
% \amnote{say something about how this affect computational complexity.}
% \jmei{This naively results in a storage complexity of $O(mn)$ and computational complexity of $O(m^2 n)$ for applying $(\tilde{\mathbf{K}}+\sigma^2\mathbf{I})^{-1}$ e.g. using the Woodbury matrix identity. These can be further reduced due to sparsity in $\mathbf{w}$.}
In order to learn kernel hyperparameters, one can maximize the SKI approximation to the log-likelihood (henceforth the SKI log-likelihood)
\begin{align*}
\tilde{\mathcal{L}}(\theta;X)
&=-\frac{1}{2}\textbf{y}^\top (\tilde{\textbf{K}}+\sigma^2 \textbf{I})^{-1}\textbf{y}-\frac{1}{2}\log \vert \tilde{\textbf{K}}+\sigma^2 \textbf{I}\vert -\frac{N}{2}\log (2\pi)
\end{align*}
Given the SKI kernel $\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ with learned hyperparameters, one can do posterior inference of the SKI approximations to the mean $\tilde{\boldsymbol{\mu}}(\cdot)$ and covariance $\tilde{\boldsymbol{\Sigma}}(\cdot)$ at a set of $T$ test points $\cdot$ as
\begin{align*}
\tilde{\boldsymbol{\mu}}(\cdot)&=\tilde{\mathbf{K}}_{\cdot, \mathbf{X}}\left(\tilde{\mathbf{K}}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\\
\tilde{\boldsymbol{\Sigma}}(\cdot)&=\tilde{\textbf{K}}_{\cdot,\cdot}+\sigma^2I-\tilde{\textbf{K}}_{\cdot,\textbf{X}}^\top (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}
\end{align*}
where $\tilde{\textbf{K}}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ is the matrix of SKI kernels between test points and training points and $\tilde{\textbf{K}}_{\cdot,\cdot}\in \mathbb{R}^{T\times T}$ is the SKI Gram matrix for the test points.
\subsection{Convolutional Cubic Interpolation}
\begin{definition}\label{def:cubic-interpolation-kernel}
The cubic convolutional interpolation kernel $u:\mathbb{R}\rightarrow\mathbb{R}$ is given by
$$
u(s)\equiv\begin{cases}
1, & s=0\\
\frac{3}{2}\vert s\vert^3-\frac{5}{2}\vert s\vert^2+1, & 0<\vert s\vert <1\\
-\frac{1}{2}\vert s\vert^3+\frac{5}{2}\vert s\vert^2-4\vert s\vert+2, & 1<\vert s\vert<2\\
0, & \text{otherwise}
\end{cases}
$$
\end{definition}
\begin{definition}\label{def:tensor-product-cubic-interpolation}
\textcolor{blue}{JX: Notation suggestion below:}
Let $\vec{\textnormal{\textbf{x}}}=\langle x_1, ..., x_d\rangle\in\mathbb{R}^d$ be a $d$-dimensional point. Let $f:\mathbb{R}^d\rightarrow\mathbb{R}$ be a function defined on a regular grid with spacing $h$ in each dimension. Let $\vec{\textnormal{\textbf{c}}_\textnormal{\textbf{x}}}$ denote the grid point closest to $\vec{\textnormal{\textbf{x}}}$. The \textbf{tensor-product cubic convolutional interpolation function} $g:\mathbb{R}^d\rightarrow \mathbb{R}$ is defined as:
$$g(\vec{\textnormal{\textbf{x}}}) \equiv \sum_{k_1=-1}^2\ldots \sum_{k_d=-1}^2 f\left(\vec{\textnormal{\textbf{c}}_\textnormal{\textbf{x}}}+h\cdot\langle k_1,\dots,k_d\rangle\right) \prod_{j=1}^d u\left(\frac{x_j-\left(\left(\vec{\textnormal{\textbf{c}}_\textnormal{\textbf{x}}}\right)_j + h\cdot k_j\right)}{h}\right)$$
where $u$ is the cubic convolutional interpolation kernel.
\textcolor{blue}{JX: Notation suggestion ends here. Previous notation below:}
Let $x = (x^{(1)}, x^{(2)}, ..., x^{(d)}) \in \mathbb{R}^d$ be a d-dimensional point. Let $f:\mathbb{R}^d \rightarrow \mathbb{R}$ be a function defined on a regular grid with spacing $h$ in each dimension. Let $x_i$ denote the grid point closest to $x$. The \textbf{tensor-product cubic convolutional interpolation function} $g:\mathbb{R}^d\rightarrow \mathbb{R}$ is defined as:
$$
g(x) \equiv \sum_{k_1=-1}^2\ldots \sum_{k_d=-1}^2 f(x_i^{(1)} + k_1 h,\ldots,x_i^{(d)} + k_d h) \prod_{j=1}^d u\left(\frac{x^{(j)}-(x_i^{(j)} + k_j h)}{h}\right)
$$
where $u$ is the cubic convolutional interpolation kernel.
\end{definition}
% Structured Kernel Interpolation (SKI) is an efficient method for scaling Gaussian Processes (GPs) to large datasets by approximating the kernel matrix using interpolation on a structured grid. The core idea is to express the original kernel \( k(\mathbf{x}, \mathbf{x}') \) as a combination of interpolation functions and a structured kernel matrix \( \mathbf{K}_{\textbf{U}} \) defined on a set of inducing points. Specifically, the kernel can be approximated as
% \begin{align*}
% k(\mathbf{x}, \mathbf{x}') &\approx \tilde{k}(\mathbf{x}, \mathbf{x}')\\
% &= \boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}'),
% \end{align*}
% where \( \boldsymbol{w}(\mathbf{x}) \) and \( \boldsymbol{w}(\mathbf{x}') \) are interpolation vectors that map the input locations \( \mathbf{x} \) and \( \mathbf{x}' \) to the grid points. This structured representation leverages the Kronecker and Toeplitz properties of \( \mathbf{K}_{\textbf{U}} \), enabling efficient matrix operations such as inversion and multiplication through fast Fourier transforms (FFT) or other specialized algorithms.
% By utilizing SKI, the computational complexity of GP inference, which is typically \( \mathcal{O}(n^3) \) for \( n \) data points, can be reduced substantially, often to near-linear time with respect to \( n \). This makes SKI particularly suitable for large-scale applications where maintaining the fidelity of the kernel while ensuring computational tractability is crucial. Additionally, the interpolation framework allows for flexibility in choosing different interpolation schemes, further enhancing the method's applicability across various domains.
\section{Important Quantities}\label{sec:important-quantities}
This section focuses on deriving bounds for several key quantities that arise in the analysis of Structured Kernel Interpolation (SKI). These quantities play a crucial role in understanding the accuracy and efficiency of the SKI approximation and will be used later to analyze the downstream effects on Gaussian process inference. We begin in Subsection \ref{subsec:polynomial-interpolation-preliminaries} by establishing some preliminaries on multivariate polynomial interpolation, specifically focusing on tensor-product cubic convolutional interpolation. This includes formal lemmas that characterize the error behavior of this type of interpolation. In Subsubsection \ref{subsubsec:elementwise-error}, we derive a bound on the elementwise error between the true kernel and its SKI approximation. This bound is fundamental, as it directly quantifies the accuracy of the basic building block of the SKI method. Building upon this, Subsection \ref{subsec:spectral-norm-error} transitions from elementwise error to a bound on the spectral norm of the difference between the true Gram matrix and its SKI approximation. The spectral norm is particularly important because it governs the stability of matrix operations and provides an overall measure of the approximation error. Furthermore, we present a bound on the spectral norm of the SKI approximation of the kernel matrix between test and training points, which will be useful later when analyzing the error of the predictive mean and variance in Gaussian process regression. Finally, we also include a sufficient condition for the number of inducing points required to achieve a certain error tolerance $\epsilon$ and a theorem that gives a necessary condition on the error for linear time computational complexity, finding that for $d\leq 3$ the algorithm is \textit{always} linear time for sufficiently large sample sizes, regardless of desired accuracy. These results are essential for understanding the trade-off between accuracy and computational efficiency in SKI.
% \subsection{New Version: Local Polynomial Interpolation Error}
% Idea
% \begin{itemize}
% \item Using constraints for interpolation function and interpolation kernel, set up underdetermined linear system
% \item Fix one variable to unknown value $a$
% \item Express
% \end{itemize}
% \begin{itemize}
% \item Univariate
% \begin{itemize}
% \item Splines
% \item KNN
% \item Lebesgue Constant
% \end{itemize}
% \item Multivariate
% \begin{itemize}
% \item Splines
% \item KNN
% \item Lebesgue Constant
% \end{itemize}
% \end{itemize}
% \subsection{Old Version: Chebyshev Polynomial Interpolation, Multivariate}
% We first bound the absolute difference between the true and SKI kernels, respectively. We then extend that to a bound on the spectral norm of the SKI gram matrix error. Next, we show an error bound for the action of the regularized inverse of the SKI kernel, and finally show a similar result for the SKI log determinant.
% \subsection{Error of Multivariate Polynomial Interpolation}
% \amnote{To do: Justin}
% Describe \cite{Khoromskij2007}. Note that they have a parameter $\rho$ that depends on region of analytic extension. It may be worth saying something about $\rho$ for functions in RKHS under different kernel smoothness conditions, and more generally for functions with different smoothness conditions. Essentially, we'd like $\rho$ to be large. Can we argue that their assumption 3.1 holds for functions that we care about?
% \textcolor{blue}{For \textbf{radial basis functions} $g(\textbf{x}_i,\textbf{x}_j)=\hat{g}(|\textbf{x}_i-\textbf{x}_j|)$ where $\hat{g}:\mathbb{R}\to\mathbb{R}$ we consider the popular example $\hat{g}(r)=\exp\left(-\frac{r^2}{2\ell^2}\right)$ given in the KISS-GP paper. This can be analytically extended to domain $\mathbb{C}$. Clearly $\hat{g}(z)=\hat{g}(|z|)$ for any $z\in\mathbb{C}$ (as $\hat{g}$ is simply a standard Gaussian distribution). Then the maximum of $\hat{g}$ over $\mathcal{E}_\rho$ is simply $\hat{g}(0)=1$.}
% % taken over the rightmost point of the ellipse $\frac{1}{2}\left(\rho+\rho^{-1}\right)$, namely $\exp\left(-\frac{\left(\frac{1}{2}\left(\rho+\rho^{-1}\right)\right)^2}{2\ell^2}\right)$. Then we find $M_\rho(g)=$}
% We start by restating a theorem from \cite{Khoromskij2007} \amnote{not yet super rigorously restated }describing the error of multivariate polynomial interpolation. Let $g:\mathcal{X}\rightarrow \mathbb{R}$ be the target function and $\tilde{g}:\mathcal{X}\rightarrow \mathbb{R}$ be its multivariate polynomial interpolant using tensor-product interpolants. Let $M_j>0$ be an upper bound on the analytic extension in $\mathcal{E}_\rho(B_j),\rho>1$ and $M_\rho>0$ be an upper bound on those $M_j$s. Then
% \begin{align*}
% \Vert g-\tilde{g}\Vert_{\infty,T}&\leq c^d\frac{2M_\rho(g)}{\rho-1}\rho^{-L}
% \end{align*}
% \amnote{What if we have $m$ inducing points (potential interpolation points), but instead of using all of them, we only use the closest $L$. How far are we from the result using all of them? Essentially we are doing selective Chebyshev rather than full Chebyshev, local rather than full.}
\subsection{Preliminaries for Polynomial Interpolation}\label{subsec:polynomial-interpolation-preliminaries}
This subsection lays the groundwork for analyzing the error of tensor-product cubic convolutional interpolation, which is a key component in the SKI framework. We introduce establish some important properties that will be used in subsequent sections. Specifically, we define the cubic convolutional interpolation kernel, the tensor-product cubic convolutional interpolation function, and prove a lemma on how its error depends on the grid spacing and the dimensionality of the input space. This extends the analysis of \cite{keys1981cubic} to the multivariate tensor-product interpolation setting. We also present a lemma on how the sum of kernel interpolation weights depends on the number of dimensions, highlighting the curse of dimensionality that arises in tensor-product interpolation.
The next lemma describes how the error of multivariate tensor-product cubic convolutional interpolation follows the $O(h^3)$ error of \cite{keys1981cubic}, but suffers from the curse of dimensionality with exponential error growth in the number of dimensions. The proof proceeds by induction on the number of dimensions. We first establish a base case for one dimension using the known error bound for cubic convolution interpolation in 1D, derived by Keys. Then, assuming the error bound holds for \(k\) dimensions, we show that it also holds for \(k+1\) dimensions by expressing the difference between the \((k+1)\)-dimensional interpolation and the \(k\)-dimensional interpolation as a weighted sum of differences between \(k\)-dimensional interpolations and their corresponding \((k-1)\)-dimensional counterparts. This allows us to bound the error in \(k+1\) dimensions using the inductive hypothesis. Finally, we sum the errors across all dimensions to obtain the total error bound, which involves a geometric series and leads to the final result.
\begin{lemma}\label{lemma:tensor-product-interpolation-error}
Assume that the grid spacing is fixed across all dimensions, denoted by $h$. The error of tensor-product cubic convolutional interpolation is $O(c^dh^3)$ for some constant $c>0$.
\end{lemma}
\begin{proof}
\textcolor{blue}{JX: Notation suggestion below:}
For any vector $\vec{\textbf{v}}=\langle v_1,\dots,v_d\rangle\in\mathbb{R}^d$, index $j=1,\dots,d$ and scalar $\alpha\in\mathbb{R}$, let $\vec{\textbf{v}}_{[j\to\alpha]}:=\langle v_1,\dots,v_{j-1},\alpha,v_{j+1},\dots,v_d\rangle$ be the vector defined by replacing the $j$-th index of $\vec{\textbf{v}}$ with $\alpha$.
We define a sequence of intermediate interpolation functions. Let $g_0(\vec{\textbf{v}})\equiv f(\vec{\textbf{v}})$ be the original function. For $j=1,\dots,d$, we recursively define $g_j(\vec{\textbf{v}})$ as the function obtained by interpolating $g_{j-1}$ along the $j$-th dimension using the cubic convolution kernel $u$:
$$g_j(\vec{\textbf{v}}) \equiv \sum_{k_j=-1}^2 g_{j-1}\left(\vec{\textbf{v}}_{\left[j\to\left(\vec{\textbf{c}_\textbf{v}}\right)_j\right]}+\left(h\cdot k_j\right)\cdot\vec{\textbf{e}}_j\right) u\left(\frac{\vec{\textbf{v}}_j-\left(\left(\vec{\textbf{c}_\textbf{v}}\right)_j+h\cdot k_j\right)}{h}\right),$$
where $\vec{\textbf{c}_\textbf{v}}$ is the grid point closest to $\vec{\textbf{v}}$ and $\vec{\textbf{e}}_j$ is a standard basis vector.
\textcolor{blue}{JX: Notation suggestion ends here. Previous notation below:}
We first define a sequence of intermediate interpolation functions. Let $g_0(x) \equiv f(x)$ be the original function. For $i = 1, \ldots, d$, we recursively define $g_i(x)$ as the function obtained by interpolating $g_{i-1}$ along the $i$-th dimension using the cubic convolution kernel $u$:
$$
g_i(x) \equiv \sum_{k_i=-1}^2 g_{i-1}(x^{(1)}, \ldots, x^{(i-1)}, x_i^{(i)} + k_i h, x^{(i+1)} \ldots, x^{(d)}) u\left(\frac{x^{(i)}-(x_i^{(i)} + k_i h)}{h}\right).
$$
Thus, $g_1(x)$ interpolates $f$ along the first dimension, $g_2(x)$ interpolates $g_1$ along the second dimension, and so on, until $g_d(x) = g(x)$ is the final tensor-product interpolated function.
We analyze the error accumulation across multiple dimensions using induction. Using \cite{keys1981cubic}, the error introduced by interpolating a sufficiently smooth function \amnote{define sufficiently smooth} \textcolor{blue}{JX: sufficiently smooth = continuous 3rd derivative} along a single dimension with the cubic convolution kernel is uniformly bounded over the interval domain by $Kh^3$ for some constant $K > 0$, provided the grid spacing $h$ is sufficiently small\amnote{what is the condition actually for big $O$?}. This gives us the base case:
$$
|g_1(x) - g_0(x)| \leq Kh^3
$$
For the inductive step, assume that for some $i=k$ the error is uniformly bounded by
$$
|g_k(x) - g_{k-1}(x)| \leq c^{k-1}Kh^3.
$$
We want to show that this bound also holds for $i=k+1$. We can express the difference $g_{k+1}(x) - g_k(x)$ as follows:
\begin{align*}
g_{k+1}(x) - g_k(x) &= \sum_{k_{k+1}=-1}^2 g_k(x^{(1)}, \ldots, x^{(k)}, x_i^{(k+1)} + k_{k+1}h, \ldots, x^{(d)}) u\left(\frac{x^{(k+1)} - (x_i^{(k+1)} + k_{k+1}h)}{h}\right) \\
&\quad - g_k(x^{(1)}, \ldots, x^{(d)}) \\
&= \sum_{k_{k+1}=-1}^2 \left[\sum_{k_k=-1}^2 g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x_i^{(k+1)} + k_{k+1}h, \ldots, x^{(d)}) u\left(\frac{x^{(k)} - (x_i^{(k)} + k_k h)}{h}\right)\right] \\
&\quad \cdot u\left(\frac{x^{(k+1)} - (x_i^{(k+1)} + k_{k+1}h)}{h}\right) \\
&\quad - \sum_{k_k=-1}^2 g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)}) u\left(\frac{x^{(k)} - (x_i^{(k)} + k_k h)}{h}\right) \\
&= \sum_{k_k=-1}^2 u\left(\frac{x^{(k)} - (x_i^{(k)} + k_k h)}{h}\right) \left[\sum_{k_{k+1}=-1}^2 g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x_i^{(k+1)} + k_{k+1}h, \ldots, x^{(d)}) \right. \\
&\quad \left. \cdot u\left(\frac{x^{(k+1)} - (x_i^{(k+1)} + k_{k+1}h)}{h}\right) - g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)})\right].
\end{align*}
The inner term in the last expression represents the difference between interpolating $g_{k-1}$ along the $(k+1)$-th dimension and $g_{k-1}$ itself, evaluated at $(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)})$. This can be written as:
\begin{align*}
&\sum_{k_{k+1}=-1}^2 g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x_i^{(k+1)} + k_{k+1}h, \ldots, x^{(d)}) u\left(\frac{x^{(k+1)} - (x_i^{(k+1)} + k_{k+1}h)}{h}\right) \\
&\quad - g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)}) \\
&= g_k(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)}) - g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)}).
\end{align*}
Therefore, we can bound the error as follows:
\begin{align*}
|g_{k+1}(x) - g_k(x)| &\leq \left|\sum_{k_k=-1}^2 u\left(\frac{x^{(k)} - (x_i^{(k)} + k_k h)}{h}\right)\right| \cdot |g_k(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)}) \\
&\quad - g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)})|.
\end{align*}
Let $c>0$ be a uniform upper bound for $\left|\sum_{k_k=-1}^2 u\left(\frac{x^{(k)} - (x_i^{(k)} + k_k h)}{h}\right)\right|$, which exists because $u$ is bounded. By the inductive hypothesis, we have $|g_k(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)}) - g_{k-1}(x^{(1)}, \ldots, x^{(k-1)}, x_i^{(k)} + k_k h, x^{(k+1)}, \ldots, x^{(d)})| \leq c^{k-1}Kh^3$. Thus,
$$
|g_{k+1}(x) - g_k(x)| \leq c \cdot c^{k-1}Kh^3 = c^k Kh^3.
$$
This completes the inductive step.
Finally, we bound the total error $|g(x) - f(x)| = |g_d(x) - g_0(x)|$ by summing the errors introduced at each interpolation step:
$$
|g(x) - f(x)| \leq \sum_{i=1}^d |g_i(x) - g_{i-1}(x)| \leq \sum_{i=1}^d c^{i-1}Kh^3 = Kh^3 \sum_{i=0}^{d-1} c^i.
$$
The last sum is a geometric series, which evaluates to $Kh^3 \frac{1 - c^d}{1 - c}$. For a fixed $c>1$ (independent of $d$), this expression is $O(c^{d})$ when $d$ is large. Therefore, tensor-product cubic convolutional interpolation has $O(c^d h^3)$ error.
\end{proof}
\begin{lemma}\label{lemma:cubic-interpolation-weights-curse-dimensionality}
Let $u: \mathbb{R} \rightarrow \mathbb{R}$ be a one-dimensional kernel function such that for some constant $c > 0$, the sum of weights in a 1D kernel regression is bounded by $c$ over its domain. That is, for any $x \in \mathbb{R}$ and a set of data points $\{x_i\}_{i=1}^n \subset \mathbb{R}$,
$$
\sum_{i=1}^n u\left(\frac{x - x_i}{h}\right) \leq c,
$$
where $h > 0$ is the bandwidth.
Let $u_d: \mathbb{R}^d \rightarrow \mathbb{R}$ be a d-dimensional product kernel defined as:
$$
u_d\left(\frac{x - x_i}{h}\right) = \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right),
$$
where $x = (x^{(1)}, x^{(2)}, ..., x^{(d)}) \in \mathbb{R}^d$ and $x_i = (x_i^{(1)}, x_i^{(2)}, ..., x_i^{(d)}) \in \mathbb{R}^d$ are d-dimensional points. Assume the data points $\{x_i\}_{i=1}^n$ ($n$ may differ from the univariate case) lie on a fixed d-dimensional grid $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$, where each $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is a finite set of $n_j$ grid points along the j-th dimension for $j = 1, 2, ..., d$.
Then, for any $x \in \mathbb{R}^d$, the sum of weights in the d-dimensional kernel regression is bounded by $c^d$:
$$
\sum_{i=1}^n u_d\left(\frac{x - x_i}{h}\right) \leq c^d.
$$
\end{lemma}
\begin{proof}
Let the fixed d-dimensional grid be defined by the Cartesian product of d sets of 1-dimensional grid points: $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$, where $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is the set of grid points along the j-th dimension.
We start with the sum of weights in the d-dimensional case:
$$
\sum_{i=1}^n u_d\left(\frac{x - x_i}{h}\right) = \sum_{i=1}^n \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right)
$$
Since the data points lie on the fixed grid $G$, we can rewrite the outer sum as a nested sum over the grid points in each dimension:
$$
\sum_{i=1}^n \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right) = \sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right)
$$
Now we can change the order of summation and product, as proven in Lemma \ref{lemma:switch-sum-product}:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) \right)
$$
By the assumption of the lemma, we know that for each dimension $j$, the sum of weights is bounded by $c$. Note that $\{p_{k_j}^{(j)}\}_{k_j=1}^{n_j}$ is simply a set of points in $\mathbb{R}$, thus:
$$
\sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) \leq c
$$
Therefore, we have:
$$
\prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) \right) \leq \prod_{j=1}^d c = c^d
$$
Thus, we have shown that:
$$
\sum_{i=1}^n u_d\left(\frac{x - x_i}{h}\right) \leq c^d
$$
\end{proof}
\subsection{Error Bounds for the SKI Kernel}\label{subsec:ski-kernel-error-bounds}
\subsubsection{Elementwise}\label{subsubsec:elementwise-error}
The following Lemma allows us to bound the absolute difference between the true and SKI kernels \textit{uniformly} in terms of the number of inducing points, an upper bound on the absolute sum of interpolation weights (in the Lagrange case this would be the Lebesgue constant), and the error due to polynomial interpolation.
\begin{lemma}\label{lemma:ski-kernel-elementwise-error}
The SKI kernel $\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ with $m$ inducing points and interpolation degree $L-1$ has error
\begin{align*}
\vert k(\textbf{x},\textbf{x}')-\tilde{k}(\textbf{x},\textbf{x}')\vert&\leq \delta_{m,L}+\sqrt{L}c^d\delta_{m,L}
% &=c^d\frac{2M_\rho(g)}{\rho-1}\rho^{-L}+\sqrt{L}c^{2d} \frac{2M_\rho(g)}{\rho-1}\rho^{-L}\\
% &=O\left(c^d\sqrt{L}\rho^{-L}\right)
\end{align*}
where $\delta_{m,L}$ is an upper bound on the error of multivariate convolutional interpolation with $m$ inducing points and interpolation degree $L-1$ and $c>0$ is an upper bound on the absolute sum of the weights for 1d kernel interpolation. If the convolutional interpolation is cubic \cite{keys1981cubic}, then we have
\begin{align*}
\vert k(\textbf{x},\textbf{x}')-\tilde{k}(\textbf{x},\textbf{x}')\vert&\leq \sqrt{L}O(c^{2d}h^3)
\end{align*}
\end{lemma}
\begin{proof}
Recall that SKI approximates the kernel as
\begin{align*}
k(\mathbf{x}, \mathbf{x}') &\approx \tilde{k}(\mathbf{x}, \mathbf{x}')\\
&= \boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}'),
\end{align*}
Let $\textbf{K}_{\textbf{U},\textbf{x}'}\in \mathbb{R}^m$ be the vector of kernels between the inducing points and the vector $\textbf{x}'$
\begin{align}
\vert k(\mathbf{x}, \mathbf{x}') -\tilde{k}(\mathbf{x}, \mathbf{x}')\vert &= \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} +\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\nonumber\\
&\leq \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\nonumber\\
&\leq \delta_{m,L}+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert \textrm{ since $\vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert$ is a single polynomial interpolation}\label{eqn:applying-single-poly-interp}
\end{align}
Now note that $\textbf{w}(x)\in \mathbb{R}^m$ is a sparse matrix with at most $L$ non-zero entries. Thus, letting $\tilde{\textbf{w}}(x)\in \mathbb{R}^L$ be the non-zero entries of $\textbf{w}(x)$ and similarly $\tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}\in \mathbb{R}^L$ be the entries of $\textbf{K}_{\textbf{U},\textbf{x}'}$ in the dimensions corresponding to non-zero entries of $\textbf{w}(x)\in \mathbb{R}^m$, while $\tilde{\textbf{K}}_{\textbf{U}}\in \mathbb{R}^{L\times m}$ is the analogous matrix for $\textbf{K}_{\textbf{U}}$, we have
\begin{align}
\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert&=\vert \tilde{\textbf{w}}(\textbf{x})^\top \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\textbf{w}}(\textbf{x})^\top \tilde{\textbf{K}}_\textbf{U}\textbf{w}(\textbf{x}')\vert\nonumber\\
&\leq \Vert \tilde{\textbf{w}}(\textbf{x})\Vert_2\Vert \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\mathbf{K}}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2\nonumber\\
&\leq c^d\sqrt{L}\Vert \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\mathbf{K}}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_\infty\nonumber\\
&\leq \sqrt{L}c^d\delta_{m,L}\label{eqn:applying-sparsity-single-kernel-evaluation}
\end{align}
where the last line follows as each element of $\mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')$ is a polynomial interpolation approximating each element of $\textbf{K}_{\textbf{U},\textbf{x}'}$. Plugging Eqn. \ref{eqn:applying-sparsity-single-kernel-evaluation} into Eqn. \ref{eqn:applying-single-poly-interp} gives us the desired initial result and Lemma \ref{lemma:tensor-product-interpolation-error} gives us the result when the convolutional kernel is cubic.
\end{proof}
% The following Lemma allows us to bound the absolute difference between the true and SKI kernels \textit{uniformly} in terms of the number of inducing points, the Lebesgue constant, which bounds the absolute sum of polynomial interpolation weights, and the error due to polynomial interpolation.
% \begin{lemma}\label{lemma:ski-kernel-elementwise-error}
% The SKI kernel $\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ with $m$ inducing points and interpolation degree $L$ has error
% \begin{align*}
% \vert k(\textbf{x},\textbf{x}')-\tilde{k}(\textbf{x},\textbf{x}')\vert&\leq \delta_{m,L}+mc(T)\delta_{m,L}
% \end{align*}
% where $\delta_{m,L}$ is an upper bound on the error of multivariate polynomial interpolation with $m$ inducing points and interpolation degree $L$ and $c(T)$ is the Lebesgue constant.
% \end{lemma}
% \begin{proof}
% Recall that SKI approximates the kernel as
% \begin{align*}
% k(\mathbf{x}, \mathbf{x}') &\approx \tilde{k}(\mathbf{x}, \mathbf{x}')\\
% &= \boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}'),
% \end{align*}
% Let $\textbf{U}$ be the set of inducing points and $\textbf{K}_{\textbf{U},\textbf{x}'}$ be the vector of kernels between the inducing points and the vector $\textbf{x}'$
% \begin{align*}
% \vert k(\mathbf{x}, \mathbf{x}') -\tilde{k}(\mathbf{x}, \mathbf{x}')\vert &= \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} +\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\\
% &\leq \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\\
% &\leq \delta_{m,L}+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert \textrm{ since $\vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert$ is a single polynomial interpolation}\\
% &\leq \delta_{m,L}+\Vert \textbf{w}(\textbf{x})\Vert_2 \Vert \textbf{K}_{\textbf{U},\textbf{x}'}- \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2\\
% &\leq \delta_{m,L}+\Vert \textbf{w}(\textbf{x})\Vert_1 \Vert \textbf{K}_{\textbf{U},\textbf{x}'}- \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2\\
% &\leq \delta_{m,L}+c(T)\Vert \textbf{K}_{\textbf{U},\textbf{x}'}- \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2\\
% &\leq \delta_{m,L}+c(T)\Vert \textbf{K}_{\textbf{U},\textbf{x}'}- \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_1\\
% &\leq \delta_{m,L}+mc(T) \delta_{m,L}
% \end{align*}
% where the last line follows as each element of $\mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')$ is a polynomial interpolation approximating each element of $\textbf{K}_{\textbf{U},\textbf{x}'}$.
% \amnote{We would love to have this not explicitly depend on $m$ here}
% \amnote{Below we try bounding $\Vert \textbf{K}_{\textbf{U},\textbf{x}'}- \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2$ using $l^\infty$ norm instead.}
% \begin{align*}
% \Vert \textbf{K}_{\textbf{U},\textbf{x}'}- \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2&\leq \sqrt{m}\Vert \textbf{K}_{\textbf{U},\textbf{x}'}- \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_\infty\\
% &\leq \sqrt{m}\delta_{m,L}
% \end{align*}
% which has $\sqrt{m}$ dependence instead of $m$ dependence.
% \end{proof}
% \amnote{Plug in \cite{sauer1995multivariate}} \textcolor{blue}{Justin: working on this as of 10/16}
% \amnote{Lebesgue constant may be relevant. We want to find a paper with it for multivariate Chebyshev nodes.}
% \subsection{Old Version}
% We focus first on the stationary case. Assume that $u_{A_j},u_{B_j}$ are the two closest inducing points to $x_j$ and that $u_{A_j}<x_j<u_{B_j}$. Let $\Tilde{k}(x_i,x_j)$ be the SKI approximation. Let
% \begin{align*}
% \Bar{k}(x_i,x_j)&=w_j k(x_i,u_{A_j})+(1-w_j)k(x_i,u_{B_j}).
% \end{align*}
% which linearly interpolates $k(x_i,x_j)$ between $k(x_i,u_{A_j})$ and $k(x_i,u_{B_j})$. Recall that if $k$ is twice differentiable in the second argument, then the error is
% \begin{align*}
% \frac{(x_j-u_{A_j})(u_{B_j}-x_j)}{2}\frac{\partial^2 k}{\partial x^2}(x_i,x)
% \end{align*}
% for some $x\in [u_{A_j},u_{B_j}]$.
% Then
% \begin{align*}
% \vert k(x_i,x_j)-\Tilde{k}(x_i,x_j)\vert &\leq \vert k(x_i,x_j)-\Bar{k}(x_i,x_j)\vert + \vert \Bar{k}(x_i,x_j)-\Tilde{k}(x_i,x_j)\vert
% \end{align*}
% Now the first term above has the error from a single linear interpolation. The second has linear interpolation twice. Let's look at this more carefully.
% \begin{align*}
% \Bar{k}(x_i,x_j)-\Tilde{k}(x_i,x_j)&=w_j (k(x_i,u_{A_j})-\Tilde{k}(x_i,u_{A_j}))+(1-w_j)(k(x_i,u_{B_j})-\Tilde{k}(x_i,u_{B_j}))
% \end{align*}
% where each of $k(x_i,u_{A_j})-\Tilde{k}(x_i,u_{A_j})$ and has linear interpolation error. We can do the same idea for polynomial interpolation, and can plug in the result of eqn 3.18 from \cite{sauer1995multivariate}
% when doing so \amnote{need to read the paper and understand, will take some effort}.
\subsubsection{Spectral Norm Error}\label{subsubsec:spectral-norm-error}
We now transition from elementwise error bounds for the SKI kernel approximation to bounds on the spectral norm of the difference between the true Gram matrix and its SKI approximation. This is both of independent interest but will also be important to analyzing the error of the SKI log-determinant and action of the SKI regularized inverse: both important quantities for hyperparameter optimization and posterior inference. We also provide a bound on the spectral norms of both the SKI train/test kernel matrix's approximation error and the matrix itself. These are useful when analyzing the error of the predictive mean and variance in Gaussian process regression.
\begin{proposition}\label{prop:spectral-norm}
Let $\gamma_{n,m,L}\equiv n(\delta_{m,L}+\sqrt{L}c^d\delta_{m,L})$, where $\delta_{m,L}$ is an upper bound on the error of multivariate interpolation with $m$ inducing points and interpolation degree $L-1$, and $c>0$ is an upper bound on the absolute sum of the weights for 1D kernel interpolation. Then, for the SKI approximation $\tilde{\textbf{K}}$ of the true Gram matrix $\textbf{K}$, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq \gamma_{n,m,L}
\end{align*}
and for tensor-product cubic convolutional interpolation we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq O(nc^{2d} h^3)
\end{align*}
\end{proposition}
\begin{proof}
Recall that for any matrix $\textbf{A}$, $\Vert \textbf{A} \Vert_2 \leq \sqrt{\Vert \textbf{A} \Vert_1 \Vert \textbf{A} \Vert_\infty}$. Since $\textbf{K}-\tilde{\textbf{K}}$ is symmetric, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq \sqrt{\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_1\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty} = \Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty
\end{align*}
Furthermore, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty$ is the maximum absolute row sum of $\textbf{K}-\tilde{\textbf{K}}$. Since there are $n$ rows and, by Lemma \ref{lemma:ski-kernel-elementwise-error}, each element of $\textbf{K} - \tilde{\textbf{K}}$ is bounded by $\delta_{m,L}+\sqrt{L}c^d\delta_{m,L}$ in absolute value, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty &\leq n \left(\delta_{m,L}+\sqrt{L} c^d\delta_{m,L}\right) = \gamma_{n,m,L}.
\end{align*}
Therefore, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2 \leq \gamma_{n,m,L}$.
\end{proof}
\begin{lemma}\label{lemma:test-train-kernel-matrix-error}
Let $\textbf{K}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ be the matrix of kernel evaluations between $T$ test points and $n$ training points, and let $\tilde{\textbf{K}}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ be the corresponding SKI approximation. Then
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq \max(\gamma_{T,m,L},\gamma_{n,m,L}).
\end{align*}
\end{lemma}
\begin{proof}
Using the same reasoning as in Proposition \ref{prop:spectral-norm}, we have
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq \sqrt{\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty} \\
&\leq \max \left(\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1, \Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty\right).
\end{align*}
Now, $\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1$ is the maximum absolute column sum, which is less than or equal to $T(\delta_{m,L} + \sqrt{L}c^d\delta_{m,L}) = \gamma_{T,m,L}$. Similarly, $\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty$ is the maximum absolute row sum, which is upper bounded by $n(\delta_{m,L} + \sqrt{L}c^d\delta_{m,L}) = \gamma_{n,m,L}$. Therefore,
$$
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq \max(\gamma_{T,m,L},\gamma_{n,m,L}).
$$
\end{proof}
\begin{lemma}\label{lemma:ski-test-train-bound}
Let $\tilde{\textbf{K}}_{\cdot,\textbf{X}} \in \mathbb{R}^{T \times n}$ be the matrix of SKI kernel evaluations between $T$ test points and $n$ training points, where the SKI approximation uses $m$ inducing points. Let $\textbf{W}(\cdot) \in \mathbb{R}^{T \times m}$ and $\textbf{W}(\textbf{X}) \in \mathbb{R}^{n \times m}$ be the matrices of interpolation weights for the test points and training points, respectively. Assume that the interpolation scheme is such that the sum of the absolute values of the interpolation weights for any point is bounded by $c^d$, where $c>0$ is a constant. Let $\textbf{K}_{\textbf{U}} \in \mathbb{R}^{m \times m}$ be the kernel matrix evaluated at the inducing points. If the kernel function $k$ is bounded such that $|k(\textbf{x}, \textbf{x}')| \leq M$ for all $\textbf{x}, \textbf{x}'$ in the domain of interest, then:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq \sqrt{Tn} m c^{2d} M
$$
\end{lemma}
\begin{proof}
By the definition of the SKI approximation and the submultiplicativity of the spectral norm, we have:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 = \Vert \textbf{W}(\cdot)\textbf{K}_{\textbf{U}}(\textbf{W}(\textbf{X}))^\top\Vert_2 \leq \Vert \textbf{W}(\cdot)\Vert_2 \Vert\textbf{K}_{\textbf{U}}\Vert_2 \Vert\textbf{W}(\textbf{X})\Vert_2
$$
We now bound each term.
1. **Bounding $\Vert \textbf{W}(\cdot)\Vert_2$ and $\Vert \textbf{W}(\textbf{X})\Vert_2$:**
Since the spectral norm is induced by the Euclidean norm, and using the assumption that the sum of absolute values of interpolation weights for any point is bounded by $c^d$, we have
$$\Vert \textbf{W}(\cdot)\Vert_2 \leq \sqrt{\Vert \textbf{W}(\cdot)\Vert_1 \Vert \textbf{W}(\cdot)\Vert_\infty} \leq \sqrt{T c^d \cdot c^d} = \sqrt{T} c^d.$$
Similarly, $\Vert\textbf{W}(\textbf{X})\Vert_2 \leq \sqrt{n}c^d$.
2. **Bounding $\Vert\textbf{K}_{\textbf{U}}\Vert_2$:**
Since $\textbf{K}_{\textbf{U}}$ is symmetric, $\Vert \textbf{K}_{\textbf{U}} \Vert_2 \leq \Vert \textbf{K}_{\textbf{U}} \Vert_\infty$. Each entry of $\textbf{K}_{\textbf{U}}$ is bounded by $M$ (by the boundedness of $k$), and each row has $m$ entries, so $\Vert \textbf{K}_{\textbf{U}} \Vert_\infty \leq mM$. Thus, $\Vert\textbf{K}_{\textbf{U}}\Vert_2 \leq mM$.
Combining these bounds, we get:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq (\sqrt{T} c^d) (mM) (\sqrt{n} c^d) = \sqrt{Tn} m c^{2d} M
$$
as required.
\end{proof}
% \begin{lemma}
% \begin{align*}
% \Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq
% \end{align*}
% \end{lemma}
% \begin{proof}
% \begin{align*}
% \Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&=\Vert \textbf{W}(\cdot)\textbf{K}_{\textbf{U}}(\textbf{W}(\textbf{X}))^\top\Vert_2\\
% &\leq \Vert \textbf{W}(\cdot)\Vert_2\Vert\textbf{K}_{\textbf{U}}\Vert_2 \Vert(\textbf{W}(\textbf{X}))^\top\Vert_2
% \end{align*}
% Note that $\Vert \textbf{W}(\cdot)\Vert_2\leq \sqrt{\Vert \textbf{W}(\cdot)\Vert_1\Vert \textbf{W}(\cdot)\Vert_\infty}\leq \sqrt{T\Vert\textbf{W}(\cdot)\Vert_\infty \Vert\textbf{W}(\cdot)\Vert_\infty}=\sqrt{T}\Vert\textbf{W}(\cdot)\Vert_\infty=\sqrt{T}c^d$ and $\Vert \textbf{W}(\textbf{X})\Vert\leq \sqrt{n}c^d$
% and thus
% \begin{align*}
% \Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&=\Vert \textbf{W}(\cdot)\textbf{K}_{\textbf{U}}(\textbf{W}(\textbf{X}))^\top\Vert_2\\
% &\leq \Vert \textbf{W}(\cdot)\Vert_2\Vert\textbf{K}_{\textbf{U}}\Vert_2 \Vert(\textbf{W}(\textbf{X}))^\top\Vert_2\\
% &\leq \sqrt{Tn}c^{2d}\Vert\textbf{K}_{\textbf{U}}\Vert_2\\
% &\leq \sqrt{Tn}c^{2d}\Vert \textbf{K}_{\textbf{U}}\Vert_\infty\textrm{ by symmetry of $\textbf{K}_{\textbf{U}}$ and $\Vert \cdot\Vert_2\leq \sqrt{\Vert \cdot\Vert_1\Vert\cdot\Vert_\infty}$}\\
% &\leq m\sqrt{Tn}c^{2d}M
% \end{align*}
% where $M>0$ is an upper bound on the values taken by the kernel on the compact set of interest.
% \end{proof}
\subsection{Achieving a Desired Error in Spectral Norm}\label{subsec:desired-error-spectral-norm}
In this subsection, a lower bound on the number of inducing points $m$ required to achieve a desired error tolerance $\epsilon > 0$ for the SKI Gram matrix when using cubic convolutional interpolation. We assume the domain is $[-D, D]^d$, where $D > 0$ is a constant that determines the size of the domain.
We can directly determine a sufficient number of inducing points needed for a given $\epsilon > 0$ and domain $[-D, D]^d$.
\begin{corollary}\label{cor:inducing-points-count-alt}
If $L=4$ (cubic interpolation), the domain is $[-D, D]^d$, and we use convolutional cubic interpolation \cite{keys1981cubic}, then to achieve a spectral norm error of $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$, it is sufficient to choose the number of inducing points $m$ such that:
$$
m \geq \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
$$
for some constant $K'$ that depends only on the kernel function and the interpolation scheme.
\end{corollary}
\begin{proof}
We want to choose $m$ such that the spectral norm error $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$. From Proposition \ref{prop:spectral-norm}, we have:
$$
\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq n(1 + \sqrt{L}c^d) \delta_{m,L}
$$
For cubic interpolation ($L=4$), Lemma \ref{lemma:tensor-product-interpolation-error}, combined with the analysis in Lemma \ref{lemma:tensor-product-interpolation-error}, gives us:
$$
\delta_{m,L} \leq K' c^{2d} h^3
$$
where $K'$ is a constant that depends only on the kernel function (through its derivatives) and the interpolation scheme, but not on $n$, $m$, $h$, or $d$.
Therefore, a sufficient condition to ensure $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$ is:
\begin{equation} \label{eq:sufficient_condition_final}
n(1 + 2c^d) K' c^{2d} h^3 \leq \epsilon
\end{equation}
Since the inducing points are placed on a regular grid with spacing $h$ in each dimension, and the domain is $[-D,D]^d$ and assuming that $2D\mod h\equiv 0$, the number of inducing points $m$ satisfies:
$$
m = \left(\frac{2D}{h}\right)^d
$$
We can rearrange this to get:
$$
h = \frac{2D}{m^{1/d}}
$$
Substituting this into the sufficient condition \eqref{eq:sufficient_condition_final}, we get:
$$
n (1 + 2c^d) K' c^{2d} \left(\frac{2D}{m^{1/d}}\right)^3 \leq \epsilon
$$
Rearranging to isolate $m$, we obtain:
$$
m^{3/d} \geq \frac{n}{\epsilon} (1 + 2c^d) K' c^{2d} (8D^3)
$$
$$
m \geq \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
$$
\end{proof}
This result shows that the number of inducing points grows
\begin{itemize}
\item At a cube root rate with the number of observations $n$ and as error tolerance $\epsilon$ decreases. Thus, as we want a tighter error tolerance or have more observations we need more inducing points.
\item Linearly with the volume of the domain $(2D)^d$. Thus, if our observations are concentrated in a small region and we select an appropriately sized domain to cover it we need fewer inducing points.
\item Exponentially with the dimension $d$.
\end{itemize}
\subsection{Lower Bound on Error for Linear Time}\label{subsec:linear-time}
The next theorem establishes a condition on the spectral norm error, $\epsilon$, that ensures linear-time $O(n)$ computational complexity for SKI. The core idea is to choose the number of inducing points, $m$, large enough to satisfy the error bound based on Corollary \ref{cor:inducing-points-count-alt}, while simultaneously ensuring that $m$ does not grow faster than $O(n/\log n)$, thus ensuring linear-time complexity. By combining these two requirements, we can derive a lower bound for $\epsilon$ as a function of $n$, the dimensionality $d$, the domain size $D$, and the constant $c$ related to the interpolation weights.
Interestingly, the theorem implies a fundamental difference between two dimensionality regimes. For $d \leq 3$, the lower bound on $m$ required for a fixed $\epsilon$ grows more slowly than $n/\log n$. This means that for any fixed $\epsilon > 0$, SKI with cubic interpolation is guaranteed to be a linear-time algorithm for sufficiently large $n$. In contrast, for $d > 3$, the lower bound on $m$ required for a fixed $\epsilon$ eventually grows faster than $n/\log n$. Thus, to maintain linear-time complexity for $d > 3$, we must allow the error $\epsilon$ to increase with $n$. This demonstrates that the ``curse of dimensionality" significantly impacts the scalability of SKI, making it challenging to achieve both high accuracy and linear-time complexity in higher dimensions. The theorem quantifies this trade-off, showing precisely how much error growth is necessary as $n$ and $d$ increase.
\begin{theorem}[Sufficient Condition for Linear Time Complexity]
Consider $d\geq 3$. If the domain is $[-D, D]^d$, we use convolutional cubic interpolation ($L=4$), and we choose $\epsilon>0$ satisfying
\begin{equation} \label{eq:epsilon_condition}
\epsilon \geq \frac{(1 + 2c^d) K' 8 c^{2d} D^3}{C^{3/d}} \cdot \frac{n (\log n)^{3/d}}{n^{3/d}}
\end{equation}
for some constants $K,C>0$ that depend on the kernel function and the interpolation scheme. then if we choose
\begin{align*}
m &= \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
\end{align*}
then we have both $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2\leq \epsilon$ and
SKI computational complexity of $O(n)$.
\end{theorem}
\begin{proof}
The idea is that we want to have an error sufficiently large so that when we choose $m$ based on Corollary \ref{cor:inducing-points-count-alt}, we have $m=O\left(\frac{n}{\log n}\right)$.
Assume that
$$
\epsilon \geq \frac{(1 + 2c^d) K' 8 c^{2d} D^3}{C^{3/d}} \cdot \frac{n (\log n)^{3/d}}{n^{3/d}}.
$$
Rearranging this we obtain
\begin{align*}
\left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}&\leq C\frac{n}{\log n}.\\
&=O\left(\frac{n}{\log n}\right).
\end{align*}
Now taking
\begin{align*}
m &= \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
\end{align*}
we have that $m=O\left(\frac{n}{\log n}\right)$ and by Corollary \ref{cor:inducing-points-count-alt}, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2\leq \epsilon$. Now plugging in $\frac{n}{\log n}$ into $m\log m$ we obtain
\begin{align*}
O\left(m\log m\right)&=O\left(\frac{n}{\log n}\log \frac{n}{\log n}\right)\\
&=O\left(\frac{n}{\log n}\log n-\frac{n}{\log n}\log \log n\right)\\
&=O(n)
\end{align*}
as desired.
\end{proof}
\subsection{Additional Quantities}\label{subsec:additional-quantities}
\subsubsection{Action of Regularized Inverse}\label{subsubsec:action-regularized-inverse}
\begin{lemma}\label{lemma:action-inverse-error}
Let \(\mathbf{\tilde{K}}\) be the SKI approximation of the kernel matrix \(\mathbf{K}\), and let \(\sigma^2\) be the regularization parameter. For a vector \(\mathbf{y}\), the error in the action of the regularized inverse can be bounded as follows:
\begin{align*}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y}\right\|_2 &\leq \frac{\gamma_{n, m, L}}{\sigma^4} \|\mathbf{y}\|_2
% \\
% &=\frac{O(n\sqrt{L}\rho^{-L})}{\sigma^4}
\end{align*}
\end{lemma}
\begin{proof}
Note that
\[
\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} = \left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} (\mathbf{K} - \mathbf{\tilde{K}}) \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}
\]
Taking the spectral norm, we have
\[
\begin{aligned}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 &\leq \left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \|\mathbf{K} - \mathbf{\tilde{K}}\|_2 \left\|\left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \\
&\leq \gamma_{n, m, L}\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \left\|\left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2\textrm{ by Lemma \ref{proposition:spectral-norm}}\\
&\leq \frac{\gamma_{n, m, L}}{\sigma^4}
\end{aligned}
\]
Thus,
\[
\begin{aligned}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y}\right\|_2 &\leq \frac{\gamma_{n, m, L}}{\sigma^4} ||\mathbf {y}||_2
\end{aligned}
\]
The last inequality follows from the bounds on the inverse norms and the spectral norm error bound for \(K - \tilde{K}\).
\end{proof}
\subsubsection{Log Determinant}\label{subsubsec:log-determinant}
\begin{lemma}\label{lemma:log-determinant-error}
Let \(\mathbf{\tilde{K}}\) be the SKI approximation of the kernel matrix \(\mathbf{K}\), and let \(\sigma^2\) be the regularization parameter. The error in the log-determinant of the regularized SKI matrix can be bounded as follows:
\[
\left|\log \det(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - \log \det(\mathbf{K} + \sigma^2 \mathbf{I})\right| \leq \frac{\gamma_{n, m, L}}{\sigma^2}.
\]
\end{lemma}
\begin{proof}
We begin by applying the functional Mean Value Theorem for matrices, which implies that for some matrix \(Z = (\mathbf{\tilde{K}} + \alpha (\mathbf{K} - \mathbf{\tilde{K}})) + \sigma^2 \mathbf{I}\) with \(\alpha \in [0, 1]\),
\[
|f(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - f(\mathbf{K} + \sigma^2 \mathbf{I})| \leq \|f'(Z)\|_2 \|\mathbf{K} - \mathbf{\tilde{K}}\|_2
\]
where \(f(X) = \log |X|\) and thus \(f'(X) = X^{-\top}\).
Thus,
\[
|\log \det(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - \log \det(\mathbf{K} + \sigma^2 \mathbf{I})| \leq \left\| (\mathbf{\tilde{K}} + \alpha (\mathbf{K} - \mathbf{\tilde{K}}) + \sigma^2 I)^{-1} \right\|_2 ||\mathbf {K} - {\tilde {\mathbf {K}}}||_2\label{eqn:initial-log-det-bound}
\]
Using Claim 1, which bounds the eigenvalues of a convex combination of matrices, we have:
\[
\lambda_{\min}(Z) = (\lambda_{\min} (\mathbf{\tilde{K}})+ (1-\alpha)\lambda_{\min} (\mathbf {K}))+\sigma^2 > 0
\]
Therefore since $\mathbf{\tilde{K}}$ and $(\mathbf {K}$ are SPD and thus have positive eigenvalues,
\[
\left\| (\mathbf{\tilde{K}} + (1-\alpha)(\mathbf {K}- {\tilde {\mathbf {K}}})+ {\sigma^2 I})^{-1} )^{-1}\right\|_2 < {\frac {1}{ {\sigma^2}}}
\]
Substituting these bounds into Eqn. \ref{eqn:initial-log-det-bound} gives:
\[
|\log \det (\tilde{\mathbf {K}}+ {\sigma^2 I})- \log \det ({\mathbf {K}}+ {\sigma^2 I})| < {\frac {1}{ {\sigma^2}}} ||{\mathbf {K}- {\tilde {\mathbf {K}}}}||_2
\leq {\frac {1}{ {\sigma^2}}} {\gamma_{n,m,L}}
\]
\end{proof}
% First, recall the functional MVT (will write in formal way at some point).
% \begin{align*}
% \vert f(X)-f(Y)\vert&\leq \Vert f'(Z)\Vert_2 \Vert X-Y\Vert_2
% \end{align*}
% for some $Z=X+\alpha(Y-X),\alpha \in [0,1]$. Then letting $f(X)=\log \vert X\vert$ and noting that $f'(X)=X^{-\top}$,
% \begin{align*}
% \vert \log |\Tilde{K}|-\log |K|\vert&\leq \Vert (\Tilde{K}+\alpha (K-\Tilde{K}))^{-1}\Vert_2 \Vert K-\Tilde{K}\Vert_2,\alpha \in [0,1]\\
% &\leq \frac{1}{\min(\lambda_n,\tilde{\lambda}_n)}\Vert K-\Tilde{K}\Vert_2\\
% &\leq \frac{1}{\min(\lambda_n,\tilde{\lambda}_n)}\gamma_{n,m,L}
% \end{align*}
% for some $\alpha \in [0,1]$.
% However, I'm not sure how best to bound $\Vert (\Tilde{K}+\alpha (K-\Tilde{K}))^{-1}\Vert_2$.
% \jmei{[[This looks good, eigs for the convex combination should be in the interval $[\min(\lambda_n, \tilde{\lambda}_n), \max(\lambda_1,\tilde{\lambda}_1)]$, so we could use $\Vert (\Tilde{K}+\alpha (K-\Tilde{K}))^{-1}\Vert_2 \le 1/\min(\lambda_n, \tilde{\lambda}_n) \le 1/\sigma^2$.]]}
% \subsubsection{Another Potential Approach}
% Another approach is: let $\lambda_1,\ldots,\lambda_n$ be the eigenvalues of $\Tilde{K}K^{-1}$
% \begin{align*}
% \vert \log |\Tilde{K}|-\log |K|\vert&=\left\vert \log \frac{|\Tilde{K}|}{|K|}\right\vert\\
% &=\left\vert \log \vert \Tilde{K}K^{-1}\vert\right\vert\\
% &=\vert \log \prod_{i=1}^n \lambda_i \vert \\
% &=\vert \sum_{i=1}^n \log \lambda_i \vert\\
% &\leq \sum_{i=1}^n \vert \log \lambda_i \vert
% \end{align*}
% Now let $a=1,b=\lambda_i$. By the MVT, there exists some $c_i$ between the two such that
% \begin{align*}
% \log \lambda_i-\log 1&=\frac{1}{c_i}(\lambda_i-1)\\
% \log \lambda_i &=\frac{\lambda_i-1}{c_i}
% \end{align*}
% so that letting $m$ \jmei{??}
% \begin{align*}
% \vert \log |\Tilde{K}|-\log |K|\vert&\leq \sum_{i=1}^n \left\vert \frac{\lambda_i-1}{c_i}\right\vert\\
% &\leq \sum_{i=1}^n \frac{\vert \lambda_i-1\vert}{c_i}
% \end{align*}
% Now note that if $\Vert \Tilde{K}K^{-1}-I\Vert_2\leq \delta <1$, then letting $\rho$ be the spectral radius (largest absolute value of its eigenvalues), we have
% \begin{align*}
% \max_i \vert \lambda_i-1\vert&=\rho(\Tilde{K}K^{-1}-I)\\
% &\leq \sigma_{\max}(\Tilde{K}K^{-1}-I)\\
% &\leq \delta
% \end{align*}
% We now claim that all of the eigenvalues $\lambda_i$ are non-negative, so that there exists $m>0$ such that $\forall i\in \{k\in \mathbb{N},k\leq n\}$. Thus we have
% \begin{align*}
% \vert \log |\Tilde{K}|-\log |K|\vert&\leq \frac{n\delta}{m}
% \end{align*}
% We want to bound $\delta$.
% \begin{align*}
% \Vert \Tilde{K}K^{-1}-I\Vert_2&=\Vert (\Tilde{K}-K)K^{-1}\Vert_2\\
% &\leq \Vert\Tilde{K}-K\Vert_2 \Vert K^{-1}\Vert_2\\
% &\leq \frac{1}{\sigma_{\min}(K)} \Vert\Tilde{K}-K\Vert_2
% \end{align*}
% Thus we have
% \begin{align*}
% \vert \log |\Tilde{K}|-\log |K|\vert&\leq \frac{n}{m}\frac{1}{\sigma_{\min}(K)} \Vert\Tilde{K}-K\Vert_2
% \end{align*}
% \subsubsection{Playing around}
% Here, let $\lambda_i$ be eigvals of $K$. Using the inequality $1-\frac{1}{x}\le\log(x)\le x - 1$, we have
% \[
% \log|\tilde{K}| = \sum_i \log \tilde{\lambda}_i \le \sum_i (\tilde{\lambda}_i - 1) = \textrm{tr}(\tilde{K})-n
% \]
% and
% \[
% \log|K| \ge \sum_i (1 - \frac{1}{\lambda_i}) = n - \textrm{tr}(K^{-1})
% \]
% so that
% \[
% \log|\tilde{K}| - \log|K| \le -2n + \textrm{tr}(\tilde{K}) + \textrm{tr}(K^{-1})
% \]
% hm, that's not at all what we want.
\section{Gaussian Processes Applications}\label{sec:gp-applications}
\subsection{Kernel Hyperparameter Estimation and Posterior Inference}
\subsubsection{Difference in Log-Likelihoods}
\begin{lemma}\label{lemma:difference-in-log-likelihoods}
Let \( \mathbf{K} \) be the true kernel matrix, \( \tilde{\mathbf{K}} \) be the SKI approximation of the kernel matrix using \( m \) inducing points and interpolation degree \( L-1 \), and \( \sigma^2 \) be the regularization parameter (noise variance). Let \( \mathbf{y} \) be the vector of observed target values, and \( n \) be the number of data points. Let \( \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) \) be the true log-likelihood and \( \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) \) be the SKI approximation of the log-likelihood, both for a mean-zero Gaussian Process prior. Then, the absolute difference between the true and SKI log-likelihoods is bounded as follows:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2 + \frac{\gamma_{n, m, L}}{\sigma^2} \right)
\end{equation}
where \( \gamma_{n, m, L} \equiv n(\delta_{m, L} + \sqrt{L} c^d \delta_{m, L}) \), \( \delta_{m, L} \) is an upper bound on the error of multivariate interpolation with \( m \) inducing points and interpolation degree \( L-1 \), and \( c > 0 \) is an upper bound on the absolute sum of the weights for 1D kernel interpolation.
Furthermore, if the convolutional cubic interpolation kernel (\( L=4 \)) is used with grid spacing \( h \), and the domain is contained in $[-D, D]^d$, then:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{n(2O(c^{2d}h^3))}{\sigma^4} \| \mathbf{y} \|^2 + \frac{n(2O(c^{2d}h^3))}{\sigma^2} \right)
\end{equation}
\end{lemma}
\begin{proof}
We begin by considering the log-likelihood of a mean-zero Gaussian Process, given by:
\begin{equation}
\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi)
\end{equation}
The corresponding SKI approximation is:
\begin{equation}
\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi)
\end{equation}
Our goal is to bound the absolute difference \( |\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})| \). Applying the triangle inequality, we have:
\begin{align}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | &= \left| \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} + \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| \right| \nonumber \\
&\leq \left| \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \right| + \left| \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right|
\end{align}
We now leverage Lemma \ref{lemma:action-inverse-error} and Lemma \ref{lemma:log-determinant-error} to bound the two terms on the right-hand side. Lemma \ref{lemma:action-inverse-error} provides a bound on the difference between the action of the true and approximated regularized inverses:
\begin{equation}
\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \|_2 \leq \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|_2
\end{equation}
This allows us to bound the first term:
\begin{equation}
\left| \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \right| \leq \frac{1}{2} \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2
\end{equation}
Lemma \ref{lemma:log-determinant-error} bounds the difference in the log-determinants:
\begin{equation}
| \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \log|\mathbf{K} + \sigma^2 \mathbf{I}| | \leq \frac{\gamma_{n, m, L}}{\sigma^2}
\end{equation}
Thus, the second term is bounded by:
\begin{equation}
\left| \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right| \leq \frac{1}{2} \frac{\gamma_{n, m, L}}{\sigma^2}
\end{equation}
Combining these bounds, we obtain the overall bound on the difference in log-likelihoods:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2 + \frac{\gamma_{n, m, L}}{\sigma^2} \right)
\end{equation}
Now, let us consider the specific case of using a convolutional cubic interpolation kernel (\( L=4 \)). From Lemma \ref{lemma:tensor-product-interpolation-error}, we have \( \delta_{m, L} = O(c^{2d}h^3) \). Consequently, from Corollary \ref{cor:spectral-norm-error-alt}:
\begin{equation}
\gamma_{n, m, L} \equiv n(\delta_{m, L} + \sqrt{L} c^d \delta_{m, L}) = n(O(c^{2d}h^3) + 2c^d O(c^{2d}h^3)) = n(2O(c^{2d}h^3))
\end{equation}
Substituting this into our general bound, we obtain the specific bound for cubic interpolation:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{n(2O(c^{2d}h^3))}{\sigma^4} \| \mathbf{y} \|^2 + \frac{n(2O(c^{2d}h^3))}{\sigma^2} \right)
\end{equation}
This concludes the proof.
\end{proof}
% We are interested in
% \begin{align*}
% &\vert \mathcal{L}(\theta;\textbf{X},\textbf{y})-\mathcal{L}(\theta;\textbf{X},\textbf{y})\vert \\
% &=\left\vert -\frac{1}{2}\textbf{y}^\top (\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}-\frac{1}{2}\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert -\frac{N}{2}\log (2\pi)-\left(-\frac{1}{2}\textbf{y}^\top (\tilde{\textbf{K}}+\sigma^2 \textbf{I})^{-1}\textbf{y}-\frac{1}{2}\log \vert \tilde{\textbf{K}}+\sigma^2 \textbf{I}\vert -\frac{N}{2}\log (2\pi)\right)\right\vert\\
% &\leq \left\vert \frac{1}{2}\textbf{y}^\top (\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}-\frac{1}{2}\textbf{y}^\top (\tilde{\textbf{K}}+\sigma^2 \textbf{I})^{-1}\textbf{y}\right\vert+\left\vert\frac{1}{2}\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert-\frac{1}{2}\log \vert \tilde{\textbf{K}}+\sigma^2 \textbf{I}\vert\right \vert\\
% &\leq \frac{1}{2}\frac{\gamma_{n, m, L}}{\sigma^4} \Vert \textbf{y}\Vert^2+\left\vert\frac{1}{2}\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert-\frac{1}{2}\log \vert \tilde{\textbf{K}}+\sigma^2 \textbf{I}\vert\right \vert\textrm{, Lemma \ref{lemma:action-inverse-error}}\\
% &\leq \frac{1}{2}\left(\frac{\gamma_{n, m, L}}{\sigma^4} \Vert \textbf{y}\Vert^2+\frac{\gamma_{n,m,L}}{\sigma^2}\right)\textrm{, Lemma \ref{lemma:log-determinant-error}}
% \end{align*}
\subsubsection{Difference in MLEs}
We formalize the intuition that a good initialization for gradient ascent on the SKI log-likelihood, combined with a sufficiently accurate SKI approximation, leads to a SKI MLE that is close to the true MLE. We achieve this by first bounding the difference in gradients between the true and SKI log-likelihoods. Then, under the assumption of strong concavity of the true log-likelihood, we show that if the initialization is close to the true MLE, the resulting SKI MLE will also be close to the true MLE.
\begin{assumption}[Strong Concavity of the Population Log-Likelihood]\label{assumption:strong-concavity}
The population log-likelihood $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu > 0$ in a neighborhood $\mathcal{D}$ of the true hyperparameters $\theta^*$. That is, for all $\theta_1, \theta_2 \in \mathcal{D}$,
$$
\bar{\mathcal{L}}(\theta_2) \leq \bar{\mathcal{L}}(\theta_1) + \nabla \bar{\mathcal{L}}(\theta_1)^\top (\theta_2 - \theta_1) - \frac{\mu}{2} \| \theta_2 - \theta_1 \|_2^2.
$$
\end{assumption}
\begin{proposition}\label{prop:strong_concavity_hessian}
A function is strongly concave if and only if the Hessian matrix $H_f(\theta)$ satisfies $H_f(\theta) \preceq -\mu I$ for all $\theta \in \mathcal{D}$, where $\preceq$ denotes the negative semidefinite ordering.
\end{proposition}
\begin{proof}
See \cite{boyd2004convex}.
\end{proof}
\begin{assumption}[Local Identifiability of Hyperparameters]\label{assumption:local-identifiability}
Let $P(\cdot | \mathbf{X}; \theta)$ denote the Gaussian process predictive distribution at a set of input points $\mathbf{X}$ given hyperparameters $\theta$. There exists a neighborhood $\mathcal{D}$ of the true hyperparameters $\theta^*$ such that for any $\theta \in \mathcal{D}$ with $\theta \neq \theta^*$, there exists a finite set of input points $\mathbf{X} = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$, a set $A$ in the Borel sigma-algebra of $\mathbb{R}^n$, and a $\delta > 0$ such that:
$$
|P(\mathbf{y} \in A | \mathbf{X}; \theta) - P(\mathbf{y} \in A | \mathbf{X}; \theta^*)| > \delta
$$
and
$$
P^*_X(\mathbf{X}) > 0,
$$
where $\mathbf{y} = (y_1, \ldots, y_n)$ is the vector of corresponding output values, and $P^*_X$ is the marginal distribution of inputs under $P^*$.
\end{assumption}
% \begin{definition}[Gradient Smoothness]
% \label{def:gradient_smoothness}
% Let $\mathcal{D} \subseteq \mathbb{R}^d$ be an open convex set. A differentiable function $f: \mathcal{D} \rightarrow \mathbb{R}$ is said to have a \emph{Lipschitz continuous gradient} (or a \emph{smooth gradient}) with parameter $L > 0$ in $\mathcal{D}$ if for all $\theta_1, \theta_2 \in \mathcal{D}$:
% \begin{equation}
% \| \nabla f(\theta_2) - \nabla f(\theta_1) \|_2 \leq L \| \theta_2 - \theta_1 \|_2,
% \end{equation}
% where $\nabla f(\theta)$ denotes the gradient of $f$ evaluated at $\theta$, and $\| \cdot \|_2$ denotes the Euclidean norm.
% \end{definition}
% \begin{proposition}\label{prop:gradient_smoothness_hessian}
% If $f$ is twice continuously differentiable, $f$ has a Lipschitz continuous gradient with parameter $L$ in $\mathcal{D}$ if and only if
% \begin{equation}
% \| H_f(\theta) \|_2 \leq L \quad \forall \theta \in \mathcal{D},
% \end{equation}
% where $H_f(\theta)$ is the Hessian matrix of $f$ at $\theta$, and $\| \cdot \|_2$ denotes the spectral norm.
% \end{proposition}
% \begin{proof}
% See \cite{boyd2004convex}.
% \end{proof}
\begin{lemma}[Bound on Derivative of SKI Kernel Error using Kernel Property of Derivative]
\label{lemma:ski_kernel_derivative_error_kernel}
Let $\mathcal{X} \subset \mathbb{R}^d$ be the input domain, and let $\Theta \subset \mathbb{R}^p$ be the domain of the kernel hyperparameters. Let $k_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be a kernel function parameterized by $\theta \in \Theta$, and let $\tilde{k}_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be its SKI approximation using $m$ inducing points and interpolation degree $L-1$. Assume that the partial derivative of $k_{\theta}$ with respect to a hyperparameter $\theta_i$, denoted as $k'_{\theta}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$, is also a valid SPD kernel. Let $\tilde{k}'_{\theta}(x,x')$ be the SKI approximation of $k'_{\theta}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$.
Then, for all $x, x' \in \mathcal{X}$, all $\theta \in \Theta$, the following inequality holds:
\begin{equation}
\left\vert \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}-\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i}\right\vert = \left\vert k'_{\theta}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \leq\delta_{m,L}'+\sqrt{L}c^d\delta_{m,L}',
\end{equation}
where $\delta_{m,L}'$ is an upper bound on the error of the SKI approximation of the kernel $k'_{\theta}(x,x')$ with $m$ inducing points and interpolation degree $L-1$, as defined in Lemma \ref{lemma:ski-kernel-elementwise-error}.
\end{lemma}
\begin{proof}
By assumption, $k'_{\theta}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$ is a valid SPD kernel. The SKI approximation of $k'_{\theta}(x, x')$ using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}(x, x')$ is given by $\tilde{k}'_{\theta}(x, x')$. For the kernel $k'_{\theta}(x, x')$, we have:
\begin{align*}
\left\vert k'_{\theta}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \leq \delta_{m,L}',
\end{align*}
where $\delta_{m,L}'$ is the upper bound on the error of the SKI approximation of $k'_{\theta}(x, x')$ as defined in Lemma \ref{lemma:ski-kernel-elementwise-error}.
Now, we need to show that $\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta}(x, x')$. Recall that the SKI approximation $\tilde{k}_{\theta}(x, x')$ is a linear combination of kernel evaluations at inducing points, with weights that depend on $x$ and $x'$:
\begin{align*}
\tilde{k}_{\theta}(x, x') = \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k_{\theta}(u_j, u_l)
\end{align*}
where $w_{jl}(x, x')$ are the interpolation weights. Taking the partial derivative with respect to $\theta_i$, we get:
\begin{align*}
\frac{\partial \tilde{k}_{\theta}(x, x')}{\partial \theta_i} &= \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') \frac{\partial k_{\theta}(u_j, u_l)}{\partial \theta_i} \\
&= \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k'_{\theta}(u_j, u_l).
\end{align*}
This is precisely the SKI approximation of the kernel $k'_{\theta}(x, x')$ using the same inducing points and weights:
\begin{align*}
\tilde{k}'_{\theta}(x, x') = \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k'_{\theta}(u_j, u_l).
\end{align*}
Therefore, $\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta}(x, x')$.
Substituting this into our inequality, we get:
\begin{align*}
\left\vert \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}-\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i}\right\vert &= \left\vert k'_{\theta}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \\
&\leq \delta_{m,L}'+\sqrt{L}c^d\delta_{m,L}'.
\end{align*}
\end{proof}
\begin{proposition}[Bound on Spectral Norm of Gradient Difference]
\label{prop:gradient_spectral_norm_bound}
Let $\mathcal{X} \subset \mathbb{R}^d$ be the input domain, and let $\Theta \subset \mathbb{R}^p$ be the domain of the kernel hyperparameters. Let $k_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be a kernel function parameterized by $\theta \in \Theta$, and let $\tilde{k}_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be its SKI approximation using $m$ inducing points and interpolation degree $L-1$. Let $K \in \mathbb{R}^{n \times n}$ and $\tilde{K} \in \mathbb{R}^{n \times n}$ be the corresponding kernel matrices for a dataset of $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$.
Assume that for each $i \in \{1, ..., p\}$, the partial derivative of $k_{\theta}$ with respect to $\theta_i$, denoted as $k'_{\theta,i}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$, is a valid SPD kernel. Let $\tilde{k}'_{\theta,i}(x,x')$ be the SKI approximation of $k'_{\theta,i}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$.
Then, the spectral norm of the difference between the gradient of the true kernel matrix and the gradient of the SKI kernel matrix is bounded by:
\begin{equation}
\| \nabla_\theta K - \nabla_\theta \tilde{K} \|_2 \leq \sqrt{p} \max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i},
\end{equation}
where $\nabla_\theta K$ and $\nabla_\theta \tilde{K}$ are represented as $p \times n^2$ matrices using the vec-notation (denominator layout), $p$ is the number of hyperparameters, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ (analogous to Proposition \ref{prop:spectral-norm}, but for the kernel $k'_{\theta,i}$).
\end{proposition}
\begin{proof}
We represent the gradients $\nabla_\theta K$ and $\nabla_\theta \tilde{K}$ as $p \times n^2$ matrices using the vec-notation (denominator layout):
\begin{align*}
\nabla_\theta K = \begin{bmatrix} \frac{\partial \text{vec}(K)}{\partial \theta_1} & \frac{\partial \text{vec}(K)}{\partial \theta_2} & \cdots & \frac{\partial \text{vec}(K)}{\partial \theta_p} \end{bmatrix}^T, \\
\nabla_\theta \tilde{K} = \begin{bmatrix} \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_1} & \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_2} & \cdots & \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_p} \end{bmatrix}^T.
\end{align*}
Let $K'_{\theta,i}$ be the kernel matrix corresponding to the kernel $k'_{\theta,i}(x,x') = \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}$, and let $\tilde{K}'_{\theta,i}$ be the kernel matrix corresponding to its SKI approximation $\tilde{k}'_{\theta,i}(x,x')$.
From Lemma \ref{lemma:ski_kernel_derivative_error_kernel}, we have:
\begin{equation}
\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta,i}(x, x')
\end{equation}
Thus, the $i$-th row of $\nabla_\theta K - \nabla_\theta \tilde{K}$ can be written as:
\begin{equation}
\frac{\partial \text{vec}(K)}{\partial \theta_i} - \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_i} = \text{vec}(K'_{\theta,i}) - \text{vec}(\tilde{K}'_{\theta,i}).
\end{equation}
Therefore, we can write:
\begin{equation}
\nabla_\theta K - \nabla_\theta \tilde{K} = \begin{bmatrix}
\text{vec}(K'_{\theta,1}) - \text{vec}(\tilde{K}'_{\theta,1}) \\
\vdots \\
\text{vec}(K'_{\theta,p}) - \text{vec}(\tilde{K}'_{\theta,p})
\end{bmatrix}^T
\end{equation}
Then, we have that:
\begin{align}
(\nabla_\theta K - \nabla_\theta \tilde{K})^T = \begin{bmatrix}
\text{vec}(K'_{\theta,1}) - \text{vec}(\tilde{K}'_{\theta,1}) &
\cdots &
\text{vec}(K'_{\theta,p}) - \text{vec}(\tilde{K}'_{\theta,p})
\end{bmatrix}
\end{align}
Now we bound the spectral norm of $(\nabla_\theta K - \nabla_\theta \tilde{K})^T$:
\begin{align*}
\| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_2 &\leq \| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_F \\
&= \left( \sum_{i=1}^p \| \text{vec}(K'_{\theta,i}) - \text{vec}(\tilde{K}'_{\theta,i}) \|_2^2 \right)^{1/2} \\
&= \left( \sum_{i=1}^p \| K'_{\theta,i} - \tilde{K}'_{\theta,i} \|_F^2 \right)^{1/2} \\
&\leq \left( \sum_{i=1}^p \left( \max_{j \in \{1,\ldots,p\}} \| K'_{\theta,j} - \tilde{K}'_{\theta,j} \|_2 \right)^2 \right)^{1/2}
\end{align*}
By Proposition \ref{prop:spectral-norm}, we have a bound on the spectral norm difference between a kernel matrix and its SKI approximation. Let $\gamma'_{n,m,L,i}$ be the corresponding bound for the kernel $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$. Then:
\begin{equation}
\| K'_{\theta,i} - \tilde{K}'_{\theta,i} \|_2 \leq \gamma'_{n,m,L,i}
\end{equation}
for each $i \in \{1, \ldots, p\}$. Thus,
\begin{align*}
\| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_2 &\leq \left( \sum_{i=1}^p (\gamma'_{n,m,L,i})^2 \right)^{1/2} \\
&\leq \left( \sum_{i=1}^p \left(\max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}\right)^2 \right)^{1/2} \\
&= \left( p \left(\max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}\right)^2 \right)^{1/2} \\
&= \sqrt{p} \max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}
\end{align*}
This completes the proof.
\end{proof}
\begin{lemma}[Corrected Bound on Gradient Difference]\label{lemma:gradient-difference-bound-corrected}
Let $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ be the true log-likelihood and $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$ be the SKI approximation of the log-likelihood. Let $\nabla \mathcal{L}(\theta)$ and $\nabla \tilde{\mathcal{L}}(\theta)$ denote their respective gradients with respect to $\theta$. Then, for any $\theta$,
$$
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 \leq \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right],
$$
where $\gamma_{n,m,L}$ bounds the elementwise difference between $\mathbf{K}$ and $\tilde{\mathbf{K}}$, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ and $C_1(\theta)$ are constants that depend on $\theta$ and the derivatives of the kernel function.
\end{lemma}
\begin{proof}
We start with the expressions for the gradients:
$$
\nabla \mathcal{L}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
$$
\nabla \tilde{\mathcal{L}}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
Thus, the difference is:
\begin{align*}
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 &= \left\| \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| \right) \right. \\
&\quad \left. - \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&\leq \underbrace{\left\| \nabla \left( \frac{1}{2} \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2}_{T_1} \\
&\quad + \underbrace{\left\| \frac{1}{2} \nabla \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2}_{T_2}.
\end{align*}
We will bound $T_1$ and $T_2$ separately.
**Bounding $T_1$:**
Using the chain rule and the matrix derivative identity $\frac{\partial}{\partial \theta} \mathbf{X}^{-1} = -\mathbf{X}^{-1} (\frac{\partial \mathbf{X}}{\partial \theta}) \mathbf{X}^{-1}$, we have
\begin{align*}
T_1 &= \frac{1}{2} \left\| \nabla_\theta \left( \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2 \\
&= \frac{1}{2} \left\| \sum_{i=1}^n \sum_{j=1}^n y_i y_j \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2 \\
&\leq \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n |y_i y_j| \left\| \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2.
\end{align*}
Now, applying the matrix derivative identity, we get:
\begin{align*}
&\left\| \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} + \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\right.\\
&\quad \left.\left. + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \nabla_\theta \mathbf{K} (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\right.\\
&\quad \left.\left. - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&\leq \underbrace{\left\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(a)} \\
&\quad + \underbrace{\left\| \left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right)(\nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(b)} \\
&\quad + \underbrace{\left\|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}(\nabla_\theta \mathbf{K})\left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right) \right\|_{2, }}_{(c)}\\
\end{align*}
We now explicitly bound (a), (b), and (c).
\begin{align*}
(a) &\leq \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2 \| \nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} \|_2 \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2\\
&\leq \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2^2 \| \nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} \|_2 \\
&\leq \frac{1}{\sigma^4} \| \nabla_\theta \mathbf{K} - \nabla_\theta \tilde{\mathbf{K}} \|_2\text{ Proposition }\ref{prop:gradient_spectral_norm_bound}\\
&\leq \frac{1}{\sigma^4}\sqrt{p} \max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}.
\end{align*}
\begin{align*}
(b) &\leq \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \|\nabla_\theta \mathbf{K}\|_2\text{ proof of Lemma \ref{lemma:action-inverse-error}}.
\end{align*}
\begin{align*}
(c) &\leq \|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \|\nabla_\theta \mathbf{K}\|_2\text{ proof of Lemma \ref{lemma:action-inverse-error}}.
\end{align*}
\amnote{We need some sort of assumption here, some kind of continuously differentiable}$\|\nabla_\theta \mathbf{K}\|$ can be bounded by a constant, say $C_1(\theta)$. Thus, we can bound $T_1$ as:
\begin{align*}
T_1 &\leq \sum_{i,j} \vert y_iy_j\vert \frac{1}{\sigma^4} \left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)\\
&\leq \frac{1}{\sigma^4}\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)\sum_{i,j} \vert y_iy_j\vert \\
&=\frac{\Vert y\Vert_1^2}{\sigma^4}\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)
\end{align*}
**Bounding $T_2$:**
Using the identity $\nabla \log |\mathbf{X}| = (\mathbf{X}^{-1})^\top$, we have
\begin{align*}
T_2 &= \frac{1}{2} \left\| \nabla_\theta \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&= \frac{1}{2} \left\| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_2.
\end{align*}
We can rewrite the difference as:
$$
(\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} = (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\tilde{\mathbf{K}} - \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}
$$
Then, applying Lemma \ref{lemma:action-inverse-error} we obtain
\begin{align*}
T_2 &\leq \frac{1}{2} \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2 \| \tilde{\mathbf{K}} - \mathbf{K} \|_2 \| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \|_2 \\
&\leq \frac{\gamma_{n,m,L}}{2\sigma^4}
\end{align*}
where $C_2(\theta)$ is another constant depending on $\theta$ (from the derivatives of the kernel when applying Lemma \ref{lemma:action-inverse-error}).
**Combining the Bounds:**
Combining the bounds for $T_1$ and $T_2$, we have
$$
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 \leq \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right].
$$
\end{proof}
\begin{proposition}[One-Step Improvement on True Log-Likelihood]\label{prop:one-step-improvement-on-true-log-likelihood}
Let $\theta^*\in \mathcal{D}$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient in $\mathcal{D}$.
If the step size $\eta$ satisfies $\eta \leq \frac{1}{L}$ (where $L$ is the Lipschitz constant of the gradient of the true log-likelihood) and $\eta\mu < 1$,
then starting at $\theta_0\in \mathcal{D}$, after one step of projected gradient ascent on $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$, yielding $\theta_1$, we have:
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2
$$
\end{proposition}
\begin{proof}
By the optimality of $\theta^*$, we have $\nabla \mathcal{L}(\theta^*) = 0$.
By the $\mu$-strong concavity of $\mathcal{L}$, we have for any $\theta$:
$$
\mathcal{L}(\theta) \leq \mathcal{L}(\theta^*) + \nabla \mathcal{L}(\theta^*)^\top (\theta - \theta^*) - \frac{\mu}{2} \| \theta - \theta^* \|_2^2 = \mathcal{L}(\theta^*) - \frac{\mu}{2} \| \theta - \theta^* \|_2^2
$$
Rearranging, we get:
\begin{equation} \label{eq:strong_concavity_alt}
\| \theta - \theta^* \|_2^2 \leq \frac{2}{\mu} (\mathcal{L}(\theta^*) - \mathcal{L}(\theta))
\end{equation}
Since $\theta_1$ is obtained by one step of projected gradient ascent from $\theta_0$, we have:
$$
\theta_1 = \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \mathcal{L}(\theta_0))
$$
By the properties of projection onto a convex set, we have:
$$
\| \theta_1 - \theta^* \|_2^2 \leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2^2
$$
Expanding the right-hand side, we get:
\begin{align*}
\| \theta_1 - \theta^* \|_2^2 &\leq \| \theta_0 - \theta^* \|_2^2 + 2 \eta (\theta_0 - \theta^*)^\top \nabla \mathcal{L}(\theta_0) + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \\
&\leq \| \theta_0 - \theta^* \|_2^2 + 2 \eta (\mathcal{L}(\theta_0) - \mathcal{L}(\theta^*) + \frac{\mu}{2} \| \theta_0 - \theta^* \|_2^2) + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \\
&\text{(by strong concavity, with } \nabla \mathcal{L}(\theta^*) = 0 \text{)}\\
&\leq \| \theta_0 - \theta^* \|_2^2 - \eta \mu \| \theta_0 - \theta^* \|_2^2 + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \quad \text{(by \eqref{eq:strong_concavity_alt})} \\
&= (1 - \eta \mu) \| \theta_0 - \theta^* \|_2^2 + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2
\end{align*}
By the Lipschitz continuity of the gradient, we have:
$$
\| \nabla \mathcal{L}(\theta_0) \|_2 = \| \nabla \mathcal{L}(\theta_0) - \nabla \mathcal{L}(\theta^*) \|_2 \leq L \| \theta_0 - \theta^* \|_2
$$
Thus,
$$
\| \theta_1 - \theta^* \|_2^2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2^2 + \eta^2 L^2 \| \theta_0 - \theta^* \|_2^2 = (1 - \eta \mu + \eta^2 L^2) \| \theta_0 - \theta^* \|_2^2
$$
Taking the square root of both sides:
$$
\| \theta_1 - \theta^* \|_2 \leq \sqrt{1 - \eta \mu + \eta^2 L^2} \| \theta_0 - \theta^* \|_2
$$
If we choose $\eta \leq \frac{1}{L}$, we have:
$$
\sqrt{1 - \eta \mu + \eta^2 L^2} \leq \sqrt{1-2\eta \mu+\eta^2\mu^2} = 1 - \eta \mu
$$
Thus,
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2
$$
as long as $\eta \leq \frac{1}{L}$.
\end{proof}
\begin{proposition}[MLE Closeness via One-Step Improvement]\label{prop:mle-closeness-one-step}
Let $\theta^*$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient near $\theta^*$. Assume that the gradient difference is bounded:
$$
\| \nabla \tilde{\mathcal{L}}(\theta) - \nabla \mathcal{L}(\theta) \|_2 \leq \epsilon_g:= \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\frac{1}{\sqrt{p}}\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right]
$$
for all $\theta\in \mathcal{D}$.
If the step size $\eta>0$ satisfies $\eta \leq \frac{1}{\tilde{L}}$ (where $L$ is the Lipschitz constant of the gradient of the log-likelihood) $\mu\eta\leq 1$:
$$
\| \theta_0 - \theta^* \|_2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2+\eta \epsilon_g
$$
Then, after one step of projected gradient ascent on $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$, yielding $\theta_1$, we have:
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2 + \eta \epsilon_g
$$
\end{proposition}
\begin{proof}
Let $\theta_1$ be the result of one step of projected gradient ascent on $\tilde{\mathcal{L}}$ from $\theta_0$ with step size $\eta$:
$$
\theta_1 = \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0))
$$
where $\mathcal{D}$ is a convex set containing $\theta^*$.
By the triangle inequality, we have:
\begin{align*}
\| \theta_1 - \theta^* \|_2 &= \| \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0)) - \theta^* \|_2 \\
&\leq \| \theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0) - \theta^* \|_2 \\
&= \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* + \eta (\nabla \tilde{\mathcal{L}}(\theta_0) - \nabla \mathcal{L}(\theta_0)) \|_2 \\
&\leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 + \eta \| \nabla \tilde{\mathcal{L}}(\theta_0) - \nabla \mathcal{L}(\theta_0) \|_2 \\
&\leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 + \eta \epsilon_g \quad \text{(by the gradient difference bound)}
\end{align*}
Now, let's consider the term $\| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2$. This represents the distance to $\theta^*$ after taking one step of gradient ascent on the true log-likelihood $\mathcal{L}$. By Proposition \ref{prop:one-step-improvement-on-true-log-likelihood}, if $\| \theta_0 - \theta^* \|_2 \leq \frac{\eta(2\mu - \eta L)}{2}$ and $\eta \leq \frac{1}{L}$, we have:
$$
\| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2
$$
Therefore,
\begin{align*}
\| \theta_1 - \theta^* \|_2 &\leq (1-\eta\mu)\|\theta_0-\theta^*\|_2+\eta\epsilon_g
\end{align*}
\end{proof}
\begin{theorem}[Closeness of SKI MLE to True MLE]\label{thm:mle-closeness-gradient-ascent}
Let $\theta^*$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient in $\mathcal{D}$. Assume that the gradient difference between the true log-likelihood and the SKI log-likelihood is bounded by
$$
\| \nabla \tilde{\mathcal{L}}(\theta) - \nabla \mathcal{L}(\theta) \|_2 \leq \epsilon_g := \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\frac{1}{\sqrt{p}}\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right]
$$
for all $\theta$ in a neighborhood of $\theta^*$, where $\gamma_{n,m,L}$, $\gamma'_{n,m,L,i}$ are defined as in Proposition \ref{prop:spectral-norm} and Proposition \ref{prop:gradient_spectral_norm_bound}, respectively, and $C$ is a bound on $C_1(\theta)$ in this neighborhood. Let $\tilde{\theta}$ be the SKI MLE obtained by running projected gradient ascent on $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$ from initialization $\theta_0$ with a fixed step size $\eta$.
If the step size $\eta$ satisfies $0 < \eta \leq \frac{1}{L}$ and $\eta \mu < 1$, then, after $t$ steps of projected gradient ascent, the iterate $\theta_t$ satisfies
$$
\| \theta_t - \theta^* \|_2 \leq (1 - \eta\mu)^t \| \theta_0 - \theta^* \|_2 + \frac{\epsilon_g}{\eta\mu}.
$$
\end{theorem}
\begin{proof}
Consider that starting from $\theta_t$ and applying one step of gradient ascent, by Proposition \ref{prop:mle-closeness-one-step}
\begin{align*}
\Vert \theta_{t+1}-\theta^*\Vert_2 &\leq (1-\eta\mu)\Vert \theta_t-\theta^*\Vert_2+\eta \epsilon_g
\end{align*}
and iterating we have
\begin{align*}
\Vert \theta_t-\theta^*\Vert_2&\leq (1-\eta\mu)^t\Vert \theta_0-\theta^*\Vert_2+\sum_{s=0}^{t-1} (1-\eta\mu)^s \eta \epsilon_g\\
&\leq (1-\eta\mu)^t\Vert \theta_0-\theta^*\Vert_2+\frac{\eta\epsilon_G}{\eta\mu}\text{ geometric series}
\end{align*}
\end{proof}
\amnote{Now we would like to show that for some kernel (RBF probably), there exists a region around true parameter such that the log-likelihood is w.h.p. both strongly concave and has L-Lipschitz continuous gradients.}
% \begin{lemma}[Negative Definiteness of the Hessian at the True Hyperparameters]\label{lemma:negative-definite-hessian}
% Under the assumptions of Theorem \ref{thm:local-strong-concavity-general}, the Hessian matrix <span class="math-inline">\\mathbf\{H\}\(\\theta\)</span> of the population log-likelihood <span class="math-inline">\\mathbb\{E\}\_\{P^\*\}\[\\mathcal\{L\}\(\\theta; \\mathbf\{X\}, \\mathbf\{Y\}\)\]</span> evaluated at the true hyperparameters <span class="math-inline">\\theta^\*</span> is negative definite.
% \end{lemma}
% \begin{proof}
% Let <span class="math-inline">\\bar\{\\mathcal\{L\}\}\(\\theta\) \= \\mathbb\{E\}\_\{P^\*\}\[\\mathcal\{L\}\(\\theta; \\mathbf\{X\}, \\mathbf\{Y\}\)\]</span> denote the population log-likelihood. The Hessian matrix <span class="math-inline">\\mathbf\{H\}\(\\theta\)</span> has entries given by:
% $$
% \mathbf{H}_{ij}(\theta) = \frac{\partial^2 \bar{\mathcal{L}}(\theta)}{\partial \theta_i \partial \theta_j}.
% $$
% We want to show that <span class="math-inline">\\mathbf\{H\}\(\\theta^\*\)</span> is negative definite. It is equivalent to showing that the negative Hessian, <span class="math-inline">\-\\mathbf\{H\}\(\\theta^\*\)</span>, which is also known as the *expected Fisher information matrix*, is positive definite.
% The log-likelihood for a set of observations <span class="math-inline">\\\{\(\\mathbf\{x\}\_i, y\_i\)\\\}\_\{i\=1\}^n</span> (assuming a zero mean function) is:
% $$
% \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\mathbf{K}_\theta + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi),
% $$
% where <span class="math-inline">\\mathbf\{K\}\_\\theta</span> is the kernel matrix evaluated at the points in <span class="math-inline">\\mathbf\{X\}</span>, and <span class="math-inline">\\sigma^2</span> is the noise variance.
% \textbf{1. First-Order Derivatives:**}
% The gradient of the log-likelihood has components:
% $$
% \frac{\partial \mathcal{L}}{\partial \theta_i} = \frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right).
% $$
% \textbf{2. Second-Order Derivatives:**}
% The second-order derivatives can be computed using matrix calculus identities. After some calculations (details in Appendix A), we arrive at the following expression:
% \begin{align*}
% \frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left[ \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \\
% &\quad + \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) \right] - \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) \right].
% \end{align*}
% \textbf{3. Population Log-Likelihood and Hessian:**}
% The population log-likelihood is <span class="math-inline">\\bar\{\\mathcal\{L\}\}\(\\theta\) \= \\mathbb\{E\}\_\{P^\*\}\[\\mathcal\{L\}\(\\theta; \\mathbf\{X\}, \\mathbf\{Y\}\)\]</span>. The Hessian of the population log-likelihood is:
% $$
% \mathbf{H}(\theta) = \mathbb{E}_{P^*}[\nabla^2_\theta \mathcal{L}(\theta; \mathbf{X}, \mathbf{y})],
% $$
% where <span class="math-inline">\\nabla^2\_\\theta</span> denotes the Hessian with respect to <span class="math-inline">\\theta</span>.
% \textbf{4. Expected Fisher Information:**}
% The expected Fisher information matrix is <span class="math-inline">\\mathbf\{I\}\(\\theta^\*\) \= \-\\mathbf\{H\}\(\\theta^\*\)</span>. We want to show that <span class="math-inline">\\mathbf\{I\}\(\\theta^\*\)</span> is positive definite.
% \textbf{5. Positive Definiteness:**}
% To show that <span class="math-inline">\\mathbf\{I\}\(\\theta^\*\)</span> is positive definite, we need to show that for any non-zero vector <span class="math-inline">\\mathbf\{v\} \\in \\mathbb\{R\}^p</span>:
% $$
% \mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0.
% $$
% Substituting <span class="math-inline">\\theta \= \\theta^\*</span> and taking the expectation with respect to <span class="math-inline">P^\*</span>, we get:
% $$
% \mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = -\mathbb{E}_{P^*}[\mathbf{v}^\top \nabla^2_\theta \mathcal{L}(\theta^*; \mathbf{X}, \mathbf{y}) \mathbf{v}].
% $$
% Since <span class="math-inline">\\mathbf\{Y\}\|\\mathbf\{X\} \\sim \\mathcal\{N\}\(\\mathbf\{0\}, \\mathbf\{K\}\_\{\\theta^\*\} \+ \\sigma^2 \\mathbf\{I\}\)</span>, we have <span class="math-inline">\\mathbb\{E\}\_\{P^\*\}\[\\mathbf\{Y\}\\mathbf\{Y\}^\\top \| \\mathbf\{X\}\] \= \\mathbf\{K\}\_\{\\theta^\*\} \+ \\sigma^2 \\mathbf\{I\}</span>. Using this, we can simplify the expectation of the quadratic form involving <span class="math-inline">\\mathbf\{Y\}</span> (details in Appendix B). After algebraic manipulation, we obtain:
% $$
% \mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
% $$
% where <span class="math-inline">\\mathbf\{A\} \= \\sum\_\{i\=1\}^p v\_i \\frac\{\\partial \\mathbf\{K\}\_\{\\theta^\*\}\}\{\\partial \\theta\_i\}</span>.
% \textbf{6. Using Properties of Trace and Positive Semidefiniteness:**}
% Now, note that <span class="math-inline">\(\\mathbf\{K\}\_\{\\theta^\*\} \+ \\sigma^2 \\mathbf\{I\}\)^\{\-1\}</span> is positive definite and thus has a positive definite square root. Let <span class="math-inline">\\mathbf\{B\} \= \(\\mathbf\{K\}\_\{\\theta^\*\} \+ \\sigma^2 \\mathbf\{I\}\)^\{\-1/2\}</span>. Then, we can rewrite the expression as:
% $$
% \mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( \mathbf{B} \mathbf{A} \mathbf{B} \mathbf{B} \mathbf{A} \mathbf{B} \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B}) (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right].
% $$
% Since <span class="math-inline">\(\\mathbf\{B\}\\mathbf\{A\}\\mathbf\{B\}\)^\\top \(\\mathbf\{B\}\\mathbf\{A\}\\mathbf\{B\}\)</span> is positive semidefinite, its trace is non-negative. Moreover, the trace is zero if and only if <span class="math-inline">\\mathbf\{B\}\\mathbf\{A\}\\mathbf\{B\} \= \\mathbf\{0\}</span>.
% \end{proof}
\begin{theorem}[Local Strong Concavity of Population Log-Likelihood - RBF Kernel, Restricted Parameter Space]\label{thm:local-strong-concavity-restricted}
Let $P^*$ be the true data generating distribution, and let $\theta^* = (\sigma_{f*}, \ell_*)$ be the maximizer of the population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ for a Gaussian process with an RBF kernel:
$$
k_\theta(x, x') = \sigma_f^2 \exp\left(-\frac{\|x - x'\|_2^2}{2\ell^2}\right),
$$
where $\theta = (\sigma_f, \ell)$ are the signal variance and lengthscale parameters, respectively.
Assume the following:
1. **Restricted Parameter Space:** We restrict the parameter space to a compact set $\Theta$ such that for all $\theta \in \Theta$, we have $\sigma_f \in [\sigma_{f, min}, \sigma_{f, max}]$ and $\ell \in [\ell_{min}, \ell_{max}]$, where $0 < \sigma_{f, min} < \sigma_{f, max} < \infty$ and $0 < \ell_{min} < \ell_{max} < \infty$. Furthermore, $\theta^* \in \Theta$.
2. **Smoothness and Boundedness:**
* The true data generating process is of the form $y_i = \xi(x_i) + \epsilon_i$, where $\xi \sim GP(0, k_{\theta^*})$ and $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are i.i.d. noise.
* The marginal distribution of the inputs $P^*_X$ has a density $p(x)$ with respect to the Lebesgue measure on $\mathbb{R}^d$ that is bounded away from zero on a compact set $\mathcal{X} \subset \mathbb{R}^d$, i.e., there exists $\rho > 0$ such that $p(x) \geq \rho$ for all $x \in \mathcal{X}$.
* The function $\xi$ and its first and second partial derivatives are bounded on $\mathcal{X}$.
3. **Regularity of the RBF Kernel:** The RBF kernel $k_\theta(x,x')$ and its first, second, and third partial derivatives with respect to $\theta$ are continuous and bounded on $\mathcal{X} \times \mathcal{X}$ for $\theta \in \Theta$.
Then, there exists a neighborhood $\mathcal{D} \subseteq \Theta$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the following hold:
* **(a) Local Strong Concavity:** The population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu$ within $\mathcal{D}$, i.e., for all $\theta_1, \theta_2 \in \mathcal{D}$:
$$
\mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{y})] \leq \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})] + \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})]^\top (\theta_2 - \theta_1) - \frac{\mu}{2} \| \theta_2 - \theta_1 \|_2^2.
$$
* **(b) Lipschitz Gradient:** The gradient of the population log-likelihood $\nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is Lipschitz continuous with constant $L$ within $\mathcal{D}$, i.e., for all $\theta_1, \theta_2 \in \mathcal{D}$:
$$
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{Y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{Y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2.
$$
\end{theorem}
\begin{proof}
The proof follows a similar structure to the previous attempts, but crucially, we now work within a restricted parameter space.
**1. Define the Population Log-Likelihood:**
As before, let $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{Y})]$ denote the population log-likelihood.
**2. Consider the Hessian:**
Let $\mathbf{H}(\theta)$ be the Hessian of $\bar{\mathcal{L}}(\theta)$. We will show that $\mathbf{H}(\theta^*)$ is negative definite.
**3. Expected Fisher Information:**
The expected Fisher information matrix is $\mathbf{I}(\theta^*) = -\mathbf{H}(\theta^*)$. We will show that $\mathbf{I}(\theta^*)$ is positive definite.
**4. Positive Definiteness of $I(\theta^*)$:**
We want to show that for any non-zero vector $\mathbf{v} \in \mathbb{R}^2$, $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0$. Following the same steps as in the previous attempt, we arrive at:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
$$
where $\mathbf{A} = v_1 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \sigma_f^2} + v_2 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \ell}$.
As before, we can rewrite this as:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right],
$$
where $\mathbf{B} = (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2}$.
**5. Key Argument - Restricted Parameter Space and Linear Independence:**
Now, we leverage the restricted parameter space $\Theta$. Since $\sigma_f$ is bounded away from 0 and $\ell$ is bounded away from both 0 and $\infty$, the eigenvalues of $\mathbf{K}_{\theta^*}$ are bounded away from 0 and $\infty$. This implies that $\mathbf{B}$ and $\mathbf{B}^{-1}$ are well-defined and have bounded norms.
Suppose for contradiction that $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = 0$ for some non-zero $\mathbf{v}$. This implies that $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$ almost surely, and consequently, $\mathbf{A} = \mathbf{0}$ almost surely.
Thus, we have:
$$
v_1 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \sigma_f^2} + v_2 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \ell} = \mathbf{0} \quad \text{a.s.}
$$
Now, we need to show that this is impossible for non-zero $\mathbf{v}$ under our assumptions. We can write this as:
$$
v_1 \frac{\partial k_{\theta^*}}{\partial \sigma_f^2}(x_i,x_j) + v_2 \frac{\partial k_{\theta^*}}{\partial \ell}(x_i,x_j) = 0
$$
for all $x_i,x_j$ in the support of $P_X^*$.
Using the derivatives of the RBF kernel that we computed earlier, this becomes:
$$
v_1 \exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right) + v_2 \sigma_{f*}^2 \exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right) \left(\frac{\|x_i - x_j\|_2^2}{\ell_*^3}\right) = 0
$$
for all $x_i, x_j$ in the support of $P_X^*$.
Dividing by $\exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right)$ (which is always non-zero), we get:
$$
v_1 + v_2 \sigma_{f*}^2 \frac{\|x_i - x_j\|_2^2}{\ell_*^3} = 0
$$
Since $\sigma_{f*} > 0$ and $\ell_* > 0$, the only way for this to hold for all $x_i, x_j$ in a set of non-zero measure under $P^*_X$ is if $v_1 = v_2 = 0$.
This is because if $v_2 \neq 0$, then the term $\frac{\|x_i - x_j\|_2^2}{\ell_*^3}$ can be varied continuously by changing $x_i$ and $x_j$, which makes it impossible for the equation to hold unless $v_1$ is also zero.
Thus, we have reached a contradiction. Therefore, for any non-zero $\mathbf{v}$, we must have $\mathbf{A} \neq \mathbf{0}$ almost surely, and hence $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0$.
**6. Negative Definite Hessian and Strong Concavity:**
Since $\mathbf{I}(\theta^*)$ is positive definite, $\mathbf{H}(\theta^*)$ is negative definite. By continuity of the Hessian (due to the regularity assumption on the kernel), there exists a neighborhood $\mathcal{D} \subseteq \Theta$ of $\theta^*$ such that $\mathbf{H}(\theta)$ is negative definite for all $\theta \in \mathcal{D}$. Moreover, there exists a constant $\mu > 0$ such that $-\mathbf{H}(\theta) \succeq \mu \mathbf{I}$ for all $\theta \in \mathcal{D}$. This implies that the population log-likelihood is strongly concave with parameter $\mu$ within $\mathcal{D}$.
**7. Lipschitz Continuity of the Gradient:**
The Lipschitz continuity of the gradient follows from the boundedness of the third-order derivatives of the population log-likelihood in the neighborhood $\mathcal{D}$. This can be shown using the regularity assumptions on the kernel (Assumption 3) and the boundedness assumption (Assumption 2). Since the third-order derivatives are bounded, there exists a constant $L < \infty$ such that:
$$
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{Y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{Y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2
$$
for all $\theta_1, \theta_2 \in \mathcal{D}$.
**Conclusion:**
We have shown that under the given assumptions, there exists a neighborhood $\mathcal{D}$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the population log-likelihood is strongly concave with parameter $\mu$ and has a Lipschitz continuous gradient with constant $L$ within $\mathcal{D}$. This completes the proof.
\end{proof}
\begin{theorem}[Local Strong Concavity of Population Log-Likelihood - General Case]\label{thm:local-strong-concavity-general}
Let $P^*$ be the true data generating distribution, and let $\theta^*$ be the maximizer of the population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ for a Gaussian process with kernel $k_\theta(x, x')$.
Assume the following:
1. **Identifiability:** The hyperparameters $\theta^*$ are identifiable from the data generating distribution $P^*$.
2. **Smoothness and Boundedness:**
* The true data generating process is of the form $y_i = \xi(x_i) + \epsilon_i$, where $\xi \sim GP(0, k_{\theta^*})$ and $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are i.i.d. noise.
* The marginal distribution of the inputs $P^*_X$ has a density $p(x)$ with respect to the Lebesgue measure on $\mathbb{R}^d$ that is bounded away from zero on a compact set $\mathcal{X} \subset \mathbb{R}^d$.
* The function $\xi$ and its first and second partial derivatives are bounded on $\mathcal{X}$.
3. **Regularity of the Kernel:** The kernel $k_\theta(x,x')$ and its first, second, and third partial derivatives with respect to $\theta$ are continuous and bounded on $\mathcal{X} \times \mathcal{X}$ for $\theta$ in a neighborhood of $\theta^*$.
4. **Linear Independence of Derivatives:** For any non-zero vector $\mathbf{v}$ of the same dimension as $\theta$, the following holds almost surely with respect to $P^*_X$:
$$
\sum_{i} v_i \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \neq \mathbf{0}
$$
where $\mathbf{K}_\theta$ is the kernel matrix evaluated at a set of input points drawn from $P^*_X$.
Then, there exists a neighborhood $\mathcal{D}$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the following hold:
* **(a) Local Strong Concavity:** The population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu$ within $\mathcal{D}$.
* **(b) Lipschitz Gradient:** The gradient of the population log-likelihood $\nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is Lipschitz continuous with constant $L$ within $\mathcal{D}$.
\end{theorem}
\begin{proof}
Let $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ denote the population log-likelihood. The Hessian matrix $\mathbf{H}(\theta)$ has entries given by:
$$
\mathbf{H}_{ij}(\theta) = \frac{\partial^2 \bar{\mathcal{L}}(\theta)}{\partial \theta_i \partial \theta_j}.
$$
We want to show that $\mathbf{H}(\theta^*)$ is negative definite. It is equivalent to showing that the negative Hessian, $-\mathbf{H}(\theta^*)$, which is also known as the *expected Fisher information matrix*, is positive definite.
The log-likelihood for a set of observations $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$ (assuming a zero mean function) is:
$$
\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\mathbf{K}_\theta + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi),
$$
where $\mathbf{K}_\theta$ is the kernel matrix evaluated at the points in $\mathbf{X}$, and $\sigma^2$ is the noise variance.
\textbf{1. First-Order Derivatives:**}
The gradient of the log-likelihood has components:
$$
\frac{\partial \mathcal{L}}{\partial \theta_i} = \frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right).
$$
\textbf{2. Second-Order Derivatives:**}
The second-order derivatives can be computed using matrix calculus identities. After some calculations (details in Appendix A), we arrive at the following expression:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left[ \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \\
&\quad + \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) \right] - \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) \right].
\end{align*}
\textbf{3. Population Log-Likelihood and Hessian:**}
The population log-likelihood is $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$. The Hessian of the population log-likelihood is:
$$
\mathbf{H}(\theta) = \mathbb{E}_{P^*}[\nabla^2_\theta \mathcal{L}(\theta; \mathbf{X}, \mathbf{y})],
$$
where $\nabla^2_\theta$ denotes the Hessian with respect to $\theta$.
\textbf{4. Expected Fisher Information:**}
The expected Fisher information matrix is $\mathbf{I}(\theta^*) = -\mathbf{H}(\theta^*)$. We want to show that $\mathbf{I}(\theta^*)$ is positive definite.
\textbf{5. Positive Definiteness:**}
To show that $\mathbf{I}(\theta^*)$ is positive definite, we need to show that for any non-zero vector $\mathbf{v} \in \mathbb{R}^2$:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0.
$$
Substituting $\theta = \theta^*$ and taking the expectation with respect to $P^*$, we get:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = -\mathbb{E}_{P^*}[\mathbf{v}^\top \nabla^2_\theta \mathcal{L}(\theta^*; \mathbf{X}, \mathbf{y}) \mathbf{v}].
$$
Since $\mathbf{y}|\mathbf{X} \sim \mathcal{N}(\mathbf{0}, \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})$, we have $\mathbb{E}_{P^*}[\mathbf{y}\mathbf{y}^\top | \mathbf{X}] = \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I}$. Using this, we can simplify the expectation of the quadratic form involving $\mathbf{y}$ (details in Appendix B). After algebraic manipulation, we obtain:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
$$
where $\mathbf{A} = \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}$.
\textbf{6. Using Properties of Trace and Positive Semidefiniteness:**}
Now, note that $(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}$ is positive definite and thus has a positive definite square root. Let $\mathbf{B} = (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2}$. Then, we can rewrite the expression as:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( \mathbf{B} \mathbf{A} \mathbf{B} \mathbf{B} \mathbf{A} \mathbf{B} \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B}) (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right].
$$
Since $(\mathbf{B}\mathbf{A}\mathbf{B})^\top (\mathbf{B}\mathbf{A}\mathbf{B})$ is positive semidefinite, its trace is non-negative. Moreover, the trace is zero if and only if $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$.
\textbf{7. Using Identifiability and Smoothness:**}
Here, we use the identifiability assumption. If $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$, then $(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2} = \mathbf{0}$, which implies $\mathbf{A} = \mathbf{0}$ (since the inverse square root is invertible).
$\mathbf{A} = \mathbf{0}$ implies:
$$
\sum_{i=1}^p v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}=\mathbf{0}
$$
This means that a linear combination of the derivatives of the kernel matrix with respect to the hyperparameters is zero. However, under the identifiability assumption and the smoothness of the data distribution, this can only happen if $v_1 = v_2 = 0$ (i.e., $\mathbf{v}$ is the zero vector). If different hyperparameters always lead to different predictive distributions, then their derivatives must be linearly independent.
Therefore, for any non-zero $\mathbf{v}$, we have $\mathbf{B}\mathbf{A}\mathbf{B} \neq \mathbf{0}$, and thus:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right] > 0.
$$
This shows that $\mathbf{I}(\theta^*)$ is positive definite, and consequently, $\mathbf{H}(\theta^*)$ is negative definite.
\textbf{Step 8: Establish Lipschitz continuity of the gradient.}
The Lipschitz continuity of the gradient follows from the boundedness of the third-order derivatives of the population log-likelihood in the neighborhood $\mathcal{D}$. This can be shown using the regularity assumptions on the kernel (Assumption 3) and the boundedness assumption (Assumption 2). Since the third-order derivatives are bounded, there exists a constant $L < \infty$ such that
\[
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2
\]
for all $\theta_1, \theta_2 \in \mathcal{D}$.
\end{proof}
% \begin{itemize}
% \item Need: true log-likelihood locally strongly concave. The log-likelihood is given by
% \begin{align*}
% \mathcal{L}(\theta;X)&=\log p(y|X)\\
% &=-\frac{1}{2}y^\top K_n^{-1}y-\frac{1}{2}\log \vert K_n\vert -\frac{N}{2}\log (2\pi)
% \end{align*}
% where and locally strongly concave means that the Hessian's smallest eigenvalue has a positive lower bound.
% \begin{align*}
% f(
% \end{align*}
% \item Then if log-likelihoods are close (they are since actions and log-determinants are close), the learned parameters are close.
% \end{itemize}
\subsection{Error of GP Predictive Posterior Parameters}
\subsubsection{Mean}
\amnote{Wrong now that it's a set of test points instead of a single test point.}
\begin{lemma}
Let $\boldsymbol{\mu}(\cdot)$ be the GP posterior mean at a set of test points $\cdot$ and $\tilde{\boldsymbol{\mu}}(\cdot)$ be the SKI posterior mean at those points. Then the SKI posterior mean $l^2$ error is bounded by:%\(\mathbf{K}_{\cdot, \mathbf{X}}\left(\mathbf{K}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\), \(\tilde{\mathbf{K}}_{\cdot, \mathbf{X}}\left(\tilde{\mathbf{K}}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\)
\begin{align*}
\Vert \tilde{\boldsymbol{\mu}}(\cdot)- \boldsymbol{\mu}(\cdot)\Vert_2&\leq \frac{1}{\sigma^2}\left(\min(\gamma_{T,m,L},\gamma_{n,m,L})+\frac{M}{\sigma^2}\gamma_{n,m,L}\right)
\end{align*}
% \[
% \left\|\mathbf{K}_{\cdot, \mathbf{X}}\left(\mathbf{K}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}-\tilde{\mathbf{K}}_{\cdot, \mathbf{X}}\left(\tilde{\mathbf{K}}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\right\|_{2} \leq \frac{1}{\sigma^2}\left(\min(\gamma_{T,m,L},\gamma_{n,m,L})+\frac{M}{\sigma^2}\gamma_{n,m,L}\right)
% \]
\end{lemma}
\begin{proof}
We start by expressing the difference between the true and SKI posterior means:
\begin{align*}
& \left\|\mathbf{K}_{\cdot,\mathbf{X}} \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} \right\|_{2} \\
&= \left\| \left(\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \right) \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} + \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left[ \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right] \mathbf{y} \right\|_{2}
\end{align*}
Applying the triangle inequality gives:
\begin{align*}
& \leq \left\| \left(\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \right) \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \| \mathbf{y} \|_{2} + \left\| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right) \mathbf{y} \right\|_{2} \\
&\leq \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} \left\| \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \| \mathbf{y} \|_{2} + \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} \left\| \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \cdot \| \mathbf{y} \|_{2} \\
& \leq \frac{ \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{2} } \| \mathbf{y} \|_{2} + \gamma_{n, m, L}\frac{ \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{4} } \| \mathbf{y} \|_{2} \textrm{ by Lemma \ref{lemma:action-inverse-error}}\\
& \leq \left( \frac{ \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{2} } + \frac{ \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{4} } \gamma_{n, m, L} \right) \| \mathbf{y} \|_{2}\\
&\leq \left(\frac{\max(\gamma_{T,m,L},\gamma_{n,m,L})}{\sigma^2}+\frac{\Vert \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \Vert_2}{\sigma^4}\gamma_{n,m,L}\right)\Vert \textbf{y}\Vert_2
\end{align*}
\end{proof}
\begin{lemma}[Bound on the Error of SKI Posterior Covariance]
Let $\boldsymbol{\Sigma}(\cdot)$ be the GP posterior covariance matrix at a set of test points $\cdot\in \mathbb{R}^{T\times d}$ and $\tilde{\boldsymbol{\Sigma}}(\cdot)$ be its SKI approximation. Assume that the true kernel satisfies the condition that $|k(\mathbf{x}, \mathbf{x}')| \leq M$ for all $\mathbf{x}, \mathbf{x}'$ in the domain of interest, so that $\Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}M$. Then
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \gamma_{T,m,L} + \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L})\\
&+ \frac{\gamma_{n,m,L}}{\sigma^4}Tn c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}),
\end{align*}
where $\gamma_{T,m,L}$ and $\gamma_{n,m,L}$ are defined as in Proposition \ref{prop:spectral-norm}, $c$ is an upper bound on the absolute sum of the weights for 1D kernel interpolation, $m$ is the number of inducing points, and $\sigma^2$ is the observation noise variance.
\end{lemma}
\begin{proof}
First, note that
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \Vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\Vert_2\\
&+ \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \gamma_{T,m,L} + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2,
\end{align*}
where we used Proposition \ref{prop:spectral-norm} and the fact that $\Vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\Vert_2 \leq \gamma_{T,m,L}$.
Now, we bound the second term:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&+ \Vert\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}(\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot})\Vert_2 + \Vert (\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1})\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert (\textbf{K}+\sigma^2 I)^{-1} \Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2\\
&\leq \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2,
\end{align*}
where we used the fact that $(\textbf{K}+\sigma^2 I)^{-1} \preceq \frac{1}{\sigma^2}I$.
Next, we bound the term $\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2$:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \\
&= \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1} - \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} + \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1} - \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&= \Vert \textbf{K}_{\cdot,\textbf{X}} [(\textbf{K}+\sigma^2 I)^{-1} - (\tilde{\textbf{K}}+\sigma^2 I)^{-1}] \Vert_2 + \Vert (\textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}}) (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert (\textbf{K}+\sigma^2 I)^{-1} - (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \Vert (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \frac{\gamma_{n,m,L}}{\sigma^4} + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \frac{1}{\sigma^2},
\end{align*}
where we used Lemma \ref{lemma:action-inverse-error} in the last inequality.
Substituting this back into the main inequality, we get:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \left(\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \frac{\gamma_{n,m,L}}{\sigma^4} + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \frac{1}{\sigma^2}\right) \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2\\
&= \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \frac{\gamma_{n,m,L}}{\sigma^4}\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 + \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2.
\end{align*}
Using Lemma \ref{lemma:test-train-kernel-matrix-error} and the fact that $\Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \leq \max(\gamma_{T,m,L},\gamma_{n,m,L})$ and that $\textbf{K}_{\cdot, \textbf{X}} = \textbf{K}_{\textbf{X},\cdot}^\top$, we have $\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 = \Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2$. Also, by assumption, $\Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}M$. Using Lemma \ref{lemma:ski-test-train-bound}, we have $\Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}mc^{2d}M$. Substituting these bounds, we get:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) + \frac{\gamma_{n,m,L}}{\sigma^4}(\sqrt{Tn}M)(\sqrt{Tn} m c^{2d} M) + \frac{1}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) (\sqrt{Tn} m c^{2d} M) \\
&= \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) + \frac{\gamma_{n,m,L}}{\sigma^4}Tn m c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}).
\end{align*}
Finally, substituting this back into the original inequality, we obtain the desired bound:
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \gamma_{T,m,L} + \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L})\\
&+ \frac{\gamma_{n,m,L}}{\sigma^4}Tn m c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}).
\end{align*}
This completes the proof.
\end{proof}
% \subsubsection{Variance}
% \begin{lemma}
% Let $\boldsymbol{\Sigma}(\cdot)$ be the GP posterior covariance matrix at a set of test points $\cdot\in \mathbb{R}^{T\times d}$ and $\tilde{\boldsymbol{\Sigma}}(\cdot)$ be its SKI approximation. Then
% \begin{align*}
% \Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert&\leq
% \end{align*}
% % \begin{align*}
% % \vert \textbf{K}_{\cdot,\cdot}+\sigma^2I-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-(\tilde{\textbf{K}}_{\cdot,\cdot}+\sigma^2I-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot})\vert &\leq
% % \end{align*}
% \end{lemma}
% \begin{proof}
% First note that
% \begin{align*}
% \Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert&\leq \vert \textbf{K}_{\cdot,\cdot}+\sigma^2I-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-(\tilde{\textbf{K}}_{\cdot,\cdot}+\sigma^2I-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot})\vert \\
% &\leq\vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\vert+\vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\vert\\
% &\leq \gamma_{T,m,L}+\vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\vert\textrm{ Lemma \ref{proposition:spectral-norm}}
% \end{align*}
% Isolating the 2nd term, we have
% \begin{align*}
% &\vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\vert\\
% &\leq \vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}+\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\vert\Vert \textbf{y}\Vert_2
% \end{align*}
% \amnote{Just to simplify}
% \begin{align*}
% \vert abc-def\vert &=\vert abc-abf+abf-def\vert
% \end{align*}
% \end{proof}
% The GP posterior covariance at a test point is
% \begin{align*}
% k(\cdot,\cdot)+\sigma^2-K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}K(\cdot,x)
% \end{align*}
% and the SKI one is
% \begin{align*}
% \tilde{k}(\cdot,\cdot)+\sigma^2-\tilde{K}_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}(\cdot,x).
% \end{align*}
% Then
% \begin{align*}
% &\vert K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}K_{\cdot,x}-\tilde{K}_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}\vert\\
% &=\vert K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}K_{\cdot,x}-K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}+K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}\\
% &\qquad-K_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}+K_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}-\tilde{K}_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}\vert\\
% &\leq \vert K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}K_{\cdot,x}-K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}\vert +\vert K_{\cdot,x}^\top (K+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}-K_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}\vert\\
% &\qquad+\vert K_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}-\tilde{K}_{\cdot,x}^\top (\tilde{K}+\sigma^2 I)^{-1}\tilde{K}_{\cdot,x}\vert\\
% &\leq \Vert K_{\cdot,x}\Vert_2 \Vert (K+\sigma^2 I)^{-1}\Vert_2\Vert K_{\cdot,x}-\tilde{K}_{\cdot,x}\Vert_2+\Vert K_{\cdot,x}-\tilde{K}_{\cdot,x}\Vert_2\Vert \tilde{K}_{\cdot,x}\Vert\Vert (\tilde{K}+\sigma^2 I)^{-1}\Vert\\
% &\qquad+\Vert K_{\cdot,x}\Vert \Vert \Vert (\tilde{K}+\sigma^2 I)^{-1}-(K+\sigma^2 I)^{-1}\Vert \Vert \tilde{K}_{\cdot,x}\Vert\\
% &\leq \frac{1}{\sigma^2}\Vert K_{\cdot,x}\Vert_2 \Vert K_{\cdot,x}-\tilde{K}_{\cdot,x}\Vert_2+\frac{1}{\sigma^2}\Vert K_{\cdot,x}-\tilde{K}_{\cdot,x}\Vert_2\Vert \tilde{K}_{\cdot,x}\Vert\\
% &\qquad+\frac{1}{\sigma^4}\gamma_{n,m,L}\Vert K_{\cdot,x}\Vert \Vert\Vert \tilde{K}_{\cdot,x}\Vert
% \end{align*}
% Note that
% \begin{align*}
% \tilde{K}_{\cdot,x}&=WK_{\textbf{U}} w(\textbf{x})
% \end{align*}
% % \amnote{some scribbles to help me understand}
% % \begin{align*}
% % \vert x^\top A x-\tilde{x}^\top \tilde{A}\tilde{x}\vert &=\vert x^\top Ax-x^\top A\tilde{x}+x^\top A\tilde x-\tilde{x}^\top \tilde{A} \tilde{x}\vert\\
% % &=\vert x^\top A x-x^\top A\tilde{x}+x^\top A \tilde{x}-x^\top \tilde{A}\tilde{x}+x^\top \tilde{A}\tilde{x}-\tilde{x}^\top \tilde{A}\tilde{x}\vert\\
% % &\leq \Vert x^\top A\Vert \Vert x-\tilde{x}\Vert+\Vert x\Vert \Vert A-\tilde{A}\Vert \Vert \tilde{x}\Vert +\Vert x-\tilde{x}\Vert\Vert \tilde{A}\tilde{x}\Vert
% % \end{align*}
% \section{Discussion}\label{sec:discussion}
% \section{KL Divergence}
% \begin{lemma}
% \textbf{Bound on the KL Divergence Between True and SKI Predictive Posteriors}
% Let \( \mathcal{P} = \mathcal{N}(\boldsymbol{\mu}_P, \boldsymbol{\Sigma}_P) \) be the true Gaussian Process (GP) posterior and \( \mathcal{Q} = \mathcal{N}(\boldsymbol{\mu}_Q, \boldsymbol{\Sigma}_Q) \) be the SKI-approximated GP posterior. Assume that the kernel approximation satisfies
% \[
% \|\boldsymbol{\Sigma}_P - \boldsymbol{\Sigma}_Q\|_2 \leq \gamma_{n, m, L}
% \]
% and
% \[
% \|\boldsymbol{\mu}_P - \boldsymbol{\mu}_Q\|_2 \leq \delta_{\mu},
% \]
% where \( \gamma_{n, m, L} \) and \( \delta_{\mu} \) are bounds derived from the SKI approximation error.
% Then, the Kullback-Leibler (KL) divergence between \( \mathcal{P} \) and \( \mathcal{Q} \) is bounded as
% \[
% \text{KL}(\mathcal{P} \| \mathcal{Q}) \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\lambda_{\min}(\boldsymbol{\Sigma}_Q)} + \frac{\delta_{\mu}^2}{\lambda_{\min}(\boldsymbol{\Sigma}_Q)} \right),
% \]
% where \( \lambda_{\min}(\boldsymbol{\Sigma}_Q) \) denotes the smallest eigenvalue of \( \boldsymbol{\Sigma}_Q \).
% \end{lemma}
% \begin{proof}
% The KL divergence between two multivariate Gaussian distributions \( \mathcal{P} = \mathcal{N}(\boldsymbol{\mu}_P, \boldsymbol{\Sigma}_P) \) and \( \mathcal{Q} = \mathcal{N}(\boldsymbol{\mu}_Q, \boldsymbol{\Sigma}_Q) \) is given by
% \[
% \text{KL}(\mathcal{P} \| \mathcal{Q}) = \frac{1}{2} \left( \log \frac{\det \boldsymbol{\Sigma}_Q}{\det \boldsymbol{\Sigma}_P} - k + \text{tr}\left(\boldsymbol{\Sigma}_Q^{-1} \boldsymbol{\Sigma}_P\right) + (\boldsymbol{\mu}_Q - \boldsymbol{\mu}_P)^\top \boldsymbol{\Sigma}_Q^{-1} (\boldsymbol{\mu}_Q - \boldsymbol{\mu}_P) \right),
% \]
% where \( k \) is the dimensionality of the Gaussian distributions.
% \textbf{Bounding the Determinant Ratio and Trace Term:}
% Using the matrix inequality for positive definite matrices, we have
% \[
% \log \frac{\det \boldsymbol{\Sigma}_Q}{\det \boldsymbol{\Sigma}_P} \leq \frac{\text{tr}(\boldsymbol{\Sigma}_Q^{-1} \boldsymbol{\Sigma}_P)}{k} - 1.
% \]
% Additionally, since \( \|\boldsymbol{\Sigma}_P - \boldsymbol{\Sigma}_Q\|_2 \leq \gamma_{n, m, L} \), we can bound the trace term as
% \[
% \text{tr}\left(\boldsymbol{\Sigma}_Q^{-1} \boldsymbol{\Sigma}_P\right) \leq \frac{\gamma_{n, m, L}}{\lambda_{\min}(\boldsymbol{\Sigma}_Q)} + k.
% \]
% \textbf{Bounding the Quadratic Term:}
% For the mean difference term, by the Cauchy-Schwarz inequality,
% \[
% (\boldsymbol{\mu}_Q - \boldsymbol{\mu}_P)^\top \boldsymbol{\Sigma}_Q^{-1} (\boldsymbol{\mu}_Q - \boldsymbol{\mu}_P) \leq \frac{\|\boldsymbol{\mu}_Q - \boldsymbol{\mu}_P\|_2^2}{\lambda_{\min}(\boldsymbol{\Sigma}_Q)} = \frac{\delta_{\mu}^2}{\lambda_{\min}(\boldsymbol{\Sigma}_Q)}.
% \]
% \textbf{Combining the Bounds:}
% Substituting the bounds into the KL divergence expression,
% \[
% \text{KL}(\mathcal{P} \| \mathcal{Q}) \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\lambda_{\min}(\boldsymbol{\Sigma}_Q)} + \frac{\delta_{\mu}^2}{\lambda_{\min}(\boldsymbol{\Sigma}_Q)} \right).
% \]
% Thus, the KL divergence is bounded as stated.
% \end{proof}
% \amnote{Some chatGPT notes}
% The Kullback-Leibler (KL) divergence between two multivariate Gaussian distributions \( P = \mathcal{N}(\boldsymbol{\mu}_P, \boldsymbol{\Sigma}_P) \) and \( Q = \mathcal{N}(\boldsymbol{\mu}_Q, \boldsymbol{\Sigma}_Q) \) quantifies the difference between them and is defined as
% \[
% \text{KL}(P \parallel Q) = \frac{1}{2} \left( \ln\left(\frac{\det \boldsymbol{\Sigma}_Q}{\det \boldsymbol{\Sigma}_P}\right) - k + \text{tr}\left(\boldsymbol{\Sigma}_Q^{-1} \boldsymbol{\Sigma}_P\right) + (\boldsymbol{\mu}_Q - \boldsymbol{\mu}_P)^\top \boldsymbol{\Sigma}_Q^{-1} (\boldsymbol{\mu}_Q - \boldsymbol{\mu}_P) \right),
% \]
% where k is the dimensionality of the Gaussian distributions. This expression accounts for differences in both the mean vectors and the covariance matrices of P and Q . The first term involving the determinants measures the discrepancy in the volume or spread of the distributions, the trace term assesses the alignment of the covariance structures, and the quadratic term evaluates the distance between the means relative to the covariance of Q . Overall, the KL divergence provides a comprehensive metric for how one Gaussian distribution diverges from another.
% \section{Things to Study}
% \begin{enumerate}
% \item Linear interpolation: technique, geometric intuition, how it's derived, error bound.
% \item Same for polynomial interpolation.
% \item Then multivariate linear interpolation.
% \end{enumerate}
\bibliography{main}
\bibliographystyle{icml2021}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% DELETE THIS PART. DO NOT PLACE CONTENT AFTER THE REFERENCES!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix
\section{Auxiliary Technical Results}
\begin{lemma}\label{lemma:switch-sum-product}
Given a function $f: \mathbb{R}^d \rightarrow \mathbb{R}$ of the form $f(x_1, x_2, ..., x_d) = \prod_{j=1}^d f_j(x_j)$, where each $f_j: \mathbb{R} \rightarrow \mathbb{R}$. Let $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$ be a fixed d-dimensional grid, where each $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is a finite set of $n_j$ grid points along the j-th dimension for $j = 1, 2, ..., d$. Then the following equality holds:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d f_j(p_{k_j}^{(j)}) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\end{lemma}
\begin{proof}
\textbf{By Induction on d (the number of dimensions):}
\textbf{Base Case (d = 1):}
When $d=1$, the statement becomes:
$$
\sum_{k_1=1}^{n_1} f_1(p_{k_1}^{(1)}) = \sum_{k_1=1}^{n_1} f_1(p_{k_1}^{(1)})
$$
This is trivially true.
\textbf{Inductive Hypothesis:}
Assume the statement holds for $d = m$, i.e.,
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \prod_{j=1}^m f_j(p_{k_j}^{(j)}) = \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\textbf{Inductive Step:}
We need to show that the statement holds for $d = m+1$. Consider the left-hand side for $d = m+1$:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_{m+1}=1}^{n_{m+1}} \prod_{j=1}^{m+1} f_j(p_{k_j}^{(j)})
$$
We can rewrite this as:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \left( \sum_{k_{m+1}=1}^{n_{m+1}} \left( \prod_{j=1}^m f_j(p_{k_j}^{(j)}) \right) f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right)
$$
Notice that the inner sum (over $k_{m+1}$) does not depend on $k_1, k_2, ..., k_m$. Thus, for any fixed values of $k_1, k_2, ..., k_m$, we can treat $\prod_{j=1}^m f_j(p_{k_j}^{(j)})$ as a constant. Let $C(k_1, ..., k_m) = \prod_{j=1}^m f_j(p_{k_j}^{(j)})$. Then we have:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \left( C(k_1, ..., k_m) \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right)
$$
Now, the inner sum $\sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)})$ is a constant with respect to $k_1, ..., k_m$. Let's call this constant $S_{m+1}$. So we have:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} C(k_1, ..., k_m) S_{m+1} = S_{m+1} \sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \prod_{j=1}^m f_j(p_{k_j}^{(j)})
$$
By the inductive hypothesis, we can replace the nested sums with a product:
$$
S_{m+1} \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right) = \left( \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right) \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
Rearranging the terms, we get:
$$
\prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right) \left( \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right) = \prod_{j=1}^{m+1} \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
This is the right-hand side of the statement for $d = m+1$. Thus, the statement holds for $d = m+1$.
\textbf{Conclusion:}
By induction, the statement holds for all $d \geq 1$. Therefore,
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d f_j(p_{k_j}^{(j)}) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\end{proof}
\begin{claim}\label{claim:convex-combo-eigenvalues}
Given a convex combination \(\mathbf{C} = \alpha \mathbf{A} + (1-\alpha) \mathbf{B}\), where \(\alpha \in [0,1]\), and \(\mathbf{A}\) and \(\mathbf{B}\) are symmetric matrices, the eigenvalues of \(\mathbf{C}\) lie in the interval \(\left[\min \left(\lambda_n(\mathbf{A}), \lambda_n(\mathbf{B})\right), \max \left(\lambda_1(\mathbf{A}), \lambda_1(\mathbf{B})\right)\right]\).
% Given a convex combination $C=\alpha A+(1-\alpha)B,\alpha \in [0,1]$, the eigenvalues of $C$ lie in the interval $[\min(\lambda(A)_n, \lambda(B)_n), \max(\lambda(A)_1,\lambda(B)_1)]$
\end{claim}
\begin{proof}
First, recall that for a symmetric matrix \(\mathbf{A}\), the Rayleigh quotient \(R(\mathbf{A}, \mathbf{x}) = \frac{\mathbf{x}^{\top} \mathbf{A} \mathbf{x}}{\mathbf{x}^{\top} \mathbf{x}}\) is bounded by the smallest and largest eigenvalues of \(\mathbf{A}\):
\[
\lambda_n(\mathbf{A}) \leq R(\mathbf{A}, \mathbf{x}) \leq \lambda_1(\mathbf{A})
\]
Consider the Rayleigh quotient for the matrix \(\mathbf{C}\):
\[
R(\mathbf{C}, \mathbf{x}) = \frac{\mathbf{x}^{\top} (\alpha \mathbf{A} + (1-\alpha) \mathbf{B}) \mathbf{x}}{\mathbf{x}^{\top} \mathbf{x}} = \alpha R(\mathbf{A}, \mathbf{x}) + (1-\alpha) R(\mathbf{B}, \mathbf{x})
\]
Since \(R(\mathbf{A}, \mathbf{x})\) and \(R(\mathbf{B}, \mathbf{x})\) are bounded by their respective eigenvalues, we have:
\[
R(\mathbf{C}, \mathbf{x}) \leq \alpha \lambda_1(\mathbf{A}) + (1-\alpha) \lambda_1(\mathbf{B})
\]
which implies:
\[
R(\mathbf{C}, x) \leq \max(\lambda_1(\textbf{A}), \lambda_1(\textbf{B}))
\]
Similarly,
\[
R(\textbf{C}, \textbf{x}) \geq \min(\lambda_n(\textbf{A}), \lambda_n(\textbf{B}))
\]
Thus, the eigenvalues of \(\textbf{C} = \alpha \textbf{A} + (1-\alpha)\textbf{B}\) are bounded by:
\[
\min(\lambda_n(\textbf{A}), \lambda_n(\textbf{B})) \leq \lambda(\textbf{C}) \leq \max(\lambda_1(\textbf{A}), \lambda_1(\textbf{B}))
\]
% First recall that for SPD matrices, the Rayleigh quotient $R(A,x)=\frac{x^\top Ax}{x^\top x}$ is bounded by the largest and smallest eigenvalues of $A$.
% Consider
% \begin{align*}
% \frac{x^\top (\alpha A+(1-\alpha)B)x}{x^\top x}&=\alpha \frac{x^\top Ax}{x^\top x}+(1-\alpha)\frac{x^\top Bx}{x^\top x}
% \end{align*}
% Now
% \begin{align*}
% \alpha \frac{x^\top Ax}{x^\top x}+(1-\alpha)\frac{x^\top Bx}{x^\top x}&\leq \alpha \lambda_1(A)+(1-\alpha)\lambda_1(B)\\
% &\leq \max(\lambda_1(A),\lambda_1(B))
% \end{align*}
% so that $R(C,x)\leq \max(\lambda_1(A),\lambda_1(B))$. A similar reasoning can show that $R(C,x)\geq \min(\lambda_n(A),\lambda_n(B))$. Thus, the eigenvalues of $C$ lie in the interval $[\min(\lambda(A)_n, \lambda(B)_n), \max(\lambda(A)_1,\lambda(B)_1)]$
\end{proof}
\textbf{Appendix A: Second-Order Derivatives of Log-Likelihood (Details)}
Here, we provide the detailed calculations for the second-order derivatives of the log-likelihood with respect to $\theta_i$ and $\theta_j$.
We start with the first-order derivative:
$$
\frac{\partial \mathcal{L}}{\partial \theta_i} = \frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right).
$$
Now, we differentiate again with respect to $\theta_j$. We will use the following matrix calculus identities:
* $\frac{\partial}{\partial \theta} (\mathbf{A}^{-1}) = -\mathbf{A}^{-1} (\frac{\partial \mathbf{A}}{\partial \theta}) \mathbf{A}^{-1}$
* $\frac{\partial}{\partial \theta} \text{tr}(\mathbf{A}\mathbf{B}) = \text{tr}(\frac{\partial \mathbf{A}}{\partial \theta} \mathbf{B}) + \text{tr}(\mathbf{A} \frac{\partial \mathbf{B}}{\partial \theta})$
* $\frac{\partial}{\partial \theta} \log|\mathbf{A}| = \text{tr}(\mathbf{A}^{-1} \frac{\partial \mathbf{A}}{\partial \theta})$
Applying these identities, we obtain:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= \frac{1}{2} \mathbf{y}^\top \frac{\partial}{\partial \theta_j} \left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \right] \mathbf{y} \\
&\quad - \frac{1}{2} \frac{\partial}{\partial \theta_j} \left[ \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right]
\end{align*}
Let's compute each term separately. For the first term:
\begin{align*}
&\frac{\partial}{\partial \theta_j} \left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \right] \\
&= \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial}{\partial \theta_j} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \\
&= -(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad - (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}
\end{align*}
For the second term:
\begin{align*}
&\frac{\partial}{\partial \theta_j} \left[ \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] \\
&= \text{tr}\left( \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \right) + \text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial}{\partial \theta_j} \left[\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right] \right) \\
&= -\text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \right) + \text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} \right)
\end{align*}
Putting everything together, we get the expression for the second-order derivative:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left[ \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \\
&\quad + \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) \right] - \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) \right].
\end{align*}
\textbf{Appendix B: Simplification of the Quadratic Form (Details)}
Here, we provide the detailed algebraic manipulations to arrive at the expression involving the trace.
We start with the quadratic form:
Substituting the expression for the second-order derivative and evaluating at $\theta=\theta^*$, we get:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v}
&= -\mathbb{E}_{P^*} \left[ \sum_{i=1}^2 \sum_{j=1}^2 v_i v_j \frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j}\bigg|_{\theta = \theta^*} \right].
\end{align*}
Now, we substitute the expression for the second-order derivatives and use the fact that $\mathbb{E}_{P^*}[\mathbf{y}\mathbf{y}^\top | \mathbf{X}] = \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I}$:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} &= \frac{1}{2} \mathbb{E}_{P^*_X} \left[ \sum_{i=1}^2 \sum_{j=1}^2 v_i v_j \left( \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right) (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right) \right] - \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \frac{\partial^2 \mathbf{K}_{\theta^*}}{\partial \theta_i \partial \theta_j} \right] \right. \right. \\
&\quad \left. \left. + \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left\{ \frac{\partial^2 \mathbf{K}_{\theta^*}}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right)(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right)(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right) \right\} \right] \right) \right].
\end{align*}
After simplifying and rearranging terms, we get:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} &= \frac{1}{2} \mathbb{E}_{P^*_X} \left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left( \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i} \right) (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left( \sum_{j=1}^2 v_j \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j} \right) \right) \right] \\
&= \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
\end{align*}
where $\mathbf{A} = \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}$. This is the expression used in the proof.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \section{Local Polynomial Interpolation}
% If we use all $m$ points for interpolation a function $f$ with a polynomial $p$, then we have
% \begin{align*}
% f(x)-p(x)&=\frac{1}{m!}f^{(m)}(\xi_x)\prod_{i=1}^m (x-x_i).
% \end{align*}
% On the other hand, if we use the \textit{closest} $L$ points, then we have, for some sub-interval $[c,d]\subseteq [a,b]$,
% \begin{align*}
% f(x)-p(x)&=\frac{1}{L!}f^{(L)}(\xi_x)\prod_{i=1}^L(x-x_{c_i})
% \end{align*}
% where $c_i$ are the indices of the closest points to $x$ in the original $m$ points. We thus have to tradeoff three contributors to an upper bound on the error:
% \begin{itemize}
% \item $\frac{1}{L!}>\frac{1}{m!}$
% \item $\prod_{i=1}^L(x-x_{c_i})\leq \prod_{i=1}^m (x-x_i)$ \textit{and} particularly, the LHS uses the closest terms so the inner terms in the product will tend to be smaller (how do we say this formally?)
% \item it's difficult to compare $f^{(L)}(\xi_x)$ on the subinterval to $f^{(m)}(\xi_x)$ on the full interval.
% \item derivative growth control and distance factor
% \end{itemize}
% If we increase $m$ while fixing $L$, then the derivative growth over the full grid doesn't get worse, while the distance factor shrinks. For simplicity, consider a 1d setting where $m\mod L\equiv 0$, e.g. $m$ is a multiple of $L$, $L$ is odd, and $x_1\equiv 0$ and $x_m\equiv 1$. Assume further that $x$ is in the `middle interval' \amnote{as Justin suggested}. Then
% \begin{align*}
% \prod_{i=1}^L(x-x_{c_i})&\leq \prod_{i=1}^L \frac{L-1}{m}\\
% &\leq \left(\frac{L-1}{kL}\right)^L\\
% &\leq k^{-L}
% \end{align*}
\end{document}
% This document was modified from the file originally made available by
% Pat Langley and Andrea Danyluk for ICML-2K. This version was created
% by Iain Murray in 2018, and modified by Alexandre Bouchard in
% 2019 and 2021. Previous contributors include Dan Roy, Lise Getoor and Tobias
% Scheffer, which was slightly modified from the 2010 version by
% Thorsten Joachims & Johannes Fuernkranz, slightly modified from the
% 2009 version by Kiri Wagstaff and Sam Roweis's 2008 version, which is
% slightly modified from Prasad Tadepalli's 2007 version which is a
% lightly changed version of the previous year's version by Andrew
% Moore, which was in turn edited from those of Kristian Kersting and
% Codrina Lauth. Alex Smola contributed to the algorithmic style files.
% \textcolor{blue}{JX: a bit of soft intuition on inducing point spacing, mainly for myself. (\textbf{Skip the first two paragraphs to get to the useful result.}) Let's start with univariate: when inducing points $u_0<\dots<u_k$ are regularly spaced, polynomial interpolants suffer errors near the lowest and highest points (\textbf{Runge's phenomenon}) that \textit{increase} with interpolant degree. This is because in order to match the function at all inducing points, univariate Lagrange interpolants are proportional to $\prod_{i=0}^k(x-u_i)$ (where $x$ is the point to be interpolated). However, this means that when $x$ is near $u_0$ or $u_k$, many of these factors grow large while only a few are small, exploding the product and diverting the interpolant ``off course". As polynomial degree increases, so does the number of large factors near $u_1$ and $u_k$. The intuition is exactly the same for multivariate interpolation, since Lagrange interpolants are defined similarly. \textit{Sidenote:} I bet this is why the Lebesgue constant grows exponentially in relation to degree when inducing points are equidistant. Each added degree adds one large factor to the product.}
% \textcolor{blue}{\textbf{Chebyshev-Gauss nodes} (which I think are slightly different from Chebyshev nodes? but the idea is the same) fix this issue by pushing $u_0$ and $u_1$ extremely close together, so when $x$ is near $u_0$, $x-u_0$ and $x-u_1$ are so tiny that the interpolant stays ``on course". That's why Chebyshev-Gauss nodes are defined as $\xi_i=\cos(\frac{\pi i}{N})$ for $0\le i\le k$: near the ends, $\cos$ changes very slowly. (I've seen multiple slightly different definitions of Chebyshev nodes, but they all use cosine.) This choice of inducing points has been proven to be at least almost optimal in minimizing the Lebesgue constant.}
% \textcolor{blue}{Finally to the math. In short, \textit{Theorem 3.3} in \textbf{https://personal-homepages.mis.mpg.de/bokh/lec3.pdf} says if $f$ is infinitely differentiable on $[-1,1]^d$ and we pick $(N+1)^d$ Chebyshev-Gauss nodes $$\xi_{i_1,\dots,i_d}=\left(cos\frac{\pi i_1}{N},\dots,\cos\frac{\pi i_d}{N}\right)$$ with all $i_k$ being nonnegative integers not more than $N$, then the interpolation error is at most $$\Lambda_N^d\frac{2M_\rho(f)}{\rho-1}\rho^{-N}$$ where $\Lambda_N$ is the Lebesgue constant of a univariate interpolant for $N+1$ inducing points, $\rho>1$ is the sum of the semi-axes of ellipse $E_\rho\subset\mathbb{C}$ with foci $\pm1$, and $M_\rho(f)$ is the maximum across all $1\le j\le d$ of the maximum output of the analytic extension of $f$ to a domain where $x_j$ and only $x_j$ can now be any complex number in $E_\rho$. (We assume this analytic extension exiss for all $j$.)}
% \textcolor{blue}{With Chebyshev-Gauss nodes $\Lambda_N\le\frac{2}{\pi}\log N+1$. \textbf{I'm very, very sorry - I'm a little stuck on linking this back to Lemma 2. I'll continue at 3 PM CST 10/30/2024.} I \textit{believe} that we find an upper bound for $M_\rho(f)$ by using information about $\textbf{K}$, after which the bound becomes a tractable expression in terms of $\rho$ and $N$. This is because both the linked slideshow and this related link \textbf{https://www.chebfun.org/examples/approx/EntireBound.html} use example functions where $M_\rho(f)$ can be expressed in terms of $\rho$.}
% \textcolor{blue}{\textbf{This is probably where the matrix sparsity comes in:} to give us information on $M_\rho(f)$.}
% \amnote{the goal isn't to relate their result to Lemma 2 (eventually we will), it's really just to relate it to Lemma 1.}
% \amnote{https://mathoverflow.net/questions/426184/upper-bound-for-spectral-norm-of-the-covariance-matrix-of-a-certain-gaussian-vec}
% \begin{corollary}
% We have that $\gamma_{n,m,L}=O(n\sqrt{L}\rho^{-L})$
% \end{corollary}
% \amnote{Issue: this ignores the Lebesgue constant.}
% \begin{proof}
% Recall that $\delta_{m,L}=c^d\frac{2M_\rho(g)}{\rho-1}\rho^{-L}$
% \begin{align*}
% \gamma_{n,m,L}&\equiv n(\delta_{m,L}+\sqrt{L}c^d\delta_{m,L})\\
% &=n\left(c^d\frac{2M_\rho(g)}{\rho-1}\rho^{-L}+\sqrt{L}c^{2d}\frac{2M_\rho(g)}{\rho-1}\rho^{-L}\right)
% \end{align*}
% \end{proof}
% \begin{corollary}
% If we want $\gamma_{n,m,L}<\epsilon$, then we need $L=O(\log \frac{n}{\epsilon})$
% \end{corollary}
% \begin{proof}
% We need to analyze the inequality
% \begin{align*}
% n\sqrt{L}\rho^{-L}&<\epsilon\\
% \sqrt{L}\rho^{-L}&<\frac{\epsilon}{n}\\
% \frac{1}{2}\log L-L\log \rho&<\log \frac{\epsilon}{n}
% \end{align*}
% and dropping lower order terms of $L$ we have
% \begin{align*}
% L>\left(\log\frac{n}{\epsilon}\right)\frac{1}{\log \rho}
% \end{align*}
% and thus
% \begin{align*}
% L&=O\left(\log\frac{n}{\epsilon}\right)
% \end{align*}
% \end{proof}
% \subsection{From Elementwise to Spectral Norm}
% In this subsection, we transition from elementwise error bounds for the SKI kernel approximation to bounds on the spectral norm of the difference between the true Gram matrix and its SKI approximation. The spectral norm is a crucial quantity in many applications involving kernel methods, as it governs the stability and accuracy of operations such as matrix inversion and solving linear systems. It is especially important because it provides guarantees on the overall error of the approximation, as opposed to just individual entries. We also provide a bound on the spectral norm of the SKI approximation of the kernel matrix between test and training points. This is useful when analyzing the error of the predictive mean and variance in Gaussian process regression.
% \begin{proposition}\label{prop:spectral-norm}
% Let $\gamma_{n,m,L}\equiv n(\delta_{m,L}+\sqrt{L}c^d\delta_{m,L})$. Then
% \begin{align*}
% \Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq \gamma_{n,m,L}
% \end{align*}
% \end{proposition}
% \begin{proof}
% Recall that
% \begin{align*}
% \Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq \sqrt{\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_1\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty}\\
% &=\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty\textrm{ since $\textbf{K}-\tilde{\textbf{K}}$ symmetric}\\
% &\leq n \max(\textbf{K}-\tilde{\textbf{K}})\textrm{ where the max is elementwise}\\
% &\leq n \left(\delta_{m,L}+\sqrt{L} c^d\delta_{m,L}\right)\\
% &=\gamma_{n,m,L}
% \end{align*}
% \end{proof}
% \begin{lemma}\label{lemma:test-train-kernel-matrix-error}
% Let $\textbf{K}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ be the matrix of kernel evaluations between test and training points and $\tilde{\textbf{K}}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ be the corresponding SKI approximation. Then
% \begin{align*}
% \Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq \max(\gamma_{T,m,L},\gamma_{n,m,L}).
% \end{align*}
% \end{lemma}
% \begin{proof}
% \begin{align*}
% \Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq \sqrt{\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty}\\
% &\leq \max \left(\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty\right)\\
% &\leq \max(\gamma_{T,m,L},\gamma_{n,m,L})
% \end{align*}
% \end{proof}
% \begin{lemma}
% Let $\tilde{\textbf{K}}_{\cdot,\textbf{X}} \in \mathbb{R}^{T \times n}$ be the matrix of SKI kernel evaluations between $T$ test points and $n$ training points, where the SKI approximation uses $m$ inducing points. Let $\textbf{W}(\cdot) \in \mathbb{R}^{T \times m}$ and $\textbf{W}(\textbf{X}) \in \mathbb{R}^{n \times m}$ be the matrices of interpolation weights for the test points and training points, respectively. Assume that the interpolation scheme is such that the sum of the absolute values of the interpolation weights for any point is bounded by $c^d$, where $c>0$ is a constant. Let $\textbf{K}_{\textbf{U}} \in \mathbb{R}^{m \times m}$ be the kernel matrix evaluated at the inducing points. If the kernel function $k$ is bounded such that $|k(\textbf{x}, \textbf{x}')| \leq M$ for all $\textbf{x}, \textbf{x}'$ in the domain of interest, then:
% $$
% \Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq \sqrt{Tn} m c^{2d} M
% $$
% \end{lemma}
% \begin{proof}
% By the definition of the SKI approximation and the submultiplicativity of the spectral norm, we have:
% $$
% \Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 = \Vert \textbf{W}(\cdot)\textbf{K}_{\textbf{U}}(\textbf{W}(\textbf{X}))^\top\Vert_2 \leq \Vert \textbf{W}(\cdot)\Vert_2 \Vert\textbf{K}_{\textbf{U}}\Vert_2 \Vert\textbf{W}(\textbf{X})\Vert_2
% $$
% We now bound each term.
% 1. **Bounding $\Vert \textbf{W}(\cdot)\Vert_2$ and $\Vert \textbf{W}(\textbf{X})\Vert_2$:**
% Since the spectral norm is induced by the Euclidean norm, and using the assumption that the sum of absolute values of interpolation weights for any point is bounded by $c^d$, we have
% $$\Vert \textbf{W}(\cdot)\Vert_2 \leq \sqrt{\Vert \textbf{W}(\cdot)\Vert_1 \Vert \textbf{W}(\cdot)\Vert_\infty} \leq \sqrt{T c^d \cdot c^d} = \sqrt{T} c^d.$$
% Similarly, $\Vert\textbf{W}(\textbf{X})\Vert_2 \leq \sqrt{n}c^d$.
% 2. **Bounding $\Vert\textbf{K}_{\textbf{U}}\Vert_2$:**
% Since $\textbf{K}_{\textbf{U}}$ is symmetric, $\Vert \textbf{K}_{\textbf{U}} \Vert_2 \leq \Vert \textbf{K}_{\textbf{U}} \Vert_\infty$. Each entry of $\textbf{K}_{\textbf{U}}$ is bounded by $M$ (by the boundedness of $k$), and each row has $m$ entries, so $\Vert \textbf{K}_{\textbf{U}} \Vert_\infty \leq mM$. Thus, $\Vert\textbf{K}_{\textbf{U}}\Vert_2 \leq mM$.
% Combining these bounds, we get:
% $$
% \Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq (\sqrt{T} c^d) (mM) (\sqrt{n} c^d) = \sqrt{Tn} m c^{2d} M
% $$
% as required.
% \end{proof}
\amnote{Now we would like to show that for some kernel (RBF probably), there exists a region around true parameter such that the log-likelihood is w.h.p. both strongly concave and has L-Lipschitz continuous gradients.}
\begin{theorem}[Local Strong Concavity of Population Log-Likelihood - RBF Kernel, Restricted Parameter Space]\label{thm:local-strong-concavity-restricted}
Let $P^*$ be the true data generating distribution, and let $\theta^* = (\sigma_{f*}, \ell_*)$ be the maximizer of the population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ for a Gaussian process with an RBF kernel:
$$
k_\theta(x, x') = \sigma_f^2 \exp\left(-\frac{\|x - x'\|_2^2}{2\ell^2}\right),
$$
where $\theta = (\sigma_f, \ell)$ are the signal variance and lengthscale parameters, respectively.
Assume the following:
1. **Restricted Parameter Space:** We restrict the parameter space to a compact set $\Theta$ such that for all $\theta \in \Theta$, we have $\sigma_f \in [\sigma_{f, min}, \sigma_{f, max}]$ and $\ell \in [\ell_{min}, \ell_{max}]$, where $0 < \sigma_{f, min} < \sigma_{f, max} < \infty$ and $0 < \ell_{min} < \ell_{max} < \infty$. Furthermore, $\theta^* \in \Theta$.
2. **Smoothness and Boundedness:**
* The true data generating process is of the form $y_i = \xi(x_i) + \epsilon_i$, where $\xi \sim GP(0, k_{\theta^*})$ and $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are i.i.d. noise.
* The marginal distribution of the inputs $P^*_X$ has a density $p(x)$ with respect to the Lebesgue measure on $\mathbb{R}^d$ that is bounded away from zero on a compact set $\mathcal{X} \subset \mathbb{R}^d$, i.e., there exists $\rho > 0$ such that $p(x) \geq \rho$ for all $x \in \mathcal{X}$.
* The function $\xi$ and its first and second partial derivatives are bounded on $\mathcal{X}$.
3. **Regularity of the RBF Kernel:** The RBF kernel $k_\theta(x,x')$ and its first, second, and third partial derivatives with respect to $\theta$ are continuous and bounded on $\mathcal{X} \times \mathcal{X}$ for $\theta \in \Theta$.
Then, there exists a neighborhood $\mathcal{D} \subseteq \Theta$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the following hold:
* **(a) Local Strong Concavity:** The population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu$ within $\mathcal{D}$, i.e., for all $\theta_1, \theta_2 \in \mathcal{D}$:
$$
\mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{y})] \leq \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})] + \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})]^\top (\theta_2 - \theta_1) - \frac{\mu}{2} \| \theta_2 - \theta_1 \|_2^2.
$$
* **(b) Lipschitz Gradient:** The gradient of the population log-likelihood $\nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is Lipschitz continuous with constant $L$ within $\mathcal{D}$, i.e., for all $\theta_1, \theta_2 \in \mathcal{D}$:
$$
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{Y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{Y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2.
$$
\end{theorem}
\begin{proof}
The proof follows a similar structure to the previous attempts, but crucially, we now work within a restricted parameter space.
**1. Define the Population Log-Likelihood:**
As before, let $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{Y})]$ denote the population log-likelihood.
**2. Consider the Hessian:**
Let $\mathbf{H}(\theta)$ be the Hessian of $\bar{\mathcal{L}}(\theta)$. We will show that $\mathbf{H}(\theta^*)$ is negative definite.
**3. Expected Fisher Information:**
The expected Fisher information matrix is $\mathbf{I}(\theta^*) = -\mathbf{H}(\theta^*)$. We will show that $\mathbf{I}(\theta^*)$ is positive definite.
**4. Positive Definiteness of $I(\theta^*)$:**
We want to show that for any non-zero vector $\mathbf{v} \in \mathbb{R}^2$, $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0$. Following the same steps as in the previous attempt, we arrive at:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
$$
where $\mathbf{A} = v_1 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \sigma_f^2} + v_2 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \ell}$.
As before, we can rewrite this as:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right],
$$
where $\mathbf{B} = (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2}$.
**5. Key Argument - Restricted Parameter Space and Linear Independence:**
Now, we leverage the restricted parameter space $\Theta$. Since $\sigma_f$ is bounded away from 0 and $\ell$ is bounded away from both 0 and $\infty$, the eigenvalues of $\mathbf{K}_{\theta^*}$ are bounded away from 0 and $\infty$. This implies that $\mathbf{B}$ and $\mathbf{B}^{-1}$ are well-defined and have bounded norms.
Suppose for contradiction that $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = 0$ for some non-zero $\mathbf{v}$. This implies that $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$ almost surely, and consequently, $\mathbf{A} = \mathbf{0}$ almost surely.
Thus, we have:
$$
v_1 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \sigma_f^2} + v_2 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \ell} = \mathbf{0} \quad \text{a.s.}
$$
Now, we need to show that this is impossible for non-zero $\mathbf{v}$ under our assumptions. We can write this as:
$$
v_1 \frac{\partial k_{\theta^*}}{\partial \sigma_f^2}(x_i,x_j) + v_2 \frac{\partial k_{\theta^*}}{\partial \ell}(x_i,x_j) = 0
$$
for all $x_i,x_j$ in the support of $P_X^*$.
Using the derivatives of the RBF kernel that we computed earlier, this becomes:
$$
v_1 \exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right) + v_2 \sigma_{f*}^2 \exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right) \left(\frac{\|x_i - x_j\|_2^2}{\ell_*^3}\right) = 0
$$
for all $x_i, x_j$ in the support of $P_X^*$.
Dividing by $\exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right)$ (which is always non-zero), we get:
$$
v_1 + v_2 \sigma_{f*}^2 \frac{\|x_i - x_j\|_2^2}{\ell_*^3} = 0
$$
Since $\sigma_{f*} > 0$ and $\ell_* > 0$, the only way for this to hold for all $x_i, x_j$ in a set of non-zero measure under $P^*_X$ is if $v_1 = v_2 = 0$.
This is because if $v_2 \neq 0$, then the term $\frac{\|x_i - x_j\|_2^2}{\ell_*^3}$ can be varied continuously by changing $x_i$ and $x_j$, which makes it impossible for the equation to hold unless $v_1$ is also zero.
Thus, we have reached a contradiction. Therefore, for any non-zero $\mathbf{v}$, we must have $\mathbf{A} \neq \mathbf{0}$ almost surely, and hence $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0$.
**6. Negative Definite Hessian and Strong Concavity:**
Since $\mathbf{I}(\theta^*)$ is positive definite, $\mathbf{H}(\theta^*)$ is negative definite. By continuity of the Hessian (due to the regularity assumption on the kernel), there exists a neighborhood $\mathcal{D} \subseteq \Theta$ of $\theta^*$ such that $\mathbf{H}(\theta)$ is negative definite for all $\theta \in \mathcal{D}$. Moreover, there exists a constant $\mu > 0$ such that $-\mathbf{H}(\theta) \succeq \mu \mathbf{I}$ for all $\theta \in \mathcal{D}$. This implies that the population log-likelihood is strongly concave with parameter $\mu$ within $\mathcal{D}$.
**7. Lipschitz Continuity of the Gradient:**
The Lipschitz continuity of the gradient follows from the boundedness of the third-order derivatives of the population log-likelihood in the neighborhood $\mathcal{D}$. This can be shown using the regularity assumptions on the kernel (Assumption 3) and the boundedness assumption (Assumption 2). Since the third-order derivatives are bounded, there exists a constant $L < \infty$ such that:
$$
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{Y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{Y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2
$$
for all $\theta_1, \theta_2 \in \mathcal{D}$.
**Conclusion:**
We have shown that under the given assumptions, there exists a neighborhood $\mathcal{D}$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the population log-likelihood is strongly concave with parameter $\mu$ and has a Lipschitz continuous gradient with constant $L$ within $\mathcal{D}$. This completes the proof.
\end{proof}
\begin{theorem}[Local Strong Concavity of Population Log-Likelihood - General Case]\label{thm:local-strong-concavity-general}
Let $P^*$ be the true data generating distribution, and let $\theta^*$ be the maximizer of the population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ for a Gaussian process with kernel $k_\theta(x, x')$.
Assume the following:
1. **Identifiability:** The hyperparameters $\theta^*$ are identifiable from the data generating distribution $P^*$.
2. **Smoothness and Boundedness:**
* The true data generating process is of the form $y_i = \xi(x_i) + \epsilon_i$, where $\xi \sim GP(0, k_{\theta^*})$ and $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are i.i.d. noise.
* The marginal distribution of the inputs $P^*_X$ has a density $p(x)$ with respect to the Lebesgue measure on $\mathbb{R}^d$ that is bounded away from zero on a compact set $\mathcal{X} \subset \mathbb{R}^d$.
* The function $\xi$ and its first and second partial derivatives are bounded on $\mathcal{X}$.
3. **Regularity of the Kernel:** The kernel $k_\theta(x,x')$ and its first, second, and third partial derivatives with respect to $\theta$ are continuous and bounded on $\mathcal{X} \times \mathcal{X}$ for $\theta$ in a neighborhood of $\theta^*$.
4. **Linear Independence of Derivatives:** For any non-zero vector $\mathbf{v}$ of the same dimension as $\theta$, the following holds almost surely with respect to $P^*_X$:
$$
\sum_{i} v_i \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \neq \mathbf{0}
$$
where $\mathbf{K}_\theta$ is the kernel matrix evaluated at a set of input points drawn from $P^*_X$.
Then, there exists a neighborhood $\mathcal{D}$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the following hold:
* **(a) Local Strong Concavity:** The population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu$ within $\mathcal{D}$.
* **(b) Lipschitz Gradient:** The gradient of the population log-likelihood $\nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is Lipschitz continuous with constant $L$ within $\mathcal{D}$.
\end{theorem}
\begin{proof}
Let $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ denote the population log-likelihood. The Hessian matrix $\mathbf{H}(\theta)$ has entries given by:
$$
\mathbf{H}_{ij}(\theta) = \frac{\partial^2 \bar{\mathcal{L}}(\theta)}{\partial \theta_i \partial \theta_j}.
$$
We want to show that $\mathbf{H}(\theta^*)$ is negative definite. It is equivalent to showing that the negative Hessian, $-\mathbf{H}(\theta^*)$, which is also known as the *expected Fisher information matrix*, is positive definite.
The log-likelihood for a set of observations $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$ (assuming a zero mean function) is:
$$
\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\mathbf{K}_\theta + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi),
$$
where $\mathbf{K}_\theta$ is the kernel matrix evaluated at the points in $\mathbf{X}$, and $\sigma^2$ is the noise variance.
\textbf{1. First-Order Derivatives:**}
The gradient of the log-likelihood has components:
$$
\frac{\partial \mathcal{L}}{\partial \theta_i} = \frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right).
$$
\textbf{2. Second-Order Derivatives:**}
The second-order derivatives can be computed using matrix calculus identities. After some calculations (details in Appendix A), we arrive at the following expression:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left[ \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \\
&\quad + \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) \right] - \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) \right].
\end{align*}
\textbf{3. Population Log-Likelihood and Hessian:**}
The population log-likelihood is $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$. The Hessian of the population log-likelihood is:
$$
\mathbf{H}(\theta) = \mathbb{E}_{P^*}[\nabla^2_\theta \mathcal{L}(\theta; \mathbf{X}, \mathbf{y})],
$$
where $\nabla^2_\theta$ denotes the Hessian with respect to $\theta$.
\textbf{4. Expected Fisher Information:**}
The expected Fisher information matrix is $\mathbf{I}(\theta^*) = -\mathbf{H}(\theta^*)$. We want to show that $\mathbf{I}(\theta^*)$ is positive definite.
\textbf{5. Positive Definiteness:**}
To show that $\mathbf{I}(\theta^*)$ is positive definite, we need to show that for any non-zero vector $\mathbf{v} \in \mathbb{R}^2$:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0.
$$
Substituting $\theta = \theta^*$ and taking the expectation with respect to $P^*$, we get:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = -\mathbb{E}_{P^*}[\mathbf{v}^\top \nabla^2_\theta \mathcal{L}(\theta^*; \mathbf{X}, \mathbf{y}) \mathbf{v}].
$$
Since $\mathbf{y}|\mathbf{X} \sim \mathcal{N}(\mathbf{0}, \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})$, we have $\mathbb{E}_{P^*}[\mathbf{y}\mathbf{y}^\top | \mathbf{X}] = \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I}$. Using this, we can simplify the expectation of the quadratic form involving $\mathbf{y}$ (details in Appendix B). After algebraic manipulation, we obtain:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
$$
where $\mathbf{A} = \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}$.
\textbf{6. Using Properties of Trace and Positive Semidefiniteness:**}
Now, note that $(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}$ is positive definite and thus has a positive definite square root. Let $\mathbf{B} = (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2}$. Then, we can rewrite the expression as:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( \mathbf{B} \mathbf{A} \mathbf{B} \mathbf{B} \mathbf{A} \mathbf{B} \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B}) (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right].
$$
Since $(\mathbf{B}\mathbf{A}\mathbf{B})^\top (\mathbf{B}\mathbf{A}\mathbf{B})$ is positive semidefinite, its trace is non-negative. Moreover, the trace is zero if and only if $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$.
\textbf{7. Using Identifiability and Smoothness:**}
Here, we use the identifiability assumption. If $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$, then $(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2} = \mathbf{0}$, which implies $\mathbf{A} = \mathbf{0}$ (since the inverse square root is invertible).
$\mathbf{A} = \mathbf{0}$ implies:
$$
\sum_{i=1}^p v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}=\mathbf{0}
$$
This means that a linear combination of the derivatives of the kernel matrix with respect to the hyperparameters is zero. However, under the identifiability assumption and the smoothness of the data distribution, this can only happen if $v_1 = v_2 = 0$ (i.e., $\mathbf{v}$ is the zero vector). If different hyperparameters always lead to different predictive distributions, then their derivatives must be linearly independent.
Therefore, for any non-zero $\mathbf{v}$, we have $\mathbf{B}\mathbf{A}\mathbf{B} \neq \mathbf{0}$, and thus:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right] > 0.
$$
This shows that $\mathbf{I}(\theta^*)$ is positive definite, and consequently, $\mathbf{H}(\theta^*)$ is negative definite.
\textbf{Step 8: Establish Lipschitz continuity of the gradient.}
The Lipschitz continuity of the gradient follows from the boundedness of the third-order derivatives of the population log-likelihood in the neighborhood $\mathcal{D}$. This can be shown using the regularity assumptions on the kernel (Assumption 3) and the boundedness assumption (Assumption 2). Since the third-order derivatives are bounded, there exists a constant $L < \infty$ such that
\[
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2
\]
for all $\theta_1, \theta_2 \in \mathcal{D}$.
\end{proof}
\amnote{What if we don't have strong concavity? Can we still obtain a weaker guarantee?}
\subsubsection{MLE}
\begin{theorem}
\end{theorem}
\begin{proof}
Noting that $\theta^+=\theta+t\nabla \tilde{\mathcal{L}}(\theta)$
\begin{align*}
\mathcal{L}(\theta^+)&\geq \mathcal{L}(\theta)+\nabla \mathcal{L}(\theta)^\top(\theta^+-\theta)-\frac{1}{2}L\Vert \theta^+-\theta\Vert^2\\
&= \mathcal{L}(\theta)+\nabla \mathcal{L}(\theta)^\top(t\nabla \tilde{\mathcal{L}}(\theta))-\frac{1}{2}tL\Vert\nabla \tilde{\mathcal{L}}(\theta)\Vert^2\\
&= \mathcal{L}(\theta)+t\nabla \mathcal{L}(\theta)^\top(\nabla\mathcal{L}(\theta)+\nabla \tilde{\mathcal{L}}(\theta)-\nabla\mathcal{L}(\theta))-\frac{1}{2}tL\Vert\nabla \mathcal{L}(\theta)+\nabla \tilde{\mathcal{L}}(\theta)-\nabla\mathcal{L}(\theta)\Vert^2\\
&\geq \mathcal{L}(\theta)+\left(t-\frac{1}{2}L\right)\Vert \nabla \mathcal{L}(\theta)\Vert^2-t\Vert\mathcal{L}(\theta)\Vert\Vert \nabla\tilde{\mathcal{L}}(\theta)-\nabla\mathcal{L}(\theta)\Vert-\frac{1}{2}tL\Vert \nabla\tilde{\mathcal{L}}(\theta)-\nabla\mathcal{L}(\theta)\Vert^2
\end{align*}
\end{proof}
% \subsubsection{Difference in Log-Likelihoods}
% \begin{lemma}\label{lemma:difference-in-log-likelihoods}
% Let \( \mathbf{K} \) be the true kernel matrix, \( \tilde{\mathbf{K}} \) be the SKI approximation of the kernel matrix using \( m \) inducing points and interpolation degree \( L-1 \), and \( \sigma^2 \) be the regularization parameter (noise variance). Let \( \mathbf{y} \) be the vector of observed target values, and \( n \) be the number of data points. Let \( \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) \) be the true log-likelihood and \( \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) \) be the SKI approximation of the log-likelihood, both for a mean-zero Gaussian Process prior. Then, the absolute difference between the true and SKI log-likelihoods is bounded as follows:
% \begin{equation}
% | \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2 + \frac{\gamma_{n, m, L}}{\sigma^2} \right)
% \end{equation}
% where \( \gamma_{n, m, L} \equiv n(\delta_{m, L} + \sqrt{L} c^d \delta_{m, L}) \), \( \delta_{m, L} \) is an upper bound on the error of multivariate interpolation with \( m \) inducing points and interpolation degree \( L-1 \), and \( c > 0 \) is an upper bound on the absolute sum of the weights for 1D kernel interpolation.
% Furthermore, if the convolutional cubic interpolation kernel (\( L=4 \)) is used with grid spacing \( h \), and the domain is contained in $[-D, D]^d$, then:
% \begin{equation}
% | \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{n(2O(c^{2d}h^3))}{\sigma^4} \| \mathbf{y} \|^2 + \frac{n(2O(c^{2d}h^3))}{\sigma^2} \right)
% \end{equation}
% \end{lemma}
% \begin{proof}
% We begin by considering the log-likelihood of a mean-zero Gaussian Process, given by:
% \begin{equation}
% \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi)
% \end{equation}
% The corresponding SKI approximation is:
% \begin{equation}
% \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi)
% \end{equation}
% Our goal is to bound the absolute difference \( |\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})| \). Applying the triangle inequality, we have:
% \begin{align}
% | \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | &= \left| \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} + \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| \right| \nonumber \\
% &\leq \left| \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \right| + \left| \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right|
% \end{align}
% We now leverage Lemma \ref{lemma:action-inverse-error} and Lemma \ref{lemma:log-determinant-error} to bound the two terms on the right-hand side. Lemma \ref{lemma:action-inverse-error} provides a bound on the difference between the action of the true and approximated regularized inverses:
% \begin{equation}
% \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \|_2 \leq \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|_2
% \end{equation}
% This allows us to bound the first term:
% \begin{equation}
% \left| \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \right| \leq \frac{1}{2} \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2
% \end{equation}
% Lemma \ref{lemma:log-determinant-error} bounds the difference in the log-determinants:
% \begin{equation}
% | \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \log|\mathbf{K} + \sigma^2 \mathbf{I}| | \leq \frac{\gamma_{n, m, L}}{\sigma^2}
% \end{equation}
% Thus, the second term is bounded by:
% \begin{equation}
% \left| \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right| \leq \frac{1}{2} \frac{\gamma_{n, m, L}}{\sigma^2}
% \end{equation}
% Combining these bounds, we obtain the overall bound on the difference in log-likelihoods:
% \begin{equation}
% | \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2 + \frac{\gamma_{n, m, L}}{\sigma^2} \right)
% \end{equation}
% Now, let us consider the specific case of using a convolutional cubic interpolation kernel (\( L=4 \)). From Lemma \ref{lemma:tensor-product-interpolation-error}, we have \( \delta_{m, L} = O(c^{2d}h^3) \). Consequently, from Corollary \ref{cor:spectral-norm-error-alt}:
% \begin{equation}
% \gamma_{n, m, L} \equiv n(\delta_{m, L} + \sqrt{L} c^d \delta_{m, L}) = n(O(c^{2d}h^3) + 2c^d O(c^{2d}h^3)) = n(2O(c^{2d}h^3))
% \end{equation}
% Substituting this into our general bound, we obtain the specific bound for cubic interpolation:
% \begin{equation}
% | \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{n(2O(c^{2d}h^3))}{\sigma^4} \| \mathbf{y} \|^2 + \frac{n(2O(c^{2d}h^3))}{\sigma^2} \right)
% \end{equation}
% This concludes the proof.
% \end{proof}
% \begin{assumption}[Local Identifiability of Hyperparameters]\label{assumption:local-identifiability}
% Let $P(\cdot | \mathbf{X}; \theta)$ denote the Gaussian process predictive distribution at a set of input points $\mathbf{X}$ given hyperparameters $\theta$. There exists a neighborhood $\mathcal{D}$ of the true hyperparameters $\theta^*$ such that for any $\theta \in \mathcal{D}$ with $\theta \neq \theta^*$, there exists a finite set of input points $\mathbf{X} = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$, a set $A$ in the Borel sigma-algebra of $\mathbb{R}^n$, and a $\delta > 0$ such that:
% $$
% |P(\mathbf{y} \in A | \mathbf{X}; \theta) - P(\mathbf{y} \in A | \mathbf{X}; \theta^*)| > \delta
% $$
% and
% $$
% P^*_X(\mathbf{X}) > 0,
% $$
% where $\mathbf{y} = (y_1, \ldots, y_n)$ is the vector of corresponding output values, and $P^*_X$ is the marginal distribution of inputs under $P^*$.
% \end{assumption}
% \begin{lemma}[Local Identifiability of Signal Variance]
% Let $\theta = (\sigma^2, \ell^*)$ and $\theta^* = (\sigma^{*2}, \ell^*)$ be two sets of hyperparameters for a Gaussian process with an RBF kernel, where $\sigma^2 \neq \sigma^{*2}$ and the lengthscale $\ell^*$ is fixed. Let $\mathbf{X}_D = \{\mathbf{x}_1, \ldots, \mathbf{x}_N\}$ be a set of $N$ distinct training input points in $\mathbb{R}^d$. Then, there exists an additional input point $\mathbf{x}_{N+1}$ such that the predictive variances $\Sigma(\mathbf{x}_{N+1}; \theta)$ and $\Sigma(\mathbf{x}_{N+1}; \theta^*)$ are different, i.e., $|\Sigma(\mathbf{x}_{N+1}; \theta) - \Sigma(\mathbf{x}_{N+1}; \theta^*)| > 0$.
% \end{lemma}
% \begin{proof}
% \begin{enumerate}
% \item \textbf{Predictive Variance:} The predictive variance at $\mathbf{x}_{N+1}$ under hyperparameters $\theta = (\sigma^2, \ell^*)$ is given by:
% $$
% \Sigma(\mathbf{x}_{N+1}; \sigma^2) = \sigma^2 - \sigma^2 \mathbf{k}(\mathbf{x}_{N+1}, \mathbf{X}_D; \ell^*) K(\mathbf{X}_D, \mathbf{X}_D; \ell^*)^{-1} \mathbf{k}(\mathbf{X}_D, \mathbf{x}_{N+1}; \ell^*)
% $$
% Similarly, under $\theta^* = (\sigma^{*2}, \ell^*)$, the predictive variance is:
% $$
% \Sigma(\mathbf{x}_{N+1}; \sigma^{*2}) = \sigma^{*2} - \sigma^{*2} \mathbf{k}(\mathbf{x}_{N+1}, \mathbf{X}_D; \ell^*) K(\mathbf{X}_D, \mathbf{X}_D; \ell^*)^{-1} \mathbf{k}(\mathbf{X}_D, \mathbf{x}_{N+1}; \ell^*)
% $$
% \item \textbf{Difference in Variances:} Subtracting the two variances, we get:
% \begin{align*}
% \Sigma(\mathbf{x}_{N+1}; \sigma^2) - \Sigma(\mathbf{x}_{N+1}; \sigma^{*2}) &= (\sigma^2 - \sigma^{*2}) - (\sigma^2 - \sigma^{*2}) \mathbf{c} \\
% &= (\sigma^2 - \sigma^{*2})(1 - \mathbf{c})
% \end{align*}
% where $\mathbf{c} = \mathbf{k}(\mathbf{x}_{N+1}, \mathbf{X}_D; \ell^*) K(\mathbf{X}_D, \mathbf{X}_D; \ell^*)^{-1} \mathbf{k}(\mathbf{X}_D, \mathbf{x}_{N+1}; \ell^*)$ is a constant that does not depend on the signal variance.
% \item \textbf{Non-Zero Difference:} Since $\sigma^2 \neq \sigma^{*2}$, the difference $\Sigma(\mathbf{x}_{N+1}; \sigma^2) - \Sigma(\mathbf{x}_{N+1}; \sigma^{*2})$ will be non-zero if $1 - \mathbf{c} \neq 0$.
% By Cauchy-Schwarz, $\mathbf{c} \leq k(\mathbf{x}_{N+1},\mathbf{x}_{N+1}) = \sigma^{*2}$. Also, since $K$ is positive definite, $\mathbf{c} \ge 0$.
% We need to choose $\mathbf{x}_{N+1}$ such that $\mathbf{c} \neq 1$.
% To ensure this, we can choose $\mathbf{x}_{N+1}$ such that $\mathbf{k}(\mathbf{x}_{N+1}, \mathbf{X}_D; \ell^*)$ is not in the column space of $K(\mathbf{X}_D, \mathbf{X}_D; \ell^*)$. This is generally true if the training points are distinct and we choose $\mathbf{x}_{N+1}$ to be sufficiently far from the training points. For example, we can choose $\mathbf{x}_{N+1}$ such that $\mathbf{k}(\mathbf{x}_{N+1}, \mathbf{X}_D; \ell^*)$ is very close to zero.
% \item \textbf{Conclusion:} Under the assumption of distinct training points, we can always find an $\mathbf{x}_{N+1}$ such that $1 - \mathbf{c} \neq 0$, and thus the predictive variances under $\sigma^2$ and $\sigma^{*2}$ are different. This implies local identifiability of the signal variance.
% \end{enumerate}
% \end{proof}
\begin{itemize}
\item Describe the importance of Gaussian processes and kernel methods, and their limitations.
\begin{itemize}
\item Gaussian Processes (GPs) are an important class of stochastic processes used in machine learning and statistics, particularly spatial data analysis, time series forecasting and bioinformatics. GPs offer a non-parametric framework for modeling distributions over functions, thus enabling both flexibility and uncertainty quantification. These capabilities, combined with the ability to incorporate prior knowledge and specify relationships by choice of kernel function, make Gaussian Processes effective for regression, classification, and optimization.
\item However, they have substantial computational and memory bottlenecks. Both training and inference require computing the action of the inverse kernel gram matrix, while training requires computing its log-determinant: both are $O(n^3)$ operations, where $n$ is the sample size. Further, storing the full gram matrix requires $O(n^2)$ memory. These bottlenecks necessitate scalable approximations to extend the Gaussian Processes to larger datasets.
\end{itemize}
\item Describe Nystr{\"o}m and its limitations
\item Describe SKI, how its been used in many places, but lacks theoretical guarantees
\begin{itemize}
\item Structured Kernel Interpolation (SKI) is an efficient method for scaling Gaussian Processes (GPs) to large datasets by approximating the kernel matrix using interpolation on a set of inducing points. The core idea is to express the original kernel \( k(\mathbf{x}, \mathbf{x}') \) as a combination of interpolation functions and a structured kernel matrix \( \mathbf{K}_g \) defined on a set of inducing points. However, despite its effectiveness, popularity (it has over $600$ citations, a large number for a GP paper) and ready software availability (\cite{gardner2018gpytorch} has 3.5k stars on github), it currently lacks theoretical analysis of even the error bound of the kernel approximation, much less how those errors propagate and influence Gaussian process training and inference.
\end{itemize}
\item In this paper, we bridge the gap between practice and theory for SKI.
\item Describe three contributions
\begin{itemize}
\item Error analysis for the kernel approximation, the action of the regularized Kernel inverse and the log determinant. We do this under both grid and Chebyshev spacing
\item Comparison of these results to Nystr{\"o}m, describing how they are better \amnote{hopefully}.
\item Analysis of important special cases, including stationary kernels and
\end{itemize}
\end{itemize}
% \begin{lemma}\label{lemma:ga-progress}
% For a gradient ascent step on the SKI log-likelihood, the following inequality holds for the true log-likelihood:
% \begin{align*}
% \mathcal{L}(\theta_{k+1})&\geq \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon}{6} \right)^2 - \frac{13\eta\epsilon_G^2}{48}.
% \end{align*}
% Further, if $\Vert \nabla \mathcal{L}(\theta_k)\Vert>\frac{5+\sqrt{13}}{6}\epsilon_G$, then a gradient ascent step on the SKI log-likelihood will monotonically increase the true log-likelihood.
% \end{lemma}
% \begin{proof}
% Define $\delta_k=\nabla\tilde{\mathcal{L}}(\theta_k)-\nabla \mathcal{L}(\theta_k)$. Recall the ascent (descent) Lemma from optimization, which states that due to $\mu$-smoothness,
% \[ \mathcal{L}(\theta_{k+1}) \geq \mathcal{L}(\theta_k) + \nabla \mathcal{L}(\theta_k)^T (\theta_{k+1} - \theta_k) - \frac{\mu}{2} \| \theta_{k+1} - \theta_k \|^2 \].
% Substituting the update rule:
% \begin{align*}
% \mathcal{L}(\theta_{k+1}) &\geq \mathcal{L}(\theta_k) + \eta \| \nabla \mathcal{\mu}(\theta_k) \|^2 + \eta \nabla \mathcal{L}(\theta_k)^T \delta_k - \frac{\mu \eta^2}{2} \| \nabla \mathcal{\mu}(\theta_k) + \delta_k \|^2 \\
% &\geq \mathcal{L}(\theta_k) + \eta \| \nabla \mathcal{\mu}(\theta_k) \|^2 + \eta \nabla \mathcal{L}(\theta_k)^T \delta_k - \frac{\mu \eta^2 }{2}\| \nabla \mathcal{\mu}(\theta_k) \|^2 - \frac{\mu\eta^2}{2} \| \delta_k \|^2 - \mu\eta^2\nabla \mathcal{\mu}(\theta_k)^T \delta_k.
% \end{align*}
% Using Cauchy-Schwarz and the bound on $\| \delta_k \|$:
% \[ \nabla \mathcal{L}(\theta_k)^T \delta_k \geq - \| \nabla \mathcal{L}(\theta_k) \| \| \delta_k \| \geq - \epsilon_G \| \nabla \mathcal{L}(\theta_k) \| \]
% Now letting $\eta=\frac{1}{2L}$, we have
% \begin{align*}
% \mathcal{L}(\theta_{k+1}) &\geq \mathcal{L}(\theta_k) + \eta \| \nabla \mathcal{L}(\theta_k) \|^2 -\epsilon_G\eta \Vert\nabla \mathcal{L}(\theta_k)\Vert - \frac{\mu \eta^2 }{2}\| \nabla \mathcal{L}(\theta_k) \|^2 - \frac{\mu\eta^2}{2} \epsilon_G^2 - \mu\eta^2\epsilon_G \Vert\nabla \mathcal{L}(\theta_k)\Vert\\
% &=\mathcal{L}(\theta_k)+\eta\left(1-\frac{\mu\eta}{2}\right)\Vert \nabla \mathcal{L}(\theta_k)\Vert^2-\epsilon_G\eta(1+\mu\eta)\Vert \nabla \mathcal{L}(\theta_k)\Vert-\frac{\mu\eta^2}{2}\epsilon^2
% \end{align*}
% Now $\text{Substituting }\eta=\frac{1}{2\mu}\text{ we get:}$
% \begin{align*}
% \mathcal{L}(\theta_{k+1}) &\geq \mathcal{L}(\theta_k) + \frac{3\eta}{4} \Vert \nabla \mathcal{L}(\theta_k) \Vert^2 - \frac{3\epsilon_G\eta}{4} \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{\eta\epsilon_G^2}{4}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert^2 - \frac{5\epsilon_G}{3}\Vert \nabla \mathcal{L}(\theta_k) \Vert \right) - \frac{\eta\epsilon_G^2}{4}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert^2 - \frac{5\epsilon_G}{3}\Vert \nabla \mathcal{L}(\theta_k) \Vert + \left(\frac{5\epsilon_G}{6}\right)^2 - \left(\frac{5\epsilon_G}{6}\right)^2 \right) - \frac{\eta\epsilon_G^2}{4}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon_G}{6} \right)^2 - \frac{3\eta}{4}\left(\frac{5\epsilon_G}{6}\right)^2 - \frac{\eta\epsilon_G^2}{4}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon_G}{6} \right)^2 - \frac{25\eta\epsilon_G^2}{48} - \frac{12\eta\epsilon_G^2}{48}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon_G}{6} \right)^2 - \frac{13\eta\epsilon_G^2}{48}
% \end{align*}
% and the term $\frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon_G}{6} \right)^2 - \frac{13\eta\epsilon_G^2}{48}$ will be positive if $\Vert \nabla \mathcal{L}(\theta_k)\Vert>\frac{5+\sqrt{13}}{6}\epsilon_G$, implying the second claim.
% \end{proof}
% \begin{lemma}
% For $k$ sufficiently large, we have
% \begin{align*}
% \Vert \nabla \mathcal{L}(\theta_k)\Vert &\leq \ldots \epsilon_G
% \end{align*}
% \end{lemma}
% \begin{proof}
% \end{proof}
% \section{Convergence Analysis with Approximate Log-Likelihood (Formal Proof)}
% We formally prove that gradient ascent on an approximate log-likelihood with a bounded gradient difference converges to a neighborhood of a stationary point of the true log-likelihood.
% \subsection{Assumptions}
% \begin{itemize}
% \item The true log-likelihood $\mathcal{L}(\theta)$ is $\mu$-smooth.
% \item The gradient difference is bounded: $\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \| \leq \epsilon$ for all $\theta \in \mathbb{R}^d$, where $\epsilon \geq 0$.
% \item Gradient ascent updates on the approximate log-likelihood $\tilde{\mathcal{L}}(\theta)$:
% \[ \theta_{k+1} = \theta_k + \eta \nabla \tilde{\mathcal{L}}(\theta_k) \]
% \item Step size: $0 < \eta \leq \frac{1}{2\mu}$
% \item $\mathcal{L}(\theta)$ is upper bounded by some constant $\mathcal{L}^*$.
% \end{itemize}
% \subsection{Goal}
% Show that gradient ascent on $\tilde{\mathcal{L}}(\theta)$ leads to a point where $\| \nabla \mathcal{L}(\theta_k) \|$ is approximately on the order of $\epsilon$.
% \subsection{Proof}
% \begin{enumerate}
% \item \textbf{Applying Smoothness:}
% From the $\mu$-smoothness of $\mathcal{L}(\theta)$, we have the following ascent lemma:
% \[ \mathcal{L}(\theta_{k+1}) \leq \mathcal{L}(\theta_k) + \nabla \mathcal{L}(\theta_k)^T (\theta_{k+1} - \theta_k) + \frac{\mu}{2} \| \theta_{k+1} - \theta_k \|^2 \]
% Using the update rule $\theta_{k+1} = \theta_k + \eta \nabla \tilde{\mathcal{L}}(\theta_k)$ and substituting $\nabla \tilde{\mathcal{L}}(\theta_k) = \nabla \mathcal{L}(\theta_k) + \delta_k$ where $\delta_k = \nabla \tilde{\mathcal{L}}(\theta_k) - \nabla \mathcal{L}(\theta_k)$ and $\| \delta_k \| \leq \epsilon$, we get:
% \begin{align*}
% \mathcal{L}(\theta_{k+1}) &\geq \mathcal{L}(\theta_k) + \eta \| \nabla \mathcal{L}(\theta_k) \|^2 + \eta \nabla \mathcal{L}(\theta_k)^T \delta_k - \frac{\mu \eta^2}{2} \| \nabla \mathcal{L}(\theta_k) + \delta_k \|^2 \\
% &\geq \mathcal{L}(\theta_k) + \eta \| \nabla \mathcal{L}(\theta_k) \|^2 + \eta \nabla \mathcal{L}(\theta_k)^T \delta_k - \frac{\mu \eta^2 }{2}\| \nabla \mathcal{L}(\theta_k) \|^2 - \frac{\mu\eta^2}{2} \| \delta_k \|^2 - \mu\eta^2\nabla \mathcal{L}(\theta_k)^T \delta_k
% \end{align*}
% \item \textbf{Bounding the Difference Term:}
% Using Cauchy-Schwarz and the bound on $\| \delta_k \|$:
% \[ \nabla \mathcal{L}(\theta_k)^T \delta_k \geq - \| \nabla \mathcal{L}(\theta_k) \| \| \delta_k \| \geq - \epsilon \| \nabla \mathcal{L}(\theta_k) \| \]
% Now letting $\eta=\frac{1}{2\mu}$, we have
% \begin{align*}
% \mathcal{L}(\theta_{k+1}) &\geq \mathcal{L}(\theta_k) + \eta \| \nabla \mathcal{L}(\theta_k) \|^2 -\epsilon\eta \Vert\nabla \mathcal{L}(\theta_k)\Vert - \frac{\mu \eta^2 }{2}\| \nabla \mathcal{L}(\theta_k) \|^2 - \frac{\mu\eta^2}{2} \epsilon^2 - \mu\eta^2\epsilon \Vert\nabla \mathcal{L}(\theta_k)\Vert\\
% &=\mathcal{L}(\theta_k)+\eta\left(1-\frac{\mu\eta}{2}\right)\Vert \nabla \mathcal{L}(\theta_k)\Vert^2-\epsilon\eta(1+\mu\eta)\Vert \nabla \mathcal{L}(\theta_k)\Vert-\frac{\mu\eta^2}{2}\epsilon^2
% \end{align*}
% Now $\text{Substituting }\eta=\frac{1}{2\mu}\text{ we get:}$
% \begin{align*}
% \mathcal{L}(\theta_{k+1}) &\geq \mathcal{L}(\theta_k) + \frac{3\eta}{4} \Vert \nabla \mathcal{L}(\theta_k) \Vert^2 - \frac{5\epsilon\eta}{4} \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{\eta\epsilon^2}{8}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert^2 - \frac{5\epsilon}{3}\Vert \nabla \mathcal{L}(\theta_k) \Vert \right) - \frac{\eta\epsilon^2}{8}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert^2 - \frac{5\epsilon}{3}\Vert \nabla \mathcal{L}(\theta_k) \Vert + \left(\frac{5\epsilon}{6}\right)^2 - \left(\frac{5\epsilon}{6}\right)^2 \right) - \frac{\eta\epsilon^2}{8}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon}{6} \right)^2 - \frac{3\eta}{4}\left(\frac{5\epsilon}{6}\right)^2 - \frac{\eta\epsilon^2}{8}\\
% &= \mathcal{L}(\theta_k) + \frac{3\eta}{4} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon}{6} \right)^2 - \frac{25\eta\epsilon^2}{48} - \frac{6\eta\epsilon^2}{48}\\
% &= \mathcal{L}(\theta_k) + \frac{3}{8\mu} \left( \Vert \nabla \mathcal{L}(\theta_k) \Vert - \frac{5\epsilon}{3} \right)^2 - \frac{31\epsilon^2}{96\mu}
% \end{align*}
% \item \textbf{Rearranging:}
% \[ \mathcal{L}(\theta_k) - \mathcal{L}(\theta_{k+1}) \leq \frac{31\epsilon^2}{96\mu} - \frac{3}{8\mu} \left( \| \nabla \mathcal{L}(\theta_k) \| - \frac{5\epsilon}{3} \right)^2 \]
% \[ \frac{3}{8\mu} \left( \| \nabla \mathcal{L}(\theta_k) \| - \frac{5\epsilon}{6} \right)^2 \leq \mathcal{L}(\theta_{k+1}) - \mathcal{L}(\theta_k) + \frac{31\epsilon^2}{96\mu} \]
% \item \textbf{Telescoping Sum:} Summing the inequality from $k=0$ to $K-1$:
% \[ \sum_{k=0}^{K-1} \frac{3}{8\mu} \left( \| \nabla \mathcal{L}(\theta_k) \| - \frac{5\epsilon}{6} \right)^2 \leq \sum_{k=0}^{K-1} \left( \mathcal{L}(\theta_{k+1}) - \mathcal{L}(\theta_k) \right) + \sum_{k=0}^{K-1} \frac{31\epsilon^2}{96\mu} \]
% \[ \frac{3}{8\mu} \sum_{k=0}^{K-1} \left( \| \nabla \mathcal{L}(\theta_k) \| - \frac{5\epsilon}{6} \right)^2 \leq \mathcal{L}(\theta_K) - \mathcal{L}(\theta_0) + \frac{31K\epsilon^2}{96\mu} \]
% Since $\mathcal{L}(\theta)$ is upper bounded by $\mathcal{L}^*$, we have:
% \[ \frac{3}{8\mu} \sum_{k=0}^{K-1} \left( \| \nabla \mathcal{L}(\theta_k) \| - \frac{5\epsilon}{6} \right)^2 \leq \mathcal{L}^* - \mathcal{L}(\theta_0) + \frac{31K\epsilon^2}{96\mu} \]
% \item \textbf{Bounding the Average:} Dividing by $K\frac{3}{8\mu}$:
% \[ \frac{1}{K} \sum_{k=0}^{K-1} \left( \| \nabla \mathcal{L}(\theta_k) \| - \frac{5\epsilon}{6} \right)^2 \leq \frac{8\mu(\mathcal{L}^* - \mathcal{L}(\theta_0))}{3K} + \frac{31\epsilon^2}{36} \]
% Since this inequality holds for all $K$, we can take the limit inferior:
% \[ \liminf_{K \to \infty} \frac{1}{K} \sum_{k=0}^{K-1} \left( \| \nabla \mathcal{L}(\theta_k) \| - \frac{5\epsilon}{6} \right)^2 \leq \frac{31\epsilon^2}{36} \]
% \item \textbf{Implication:} The left-hand side is an average of non-negative terms. The limit inferior of an average of non-negative terms is 0 only if the terms themselves converge to 0. Thus, the above inequality implies that there exists a subsequence of $\{\theta_k\}$ such that:
% \[ \lim_{j \to \infty} \left( \| \nabla \mathcal{L}(\theta_{k_j}) \| - \frac{5\epsilon}{6} \right)^2 = c^2 \]
% where $c^2 \leq \frac{31\epsilon^2}{36}$.
% This means there exists a subsequence that converges to a point where:
% \[ \| \nabla \mathcal{L}(\theta_{k_j}) \| \leq c + \frac{5\epsilon}{6} \]
% \[ -\left(c + \frac{5\epsilon}{6}\right) \leq \| \nabla \mathcal{L}(\theta_{k_j}) \| \leq c + \frac{5\epsilon}{6} \]
% So we know that $c + \frac{5\epsilon}{6} \leq \sqrt{\frac{31}{36}}\epsilon + \frac{5\epsilon}{6} = \left( \frac{5}{6} + \frac{\sqrt{31}}{6} \right) \epsilon$
% So we know that there exists a subsequence that converges to a point where:
% \[ \| \nabla \mathcal{L}(\theta_{k_j}) \| \leq \left( \frac{5}{6} + \frac{\sqrt{31}}{6} \right) \epsilon \]
% Since we are assuming we are in a compact space, then we know that the convergence of this subsequence implies the convergence of the entire sequence.
% \end{enumerate}
% \subsection{Implications}
% \begin{itemize}
% \item \textbf{Convergence to a Neighborhood:} Gradient ascent on the approximate log-likelihood will converge to a region where the true gradient norm $\| \nabla \mathcal{L}(\theta_k) \|$ is approximately on the order of $\epsilon$.
% \item \textbf{Accuracy Depends on $\epsilon$:} The accuracy is directly tied to the accuracy of the approximate log-likelihood's gradient.
% \item \textbf{Step Size:} A step size $\eta = \frac{1}{2\mu}$ is used in this analysis.
% \end{itemize}
% \subsection{Further Considerations}
% \begin{itemize}
% \item Smoothness of $\tilde{\mathcal{L}}(\theta)$ might also play a role.
% \item Structure of the approximation error could lead to tighter bounds.
% \end{itemize}
% The inequality
% \[ \frac{3}{8\mu}\left(\Vert \nabla \mathcal{L}(\theta_k)\Vert-\frac{5\epsilon}{3}\right)^2-\frac{31\epsilon^2}{96\mu}>0 \]
% holds when:
% \[ \Vert \nabla \mathcal{L}(\theta_k) \Vert > \frac{5 + \sqrt{31}}{6}\epsilon \approx 1.76\epsilon \]
% This means that the norm of the true gradient needs to be sufficiently large compared to the error bound $\epsilon$ for the inequality to hold. In the context of gradient ascent, this condition ensures that the algorithm is still making progress in the right direction, despite the approximation, as long as the true gradient is not too small.
% In conclusion, using an approximate log-likelihood with a bounded gradient difference, gradient ascent can still make progress and converge to a neighborhood of a stationary point of the true log-likelihood. The size of this neighborhood is determined by the accuracy of the gradient approximation.
% \section{Convergence Rate Analysis for Gradient Norms in the Proxy Setting (Corrected)}
% We analyze the convergence rate of the sum of squared gradient norms when using projected gradient ascent on an approximate log-likelihood with a bounded gradient difference.
% \subsection{Assumptions}
% \begin{itemize}
% \item The true log-likelihood $L(\theta)$ is $L$-smooth.
% \item The gradient difference is bounded: $\| \nabla L(\theta) - \nabla L_{approx}(\theta) \| \leq \epsilon$ for all $\theta \in \mathcal{D}$.
% \item Projected gradient ascent updates on the approximate log-likelihood $L_{approx}(\theta)$:
% \[ \theta_{k+1} = \Pi_{\mathcal{D}}(\theta_k + \eta \nabla L_{approx}(\theta_k)) \]
% \item Step size: $0 < \eta \leq \frac{1}{2L}$
% \item $L(\theta)$ is upper bounded by some constant $L^*$ on the domain $\mathcal{D}$.
% \item $\mathcal{D}$ is a convex set.
% \end{itemize}
% \subsection{Goal}
% Obtain a result analogous to $\frac{1}{K} \sum_{k=0}^{K-1} \| \nabla L(\theta_k) \|^2 = O\left(\frac{1}{K}\right)$, but in the proxy setting.
% \subsection{Derivation}
% \begin{enumerate}
% \item \textbf{Smoothness and Update:} From the $L$-smoothness of $L(\theta)$ and our previous steps, we have:
% \[ L(\theta_{k+1}) \geq L(\theta_k) + \frac{\eta}{2} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 - \frac{3\eta\epsilon^2}{8} \]
% Rearranging:
% \[ \frac{\eta}{2} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq L(\theta_{k+1}) - L(\theta_k) + \frac{3\eta\epsilon^2}{8} \]
% \item \textbf{Telescoping Sum:} Summing from $k=0$ to $K-1$:
% \[ \frac{\eta}{2} \sum_{k=0}^{K-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq L(\theta_K) - L(\theta_0) + \frac{3K\eta\epsilon^2}{8} \]
% Since $L(\theta) \leq L^*$, we have:
% \[ \frac{\eta}{2} \sum_{k=0}^{K-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq L^* - L(\theta_0) + \frac{3K\eta\epsilon^2}{8} \]
% \item \textbf{Bounding the Average:} Dividing by $K\eta/2$:
% \[ \frac{1}{K} \sum_{k=0}^{K-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq \frac{2(L^* - L(\theta_0))}{K\eta} + \frac{3\epsilon^2}{4} \]
% \item \textbf{Splitting the Sum:} Let $K'$ be the smallest index such that for all $k \geq K'$, $\| \nabla L(\theta_k) \| \leq \left( \frac{1}{2} + \frac{\sqrt{3}}{2} \right) \epsilon$. We can split the sum into two parts:
% \[ \frac{1}{K} \sum_{k=0}^{K'-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 + \frac{1}{K} \sum_{k=K'}^{K-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq \frac{2(L^* - L(\theta_0))}{K\eta} + \frac{3\epsilon^2}{4} \]
% \item \textbf{Bounding the Second Term:} For $k \geq K'$, we have:
% \[ \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq \left( \left( \frac{1}{2} + \frac{\sqrt{3}}{2} \right) \epsilon - \frac{\epsilon}{2} \right)^2 = \frac{3\epsilon^2}{4} \]
% Therefore:
% \[ \frac{1}{K} \sum_{k=K'}^{K-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq \frac{1}{K} \sum_{k=K'}^{K-1} \frac{3\epsilon^2}{4} \leq \frac{3\epsilon^2}{4} \]
% \item \textbf{Combining and Simplifying:} Substituting back into the inequality from step 4:
% \[ \frac{1}{K} \sum_{k=0}^{K'-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 + \frac{1}{K} \sum_{k=K'}^{K-1} \frac{3\epsilon^2}{4} \leq \frac{2(L^* - L(\theta_0))}{K\eta} + \frac{3\epsilon^2}{4} \]
% \[ \frac{1}{K} \sum_{k=0}^{K'-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq \frac{2(L^* - L(\theta_0))}{K\eta} \]
% We know that for $k < K'$, that the gradient norm is at least greater than $\frac{\epsilon}{2}$, so we can say that $\left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 > 0$ for $k < K'$. Thus, for sufficiently large $K$:
% \[ \frac{1}{K} \sum_{k=0}^{K'-1} \left( \| \nabla L(\theta_k) \| - \frac{\epsilon}{2} \right)^2 \leq \frac{2(L^* - L(\theta_0))}{K\eta} \]
% This implies that $K' \sim O(\frac{1}{K})$ since the other term does not decrease with $K$. Now we can rewrite the equation from step 4:
% \[ \frac{1}{K} \sum_{k=0}^{K-1} \| \nabla L(\theta_k) \|^2 \leq \frac{2(L^* - L(\theta_0))}{K\eta} + \frac{3\epsilon^2}{4} + \frac{1}{K} \sum_{k=0}^{K'-1} \epsilon \| \nabla L(\theta_k) \| + \frac{1}{K} \sum_{k=K'}^{K-1} \epsilon \| \nabla L(\theta_k) \| \]
% Since $K' \sim O(\frac{1}{K})$ and since we can assume that $\| \nabla L(\theta_k) \|$ is upper bounded by some constant in a compact domain, then the term $\frac{1}{K} \sum_{k=0}^{K'-1} \epsilon \| \nabla L(\theta_k) \|$ goes to 0 as $K$ goes to infinity. Further, $\frac{1}{K} \sum_{k=K'}^{K-1} \epsilon \| \nabla L(\theta_k) \| \leq \frac{1}{K} \sum_{k=K'}^{K-1} \left( \frac{1}{2} + \frac{\sqrt{3}}{2} \right) \epsilon^2 \leq \left( \frac{1}{2} + \frac{\sqrt{3}}{2} \right) \epsilon^2$
% \end{enumerate}
% \subsection{Result}
% For sufficiently large $K$, we have:
% \[ \frac{1}{K} \sum_{k=0}^{K-1} \| \nabla L(\theta_k) \|^2 \leq \frac{2(L^* - L(\theta_0))}{K\eta} + \left( \frac{5 + 2\sqrt{3}}{4} \right) \epsilon^2 \]
% \subsection{Interpretation}
% \begin{itemize}
% \item The result is essentially the same as before, but the derivation is more careful about when the inequality $\| \nabla L(\theta_k) \| \leq \left( \frac{1}{2} + \frac{\sqrt{3}}{2} \right) \epsilon$ holds.
% \item We still have a $\frac{1}{K}$ term that goes to zero as $K$ increases and a constant error term that depends on $\epsilon^2$.
% \item The convergence is still to a neighborhood where the average squared gradient norm is influenced by $\epsilon$.
% \end{itemize}
% In conclusion, by carefully accounting for the conditions under which the gradient norm bound holds, we arrive at a similar convergence result for the sum of squared gradient norms in the proxy setting. The analysis highlights the role of the approximation error $\epsilon$ in determining the final convergence behavior.
\subsection{Second-Order Derivatives of Log-Likelihood (Details)}
Here, we provide the detailed calculations for the second-order derivatives of the log-likelihood with respect to $\theta_i$ and $\theta_j$.
We start with the first-order derivative:
$$
\frac{\partial \mathcal{L}}{\partial \theta_i} = \frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right).
$$
Now, we differentiate again with respect to $\theta_j$. We will use the following matrix calculus identities:
* $\frac{\partial}{\partial \theta} (\mathbf{A}^{-1}) = -\mathbf{A}^{-1} (\frac{\partial \mathbf{A}}{\partial \theta}) \mathbf{A}^{-1}$
* $\frac{\partial}{\partial \theta} \text{tr}(\mathbf{A}\mathbf{B}) = \text{tr}(\frac{\partial \mathbf{A}}{\partial \theta} \mathbf{B}) + \text{tr}(\mathbf{A} \frac{\partial \mathbf{B}}{\partial \theta})$
* $\frac{\partial}{\partial \theta} \log|\mathbf{A}| = \text{tr}(\mathbf{A}^{-1} \frac{\partial \mathbf{A}}{\partial \theta})$
Applying these identities, we obtain:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= \frac{1}{2} \mathbf{y}^\top \frac{\partial}{\partial \theta_j} \left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \right] \mathbf{y} \\
&\quad - \frac{1}{2} \frac{\partial}{\partial \theta_j} \left[ \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right]
\end{align*}
Let's compute each term separately. For the first term:
\begin{align*}
&\frac{\partial}{\partial \theta_j} \left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \right] \\
&= \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial}{\partial \theta_j} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \\
&= -(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad - (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}
\end{align*}
For the second term:
\begin{align*}
&\frac{\partial}{\partial \theta_j} \left[ \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] \\
&= \text{tr}\left( \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \right) + \text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial}{\partial \theta_j} \left[\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right] \right) \\
&= -\text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \right) + \text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} \right)
\end{align*}
Putting everything together, we get the expression for the second-order derivative:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left[ \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \\
&\quad + \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) \right] - \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) \right].
\end{align*}
\subsection{Appendix B: Simplification of the Quadratic Form (Details)}
Here, we provide the detailed algebraic manipulations to arrive at the expression involving the trace. We start with the quadratic form:
Substituting the expression for the second-order derivative and evaluating at $\theta=\theta^*$, we get:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v}
&= -\mathbb{E}_{P^*} \left[ \sum_{i=1}^2 \sum_{j=1}^2 v_i v_j \frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j}\bigg|_{\theta = \theta^*} \right].
\end{align*}
Now, we substitute the expression for the second-order derivatives and use the fact that $\mathbb{E}_{P^*}[\mathbf{y}\mathbf{y}^\top | \mathbf{X}] = \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I}$:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} &= \frac{1}{2} \mathbb{E}_{P^*_X} \left[ \sum_{i=1}^2 \sum_{j=1}^2 v_i v_j \left( \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right) (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right) \right] - \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \frac{\partial^2 \mathbf{K}_{\theta^*}}{\partial \theta_i \partial \theta_j} \right] \right. \right. \\
&\quad \left. \left. + \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left\{ \frac{\partial^2 \mathbf{K}_{\theta^*}}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right)(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right)(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right) \right\} \right] \right) \right].
\end{align*}
After simplifying and rearranging terms, we get:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} &= \frac{1}{2} \mathbb{E}_{P^*_X} \left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left( \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i} \right) (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left( \sum_{j=1}^2 v_j \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j} \right) \right) \right] \\
&= \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
\end{align*}
where $\mathbf{A} = \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}$. This is the expression used in the proof.
% We formalize the intuition that a good initialization for gradient ascent on the SKI log-likelihood, combined with a sufficiently accurate SKI approximation, leads to a SKI MLE that is close to the true MLE. We achieve this by first bounding the difference in partial derivatives between the true and SKI log-likelihoods, assuming that the partial derivatives themselves are SPD kernels. Then, under the assumption of one point strong concavity of the true log-likelihood at the true parameter, we show that if the initialization is close to the true MLE, the resulting SKI MLE will also be close to the true MLE. Finally, we show that for the Gaussian RBF kernel, for the signal variance parameter 1) the derivative of the kernel with respect to it is itself an SPD kernel 2) local strong concavity holds when learning it while fixing the lengthscale parameter. We leave the highly challenging case of the lengthscale to future work.
\end{document}
%%%%%%%% ICML 2021 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2021} with \usepackage[nohyperref]{icml2021} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\newcommand{\jmei}[1]{\textcolor{magenta}{#1}}
\newcommand{\amnote}[1]{\textcolor{red}{#1}}
% Use the following line for the initial blind version submitted for review:
\newtheorem{claim}{Claim}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem{corollary}{Corollary}
\newtheorem{proposition}{Proposition}
\newtheorem{theorem}{Theorem}
\usepackage{icml2021}
\newtheorem{assumption}{Assumption}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{icml2021}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Error Bounds for Structured Kernel Interpolation via Multivariate Polynomial Interpolation Analysis}
\begin{document}
\twocolumn[
\icmltitle{Error Bounds for Structured Kernel Interpolation via Multivariate Polynomial Interpolation Analysis}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2021
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Aeiau Zzzz}{equal,to}
\icmlauthor{Bauiu C.~Yyyy}{equal,to,goo}
\icmlauthor{Cieua Vvvvv}{goo}
\icmlauthor{Iaesut Saoeu}{ed}
\icmlauthor{Fiuea Rrrr}{to}
\icmlauthor{Tateu H.~Yasehe}{ed,to,goo}
\icmlauthor{Aaoeu Iasoh}{goo}
\icmlauthor{Buiui Eueu}{ed}
\icmlauthor{Aeuia Zzzz}{ed}
\icmlauthor{Bieea C.~Yyyy}{to,goo}
\icmlauthor{Teoau Xxxx}{ed}
\icmlauthor{Eee Pppp}{ed}
\end{icmlauthorlist}
\icmlaffiliation{to}{Department of Computation, University of Torontoland, Torontoland, Canada}
\icmlaffiliation{goo}{Googol ShallowMind, New London, Michigan, USA}
\icmlaffiliation{ed}{School of Computation, University of Edenborrow, Edenborrow, United Kingdom}
\icmlcorrespondingauthor{Cieua Vvvvv}{c.vvvvv@googol.com}
\icmlcorrespondingauthor{Eee Pppp}{ep@eden.co.uk}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\onecolumn
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
%\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
\printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
\begin{abstract}
Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} is helps scale Gaussian Processes (GPs) to large datasets by approximating the kernel matrix via interpolation at inducing points, leading to a computational complexity of $O(n+m\log m)$, where $n$ is the sample size and $m$ is the number of inducing points. Despite its widespread use and effectiveness, SKI lacks rigorous theoretical error analysis. In particular, for a given error tolerance for the approximate kernel matrix, how many inducing points $m$ do we need, and will $O(m\log m)\leq O(n)$ hold so that the computational complexity is linear as claimed? This paper addresses this gap by deriving error bounds for the SKI kernel approximation and analyzing the impact on GP training and inference. We: (1) establish error bounds for the SKI kernel and its gram matrix; (2) examine how these errors affect maximum likelihood estimation of mean function and kernel hyperparameters; (3) evaluate the error of SKI GP posterior means and variances. Interestingly, we identify two dimensionality regimes relating SKI gram matrix error to computational complexity. For $d\leq 3$, as long as the sample size grows as we want to \textit{decrease} error, we maintain linear time. For $d>3$, the error must \textit{increase} with the sample size to maintain linear time. Our theoretical analysis thus provides crucial insights into the scalability-accuracy trade-offs of SKI and how they change with the feature dimensionality, establishing precise conditions under which linear-time GP inference can be achieved while maintaining controlled approximation error.
\end{abstract}
\section{Introduction}\label{sec:introduction}
Gaussian Processes (GPs) are an important class of stochastic processes used in machine learning and statistics, with use cases including spatial data analysis \citep{liu2021missing}, time series forecasting \citep{girard2002gaussian} and bioinformatics \citep{luo2023diseasegps}. GPs offer a non-parametric framework for modeling distributions over functions, enabling both flexibility and uncertainty quantification. These capabilities, combined with the ability to incorporate prior knowledge and specify relationships by choice of kernel function, make Gaussian Processes effective for regression, classification, and optimization.
However, GPs have substantial computational and memory bottlenecks. Both training and inference require computing the action of the inverse kernel gram matrix, while training requires computing its log-determinant: both are $O(n^3)$ operations, where $n$ is the sample size. Further, storing the full gram matrix requires $O(n^2)$ memory. These bottlenecks require scalable approximations to extend Gaussian Processes to larger datasets.
Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} is an efficient method for scaling Gaussian Processes (GPs) to large datasets by approximating the kernel matrix using interpolation on a set of inducing points. For stationary kernels, this requires $O(n+m \log m)$ computational complexity. The core idea is to express the original kernel as a combination of interpolation functions and a kernel matrix defined on a set of inducing points. However, despite its effectiveness, popularity (over $600$ citations, a large number for a GP paper) and high quality software availability (\cite{gardner2018gpytorch} has 3.5k stars on github), it currently lacks theoretical analysis. A key initial question is, given a fixed error bound for the SKI gram matrix and their use of cubic interpolation, how many inducing points are required to achieve that error bound? Given the required value of $m$ as a function of $n$, for what error tolerance is $O(n+m\log m)$ still linear? Following this, how do these errors propagate to the SKI log-likelihood and posterior inference?
In this paper, we begin to bridge the gap between practice and a theoretical understanding of SKI. We have three primary contributions: 1) error bounds for the SKI kernel and relevant quantities, including the spectral norm of the difference between the true and SKI gram matrices. 2) SKI estimation analysis: maximum likelihood estimation (MLE) of kernel hyperparameters assuming a good initialization. 3) SKI inference analysis: the error of the GP posterior means and variances at test points. Interestingly, we identify two dimensionality regimes relating SKI gram matrix error to computational complexity. For $d\leq 3$, as long as the sample size grows as we want to \textit{decrease} error, we maintain linear time. For $d>3$, the error must \textit{increase} with the sample size to maintain linear time. \amnote{not sure this next statement is true need to check.}By our downstream analysis, this implies that for small dimensionality and kernels exhibiting desirable regularity conditions, we can have arbitrarily small error in the SKI gram matrix, in our estimated parameters and in posterior inference all in linear time as long as the sample size is sufficiently large.
In section \ref{sec:related} we describe related work. In section \ref{sec:ski-background} we give a brief background on SKI. In section \ref{sec:important-quantities} we bound the error of important quantities: specifically the SKI kernel error, the SKI gram matrix error and the error of the SKI inverse's action and the SKI log-determinant. In section \ref{sec:gp-applications} we use these to analyze the error of the SKI MLE and the KL divergence, total variation and Wasserstein distances between the true and SKI predictive posteriors. We conclude in section \ref{sec:discussion} by summarizing our results and discussing limitations and future work.
For papers
\section{Related Work}\label{sec:related}
We can divide related works into three groups: those theoretically analyzing Gaussian process regression or kernel methods when using approximate kernels, SKI and its extensions, and papers developing techniques we use to obtain our guarantees. In the first group, the most relevant works are \cite{burt2019rates,burt2020convergence}, where they analyzed the sparse variational GP framework and derived bounds on the Kullback-Leibler divergence between the true posterior and the variational approximate posterior. \cite{moreno2023ski} gave bounds on the approximation error of the SKI Gram matrix. However, they only handled the case of univariate features and only bounded how much worse the SKI Gram matrix can be than the Nystr{\"o}m one. Further, they did not analyze the downstream effects on the approximate MLE or GP posterior. Also relevant are \cite{wynne2022variational,wild2021connections}, who gave a Banach space view of sparse variational GPs and connected them to the Nystr{\"o}m method, respectively. Only one of these papers \cite{moreno2023ski} treated SKI specifically, and it only covered a very special case setting. % We briefly discuss and compare to the Nystr{\"o}m method based on their results, while leaving a full Banach space analysis to future work.
%We obtain analogous results when using the SKI kernel.
In the second group, the foundational work by \cite{wilson2015kernel} that we analyze introduced SKI as a scalable method for large-scale GP inference. \cite{kapoor2021skiing} extended SKI to high-dimensional settings using the permutohedral lattice \cite{yadav2022kernel} developed a sparse grid approach to kernel interpolation that also helps address the curse of dimensionality. Most recently, \cite{ban2024malleable} proposed a flexible adaptation of SKI that allows for more adaptive kernel structures\amnote{check if these are correct interpretations}. We focus on the technique of \cite{wilson2015kernel} in this paper, but future work could extend to the settings of the latter papers.
Also relevant are papers where we leverage or extend their proof techniques. We require a multivariate extension to the error analysis of \cite{keys1981cubic} for convolutional cubic interpolation, which we derive. Further, when analyzing the hyperparameter MLE error of gradient ascent on the SKI log-likelihood, we use a strategy that is commonly used in the expectation maximization theory literature \cite{balakrishnan2017statistical,wu2016convergence,moreno2020robust} where they decompose the EM algorithm error into a population error, which decays geometrically, and a sample error, which involves a geometric sum. We instead analyze the error of gradient ascent on the SKI log-likelihood. We decompose into gradient ascent on the true log-likelihood, which decays geometrically and the error due to using the SKI log-likelihood, which also involves a geometric sum.
\section{Gaussian Processes, Structured Kernel Interpolation and Convolutional Cubic Interpolation}\label{sec:ski-background}
This section provides background on Gaussian Processes (GPs) and two key techniques for enabling scalable inference: Structured Kernel Interpolation (SKI) and Convolutional Cubic Interpolation. SKI \cite{wilson2015kernel} addresses GPs scalability issue by approximating the kernel matrix through interpolation on a set of inducing points, leveraging the efficiency of convolutional kernels. In particular, cubic convolutional kernels, as detailed in \cite{keys1981cubic}, provide a smooth and accurate interpolation scheme that forms the foundation of the SKI framework. In this paper, we focus on this cubic case as it is used by SKI. Future work may extend this to study higher-order interpolation methods. Here, we formally define these concepts and lay the groundwork for the subsequent error analysis.
\subsection{Gaussian Processes}\label{sec:gp-background}
A Gaussian process $\xi\sim \textrm{GP}(\nu,k_\theta)$ is a stochastic process $\{\xi(\textbf{x})\}_{\textbf{x}\in \mathcal{X}}$ such that any finite subcollection $\{\xi(\textbf{x}_i)\}_{i=1}^n$ is multivariate Gaussian distributed. In machine learning, we generally assume that we have index locations $\textbf{x}_i\in \mathbb{R}^d$ and observations $y_i\in \mathbb{R}$ for a set of training points $i=1,\ldots,n$ such that
\begin{align*}
y_i&=\xi(\textbf{x}_i)+\epsilon_i,\epsilon_i\sim \mathcal{N}(0,\sigma^2).
\end{align*}
where $\nu:\mathcal{X}\rightarrow\mathbb{R}$, $k_\theta:\mathcal{X}\times \mathcal{X}\rightarrow\mathbb{R}$ are the prior mean and covariance functions, respectively, with $\nu$ the mean function generally assumed to be $0$ everywhere and $k$ an SPD kernel with hyperparameters $\theta$. Given $\{\textbf{x}_i,y_i\}_{i=1}^n$ we are primariliy interested in two tasks: 1) estimate the kernel hyperparameters, given an assumed form for $k_\theta$ (e.g. RBF kernel) 2) do Bayesian inference for the posterior mean $\boldsymbol{\mu}(\cdot)\in \mathbb{R}^{T}$ and covariance $\boldsymbol{\Sigma}(\cdot)\in \mathbb{R}^{T\times T}$ at a set of test points $\{\textbf{x}_t\}_{t=1}^T$. Assuming $\nu\equiv 0$ (a mean-zero GP prior), for 1), one maximizes the log-likelihood
\begin{align}
\mathcal{L}(\theta;X) = -\frac{1}{2}\textbf{y}^\top (\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}-\frac{1}{2}\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert -\frac{N}{2}\log (2\pi)\label{eqn:log-likelihood}
\end{align}
where $\textbf{K}\in \mathbb{R}^{n\times n}$ with entries $\textbf{K}_{ij}=k_\theta(\textbf{x}_i,\textbf{x}_j)$ is the Gram matrix for the training dataset. For 2), given the kernel function and known observation variance $\sigma^2$, the posterior mean and covariance are given by
\begin{align}
\boldsymbol{\mu}(\cdot) &= \mathbf{K}_{\cdot, \mathbf{X}}\left(\mathbf{K}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\label{eqn:posterior-mean}\\
\boldsymbol{\Sigma}(\cdot) &= \textbf{K}_{\cdot,\cdot}+\sigma^2I-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}\label{eqn:posterior-covariance}
\end{align}
where $\textbf{K}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ is the matrix of kernel evaluations between test and training points. However, between the log-likelihood and the posteriors, one first needs to compute the action of the inverse of the regularized Gram matrix, $(\textbf{K}+\sigma^2 \textbf{I})^{-1}\textbf{y}$. Second, one needs to compute the log-determinant $\log \vert \textbf{K}+\sigma^2 \textbf{I}\vert$. These are both $O(n^3)$ computationally and $O(n^2)$ memory.
\subsection{Structured Kernel Interpolation}\label{subsec:ski}
Structured kernel interpolation \citep{wilson2015kernel} or (SKI) addresses these computational and memory bottlenecks by approximating the original kernel function \(k_\theta:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R},\mathcal{X}\subseteq\mathbb{R}^d\) by interpolating kernel values at a chosen set of inducing points \(\mathbf{U}=\left(\begin{array}{c}
\textbf{u}_1^\top\\
\vdots\\
\textbf{u}_m^\top
\end{array}\right)\in \mathbb{R}^{m\times d}\). The approximate kernel function \(\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow\mathbb{R}\) can be expressed as:
\[
\tilde{k}(\mathbf{x}, \mathbf{x}') = \mathbf{w}(\mathbf{x})^{\top} \mathbf{K}_{\mathbf{U}} \mathbf{w}(\mathbf{x}')
\]
where \(\mathbf{K}_{\mathbf{U}}\in \mathbb{R}^{m\times m}\) is the kernel matrix computed on the inducing points, and \(\mathbf{w}(\mathbf{x}),\mathbf{w}(\mathbf{x}')\in \mathbb{R}^m\) are vectors of interpolation weights using (usually cubic) convolutional kernel $u:\mathbb{R}\rightarrow \mathbb{R}$ for the points $\textbf{x}$ and $\textbf{x}'$, respectively. One can then form the SKI Gram matrix $\tilde{\textbf{K}}=\textbf{W}\textbf{K}_\textbf{U}\textbf{W}^\top$ with $\textbf{W}$ a \textit{sparse} matrix of $L$ interpolation weights per row for a polynomial of degree $L-1$. By exploiting the sparsity of each row, for stationary kernels this leads to a computational complexity of $O(nL+m\log m)$ and a memory complexity of $O(nL+m)$.
In order to learn kernel hyperparameters, one can maximize the SKI approximation to the log-likelihood (henceforth the SKI log-likelihood)
\begin{align*}
\tilde{\mathcal{L}}(\theta;X)
&=-\frac{1}{2}\textbf{y}^\top (\tilde{\textbf{K}}+\sigma^2 \textbf{I})^{-1}\textbf{y}-\frac{1}{2}\log \vert \tilde{\textbf{K}}+\sigma^2 \textbf{I}\vert -\frac{N}{2}\log (2\pi)
\end{align*}
Given the SKI kernel $\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ with learned hyperparameters, one can do posterior inference of the SKI approximations to the mean $\tilde{\boldsymbol{\mu}}(\cdot)$ and covariance $\tilde{\boldsymbol{\Sigma}}(\cdot)$ at a set of $T$ test points $\cdot$ as
\begin{align*}
\tilde{\boldsymbol{\mu}}(\cdot)&=\tilde{\mathbf{K}}_{\cdot, \mathbf{X}}\left(\tilde{\mathbf{K}}+\sigma^{2} \mathbf{I}\right)^{-1} \mathbf{y}\\
\tilde{\boldsymbol{\Sigma}}(\cdot)&=\tilde{\textbf{K}}_{\cdot,\cdot}+\sigma^2I-\tilde{\textbf{K}}_{\cdot,\textbf{X}}^\top (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}
\end{align*}
where $\tilde{\textbf{K}}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ is the matrix of SKI kernels between test points and training points and $\tilde{\textbf{K}}_{\cdot,\cdot}\in \mathbb{R}^{T\times T}$ is the SKI Gram matrix for the test points.
\subsection{Convolutional Cubic Interpolation}\label{subsec:convolutional-cubic-interpolation}
Convolutional cubic interpolation \citep{keys1981cubic} is a method used to approximate the value of a function at arbitrary points within a domain, given its values on a regular grid. It leverages a cubic convolution kernel to produce a smooth, continuous estimate of the underlying function. The method calculates a weighted average of the known values, where the weights are determined by the cubic convolution kernel, which is a piecewise polynomial function specifically designed so that the approximation interpolates and is continuously differentiable. We formalize this using the definitions of the cubic convolutional interpolation kernel and the tensor-product cubic convolutional function below.
\begin{definition}\label{def:cubic-interpolation-kernel}
The cubic convolutional interpolation kernel $u:\mathbb{R}\rightarrow\mathbb{R}$ is given by
$$
u(s)\equiv\begin{cases}
1, & s=0\\
\frac{3}{2}\vert s\vert^3-\frac{5}{2}\vert s\vert^2+1, & 0<\vert s\vert <1\\
-\frac{1}{2}\vert s\vert^3+\frac{5}{2}\vert s\vert^2-4\vert s\vert+2, & 1<\vert s\vert<2\\
0, & \text{otherwise}
\end{cases}
$$
\end{definition}
\begin{definition}\label{def:tensor-product-cubic-interpolation-alt1}
Let $\mathbf{x} = (x_1, x_2, ..., x_d) \in \mathbb{R}^d$ be a d-dimensional point. Let $f:\mathbb{R}^d \rightarrow \mathbb{R}$ be a function defined on a regular grid with spacing $h$ in each dimension. Let $\mathbf{c_x}$ denote the grid point closest to $\mathbf{x}$. The tensor-product cubic convolutional interpolation function $g:\mathbb{R}^d\rightarrow \mathbb{R}$ is defined as:
$$
g(\mathbf{x}) \equiv \sum_{\mathbf{k} \in \{-1, 0, 1, 2\}^d} f(\mathbf{c_x} + h\mathbf{k}) \prod_{j=1}^d u\left(\frac{x_j - (\mathbf{c_x})_j - h k_j}{h}\right),
$$
where $u$ is the cubic convolutional interpolation kernel (defined in Definition \ref{def:cubic-interpolation-kernel}), and $\mathbf{k} = (k_1, \ldots, k_d)$ is a vector of integer indices.
\end{definition}
\section{Important Quantities}\label{sec:important-quantities}
This section derives bounds for key quantities in Structured Kernel Interpolation (SKI). Section \ref{subsec:polynomial-interpolation-preliminaries} covers polynomial interpolation preliminaries, focusing on multivariate tensor-product cubic convolutional interpolation and its error characteristics. Section \ref{subsubsec:elementwise-error} provides a bound on the elementwise error between the true kernel and its SKI approximation, foundational for SKI accuracy. In Section \ref{subsubsec:spectral-norm-error}, we extend this to the spectral norm error of the SKI approximation for the training Gram matrix and train-test kernel matrix, key for expressing various downstream errors. Finally, we present conditions on the number of inducing points for achieving specific error tolerance $\epsilon$ and a theorem on error necessary for linear time complexity, noting linear time always holds for $d\leq 3$ with sufficiently large samples.
\subsection{Preliminaries for Polynomial Interpolation}\label{subsec:polynomial-interpolation-preliminaries}
This subsection analyzes the error of multivariate tensor-product cubic convolutional interpolation, a crucial part of the SKI framework. We define the interpolation kernel and function, and establish a lemma on how its error relates to grid spacing and input dimensionality, extending the work of \cite{keys1981cubic} to multivariate settings. Additionally, we highlight the curse of dimensionality for kernel interpolation weights.
The next lemma shows that multivariate tensor-product cubic convolutional interpolation retains error cubic in the grid spacing of \cite{keys1981cubic}, but exhibits exponential error growth with increasing dimensions. The proof uses induction on dimensions, starting with the 1D case from Keys.
\begin{lemma}\label{lemma:tensor-product-interpolation-error}
Assume that the grid spacing is fixed across all dimensions, denoted by $h$. The error of tensor-product cubic convolutional interpolation is $O(c^dh^3)$ for some constant $c>0$.
\end{lemma}
\begin{proof}
We define a sequence of intermediate interpolation functions. Let $g_0(\mathbf{x}) \equiv f(\mathbf{x})$ be the original function. For $i = 1, \ldots, d$, we recursively define $g_i(\mathbf{x})$ as the function obtained by interpolating $g_{i-1}$ along the $i$-th dimension using the cubic convolution kernel $u$:
$$
g_i(\mathbf{x}) \equiv \sum_{k=-1}^2 g_{i-1}\left(\mathbf{x} + \left( (\mathbf{c_x})_i - x_i + kh\right)\mathbf{e}_i \right) u\left(\frac{x_i - (\mathbf{c_x})_i - kh}{h}\right).
$$
Here, $\mathbf{c_x}$ is the grid point closest to $\mathbf{x}$, and $\mathbf{e}_i$ is the $i$-th standard basis vector. Thus, $g_1(\mathbf{x})$ interpolates $f$ along the first dimension, $g_2(\mathbf{x})$ interpolates $g_1$ along the second dimension, and so on, until $g_d(\mathbf{x}) = g(\mathbf{x})$ is the final tensor-product interpolated function.
We analyze the error accumulation across multiple dimensions using induction. Using \cite{keys1981cubic}, the error introduced by interpolating a thrice continuous differentiable function along a single dimension with the cubic convolution kernel is uniformly bounded over the interval domain by $Kh^3$ for some constant $K > 0$, provided the grid spacing $h$ is sufficiently small. This gives us the base case:
$$
|g_1(\mathbf{x}) - g_0(\mathbf{x})| \leq Kh^3.
$$
For the inductive step, assume that for some $i=k$ the error is uniformly bounded by
$$
|g_k(\mathbf{x}) - g_{k-1}(\mathbf{x})| \leq c^{k-1}Kh^3.
$$
We want to show that this bound also holds for $i=k+1$. We can express the difference $g_{k+1}(\mathbf{x}) - g_k(\mathbf{x})$ as follows:
\begin{align*}
g_{k+1}(\mathbf{x}) - g_k(\mathbf{x}) &= \sum_{k_{k+1}=-1}^2 g_k\left(\mathbf{x} + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) \\
&\quad - g_k(\mathbf{x}) \\
&= \sum_{k_{k+1}=-1}^2 \left[\sum_{k_k=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) \right. \\
&\quad \left. u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right)\right] u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) \\
&\quad - \sum_{k_k=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right) \\
&= \sum_{k_k=-1}^2 u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right) \left[\sum_{k_{k+1}=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) \right. \\
&\quad \left. u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k\right)\right].
\end{align*}
The inner term in the last expression represents the difference between interpolating $g_{k-1}$ along the $(k+1)$-th dimension and $g_{k-1}$ itself, evaluated at $\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k$. This can be written as:
\begin{align*}
&\sum_{k_{k+1}=-1}^2 g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k + ((\mathbf{c_x})_{k+1} - x_{k+1} + k_{k+1}h) \mathbf{e}_{k+1} \right) u\left(\frac{x_{k+1} - (\mathbf{c_x})_{k+1} - k_{k+1}h}{h}\right) \\
&\quad - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k\right) \\
&= g_k\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right).
\end{align*}
Therefore, we can bound the error as follows:
\begin{align*}
|g_{k+1}(\mathbf{x}) - g_k(\mathbf{x})| &\leq \left|\sum_{k_k=-1}^2 u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right)\right| \cdot \left|g_k\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right)\right|.
\end{align*}
Let $c>0$ be a uniform upper bound for $\left|\sum_{k_k=-1}^2 u\left(\frac{x_k - (\mathbf{c_x})_k - k_k h}{h}\right)\right|$, which exists because $u$ is bounded. By the inductive hypothesis, we have $\left|g_k\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right) - g_{k-1}\left(\mathbf{x} + ((\mathbf{c_x})_k - x_k + k_k h) \mathbf{e}_k \right)\right| \leq c^{k-1}Kh^3$. Thus,
$$
|g_{k+1}(\mathbf{x}) - g_k(\mathbf{x})| \leq c \cdot c^{k-1}Kh^3 = c^k Kh^3.
$$
This completes the inductive step.
Finally, we bound the total error $|g(\mathbf{x}) - f(\mathbf{x})| = |g_d(\mathbf{x}) - g_0(\mathbf{x})|$ by summing the errors introduced at each interpolation step:
$$
|g(\mathbf{x}) - f(\mathbf{x})| \leq \sum_{i=1}^d |g_i(\mathbf{x}) - g_{i-1}(\mathbf{x})| \leq \sum_{i=1}^d c^{i-1}Kh^3 = Kh^3 \sum_{i=0}^{d-1} c^i.
$$
The last sum is a geometric series, which evaluates to $Kh^3 \frac{1 - c^d}{1 - c}$. For a fixed $c>1$ (independent of $d$), this expression is $O(c^{d})$ when $d$ is large. Therefore, tensor-product cubic convolutional interpolation has $O(c^d h^3)$ error.
\end{proof}
The next lemma shows that when using a product kernel for $d$-dimensional kernel regression (where cubic convolutional interpolation is a special case), the sum of weights suffers from the curse of dimensionality. The proof strategy involves expressing the multi-dimensional sum as a product of sums over each individual dimension, leveraging the initial condition on the one-dimensional bound for each dimension, and taking advantage of the structure of the Cartesian grid.
\begin{lemma}\label{lemma:cubic-interpolation-weights-curse-dimensionality}
Let $u: \mathbb{R} \rightarrow \mathbb{R}$ be a one-dimensional kernel function such that for some constant $c > 0$, the sum of weights in a 1D kernel regression is bounded by $c$ over its domain. That is, for any $x \in \mathbb{R}$ and a set of data points $\{x_i\}_{i=1}^n \subset \mathbb{R}$,
$$
\sum_{i=1}^n u\left(\frac{x - x_i}{h}\right) \leq c,
$$
where $h > 0$ is the bandwidth.
Let $u_d: \mathbb{R}^d \rightarrow \mathbb{R}$ be a d-dimensional product kernel defined as:
$$
u_d\left(\frac{x - x_i}{h}\right) = \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right),
$$
where $x = (x^{(1)}, x^{(2)}, ..., x^{(d)}) \in \mathbb{R}^d$ and $x_i = (x_i^{(1)}, x_i^{(2)}, ..., x_i^{(d)}) \in \mathbb{R}^d$ are d-dimensional points. Assume the data points $\{x_i\}_{i=1}^n$ ($n$ may differ from the univariate case) lie on a fixed d-dimensional grid $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$, where each $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is a finite set of $n_j$ grid points along the j-th dimension for $j = 1, 2, ..., d$.
Then, for any $x \in \mathbb{R}^d$, the sum of weights in the d-dimensional kernel regression is bounded by $c^d$:
$$
\sum_{i=1}^n u_d\left(\frac{x - x_i}{h}\right) \leq c^d.
$$
\end{lemma}
\begin{proof}
Let the fixed d-dimensional grid be defined by the Cartesian product of d sets of 1-dimensional grid points: $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$, where $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is the set of grid points along the j-th dimension.
We start with the sum of weights in the d-dimensional case:
$$
\sum_{i=1}^n u_d\left(\frac{x - x_i}{h}\right) = \sum_{i=1}^n \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right)
$$
Since the data points lie on the fixed grid $G$, we can rewrite the outer sum as a nested sum over the grid points in each dimension:
$$
\sum_{i=1}^n \prod_{j=1}^d u\left(\frac{x^{(j)} - x_i^{(j)}}{h}\right) = \sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right)
$$
Now we can change the order of summation and product, as proven in Lemma \ref{lemma:switch-sum-product}:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) \right)
$$
By the assumption of the lemma, we know that for each dimension $j$, the sum of weights is bounded by $c$. Note that $\{p_{k_j}^{(j)}\}_{k_j=1}^{n_j}$ is simply a set of points in $\mathbb{R}$, thus:
$$
\sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) \leq c
$$
Therefore, we have:
$$
\prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} u\left(\frac{x^{(j)} - p_{k_j}^{(j)}}{h}\right) \right) \leq \prod_{j=1}^d c = c^d
$$
Thus, we have shown that:
$$
\sum_{i=1}^n u_d\left(\frac{x - x_i}{h}\right) \leq c^d
$$
\end{proof}
\subsection{Error Bounds for the Ski Kernel}\label{subsec:ski-kernel-error-bounds}
This subsection analyzes the error introduced by the SKI approximation of the kernel function. We first derive bounds on the elementwise difference between the true kernel \(k(\textbf{x},\textbf{x}')\) and its SKI approximation \(\tilde{k}(\textbf{x},\textbf{x}')\). We then apply these elementwise bounds to derive further error bounds on the SKI kernel matrices, which will be crucial for understanding the downstream effects of the SKI approximation on Gaussian process hyperparameter estimation and posterior inference.
\subsubsection{Elementwise}\label{subsubsec:elementwise-error}
The following Lemma allows us to bound the absolute difference between the true and SKI kernels \textit{uniformly} in terms of the number of inducing points, an upper bound on the absolute sum of interpolation weights (in the Lagrange case this would be the Lebesgue constant), and the error due to polynomial interpolation.
\begin{lemma}\label{lemma:ski-kernel-elementwise-error}
The SKI kernel $\tilde{k}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ with $m$ inducing points and interpolation degree $L-1$ has error
\begin{align*}
\vert k(\textbf{x},\textbf{x}')-\tilde{k}(\textbf{x},\textbf{x}')\vert&\leq \delta_{m,L}+\sqrt{L}c^d\delta_{m,L}
% &=c^d\frac{2M_\rho(g)}{\rho-1}\rho^{-L}+\sqrt{L}c^{2d} \frac{2M_\rho(g)}{\rho-1}\rho^{-L}\\
% &=O\left(c^d\sqrt{L}\rho^{-L}\right)
\end{align*}
where $\delta_{m,L}$ is an upper bound on the error of multivariate convolutional interpolation with $m$ inducing points and interpolation degree $L-1$ and $c>0$ is an upper bound on the absolute sum of the weights for 1d kernel interpolation. If the convolutional interpolation is cubic \cite{keys1981cubic}, then we have
\begin{align*}
\vert k(\textbf{x},\textbf{x}')-\tilde{k}(\textbf{x},\textbf{x}')\vert&\leq \sqrt{L}O(c^{2d}h^3)
\end{align*}
\end{lemma}
\begin{proof}
Recall that SKI approximates the kernel as
\begin{align*}
k(\mathbf{x}, \mathbf{x}') &\approx \tilde{k}(\mathbf{x}, \mathbf{x}')\\
&= \boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}'),
\end{align*}
Let $\textbf{K}_{\textbf{U},\textbf{x}'}\in \mathbb{R}^m$ be the vector of kernels between the inducing points and the vector $\textbf{x}'$
\begin{align}
\vert k(\mathbf{x}, \mathbf{x}') -\tilde{k}(\mathbf{x}, \mathbf{x}')\vert &= \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} +\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\nonumber\\
&\leq \vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert\nonumber\\
&\leq \delta_{m,L}+\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert \textrm{ since $\vert k(\mathbf{x}, \mathbf{x}')-\boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'} \vert$ is a single polynomial interpolation}\label{eqn:applying-single-poly-interp}
\end{align}
Now note that $\textbf{w}(x)\in \mathbb{R}^m$ is a sparse matrix with at most $L$ non-zero entries. Thus, letting $\tilde{\textbf{w}}(x)\in \mathbb{R}^L$ be the non-zero entries of $\textbf{w}(x)$ and similarly $\tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}\in \mathbb{R}^L$ be the entries of $\textbf{K}_{\textbf{U},\textbf{x}'}$ in the dimensions corresponding to non-zero entries of $\textbf{w}(x)\in \mathbb{R}^m$, while $\tilde{\textbf{K}}_{\textbf{U}}\in \mathbb{R}^{L\times m}$ is the analogous matrix for $\textbf{K}_{\textbf{U}}$, we have
\begin{align}
\vert \boldsymbol{w}(\mathbf{x})^\top \textbf{K}_{\textbf{U},\textbf{x}'}-\boldsymbol{w}(\mathbf{x})^\top \mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\vert&=\vert \tilde{\textbf{w}}(\textbf{x})^\top \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\textbf{w}}(\textbf{x})^\top \tilde{\textbf{K}}_\textbf{U}\textbf{w}(\textbf{x}')\vert\nonumber\\
&\leq \Vert \tilde{\textbf{w}}(\textbf{x})\Vert_2\Vert \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\mathbf{K}}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_2\nonumber\\
&\leq c^d\sqrt{L}\Vert \tilde{\textbf{K}}_{\textbf{U},\textbf{x}'}-\tilde{\mathbf{K}}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')\Vert_\infty\nonumber\\
&\leq \sqrt{L}c^d\delta_{m,L}\label{eqn:applying-sparsity-single-kernel-evaluation}
\end{align}
where the last line follows as each element of $\mathbf{K}_{\textbf{U}} \boldsymbol{w}(\mathbf{x}')$ is a polynomial interpolation approximating each element of $\textbf{K}_{\textbf{U},\textbf{x}'}$. Plugging Eqn. \ref{eqn:applying-sparsity-single-kernel-evaluation} into Eqn. \ref{eqn:applying-single-poly-interp} gives us the desired initial result and Lemma \ref{lemma:tensor-product-interpolation-error} gives us the result when the convolutional kernel is cubic.
\end{proof}
\subsubsection{Spectral Norm Error}\label{subsubsec:spectral-norm-error}
We now transition from elementwise error bounds for the SKI kernel approximation to bounds on the spectral norm of the difference between the true Gram matrix and its SKI approximation. This is both of independent interest but will also be important to analyzing the error of the SKI log-determinant and action of the SKI regularized inverse: both important quantities for hyperparameter optimization and posterior inference. We also provide a bound on the spectral norms of both the SKI train/test kernel matrix's approximation error and the matrix itself. These are useful when analyzing the error of the predictive mean and variance in Gaussian process regression.
\begin{proposition}\label{prop:spectral-norm}
Let $\gamma_{n,m,L}\equiv n(\delta_{m,L}+\sqrt{L}c^d\delta_{m,L})$, where $\delta_{m,L}$ is an upper bound on the error of multivariate interpolation with $m$ inducing points and interpolation degree $L-1$, and $c>0$ is an upper bound on the absolute sum of the weights for 1D kernel interpolation. Then, for the SKI approximation $\tilde{\textbf{K}}$ of the true Gram matrix $\textbf{K}$, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq \gamma_{n,m,L}
\end{align*}
and for tensor-product cubic convolutional interpolation we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq O(nc^{2d} h^3)
\end{align*}
\end{proposition}
\begin{proof}
Recall that for any matrix $\textbf{A}$, $\Vert \textbf{A} \Vert_2 \leq \sqrt{\Vert \textbf{A} \Vert_1 \Vert \textbf{A} \Vert_\infty}$. Since $\textbf{K}-\tilde{\textbf{K}}$ is symmetric, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2&\leq \sqrt{\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_1\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty} = \Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty
\end{align*}
Furthermore, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty$ is the maximum absolute row sum of $\textbf{K}-\tilde{\textbf{K}}$. Since there are $n$ rows and, by Lemma \ref{lemma:ski-kernel-elementwise-error}, each element of $\textbf{K} - \tilde{\textbf{K}}$ is bounded by $\delta_{m,L}+\sqrt{L}c^d\delta_{m,L}$ in absolute value, we have
\begin{align*}
\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_\infty &\leq n \left(\delta_{m,L}+\sqrt{L} c^d\delta_{m,L}\right) = \gamma_{n,m,L}.
\end{align*}
Therefore, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2 \leq \gamma_{n,m,L}$.
\end{proof}
\begin{lemma}\label{lemma:test-train-kernel-matrix-error}
Let $\textbf{K}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ be the matrix of kernel evaluations between $T$ test points and $n$ training points, and let $\tilde{\textbf{K}}_{\cdot,\textbf{X}}\in \mathbb{R}^{T\times n}$ be the corresponding SKI approximation. Then
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq \max(\gamma_{T,m,L},\gamma_{n,m,L}).
\end{align*}
\end{lemma}
\begin{proof}
Using the same reasoning as in Proposition \ref{prop:spectral-norm}, we have
\begin{align*}
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2&\leq \sqrt{\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty} \\
&\leq \max \left(\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1, \Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty\right).
\end{align*}
Now, $\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_1$ is the maximum absolute column sum, which is less than or equal to $T(\delta_{m,L} + \sqrt{L}c^d\delta_{m,L}) = \gamma_{T,m,L}$. Similarly, $\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_\infty$ is the maximum absolute row sum, which is upper bounded by $n(\delta_{m,L} + \sqrt{L}c^d\delta_{m,L}) = \gamma_{n,m,L}$. Therefore,
$$
\Vert \textbf{K}_{\cdot,\textbf{X}}-\tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq \max(\gamma_{T,m,L},\gamma_{n,m,L}).
$$
\end{proof}
\begin{lemma}\label{lemma:ski-test-train-bound}
Let $\tilde{\textbf{K}}_{\cdot,\textbf{X}} \in \mathbb{R}^{T \times n}$ be the matrix of SKI kernel evaluations between $T$ test points and $n$ training points, where the SKI approximation uses $m$ inducing points. Let $\textbf{W}(\cdot) \in \mathbb{R}^{T \times m}$ and $\textbf{W}(\textbf{X}) \in \mathbb{R}^{n \times m}$ be the matrices of interpolation weights for the test points and training points, respectively. Assume that the interpolation scheme is such that the sum of the absolute values of the interpolation weights for any point is bounded by $c^d$, where $c>0$ is a constant. Let $\textbf{K}_{\textbf{U}} \in \mathbb{R}^{m \times m}$ be the kernel matrix evaluated at the inducing points. If the kernel function $k$ is bounded such that $|k(\textbf{x}, \textbf{x}')| \leq M$ for all $\textbf{x}, \textbf{x}'$ in the domain of interest, then:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq \sqrt{Tn} m c^{2d} M
$$
\end{lemma}
\begin{proof}
By the definition of the SKI approximation and the submultiplicativity of the spectral norm, we have:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 = \Vert \textbf{W}(\cdot)\textbf{K}_{\textbf{U}}(\textbf{W}(\textbf{X}))^\top\Vert_2 \leq \Vert \textbf{W}(\cdot)\Vert_2 \Vert\textbf{K}_{\textbf{U}}\Vert_2 \Vert\textbf{W}(\textbf{X})\Vert_2
$$
We now bound each term.
1. **Bounding $\Vert \textbf{W}(\cdot)\Vert_2$ and $\Vert \textbf{W}(\textbf{X})\Vert_2$:**
Since the spectral norm is induced by the Euclidean norm, and using the assumption that the sum of absolute values of interpolation weights for any point is bounded by $c^d$, we have
$$\Vert \textbf{W}(\cdot)\Vert_2 \leq \sqrt{\Vert \textbf{W}(\cdot)\Vert_1 \Vert \textbf{W}(\cdot)\Vert_\infty} \leq \sqrt{T c^d \cdot c^d} = \sqrt{T} c^d.$$
Similarly, $\Vert\textbf{W}(\textbf{X})\Vert_2 \leq \sqrt{n}c^d$.
2. **Bounding $\Vert\textbf{K}_{\textbf{U}}\Vert_2$:**
Since $\textbf{K}_{\textbf{U}}$ is symmetric, $\Vert \textbf{K}_{\textbf{U}} \Vert_2 \leq \Vert \textbf{K}_{\textbf{U}} \Vert_\infty$. Each entry of $\textbf{K}_{\textbf{U}}$ is bounded by $M$ (by the boundedness of $k$), and each row has $m$ entries, so $\Vert \textbf{K}_{\textbf{U}} \Vert_\infty \leq mM$. Thus, $\Vert\textbf{K}_{\textbf{U}}\Vert_2 \leq mM$.
Combining these bounds, we get:
$$
\Vert \tilde{\textbf{K}}_{\cdot,\textbf{X}}\Vert_2 \leq (\sqrt{T} c^d) (mM) (\sqrt{n} c^d) = \sqrt{Tn} m c^{2d} M
$$
as required.
\end{proof}
\subsection{Achieving a Desired Error in Spectral Norm}\label{subsec:desired-error-spectral-norm}
Here, we derive a lower bound on the number of inducing points $m$ required to achieve a desired error tolerance $\epsilon > 0$ for the SKI Gram matrix when using cubic convolutional interpolation. We assume the domain is $[-D, D]^d$, where $D > 0$ is a constant that determines the size of the domain.
We can directly determine a sufficient number of inducing points needed for a given $\epsilon > 0$ and domain $[-D, D]^d$.
\begin{corollary}\label{cor:inducing-points-count-alt}
If $L=4$ (cubic interpolation), the domain is $[-D, D]^d$, and we use convolutional cubic interpolation \cite{keys1981cubic}, then to achieve a spectral norm error of $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$, it is sufficient to choose the number of inducing points $m$ such that:
$$
m \geq \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
$$
for some constant $K'$ that depends only on the kernel function and the interpolation scheme.
\end{corollary}
\begin{proof}
We want to choose $m$ such that the spectral norm error $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$. From Proposition \ref{prop:spectral-norm}, we have:
$$
\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq n(1 + \sqrt{L}c^d) \delta_{m,L}
$$
For cubic interpolation ($L=4$), Lemma \ref{lemma:tensor-product-interpolation-error}, combined with the analysis in Lemma \ref{lemma:tensor-product-interpolation-error}, gives us:
$$
\delta_{m,L} \leq K' c^{2d} h^3
$$
where $K'$ is a constant that depends only on the kernel function (through its derivatives) and the interpolation scheme, but not on $n$, $m$, $h$, or $d$.
Therefore, a sufficient condition to ensure $\Vert \textbf{K} - \tilde{\textbf{K}} \Vert_2 \leq \epsilon$ is:
\begin{equation} \label{eq:sufficient_condition_final}
n(1 + 2c^d) K' c^{2d} h^3 \leq \epsilon
\end{equation}
Since the inducing points are placed on a regular grid with spacing $h$ in each dimension, and the domain is $[-D,D]^d$ and assuming that $2D\mod h\equiv 0$, the number of inducing points $m$ satisfies:
$$
m = \left(\frac{2D}{h}\right)^d
$$
We can rearrange this to get:
$$
h = \frac{2D}{m^{1/d}}
$$
Substituting this into the sufficient condition \eqref{eq:sufficient_condition_final}, we get:
$$
n (1 + 2c^d) K' c^{2d} \left(\frac{2D}{m^{1/d}}\right)^3 \leq \epsilon
$$
Rearranging to isolate $m$, we obtain:
$$
m^{3/d} \geq \frac{n}{\epsilon} (1 + 2c^d) K' c^{2d} (8D^3)
$$
$$
m \geq \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
$$
\end{proof}
This result shows that the number of inducing points grows
\begin{itemize}
\item At a cube root rate with the number of observations $n$ and as error tolerance $\epsilon$ decreases. Thus, as we want a tighter error tolerance or have more observations we need more inducing points.
\item Linearly with the volume of the domain $(2D)^d$. Thus, if our observations are concentrated in a small region and we select an appropriately sized domain to cover it we need fewer inducing points.
\item Exponentially with the dimension $d$.
\end{itemize}
\subsection{Lower Bound on Error for Linear Time}\label{subsec:linear-time}
The next theorem establishes a condition on the spectral norm error, $\epsilon$, that ensures linear-time $O(n)$ computational complexity for SKI. The core idea is to choose the number of inducing points, $m$, large enough to satisfy the error bound based on Corollary \ref{cor:inducing-points-count-alt}, while simultaneously ensuring that $m$ does not grow faster than $O(n/\log n)$, thus ensuring linear-time complexity. By combining these two requirements, we can derive a lower bound for $\epsilon$ as a function of $n$, the dimensionality $d$, the domain size $D$, and the constant $c$ related to the interpolation weights.
Interestingly, the theorem implies a fundamental difference between two dimensionality regimes. For $d \leq 3$, the lower bound on $m$ required for a fixed $\epsilon$ grows more slowly than $n/\log n$. This means that for any fixed $\epsilon > 0$, SKI with cubic interpolation is guaranteed to be a linear-time algorithm for sufficiently large $n$. In contrast, for $d > 3$, the lower bound on $m$ required for a fixed $\epsilon$ eventually grows faster than $n/\log n$. Thus, to maintain linear-time complexity for $d > 3$, we must allow the error $\epsilon$ to increase with $n$. This demonstrates that the ``curse of dimensionality" significantly impacts the scalability of SKI, making it challenging to achieve both high accuracy and linear-time complexity in higher dimensions. The theorem quantifies this trade-off, showing precisely how much error growth is necessary as $n$ and $d$ increase.
\begin{theorem}[Sufficient Condition for Linear Time Complexity]
Consider $d\geq 3$. If the domain is $[-D, D]^d$, we use convolutional cubic interpolation ($L=4$), and we choose $\epsilon>0$ satisfying
\begin{equation} \label{eq:epsilon_condition}
\epsilon \geq \frac{(1 + 2c^d) K' 8 c^{2d} D^3}{C^{3/d}} \cdot \frac{n (\log n)^{3/d}}{n^{3/d}}
\end{equation}
for some constants $K,C>0$ that depend on the kernel function and the interpolation scheme. then if we choose
\begin{align*}
m &= \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
\end{align*}
then we have both $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2\leq \epsilon$ and
SKI computational complexity of $O(n)$.
\end{theorem}
\begin{proof}
The idea is that we want to have an error sufficiently large so that when we choose $m$ based on Corollary \ref{cor:inducing-points-count-alt}, we have $m=O\left(\frac{n}{\log n}\right)$.
Assume that
$$
\epsilon \geq \frac{(1 + 2c^d) K' 8 c^{2d} D^3}{C^{3/d}} \cdot \frac{n (\log n)^{3/d}}{n^{3/d}}.
$$
Rearranging this we obtain
\begin{align*}
\left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}&\leq C\frac{n}{\log n}.\\
&=O\left(\frac{n}{\log n}\right).
\end{align*}
Now taking
\begin{align*}
m &= \left( \frac{n}{\epsilon} (1 + 2c^d) K' (8 c^{2d} D^3) \right)^{d/3}
\end{align*}
we have that $m=O\left(\frac{n}{\log n}\right)$ and by Corollary \ref{cor:inducing-points-count-alt}, $\Vert \textbf{K}-\tilde{\textbf{K}}\Vert_2\leq \epsilon$. Now plugging in $\frac{n}{\log n}$ into $m\log m$ we obtain
\begin{align*}
O\left(m\log m\right)&=O\left(\frac{n}{\log n}\log \frac{n}{\log n}\right)\\
&=O\left(\frac{n}{\log n}\log n-\frac{n}{\log n}\log \log n\right)\\
&=O(n)
\end{align*}
as desired.
\end{proof}
\subsection{Additional Quantities}\label{subsec:additional-quantities}
\subsubsection{Action of Regularized Inverse}\label{subsubsec:action-regularized-inverse}
\begin{lemma}\label{lemma:action-inverse-error}
Let \(\mathbf{\tilde{K}}\) be the SKI approximation of the kernel matrix \(\mathbf{K}\), and let \(\sigma^2\) be the regularization parameter. For a vector \(\mathbf{y}\), the error in the action of the regularized inverse can be bounded as follows:
\begin{align*}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y}\right\|_2 &\leq \frac{\gamma_{n, m, L}}{\sigma^4} \|\mathbf{y}\|_2
% \\
% &=\frac{O(n\sqrt{L}\rho^{-L})}{\sigma^4}
\end{align*}
\end{lemma}
\begin{proof}
Note that
\[
\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} = \left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} (\mathbf{K} - \mathbf{\tilde{K}}) \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}
\]
Taking the spectral norm, we have
\[
\begin{aligned}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 &\leq \left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \|\mathbf{K} - \mathbf{\tilde{K}}\|_2 \left\|\left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \\
&\leq \gamma_{n, m, L}\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2 \left\|\left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1}\right\|_2\textrm{ by Lemma \ref{proposition:spectral-norm}}\\
&\leq \frac{\gamma_{n, m, L}}{\sigma^4}
\end{aligned}
\]
Thus,
\[
\begin{aligned}
\left\|\left(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y} - \left(\mathbf{K} + \sigma^2 \mathbf{I}\right)^{-1} \mathbf{y}\right\|_2 &\leq \frac{\gamma_{n, m, L}}{\sigma^4} ||\mathbf {y}||_2
\end{aligned}
\]
The last inequality follows from the bounds on the inverse norms and the spectral norm error bound for \(K - \tilde{K}\).
\end{proof}
\subsubsection{Log Determinant}\label{subsubsec:log-determinant}
\begin{lemma}\label{lemma:log-determinant-error}
Let \(\mathbf{\tilde{K}}\) be the SKI approximation of the kernel matrix \(\mathbf{K}\), and let \(\sigma^2\) be the regularization parameter. The error in the log-determinant of the regularized SKI matrix can be bounded as follows:
\[
\left|\log \det(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - \log \det(\mathbf{K} + \sigma^2 \mathbf{I})\right| \leq \frac{\gamma_{n, m, L}}{\sigma^2}.
\]
\end{lemma}
\begin{proof}
We begin by applying the functional Mean Value Theorem for matrices, which implies that for some matrix \(Z = (\mathbf{\tilde{K}} + \alpha (\mathbf{K} - \mathbf{\tilde{K}})) + \sigma^2 \mathbf{I}\) with \(\alpha \in [0, 1]\),
\[
|f(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - f(\mathbf{K} + \sigma^2 \mathbf{I})| \leq \|f'(Z)\|_2 \|\mathbf{K} - \mathbf{\tilde{K}}\|_2
\]
where \(f(X) = \log |X|\) and thus \(f'(X) = X^{-\top}\).
Thus,
\[
|\log \det(\mathbf{\tilde{K}} + \sigma^2 \mathbf{I}) - \log \det(\mathbf{K} + \sigma^2 \mathbf{I})| \leq \left\| (\mathbf{\tilde{K}} + \alpha (\mathbf{K} - \mathbf{\tilde{K}}) + \sigma^2 I)^{-1} \right\|_2 ||\mathbf {K} - {\tilde {\mathbf {K}}}||_2\label{eqn:initial-log-det-bound}
\]
Using Claim 1, which bounds the eigenvalues of a convex combination of matrices, we have:
\[
\lambda_{\min}(Z) = (\lambda_{\min} (\mathbf{\tilde{K}})+ (1-\alpha)\lambda_{\min} (\mathbf {K}))+\sigma^2 > 0
\]
Therefore since $\mathbf{\tilde{K}}$ and $(\mathbf {K}$ are SPD and thus have positive eigenvalues,
\[
\left\| (\mathbf{\tilde{K}} + (1-\alpha)(\mathbf {K}- {\tilde {\mathbf {K}}})+ {\sigma^2 I})^{-1} )^{-1}\right\|_2 < {\frac {1}{ {\sigma^2}}}
\]
Substituting these bounds into Eqn. \ref{eqn:initial-log-det-bound} gives:
\[
|\log \det (\tilde{\mathbf {K}}+ {\sigma^2 I})- \log \det ({\mathbf {K}}+ {\sigma^2 I})| < {\frac {1}{ {\sigma^2}}} ||{\mathbf {K}- {\tilde {\mathbf {K}}}}||_2
\leq {\frac {1}{ {\sigma^2}}} {\gamma_{n,m,L}}
\]
\end{proof}
\section{Gaussian Processes Applications}\label{sec:gp-applications}
\subsection{Kernel Hyperparameter Estimation and Posterior Inference}
\subsubsection{Difference in Log-Likelihoods}
\begin{lemma}\label{lemma:difference-in-log-likelihoods}
Let \( \mathbf{K} \) be the true kernel matrix, \( \tilde{\mathbf{K}} \) be the SKI approximation of the kernel matrix using \( m \) inducing points and interpolation degree \( L-1 \), and \( \sigma^2 \) be the regularization parameter (noise variance). Let \( \mathbf{y} \) be the vector of observed target values, and \( n \) be the number of data points. Let \( \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) \) be the true log-likelihood and \( \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) \) be the SKI approximation of the log-likelihood, both for a mean-zero Gaussian Process prior. Then, the absolute difference between the true and SKI log-likelihoods is bounded as follows:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2 + \frac{\gamma_{n, m, L}}{\sigma^2} \right)
\end{equation}
where \( \gamma_{n, m, L} \equiv n(\delta_{m, L} + \sqrt{L} c^d \delta_{m, L}) \), \( \delta_{m, L} \) is an upper bound on the error of multivariate interpolation with \( m \) inducing points and interpolation degree \( L-1 \), and \( c > 0 \) is an upper bound on the absolute sum of the weights for 1D kernel interpolation.
Furthermore, if the convolutional cubic interpolation kernel (\( L=4 \)) is used with grid spacing \( h \), and the domain is contained in $[-D, D]^d$, then:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{n(2O(c^{2d}h^3))}{\sigma^4} \| \mathbf{y} \|^2 + \frac{n(2O(c^{2d}h^3))}{\sigma^2} \right)
\end{equation}
\end{lemma}
\begin{proof}
We begin by considering the log-likelihood of a mean-zero Gaussian Process, given by:
\begin{equation}
\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi)
\end{equation}
The corresponding SKI approximation is:
\begin{equation}
\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi)
\end{equation}
Our goal is to bound the absolute difference \( |\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})| \). Applying the triangle inequality, we have:
\begin{align}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | &= \left| \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} + \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| \right| \nonumber \\
&\leq \left| \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \right| + \left| \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right|
\end{align}
We now leverage Lemma \ref{lemma:action-inverse-error} and Lemma \ref{lemma:log-determinant-error} to bound the two terms on the right-hand side. Lemma \ref{lemma:action-inverse-error} provides a bound on the difference between the action of the true and approximated regularized inverses:
\begin{equation}
\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \|_2 \leq \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|_2
\end{equation}
This allows us to bound the first term:
\begin{equation}
\left| \frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \right| \leq \frac{1}{2} \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2
\end{equation}
Lemma \ref{lemma:log-determinant-error} bounds the difference in the log-determinants:
\begin{equation}
| \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \log|\mathbf{K} + \sigma^2 \mathbf{I}| | \leq \frac{\gamma_{n, m, L}}{\sigma^2}
\end{equation}
Thus, the second term is bounded by:
\begin{equation}
\left| \frac{1}{2} \log|\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{1}{2} \log|\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right| \leq \frac{1}{2} \frac{\gamma_{n, m, L}}{\sigma^2}
\end{equation}
Combining these bounds, we obtain the overall bound on the difference in log-likelihoods:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{\gamma_{n, m, L}}{\sigma^4} \| \mathbf{y} \|^2 + \frac{\gamma_{n, m, L}}{\sigma^2} \right)
\end{equation}
Now, let us consider the specific case of using a convolutional cubic interpolation kernel (\( L=4 \)). From Lemma \ref{lemma:tensor-product-interpolation-error}, we have \( \delta_{m, L} = O(c^{2d}h^3) \). Consequently, from Corollary \ref{cor:spectral-norm-error-alt}:
\begin{equation}
\gamma_{n, m, L} \equiv n(\delta_{m, L} + \sqrt{L} c^d \delta_{m, L}) = n(O(c^{2d}h^3) + 2c^d O(c^{2d}h^3)) = n(2O(c^{2d}h^3))
\end{equation}
Substituting this into our general bound, we obtain the specific bound for cubic interpolation:
\begin{equation}
| \mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) - \tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y}) | \leq \frac{1}{2} \left( \frac{n(2O(c^{2d}h^3))}{\sigma^4} \| \mathbf{y} \|^2 + \frac{n(2O(c^{2d}h^3))}{\sigma^2} \right)
\end{equation}
This concludes the proof.
\end{proof}
\subsubsection{Difference in MLEs}
We formalize the intuition that a good initialization for gradient ascent on the SKI log-likelihood, combined with a sufficiently accurate SKI approximation, leads to a SKI MLE that is close to the true MLE. We achieve this by first bounding the difference in gradients between the true and SKI log-likelihoods. Then, under the assumption of strong concavity of the true log-likelihood, we show that if the initialization is close to the true MLE, the resulting SKI MLE will also be close to the true MLE.
\begin{assumption}[Strong Concavity of the Population Log-Likelihood]\label{assumption:strong-concavity}
The population log-likelihood $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu > 0$ in a neighborhood $\mathcal{D}$ of the true hyperparameters $\theta^*$. That is, for all $\theta_1, \theta_2 \in \mathcal{D}$,
$$
\bar{\mathcal{L}}(\theta_2) \leq \bar{\mathcal{L}}(\theta_1) + \nabla \bar{\mathcal{L}}(\theta_1)^\top (\theta_2 - \theta_1) - \frac{\mu}{2} \| \theta_2 - \theta_1 \|_2^2.
$$
\end{assumption}
\begin{proposition}\label{prop:strong_concavity_hessian}
A function is strongly concave if and only if the Hessian matrix $H_f(\theta)$ satisfies $H_f(\theta) \preceq -\mu I$ for all $\theta \in \mathcal{D}$, where $\preceq$ denotes the negative semidefinite ordering.
\end{proposition}
\begin{proof}
See \cite{boyd2004convex}.
\end{proof}
\begin{assumption}[Local Identifiability of Hyperparameters]\label{assumption:local-identifiability}
Let $P(\cdot | \mathbf{X}; \theta)$ denote the Gaussian process predictive distribution at a set of input points $\mathbf{X}$ given hyperparameters $\theta$. There exists a neighborhood $\mathcal{D}$ of the true hyperparameters $\theta^*$ such that for any $\theta \in \mathcal{D}$ with $\theta \neq \theta^*$, there exists a finite set of input points $\mathbf{X} = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$, a set $A$ in the Borel sigma-algebra of $\mathbb{R}^n$, and a $\delta > 0$ such that:
$$
|P(\mathbf{y} \in A | \mathbf{X}; \theta) - P(\mathbf{y} \in A | \mathbf{X}; \theta^*)| > \delta
$$
and
$$
P^*_X(\mathbf{X}) > 0,
$$
where $\mathbf{y} = (y_1, \ldots, y_n)$ is the vector of corresponding output values, and $P^*_X$ is the marginal distribution of inputs under $P^*$.
\end{assumption}
\begin{lemma}[Bound on Derivative of SKI Kernel Error using Kernel Property of Derivative]
\label{lemma:ski_kernel_derivative_error_kernel}
Let $\mathcal{X} \subset \mathbb{R}^d$ be the input domain, and let $\Theta \subset \mathbb{R}^p$ be the domain of the kernel hyperparameters. Let $k_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be a kernel function parameterized by $\theta \in \Theta$, and let $\tilde{k}_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be its SKI approximation using $m$ inducing points and interpolation degree $L-1$. Assume that the partial derivative of $k_{\theta}$ with respect to a hyperparameter $\theta_i$, denoted as $k'_{\theta}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$, is also a valid SPD kernel. Let $\tilde{k}'_{\theta}(x,x')$ be the SKI approximation of $k'_{\theta}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$.
Then, for all $x, x' \in \mathcal{X}$, all $\theta \in \Theta$, the following inequality holds:
\begin{equation}
\left\vert \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}-\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i}\right\vert = \left\vert k'_{\theta}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \leq\delta_{m,L}'+\sqrt{L}c^d\delta_{m,L}',
\end{equation}
where $\delta_{m,L}'$ is an upper bound on the error of the SKI approximation of the kernel $k'_{\theta}(x,x')$ with $m$ inducing points and interpolation degree $L-1$, as defined in Lemma \ref{lemma:ski-kernel-elementwise-error}.
\end{lemma}
\begin{proof}
By assumption, $k'_{\theta}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$ is a valid SPD kernel. The SKI approximation of $k'_{\theta}(x, x')$ using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}(x, x')$ is given by $\tilde{k}'_{\theta}(x, x')$. For the kernel $k'_{\theta}(x, x')$, we have:
\begin{align*}
\left\vert k'_{\theta}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \leq \delta_{m,L}',
\end{align*}
where $\delta_{m,L}'$ is the upper bound on the error of the SKI approximation of $k'_{\theta}(x, x')$ as defined in Lemma \ref{lemma:ski-kernel-elementwise-error}.
Now, we need to show that $\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta}(x, x')$. Recall that the SKI approximation $\tilde{k}_{\theta}(x, x')$ is a linear combination of kernel evaluations at inducing points, with weights that depend on $x$ and $x'$:
\begin{align*}
\tilde{k}_{\theta}(x, x') = \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k_{\theta}(u_j, u_l)
\end{align*}
where $w_{jl}(x, x')$ are the interpolation weights. Taking the partial derivative with respect to $\theta_i$, we get:
\begin{align*}
\frac{\partial \tilde{k}_{\theta}(x, x')}{\partial \theta_i} &= \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') \frac{\partial k_{\theta}(u_j, u_l)}{\partial \theta_i} \\
&= \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k'_{\theta}(u_j, u_l).
\end{align*}
This is precisely the SKI approximation of the kernel $k'_{\theta}(x, x')$ using the same inducing points and weights:
\begin{align*}
\tilde{k}'_{\theta}(x, x') = \sum_{j=1}^m \sum_{l=1}^m w_{jl}(x, x') k'_{\theta}(u_j, u_l).
\end{align*}
Therefore, $\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta}(x, x')$.
Substituting this into our inequality, we get:
\begin{align*}
\left\vert \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}-\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i}\right\vert &= \left\vert k'_{\theta}(x, x') - \tilde{k}'_{\theta}(x, x') \right\vert \\
&\leq \delta_{m,L}'+\sqrt{L}c^d\delta_{m,L}'.
\end{align*}
\end{proof}
\begin{proposition}[Bound on Spectral Norm of Gradient Difference]
\label{prop:gradient_spectral_norm_bound}
Let $\mathcal{X} \subset \mathbb{R}^d$ be the input domain, and let $\Theta \subset \mathbb{R}^p$ be the domain of the kernel hyperparameters. Let $k_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be a kernel function parameterized by $\theta \in \Theta$, and let $\tilde{k}_{\theta}: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ be its SKI approximation using $m$ inducing points and interpolation degree $L-1$. Let $K \in \mathbb{R}^{n \times n}$ and $\tilde{K} \in \mathbb{R}^{n \times n}$ be the corresponding kernel matrices for a dataset of $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$.
Assume that for each $i \in \{1, ..., p\}$, the partial derivative of $k_{\theta}$ with respect to $\theta_i$, denoted as $k'_{\theta,i}(x, x') = \frac{\partial k_{\theta}(x, x')}{\partial \theta_i}$, is a valid SPD kernel. Let $\tilde{k}'_{\theta,i}(x,x')$ be the SKI approximation of $k'_{\theta,i}(x,x')$, using the same inducing points and interpolation scheme as $\tilde{k}_{\theta}$.
Then, the spectral norm of the difference between the gradient of the true kernel matrix and the gradient of the SKI kernel matrix is bounded by:
\begin{equation}
\| \nabla_\theta K - \nabla_\theta \tilde{K} \|_2 \leq \sqrt{p} \max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i},
\end{equation}
where $\nabla_\theta K$ and $\nabla_\theta \tilde{K}$ are represented as $p \times n^2$ matrices using the vec-notation (denominator layout), $p$ is the number of hyperparameters, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ (analogous to Proposition \ref{prop:spectral-norm}, but for the kernel $k'_{\theta,i}$).
\end{proposition}
\begin{proof}
We represent the gradients $\nabla_\theta K$ and $\nabla_\theta \tilde{K}$ as $p \times n^2$ matrices using the vec-notation (denominator layout):
\begin{align*}
\nabla_\theta K = \begin{bmatrix} \frac{\partial \text{vec}(K)}{\partial \theta_1} & \frac{\partial \text{vec}(K)}{\partial \theta_2} & \cdots & \frac{\partial \text{vec}(K)}{\partial \theta_p} \end{bmatrix}^T, \\
\nabla_\theta \tilde{K} = \begin{bmatrix} \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_1} & \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_2} & \cdots & \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_p} \end{bmatrix}^T.
\end{align*}
Let $K'_{\theta,i}$ be the kernel matrix corresponding to the kernel $k'_{\theta,i}(x,x') = \frac{\partial k_{\theta}(x,x')}{\partial \theta_i}$, and let $\tilde{K}'_{\theta,i}$ be the kernel matrix corresponding to its SKI approximation $\tilde{k}'_{\theta,i}(x,x')$.
From Lemma \ref{lemma:ski_kernel_derivative_error_kernel}, we have:
\begin{equation}
\frac{\partial \tilde{k}_{\theta}(x,x')}{\partial \theta_i} = \tilde{k}'_{\theta,i}(x, x')
\end{equation}
Thus, the $i$-th row of $\nabla_\theta K - \nabla_\theta \tilde{K}$ can be written as:
\begin{equation}
\frac{\partial \text{vec}(K)}{\partial \theta_i} - \frac{\partial \text{vec}(\tilde{K})}{\partial \theta_i} = \text{vec}(K'_{\theta,i}) - \text{vec}(\tilde{K}'_{\theta,i}).
\end{equation}
Therefore, we can write:
\begin{equation}
\nabla_\theta K - \nabla_\theta \tilde{K} = \begin{bmatrix}
\text{vec}(K'_{\theta,1}) - \text{vec}(\tilde{K}'_{\theta,1}) \\
\vdots \\
\text{vec}(K'_{\theta,p}) - \text{vec}(\tilde{K}'_{\theta,p})
\end{bmatrix}^T
\end{equation}
Then, we have that:
\begin{align}
(\nabla_\theta K - \nabla_\theta \tilde{K})^T = \begin{bmatrix}
\text{vec}(K'_{\theta,1}) - \text{vec}(\tilde{K}'_{\theta,1}) &
\cdots &
\text{vec}(K'_{\theta,p}) - \text{vec}(\tilde{K}'_{\theta,p})
\end{bmatrix}
\end{align}
Now we bound the spectral norm of $(\nabla_\theta K - \nabla_\theta \tilde{K})^T$:
\begin{align*}
\| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_2 &\leq \| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_F \\
&= \left( \sum_{i=1}^p \| \text{vec}(K'_{\theta,i}) - \text{vec}(\tilde{K}'_{\theta,i}) \|_2^2 \right)^{1/2} \\
&= \left( \sum_{i=1}^p \| K'_{\theta,i} - \tilde{K}'_{\theta,i} \|_F^2 \right)^{1/2} \\
&\leq \left( \sum_{i=1}^p \left( \max_{j \in \{1,\ldots,p\}} \| K'_{\theta,j} - \tilde{K}'_{\theta,j} \|_2 \right)^2 \right)^{1/2}
\end{align*}
By Proposition \ref{prop:spectral-norm}, we have a bound on the spectral norm difference between a kernel matrix and its SKI approximation. Let $\gamma'_{n,m,L,i}$ be the corresponding bound for the kernel $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$. Then:
\begin{equation}
\| K'_{\theta,i} - \tilde{K}'_{\theta,i} \|_2 \leq \gamma'_{n,m,L,i}
\end{equation}
for each $i \in \{1, \ldots, p\}$. Thus,
\begin{align*}
\| (\nabla_\theta K - \nabla_\theta \tilde{K})^T \|_2 &\leq \left( \sum_{i=1}^p (\gamma'_{n,m,L,i})^2 \right)^{1/2} \\
&\leq \left( \sum_{i=1}^p \left(\max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}\right)^2 \right)^{1/2} \\
&= \left( p \left(\max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}\right)^2 \right)^{1/2} \\
&= \sqrt{p} \max_{j \in \{1,\ldots,p\}} \gamma'_{n,m,L,j}
\end{align*}
This completes the proof.
\end{proof}
\begin{lemma}[Corrected Bound on Gradient Difference]\label{lemma:gradient-difference-bound-corrected}
Let $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ be the true log-likelihood and $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$ be the SKI approximation of the log-likelihood. Let $\nabla \mathcal{L}(\theta)$ and $\nabla \tilde{\mathcal{L}}(\theta)$ denote their respective gradients with respect to $\theta$. Then, for any $\theta$,
$$
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 \leq \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right],
$$
where $\gamma_{n,m,L}$ bounds the elementwise difference between $\mathbf{K}$ and $\tilde{\mathbf{K}}$, and $\gamma'_{n,m,L,i}$ is the bound on the spectral norm difference between the kernel matrices corresponding to $k'_{\theta,i}$ and its SKI approximation $\tilde{k}'_{\theta,i}$ and $C_1(\theta)$ are constants that depend on $\theta$ and the derivatives of the kernel function.
\end{lemma}
\begin{proof}
We start with the expressions for the gradients:
$$
\nabla \mathcal{L}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
$$
\nabla \tilde{\mathcal{L}}(\theta) = \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi) \right).
$$
Thus, the difference is:
\begin{align*}
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 &= \left\| \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\mathbf{K} + \sigma^2 \mathbf{I}| \right) \right. \\
&\quad \left. - \nabla \left( -\frac{1}{2} \mathbf{y}^\top (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&\leq \underbrace{\left\| \nabla \left( \frac{1}{2} \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2}_{T_1} \\
&\quad + \underbrace{\left\| \frac{1}{2} \nabla \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2}_{T_2}.
\end{align*}
We will bound $T_1$ and $T_2$ separately.
**Bounding $T_1$:**
Using the chain rule and the matrix derivative identity $\frac{\partial}{\partial \theta} \mathbf{X}^{-1} = -\mathbf{X}^{-1} (\frac{\partial \mathbf{X}}{\partial \theta}) \mathbf{X}^{-1}$, we have
\begin{align*}
T_1 &= \frac{1}{2} \left\| \nabla_\theta \left( \mathbf{y}^\top \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right) \mathbf{y} \right) \right\|_2 \\
&= \frac{1}{2} \left\| \sum_{i=1}^n \sum_{j=1}^n y_i y_j \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2 \\
&\leq \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n |y_i y_j| \left\| \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2.
\end{align*}
Now, applying the matrix derivative identity, we get:
\begin{align*}
&\left\| \nabla_\theta \left( (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}_{ij} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}_{ij} \right) \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} + \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\right.\\
&\quad \left.\left. + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \nabla_\theta \mathbf{K} (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&= \left\| \left( -(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right.\right.\\
&\quad \left.\left. - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \right)_{ij} \right\|_2 \\
&\leq \underbrace{\left\| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(a)} \\
&\quad + \underbrace{\left\| \left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right)(\nabla_\theta \mathbf{K}) (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_{2}}_{(b)} \\
&\quad + \underbrace{\left\|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}(\nabla_\theta \mathbf{K})\left((\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\right) \right\|_{2, }}_{(c)}\\
\end{align*}
We now explicitly bound (a), (b), and (c).
\begin{align*}
(a) &\leq \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2 \| \nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} \|_2 \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2\\
&\leq \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2^2 \| \nabla_\theta \tilde{\mathbf{K}} - \nabla_\theta \mathbf{K} \|_2 \\
&\leq \frac{1}{\sigma^4} \| \nabla_\theta \mathbf{K} - \nabla_\theta \tilde{\mathbf{K}} \|_2\text{ Proposition }\ref{prop:gradient_spectral_norm_bound}\\
&\leq \frac{1}{\sigma^4}\sqrt{p} \max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}.
\end{align*}
\begin{align*}
(b) &\leq \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \|\nabla_\theta \mathbf{K}\|_2\text{ proof of Lemma \ref{lemma:action-inverse-error}}.
\end{align*}
\begin{align*}
(c) &\leq \|(\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \\
&\leq \frac{1}{\sigma^2} \|(\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} - (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}\|_2 \|\nabla_\theta \mathbf{K}\|_2 \\
&\leq \frac{\gamma_{n,m,L}}{\sigma^4} \|\nabla_\theta \mathbf{K}\|_2\text{ proof of Lemma \ref{lemma:action-inverse-error}}.
\end{align*}
\amnote{We need some sort of assumption here, some kind of continuously differentiable}$\|\nabla_\theta \mathbf{K}\|$ can be bounded by a constant, say $C_1(\theta)$. Thus, we can bound $T_1$ as:
\begin{align*}
T_1 &\leq \sum_{i,j} \vert y_iy_j\vert \frac{1}{\sigma^4} \left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)\\
&\leq \frac{1}{\sigma^4}\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)\sum_{i,j} \vert y_iy_j\vert \\
&=\frac{\Vert y\Vert_1^2}{\sigma^4}\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)
\end{align*}
**Bounding $T_2$:**
Using the identity $\nabla \log |\mathbf{X}| = (\mathbf{X}^{-1})^\top$, we have
\begin{align*}
T_2 &= \frac{1}{2} \left\| \nabla_\theta \left( \log |\mathbf{K} + \sigma^2 \mathbf{I}| - \log |\tilde{\mathbf{K}} + \sigma^2 \mathbf{I}| \right) \right\|_2 \\
&= \frac{1}{2} \left\| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \right\|_2.
\end{align*}
We can rewrite the difference as:
$$
(\mathbf{K} + \sigma^2 \mathbf{I})^{-1} - (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} = (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} (\tilde{\mathbf{K}} - \mathbf{K}) (\mathbf{K} + \sigma^2 \mathbf{I})^{-1}
$$
Then, applying Lemma \ref{lemma:action-inverse-error} we obtain
\begin{align*}
T_2 &\leq \frac{1}{2} \| (\tilde{\mathbf{K}} + \sigma^2 \mathbf{I})^{-1} \|_2 \| \tilde{\mathbf{K}} - \mathbf{K} \|_2 \| (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \|_2 \\
&\leq \frac{\gamma_{n,m,L}}{2\sigma^4}
\end{align*}
where $C_2(\theta)$ is another constant depending on $\theta$ (from the derivatives of the kernel when applying Lemma \ref{lemma:action-inverse-error}).
**Combining the Bounds:**
Combining the bounds for $T_1$ and $T_2$, we have
$$
\| \nabla \mathcal{L}(\theta) - \nabla \tilde{\mathcal{L}}(\theta) \|_2 \leq \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C_1(\theta)\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right].
$$
\end{proof}
\begin{proposition}[One-Step Improvement on True Log-Likelihood]\label{prop:one-step-improvement-on-true-log-likelihood}
Let $\theta^*\in \mathcal{D}$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient in $\mathcal{D}$.
If the step size $\eta$ satisfies $\eta \leq \frac{1}{L}$ (where $L$ is the Lipschitz constant of the gradient of the true log-likelihood) and $\eta\mu < 1$,
then starting at $\theta_0\in \mathcal{D}$, after one step of projected gradient ascent on $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$, yielding $\theta_1$, we have:
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2
$$
\end{proposition}
\begin{proof}
By the optimality of $\theta^*$, we have $\nabla \mathcal{L}(\theta^*) = 0$.
By the $\mu$-strong concavity of $\mathcal{L}$, we have for any $\theta$:
$$
\mathcal{L}(\theta) \leq \mathcal{L}(\theta^*) + \nabla \mathcal{L}(\theta^*)^\top (\theta - \theta^*) - \frac{\mu}{2} \| \theta - \theta^* \|_2^2 = \mathcal{L}(\theta^*) - \frac{\mu}{2} \| \theta - \theta^* \|_2^2
$$
Rearranging, we get:
\begin{equation} \label{eq:strong_concavity_alt}
\| \theta - \theta^* \|_2^2 \leq \frac{2}{\mu} (\mathcal{L}(\theta^*) - \mathcal{L}(\theta))
\end{equation}
Since $\theta_1$ is obtained by one step of projected gradient ascent from $\theta_0$, we have:
$$
\theta_1 = \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \mathcal{L}(\theta_0))
$$
By the properties of projection onto a convex set, we have:
$$
\| \theta_1 - \theta^* \|_2^2 \leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2^2
$$
Expanding the right-hand side, we get:
\begin{align*}
\| \theta_1 - \theta^* \|_2^2 &\leq \| \theta_0 - \theta^* \|_2^2 + 2 \eta (\theta_0 - \theta^*)^\top \nabla \mathcal{L}(\theta_0) + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \\
&\leq \| \theta_0 - \theta^* \|_2^2 + 2 \eta (\mathcal{L}(\theta_0) - \mathcal{L}(\theta^*) + \frac{\mu}{2} \| \theta_0 - \theta^* \|_2^2) + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \\
&\text{(by strong concavity, with } \nabla \mathcal{L}(\theta^*) = 0 \text{)}\\
&\leq \| \theta_0 - \theta^* \|_2^2 - \eta \mu \| \theta_0 - \theta^* \|_2^2 + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2 \quad \text{(by \eqref{eq:strong_concavity_alt})} \\
&= (1 - \eta \mu) \| \theta_0 - \theta^* \|_2^2 + \eta^2 \| \nabla \mathcal{L}(\theta_0) \|_2^2
\end{align*}
By the Lipschitz continuity of the gradient, we have:
$$
\| \nabla \mathcal{L}(\theta_0) \|_2 = \| \nabla \mathcal{L}(\theta_0) - \nabla \mathcal{L}(\theta^*) \|_2 \leq L \| \theta_0 - \theta^* \|_2
$$
Thus,
$$
\| \theta_1 - \theta^* \|_2^2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2^2 + \eta^2 L^2 \| \theta_0 - \theta^* \|_2^2 = (1 - \eta \mu + \eta^2 L^2) \| \theta_0 - \theta^* \|_2^2
$$
Taking the square root of both sides:
$$
\| \theta_1 - \theta^* \|_2 \leq \sqrt{1 - \eta \mu + \eta^2 L^2} \| \theta_0 - \theta^* \|_2
$$
If we choose $\eta \leq \frac{1}{L}$, we have:
$$
\sqrt{1 - \eta \mu + \eta^2 L^2} \leq \sqrt{1-2\eta \mu+\eta^2\mu^2} = 1 - \eta \mu
$$
Thus,
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2
$$
as long as $\eta \leq \frac{1}{L}$.
\end{proof}
\begin{proposition}[MLE Closeness via One-Step Improvement]\label{prop:mle-closeness-one-step}
Let $\theta^*$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient near $\theta^*$. Assume that the gradient difference is bounded:
$$
\| \nabla \tilde{\mathcal{L}}(\theta) - \nabla \mathcal{L}(\theta) \|_2 \leq \epsilon_g:= \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\frac{1}{\sqrt{p}}\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right]
$$
for all $\theta\in \mathcal{D}$.
If the step size $\eta>0$ satisfies $\eta \leq \frac{1}{\tilde{L}}$ (where $L$ is the Lipschitz constant of the gradient of the log-likelihood) $\mu\eta\leq 1$:
$$
\| \theta_0 - \theta^* \|_2 \leq (1 - \eta \mu) \| \theta_0 - \theta^* \|_2+\eta \epsilon_g
$$
Then, after one step of projected gradient ascent on $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$, yielding $\theta_1$, we have:
$$
\| \theta_1 - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2 + \eta \epsilon_g
$$
\end{proposition}
\begin{proof}
Let $\theta_1$ be the result of one step of projected gradient ascent on $\tilde{\mathcal{L}}$ from $\theta_0$ with step size $\eta$:
$$
\theta_1 = \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0))
$$
where $\mathcal{D}$ is a convex set containing $\theta^*$.
By the triangle inequality, we have:
\begin{align*}
\| \theta_1 - \theta^* \|_2 &= \| \text{Proj}_{\mathcal{D}} (\theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0)) - \theta^* \|_2 \\
&\leq \| \theta_0 + \eta \nabla \tilde{\mathcal{L}}(\theta_0) - \theta^* \|_2 \\
&= \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* + \eta (\nabla \tilde{\mathcal{L}}(\theta_0) - \nabla \mathcal{L}(\theta_0)) \|_2 \\
&\leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 + \eta \| \nabla \tilde{\mathcal{L}}(\theta_0) - \nabla \mathcal{L}(\theta_0) \|_2 \\
&\leq \| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 + \eta \epsilon_g \quad \text{(by the gradient difference bound)}
\end{align*}
Now, let's consider the term $\| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2$. This represents the distance to $\theta^*$ after taking one step of gradient ascent on the true log-likelihood $\mathcal{L}$. By Proposition \ref{prop:one-step-improvement-on-true-log-likelihood}, if $\| \theta_0 - \theta^* \|_2 \leq \frac{\eta(2\mu - \eta L)}{2}$ and $\eta \leq \frac{1}{L}$, we have:
$$
\| \theta_0 + \eta \nabla \mathcal{L}(\theta_0) - \theta^* \|_2 \leq (1 - \mu \eta) \| \theta_0 - \theta^* \|_2
$$
Therefore,
\begin{align*}
\| \theta_1 - \theta^* \|_2 &\leq (1-\eta\mu)\|\theta_0-\theta^*\|_2+\eta\epsilon_g
\end{align*}
\end{proof}
\begin{theorem}[Closeness of SKI MLE to True MLE]\label{thm:mle-closeness-gradient-ascent}
Let $\theta^*$ be the true MLE maximizing $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$. Assume $\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})$ is $\mu$-strongly concave and has $L$-Lipschitz continuous gradient in $\mathcal{D}$. Assume that the gradient difference between the true log-likelihood and the SKI log-likelihood is bounded by
$$
\| \nabla \tilde{\mathcal{L}}(\theta) - \nabla \mathcal{L}(\theta) \|_2 \leq \epsilon_g := \frac{1}{\sigma^4}\left[\Vert y\Vert_1^2\left(\frac{1}{\sqrt{p}}\max_{i \in \{1,\ldots,p\}} \gamma'_{n,m,L,i}+2 C\gamma_{n,m,L}\right)+\frac{1}{2}\gamma_{n,m,L}\right]
$$
for all $\theta$ in a neighborhood of $\theta^*$, where $\gamma_{n,m,L}$, $\gamma'_{n,m,L,i}$ are defined as in Proposition \ref{prop:spectral-norm} and Proposition \ref{prop:gradient_spectral_norm_bound}, respectively, and $C$ is a bound on $C_1(\theta)$ in this neighborhood. Let $\tilde{\theta}$ be the SKI MLE obtained by running projected gradient ascent on $\tilde{\mathcal{L}}(\theta; \mathbf{X}, \mathbf{y})$ from initialization $\theta_0$ with a fixed step size $\eta$.
If the step size $\eta$ satisfies $0 < \eta \leq \frac{1}{L}$ and $\eta \mu < 1$, then, after $t$ steps of projected gradient ascent, the iterate $\theta_t$ satisfies
$$
\| \theta_t - \theta^* \|_2 \leq (1 - \eta\mu)^t \| \theta_0 - \theta^* \|_2 + \frac{\epsilon_g}{\eta\mu}.
$$
\end{theorem}
\begin{proof}
Consider that starting from $\theta_t$ and applying one step of gradient ascent, by Proposition \ref{prop:mle-closeness-one-step}
\begin{align*}
\Vert \theta_{t+1}-\theta^*\Vert_2 &\leq (1-\eta\mu)\Vert \theta_t-\theta^*\Vert_2+\eta \epsilon_g
\end{align*}
and iterating we have
\begin{align*}
\Vert \theta_t-\theta^*\Vert_2&\leq (1-\eta\mu)^t\Vert \theta_0-\theta^*\Vert_2+\sum_{s=0}^{t-1} (1-\eta\mu)^s \eta \epsilon_g\\
&\leq (1-\eta\mu)^t\Vert \theta_0-\theta^*\Vert_2+\frac{\eta\epsilon_G}{\eta\mu}\text{ geometric series}
\end{align*}
\end{proof}
\amnote{Now we would like to show that for some kernel (RBF probably), there exists a region around true parameter such that the log-likelihood is w.h.p. both strongly concave and has L-Lipschitz continuous gradients.}
\begin{theorem}[Local Strong Concavity of Population Log-Likelihood - RBF Kernel, Restricted Parameter Space]\label{thm:local-strong-concavity-restricted}
Let $P^*$ be the true data generating distribution, and let $\theta^* = (\sigma_{f*}, \ell_*)$ be the maximizer of the population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ for a Gaussian process with an RBF kernel:
$$
k_\theta(x, x') = \sigma_f^2 \exp\left(-\frac{\|x - x'\|_2^2}{2\ell^2}\right),
$$
where $\theta = (\sigma_f, \ell)$ are the signal variance and lengthscale parameters, respectively.
Assume the following:
1. **Restricted Parameter Space:** We restrict the parameter space to a compact set $\Theta$ such that for all $\theta \in \Theta$, we have $\sigma_f \in [\sigma_{f, min}, \sigma_{f, max}]$ and $\ell \in [\ell_{min}, \ell_{max}]$, where $0 < \sigma_{f, min} < \sigma_{f, max} < \infty$ and $0 < \ell_{min} < \ell_{max} < \infty$. Furthermore, $\theta^* \in \Theta$.
2. **Smoothness and Boundedness:**
* The true data generating process is of the form $y_i = \xi(x_i) + \epsilon_i$, where $\xi \sim GP(0, k_{\theta^*})$ and $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are i.i.d. noise.
* The marginal distribution of the inputs $P^*_X$ has a density $p(x)$ with respect to the Lebesgue measure on $\mathbb{R}^d$ that is bounded away from zero on a compact set $\mathcal{X} \subset \mathbb{R}^d$, i.e., there exists $\rho > 0$ such that $p(x) \geq \rho$ for all $x \in \mathcal{X}$.
* The function $\xi$ and its first and second partial derivatives are bounded on $\mathcal{X}$.
3. **Regularity of the RBF Kernel:** The RBF kernel $k_\theta(x,x')$ and its first, second, and third partial derivatives with respect to $\theta$ are continuous and bounded on $\mathcal{X} \times \mathcal{X}$ for $\theta \in \Theta$.
Then, there exists a neighborhood $\mathcal{D} \subseteq \Theta$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the following hold:
* **(a) Local Strong Concavity:** The population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu$ within $\mathcal{D}$, i.e., for all $\theta_1, \theta_2 \in \mathcal{D}$:
$$
\mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{y})] \leq \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})] + \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})]^\top (\theta_2 - \theta_1) - \frac{\mu}{2} \| \theta_2 - \theta_1 \|_2^2.
$$
* **(b) Lipschitz Gradient:** The gradient of the population log-likelihood $\nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is Lipschitz continuous with constant $L$ within $\mathcal{D}$, i.e., for all $\theta_1, \theta_2 \in \mathcal{D}$:
$$
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{Y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{Y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2.
$$
\end{theorem}
\begin{proof}
The proof follows a similar structure to the previous attempts, but crucially, we now work within a restricted parameter space.
**1. Define the Population Log-Likelihood:**
As before, let $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{Y})]$ denote the population log-likelihood.
**2. Consider the Hessian:**
Let $\mathbf{H}(\theta)$ be the Hessian of $\bar{\mathcal{L}}(\theta)$. We will show that $\mathbf{H}(\theta^*)$ is negative definite.
**3. Expected Fisher Information:**
The expected Fisher information matrix is $\mathbf{I}(\theta^*) = -\mathbf{H}(\theta^*)$. We will show that $\mathbf{I}(\theta^*)$ is positive definite.
**4. Positive Definiteness of $I(\theta^*)$:**
We want to show that for any non-zero vector $\mathbf{v} \in \mathbb{R}^2$, $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0$. Following the same steps as in the previous attempt, we arrive at:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
$$
where $\mathbf{A} = v_1 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \sigma_f^2} + v_2 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \ell}$.
As before, we can rewrite this as:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right],
$$
where $\mathbf{B} = (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2}$.
**5. Key Argument - Restricted Parameter Space and Linear Independence:**
Now, we leverage the restricted parameter space $\Theta$. Since $\sigma_f$ is bounded away from 0 and $\ell$ is bounded away from both 0 and $\infty$, the eigenvalues of $\mathbf{K}_{\theta^*}$ are bounded away from 0 and $\infty$. This implies that $\mathbf{B}$ and $\mathbf{B}^{-1}$ are well-defined and have bounded norms.
Suppose for contradiction that $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = 0$ for some non-zero $\mathbf{v}$. This implies that $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$ almost surely, and consequently, $\mathbf{A} = \mathbf{0}$ almost surely.
Thus, we have:
$$
v_1 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \sigma_f^2} + v_2 \frac{\partial \mathbf{K}_{\theta^*}}{\partial \ell} = \mathbf{0} \quad \text{a.s.}
$$
Now, we need to show that this is impossible for non-zero $\mathbf{v}$ under our assumptions. We can write this as:
$$
v_1 \frac{\partial k_{\theta^*}}{\partial \sigma_f^2}(x_i,x_j) + v_2 \frac{\partial k_{\theta^*}}{\partial \ell}(x_i,x_j) = 0
$$
for all $x_i,x_j$ in the support of $P_X^*$.
Using the derivatives of the RBF kernel that we computed earlier, this becomes:
$$
v_1 \exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right) + v_2 \sigma_{f*}^2 \exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right) \left(\frac{\|x_i - x_j\|_2^2}{\ell_*^3}\right) = 0
$$
for all $x_i, x_j$ in the support of $P_X^*$.
Dividing by $\exp\left(-\frac{\|x_i - x_j\|_2^2}{2\ell_*^2}\right)$ (which is always non-zero), we get:
$$
v_1 + v_2 \sigma_{f*}^2 \frac{\|x_i - x_j\|_2^2}{\ell_*^3} = 0
$$
Since $\sigma_{f*} > 0$ and $\ell_* > 0$, the only way for this to hold for all $x_i, x_j$ in a set of non-zero measure under $P^*_X$ is if $v_1 = v_2 = 0$.
This is because if $v_2 \neq 0$, then the term $\frac{\|x_i - x_j\|_2^2}{\ell_*^3}$ can be varied continuously by changing $x_i$ and $x_j$, which makes it impossible for the equation to hold unless $v_1$ is also zero.
Thus, we have reached a contradiction. Therefore, for any non-zero $\mathbf{v}$, we must have $\mathbf{A} \neq \mathbf{0}$ almost surely, and hence $\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0$.
**6. Negative Definite Hessian and Strong Concavity:**
Since $\mathbf{I}(\theta^*)$ is positive definite, $\mathbf{H}(\theta^*)$ is negative definite. By continuity of the Hessian (due to the regularity assumption on the kernel), there exists a neighborhood $\mathcal{D} \subseteq \Theta$ of $\theta^*$ such that $\mathbf{H}(\theta)$ is negative definite for all $\theta \in \mathcal{D}$. Moreover, there exists a constant $\mu > 0$ such that $-\mathbf{H}(\theta) \succeq \mu \mathbf{I}$ for all $\theta \in \mathcal{D}$. This implies that the population log-likelihood is strongly concave with parameter $\mu$ within $\mathcal{D}$.
**7. Lipschitz Continuity of the Gradient:**
The Lipschitz continuity of the gradient follows from the boundedness of the third-order derivatives of the population log-likelihood in the neighborhood $\mathcal{D}$. This can be shown using the regularity assumptions on the kernel (Assumption 3) and the boundedness assumption (Assumption 2). Since the third-order derivatives are bounded, there exists a constant $L < \infty$ such that:
$$
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{Y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{Y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2
$$
for all $\theta_1, \theta_2 \in \mathcal{D}$.
**Conclusion:**
We have shown that under the given assumptions, there exists a neighborhood $\mathcal{D}$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the population log-likelihood is strongly concave with parameter $\mu$ and has a Lipschitz continuous gradient with constant $L$ within $\mathcal{D}$. This completes the proof.
\end{proof}
\begin{theorem}[Local Strong Concavity of Population Log-Likelihood - General Case]\label{thm:local-strong-concavity-general}
Let $P^*$ be the true data generating distribution, and let $\theta^*$ be the maximizer of the population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ for a Gaussian process with kernel $k_\theta(x, x')$.
Assume the following:
1. **Identifiability:** The hyperparameters $\theta^*$ are identifiable from the data generating distribution $P^*$.
2. **Smoothness and Boundedness:**
* The true data generating process is of the form $y_i = \xi(x_i) + \epsilon_i$, where $\xi \sim GP(0, k_{\theta^*})$ and $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are i.i.d. noise.
* The marginal distribution of the inputs $P^*_X$ has a density $p(x)$ with respect to the Lebesgue measure on $\mathbb{R}^d$ that is bounded away from zero on a compact set $\mathcal{X} \subset \mathbb{R}^d$.
* The function $\xi$ and its first and second partial derivatives are bounded on $\mathcal{X}$.
3. **Regularity of the Kernel:** The kernel $k_\theta(x,x')$ and its first, second, and third partial derivatives with respect to $\theta$ are continuous and bounded on $\mathcal{X} \times \mathcal{X}$ for $\theta$ in a neighborhood of $\theta^*$.
4. **Linear Independence of Derivatives:** For any non-zero vector $\mathbf{v}$ of the same dimension as $\theta$, the following holds almost surely with respect to $P^*_X$:
$$
\sum_{i} v_i \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \neq \mathbf{0}
$$
where $\mathbf{K}_\theta$ is the kernel matrix evaluated at a set of input points drawn from $P^*_X$.
Then, there exists a neighborhood $\mathcal{D}$ of $\theta^*$ and constants $\mu > 0$ and $L < \infty$ such that the following hold:
* **(a) Local Strong Concavity:** The population log-likelihood $\mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is strongly concave with parameter $\mu$ within $\mathcal{D}$.
* **(b) Lipschitz Gradient:** The gradient of the population log-likelihood $\nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ is Lipschitz continuous with constant $L$ within $\mathcal{D}$.
\end{theorem}
\begin{proof}
Let $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$ denote the population log-likelihood. The Hessian matrix $\mathbf{H}(\theta)$ has entries given by:
$$
\mathbf{H}_{ij}(\theta) = \frac{\partial^2 \bar{\mathcal{L}}(\theta)}{\partial \theta_i \partial \theta_j}.
$$
We want to show that $\mathbf{H}(\theta^*)$ is negative definite. It is equivalent to showing that the negative Hessian, $-\mathbf{H}(\theta^*)$, which is also known as the *expected Fisher information matrix*, is positive definite.
The log-likelihood for a set of observations $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$ (assuming a zero mean function) is:
$$
\mathcal{L}(\theta; \mathbf{X}, \mathbf{y}) = -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \log|\mathbf{K}_\theta + \sigma^2 \mathbf{I}| - \frac{n}{2} \log(2\pi),
$$
where $\mathbf{K}_\theta$ is the kernel matrix evaluated at the points in $\mathbf{X}$, and $\sigma^2$ is the noise variance.
\textbf{1. First-Order Derivatives:**}
The gradient of the log-likelihood has components:
$$
\frac{\partial \mathcal{L}}{\partial \theta_i} = \frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right).
$$
\textbf{2. Second-Order Derivatives:**}
The second-order derivatives can be computed using matrix calculus identities. After some calculations (details in Appendix A), we arrive at the following expression:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left[ \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \\
&\quad + \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) \right] - \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) \right].
\end{align*}
\textbf{3. Population Log-Likelihood and Hessian:**}
The population log-likelihood is $\bar{\mathcal{L}}(\theta) = \mathbb{E}_{P^*}[\mathcal{L}(\theta; \mathbf{X}, \mathbf{y})]$. The Hessian of the population log-likelihood is:
$$
\mathbf{H}(\theta) = \mathbb{E}_{P^*}[\nabla^2_\theta \mathcal{L}(\theta; \mathbf{X}, \mathbf{y})],
$$
where $\nabla^2_\theta$ denotes the Hessian with respect to $\theta$.
\textbf{4. Expected Fisher Information:**}
The expected Fisher information matrix is $\mathbf{I}(\theta^*) = -\mathbf{H}(\theta^*)$. We want to show that $\mathbf{I}(\theta^*)$ is positive definite.
\textbf{5. Positive Definiteness:**}
To show that $\mathbf{I}(\theta^*)$ is positive definite, we need to show that for any non-zero vector $\mathbf{v} \in \mathbb{R}^2$:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} > 0.
$$
Substituting $\theta = \theta^*$ and taking the expectation with respect to $P^*$, we get:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = -\mathbb{E}_{P^*}[\mathbf{v}^\top \nabla^2_\theta \mathcal{L}(\theta^*; \mathbf{X}, \mathbf{y}) \mathbf{v}].
$$
Since $\mathbf{y}|\mathbf{X} \sim \mathcal{N}(\mathbf{0}, \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})$, we have $\mathbb{E}_{P^*}[\mathbf{y}\mathbf{y}^\top | \mathbf{X}] = \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I}$. Using this, we can simplify the expectation of the quadratic form involving $\mathbf{y}$ (details in Appendix B). After algebraic manipulation, we obtain:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
$$
where $\mathbf{A} = \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}$.
\textbf{6. Using Properties of Trace and Positive Semidefiniteness:**}
Now, note that $(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}$ is positive definite and thus has a positive definite square root. Let $\mathbf{B} = (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2}$. Then, we can rewrite the expression as:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( \mathbf{B} \mathbf{A} \mathbf{B} \mathbf{B} \mathbf{A} \mathbf{B} \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B}) (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right] = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right].
$$
Since $(\mathbf{B}\mathbf{A}\mathbf{B})^\top (\mathbf{B}\mathbf{A}\mathbf{B})$ is positive semidefinite, its trace is non-negative. Moreover, the trace is zero if and only if $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$.
\textbf{7. Using Identifiability and Smoothness:**}
Here, we use the identifiability assumption. If $\mathbf{B}\mathbf{A}\mathbf{B} = \mathbf{0}$, then $(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1/2} = \mathbf{0}$, which implies $\mathbf{A} = \mathbf{0}$ (since the inverse square root is invertible).
$\mathbf{A} = \mathbf{0}$ implies:
$$
\sum_{i=1}^p v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}=\mathbf{0}
$$
This means that a linear combination of the derivatives of the kernel matrix with respect to the hyperparameters is zero. However, under the identifiability assumption and the smoothness of the data distribution, this can only happen if $v_1 = v_2 = 0$ (i.e., $\mathbf{v}$ is the zero vector). If different hyperparameters always lead to different predictive distributions, then their derivatives must be linearly independent.
Therefore, for any non-zero $\mathbf{v}$, we have $\mathbf{B}\mathbf{A}\mathbf{B} \neq \mathbf{0}$, and thus:
$$
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} = \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{B} \mathbf{A} \mathbf{B})^\top (\mathbf{B} \mathbf{A} \mathbf{B}) \right) \right] > 0.
$$
This shows that $\mathbf{I}(\theta^*)$ is positive definite, and consequently, $\mathbf{H}(\theta^*)$ is negative definite.
\textbf{Step 8: Establish Lipschitz continuity of the gradient.}
The Lipschitz continuity of the gradient follows from the boundedness of the third-order derivatives of the population log-likelihood in the neighborhood $\mathcal{D}$. This can be shown using the regularity assumptions on the kernel (Assumption 3) and the boundedness assumption (Assumption 2). Since the third-order derivatives are bounded, there exists a constant $L < \infty$ such that
\[
\| \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_1; \mathbf{X}, \mathbf{y})] - \nabla_\theta \mathbb{E}_{P^*}[\mathcal{L}(\theta_2; \mathbf{X}, \mathbf{y})] \|_2 \leq L \| \theta_1 - \theta_2 \|_2
\]
for all $\theta_1, \theta_2 \in \mathcal{D}$.
\end{proof}
\subsection{Error of GP Predictive Posterior Parameters}
\subsubsection{Mean}
\amnote{Wrong now that it's a set of test points instead of a single test point.}
\begin{lemma}
Let $\boldsymbol{\mu}(\cdot)$ be the GP posterior mean at a set of test points $\cdot$ and $\tilde{\boldsymbol{\mu}}(\cdot)$ be the SKI posterior mean at those points. Then the SKI posterior mean $l^2$ error is bounded by:
\begin{align*}
\Vert \tilde{\boldsymbol{\mu}}(\cdot)- \boldsymbol{\mu}(\cdot)\Vert_2&\leq \frac{1}{\sigma^2}\left(\min(\gamma_{T,m,L},\gamma_{n,m,L})+\frac{M}{\sigma^2}\gamma_{n,m,L}\right)
\end{align*}
\end{lemma}
\begin{proof}
We start by expressing the difference between the true and SKI posterior means:
\begin{align*}
& \left\|\mathbf{K}_{\cdot,\mathbf{X}} \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} \right\|_{2} \\
&= \left\| \left(\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \right) \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \mathbf{y} + \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left[ \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right] \mathbf{y} \right\|_{2}
\end{align*}
Applying the triangle inequality gives:
\begin{align*}
& \leq \left\| \left(\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \right) \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \| \mathbf{y} \|_{2} + \left\| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \left( \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right) \mathbf{y} \right\|_{2} \\
&\leq \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} \left\| \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \| \mathbf{y} \|_{2} + \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} \left\| \left( \mathbf{K} + \sigma^{2} \mathbf{I} \right)^{-1} - \left( \tilde{\mathbf{K}} + \sigma^{2} \mathbf{I} \right)^{-1} \right\|_{2} \cdot \| \mathbf{y} \|_{2} \\
& \leq \frac{ \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{2} } \| \mathbf{y} \|_{2} + \gamma_{n, m, L}\frac{ \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{4} } \| \mathbf{y} \|_{2} \textrm{ by Lemma \ref{lemma:action-inverse-error}}\\
& \leq \left( \frac{ \|\mathbf{K}_{\cdot,\mathbf{X}} - \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{2} } + \frac{ \| \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \|_{2} }{ \sigma^{4} } \gamma_{n, m, L} \right) \| \mathbf{y} \|_{2}\\
&\leq \left(\frac{\max(\gamma_{T,m,L},\gamma_{n,m,L})}{\sigma^2}+\frac{\Vert \tilde{\mathbf{K}}_{\cdot, \mathbf{X}} \Vert_2}{\sigma^4}\gamma_{n,m,L}\right)\Vert \textbf{y}\Vert_2
\end{align*}
\end{proof}
\begin{lemma}[Bound on the Error of SKI Posterior Covariance]
Let $\boldsymbol{\Sigma}(\cdot)$ be the GP posterior covariance matrix at a set of test points $\cdot\in \mathbb{R}^{T\times d}$ and $\tilde{\boldsymbol{\Sigma}}(\cdot)$ be its SKI approximation. Assume that the true kernel satisfies the condition that $|k(\mathbf{x}, \mathbf{x}')| \leq M$ for all $\mathbf{x}, \mathbf{x}'$ in the domain of interest, so that $\Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}M$. Then
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \gamma_{T,m,L} + \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L})\\
&+ \frac{\gamma_{n,m,L}}{\sigma^4}Tn c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}),
\end{align*}
where $\gamma_{T,m,L}$ and $\gamma_{n,m,L}$ are defined as in Proposition \ref{prop:spectral-norm}, $c$ is an upper bound on the absolute sum of the weights for 1D kernel interpolation, $m$ is the number of inducing points, and $\sigma^2$ is the observation noise variance.
\end{lemma}
\begin{proof}
First, note that
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \Vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\Vert_2\\
&+ \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \gamma_{T,m,L} + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2,
\end{align*}
where we used Proposition \ref{prop:spectral-norm} and the fact that $\Vert \textbf{K}_{\cdot,\cdot}-\tilde{\textbf{K}}_{\cdot,\cdot}\Vert_2 \leq \gamma_{T,m,L}$.
Now, we bound the second term:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&+ \Vert\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}(\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot})\Vert_2 + \Vert (\textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1})\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert (\textbf{K}+\sigma^2 I)^{-1} \Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2\\
&\leq \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2,
\end{align*}
where we used the fact that $(\textbf{K}+\sigma^2 I)^{-1} \preceq \frac{1}{\sigma^2}I$.
Next, we bound the term $\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2$:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \\
&= \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1} - \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} + \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 \\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1} - \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&= \Vert \textbf{K}_{\cdot,\textbf{X}} [(\textbf{K}+\sigma^2 I)^{-1} - (\tilde{\textbf{K}}+\sigma^2 I)^{-1}] \Vert_2 + \Vert (\textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}}) (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert (\textbf{K}+\sigma^2 I)^{-1} - (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2 + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \Vert (\tilde{\textbf{K}}+\sigma^2 I)^{-1} \Vert_2\\
&\leq \Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \frac{\gamma_{n,m,L}}{\sigma^4} + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \frac{1}{\sigma^2},
\end{align*}
where we used Lemma \ref{lemma:action-inverse-error} in the last inequality.
Substituting this back into the main inequality, we get:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \left(\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \frac{\gamma_{n,m,L}}{\sigma^4} + \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \frac{1}{\sigma^2}\right) \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2\\
&= \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}}\Vert_2 \Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 + \frac{\gamma_{n,m,L}}{\sigma^4}\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 + \frac{1}{\sigma^2} \Vert \textbf{K}_{\cdot,\textbf{X}} - \tilde{\textbf{K}}_{\cdot,\textbf{X}} \Vert_2 \Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2.
\end{align*}
Using Lemma \ref{lemma:test-train-kernel-matrix-error} and the fact that $\Vert \textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \leq \max(\gamma_{T,m,L},\gamma_{n,m,L})$ and that $\textbf{K}_{\cdot, \textbf{X}} = \textbf{K}_{\textbf{X},\cdot}^\top$, we have $\Vert \textbf{K}_{\cdot,\textbf{X}} \Vert_2 = \Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2$. Also, by assumption, $\Vert \textbf{K}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}M$. Using Lemma \ref{lemma:ski-test-train-bound}, we have $\Vert \tilde{\textbf{K}}_{\textbf{X},\cdot} \Vert_2 \leq \sqrt{Tn}mc^{2d}M$. Substituting these bounds, we get:
\begin{align*}
&\Vert \textbf{K}_{\cdot,\textbf{X}} (\textbf{K}+\sigma^2 I)^{-1}\textbf{K}_{\textbf{X},\cdot}-\tilde{\textbf{K}}_{\cdot,\textbf{X}} (\tilde{\textbf{K}}+\sigma^2 I)^{-1}\tilde{\textbf{K}}_{\textbf{X},\cdot}\Vert_2 \\
&\leq \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) + \frac{\gamma_{n,m,L}}{\sigma^4}(\sqrt{Tn}M)(\sqrt{Tn} m c^{2d} M) + \frac{1}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) (\sqrt{Tn} m c^{2d} M) \\
&= \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}) + \frac{\gamma_{n,m,L}}{\sigma^4}Tn m c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}).
\end{align*}
Finally, substituting this back into the original inequality, we obtain the desired bound:
\begin{align*}
\Vert \boldsymbol{\Sigma}(\cdot)-\tilde{\boldsymbol{\Sigma}}(\cdot)\Vert_2 &\leq \gamma_{T,m,L} + \frac{\sqrt{Tn}M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L})\\
&+ \frac{\gamma_{n,m,L}}{\sigma^4}Tn m c^{2d} M^2 + \frac{\sqrt{Tn} m c^{2d} M}{\sigma^2} \max(\gamma_{T,m,L},\gamma_{n,m,L}).
\end{align*}
This completes the proof.
\end{proof}
\section{Discussion}\label{sec:Discussion}
\bibliography{main}
\bibliographystyle{icml2021}
\appendix
\section{Auxiliary Technical Results}
\begin{lemma}\label{lemma:switch-sum-product}
Given a function $f: \mathbb{R}^d \rightarrow \mathbb{R}$ of the form $f(x_1, x_2, ..., x_d) = \prod_{j=1}^d f_j(x_j)$, where each $f_j: \mathbb{R} \rightarrow \mathbb{R}$. Let $G = G^{(1)} \times G^{(2)} \times ... \times G^{(d)}$ be a fixed d-dimensional grid, where each $G^{(j)} = \{p_1^{(j)}, p_2^{(j)}, ..., p_{n_j}^{(j)}\}$ is a finite set of $n_j$ grid points along the j-th dimension for $j = 1, 2, ..., d$. Then the following equality holds:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d f_j(p_{k_j}^{(j)}) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\end{lemma}
\begin{proof}
\textbf{By Induction on d (the number of dimensions):}
\textbf{Base Case (d = 1):}
When $d=1$, the statement becomes:
$$
\sum_{k_1=1}^{n_1} f_1(p_{k_1}^{(1)}) = \sum_{k_1=1}^{n_1} f_1(p_{k_1}^{(1)})
$$
This is trivially true.
\textbf{Inductive Hypothesis:}
Assume the statement holds for $d = m$, i.e.,
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \prod_{j=1}^m f_j(p_{k_j}^{(j)}) = \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\textbf{Inductive Step:}
We need to show that the statement holds for $d = m+1$. Consider the left-hand side for $d = m+1$:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_{m+1}=1}^{n_{m+1}} \prod_{j=1}^{m+1} f_j(p_{k_j}^{(j)})
$$
We can rewrite this as:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \left( \sum_{k_{m+1}=1}^{n_{m+1}} \left( \prod_{j=1}^m f_j(p_{k_j}^{(j)}) \right) f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right)
$$
Notice that the inner sum (over $k_{m+1}$) does not depend on $k_1, k_2, ..., k_m$. Thus, for any fixed values of $k_1, k_2, ..., k_m$, we can treat $\prod_{j=1}^m f_j(p_{k_j}^{(j)})$ as a constant. Let $C(k_1, ..., k_m) = \prod_{j=1}^m f_j(p_{k_j}^{(j)})$. Then we have:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \left( C(k_1, ..., k_m) \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right)
$$
Now, the inner sum $\sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)})$ is a constant with respect to $k_1, ..., k_m$. Let's call this constant $S_{m+1}$. So we have:
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} C(k_1, ..., k_m) S_{m+1} = S_{m+1} \sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_m=1}^{n_m} \prod_{j=1}^m f_j(p_{k_j}^{(j)})
$$
By the inductive hypothesis, we can replace the nested sums with a product:
$$
S_{m+1} \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right) = \left( \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right) \prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
Rearranging the terms, we get:
$$
\prod_{j=1}^m \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right) \left( \sum_{k_{m+1}=1}^{n_{m+1}} f_{m+1}(p_{k_{m+1}}^{(m+1)}) \right) = \prod_{j=1}^{m+1} \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
This is the right-hand side of the statement for $d = m+1$. Thus, the statement holds for $d = m+1$.
\textbf{Conclusion:}
By induction, the statement holds for all $d \geq 1$. Therefore,
$$
\sum_{k_1=1}^{n_1} \sum_{k_2=1}^{n_2} ... \sum_{k_d=1}^{n_d} \prod_{j=1}^d f_j(p_{k_j}^{(j)}) = \prod_{j=1}^d \left( \sum_{k_j=1}^{n_j} f_j(p_{k_j}^{(j)}) \right)
$$
\end{proof}
\begin{claim}\label{claim:convex-combo-eigenvalues}
Given a convex combination \(\mathbf{C} = \alpha \mathbf{A} + (1-\alpha) \mathbf{B}\), where \(\alpha \in [0,1]\), and \(\mathbf{A}\) and \(\mathbf{B}\) are symmetric matrices, the eigenvalues of \(\mathbf{C}\) lie in the interval \(\left[\min \left(\lambda_n(\mathbf{A}), \lambda_n(\mathbf{B})\right), \max \left(\lambda_1(\mathbf{A}), \lambda_1(\mathbf{B})\right)\right]\).
% Given a convex combination $C=\alpha A+(1-\alpha)B,\alpha \in [0,1]$, the eigenvalues of $C$ lie in the interval $[\min(\lambda(A)_n, \lambda(B)_n), \max(\lambda(A)_1,\lambda(B)_1)]$
\end{claim}
\begin{proof}
First, recall that for a symmetric matrix \(\mathbf{A}\), the Rayleigh quotient \(R(\mathbf{A}, \mathbf{x}) = \frac{\mathbf{x}^{\top} \mathbf{A} \mathbf{x}}{\mathbf{x}^{\top} \mathbf{x}}\) is bounded by the smallest and largest eigenvalues of \(\mathbf{A}\):
\[
\lambda_n(\mathbf{A}) \leq R(\mathbf{A}, \mathbf{x}) \leq \lambda_1(\mathbf{A})
\]
Consider the Rayleigh quotient for the matrix \(\mathbf{C}\):
\[
R(\mathbf{C}, \mathbf{x}) = \frac{\mathbf{x}^{\top} (\alpha \mathbf{A} + (1-\alpha) \mathbf{B}) \mathbf{x}}{\mathbf{x}^{\top} \mathbf{x}} = \alpha R(\mathbf{A}, \mathbf{x}) + (1-\alpha) R(\mathbf{B}, \mathbf{x})
\]
Since \(R(\mathbf{A}, \mathbf{x})\) and \(R(\mathbf{B}, \mathbf{x})\) are bounded by their respective eigenvalues, we have:
\[
R(\mathbf{C}, \mathbf{x}) \leq \alpha \lambda_1(\mathbf{A}) + (1-\alpha) \lambda_1(\mathbf{B})
\]
which implies:
\[
R(\mathbf{C}, x) \leq \max(\lambda_1(\textbf{A}), \lambda_1(\textbf{B}))
\]
Similarly,
\[
R(\textbf{C}, \textbf{x}) \geq \min(\lambda_n(\textbf{A}), \lambda_n(\textbf{B}))
\]
Thus, the eigenvalues of \(\textbf{C} = \alpha \textbf{A} + (1-\alpha)\textbf{B}\) are bounded by:
\[
\min(\lambda_n(\textbf{A}), \lambda_n(\textbf{B})) \leq \lambda(\textbf{C}) \leq \max(\lambda_1(\textbf{A}), \lambda_1(\textbf{B}))
\]
\end{proof}
\section{Proofs Related to Estimation}
\subsection{Second-Order Derivatives of Log-Likelihood (Details)}
Here, we provide the detailed calculations for the second-order derivatives of the log-likelihood with respect to $\theta_i$ and $\theta_j$.
We start with the first-order derivative:
$$
\frac{\partial \mathcal{L}}{\partial \theta_i} = \frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} - \frac{1}{2} \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right).
$$
Now, we differentiate again with respect to $\theta_j$. We will use the following matrix calculus identities:
* $\frac{\partial}{\partial \theta} (\mathbf{A}^{-1}) = -\mathbf{A}^{-1} (\frac{\partial \mathbf{A}}{\partial \theta}) \mathbf{A}^{-1}$
* $\frac{\partial}{\partial \theta} \text{tr}(\mathbf{A}\mathbf{B}) = \text{tr}(\frac{\partial \mathbf{A}}{\partial \theta} \mathbf{B}) + \text{tr}(\mathbf{A} \frac{\partial \mathbf{B}}{\partial \theta})$
* $\frac{\partial}{\partial \theta} \log|\mathbf{A}| = \text{tr}(\mathbf{A}^{-1} \frac{\partial \mathbf{A}}{\partial \theta})$
Applying these identities, we obtain:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= \frac{1}{2} \mathbf{y}^\top \frac{\partial}{\partial \theta_j} \left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \right] \mathbf{y} \\
&\quad - \frac{1}{2} \frac{\partial}{\partial \theta_j} \left[ \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right]
\end{align*}
Let's compute each term separately. For the first term:
\begin{align*}
&\frac{\partial}{\partial \theta_j} \left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \right] \\
&= \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial}{\partial \theta_j} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \\
&= -(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad + (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \\
&\quad - (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}
\end{align*}
For the second term:
\begin{align*}
&\frac{\partial}{\partial \theta_j} \left[ \text{tr}\left((\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] \\
&= \text{tr}\left( \frac{\partial}{\partial \theta_j} [(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}] \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \right) + \text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial}{\partial \theta_j} \left[\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right] \right) \\
&= -\text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial \mathbf{K}_\theta}{\partial \theta_i} \right) + \text{tr}\left( (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} \right)
\end{align*}
Putting everything together, we get the expression for the second-order derivative:
\begin{align*}
\frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j} &= -\frac{1}{2} \mathbf{y}^\top (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \left[ \frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right) \right] (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1} \mathbf{y} \\
&\quad + \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_i}\right)(\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_\theta}{\partial \theta_j}\right) \right] - \frac{1}{2} \text{tr}\left[ (\mathbf{K}_\theta + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial^2 \mathbf{K}_\theta}{\partial \theta_i \partial \theta_j}\right) \right].
\end{align*}
\subsection{Appendix B: Simplification of the Quadratic Form (Details)}
Here, we provide the detailed algebraic manipulations to arrive at the expression involving the trace. We start with the quadratic form:
Substituting the expression for the second-order derivative and evaluating at $\theta=\theta^*$, we get:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v}
&= -\mathbb{E}_{P^*} \left[ \sum_{i=1}^2 \sum_{j=1}^2 v_i v_j \frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j}\bigg|_{\theta = \theta^*} \right].
\end{align*}
Now, we substitute the expression for the second-order derivatives and use the fact that $\mathbb{E}_{P^*}[\mathbf{y}\mathbf{y}^\top | \mathbf{X}] = \mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I}$:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} &= \frac{1}{2} \mathbb{E}_{P^*_X} \left[ \sum_{i=1}^2 \sum_{j=1}^2 v_i v_j \left( \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right) (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right) \right] - \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \frac{\partial^2 \mathbf{K}_{\theta^*}}{\partial \theta_i \partial \theta_j} \right] \right. \right. \\
&\quad \left. \left. + \text{tr}\left[ (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left\{ \frac{\partial^2 \mathbf{K}_{\theta^*}}{\partial \theta_i \partial \theta_j} - \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right)(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right) - \left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j}\right)(\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1}\left(\frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}\right) \right\} \right] \right) \right].
\end{align*}
After simplifying and rearranging terms, we get:
\begin{align*}
\mathbf{v}^\top \mathbf{I}(\theta^*) \mathbf{v} &= \frac{1}{2} \mathbb{E}_{P^*_X} \left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left( \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i} \right) (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \left( \sum_{j=1}^2 v_j \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_j} \right) \right) \right] \\
&= \frac{1}{2} \mathbb{E}_{P^*_X}\left[ \text{tr}\left( (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} (\mathbf{K}_{\theta^*} + \sigma^2 \mathbf{I})^{-1} \mathbf{A} \right) \right],
\end{align*}
where $\mathbf{A} = \sum_{i=1}^2 v_i \frac{\partial \mathbf{K}_{\theta^*}}{\partial \theta_i}$. This is the expression used in the proof.
%%%%%%%% ICML 2025 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
\newcommand{\jmei}[1]{\textcolor{magenta}{#1}}
\newcommand{\amnote}[1]{\textcolor{red}{#1}}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2025} with \usepackage[nohyperref]{icml2025} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
% Use the following line for the initial blind version submitted for review:
%\usepackage{icml2025}
% If accepted, instead use the following line for the camera-ready submission:
\usepackage[accepted]{icml2025}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{thmtools}
\usepackage{thm-restate}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{claim}{Claim}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Error Bounds for Structured Kernel Interpolation}
\begin{document}
\twocolumn[
\icmltitle{The Price of Linear Time: Error Analysis of Structured Kernel Interpolation}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2025
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Alexander Moreno}{yyy}
\icmlauthor{Justin Xiao}{yyy}
\icmlauthor{Jonathan Mei}{yyy}
%\icmlauthor{}{sch}
%\icmlauthor{}{sch}
%\icmlauthor{}{sch}
\end{icmlauthorlist}
\icmlaffiliation{yyy}{Independent Researcher}
\icmlcorrespondingauthor{Alexander Moreno}{alexander.f.moreno@gmail.com}
%\icmlcorrespondingauthor{Firstname2 Lastname2}{first2.last2@www.uk}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in
]
\printAffiliationsAndNotice{\phantom{}}
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
%\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
%\printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
\begin{abstract}
Structured Kernel Interpolation (SKI) \cite{wilson2015kernel} helps scale Gaussian Processes (GPs) by approximating the kernel matrix via interpolation at inducing points, achieving linear computational complexity. However, it lacks rigorous theoretical error analysis. This paper bridges the gap: we prove error bounds for the SKI Gram matrix and examine the error's effect on hyperparameter estimation and posterior inference. We further provide a practical guide to selecting the number of inducing points under convolutional cubic interpolation: they should grow as $n^{d/3}$ for error control. Crucially, we identify two dimensionality regimes governing the trade-off between SKI Gram matrix spectral norm error and computational complexity. For $d \leq 3$, \textit{any} error tolerance can achieve linear time for sufficiently large sample size. For $d > 3$, the error must \textit{increase} with sample size to maintain linear time. Our analysis provides key insights into SKI's scalability-accuracy trade-offs, establishing precise conditions for achieving linear-time GP inference with controlled approximation error.
\end{abstract}
\section{Introduction}\label{sec:introduction}
\input{body/01intro}
\section{Related Work}\label{sec:related}
\input{body/02related}
\section{Gaussian Processes, Structured Kernel Interpolation and Convolutional Cubic Interpolation}\label{sec:ski-background}
\input{body/03background}
\section{Important Quantities}\label{sec:important-quantities}
\input{body/04important-quantities}
\section{Gaussian Processes Applications}\label{sec:gp-applications}
\input{body/05gp-applications}
\section{Discussion}\label{sec:discussion}
\input{body/06discussion}
\section{Broader Impact}\label{sec:broader-impact}
\input{body/07broader-impact}
\bibliography{example_paper}
\bibliographystyle{icml2025}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\onecolumn
\section{Auxiliary Technical Results}
\input{body/appendices/01auxiliary}
\section{Proofs Related to Important Quantities}
\input{body/appendices/04important-quantities}
\section{Proofs Related to Gaussian Process Applications}
\input{body/appendices/05gp-applications}
\end{document}
% This document was modified from the file originally made available by
% Pat Langley and Andrea Danyluk for ICML-2K. This version was created
% by Iain Murray in 2018, and modified by Alexandre Bouchard in
% 2019 and 2021 and by Csaba Szepesvari, Gang Niu and Sivan Sabato in 2022.
% Modified again in 2023 and 2024 by Sivan Sabato and Jonathan Scarlett.
% Previous contributors include Dan Roy, Lise Getoor and Tobias
% Scheffer, which was slightly modified from the 2010 version by
% Thorsten Joachims & Johannes Fuernkranz, slightly modified from the
% 2009 version by Kiri Wagstaff and Sam Roweis's 2008 version, which is
% slightly modified from Prasad Tadepalli's 2007 version which is a
% lightly changed version of the previous year's version by Andrew
% Moore, which was in turn edited from those of Kristian Kersting and
% Codrina Lauth. Alex Smola contributed to the algorithmic style files.
\begin{thebibliography}{34}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Alaa \& van~der Schaar(2017)Alaa and van~der Schaar]{alaa2017bayesian}
Alaa, A.~M. and van~der Schaar, M.
\newblock Bayesian inference of individualized treatment effects using
multi-task gaussian processes.
\newblock \emph{Advances in Neural Information Processing Systems}, 30, 2017.
\bibitem[Bach(2013)]{bach2013sharp}
Bach, F.
\newblock Sharp analysis of low-rank kernel matrix approximations.
\newblock In \emph{Conference on learning theory}, pp.\ 185--209. PMLR, 2013.
\bibitem[Ban et~al.(2024)Ban, Riemens, and Rajan]{ban2024malleable}
Ban, H., Riemens, E.~H., and Rajan, R.~T.
\newblock Malleable kernel interpolation for scalable structured gaussian
process.
\newblock In \emph{2024 32nd European Signal Processing Conference (EUSIPCO)},
pp.\ 997--1001. IEEE, 2024.
\bibitem[Burt et~al.(2019)Burt, Rasmussen, and Van Der~Wilk]{burt2019rates}
Burt, D., Rasmussen, C.~E., and Van Der~Wilk, M.
\newblock Rates of convergence for sparse variational gaussian process
regression.
\newblock In \emph{International Conference on Machine Learning}, pp.\
862--871. PMLR, 2019.
\bibitem[Burt et~al.(2020)Burt, Rasmussen, and Van
Der~Wilk]{burt2020convergence}
Burt, D.~R., Rasmussen, C.~E., and Van Der~Wilk, M.
\newblock Convergence of sparse variational inference in gaussian processes
regression.
\newblock \emph{Journal of Machine Learning Research}, 21\penalty0
(131):\penalty0 1--63, 2020.
\bibitem[d'Aspremont(2008)]{daspremont2008smooth}
d'Aspremont, A.
\newblock Smooth optimization with approximate gradient.
\newblock \emph{SIAM Journal on Optimization}, 19\penalty0 (3):\penalty0
1171--1183, 2008.
\bibitem[Deisenroth et~al.(2015)Deisenroth, Fox, and
Rasmussen]{deisenroth2015gaussian}
Deisenroth, M.~P., Fox, D., and Rasmussen, C.~E.
\newblock Gaussian processes for data-efficient learning in robotics and
control.
\newblock \emph{IEEE transactions on pattern analysis and machine
intelligence}, 37\penalty0 (2):\penalty0 408--423, 2015.
\bibitem[Desai et~al.(2023)Desai, Gujarathi, Parikh, Yadav, Patel, and
Batra]{desai2023deep}
Desai, A., Gujarathi, E., Parikh, S., Yadav, S., Patel, Z., and Batra, N.
\newblock Deep gaussian processes for air quality inference.
\newblock In \emph{Proceedings of the 6th Joint International Conference on
Data Science \& Management of Data (10th ACM IKDD CODS and 28th COMAD)}, pp.\
278--279, 2023.
\bibitem[Devolder et~al.(2014)Devolder, Glineur, and
Nesterov]{devolder2014first}
Devolder, O., Glineur, F., and Nesterov, Y.
\newblock First-order methods of smooth convex optimization with inexact
oracle.
\newblock \emph{Mathematical Programming}, 146\penalty0 (1):\penalty0 37--75,
2014.
\bibitem[Frazier(2018)]{frazier2018tutorial}
Frazier, P.~I.
\newblock A tutorial on bayesian optimization.
\newblock \emph{arXiv preprint arXiv:1807.02811}, 2018.
\bibitem[Gardner et~al.(2018)Gardner, Pleiss, Weinberger, Bindel, and
Wilson]{gardner2018gpytorch}
Gardner, J., Pleiss, G., Weinberger, K.~Q., Bindel, D., and Wilson, A.~G.
\newblock Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu
acceleration.
\newblock \emph{Advances in neural information processing systems}, 31, 2018.
\bibitem[Girard et~al.(2002)Girard, Rasmussen, Candela, and
Murray-Smith]{girard2002gaussian}
Girard, A., Rasmussen, C., Candela, J.~Q., and Murray-Smith, R.
\newblock Gaussian process priors with uncertain inputs application to
multiple-step ahead time series forecasting.
\newblock \emph{Advances in neural information processing systems}, 15, 2002.
\bibitem[Hensman et~al.(2013)Hensman, Fusi, and Lawrence]{hensman2013gaussian}
Hensman, J., Fusi, N., and Lawrence, N.~D.
\newblock Gaussian processes for big data.
\newblock In \emph{Uncertainty in Artificial Intelligence}, 2013.
\bibitem[Horn \& Johnson(2012)Horn and Johnson]{horn2012matrix}
Horn, R.~A. and Johnson, C.~R.
\newblock \emph{Matrix analysis}.
\newblock Cambridge university press, 2012.
\bibitem[Huang et~al.(2023)Huang, Jiang, Zhao, Wu, Li, Lyu, Feng, Wei, Zhu, Gu,
et~al.]{luo2023diseasegps}
Huang, D., Jiang, J., Zhao, T., Wu, S., Li, P., Lyu, Y., Feng, J., Wei, M.,
Zhu, Z., Gu, J., et~al.
\newblock diseasegps: auxiliary diagnostic system for genetic disorders based
on genotype and phenotype.
\newblock \emph{Bioinformatics}, 39\penalty0 (9):\penalty0 btad517, 2023.
\bibitem[Kapoor et~al.(2021)Kapoor, Finzi, Wang, and Wilson]{kapoor2021skiing}
Kapoor, S., Finzi, M., Wang, K.~A., and Wilson, A. G.~G.
\newblock Skiing on simplices: Kernel interpolation on the permutohedral
lattice for scalable gaussian processes.
\newblock In \emph{International Conference on Machine Learning}, pp.\
5279--5289. PMLR, 2021.
\bibitem[Keys(1981)]{keys1981cubic}
Keys, R.
\newblock Cubic convolution interpolation for digital image processing.
\newblock \emph{IEEE transactions on acoustics, speech, and signal processing},
29\penalty0 (6):\penalty0 1153--1160, 1981.
\bibitem[Kolmogorov(1940)]{kolmogorov1940wienersche}
Kolmogorov, A.~N.
\newblock Wienersche spiralen und einige andere interessante kurven im
hilbertschen raum.
\newblock \emph{CR (Doklady) Acad. Sci. URSS (NS)}, 26:\penalty0 115--118,
1940.
\bibitem[Lagrange(1795)]{lagrange1795lecons}
Lagrange, J.-L.
\newblock \emph{Leçons élémentaires sur les mathématiques}.
\newblock Imprimerie de la République, 1795.
\bibitem[Lin et~al.(2024)Lin, Padhy, Antoran, Tripp, Terenin, Szepesvari,
Hern{\'a}ndez-Lobato, and Janz]{lin2024stochastic}
Lin, J.~A., Padhy, S., Antoran, J., Tripp, A., Terenin, A., Szepesvari, C.,
Hern{\'a}ndez-Lobato, J.~M., and Janz, D.
\newblock Stochastic gradient descent for gaussian processes done right.
\newblock In \emph{The Twelfth International Conference on Learning
Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=fj2E5OcLFn}.
\bibitem[Liu \& Onnela(2021)Liu and Onnela]{liu2021missing}
Liu, G. and Onnela, J.-P.
\newblock Bidirectional imputation of spatial gps trajectories with missingness
using sparse online gaussian process.
\newblock \emph{Journal of the American Medical Informatics Association},
28\penalty0 (8):\penalty0 1777--1784, 2021.
\bibitem[Modell(2024)]{modell2024entrywise}
Modell, A.
\newblock Entrywise error bounds for low-rank approximations of kernel
matrices.
\newblock \emph{arXiv preprint arXiv:2405.14494}, 2024.
\bibitem[Moreno et~al.(2023)Moreno, Mei, and Walters]{moreno2023ski}
Moreno, A., Mei, J., and Walters, L.
\newblock {SKI} to go faster: Accelerating toeplitz neural networks via
asymmetric kernels.
\newblock \emph{arXiv preprint arXiv:2305.09028}, 2023.
\bibitem[Morley et~al.(2020)Morley, Machado, Burr, Cowls, Joshi, Taddeo, and
Floridi]{morley2020ethics}
Morley, J., Machado, C.~C., Burr, C., Cowls, J., Joshi, I., Taddeo, M., and
Floridi, L.
\newblock The ethics of ai in health care: a mapping review.
\newblock \emph{Social Science \& Medicine}, 260:\penalty0 113172, 2020.
\bibitem[Musco \& Musco(2017)Musco and Musco]{musco2017recursive}
Musco, C. and Musco, C.
\newblock Recursive sampling for the nystrom method.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Rasmussen \& Williams(2006)Rasmussen and
Williams]{rasmussen2006gaussian}
Rasmussen, C.~E. and Williams, C. K.~I.
\newblock \emph{Gaussian processes for machine learning}.
\newblock MIT press, 2006.
\bibitem[Snelson \& Ghahramani(2006)Snelson and Ghahramani]{snelson2006sparse}
Snelson, E. and Ghahramani, Z.
\newblock Sparse gaussian processes using pseudo-inputs.
\newblock In \emph{Advances in neural information processing systems}, pp.\
1257--1264, 2006.
\bibitem[Stonyakin et~al.(2023)Stonyakin, Kuruzov, and
Polyak]{stonyakin2023stopping}
Stonyakin, F., Kuruzov, I., and Polyak, B.
\newblock Stopping rules for gradient methods for non-convex problems with
additive noise in gradient.
\newblock \emph{Journal of Optimization Theory and Applications}, 198\penalty0
(2):\penalty0 531--551, 2023.
\bibitem[Titsias(2009)]{titsias2009variational}
Titsias, M.~K.
\newblock Variational model selection for sparse gaussian process regression.
\newblock \emph{Report, University of Manchester, UK}, 2009.
\bibitem[Wild et~al.(2021)Wild, Kanagawa, and Sejdinovic]{wild2021connections}
Wild, V., Kanagawa, M., and Sejdinovic, D.
\newblock Connections and equivalences between the nystr{\"o}m method and
sparse variational gaussian processes.
\newblock \emph{arXiv preprint arXiv:2106.01121}, 2021.
\bibitem[Wilson \& Nickisch(2015)Wilson and Nickisch]{wilson2015kernel}
Wilson, A. and Nickisch, H.
\newblock Kernel interpolation for scalable structured gaussian processes
({KISS-GP}).
\newblock In \emph{International conference on machine learning}, pp.\
1775--1784. PMLR, 2015.
\bibitem[Wynne \& Wild(2022)Wynne and Wild]{wynne2022variational}
Wynne, G. and Wild, V.
\newblock Variational gaussian processes: A functional analysis view.
\newblock In \emph{International Conference on Artificial Intelligence and
Statistics}, pp.\ 4955--4971. PMLR, 2022.
\bibitem[Yadav et~al.(2021)Yadav, Pleiss, Gardner, Weinberger, and
Wilson]{yadav2021faster}
Yadav, M., Pleiss, G., Gardner, J., Weinberger, K.~Q., and Wilson, A.~G.
\newblock Faster kernel interpolation for gaussian processes.
\newblock In \emph{Proceedings of the 38th International Conference on Machine
Learning}, pp.\ 11279--11288. PMLR, 2021.
\bibitem[Yadav et~al.(2022)Yadav, Sheldon, and Musco]{yadav2022kernel}
Yadav, M., Sheldon, D., and Musco, C.
\newblock Kernel interpolation with sparse grids.
\newblock In \emph{Advances in Neural Information Processing Systems}, 2022.
\end{thebibliography}