Kernel quadrature with randomly pivoted Cholesky

Extracted source text · arXiv 2306.03955

\documentclass[preprint]{article}

% if you need to pass options to natbib, use, e.g.:
     \PassOptionsToPackage{sort&compress,numbers}{natbib}
% before loading neurips_2023


% ready for submission
\usepackage{neurips_2023}
\usepackage[dvipdfm]{graphicx}


% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
%     \usepackage[preprint]{neurips_2023}


% to compile a camera-ready version, add the [final] option, e.g.:
%     \usepackage[final]{neurips_2023}


% to avoid loading the natbib package, add option nonatbib:
%    \usepackage[nonatbib]{neurips_2023}


\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{xcolor}         % colors
\usepackage{amsmath,amssymb,mathtools,amsthm}
\usepackage{cleveref}
\usepackage{xspace}
\usepackage{subcaption}
\usepackage{autonum}

\crefname{equation}{}{}
\Crefname{equation}{Equation}{Equations}
\newcommand\hiddenref[1]{\sbox0{\ref{#1}}}

% Macros
\newcommand{\set}[1]{\mathsf{#1}}
\newcommand{\Ltwo}{\set{L}^2}
\newcommand{\ENE}[1]{{\color{blue} [\textbf{ENE}: #1]}}
\newcommand{\EMF}[1]{{\color{red} [\textbf{EMF}: #1]}}
\newcommand{\EM}[1]{\EMF{#1}}
\renewcommand{\d}{\mathrm{d}}
\newcommand{\real}{\mathbb{R}}
\newcommand{\RKHS}{\mathcal{H}}
\newcommand{\order}{\mathcal{O}}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\mat}[1]{\boldsymbol{#1}}
\renewcommand{\vec}[1]{\boldsymbol{#1}}
\DeclareMathOperator{\prob}{\mathbb{P}}
\DeclareMathOperator{\expect}{\mathbb{E}}
\newcommand{\RPCholesky}{\textsc{RPCholesky}\xspace}
\DeclareMathOperator{\Tr}{Tr}
\DeclareMathOperator{\Unif}{\textsc{Unif}}
\newcommand{\iu}{\mathrm{i}}

% Algorithms
\usepackage{algorithm, algpseudocode, algorithmicx}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

% Specialized nottion for this document
\newcommand{\Err}{\mathrm{Err}}
\DeclareMathOperator{\sgn}{sgn}

\newtheorem{theorem}{Theorem}
\newtheorem{proposition}[theorem]{Proposition}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}

\newif\ifneurips
\neuripsfalse

\newcommand{\boxedtext}[1]{\begin{center} \vspace{0.5pc}
		\fbox{ \begin{minipage}{0.9\textwidth}
				\begin{center} #1 \end{center}
			\end{minipage}
		} \vspace{0.5pc}
	\end{center}}

\includeonly{supplement.tex}

\title{Kernel Quadrature with Randomly Pivoted Cholesky}


% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.


\author{%
  Ethan N.~Epperly and Elvira Moreno\thanks{ENE acknowledges support by NSF FRG 1952777, Carver Mead New Horizons Fund, and the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0021110. EM was supported in part by Air Force Office of Scientific Research grant FA9550-22-1-0225.} \\
  Department of Computing and Mathematical Sciences\\
  California Institute of Technology\\
  Pasadena, CA 91125 \\
  \texttt{\{eepperly,emoreno2\}@caltech.edu} \\
  % examples of more authors
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
  % \AND
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
}


\begin{document}


\maketitle


\begin{abstract}
    This paper presents new quadrature rules for functions in a reproducing kernel Hilbert space using nodes drawn by a sampling algorithm known as randomly pivoted Cholesky.
    The resulting computational procedure compares favorably to previous kernel quadrature methods, which either achieve low accuracy or require solving a computationally challenging sampling problem. Theoretical and numerical results show that randomly pivoted Cholesky is fast and achieves comparable quadrature error rates to more computationally expensive quadrature schemes based on continuous volume sampling, thinning, and recombination. 
    Randomly pivoted Cholesky is easily adapted to complicated geometries with arbitrary kernels, unlocking new potential for kernel quadrature.
\end{abstract}

\section{Introduction}

Quadrature is one of the fundamental problems in computational mathematics, with applications in Bayesian statistics \cite{Bayesian_choice}, probabilistic ODE solvers \cite{prob_ode}, reinforcement learning \cite{rob_control}, and model-based machine learning \cite{MBML}.
The task is to approximate an integral of a function $f$ by the weighted sum of $f$'s values at judiciously chosen quadrature points $s_1,\ldots,s_n$:
%
\begin{equation} \label{eq:quadrature}
    \int_{\set{X}} f(x) g(x) \, \d\mu(x) \approx \sum_{i=1}^n w_i f(s_i).
\end{equation}
%
Here, and throughout, $\set{X}$ denotes a topological space equipped with a Borel measure $\mu$ and $g \in \Ltwo(\mu)$ denotes a square-integrable function.
The goal of kernel quadrature is to select quadrature weights $w_1,\ldots,w_n \in \real$ and nodes $s_1,\ldots,s_n\in \set{X}$ which minimize the error in the approximation \cref{eq:quadrature} for all $f$ drawn from a reproducing kernel Hilbert space (RKHS) $\RKHS$ of candidate functions.

The ideal kernel quadrature scheme would satisfy three properties:
%
\begin{enumerate}
    \item \emph{Spectral accuracy.} The error of the approximation \cref{eq:quadrature} decreases at a rate governed by the eigenvalues of the reproducing kernel $k$ of $\RKHS$, with rapidly decaying eigenvalues guaranteeing rapidly decaying quadrature error.
    \item \emph{Efficiency.} The nodes $s_1,\ldots,s_n$ and weights $w_1,\ldots,w_n$ can be computed by an algorithm which is efficient in both theory and practice.
\end{enumerate}

The first two goals may be more easily achieved if one has access to a Mercer decomposition
%
\begin{equation} \label{eq:mercer}
    k(x,y) = \sum_{i=1}^\infty \lambda_i e_i(x) e_i(y),
\end{equation}
%
where $e_1,e_2,\ldots$ form an orthonormal basis of $\Ltwo(\mu)$ and the eigenvalues $\lambda_1 \ge \lambda_2 \ge \cdots $ are decreasing.
Fortunately, the Mercer decomposition is known analytically for many RKHS's $\RKHS$ on simple sets $\set{X}$ such as boxes $[0,1]^d$.
However, such a decomposition is hard to compute for general kernels $k$ and domains $\set{X}$, leading to the third of our desiderata:
%
\begin{enumerate}
    \setcounter{enumi}{2}
    \item \emph{Mercer-free.} The quadrature scheme can be efficiently implemented without access to an explicit Mercer decomposition \cref{eq:mercer}.
\end{enumerate}

Despite significant progress in the probabilistic and kernel quadrature literature (see, e.g., \cite{BBC20, BBC19a, Bel21, FD12, Bardenet2016MonteCW,bach_lever,bach_herd,DM21,DM22,GBV19,TsujiTanakaPokutta2022,PWKQ,hayakawa2023samplingbased}), the search for a kernel quadrature scheme meeting all three criteria remains ongoing.

\paragraph{Contributions.}
We present a new kernel quadrature method based on the \emph{randomly pivoted Cholesky} (\RPCholesky) sampling algorithm that achieves all three of the goals.
Our main contributions are
%
\begin{enumerate}
    \item Generalizing \RPCholesky sampling (introduced in \cite{CETW23} for finite kernel matrices) to the continuum setting and demonstrating its effectiveness for kernel quadrature.
    \item Establishing theoretical results (see \cref{thm:rpc}) which show that \RPCholesky kernel quadrature achieves near-optimal quadrature error rates.
    \item Developing efficient rejection sampling implementations (see \cref{alg:rpc_rejection,alg:rpc_rejection+optimization}) of \RPCholesky in the continuous setting, allowing kernel quadrature to be applied to general spaces $\set{X}$, measures $\mu$, and kernels $k$ with ease.
\end{enumerate}

The remainder of this introduction will sketch our proposal for \RPCholesky kernel quadrature.
A comparison with existing kernel quadrature approaches appears in \cref{sec:related}.

\paragraph{Randomly pivoted Cholesky.} Let $\RKHS$ be an RKHS with a kernel $k$ that is integrable on the diagonal
%
\begin{equation} \label{eq:integrable}
    \int_{\set{X}} k(x,x) \, \d\mu(x) < +\infty.
\end{equation}
%
\RPCholesky uses the kernel diagonal $k(x,x)$ as a \emph{sampling distribution} to pick quadrature nodes.
The first node $s_1$ is chosen to be a sample from the distribution $k(x,x) \, \d\mu(x)$, properly normalized:
%
\begin{equation} \label{eq:rpc_sample}
    s_1 \sim \frac{k(x,x) \, \d\mu(x)}{\int_{\set{X}} k(x,x) \, \d\mu(x)}.
\end{equation}
%
Having selected $s_1$, we remove its influence on the kernel, updating the entire kernel function:
%
\begin{equation} \label{eq:rpc_update}
    k'(x,y) = k(x,y) - \frac{k(x,s_1)k(s_1,y)}{k(s_1,s_1)}.
\end{equation}
%
Linear algebraically, the update \cref{eq:rpc_update} can be interpreted as Gaussian elimination, eliminating ``row'' and ``column'' $s_1$ from the ``infinite matrix'' $k$.
Probabilistically, if we interpret $k$ as the covariance function of a Gaussian process, the update \cref{eq:rpc_update} represents conditioning on the value of the process at $s_1$.
We use the updated kernel $k'$ to select the next quadrature node:
%
\begin{equation}
    s_2 \sim \frac{k'(x,x) \, \d\mu(x)}{\int_{\set{X}} k'(x,x) \, \d\mu(x)},
\end{equation}
%
whose influence is then subtracted off $k'$ as in \cref{eq:rpc_update}.
\RPCholesky continues along these lines until $n$ nodes have been selected.
The resulting algorithm is shown in \cref{alg:rpc_unoptimized}.
Having chosen the nodes $s_1,\ldots,s_n$, our choice of weights $w_1,\ldots,w_n$ is standard and is discussed in \cref{sec:kq}.

\begin{algorithm}[ht]
	\caption{\RPCholesky: unoptimized implementation \label{alg:rpc_unoptimized}}
	\begin{algorithmic}[1]
		\Require Kernel $k : \set{X} \times \set{X} \to \real$ and number of quadrature points $n$
		\Ensure Quadrature points $s_1,\ldots,s_n$
		\For{$i = 1,2,\ldots,n$}
            \State Sample $s_i$ from the probability measure $k(x,x) \, \d\mu(x) / \int_{\set{X}} k(x,x) \, \d\mu(x)$
            \State Update kernel $k \gets k - k(\cdot,s_i) k(s_i,s_i)^{-1} k(s_i,\cdot)$
        \EndFor
	\end{algorithmic}
\end{algorithm}

\RPCholesky sampling is more flexible than many kernel quadrature methods, easily adapting to general spaces $\set{X}$, measures $\mu$, and kernels $k$.
To demonstrate this flexibility, we apply \RPCholesky to the region $\set{X}$ in \cref{fig:conditional} equipped with the Mat\'ern $5/2$-kernel with bandwidth $2$ and the measure
%
\begin{equation}
    \d\mu(x,y) = (x^2+y^2)\, \d x \, \d y.
\end{equation}
%
A set of $n = 20$ quadrature nodes produced by \RPCholesky sampling (using \cref{alg:rpc_rejection}) is shown in \cref{fig:conditional}.
The cyan--pink shading shows the diagonal of the kernel after updating for the selected nodes.
We see that near the selected nodes, the updated kernel is very small, meaning that future steps of the algorithm will avoid choosing nodes in those regions.
Nodes far from any currently selected nodes have a much larger kernel value, making them more likely to be chosen in future \RPCholesky iterations.
\begin{figure}[ht]
    \centering
    
    \begin{subfigure}{0.485\textwidth}
        \centering 
        \includegraphics[height=2in]{conditional.png}
        \caption{}\label{fig:conditional}
    \end{subfigure}
    \begin{subfigure}{0.485\textwidth}
        \centering \includegraphics[height=2in]{error_moon.png}
        \caption{}\label{fig:error_moon}
    \end{subfigure}
    \caption{\textbf{\RPCholesky kernel quadrature on oddly shaped region.} \textit{Left:} Black dots show 20 points selected by \RPCholesky on a crescent-shaped region. 
    Shading shows kernel values $k((x,y),(x,y))$ after removing influence of selected points. \textit{Right:} Mean relative error for \RPCholesky kernel quadrature, iid kernel quadrature, and Monte Carlo quadrature for $f(x,y) = \sin(x)\exp(y)$ with 100 trials.
    Shaded regions show 10\%/90\% quantiles.}\label{fig:moon}
\end{figure}

The quadrature error for $f(x,y) = \sin(x)\exp(y)$, $g\equiv 1$, and different numbers $n$ of quadrature nodes for \RPCholesky kernel quadrature, kernel quadrature with nodes drawn iid from $\mu / \mu(\set{X})$, and Monte Carlo quadrature are shown in \cref{fig:error_moon}.
Other spectrally accurate kernel quadrature methods would be difficult to implement in this setting because they would require an explicit Mercer decomposition (projection DPPs and leverage scores) or an expensive sampling procedure (continuous volume sampling).
A comparison of \RPCholesky with more kernel quadrature methods on a benchmark problem is provided in \cref{fig:benchmark}.

\section{Related work} \label{sec:related}

Here, we overview past work on kernel quadrature and discuss the history of \RPCholesky sampling.

\subsection{Kernel quadrature} \label{sec:related_kq}

The goal of kernel quadrature is to provide a systematic means for designing quadrature rules on RKHS's. Relevant points of comparison are Monte Carlo \cite{MCbound} and quasi-Monte Carlo \cite{dick_pillichshammer_2010} methods, which have $\order(1/\sqrt{n})$ and $\order(\operatorname{polylog} (n)/n)$ convergence rates, respectively.

The literature on probabilistic and kernel approaches to quadrature is vast, so any short summary is necessarily incomplete.
Here, we briefly summarize some of the most prominent kernel quadrature methods, highlighting limitations that we address with our \RPCholesky approach.
We refer the reader to \cite[Tab.~1]{PWKQ} for a helpful comparison of many of the below-discussed methods. 

\paragraph{Herding.} Kernel herding schemes \cite{CWS10,bach_herd,TsujiTanakaPokutta2022,FD12} iteratively select quadrature nodes using a greedy approach.
These methods face two limitations.
First, they require the solution of a (typically) nonconvex global optimization problem at every step, which may be computationally costly.
Second, these methods exhibit slow $\order(1/n)$ quadrature error rates \cite[Prop.~4]{CWS10} (or even slower \cite{Dun80}). 
Closely related to kernel herding is sequential Bayesian quadrature \cite{FD12}.

\paragraph{Thinning.}
Thinning methods \cite{DM21,DM22,SDM22} try to select $n$ good quadrature nodes as a subset of a larger set of $N\gg n$ candidate nodes drawn from a simple distribution. 
While these methods have other desirable theoretical properties, they do not benefit from spectral accuracy. 

\paragraph{Leverage scores and projection DPPs.}
With optimal weights (\cref{sec:kq}), quadrature with nodes sampled using a projection determintal point process (DPP) \cite{BBC19a} (see also \cite{GBV19,Bardenet2016MonteCW, Bel21}) or iid from the (ridge) leverage score distribution \cite{bach_lever} achieve spectral accuracy.
However, known efficient sampling algorithms require access to the Mercer decomposition \cref{eq:mercer}, limiting the applicability of these schemes to simple spaces $\set{X}$, measures $\mu$, and kernels $k$ where the decomposition is known analytically.

\paragraph{Continuous volume sampling.}
Continuous volume sampling is the continuous analog of $n$-DPP sampling \cite{KT11}, providing quadrature nodes that achieve spectral accuracy \cite[Prop.~5]{BBC20}.
Unfortunately, continuous volume sampling is computationally challenging.
In the finite setting, the best-known algorithm \cite{CDV20a} for exact $n$-DPP sampling requires a costly $\order(|\set{X}| \cdot n^{6.5} + n^{9.5})$ operations.
Inexact samplers based on Markov chain Monte Carlo (MCMC) (e.g., \cite{AOR16,ALV22,RO19}) may be more competitive, but the best-known samplers in the continuous setting still require an expensive $\order(n^5 \log n)$ MCMC steps \cite[Thms.~1.3--1.4]{RO19}.
\RPCholesky sampling achieves similar theoretical guarantees to continuous volume sampling (see \cref{thm:rpc}) and can be efficiently and exactly sampled (\cref{alg:rpc_rejection}).

\paragraph{Recombination and convex weights.}
The paper \cite{PWKQ} (see also \cite{hayakawa2023samplingbased}) proposes two ideas for kernel quadrature when $\mu$ is a probability measure and $g\equiv 1$.
First, they suggest using a recombination algorithm (e.g., \cite{LL12}) to subselect good quadratures from $N\gg n$ candidate nodes iid sampled from $\mu$.
All of the variants of their method either fail to achieve spectral accuracy or require an explicit Mercer decomposition \cite[Tab.~1]{PWKQ}.
Second, they propose choosing weights $(w_1,\ldots,w_n)$ that are convex combination coefficients.
This choice makes the quadrature scheme robust against misspecification of the RKHS $\RKHS \not\ni f$, among other benefits.
It may be worth investigating a combination of \RPCholesky quadrature nodes with convex weights in future work.

\subsection{Randomly pivoted Cholesky}

\RPCholesky was proposed, implemented, and analyzed in \cite{CETW23} for the task of approximating an $M\times M$ positive-semidefinite matrix $\mat{A}$.
The algorithm is algebraically equivalent to applying an earlier algorithm known as \emph{adaptive sampling} \cite{DRVW06,DV06} to $\mat{A}^{1/2}$.
Despite their similarities, \RPCholesky and adaptive sampling are different algorithms: To produce a rank-$n$ approximation to an $M\times M$ matrix, \RPCholesky requires $\order(n^2M)$ operations, while adaptive sampling requires a much larger $\order(nM^2)$ operations.
See \cite[\S4.1]{CETW23} for further discussion on \RPCholesky's history.
In this paper, we introduce a continuous extension of \RPCholesky and analyze its effectiveness for kernel quadrature.

\section{Theoretical results} \label{sec:main}

In this section, we prove our main theoretical result for \RPCholesky kernel quadrature. 
We first establish the mathematical setting (\cref{sec:setting}) and introduce kernel quadrature (\cref{sec:kq}).
Then, we present our main theorem (\cref{sec:error_bound}) and discuss its proof (\cref{sec:proof_idea}).

\subsection{Mathematical setting and notation} \label{sec:setting}

Let $\mu$ be a Borel measure supported on a topological space $\set{X}$ and let $\RKHS$ be a RKHS on $\set{X}$ with continuous kernel $k : \set{X} \times \set{X} \to \real$.
We assume that $x \mapsto k(x,x)$ is integrable \cref{eq:integrable} and that $\RKHS$ is dense in $\Ltwo(\mu)$.
These assumptions imply that $\RKHS$ is compactly embedded in $\Ltwo(\mu)$, the Mercer decomposition \cref{eq:mercer} converges pointwise, and the Mercer eigenfunctions form an orthonormal basis of $\Ltwo(\mu)$ and an orthogonal basis of $\RKHS$ \cite[Thm.~3.1]{SS12}.

Define the integral operator
%
\begin{equation} \label{eq:integral_operator}
    Tf(x) = \int_{\set{X}} k(x,y) f(y) \, \d\mu(y).
\end{equation}
%
Viewed as an operator $T : \Ltwo(\mu) \to \Ltwo(\mu)$, $T$ is self-adjoint, positive semidefinite, and trace-class.

One final piece of notation: For a function $\delta : \set{X} \to \real$ and an ordered (multi)set $\set{S} = \{s_1,\ldots,s_n\}$, we let $\delta(\set{S})$ be the column vector with $i$th entry $\delta(s_i)$.
Similarly, for a symmetric bivariate function $h : \set{X} \times \set{X}\to\real$, $h(\cdot,\set{S})$ (resp.\ $h(\set{S},\cdot)$) denotes the row (resp.\ column) vector-valued function with $i$th entry $h(\cdot,s_i)$, and $h(\set{S},\set{S})$ denotes the matrix with $ij$ entry $h(s_i,s_j)$.

\subsection{Kernel quadrature} \label{sec:kq}

Following earlier works (e.g., \cite{bach_lever,BBC19a}), let us describe the kernel quadrature problem more precisely.
Given a function $g \in \Ltwo(\mu)$, we seek quadrature weights $\vec{w} = (w_1,\ldots,w_n)\in \real^n$ and nodes $\set{S} = \{ s_1,\ldots,s_n \} \subseteq \set{X}$ that minimize the maximum quadrature error over all $f \in \RKHS$ with $\norm{f}_\RKHS \le 1$:
%
\begin{equation}
    \mathrm{Err}(\set{S},\vec{w};g) \coloneqq \max_{\norm{f}_{\RKHS} \le 1} \left|\int_{\set{X}} f(x) g(x) \, \d\mu(x) - \sum_{i=1}^n w_i f(s_i)\right|.
\end{equation}
%
A short derivation (see, e.g., \cite[Eq.~(7)]{bach_lever}) yields the simple formula
%
\begin{equation}
    \mathrm{Err}(\set{S},\vec{w};g) = \norm{Tg - \sum_{i=1}^n w_i k(\cdot,s_i)}_{\RKHS}.
\end{equation}
This equation reveals that the quadrature rule minimizing $\Err(\set{S},\vec{w};g)$ is the least-squares approximation of  $Tg \in \RKHS$ by a linear combination of kernel function evaluations $\sum_{i=1}^n w_i k(\cdot,s_i)$.

If we fix the quadrature nodes $\set{S} = \{s_1,\ldots,s_n\}$, the optimal weights $\vec{w}_{\star} = (w_{\star 1},\ldots,w_{\star n}) \in \real^n$ minimizing $\mathrm{Err}(\set{S},\vec{w};g)$ are the solution of the linear system of equations
%
\begin{equation} \label{eq:weights}
    k(\set{S},\set{S})\vec{w}_\star = Tg(\set{S}).
\end{equation}
%
We use \cref{eq:weights} to select the weights throughout this paper, and denote the error with optimal weights by
%
\begin{equation} \label{eq:error_def}
    \Err(\set{S};g) \coloneqq \Err(\set{S},\vec{w}_\star;g).
\end{equation}

\subsection{Error bounds for randomly pivoted Cholesky kernel quadrature} \label{sec:error_bound}

Our main result for \RPCholesky kernel quadrature is as follows:
%
\begin{theorem} \label{thm:rpc}
    Let $\set{S} = \{s_1,\ldots,s_n\}$ be generated by the \RPCholesky sampling algorithm.
    For any function $g \in \Ltwo(\mu)$, nonnegative integer $r \ge 0$, and real number $\delta > 0$, we have
    %
    \begin{equation}
        \expect \Err^2(\set{S};g) \le \delta \cdot \norm{g}_{\Ltwo(\mu)}^2 \cdot \sum_{i=r+1}^\infty \lambda_i 
    \end{equation}
    %
    provided
    %
    \begin{equation} \label{eq:n_condition}
        n \ge r \log\left( \frac{2\lambda_1}{\delta \sum_{i=r+1}^\infty \lambda_i} \right) + \frac{2}{\delta}.
    \end{equation}
\end{theorem}

To illustrate this result, consider an RKHS with eigenvalues $\lambda_i = \order(i^{-2s})$ for $s > 1/2$.
An example is the periodic Sobolev space $H^s_{\rm per}([0,1])$ (see \cref{sec:experiments}).
By setting $\delta = 1/r$, we see that 
%
\begin{equation}
    \expect \Err^2(\set{S};g) \le \order(r^{-2s}) \norm{g}_{\Ltwo(\mu)}^2 \quad \text{for \RPCholesky with} \quad n = \Omega(r \log r).
\end{equation}
%
The optimal scheme requires $n = \Theta(r)$ nodes to achieve this bound \cite[Prop.~5 \& \S2.5]{BBC20}.
Thus, to achieve an error of $\order(r^{-2s})$, \textbf{\RPCholesky requires just logarithmically more nodes than the optimal quadrature scheme for such spaces.}
This compares favorably to continuous volume sampling, which achieves the slightly better optimal rate but is much more difficult to sample.

Having established that \RPCholesky achieves nearly optimal error rates for interesting RKHS's, we make some general comments on \cref{thm:rpc}.
First, observe that the error depends on the sum $\sum_{i=r+1}^\infty \lambda_i$ of the tail eigenvalues.
This tail sum is characteristic of spectrally accurate kernel quadrature schemes \cite{BBC19a,BBC20}.
The more distinctive feature in \cref{eq:n_condition} is the logarithmic factor.
Fortunately, this factor can never be large in ordinary computations.
Indeed, for double precision computation, the achieveable accuracy is bounded by the machine precision $2^{-52}$, so this logarithmic factor is effectively bounded:
%
\begin{equation}
    \log\left(\frac{2\lambda_1}{\delta \sum_{i=r+1}^\infty}\right)\lessapprox \log(2^{52}) < 37.
\end{equation}

\subsection{Idea of proof} \label{sec:proof_idea}

We briefly outline the proof of \cref{thm:rpc}.
Following previous works (e.g., \cite{bach_lever,hayakawa2023samplingbased}), we first utilize the connection between kernel quadrature and the Nystr\"om approximation \cite{Nys30,WS00} of the kernel $k$.

\begin{definition} \label{def:nystrom}
    For nodes $\set{S} = \{s_1,\ldots,s_n\} \subseteq \set{X}$, the \emph{Nystr\"om approximation} to the kernel $k$ is
    %
    \begin{equation} \label{eq:nystrom_kernel}
        k_\set{S}(x,y) = k(x,\set{S}) \, k(\set{S},\set{S})^\dagger \, k(\set{S},y).
    \end{equation}
    %
    Here, ${}^\dagger$ denotes the Moore--Penrose pseudoinverse.
    The Nystr\"om approximate integral operator is 
    %
    \begin{equation}
        T_{\set{S}}f \coloneqq \int_{\set{X}} k_{\set{S}}(\cdot,x) f(x) \, \d\mu(x).
    \end{equation}
\end{definition}

This operator leads to a simple formula for the quadrature rule $\sum_{i=1}^n w_i k(\cdot,s_i)$ with optimal weights \cref{eq:weights}.

\begin{proposition}
    Fix nodes $\set{S} = \{s_1,\ldots,s_n\}$.
    With the optimal weights $\vec{w}_\star \in \real^n$ \cref{eq:weights}, we have
    %
    \begin{equation} \label{eq:weights_nystrom}
        \sum_{i=1}^n w_{i\star} k(\cdot,s_i) = T_{\set{S}}g.
    \end{equation}
    %
    Consequently,
    %
    \begin{equation} \label{eq:err_nystrom}
        \Err^2(\set{S};g) = \norm{(T-T_{\set{S}})g}_{\RKHS}^2 \le \langle g, (T-T_{\set{S}})g\rangle_{\Ltwo(\mu)}.
    \end{equation}
\end{proposition}

\begin{proof}
Equation~\cref{eq:weights_nystrom} follows from a direct calculation:
%
\begin{align}
    \sum_{i=1}^n w_i k(\cdot,s_i) &= k(\cdot,\set{S})\vec{w} = \int_{\set{X}} k(\cdot,\set{S}) k(\set{S},\set{S})^{-1} k(\set{S},x) g(x) \, \d\mu(x) = T_{\set{S}}g.
\end{align}
%
To establish the second claim, use the fact \cite[\S2.1]{bach_lever} that $T^{-1/2} : \RKHS \to \Ltwo(\mu)$ is an isometry and invoke the definition \cref{eq:error_def} of $\Err(\set{S};g)$:
%
\begin{align}
    \Err^2(\set{S};g) &= \norm{(T-T_{\set{S}})g}_{\RKHS}^2  = \norm{T^{-1/2}(T-T_{\set{S}})g}_{\Ltwo(\mu)}^2 \\
    &= \langle g, (T-T_{\set{S}})T^{-1}(T-T_{\set{S}}) g \rangle_{\Ltwo(\mu)} \le \langle g, (T-T_{\set{S}}) g \rangle_{\Ltwo(\mu)}.
\end{align}
%
The last inequality follows because $T - T_{\set{S}}$ and $T_{\set{S}} = T - (T - T_{\set{S}})$ are both positive semidefinite.
\end{proof}

To finish the proof of \cref{thm:rpc}, we develop and solve a recurrence for an upper bound on the largest eigenvalue of $\expect[T - T_{\set{S}}]$; see \cref{sec:proof_of_rpc} for details.

\section{Efficient randomly pivoted Cholesky by rejection sampling}
\label{sec:rejection}

To efficiently perform \RPCholesky sampling in the continuous setting, we can use \emph{rejection sampling}.
Assume for simplicity that the measure $\mu$ is normalized so that
%
\begin{equation}
    \int_{\set{X}} k(x,x) \, \d\mu(x) = \Tr T = 1.
\end{equation}
%
We assume two forms of access to the kernel $k$:
%
\begin{enumerate}
    \item \textbf{Entry evaluations.} For any $x,y \in \set{X}$, we can evaluate $k(x,y)$.
    \item \textbf{Sampling the diagonal.} We can produce samples from the measure $k(x,x) \, \d\mu(x)$.
\end{enumerate}
%
To sample from the \RPCholesky distribution, we use $k(x,x) \, \d\mu(x)$ as a proposal distribution and accept proposal $s$ with probability 
%
\begin{equation}
    1 - \frac{k_{\set{S}}(s,s)}{k(s,s)}.
\end{equation}
%
(Recall $k_{\set{S}}$ from \cref{eq:nystrom_kernel}.)
The resulting algorithm for \RPCholesky sampling is shown in \cref{alg:rpc_rejection}.

\begin{algorithm}[ht]
	\caption{\RPCholesky with rejection sampling \label{alg:rpc_rejection}}
	\begin{algorithmic}[1]
		\Require Kernel $k : \set{X} \times \set{X} \to \real$ and number of quadrature points $n$
		\Ensure Quadrature points $s_1,\ldots,s_n$
        \State Initialize $\mat{L} \leftarrow \mat{0}_{n\times n}$, $i\gets 1$, $\set{S} \leftarrow \emptyset$ \Comment{$\mat{L}$ stores a Cholesky decomposition of $k(\set{S},\set{S})$}
        \While{$i \le n$}
            \State Sample $s_i$ from the probability measure $k(x,x) \, \d\mu(x)$
            \State $(d,\vec{c}) \leftarrow \Call{ResidualKernel}{s_i,\set{S},\mat{L},k}$ \Comment{Helper subroutine \cref{alg:helper}}
            \State Draw a uniform random variable $U \sim \Unif[0,1]$
            \If{$U < d/k(s_i,s_i)$} \Comment{Accept with probability $d/k(s_i,s_i)$}
             \State Induct $s_i$ into the selected set: $\set{S} \gets \set{S} \cup \{s_i\}$, $i \leftarrow i+1$
             \State $\mat{L}(i,1:i) \leftarrow \begin{bmatrix} \vec{c}^\top & \sqrt{d}\end{bmatrix}$
            \EndIf 
        \EndWhile
	\end{algorithmic}
\end{algorithm}

\begin{algorithm}[ht]
	\caption{Helper subroutine to evaluate residual kernel \label{alg:helper}}
	\begin{algorithmic}[1]
		\Require Point $s \in \set{X}$, set $\set{S} \subseteq \set{X}$, Cholesky factor $\mat{L}$ of $k(\set{S},\set{S})$,  and kernel $k$
		\Ensure $d = k(s,s) - k_{\set{S}}(s,s)$ and $\vec{c} = \mat{L}^{-1}k(\set{S},s)$
        \Procedure{ResidualKernel}{$s$,$\set{S}$,$\mat{L}$,$k$}
        \State $\vec{c} \leftarrow \mat{L}^{-1} k(\set{S},s)$
        \State $d \leftarrow k(s,s) - \norm{\vec{c}}^2$ \Comment{$d = k(s,s) - k_{\set{S}}(s,s)$}
        \State \Return $(d,\vec{c})$
        \EndProcedure
	\end{algorithmic}
\end{algorithm}

\begin{theorem} \label{thm:rejection}
    \Cref{alg:rpc_rejection} produces exact \RPCholesky samples.
    Let $\eta_i$ denote the trace-error of the best rank-$i$ approximation to $T$:
    %
    \begin{equation}
        \eta_i = \sum_{j=i+1}^\infty \lambda_i.
    \end{equation}
    %
    The expected runtime of \cref{alg:rpc_rejection} is at most $\order(\sum_{i=1}^n i^2 /\eta_i) \le \order(n^3/\eta_{n-1})$.
\end{theorem}

This result demonstrates that the time required by \cref{alg:rpc_rejection} increases as the Nystr\"om approximation becomes more accurate.
The algorithm works under minimal assumptions and can reach useful accuracies $\eta \ge 10^{-5}$ and sample $n = 1000$ nodes on a laptop.
\Cref{alg:rpc_rejection} may also be interesting for \RPCholesky sampling for a large finite set $|\set{X}| \ge 10^9$, since its runtime has no explicit dependence on the size of the space $\set{X}$.

To achieve higher accuracies, we can improve \cref{alg:rpc_rejection} with optimization.
Indeed, the optimal acceptance probability for \cref{alg:rpc_rejection} would be
%
\begin{equation}
    p(s_i;\alpha) = \frac{1}{\alpha} \cdot \frac{k(s_i,s_i) - k_{\set{S}}(s_i,s_i)}{k(s_i,s_i)} \quad \text{where} \quad \alpha = \max_{x \in \set{X}} \frac{k(x,x) - k_{\set{S}}(x,x)}{k(x,x)}.
\end{equation}
%
This suggests the following scheme: Initialize with $\alpha = 1$ and run the algorithm with acceptance probability $p(s_i;\alpha)$.
If we perform many loop iterations without an acceptance, we then recompute the optimal $\alpha$ by solving an optimization problem.
The resulting procedure is shown in \cref{alg:rpc_rejection+optimization}.

\begin{algorithm}[ht]
	\caption{\RPCholesky with optimized rejection sampling \label{alg:rpc_rejection+optimization}}
	\begin{algorithmic}[1]
		\Require Kernel $k : \set{X} \times \set{X} \to \real$ and number of quadrature points $n$
		\Ensure Quadrature points $s_1,\ldots,s_n$
        \State Initialize $\mat{L} \leftarrow \mat{0}_{n\times n}$, $i\gets 1$, $\set{S} \leftarrow \emptyset$, $\mathtt{trials} \gets 0$, $\alpha \gets 1$
        \While{$i \le n$}
            \State $\mathtt{trials} \gets \mathtt{trials} + 1$
            \State Sample $s_i$ from the probability measure $k(x,x) \, \d\mu(x)$
            \State $(d,\vec{c}) \leftarrow \Call{ResidualKernel}{s_i,\set{S},\mat{L},k}$ \Comment{Helper subroutine \cref{alg:helper}}
            \State Draw a uniform random variable $U \sim \Unif[0,1]$
            \If{$U < (1/\alpha) \cdot d/k(s_i,s_i)$} \Comment{Accept with probability $d/k(s_i,s_i)$}
             \State $\set{S} \gets \set{S} \cup \{s_i\}$, $i \leftarrow i+1$, $\mathtt{trials} \gets 0$
             \State $\mat{L}(i,1:i) \leftarrow \begin{bmatrix} \vec{c}^\top & \sqrt{d}\end{bmatrix}$
            \EndIf 
            \If{$\mathtt{trials} \ge \mathtt{TRIALS\_MAX}$} \Comment{We use $\mathtt{TRIALS\_MAX} \in [25,1000]$}
            \State $\alpha \gets \max_{x \in \set{X}} \Call{ResidualKernel}{x,\set{S},\mat{L},k}/k(x,x)$
            \State $\mathtt{trials} \gets 0$
            \EndIf
        \EndWhile
	\end{algorithmic}
\end{algorithm}

If the optimization problem for $\alpha$ is solved to global optimality, then \cref{alg:rpc_rejection+optimization} produces exact \RPCholesky samples.
The downside of \cref{alg:rpc_rejection+optimization} is the need to solve a (typically) nonconvex global optimization problem to compute the optimal $\alpha$.
Fortunately, in our experiments (\cref{sec:experiments}), only a small number of optimization problems ($\le10$) are needed to produce a sample of $n=1000$ nodes.
In the setting where the optimization problem is tractable, the speedup of \cref{alg:rpc_rejection+optimization} can be immense.
To produce $n = 200$ \RPCholesky samples for the space $[0,1]^3$ with the kernel \cref{eq:3d_kernel}, \cref{alg:rpc_rejection+optimization} requires just $0.19$ seconds compared to $7.47$ seconds for \cref{alg:rpc_rejection}.

\section{Comparison of methods on a benchmark example}
\label{sec:experiments}

Having demonstrated the versatility of \RPCholesky for general spaces $\set{X}$, measures $\mu$, and kernels $k$ in \cref{fig:moon}, we now present a benchmark example from the kernel quadrature literature to compare \RPCholesky with other methods.
Consider the periodic Sobolev space $H^s_{\rm per}([0,1])$ with kernel
%
\begin{equation}
    k(x,y) = 1 + 2\sum_{m = 1}^\infty m^{-2s} \cos(2\pi m(x-y)) = 1 + \frac{(-1)^{s-1}(2\pi)^{2s}}{(2s)!}B_{2s}(\{x-y\}),
\end{equation}
%
where $B_{2s}$ is the $(2s)$th Bernoulli polynomial and $\{\cdot\}$ reports the fractional part of a real number \cite[p.~7]{bach_lever}.
We also consider $[0,1]^3$ equipped with the product kernel
%
\begin{equation} \label{eq:3d_kernel}
    k^{\otimes 3}((x_1,x_2,x_3),(y_1,y_2,y_3)) = k(x_1,y_1) k(x_2,y_2)k(x_3,y_3).
\end{equation}
%
We quantify the performance of nodes $\set{S}$ and weights $\vec{w}$ using $\Err(\set{S},\vec{w};g)$, which can be computed in closed form \cite[Eq.~(14)]{PWKQ}.
We set $\mu \coloneqq \Unif[0,1]^d$ and $g\equiv 1$.

We compare the following schemes:
%
\begin{itemize}
    \item \emph{Monte Carlo, IID kernel quadrature.} Nodes $s_1,\ldots,s_n \stackrel{\mathrm{iid}}{\sim} \mu$ with uniform weights $w_i = 1/n$ (Monte Carlo) and optimal weights \cref{eq:weights} (IID).
    \item \emph{Thinning.} Nodes $s_1,\ldots,s_n$ thinned from $n^2$ iid samples from $\mu = \Unif[0,1]$ using kernel thinning \cite{DM21,DM22} with the \textsc{Compress}\texttt{++} algorithm \cite{SDM22} and with uniform weights $w_i = 1/n$.
    \item \emph{Continuous volume sampling (CVS).} Nodes $s_1,\ldots,s_n$ drawn from the volume sampling distribution \cite[Def.~1]{BBC20} by Markov chain Monte Carlo with optimal weights \cref{eq:weights}.
    \item \emph{\RPCholesky.} Nodes $s_1,\ldots,s_n$ sampled by \RPCholesky using \cref{alg:rpc_rejection+optimization} with the optimal weights \cref{eq:weights}.
    \item \emph{Positively weighted kernel quadrature (PWKQ).} Nodes and weights computed by the Nystr\"om+empirical+opt method of \cite{PWKQ}.
\end{itemize}
%
See \cref{sec:numerical_experiment_details} for more details about our numerical experiments.
Experiments were run on a MacBook Pro with a 2.4 GHz 8-Core Intel Core i9 CPU and 64 GB 2667 MHz DDR4 RAM.
Our code is available at
%
\boxedtext{\url{https://github.com/eepperly/RPCholesky-Kernel-Quadrature}.}

\begin{figure}[ht]
    \centering
    
    \begin{subfigure}{0.98\textwidth}
        \centering 
        \includegraphics[height=1.9in]{1d_errors.png} ~ \includegraphics[height=1.9in]{1d_times.png}
        \caption{$s=3$, $d = 1$}
    \end{subfigure}
    
    \begin{subfigure}{0.98\textwidth}
        \centering \includegraphics[height=1.9in]{3d_errors.png} ~ \includegraphics[height=1.9in]{3d_times.png}
        \caption{$s=3$, $d = 3$}
    \end{subfigure}
    \caption{\textbf{Benchmark example: Sobolev space.} Mean quadrature error $\Err(\set{S},\vec{w};g)$ (\emph{left}) and sampling time (\emph{right}) for different methods (100 trials) for $s = 1$, $d = 3$ (\emph{top}) and $s = d = 3$ (\emph{bottom}). Shaded regions show 10\%/90\% quantiles.}\label{fig:benchmark}
\end{figure}

Errors for the different methods are shown in \cref{fig:benchmark} (left panels).
We see that \RPCholesky consistently performs among the best methods at every value of $n$, numerically achieving the rate of convergence of the fastest method for each problem.
Particularly striking is the result that \RPCholesky sampling's performance is either very close to or better than that of continuous volume sampling, despite the latter's slightly stronger theoretical properties.

\Cref{fig:benchmark} (right panels) show the sampling times for each of the nonuniform sampling methods.
\RPCholesky sampling is the fastest by far.
To sample 128 nodes for $s = 3$ and $d=1$, \RPCholesky was \textbf{17$\times$ faster than continuous volume sampling, 170$\times$ faster than PWKQ, and 508$\times$ faster than thinning.}

To summarize our numerical evaluation (comprising \cref{fig:benchmark} and \cref{fig:moon} from the introduction), we find that \RPCholesky is among the most accurate kernel quadrature methods tested.
To us, the strongest benefits of \RPCholesky (supported by these experiments) are the method's speed and flexibility. These virtues make it possible to apply spectrally accurate kernel quadrature in scenarios where it would have been computationally intractable before.  

\section{Conclusions, Limitations, and Possibilites for Future Work}

In this article, we developed continuous \RPCholesky sampling for kernel quadrature. \Cref{thm:rpc} demonstrates \RPCholesky kernel quadrature achieves near-optimal error rates.
Numerical results (\cref{fig:benchmark}) hint that its practical performance might be even better than that suggested by our theoretical analysis and fully comparable with kernel quadrature based on the much more computationally expensive continuous volume sampling distribution.
\RPCholesky supports performant rejection sampling algorithms (\cref{alg:rpc_rejection,alg:rpc_rejection+optimization}), which facilitate implementation for general spaces $\set{X}$, measures $\mu$, and kernels $k$ with ease.

We highlight three limitations of \RPCholesky sampling that would be worth addressing in future work.
First, given the comparable performance of \RPCholesky and continuous volume sampling in practice, it would be desirable to prove stronger error bounds for \RPCholesky sampling or find counterexamples which demonstrate a separation between the methods.
Second, it would be worth developing improved sampling algorithms for \RPCholesky which avoid the need for global optimization steps.
Third, all known spectrally accurate kernel quadrature methods require integrals of the form $\int_{\set{X}} k(x,y) g(y) \, \d\mu(y)$, which may not be available.
\RPCholesky kernel quadrature requires them for the computation of the weights \cref{eq:weights}.
Developing spectrally accurate kernel quadrature schemes that avoid such integrals remains a major open problem for the field.

\section*{Acknowledgements}

We thank Eliza O'Reilly, Joel Tropp, and Robert Webber for helpful discussions and feedback.

\section*{Disclaimer}

This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.

\bibliographystyle{plain}
\bibliography{refs}

\include{supplement}

\end{document}
\newpage

\appendix

\newcommand{\twobyone}[2]{\begin{bmatrix} #1 \\ #2 \end{bmatrix}}
\newcommand{\onebytwo}[2]{\begin{bmatrix} #1 & #2 \end{bmatrix}}
\newcommand{\twobytwo}[4]{\begin{bmatrix} #1 & #2 \\ #3 & #4 \end{bmatrix}}
\newcommand{\Id}{\mathbf{I}}

\section{Proof of \cref{thm:rpc}} \label{sec:proof_of_rpc}

We begin by establishing some notation. Let $\preceq$ denote the Loewner order for bounded, linear operators on $\Ltwo(\mu)$. That is, $A \preceq B$ if $B-A$ is positive semidefinite (psd). Let $k_i$ denote the residual
%
\begin{equation}
    k_i \coloneqq k - k_{\{s_1,\ldots,s_i\}}
\end{equation}
%
for the approximate kernel associated with the first $i$ nodes.
(See \cref{prop:rpc_equivalence} below.)
The associated integral operator is 
%
\begin{equation}
    T_if \coloneqq \int_{\set{X}} k_i(\cdot,x) f(x) \,\d\mu(x).
\end{equation}
%

To analyze \RPCholesky, we begin by analyzing a single step of the procedure.
The expected value of the residual kernel after one step, $k_1$, is 
%
\begin{equation}
    \expect k_1(x,y) = k(x,y) - \frac{\int_{\set{X}} k(x,s) k(s,y) \, \d\mu(s)}{\int_{\set{X}} k(z,z) \, \d\mu(z)}.
\end{equation}
%
Consequently, the expected value of the integral operator $T_1$ is
%
\begin{equation} \label{eq:Phi}
    \Phi(T) \coloneqq \expect[T_1] = T - \frac{T^2}{\Tr T}.
\end{equation}

The map $\Phi$ enjoys several useful properties \cite[Lem.~3.2]{CETW23}.

\begin{proposition}
    The map $\Phi$ is positive, monotone, and concave with respect to the order $\preceq$ on the set of trace-class psd operators on $\Ltwo(\mu)$.
    That is, for trace-class psd operators $A,B$,
    %
    \begin{equation} \label{eq:positive_monotone}
        0\preceq A\preceq B \implies 0\preceq \Phi(A) \preceq \Phi(B)
    \end{equation}
    %
    and, for $\theta \in [0,1]$,
    %
    \begin{equation}
        \theta \Phi(A) + (1-\theta)\Phi(B) \preceq \Phi(\theta A + (1-\theta)B). \label{eq:concavity}
    \end{equation}
\end{proposition}

\newcommand{\IdOp}{\mathrm{Id}}

\begin{proof}
    We reproduce the proof from \cite[Lem.~3.2]{CETW23} for completeness. 

    \emph{\textbf{Positivity.}} Assume $A\succeq 0$.
    Observe that $\IdOp \succeq A / \Tr A$, where $\IdOp$ denotes the identity operator.
    Thus
    %
    \begin{equation}
        \Phi(A) = A \left( \IdOp - \frac{A}{\Tr A}\right) \succeq 0,
    \end{equation}
    %
    This establishes that $\Phi$ is positive: $A \succeq 0$ implies $\Phi(A) \succeq 0$.

    \emph{\textbf{Concavity.}} For $\theta \in [0,1]$, denote $\overline{\theta} \coloneqq 1 - \theta$.
    Then
    %
    \begin{equation}
        \Phi(\theta A + \overline{\theta}B) - \theta \Phi(A) - \overline{\theta} \Phi(B) = \frac{\theta \overline{\theta}}{\theta \Tr A + \overline{\theta} \Tr B} \left( \sqrt{\frac{\Tr B}{\Tr A}} A - \sqrt{\frac{\Tr A}{\Tr B}} B\right) \succeq 0.
    \end{equation}
    %
    This establishes the concavity claim \cref{eq:concavity}.

    \emph{\textbf{Monotonicity.}} 
    Suppose $0 \preceq A \preceq B$.
    Observe the map $\Phi$ is positive homogeneous: For $\theta > 0$, $\Phi(\theta A) = \theta \Phi(A)$.
    Thus,
    %
    \begin{equation}
        \Phi(B) = \Phi(A + (B-A)) = 2 \Phi \left(\frac{A + (B-A)}{2}\right).
    \end{equation}
    % 
    Invoke concavity \cref{eq:concavity} to obtain
    %
    \begin{equation}
        \Phi(B) = 2 \Phi \left(\frac{A + (B-A)}{2}\right) \succeq \Phi(A) + \Phi(B-A) \succeq \Phi(A).
    \end{equation}
    %
    In the last inequality, we use the positivity $\Phi(B-A) \succeq 0$.
    This completes the proof of \cref{eq:positive_monotone}.
\end{proof}

We can use the map $\Phi$ to bound the residual integral operator $T_i$:

\begin{proposition} \label{prop:rpc_composition}
    For $i\ge 0$, we have the bound
    \begin{equation} \label{eq:rpc_iterate}
        \expect[T_i] \preceq \Phi^{i}(T),
    \end{equation}
    %
    where $\Phi^{i}$ denotes the $i$-fold composition of $\Phi$.
\end{proposition}

\begin{proof}
The $i$th step of \RPCholesky applies one step of \RPCholesky to $k_{i-1}$.
Thus,
%
\begin{equation}
    \expect[T_i] = \expect[\expect[T_i \mid s_1,\ldots,s_{i-1}]] = \expect[\Phi(T_{i-1})].
\end{equation}
%
Now, we can apply concavity \cref{eq:concavity} to obtain
%
\begin{equation}
    \expect[T_i] = \expect[\Phi(T_{i-1})] \preceq \Phi(\expect [T_{i-1}]).
\end{equation}
%
Applying this result twice and using monotonicity \cref{eq:positive_monotone} yields:
%
\begin{equation}
    \expect[T_i] \preceq \Phi(\expect [T_{i-1}]) \text{ and } \expect[T_{i-1}] \preceq \Phi(\expect [T_{i-2}]) \implies \expect[T_i] \preceq \Phi(\Phi (\expect[T_{i-2}])).
\end{equation}
%
Iterating this argument, we obtain the desired conclusion \cref{eq:rpc_iterate}. 
%
\end{proof}

We now have the tools to prove \cref{thm:rpc}.
We actually prove the following slight refinement:

\begin{theorem} \label{thm:rpc_refined}
    Let $\set{S} = \{s_1,\ldots,s_n\}$ be generated by the \RPCholesky sampling algorithm.
    For any function $g \in \Ltwo(\mu)$, nonnegative integer $r \ge 0$, and real numbers $a,\varepsilon > 0$, we have
    %
    \begin{equation}
        \expect \Err^2(\set{S};g) \le \norm{g}_{\Ltwo(\mu)}^2 \cdot \left(a + \varepsilon\sum_{i=r+1}^\infty \lambda_i\right)
    \end{equation}
    %
    provided
    %
    \begin{equation} \label{eq:n_condition_refined}
        n \ge r \log\left( \frac{\lambda_1}{a} \right) + \frac{1}{\varepsilon}.
    \end{equation}
\end{theorem}

\Cref{thm:rpc} immediately follows from \cref{thm:rpc_refined} by taking $\varepsilon = \delta/2$ and $a = (\delta/2)\sum_{i=r+1}^\infty \lambda_i$.
We now prove \cref{thm:rpc_refined}.

\begin{proof}[Proof of \cref{thm:rpc_refined}]
    In the standing notation, $T - T_{\set{S}}$ corresponds to $T_n$. Thus, by \cref{prop:rpc_composition}, we have
    %
    \begin{equation}
        \expect \langle g, (T - T_{\set{S}}) g\rangle_{\Ltwo(\mu)} = \langle g, \expect[T_n]g\rangle_{\Ltwo(\mu)} \le \langle g, \Phi^{n}(T)g\rangle_{\Ltwo(\mu)} \le \norm{g}_{\Ltwo(\mu)}^2 \cdot \lambda_{\rm max}(\Phi^n(T)).
    \end{equation}
    %
    Thus, it is sufficient to show that 
    %
    \begin{equation} \label{eq:biggest_eigenvalue_condition}
        \lambda_{\rm max}(\Phi^n(T)) \le a + \varepsilon \sum_{i=r+1}^\infty \lambda_i
    \end{equation}
    %
    holds under condition \cref{eq:n_condition_refined}.

    Using the definition \cref{eq:Phi} of $\Phi$, we see that the eigenvalues $\lambda^{(i)}_1 \ge \lambda^{(i)}_2 \ge \cdots $ of $\Phi^i(T)$ are given by the following recurrence
    %
    \begin{equation} \label{eq:eigenvalue_recurrence}
        \lambda^{(i)}_j = \lambda^{(i-1)}_j - \frac{\left(\lambda^{(i-1)}_j\right)^2}{\sum_{\ell=1}^\infty\lambda^{(i-1)}_\ell} \quad \text{for } i,j = 1,2,\ldots
    \end{equation}
    %
    with initial condition $\lambda^{(0)}_j = \lambda_j$ for all $j$.
    From this formula, we immediately conclude that each $\lambda_j^{(i)}$ is nonnegative and nonincreasing in $i$.
    In addition, the ordinal ranking $\lambda_1^{(i)} \ge \lambda_2^{(i)} \ge \cdots $ is preserved for each $i$. 
    Indeed, if $\lambda_{j+1}^{(i-1)}-\lambda_j^{(i-1)} \ge 0$, then
    %
    \begin{align}
        \lambda^{(i)}_{j+1} - \lambda^{(i)}_j &= \lambda^{(i-1)}_{j+1} - \lambda^{(i-1)}_{j} - \frac{\left(\lambda^{(i-1)}_{j+1}\right)^2 - \left(\lambda^{(i-1)}_j\right)^2}{\sum_{\ell=1}^\infty\lambda^{(i-1)}_\ell} \\
        &= \left( \lambda^{(i-1)}_{j+1} - \lambda^{(i-1)}_{j} \right) \left(1 - \frac{\lambda^{(i-1)}_{j+1}+\lambda^{(i-1)}_{j}}{\sum_{\ell=1}^\infty\lambda^{(i-1)}_\ell}\right) \ge 0,
    \end{align}
    and the claim follows by induction on $i$. 
    To bound $\lambda_{\rm max}(\Phi^i(T)) = \lambda_1^{(i)}$, we bound the sum of eigenvalues appearing in the recurrence \cref{eq:eigenvalue_recurrence}:
    %
    \begin{equation} \label{eq:trace_bound}
        \sum_{\ell=1}^\infty\lambda^{(i-1)}_\ell = \lambda^{(i-1)}_1 + \cdots + \lambda^{(i-1)}_r + \sum_{\ell=r+1}^\infty \lambda^{(i-1)}_\ell \le r \lambda^{(i-1)}_1 + \sum_{\ell=r+1}^\infty \lambda_\ell.
    \end{equation}
    %
    To bound the first $r$ terms, we used the ordering of the eigenvalues $\lambda_1^{(i-1)} \ge \cdots \ge \lambda_r^{(i-1)}$.
    To bound the tail, we used the fact that for each $\ell\in\{r+1,r+2,\ldots\}$, the eigenvalue $\lambda^{(i-1)}_\ell$ is nonincreasing in $i$ and thus bounded by $\lambda^{(0)}_\ell = \lambda_\ell$.
    Substituting \cref{eq:trace_bound} in \cref{eq:eigenvalue_recurrence}, we obtain
    %
    \begin{equation} \label{eq:recurrence_bound}
        \lambda^{(i)}_1 \le \lambda^{(i-1)}_1 - \frac{\left(\lambda^{(i-1)}_1\right)^2}{r\lambda_1^{(i-1)} + \sum_{\ell=r+1}^\infty \lambda_\ell} \quad \text{for } i=1,2,\ldots.
    \end{equation}

    To bound the solution to the recurrence \cref{eq:recurrence_bound}, we pass to continuous time.
    Specifically, for any $t\in \mathbb{Z}_+$, $\lambda_1^{(t)}$ is bounded by the process $x(t)$ solving the initial value problem
    %
    \begin{equation}
        \frac{\d}{\d t} x(t) = - \frac{x(t)^2}{rx(t) + \sum_{\ell=r+1}^\infty \lambda_\ell}, \quad x(0) = \lambda_1.
    \end{equation}
    %
    The bound $\lambda^{(t)}_1 \le x(t)$ holds for all $t \in \mathbb{Z}_+$ because $x \mapsto -x^2/(rx + \sum_{\ell=r+1}^\infty \lambda_\ell)$ is decreasing for $x \in (0,\infty)$.
    We are interested in the time $t = t_\star$ at which $x(t_\star) = a + \varepsilon\sum_{\ell=r+1}^\infty \lambda_\ell$, which we obtain by separation of variables
    %
    \begin{align}
        t_\star &= \int_{\lambda_1}^{a + \varepsilon\sum_{\ell=r+1}^\infty \lambda_\ell} - \frac{rx + \sum_{\ell=r+1}^\infty \lambda_\ell}{x^2} \, \d x = \left[-r \log x  + \frac{\sum_{\ell=r+1}^\infty \lambda_\ell}{x}\right]_{x=\lambda_1}^{x=a + \varepsilon\sum_{\ell=r+1}^\infty \lambda_\ell} \\
        &= r \log \frac{\lambda_1}{a+\varepsilon\sum_{\ell=r+1}^\infty \lambda_\ell} + \frac{\sum_{\ell=r+1}\lambda_\ell}{a + \varepsilon\sum_{\ell=r+1}\lambda_\ell} - \frac{\sum_{\ell=r+1}\lambda_\ell}{\lambda_1} \le r \log \frac{\lambda_1}{a} + \frac{1}{\varepsilon}.
    \end{align}
    %
    Since $x$ is decreasing, the following holds for $n$ satisfying \cref{eq:n_condition}: 
    %
    \begin{equation}
        \lambda_{\rm max}(\Phi^n(T)) = \lambda^{(n)}_1 \le x(n) \le x(t_\star) = a + \varepsilon\sum_{\ell=r+1}^\infty \lambda_\ell.
    \end{equation}
    %
    This shows \cref{eq:biggest_eigenvalue_condition}, completing the proof.
\end{proof}

\section{Proof of \cref{thm:rejection}}

To see why algorithm \ref{alg:rpc_rejection} produces samples from the \RPCholesky sampling distribution, observe that at each effective iteration $i$ the matrix $\mat{L}(1:i-1,1:i-1)$ is the Cholesky factor of $k(\set{S},\set{S})$. Therefore, $d$ is equal to the residual kernel $k^{\rm res}_{\set{S}} \coloneqq k-k_\set{S}$ evaluated at $s_i$
%
\begin{equation}
    d = k^{\rm res}_{\set{S}}(s_i,s_i).
\end{equation}

At each effective iteration $i$, the candidate point $s_i$ is sampled from $k(x,x)\, \d\mu(x)$ and accepted with probability $k^{\rm res}_{\set{S}}(s_i,s_i) / k(s_i,s_i)$.
Therefore, conditional on acceptance and previous steps of the algorithm, $s_i$ is sampled from the probability measure
%
\begin{equation}
    \frac{1}{\prob\{\text{Acceptance} \mid \set{S}\}} \cdot k(x,x)\, \d\mu(x) \cdot \frac{k^{\rm res}_{\set{S}}(x,x)}{k(x,x)} = \frac{k^{\rm res}_{\set{S}}(x,x) \, \d\mu(x)}{\prob\{\text{Acceptance} \mid \set{S}\}}.
\end{equation}
%
Since this is a probability measure which must integrate to one, the probability of acceptance is 
%
\begin{equation}
    \prob\{\text{Acceptance} \mid \set{S}\} = \int_{\set{X}} k^{\rm res}_{\set{S}}(x,x) \, \d\mu(x).
\end{equation}
%
Therefore, conditional on acceptance and previous steps of the algorithm, $s_i$ is sampled from the probability measure $k^{\rm res}_{\set{S}}(x,x) \, \d\mu(x)/\int_{\set{X}} k^{\rm res}_{\set{S}}(x,x) \, \d\mu(x).$
This corresponds exactly with the \RPCholesky sampling procedure described in algorithm \ref{alg:rpc_unoptimized} (see \cref{prop:rpc_equivalence} below).

Now, we bound the runtime.
For each iteration $i$, the expected acceptance probability is bounded from below by the smallest possible trace error achievable by a rank $(i-1)$ approximation
%
\begin{equation}
    \prob\{\text{Acceptance}\} = \expect[\prob\{\text{Acceptance} \mid \set{S}\}] = \expect\left[\int_{\set{X}} k^{\rm res}_{\set{S}}(x,x) \, \d\mu(x)\right] \ge \eta_{i-1}.
\end{equation}
%
Therefore, the expected number of rejections at step $i$ is bounded by the expectation of a geometric random variable with parameter $\eta_{i-1}$, i.e., by $\eta_{i-1}^{-1}$. Since each loop iteration takes $\order(i^2)$ operations (attributable to line 4) and at most $\eta_{i-1}^{-1}$ expected iterations are required to accept at step $i$, the total expected runtime is at most $\order(\sum_{i=1}^n i^2/\eta_{i-1}) \le \order(n^3/\eta_{n-1})$. \hfill $\square$

\section{Details for numerical experiments} \label{sec:numerical_experiment_details}

In this section, we provide additional details about our numerical experiments.

\paragraph{Continuous volume sampling.} 
We are unaware of a provably correct $n$-DPP/continuous volume distribution sampler that is sufficiently efficient to facillitate numerical testing and comparisons.
The best-known sampler \cite{RO19} for the continuous setting is inexact and requires $\order(n^5 \log n)$ MCMC steps, which would be prohibitive for us.
To produce samples from the continuous volume sampling distribution for this paper, we use a continuous analog of the MCMC algorithm \cite{AOR16}.
Given a set $\set{S} = \{s_1,\ldots,s_n\}$ and assuming $\mu$ is a probability measure, one MCMC step proceeds as follows:
%
\begin{enumerate}
    \item Sample $s_{\rm new} \sim \mu$ and $s_{\rm old} \sim \Unif \set{S}$.
    \item Define $\set{S}' = \set{S} \cup \{s_{\rm new}\} \setminus \{s_{\rm old}\}$.
    \item With probability 
    %
    \begin{equation}
        \frac{1}{2} \min \left\{ 1, \frac{\det k(\set{S}',\set{S}')}{\det k(\set{S},\set{S})} \right\},
    \end{equation}
    %
    accept and set $\set{S} \leftarrow \set{S}'$.
    Otherwise, leave $\set{S}$ unchanged.
\end{enumerate}
%
In our experiments, we initialize with $s_1,\ldots,s_n$ drawn iid from $\mu$ and run for $10n$ MCMC steps.

Running for just $10n$ MCMC steps is aggressive, even in the finite setting.
Indeed, the theoretical analysis of \cite[Thm.~2, Eq.~(1.2)]{AOR16}
for $\mu = \Unif\{1,2,\ldots,|\set{X}|\}$, proves the sampler mixes to stationarity in at most $\tilde{\order}(n^2|\set{X}|)$ MCMC steps when properly initialized.
Even with just $10n$ MCMC steps, continuous volume sampling is dramatically more expensive than \RPCholesky in our experiments for $n\ge 64$.

Finally, we notice that the performance of continuous volume sampling degrades to that of iid sampling for sufficiently large $n$.
We believe that the cause for this are numerical issues in evaluating the determinants of the kernel matrices $k(\set{S},\set{S})$.
This suggests that, in addition to being faster, \RPCholesky may be more numerically robust than continuous volume sampling.

\paragraph{Thinning and positively weighted kernel quadrature.} Our kernel quadrature experiments with thinning and PWKQ use code from the authors of \cite{PWKQ} available online at \url{https://github.com/satoshi-hayakawa/kernel-quadrature}, which uses the \texttt{goodpoints} package (\url{https://github.com/microsoft/goodpoints}) by the authors of \cite{DM21,DM22,SDM22} to perform thinning.
Following the experimental setup of \cite{PWKQ}, we use $\mathfrak{g} = \min \{4,\log_2(n)\}$ for the \textsc{Compress}\texttt{++} algorithm.

\paragraph{Miscellaneous.}
\ifneurips{Our code is included in the supplementary materials and can be found online at}\else{Our code is available online at}\fi
%
\boxedtext{\url{https://github.com/eepperly/RPCholesky-Kernel-Quadrature}.}
%
To evaluate exact integrals
%
\begin{equation}
    Tg(x) = \int_{\set{X}} k(x,y) g(y) \, \d\mu(y)
\end{equation}
%
for \cref{fig:moon}, we use the 2D functionality in ChebFun \cite{DHT14}.
To compute the optimal weights \cref{eq:weights}, we add a small multiple of the identity to regularize the system:
%
\begin{equation}
    \vec{w}_{\star, \, {\rm reg}} = (k(\set{S},\set{S}) + 10\varepsilon_{\rm mach} \operatorname{trace} (k(\set{S},\set{S}))\cdot  \mathbf{I})^{-1} Tg(\set{S}).
\end{equation}
%
Here, $\varepsilon_{\rm mach} = 2^{-52}$ is the double precision machine epsilon.

\section{Randomly pivoted Cholesky and Nystr\"om approximation}

In this section, we describe the connection between the \RPCholesky algorithm and Nystr\"om approximation.
We hope that this explanation provides context that will be helpful in understanding the \RPCholesky algorithm and the proofs of \cref{thm:rpc,thm:rejection}.

To make the following discussion more clear, we begin by rewriting the basic \RPCholesky algorithm (see algorithm \ref{alg:rpc_unoptimized}) in \cref{alg:rpc_for_understanding} to avoid overwriting the kernel $k$ at each step:

\begin{algorithm}[ht]
	\caption{\RPCholesky: unoptimized implementation without overwriting \label{alg:rpc_for_understanding}}
	\begin{algorithmic}[1]
		\Require Kernel $k : \set{X} \times \set{X} \to \real$ and number of quadrature points $n$
		\Ensure Quadrature points $s_1,\ldots,s_n$ and Nystr\"om approximation $\hat{k}_n = k_{\{s_1,\ldots,s_n\}}$ to $k$
        \State Initialize $k_0 \leftarrow k$, $\hat{k}_0 \leftarrow 0$
		\For{$i = 1,2,\ldots,n$}
            \State Sample $s_i$ from the probability measure $k(x,x) \, \d\mu(x) / \int_{\set{X}} k(x,x) \, \d\mu(x)$
            \State Update Nystr\"om approximation $\hat{k}_i \leftarrow \hat{k}_{i-1} + k_{i-1}(\cdot,s_i) k_{i-1}(s_i,s_i)^{-1} k_{i-1}(s_i,\cdot)$
            \State Update kernel $k_i \leftarrow k_{i-1} - k_{i-1}(\cdot,s_i) k_{i-1}(s_i,s_i)^{-1} k_{i-1}(s_i,\cdot)$
        \EndFor
	\end{algorithmic}
\end{algorithm}

From this rewriting, we see that in addition to selecting quadrature nodes $s_1,\ldots,s_n$, the \RPCholesky algorithm builds an approximation $\hat{k}_n$ to the kernel $k$.
Indeed, one can verify that the approximation $\hat{k}_n$ produced by this algorithm is precisely the Nystr\"om approximation (\cref{def:nystrom}).

\begin{proposition} \label{prop:rpc_equivalence}
    The output $\hat{k}_n$ of \cref{alg:rpc_for_understanding} is equal to the Nystr\"om approximation $k_{\{s_1,\ldots,s_n\}}$.
\end{proposition}

\begin{proof}
    Proceed by induction on $n$.
    The base case $n = 0$ is immediate.
    Fix $n$ and let $\set{S} \coloneqq \{s_1,\ldots,s_n\}$, $s' \coloneqq s_{n+1}$, and $\set{S}' \coloneqq \{s_1,\ldots,s_n,s'\}$.
    Write the Nystr\"om approximation $k_{\set{S}'}$ as
    %
    \begin{equation}
        k_{\set{S}'} = \onebytwo{k(\cdot,\set{S})}{k(\cdot,s')} \twobytwo{k(\set{S},\set{S})}{k(\set{S},s')}{k(s',\set{S})}{k(s',s')}^{-1} \twobyone{k(\set{S},\cdot)}{k(s',\cdot)}
    \end{equation}
    %
    Compute a block $\mat{L}\mat{D}\mat{L}^\top$ factorization of the matrix in the middle of this expression:
    %
    \begin{equation}
        \twobytwo{k(\set{S},\set{S})}{k(\set{S},s')}{k(s',\set{S})}{k(s',s')} = \twobytwo{\Id}{\vec{0}}{k(s',\set{S})k(\set{S},\set{S})^{-1}}{1}\twobytwo{k(\set{S},\set{S})}{\vec{0}}{\vec{0}^\top}{k_n(s',s')}\twobytwo{\Id}{k(\set{S},\set{S})^{-1}k(\set{S},s')}{\vec{0}^\top}{1}.
    \end{equation}
    %
    Here, we've used the induction hypothesis
    %
    \begin{equation} \label{eq:ih}
        k_n(s',s') = k(s',s') - k_{\set{S}}(s',s') = k(s',s') - k(s',\set{S})k(\set{S},\set{S})^{-1}k(\set{S},s').
    \end{equation}
    %
    Thus,
    %
    \begin{align}
        k_{\set{S}'} &= \onebytwo{k(\cdot,\set{S})}{k(\cdot,s')} \twobytwo{k(\set{S},\set{S})}{k(\set{S},s')}{k(s',\set{S})}{k(s',s')}^{-1} \twobyone{k(\set{S},\cdot)}{k(s',\cdot)} \\
        &= \onebytwo{k(\cdot,\set{S})}{k(\cdot,s')}\twobytwo{\Id}{-k(\set{S},\set{S})^{-1}k(\set{S},s')}{\vec{0}^\top}{1}\twobytwo{k(\set{S},\set{S})^{-1}}{\vec{0}}{\vec{0}^\top}{k_n(s',s')^{-1}} \\
        &\qquad \cdot 
        \twobytwo{\Id}{\vec{0}}{-k(s',\set{S})k(\set{S},\set{S})^{-1}}{1}
        \twobyone{k(\set{S},\cdot)}{k(s',\cdot)} \\
        &= \onebytwo{k(\cdot,\set{S})}{k_n(\cdot,s')}\twobytwo{k(\set{S},\set{S})^{-1}}{\vec{0}}{\vec{0}^\top}{k_n(s',s')^{-1}} \twobyone{k(\set{S},\cdot)}{k_n(\cdot,s')}  \label{eq:use_ih} \\
        &= \hat{k}_n + k_n(\cdot,s') k_n(s',s')^{-1}k_n(s',\cdot) = \hat{k}_{n+1}.
    \end{align}
    %
    For \cref{eq:use_ih}, 
    we used the induction hypothesis \cref{eq:ih} again.
    Having established that $k_{\set{S}'} = k_{n+1}$, the proposition is proven.
\end{proof}
\begin{thebibliography}{10}

\bibitem{ALV22}
Nima Anari, Yang~P. Liu, and Thuy-Duong Vuong.
\newblock Optimal sublinear sampling of spanning trees and determinantal point
  processes via average-case entropic independence.
\newblock In {\em 63rd Annual Symposium on Foundations of Computer Science},
  pages 123--134. IEEE, October 2022.

\bibitem{AOR16}
Nima Anari, Shayan Oveis~Gharan, and Alireza Rezaei.
\newblock Monte {Carlo} {Markov} chain algorithms for sampling strongly
  {Rayleigh} distributions and determinantal point processes.
\newblock In {\em Proceedings of the 29th Conference on Learning Theory}, pages
  103--115. PMLR, June 2016.

\bibitem{bach_lever}
Francis Bach.
\newblock On the equivalence between kernel quadrature rules and random feature
  expansions.
\newblock {\em Journal of Machine Learning Research}, 18(1):714–751, January
  2017.

\bibitem{bach_herd}
Francis Bach, Simon Lacoste-Julien, and Guillaume Obozinski.
\newblock On the equivalence between herding and conditional gradient
  algorithms.
\newblock {\em Proceedings of the 29th International Conference on Machine
  Learning}, March 2012.

\bibitem{Bardenet2016MonteCW}
R{\'e}mi Bardenet and Adrien Hardy.
\newblock Monte {Carlo} with determinantal point processes.
\newblock {\em The Annals of Applied Probability}, 30(1):368--417, February
  2020.

\bibitem{Bel21}
Ayoub Belhadji.
\newblock An analysis of {E}rmakov--{Z}olotukhin quadrature using kernels.
\newblock In {\em Advances in Neural Information Processing Systems},
  volume~34, pages 27278--27289. Curran Associates, Inc., 2021.

\bibitem{BBC19a}
Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais.
\newblock Kernel quadrature with {DPPs}.
\newblock In {\em Advances in {Neural} {Information} {Processing} {Systems}},
  volume~32. Curran Associates, Inc., 2019.

\bibitem{BBC20}
Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais.
\newblock Kernel interpolation with continuous volume sampling.
\newblock In {\em Proceedings of the 37th {International} {Conference} on
  {Machine} {Learning}}, pages 725--735. PMLR, November 2020.

\bibitem{CDV20a}
Daniele Calandriello, Michal Derezinski, and Michal Valko.
\newblock Sampling from a $k$-{DPP} without looking at all items.
\newblock In {\em Advances in {Neural} {Information} {Processing} {Systems}},
  volume~33, pages 6889--6899. Curran Associates, Inc., 2020.

\bibitem{CETW23}
Yifan Chen, Ethan~N. Epperly, Joel~A. Tropp, and Robert~J. Webber.
\newblock Randomly pivoted {Cholesky}: {Practical} approximation of a kernel
  matrix with few entry evaluations, February 2023.
\newblock arXiv:2207.06503.

\bibitem{CWS10}
Yutian Chen, Max Welling, and Alex Smola.
\newblock Super-samples from kernel herding.
\newblock In {\em Proceedings of the Twenty-Sixth Conference on Uncertainty in
  Artificial Intelligence}, page 109–116. AUAI Press, 2010.

\bibitem{DRVW06}
Amit Deshpande, Luis Rademacher, Santosh Vempala, and Grant Wang.
\newblock Matrix approximation and projective clustering via volume sampling.
\newblock In {\em Proceedings of the seventeenth annual {ACM}-{SIAM} symposium
  on Discrete algorithm}, pages 1117--1126. ACM, 2006.

\bibitem{DV06}
Amit Deshpande and Santosh Vempala.
\newblock Adaptive sampling and fast low-rank matrix approximation.
\newblock In Josep D{\'i}az, Klaus Jansen, Jos{\'e} D.~P. Rolim, and Uri Zwick,
  editors, {\em Approximation, Randomization, and Combinatorial Optimization.
  Algorithms and Techniques}, pages 292--303. Springer Berlin Heidelberg, 2006.

\bibitem{dick_pillichshammer_2010}
Josef Dick and Friedrich Pillichshammer.
\newblock {\em Quasi-Monte Carlo integration, discrepancy and reproducing
  kernel Hilbert spaces}, page 16–45.
\newblock Cambridge University Press, 2010.

\bibitem{DHT14}
Tobin~A. Driscoll, Nicholas Hale, and Lloyd~N. Trefethen.
\newblock Chebfun guide, 2014.

\bibitem{Dun80}
J.~C. Dunn.
\newblock Convergence rates for conditional gradient sequences generated by
  implicit step length rules.
\newblock {\em SIAM Journal on Control and Optimization}, 18(5):473--487, 1980.

\bibitem{DM21}
Raaz Dwivedi and Lester Mackey.
\newblock Kernel thinning.
\newblock In {\em Proceedings of Thirty Fourth Conference on Learning Theory},
  pages 1753--1753. PMLR, August 2021.

\bibitem{DM22}
Raaz Dwivedi and Lester Mackey.
\newblock Generalized kernel thinning.
\newblock In {\em Tenth International Conference on Learning Representations},
  April 2022.

\bibitem{GBV19}
Guillaume Gautier, Rémi Bardenet, and Michal Valko.
\newblock On two ways to use determinantal point processes for {Monte} {Carlo}
  integration.
\newblock In {\em Advances in {Neural} {Information} {Processing} {Systems}},
  volume~32. Curran Associates, Inc., 2019.

\bibitem{PWKQ}
Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons.
\newblock Positively weighted kernel quadrature via subsampling.
\newblock In {\em Advances in Neural Information Processing Systems},
  volume~35, pages 6886--6900. Curran Associates, Inc., 2022.

\bibitem{hayakawa2023samplingbased}
Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons.
\newblock Sampling-based {Nystr\"om} approximation and kernel quadrature,
  January 2023.
\newblock arXiv:2301.09517.

\bibitem{FD12}
Ferenc Husz\'{a}r and David Duvenaud.
\newblock Optimally-weighted herding is {B}ayesian quadrature.
\newblock In {\em Proceedings of the Twenty-Eighth Conference on Uncertainty in
  Artificial Intelligence}, page 377–386. AUAI Press, 2012.

\bibitem{prob_ode}
Hans Kersting and Philipp Hennig.
\newblock Active uncertainty calibration in {B}ayesian {ODE} solvers.
\newblock In {\em Proceedings of the Thirty-Second Conference on Uncertainty in
  Artificial Intelligence}, page 309–318. AUAI Press, 2016.

\bibitem{KT11}
Alex Kulesza and Ben Taskar.
\newblock $k$-{DPPs}: Fixed-size determinantal point processes.
\newblock In {\em Proceedings of the 28th International Conference on Machine
  Learning}, pages 1193--1200. Omnipress, January 2011.

\bibitem{LL12}
C.~Litterer and T.~Lyons.
\newblock High order recombination and an application to cubature on {Wiener}
  space.
\newblock {\em The Annals of Applied Probability}, 22(4):1301--1327, August
  2012.

\bibitem{MBML}
Kevin~P. Murphy.
\newblock {\em Machine Learning: A Probabilistic Perspective}.
\newblock The MIT Press, 2012.

\bibitem{Nys30}
E.~J. Nyström.
\newblock Über {Die} {Praktische} {Auflösung} von {Integralgleichungen} mit
  {Anwendungen} auf {Randwertaufgaben}.
\newblock {\em Acta Mathematica}, 54(0):185--204, 1930.

\bibitem{rob_control}
Supratik Paul, Konstantinos Chatzilygeroudis, Kamil Ciosek, Jean-Baptiste
  Mouret, Michael~A. Osborne, and Shimon Whiteson.
\newblock Alternating optimisation and quadrature for robust control.
\newblock In {\em Proceedings of the Thirty-Second {AAAI} Conference on
  Artificial Intelligence, (AAAI-18), the 30th innovative Applications of
  Artificial Intelligence (IAAI-18), and the 8th {AAAI} Symposium on
  Educational Advances in Artificial Intelligence (EAAI-18)}, pages 3925--3933.
  AAAI Press, February 2018.

\bibitem{RO19}
Alireza Rezaei and Shayan Oveis~Gharan.
\newblock A polynomial time {MCMC} method for sampling from continuous
  determinantal point processes.
\newblock In {\em Proceedings of the 36th {International} {Conference} on
  {Machine} {Learning}}, pages 5438--5447. PMLR, May 2019.

\bibitem{Bayesian_choice}
Christian~P. Robert.
\newblock {\em The {Bayesian} {Choice}: {From} {Decision}-{Theoretic}
  {Foundations} to {Computational} {Implementation}}.
\newblock Springer {Texts} in {Statistics}. Springer, second edition, 2007.

\bibitem{MCbound}
Christian~P. Robert and George Casella.
\newblock {\em Monte {Carlo} {Statistical} {Methods}}.
\newblock Springer {Texts} in {Statistics}. Springer, second edition, 2004.

\bibitem{SDM22}
Abhishek Shetty, Raaz Dwivedi, and Lester Mackey.
\newblock Distribution compression in near-linear time.
\newblock In {\em Tenth International Conference on Learning Representations},
  April 2022.

\bibitem{SS12}
Ingo Steinwart and Clint Scovel.
\newblock Mercer’s theorem on general domains: {On} the interaction between
  measures, kernels, and {RKHSs}.
\newblock {\em Constructive Approximation}, 35(3):363--417, June 2012.

\bibitem{TsujiTanakaPokutta2022}
Kazuma Tsuji, Ken'ichiro Tanaka, and Sebastian Pokutta.
\newblock Pairwise conditional gradients without swap steps and sparser kernel
  herding.
\newblock In {\em Proceedings of the International Conference on Machine
  Learning}, 2022.

\bibitem{WS00}
Christopher K.~I. Williams and Matthias Seeger.
\newblock Using the {Nystr\"om} method to speed up kernel machines.
\newblock In {\em Proceedings of the 13th {International} {Conference} on
  {Neural} {Information} {Processing} {Systems}}, pages 661--667. MIT Press,
  January 2000.

\end{thebibliography}