Fast Robust Kernel Regression through Sign Gradient Descent with Early Stopping
Extracted source text · arXiv 2306.16838
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{arxiv}
\usepackage{bm}
\usepackage{url}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts} % blackboard math symbols
\usepackage{cancel}
\usepackage{graphicx}
\usepackage{natbib}
\usepackage{multirow}
\usepackage{makecell}
\usepackage{tablefootnote}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{hyperref} % hyperlinks
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\I}{\bm{I}}
\newcommand{\nv}{\bm{0}}
\newcommand{\Oo}{\mathcal{O}}
\newcommand{\Tr}{\text{Tr}}
\newcommand{\Cov}{\text{Cov}}
\newcommand{\Var}{\text{Var}}
\newcommand{\sgn}{\text{\normalfont sign}}
\newcommand{\epsv}{\bm{\varepsilon}}
\newcommand{\ktwobar}{\overline{k^*_2}}
\newcommand{\smin}{s_{\min}}
\newcommand{\sminbar}{\overline{s_{\min}}}
\newcommand{\invsigbar}{\overline{\sigma^{-1}}}
\newcommand{\smax}{s_{\max}}
\newcommand{\ns}{n^*}
\newcommand{\vv}{\bm{v}}
\newcommand{\xv}{\bm{x}}
\newcommand{\xsv}{\bm{x^*}}
\newcommand{\xpv}{\bm{x'}}
\newcommand{\xvi}{\bm{x_i}}
\newcommand{\xvj}{\bm{x_j}}
\newcommand{\xsvi}{\bm{x^*_i}}
\newcommand{\xsvj}{\bm{x^*_j}}
\newcommand{\xsvn}{\bm{x^*_0}}
\newcommand{\xvm}{\bm{x_m}}
\newcommand{\X}{\bm{X}}
\newcommand{\Xs}{\bm{X^*}}
\newcommand{\yv}{\bm{y}}
\newcommand{\ytv}{\bm{\tilde{y}}}
\newcommand{\ypv}{\bm{y^{^+}}}
\newcommand{\Ph}{\bm{\Phi}}
\newcommand{\Phs}{\bm{\Phi^*}}
\newcommand{\phiv}{\bm{\varphi}}
\newcommand{\K}{\bm{K}}
\newcommand{\kv}{\bm{k}}
\newcommand{\ksv}{\bm{k^*}}
\newcommand{\ksvt}{{\bm{k^*}}^\top}
\newcommand{\Ks}{\bm{K^*}}
\newcommand{\Kss}{\bm{K^{**}}}
\newcommand{\Ih}{\bm{\hat{I}}}
\newcommand{\Sigmaalpha}{\bm{\Sigma_\alpha}}
\newcommand{\thetahv}{\bm{\hat{\theta}}}
\newcommand{\thetanv}{\bm{\theta_0}}
\newcommand{\alphav}{\bm{\alpha}}
\newcommand{\alphahv}{\bm{\hat{\alpha}}}
\newcommand{\dalphahv}{\bm{\Delta\hat{\alpha}}}
\newcommand{\alphanv}{\bm{\alpha_0}}
\newcommand{\alphahnv}{\bm{\hat{\alpha}}^0}
\newcommand{\alphahfv}{\bm{\hat{\alpha}}_\text{\normalfont KGF}}
\newcommand{\alphahri}{\hat{\alpha}_i^\text{\normalfont KRR}}
\newcommand{\alphahfi}{\hat{\alpha}_i^\text{\normalfont KGF}}
\newcommand{\alphahni}{\hat{\alpha}_i^0}
\newcommand{\alphahrv}{\bm{\hat{\alpha}}_\text{\normalfont KRR}}
\newcommand{\alphahsgfi}{\hat{\alpha}^\text{SGF}_i}
\newcommand{\alphahsgfv}{\bm{\hat{\alpha}}^\text{SGF}}
\newcommand{\alphahinfi}{\hat{\alpha}^\infty_i}
\newcommand{\alphahinfv}{\bm{\hat{\alpha}}^\infty}
\newcommand{\betav}{\bm{\beta}}
\newcommand{\betahv}{\bm{\hat{\beta}}}
\newcommand{\betahsgfi}{\hat{\beta}^\text{SGF}_i}
\newcommand{\betahsgfv}{\bm{\hat{\beta}}^\text{SGF}}
\newcommand{\betahinfi}{\hat{\beta}^\infty_i}
\newcommand{\betahinfv}{\bm{\hat{\beta}}^\infty}
\newcommand{\fh}{\hat{f}}
\newcommand{\fhs}{\hat{f}^*}
\newcommand{\fv}{\bm{f}}
\newcommand{\fhv}{\bm{\hat{f}}}
\newcommand{\fsv}{\bm{f^*}}
\newcommand{\fhsv}{\bm{\hat{f}^*}}
\newcommand{\fnv}{\bm{f_0}}
\newcommand{\fhn}{\hat{f}^0}
\newcommand{\fhnv}{\bm{\hat{f}}^0}
\newcommand{\fhf}{\hat{f}_\text{\normalfont KGF}}
\newcommand{\fhfv}{\bm{\hat{f}}_\text{\normalfont KGF}}
\newcommand{\fhr}{\hat{f}_\text{\normalfont KRR}}
\newcommand{\fhrv}{\bm{\hat{f}}_\text{\normalfont KRR}}
\newcommand{\fpv}{\bm{f^{^+}}}
\newcommand{\fhpv}{\bm{\hat{f}^{^+}}}
\newcommand{\etahv}{\bm{\hat{\eta}}}
\newcommand{\etahnv}{\bm{\hat{\eta}_0}}
\newcommand{\U}{\bm{U}}
\newcommand{\V}{\bm{V}}
\newcommand{\Ss}{\bm{S}}
\newcommand{\A}{\bm{A}}
\newcommand{\B}{\bm{B}}
\newcommand{\C}{\bm{C}}
\newcommand{\D}{\bm{D}}
\newcommand{\DK}{\bm{\Delta_K}}
\newcommand{\Hh}{\mathcal{H}}
\newcommand{\fhp}{\hat{f}^{^+}}
\newcommand{\fp}{f^{^+}}
\newcommand{\yp}{y^{^+}}
\newcommand{\fhpinfv}{\bm{\hat{f}}^{\bm{+}\infty}}
\newcommand{\fhpsgfv}{\bm{\hat{f}}^{\bm{+}\text{SGF}}}
\newcommand{\fhpinfi}{\hat{f}^{+\infty}_i}
\newcommand{\fhpsgfi}{\hat{f}^{+\text{SGF}}_i}
\title{Solving Kernel Ridge Regression with Gradient-Based Optimization Methods}
\author{%
Oskar Allerbo \\
Department of Mathematics\\
KTH Royal Institute of Technology\\
\texttt{oallerbo@kth.se} \\
}
%\date{May 2023}
\begin{document}
\maketitle
\begin{abstract}
Kernel ridge regression, KRR, is a generalization of linear ridge regression that is non-linear in the data, but linear in the parameters.
Here, we introduce an equivalent formulation of the objective function of KRR, opening up both for using penalties other than the ridge penalty and for studying kernel ridge regression from the perspective of gradient descent.
Using a continuous-time perspective, we derive a closed-form solution for solving kernel regression with gradient descent, something we refer to as kernel gradient flow, KGF, and theoretically bound the differences between KRR and KGF, where, for the latter, regularization is obtained through early stopping. We also generalize KRR by replacing the ridge penalty with the $\ell_1$ and $\ell_\infty$ penalties, respectively, and use the fact that analogous to the similarities between KGF and KRR, $\ell_1$ regularization and forward stagewise regression (also known as coordinate descent), and $\ell_\infty$ regularization and sign gradient descent, follow similar solution paths. We can thus alleviate the need for computationally heavy algorithms based on proximal gradient descent. We show theoretically and empirically how the $\ell_1$ and $\ell_\infty$ penalties, and the corresponding gradient-based optimization algorithms, produce sparse and robust kernel regression solutions, respectively.
\end{abstract}
\textbf{Keywords:} Kernel Ridge Regression, Gradient Descent, Forward Stagewise Regression, Sign Gradient Descent, Gradient Flow
\section{Introduction}
Kernel ridge regression, KRR, is a generalization of linear ridge regression that is non-linear in the data, but linear in the parameters. As for linear ridge regression, KRR has a closed-form solution, but at the cost of inverting an $n\times n$ matrix, where $n$ is the number of training observations. The KRR estimate coincides with the posterior mean of kriging, or Gaussian process regression, \citep{krige1951statistical,matheron1963principles} and has successfully been applied within a wide range of applications \citep{zahrt2019prediction, ali2020complete, chen2021optimizing, fan2021well, le2021fingerprinting, safari2021kernel, shahsavar2021experimental, singh2021neural, wu2021increasing, chen2022kernel}.
For linear regression, many alternatives to the ridge penalty have been proposed, including the lasso penalty \citep{tibshirani1996regression}, which is known for creating sparse models. By replacing the ridge penalty of KRR with the lasso penalty \citet{roth2004generalized}, \citet{guigue2005kernel} and \citet{feng2016kernelized} have obtained non-linear regression that is sparse in the observations. However, while the ridge penalty in KRR is formulated such that KRR can alternatively be interpreted as linear ridge regression in feature space, for these formulations the analog interpretation is lost. Rather than using the lasso, or $\ell_1$, penalty, \citet{russu2016secure} and \citet{demontis2017infinity} trained kernelized support vector machines with infinity norm regularization to obtain models that are robust against adversarial data, which in the context of regression would translate to outliers. However, just as for kernelized lasso, by using their formulation, the connection to linear regression in feature space is lost.
An alternative to explicit regularization is to use an iterative optimization algorithm and stop the training before the algorithm converges. A well-known example of this is the connection between the explicitly regularized lasso model and the iterative method forward stagewise regression, which is also known as coordinate descent. The similarities between the solutions are well studied by e.g.\ \citet{efron2004least}, \citet{hastie2007forward} and \citet{tibshirani2015general}. There are also striking similarities between explicitly regularized ridge regression and gradient descent with early stopping \citep{friedman2004gradient,ali2019continuous}. Replacing explicit infinity norm regularization with an iterative optimization method with early stopping does not seem to be as well studied. However, as is shown in this paper, the solutions are similar to those of sign gradient descent with early stopping.
Explicit regularization may be replaced by early stopping for different reasons, e.g.\ to reduce computational time. Even if closed-form solutions do exist for linear and kernel ridge regression, they include the inversion of a matrix, which is an $\Oo(d^3)$ operation for a $\R^{d\times d}$ matrix, something can be avoided when by gradient descent, which has been used in the kernel setting by e.g.\ \citet{yao2007early}, \citet{raskutti2014early} and \citet{ma2017diving}.
Moreover, while for explicit regularization this calculation has to be performed once for every considered value of the regularization strength, for an iterative optimization method, all solutions, for different stopping times, are obtained by running the algorithm to convergence just once.
While the lasso and infinity norm regularized problems are still convex, in general, no closed-form solutions exist, and iterative optimization methods have to be run to convergence once for every candidate value of the regularization strength, something that tends to be computationally heavy.
In this paper, we present an equivalent formulation of KRR, which we use to obtain kernel regression with the $\ell_1$ and $\ell_\infty$ norms, in addition to the default, $\ell_2$, norm. We also use the equivalent formulation to solve kernel regression using the three different gradient-based optimization algorithms gradient descent, coordinate descent (forward stagewise regression), and sign gradient descent. We relate each iterative algorithm to one explicitly norm-based regularized solution and use gradient-based optimization with early stopping to obtain computationally efficient sparse, and robust, kernel regression.
The rest of the paper is structured as follows.
In Section \ref{sec:krr_rev}, we review kernel ridge regression, before introducing our equivalent formulation in Section \ref{sec:equ_krr}. In Section \ref{sec:kgf}, we review kernel gradient descent, introduce kernel gradient flow, and theoretically analyze the differences between KGF and KRR. In Section \ref{sec:l1linf}, we generalize KRR by replacing the $\ell_2$ penalty with the $\ell_1$ and $\ell_\infty$ penalties and introduce the related algorithms kernel coordinate descent and kernel sign gradient descent. Finally, in Section \ref{sec:exps}, we verify our findings with experiments on real and synthetic data.
Our main contributions are listed below.
\begin{itemize}
\item We present an equivalent objective function for KRR and use this formulation to generalize KRR to the $\ell_1$ and $\ell_\infty$ penalties.
\item We derive a closed-form expression for kernel gradient flow, KGF, i.e.\ solving kernel regression with gradient descent for infinitesimal step size, and bound the differences between KGF and KRR in terms of:
\begin{itemize}
\item
Estimation and prediction differences.
\item
Estimation and prediction risks.
\end{itemize}
\item We introduce computationally efficient regularization by replacing $\ell_1$ and $\ell_\infty$ regularization by
\begin{itemize}
\item kernel coordinate descent (or kernel forward stagewise regression)
\item kernel sign gradient descent and
\end{itemize}
with early stopping. We also theoretically show that kernel coordinate descent, and kernel sign gradient descent, with early stopping, correspond to sparse and robust kernel regression, respectively.
\end{itemize}
All proofs are deferred to Appendix \ref{sec:proofs}.
\section{Review of Kernel Ridge Regression}
\label{sec:krr_rev}
For a positive semi-definite kernel function, $k(\xv,\xpv)\in \R$, and $n$ paired observations, $(\xvi,y_i)_{i=1}^n\in\R^p\times\R$, presented in a design matrix, $\X=[\bm{x_1}^\top,\bm{x_2}^\top,\dots \bm{x_n}^\top]^\top\in\R^{n\times p}$, and a response vector, $\yv \in \R^n$, and for a given regularization strength, $\lambda>0$, the objective function of kernel ridge regression, KRR, is given by
\begin{equation}
\label{eq:krr_obj_alpha}
\alphahv=\argmin_{\alphav\in \R^n}\frac12\left\|\yv-\K\alphav\right\|_2^2+\frac\lambda2\|\alphav\|^2_{\K}
\end{equation}
with predictions given by
\begin{equation*}
%\label{eq:krr_s_alpha_f}
\begin{bmatrix}\fhv(\X)\\\fhv(\Xs)\end{bmatrix}=\begin{bmatrix}\K\\\Ks\end{bmatrix}\alphahv.
\end{equation*}
Here, $\fhv(\X)\in\R^{n}$ and $\fhv(\Xs)\in\R^{\ns}$ denote model predictions for the training data, $\X$, and new data, $\Xs=[\bm{x^*_1}{^\top},\bm{x^*_2}{^\top},\dots \bm{x^*_{n^*}}{^\top}]^\top\in\R^{\ns\times p}$, and $\K=\K(\X)\in \R^{n\times n}$ and $\Ks=\Ks(\Xs,\X)\in \R^{\ns\times n}$ denote two kernel matrices defined according to $\K_{ij}=k(\xvi,\xvj)$ and $\Ks_{ij}=k(\xsvi,\xvj)$. The weighted norm, $\|\vv\|_{\A}$, is defined according to $\|\vv\|_{\A}^2=\vv^\top\A\vv$ for any symmetric positive definite matrix $\A$.
The closed-form solution for $\alphahv$ is given by
\begin{equation}
\label{eq:krr_s_alpha}
\alphahv=\left(\K+\lambda\I\right)^{-1}\yv,
\end{equation}
and consequently
\begin{equation}
\label{eq:krr_s_fs}
\begin{bmatrix}\fhv(\X)\\\fhv(\Xs)\end{bmatrix}=\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\K+\lambda\I\right)^{-1}\yv.
\end{equation}
An alternative interpretation of KRR is as linear regression for a non-linear feature expansion of $\X$. According to Mercer's Theorem \citep{mercer1909xvi}, every kernel can be written as the inner product of feature expansions of its two arguments: $k(\xv,\xpv)=\phiv(\xv)^\top\phiv(\xpv)$ for $\phiv(\xv)\in \R^q$. Thus, denoting the feature expansions of the design matrix and the new data with $\Ph=\Ph(\X)\in \R^{n\times q}$ and $\Phs=\Phs(\Xs)\in \R^{\ns\times q}$, the two kernel matrices can be expressed as $\K=\Ph\Ph^\top$ and $\Ks=\Phs\Ph^\top$. Thus, for $\betav=\Ph^\top\alphav$, Equations \ref{eq:krr_obj_alpha} and \ref{eq:krr_s_fs} become
\begin{equation}
\label{eq:krr_obj_beta}
\begin{aligned}
&\betahv=\argmin_{\betav\in \R^q}\frac12\left\|\yv-\Ph\betav\right\|_2^2+\frac\lambda2\|\betav\|^2_2\\
&\begin{bmatrix}\fhv(\X)\\\fhv(\Xs)\end{bmatrix}=\begin{bmatrix}\Ph\\\Phs\end{bmatrix}\betahv,
\end{aligned}
\end{equation}
which is exactly linear ridge regression for the feature expansion of the kernel.
\section{Equivalent Formulations of Kernel Ridge Regression}
\label{sec:equ_krr}
In this section, we present an equivalent formulation of the objective function in Equation \ref{eq:krr_obj_alpha}, which provides the same solution for $\alphahv$ as in Equation \ref{eq:krr_s_alpha}. This formulation opens up for generalizing KRR by using penalties other than the ridge penalty and also provides an interesting connection to functional gradient descent \citep{mason1999boosting}.
In Appendix \ref{sec:equ_f}, we take this one step further by reformulating Equation \ref{eq:krr_obj_alpha} directly in the model predictions,
$$\begin{bmatrix}\fv\\\fsv\end{bmatrix}:=\begin{bmatrix}\fv(\X)\\\fv(\Xs)\end{bmatrix},$$
by presenting two equivalent objectives, that, when minimized with respect to $[\fv^\top, \fsv{^\top}]^\top$, generate the solution in Equation \ref{eq:krr_s_fs}. In this and the following sections, all calculations are done with respect to $\alphav$, and the corresponding expressions for $[\fv^\top, \fsv{^\top}]^\top$ are obtained through multiplication with $[\K^\top,\Ks{^\top}]^\top$. However, in Appendix \ref{sec:equ_f}, we show how the expressions for $[\fv^\top, \fsv{^\top}]^\top$ can alternatively be obtained directly without taking the detour over $\alphav$.
In Lemma \ref{thm:equ_alpha}, we show how we can move the weighted norm in Equation \ref{eq:krr_obj_alpha} from the penalty term to the reconstruction term.
\begin{lemma}
\label{thm:equ_alpha}
\begin{subequations}
\label{eq:equ_alpha}
\begin{align}
\alphahv=&\argmin_{\alphav\in \R^n}\frac12\left\|\yv-\K\alphav\right\|_2^2+\frac\lambda2\|\alphav\|^2_{\K}\label{eq:equ_alpha1}\\
=&\argmin_{\alphav\in \R^n}\frac12\left\|\yv-\K\alphav\right\|_{\K^{-1}}^2+\frac\lambda2\|\alphav\|^2_2\label{eq:equ_alpha2}\\
=&\left(\K+\lambda\I\right)^{-1}\yv.\notag
\end{align}
\end{subequations}
\end{lemma}
The alternative formulation in Equation \ref{eq:equ_alpha2}, where the reconstruction term rather than the regularization term, is weighted by the kernel matrix, has two interesting implications. First, a standard ridge penalty on the regularization term opens up for using other penalties than the $\ell_2$ norm, such as the $\ell_1$ and $\ell_\infty$ norms. Although these penalties have previously been used in the kernel setting, it has been done by replacing $\|\alphav\|_{\K}^2$ in Equation \ref{eq:krr_obj_alpha} by $\|\alphav\|_1$ or $\|\alphav\|_\infty$, thus ignoring the impact of $\K$, by acting as if the objective of KRR were to minimize $\|\yv-\K\alphav\|_2^2+\lambda\|\alphav\|_2^2$. This form has the solution $\alphahv=\K(\K^2+\lambda)^{-1}\yv$, for which the connection to linear regression in feature space is lost.
Second, the gradient of Equation \ref{eq:equ_alpha2} with respect to $\alphav$ is
\begin{equation}
\label{eq:kgd_grad_alpha}
\K\alphav-\yv+\lambda\alphav
\end{equation}
Multiplying by $[\K^\top,\Ks{^\top}]^\top$, we obtain
\begin{equation}
\label{eq:kgd_grad_f}
\begin{bmatrix}\K\\\Ks\end{bmatrix}(\fv-\yv)+\lambda\begin{bmatrix}\fv\\\fsv\end{bmatrix},
\end{equation}
which is the gradient used in functional gradient descent, where it is obtained from differentiating functionals. Here, however, it is a simple simple consequence of the equivalent objective function of KRR.
\section{Kernel Gradient Descent and Kernel Gradient Flow}
\label{sec:kgf}
In this section, we investigate solving kernel regression iteratively with gradient descent. We also use gradient descent with infinitesimal step size, known as gradient flow, to obtain a closed-form solution which we use for direct comparisons to kernel ridge regression.
The similarities between ridge regression and gradient descent with early stopping are well studied for linear regression \citep{friedman2004gradient,ali2019continuous, allerbo2022elastic}. When starting at zero, optimization time can be thought of as an inverse penalty, where longer optimization time corresponds to weaker regularization. When applying gradient descent to kernel regression, something we refer to as kernel gradient descent, KGD, we will replace explicit regularization with implicit regularization through early stopping. That is, we will use $\lambda=0$ and consider training time, $t$, as a regularizer.
With $\lambda=0$ in Equation \ref{eq:kgd_grad_alpha}, starting at $\nv$, the KGD update becomes, for step size $\eta$,
\begin{equation}
\label{eq:kgd_update}
\alphahv_{k+1}=\alphahv_k+\eta\cdot\left(\yv-\K\alphahv_k\right),\ \alphahv_0=\nv.
\end{equation}
To compare the regularization injected by early stopping to that of ridge regression, we let the optimization step size go to zero to obtain a closed-form solution, which we refer to as kernel gradient flow, KGF. Then, Equation \ref{eq:kgd_update} can be thought of as the Euler forward formulation of the differential equation in Equation \ref{eq:kgd_diff_eq},
\begin{equation}
\label{eq:kgd_diff_eq}
\frac{d \alphahv(t)}{d t}=\yv-\K\alphahv(t),\ \alphahv(0)=\nv
\end{equation}
whose solution is stated in Lemma \ref{thm:kgf_s}.
\begin{lemma}
\label{thm:kgf_s}~\\
The solution to the differential equation in Equation \ref{eq:kgd_diff_eq} is
\begin{equation}
\label{eq:kgf_s}
\alphahv(t)=(\I-\exp(-t\K))\K^{-1}\yv=:\alphahfv(t),
\end{equation}
where $\exp$ denotes the matrix exponential.
\end{lemma}
\textbf{Remark 1:} Note that $(\I-\exp(-t\K))\K^{-1}=\K^{-1}(\I-\exp(-t\K))$ is well-defined even if $\K$ is singular. The matrix exponential is defined through its Taylor approximation and from $\I-\exp(-t\K)=t\K-\frac1{2!}t^2\K^2+\dots$, a matrix $\K$ factors out, that cancels with $\K^{-1}$.\\
\textbf{Remark 2:} It is possible to generalize Lemma \ref{thm:kgf_s} to Nesterov accelerated gradient descent with momentum \citep{nesterov1983method,polyak1964some}. In this case $\exp(-t\K)$ in Equation \ref{eq:kgf_s} generalizes to $\exp\left(-\frac t{1-\gamma}\K\right)$, where $\gamma\in [0,1)$ is the strength of the momentum. See the proof in Appendix \ref{sec:proofs} for details.
To facilitate the comparisons between KGF and KRR, we rewrite Equation \ref{eq:krr_s_alpha} using Lemma \ref{thm:rewrite_alpha_krr}, from which we obtain
\begin{equation}
\label{eq:krr_equ}
\alphahrv(\lambda)=\left(\I-\left(\I+1/\lambda\cdot\K\right)^{-1}\right)\K^{-1}\yv.
\end{equation}
Since $\exp(-t\K)=\exp(t\K)^{-1}$, the KGF and KRR solutions differ only in the factor
\begin{equation}
\label{eq:exp_vs_lin}
\exp(t\K) \text{ v.s. } \I+1/\lambda\K.
\end{equation}
Thus, for $t=1/\lambda$, we can think of the ridge penalty as a first-order Taylor approximation of the gradient flow penalty.
\begin{lemma}
\label{thm:rewrite_alpha_krr}
\begin{equation*}
\label{eq:rewrite_alpha_krr}
\left(\K+\lambda\I\right)^{-1}=\left(\I-\left(\I+1/\lambda\cdot\K\right)^{-1}\right)\K^{-1}.
\end{equation*}
\end{lemma}
\subsection{Comparisons between Kernel Ridge Regression and Kernel Gradient Flow}
In this section, we compare the KRR and KGF solutions for $\lambda=1/t$. To do this, we introduce the following notation, where $\kv(\xsv)^\top=\kv(\xsv,\X)^\top\in\R^n$ is the row in $\Ks$ corresponding to $\xsv$:
\begin{equation*}
\begin{aligned}
&\fhfv(\X,t):=\K\alphahfv(t),\quad \fhf(\xsv,t):=\kv(\xsv)^\top\alphahfv(t),\\
&\fhrv(\X,\lambda):=\K\alphahrv(\lambda),\quad \fhr(\xsv,\lambda):=\kv(\xsv)^\top\alphahrv(\lambda),\\
&\alphahnv:=\alphahfv(t=\infty)=\alphahrv(\lambda=0)=\K^{-1}\yv,\\
&\fhnv(\X):=\fhfv(\X,t=\infty)=\fhrv(\X,\lambda=0)=\K\alphahnv=\yv,\\
&\fhn(\xsv):=\fhf(\xsv,t=\infty)=\fhr(\xsv,\lambda=0)=\kv(\xsv)^\top\alphahnv=\kv(\xsv)^\top\K^{-1}\yv,\\
&\alphahfi:=(\alphahfv)_i, \quad \alphahri:=(\alphahrv)_i,\quad \alphahni:=(\alphahnv)_i.
\end{aligned}
\end{equation*}
Even though not explicitly stated, all estimates depend on the training data $(\X,\yv)$.
In Proposition \ref{thm:kgf_krr_diff}, we bound the differences between the KGF and KRR solutions in terms of the non-regularized solutions, $\alphanv$, $\fhnv(\X)$ and $\fhn(\xsv)$. In parts \emph{(a)} and \emph{(b)}, where the difference between the parameter and in-sample prediction vectors are bounded, no further assumptions are needed. In parts \emph{(c)} and \emph{(d)}, with bounds on individual parameters and predictions, including out-of-sample predictions, some very reasonable assumptions have to be made on the data. For all four bounds, the larger the non-regularized value is, the larger the difference between the KGF and KRR estimates is allowed to be.
\begin{prop}~\\
\label{thm:kgf_krr_diff}
For $t\geq 0$, $\yv\in\R^n$,
\begin{itemize}
\item[(a)]
$\left\|\alphahfv(t)-\alphahrv(1/t)\right\|_2^2\leq 0.0415\cdot\|\alphahnv\|_2^2$,
\item[(b)]
$\left\|\fhfv(\X,t)-\fhrv(\X,1/t)\right\|_2^2\leq 0.0415\cdot\|\fhnv(\X)\|_2^2=0.0415\cdot\|\yv\|_2^2$.
\end{itemize}
For $t\geq 0$, $\yv=\K\alphanv+\epsv,\ \alphanv\sim (\nv, \Sigmaalpha),\ \epsv\sim (\nv, \sigma_\varepsilon^2\I)$, where $\Sigmaalpha$ and $\K$ are simultaneously diagonalizable,
\begin{itemize}
\item[(c)]
$\E_{\epsv,\alphanv}\left(\left(\alphahfi(t)-\alphahri(1/t)\right)^2\right)\leq0.0415\cdot\E_{\epsv,\alphanv}\left((\alphahni)^2\right)$, $i=1,2,\dots n$,
\item[(d)]
$\E_{\epsv,\alphanv}\left(\left(\fhf(\xsv,t)-\fhr(\xsv,1/t)\right)^2\right)\leq0.0415\cdot\E_{\epsv,\alphanv}\left(\fhn(\xsv)^2\right)$.
\end{itemize}
\end{prop}
Two typical options for $\Sigmaalpha$ are $\sigma_{\alpha}^2\I$ and $\sigma_\beta^2\K^{-1}$, where the second formulation implies $\bm{\beta_0}\sim (\nv,\sigma_\beta^2\I)$ in the feature space formulation of KRR from Equation \ref{eq:krr_obj_beta}.
In Proposition \ref{thm:kgf_krr_y}, we compare the distances to the observation vector, $\yv$, of the KGF and KRR solutions. According to parts \emph{(a)} and \emph{(c)}, for a given regularization, the KGF solution lies closer to the observations than the KRR solution does. Analogously, according to parts \emph{(b)} and \emph{(d)}, for a given regularization, the KRR solution lies closer to zero than the KGF solution does.
\begin{prop}~\\
\label{thm:kgf_krr_y}
For $t\geq 0$, $\yv\in \R^n$,
\begin{equation*}
\begin{aligned}
&\text{(a)} && \max_{\yv\neq \nv}\left(\frac{\left\|\fhfv(\X,t)-\yv\right\|_2}{\left\|\yv\right\|_2}\right)\leq\max_{\yv\neq \nv}\left(\frac{\left\|\fhrv(\X,1/t)-\yv\right\|_2}{\left\|\yv\right\|_2}\right),\\
&\text{(b)} && \max_{\yv\neq \nv}\left(\frac{\left\|\fhrv(\X,1/t)\right\|_2}{\left\|\yv\right\|_2}\right)\leq\max_{\yv\neq \nv}\left(\frac{\left\|\fhfv(\X,t)\right\|_2}{\left\|\yv\right\|_2}\right).
\end{aligned}
\end{equation*}
For $t\geq 0$, $\yv=\K\alphanv+\epsv,\ \alphanv\sim (\nv, \Sigmaalpha),\ \epsv\sim (\nv, \sigma_\varepsilon^2\I)$, where $\Sigmaalpha$ and $\K$ are simultaneously diagonalizable, and $i=1,2,\dots n$,
\begin{equation*}
\begin{aligned}
&\text{(c)} && \max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\fhf(\xvi,t)-y_i\right)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right)
\leq\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\fhr(\xvi,1/t)-y_i\right)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right),\\
&\text{(d)} && \max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\fhr(\xvi,1/t)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right)
\leq\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\fhf(\xvi,t)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right).
\end{aligned}
\end{equation*}
\end{prop}
Finally, if we assume that a true function exists, parameterized by the true $\alphanv$ as
$$\begin{bmatrix}\fnv(\X)\\\fnv(\Xs)\end{bmatrix}=\begin{bmatrix}\K\\\Ks\end{bmatrix}\alphanv,$$
and with observations according to $\yv=\fnv(\X)+\epsv$, we can calculate the expected squared differences between the estimated and true models, something that is often referred to as the risk: $\text{Risk}(\bm{\hat{\theta}};\bm{\theta_0})=\E\left(\|\bm{\hat{\theta}}-\bm{\theta_0}\|_2^2\right)$. In Proposition \ref{thm:kgf_krr_risk}, the KGF estimation and prediction risks are bounded in terms of the corresponding KRR risks. In all three cases, the KGF risk is less than 1.69 times the KRR risk.
\begin{prop}~\\
\label{thm:kgf_krr_risk}
For $t\geq 0$, $\yv=\fnv(\X)+\epsv=\K\alphanv+\epsv$, $\epsv\sim(\bm{0},\sigma_\varepsilon^2\I)$,
\begin{itemize}
\item[(a)]
For any $\alphanv$, $\text{\emph{Risk}}(\alphahfv(t);\alphanv)\leq 1.6862\cdot\text{\emph{Risk}}(\alphahrv(1/t);\alphanv)$.
\item[(b)]
For any $\alphanv$, $\text{\emph{Risk}}(\fhfv(\X,t);\fnv(\X))\leq 1.6862\cdot\text{\emph{Risk}}(\fhrv(\X,1/t);\fnv(\X))$.
\item[(c)]
For $\alphanv\sim \left(\nv,\Sigmaalpha\right)$, where $\Sigmaalpha$ and $\K$ are simultaneously diagonalizable,\\$\E_{\alphanv}\left(\text{\emph{Risk}}(\fhf(\xsv,t);f_0(\xsv))\right)\leq 1.6862\cdot\E_{\alphanv}\left(\text{\emph{Risk}}(\fhr(\xsv,1/t);f_0(\xsv))\right)$,\\
where $f_0(\xsv)\in \fnv(\Xs)$.
\end{itemize}
\end{prop}
\section{Kernel Regression with the \texorpdfstring{$\ell_\infty$}{l1} and \texorpdfstring{$\ell_1$}{linf} Norms}
%\section{Kernel Regression with the $\ell_\infty$ and $\ell_$ Norms}
\label{sec:l1linf}
In this section, we replace the squared $\ell_2$ norm of KRR in Equation \ref{eq:equ_alpha2} by the $\ell_\infty$ and $\ell_1$ norms, respectively, to obtain $\ell_\infty$ and $\ell_1$ regularized kernel regression, which we abbreviate as K$\ell_\infty$R and K$\ell_1$R, respectively. We also connect the explicitly regularized algorithms K$\ell_\infty$R and K$\ell_1$R to kernel sign gradient descent, KSGD, and kernel coordinate descent, KCD, with early stopping, similarly to how we related KRR and KGF in Section \ref{sec:kgf}. The six algorithms and their abbreviations are stated in Table \ref{tab:abbrs}.
\begin{table}
\caption{Kernel regression algorithms}
\center
\begin{tabular}{|l|l|}
\hline
Algorithm & Abbreviation\\
\hline
Kernel ridge regression ($\ell_2$ penalty) & KRR\\
Kernel regression with the $\ell_\infty$ penalty (robust) & K$\ell_\infty$R\\
Kernel regression with the $\ell_1$ penalty (sparse) & K$\ell_1$R\\
Kernel gradient descent & KGD\\
Kernel sign gradient descent (robust) & KSGD\\
Kernel coordinate descent (sparse) & KCD\\
\hline
\end{tabular}
\label{tab:abbrs}
\end{table}
The objective functions for K$\ell_\infty$R and K$\ell_1$R are, respectively
\begin{equation}
\label{eq:alphainf}
\argmin_{\alphav\in \R^n}\frac12\left\|\yv-\K\alphav\right\|_{\K^{-1}}^2+\lambda\|\alphav\|_\infty
\end{equation}
and
\begin{equation}
\label{eq:alpha1}
\argmin_{\alphav\in \R^n}\frac12\left\|\yv-\K\alphav\right\|_{\K^{-1}}^2+\lambda\|\alphav\|_1.
\end{equation}
In contrast to KRR, unless the data is uncorrelated, no closed-form solutions exist for Equations \ref{eq:alphainf} and \ref{eq:alpha1}. However, the problems are still convex and solutions can be obtained using the iterative optimization method proximal gradient descent \citep{rockafellar1976monotone}, which, in contrast to standard gradient descent, can handle the discontinuities of the gradients of the $\ell_\infty$ and $\ell_1$ norms.
%For a translational-invariant kernel, whose value decreases with the distance between the arguments (i.e.\ $k(\xv,\xpv)=k(\|\xv-\xpv\|_2)$ and $k'(\|\xv-\xpv\|_2)\leq 0$), each prediction is more affected by observations that it is close to than observations further away.
With $\ell_\infty$ regularization, a solution where all parameters contribute equally is encouraged. Thus, Equation \ref{eq:alphainf} promotes a solution where all observations contribute equally, as is exemplified in the top-middle panel of Figure \ref{fig:comp_af}. This might be a good option if outliers are present in the data.
On the contrary, $\ell_1$, or lasso, regularization promotes a sparse solution, where parameters are added sequentially to the model, with the most significant parameters included first.
A sparse solution in $\alphav$, which only includes the most significant observations, thus results in a solution that is closer to the non-regularized solution in the neighborhood of influential observations, and close to zero further away from influential observations, as is exemplified in the top-right panel of Figures \ref{fig:comp_af}. This might be a suitable penalty if there is a strong signal in combination with noise, since this regularization scheme has the capacity to discard the noise without introducing a too large bias to the signal; or to obtain a compressed solution, including only the most significant observations.
\subsection{Regularization through Early Stopping}
Applying the proximal gradient descent algorithm might be computationally heavy, especially when evaluating several different regularization strengths. In Section \ref{sec:kgf}, we showed how the solution of KGF with early stopping is very similar to that of KRR. Thus, running KGD until convergence once, all different regularization strengths between $t=0$ and $t=t_{\max}$ are obtained through this single execution of the algorithm. If something similar could be done for the $\ell_\infty$ and $\ell_1$ penalties, instead of solving the problem iteratively once for each value of $\lambda$, all different regularization strengths could be obtained through a single call of the algorithm.
The similarities between $\ell_1$ regularization (lasso) and the iterative optimization method forward stagewise regression (also known as coordinate descent) are well studied, for instance by \citet{efron2004least} and \citet{hastie2007forward}. Just as there is a connection between $\ell_2$ regularization and gradient descent with early stopping, an analog connection exists between $\ell_1$ regularization and coordinate descent with early stopping.
The topic of a gradient-based algorithm similar to $\ell_\infty$ regularization does not seem to be as well studied. However, in Proposition \ref{thm:sgn_inf}, we show that the solutions of $\ell_\infty$ regularization and sign gradient descent flow coincide for uncorrelated data. This is shown using the constrained form of regularized regression, which, due to Lagrangian duality, is equivalent to the penalized form.
We denote the gradient and its maximum component as
\begin{equation*}
\begin{aligned}
\bm{g}_k&:=\frac{\partial L(\bm{\theta}_k;\X,\yv)}{\partial \bm{\theta}_k}\\
m_k&:=\argmax_d(|\bm{g}_k|)_d,
\end{aligned}
\end{equation*}
where $L(\cdot)$ is a loss function that quantifies the reconstruction error, and where the absolute value of the gradient is evaluated element-wise.
Then, the update rules of gradient descent, coordinate descent, and sign gradient descent are stated in Equation \ref{eq:gd_updates}.
\begin{equation}
\label{eq:gd_updates}
\begin{aligned}
\text{Gradient Descent: }& &\bm{\theta}_{k+1}&=\bm{\theta}_k-\eta\cdot\bm{g}_k\\
\text{Sign Gradient Descent: }& &\bm{\theta}_{k+1}&=\bm{\theta}_k-\eta\cdot\sgn\left(\bm{g}_k\right)\\
\text{Coordinate Descent: }& &\left(\bm{\theta}_{k+1}\right)_{m_k}&=\left(\bm{\theta}_k\right)_{m_k}-\eta\cdot\sgn\left(\bm{g}_k\right)_{m_k}.
\end{aligned}
\end{equation}
\textbf{Remark 1:} For coordinate descent, in each iteration, only the coordinate corresponding to the maximal absolute gradient value is updated.\\
\textbf{Remark 2:} The name ``coordinate descent'' is sometimes also used for other, related, algorithms.
\begin{prop}~\\
\label{thm:sgn_inf}
\begin{enumerate}
\item[(a)]
Let $\betahinfv(c)$ denote the solution to
$$\min_{\betav\in\R^p}\left\|\yv-\X\betav\right\|_{2}^2 \text{ s.t. } \|\betav\|_\infty\leq c,$$
and let $\betahsgfv(t)$ denote the solution to
$$\frac{d\betav(t)}{dt}=\sgn\left(-\frac{\partial}{\partial\betav(t)}\left(\left\|\yv-\X\betav(t)\right\|_{2}^2\right)\right),\ \betav(0)=\nv.$$
When $\X^\top\X$ is diagonal, with elements $\{s_{ii}\}_{i=1}^p$, the two solutions decompose element-wise and coincide for $c=t$:
$$\betahinfi(t)=\betahsgfi(t)=\sgn((\X^\top\yv)_i/s_{ii})\cdot\min(t,|(\X^\top\yv)_i/s_{ii}|).$$
\item[(b)]
Let $\alphahinfv(c)$ denote the solution to
$$\min_{\alphav\in\R^n}\left\|\yv-\K\alphav\right\|_{\K^{-1}}^2 \text{ s.t. } \|\alphav\|_\infty\leq c,$$
and let $\alphahsgfv(t)$ denote the solution to
$$\frac{d\alphav(t)}{dt}=\sgn\left(-\frac{\partial}{\partial\alphav(t)}\left(\left\|\yv-\K\alphav(t)\right\|_{\K^{-1}}^2\right)\right),\ \alphav(0)=\nv.$$
When $\K$ is diagonal, with elements $\{k_{ii}\}_{i=1}^n$, the two solutions decompose element-wise and coincide for $c=t$:
$$\alphahinfi(t)=\alphahsgfi(t)=\sgn(y_i/k_{ii})\cdot\min(t,|y_i/k_{ii}|).$$
\end{enumerate}
\end{prop}
\textbf{Remark 1:} For infinitesimal step size, sign gradient descent, Adam \citep{kingma2014adam}, and RMSProp \citep{hinton2012neural} basically coincide \citep{balles2018dissecting}. Thus, in the uncorrelated case, $\ell_\infty$ regularization also (almost) coincides with Adam and RMSProp with infinitesimal step size.\\
\textbf{Remark 2:} It is also possible to show that, for uncorrelated data, the solutions of gradient flow and $\ell_2$ regularization, and the solutions of coordinate flow and $\ell_1$ regularization, coincide. The former is easily shown by comparing Equations \ref{eq:kgf_s} and \ref{eq:krr_equ} for $\K=\I$ and $\lambda=1/(1+e^t)$.\\
In Figures \ref{fig:comp_af} and \ref{fig:comp_a}, the similarities between explicit regularization and gradient-based optimization with early stopping are demonstrated. The solutions are very similar, although not exactly identical.
\begin{figure}
\center
\includegraphics[width=\textwidth]{figures/compare_pen_af}
\caption{Comparisons of the effects of KRR/KGF, K$\ell_\infty$R/KSGD, and K$\ell_1$R/KCD on $\fhv(\xv)$.
In the top panel, a larger regularization, or a shorter training time, is used than in the bottom panel.
For KRR/KGF, we use $t=1/\lambda$, while for the other two cases, $t$ and $\lambda$ are chosen so that the functions coincide as well as possible.\\
As proposed by Proposition \ref{thm:kgf_krr_diff}, the KGF and KRR solution differs most where the non-regularized solution is large, and as suggested by Proposition \ref{thm:kgf_krr_y}, the KGF solution tends to lie closer to the observations than the KRR solution does.\\
For K$\ell_\infty$R/KSGD, all observations tend to contribute equally to the solution, resulting in a more robust solution, less sensitive to outliers.
For K$\ell_1$R/KCD, some observations do not contribute to the solution, resulting in peaks at the more significant observations.\\
The solutions obtained through early stopping are very similar to those obtained through explicit regularization, although not exactly identical.
}
\label{fig:comp_af}
\end{figure}
\begin{figure}
\center
\includegraphics[width=\textwidth]{figures/compare_pen_a}
\caption{Comparisons of the effects of KRR/KGF, K$\ell_\infty$R/KSGD, and K$\ell_1$R/KCD on $\alphahv$.
In the top panel, a larger regularization, or a shorter training time, is used than in the bottom panel.
For KRR/KGF, we use $t=1/\lambda$, while, for the two other cases, $t$ and $\lambda$ are chosen so that the functions coincide as well as possible.\\
For K$\ell_\infty$R/KSGD, all observations tend to contribute equally to the solution, resulting in many $\hat{\alpha}_i$'s being equal in magnitude.
For K$\ell_1$R/KCD, some observations do not contribute to the solution, resulting in the corresponding $\hat{\alpha}_i$'s being 0.\\
The solutions obtained through early stopping are very similar to those obtained through explicit regularization, although not exactly identical.
}
\label{fig:comp_a}
\end{figure}
\subsection{Equivalent Interpretations of Kernel Sign Gradient Descent and Kernel Coordinate Descent}
As discussed above, using $\ell_\infty$ regularization on $\alphav$ promotes a solution where all observations contribute equally, while using $\ell_1$ regularization promotes a sparse solution, where only the most significant observations are included.
Since sign gradient descent, and coordinate descent, with early stopping provide solutions similar to those of $\ell_\infty$ and $\ell_1$ regularization, the same properties should hold for these algorithms. In Proposition \ref{thm:other_objs}, we show that this is the case by using the equivalent interpretation of kernel regression as linear regression in feature space from Equation \ref{eq:krr_obj_beta}.
\begin{prop}~\\
\label{thm:other_objs}
Solving $\min_{\alphav}\left\|\yv-\K\alphav\right\|_{\K^{-1}}^2$ for $\alphav$ with
\begin{itemize}
\item gradient descent, is equivalent of solving $\min_{\betav}\left\|\yv-\Ph\betav\right\|_2^2$ for $\betav$ with gradient descent.
\item sign gradient descent, is equivalent of solving $\min_{\betav}\left\|\yv-\Ph\betav\right\|_1$ for $\betav$ with gradient descent.
\item coordinate descent, is equivalent of solving $\min_{\betav}\left\|\yv-\Ph\betav\right\|_\infty$ for $\betav$ with gradient descent.
\end{itemize}
\end{prop}
According to Proposition \ref{thm:other_objs}, KSGD and KCD correspond to feature space regression with the $\ell_1$ and $\ell_\infty$ norms, respectively. Note that while we previously discussed different norms for the penalty term, we now consider different norms for the reconstruction term.
Regression with the $\ell_1$ norm, which is also known as robust regression, is known to be less sensitive to extreme observations than standard, $\ell_2$ norm, regression.
Vice versa, $\ell_\infty$ norm regression minimizes the maximum residual, i.e.\ more extreme observations contribute more to the solution than less extreme observations do.
Thus the takeaways of Proposition \ref{thm:other_objs} are the same as those for explicit regularization.
The implications of Proposition \ref{thm:other_objs} can also be observed in Figure \ref{fig:comp_af}.
For $\ell_2$ regression, all residuals are treated equally and all parts of the function are updated at a pace that is proportional to the distance to the solution at convergence.
For $\ell_1$ regression (KSGD), which is less sensitive to outliers, all parts of the function are initially updated at the same pace, and, in contrast to more moderate observations, the most extreme observations are not fully incorporated until the end of the training.
For $\ell_\infty$ regression (KCD), which minimizes the maximum residual, the most extreme observations are considered first, while more moderate observations are initially ignored.
\subsection{Convergence Analysis}
In Proposition \ref{thm:conv}, we present convergence analyses for KGD, KSGD, and KCD. We note that similarly to in Proposition \ref{thm:other_objs}, KSGD is related to $\ell_1$ regression, while KCD is related to $\ell_\infty$ regression.
\begin{prop}~\\
\label{thm:conv}
\begin{itemize}
\item
For KGD, where $\alphahv_{k+1}=\alphahv_k+\eta\cdot\left(\yv-\K\alphahv_k\right)$,
$$\eta< \frac2{\smax(\K)} \implies \|\yv-\K\alphahv_{k+1}\|_2<\|\yv-\K\alphahv_k\|_2,$$
where $\smax(\K)$ denotes the largest singular value of $\K$.
\item
For KCD, where $\left(\alphahv_{k+1}\right)_{m_k}=\left(\alphahv_k\right)_{m_k}+\eta\cdot\sgn\left(\yv-\K\alphahv_k\right)_{m_k}$,
$$\eta< \min_{1\leq i\leq n}\left(\left|\yv-\K\alphahv_k\right|_i\right) \implies \|\yv-\K\alphahv_{k+1}\|_\infty<\|\yv-\K\alphahv_k\|_\infty,$$
where $m_k$ denotes the coordinate with the largest absolute value, and where the absolute value is taken element-wise.
\item
For KSGD, where $\alphahv_{k+1}=\alphahv_k+\eta\cdot\sgn\left(\yv-\K\alphahv_k\right)$,
$$\eta< \min_{1\leq i\leq n}\left(\left|\yv-\K\alphahv_k\right|_i\right) \implies \|\yv-\K\alphahv_{k+1}\|_1<\|\yv-\K\alphahv_k\|_1,$$
where the absolute value is taken element-wise.
\end{itemize}
\end{prop}
\section{Experiments}
\label{sec:exps}
In this section, we demonstrate K$\ell_\infty$R, K$\ell_1$R, and the corresponding early stopping algorithms, KSGD and KCD, and compare the algorithms to KRR and KGD, on two synthetic and four real data sets.
The synthetic data were selected as follows:
For robust regression, to obtain data with outliers, 100 observations were sampled according to
\begin{equation}
\label{eq:data_sin}
x\sim\mathcal{U}(-10,10),\ y=\sin\left(\frac\pi2\cdot x\right)+\mathcal{C}(0,0.1),
\end{equation}
where $\mathcal{C}(0,\gamma)$ is the centered Cauchy distribution with scale parameter $\gamma$.
To demonstrate sparse regression on a signal that is mostly zero, with a narrow but distinct peak, 100 observations were sampled according to
\begin{equation}
\label{eq:data_gauss}
x\sim\mathcal{U}(-10,10),\ y=e^{-5\cdot x^2}+\mathcal{N}(0,0.1^2).
\end{equation}
On the first (robust) data set, we compare KSGD, K$\ell_\infty$R, KGD, and KRR, and on the second (sparse) data set, we compare KCD, K$\ell_1$R, KGD, and KRR.
The four real data sets are presented in Table \ref{tab:datasets}. For each data set, which was standardized to zero mean and unit variance, 100 random splits were created by randomly selecting 500 observations, which in turn were split 80\%/20\% into training and testing data.
When evaluating robust regression, outliers were introduced by multiplying each element, $y_i$, in the response vector by $(1+|\varepsilon_i|)$, where $\varepsilon_i\sim\mathcal{C}(0,0.01)$ is Cauchy distributed.
All experiments in this section were performed with the Gaussian kernel, $k(\xv,\xpv)=\exp\left(-\frac{\|\xv-\xpv\|^2_2}{2\sigma^2}\right)$; in Appendix \ref{sec:more_exps}, we also present results for the Laplace (Matérn-$\frac12$), Matérn-$\frac32$, Matérn-$\frac52$, and Cauchy kernels.
\begin{table}
\centering
\begin{tabular}{|l|l|}
\hline
Data set & Size, $n\times p$\\
\hline
\hline
Quality of aspen tree fibres\tablefootnote{The data set is available at \url{https://openmv.net/info/wood-fibres}.} & $25165\times 5$\\
\hline
House values in California \citep{pace1997sparse}\tablefootnote{The data set is available at \url{https://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.html}.} & $20640\times 8$\\
\hline
\makecell[l]{Protein structure as root-mean-square deviation of\\ atomic positions, taken from CASP\tablefootnote{\url{https://predictioncenter.org/}, The data set is available at \url{https://archive.ics.uci.edu/ml/datasets/Physicochemical+Properties+of+Protein+Tertiary+Structure}.}} & $45730\times 9$\\
\hline
\makecell[l]{Daily concentration of black smoke particles in the U.K.\\ in the year 2000 \citep{wood2017generalized}\tablefootnote{The data set is available at \url{https://www.maths.ed.ac.uk/~swood34}.}} & $45568\times 10$\\
\hline
\end{tabular}
\caption{Real data sets used for comparing the algorithms.}
\label{tab:datasets}
%\vspace{-.5cm}
\end{table}
For the explicitly regularized algorithms, the kernel bandwidth, $\sigma$, and regularization strengths were selected through 10-fold cross-validation with $50\times 50$ logarithmically spaced values. For the early stopping algorithms, 10-fold cross-validation was used for bandwidth selection, with the same 50 candidate bandwidths as for the explicitly regularized models. The stopping times were selected by monitoring performance on validation data.
All statistical tests were performed using Wilcoxon signed-rank tests.
The experiments were run on a cluster with Intel Xeon Gold 6130, 2.10 GHz processors.
For the iterative algorithms, we used an optimization step size of $10^{-4}$.
%We used five different kernels, four Matérn kernels, with different differentiability parameters, $\nu$ (including the Laplace and Gaussian kernels), and the Cauchy kernel. The five different kernels are stated in Table \ref{tab:kerns}.
The evaluation metrics used are
\begin{itemize}
\item
test $R^2$, i.e.\ the proportion of the variance in test $\yv$ that is explained by the model.
\item computation time in seconds.
\item model sparsity, calculated as the fraction between included and available observations (only for K$\ell_1$R/KCD).
\end{itemize}
The results are presented in Tables \ref{tab:sgd} and \ref{tab:cd}.
For robust regression (Table \ref{tab:sgd}), K$\ell_\infty$R/KSGD outperform KRR/KGD in terms of test $R^2$ on all data sets, with KSGD being significantly faster than K$\ell_\infty$R (although not as fast KRR which has a closed-form solution). KRR tends to perform better than KGD in terms of test $R^2$, which can probably be attributed to the fact that the KGF solution tends to lie closer to the observations than the KRR solution does, as suggested by Proposition \ref{thm:kgf_krr_y}. KGD is thus more sensitive to outliers than KRR is.
In contrast to KRR/KGD, K$\ell_1$R/KCD produce sparse models (in the observations, Table \ref{tab:cd}), however usually at the cost of a lower $R^2$. In this case, the early stopping algorithm, KCD, does not have the computational advantage as KSGD does, which can be attributed to the nature of coordinate descent, where only one parameter is updated in every iteration. To alleviate this limitation of coordinate descent, we also included elastic gradient descent \citep{allerbo2022elastic}, which combines coordinate and gradient descent into a computationally more efficient, although still sparse, iterative optimization algorithm. Compared to KCD, kernel elastic gradient descent, KEGD, is much faster and tends to exhibit a higher test $R^2$ at the expense of a less sparse model. We ran the KEGD algorithm with $\alpha=0.9$, where $\alpha=1$ corresponds to KCD and $\alpha=0$ corresponds to KGD, and a momentum of $\gamma=0.5$.
\begin{table}
\caption{Median and first and third quartile of test $R^2$ and computation time in seconds for KSGD, K$\ell_\infty$R, KGD, and KRR. Occasions where the algorithm performs significantly worse in terms of $R^2$/computation time than KSGD (at significance level 0.05) are marked with bold. KSGD is significantly faster than K$\ell_\infty$R. K$\ell_\infty$R/KSGD are less sensitive to outliers than KRR/KGD are and thus perform significantly better. KGD is more sensitive to outliers than KRR is.}
\center
\begin{tabular}{|l|l|l|l|}
\hline
Data & Method & $R^2$ & Computation Time [s]\\
\hline
\multirow{4}{*}{Synthetic} & KSGD & $0.96,\ (0.92, 0.98)$ & $83,\ (70, 97)$ \\
& K$\ell_\infty$R & $\bm{0.94,\ (0.88, 0.96)}$ & $\bm{1981,\ (1946, 2045)}$ \\
& KGD & $\bm{-0.40,\ (-2.93, 0.71)}$ & $\bm{1790,\ (832, 3137)}$ \\
& KRR & $\bm{0.51,\ (-0.15, 0.84)}$ & $12,\ (12, 12)$ \\
\hline
\multirow{4}{*}{\makecell{Aspen\\Fibres}} & KSGD & $0.37,\ (0.14, 0.46)$ & $212,\ (182, 262)$ \\
& K$\ell_\infty$R & $0.36,\ (0.12, 0.47)$ & $\bm{3556,\ (3175, 3796)}$ \\
& KGD & $\bm{0.21,\ (0.01, 0.39)}$ & $\bm{14950,\ (8064, 25870)}$ \\
& KRR & $\bm{0.24,\ (0.06, 0.42)}$ & $133,\ (116, 139)$ \\
\hline
\multirow{4}{*}{\makecell{California\\Housing}} & KSGD & $0.43,\ (0.22, 0.56)$ & $1708,\ (1390, 2130)$ \\
& K$\ell_\infty$R & $0.41,\ (0.18, 0.56)$ & $\bm{5945,\ (5808, 6114)}$ \\
& KGD & $\bm{0.22,\ (0.00, 0.45)}$ & $\bm{7523,\ (5651, 11283)}$ \\
& KRR & $\bm{0.33,\ (0.13, 0.50)}$ & $155,\ (149, 161)$ \\
\hline
\multirow{4}{*}{\makecell{Protein\\Structure}} & KSGD & $0.13,\ (0.06, 0.20)$ & $3994,\ (3394, 5459)$ \\
& K$\ell_\infty$R & $0.14,\ (0.03, 0.23)$ & $\bm{6586,\ (5728, 6841)}$ \\
& KGD & $\bm{-0.01,\ (-0.40, 0.13)}$ & $\bm{10768,\ (6972, 20140)}$ \\
& KRR & $\bm{0.06,\ (0.00, 0.16)}$ & $136,\ (120, 149)$ \\
\hline
\multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KSGD & $0.09,\ (0.03, 0.15)$ & $568,\ (432, 755)$ \\
& K$\ell_\infty$R & $0.09,\ (0.02, 0.17)$ & $\bm{5539,\ (5216, 5749)}$ \\
& KGD & $\bm{0.00,\ (-0.33, 0.07)}$ & $\bm{7479,\ (5777, 10585)}$ \\
& KRR & $\bm{0.03,\ (0.00, 0.09)}$ & $203,\ (179, 217)$ \\
\hline
\end{tabular}
\label{tab:sgd}
\end{table}
\begin{table}
\caption{Median and first and third quartile of test $R^2$, computation time in seconds, and sparsity for KCD, K$\ell_1$R, KGD, and KRR. Occasions where the algorithm performs significantly worse in terms of $R^2$/computation time/sparsity than KCD (at significance level 0.05) are marked with bold. In contrast to KRR/KGD, K$\ell_1$R/KCD/KEGD only includes a fraction of the observations in the model. It is not obvious which of the K$\ell_1$R and KCD algorithms performs better. KEGD generally outperforms K$\ell_1$R/KCD in terms of computation time.}
\center
\begin{tabular}{|l|l|l|l|l|}
\hline
Data & Method & $R^2$ & Computation Time [s] & Sparsity\\
\hline
\multirow{4}{*}{Synthetic} & KCD & $0.93,\ (0.85, 0.96)$ & $262,\ (206, 355)$ & $0.04,\ (0.02, 0.07)$ \\
& K$\ell_1$R & $\bm{0.91,\ (0.87, 0.94)}$ & $\bm{427,\ (404, 456)}$ & $\bm{0.47,\ (0.30, 0.74)}$ \\
& KGD & $\bm{0.80,\ (0.72, 0.87)}$ & $\bm{1894,\ (952, 2926)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& KRR & $\bm{0.86,\ (0.82, 0.89)}$ & $13,\ (12, 14)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& \textit{KEGD} & $\mathit{0.92,\ (0.84, 0.95)}$ & $\mathit{88,\ (67, 112)}$ & $\mathit{0.04,\ (0.01, 0.10)}$ \\
\hline
\multirow{4}{*}{\makecell{Aspen\\Fibres}} & KCD & $0.07,\ (0.03, 0.10)$ & $153,\ (103, 246)$ & $0.01,\ (0.00, 0.01)$ \\
& K$\ell_1$R & $\bm{0.05,\ (0.00, 0.09)}$ & $190,\ (100, 269)$ & $0.01,\ (0.00, 0.01)$ \\
& KGD & $0.58,\ (0.48, 0.63)$ & $\bm{31765,\ (23177, 42665)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& KRR & $0.58,\ (0.49, 0.64)$ & $138,\ (134, 141)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& \textit{KEGD} & $\mathit{0.07,\ (0.03, 0.10)}$ & $\mathit{54,\ (39, 77)}$ & $\mathit{0.01,\ (0.01, 0.01)}$ \\
\hline
\multirow{4}{*}{\makecell{California\\Housing}} & KCD & $0.38,\ (0.32, 0.44)$ & $37130,\ (29714, 44168)$ & $0.34,\ (0.28, 0.40)$ \\
& K$\ell_1$R & $0.50,\ (0.45, 0.56)$ & $517,\ (453, 552)$ & $0.27,\ (0.24, 0.29)$ \\
& KGD & $0.70,\ (0.63, 0.73)$ & $18198,\ (14190, 23534)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& KRR & $0.70,\ (0.67, 0.74)$ & $140,\ (121, 148)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& \textit{KEGD} & $\mathit{0.45,\ (0.39, 0.53)}$ & $\mathit{787,\ (690, 939)}$ & $\mathit{0.52,\ (0.43, 0.66)}$ \\
\hline
\multirow{4}{*}{\makecell{Protein\\Structure}} & KCD & $0.00,\ (-0.02, 0.02)$ & $5046,\ (2663, 8280)$ & $0.04,\ (0.02, 0.07)$ \\
& K$\ell_1$R & $0.29,\ (0.24, 0.34)$ & $1391,\ (1348, 1423)$ & $\bm{0.93,\ (0.92, 0.94)}$ \\
& KGD & $0.27,\ (0.20, 0.35)$ & $\bm{22160,\ (15571, 28547)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& KRR & $0.29,\ (0.23, 0.34)$ & $153,\ (150, 156)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& \textit{KEGD} & $\mathit{0.07,\ (0.03, 0.11)}$ & $\mathit{786,\ (515, 1111)}$ & $\mathit{0.35,\ (0.20, 0.52)}$ \\
\hline
\multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KCD & $0.04,\ (0.01, 0.07)$ & $2606,\ (1556, 4098)$ & $0.07,\ (0.03, 0.15)$ \\
& K$\ell_1$R & $0.09,\ (0.06, 0.12)$ & $388,\ (357, 428)$ & $\bm{0.15,\ (0.14, 0.17)}$ \\
& KGD & $0.14,\ (0.08, 0.21)$ & $\bm{9569,\ (7648, 12781)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& KRR & $0.16,\ (0.10, 0.23)$ & $220,\ (218, 223)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& \textit{KEGD} & $\mathit{0.05,\ (0.01, 0.10)}$ & $\mathit{226,\ (157, 302)}$ & $\mathit{0.16,\ (0.06, 0.30)}$ \\
\hline
\end{tabular}
\label{tab:cd}
\end{table}
In Figure \ref{fig:other_reg} we present one realization for each of the synthetic data sets.
For the first data set, KRR/KGD are more sensitive to the outliers than K$\ell_\infty$R/KSGD are, with KGD being more sensitive than KRR.
For the first data set, to perform well at the peak, KRR/KGD must also incorporate the noise in the regions where the true signal is zero. K$\ell_1$R/KCD include the most significant observations first and are thus able to perform well at the peak while still ignoring the noise in the zero regions.
\begin{figure}
\center
\includegraphics[width=\textwidth]{figures/syn_sgd_cd_expl}
\caption{Modelling the data generated by Equations \ref{eq:data_sin} (top) and \ref{eq:data_gauss} (bottom) using appropriate regularization and standard KRR/KGD.
For the first data set, K$\ell_\infty$R/KSGD are less affected by the outliers than KRR/KGD are, and KGD is more affected by the outliers than KRR is.
For the second data set, in contrast to KRR/KGD, K$\ell_1$R/KCD are able to model the peak without incorporating the noise.}
\label{fig:other_reg}
\end{figure}
\newpage
\section{Conclusions}
We introduced an equivalent formulation of kernel ridge regression and used it to define kernel regression with the $\ell_\infty$ and $\ell_1$ penalties, and for solving kernel regression with gradient-based optimization methods.
We introduced the methods kernel sign gradient descent and kernel coordinate descent and utilized the similarities between $\ell_\infty$ regularization and sign gradient descent, and between $\ell_1$ regularization and coordinate descent (forward stagewise regression), to obtain computationally efficient algorithms for sparse and robust kernel regression, respectively.
We theoretically analyzed the similarities between kernel gradient descent, kernel sign gradient descent, and coordinate descent and the corresponding explicitly regularized kernel regression methods, and compared the methods on real and synthetic data.
Our generalizations of kernel ridge regression, together with regularization through early stopping, enable computationally efficient, kernelized robust, and sparse, regression.
Although we investigated only kernel regression with the $\ell_2$, $\ell_1$, and $\ell_\infty$ penalties, many other penalties could be considered, such as the adaptive lasso \citep{zou2006adaptive}, the group lasso \citep{yuan2006model}, the exclusive lasso \citep{zhou2010exclusive} and OSCAR \citep{bondell2008simultaneous}. This is, however, left for future work.
Code is available at \url{https://github.com/allerbo/gradient_based_kernel_regression}.
\newpage
\clearpage
\newpage
\bibliography{refs}
\bibliographystyle{apalike}
\newpage
\appendix
\section{Calculations for \texorpdfstring{$[\fhv^\top, \fhsv{^\top}]^\top$}{[f, f*]}}
\label{sec:equ_f}
In this section, we revisit some of the calculations in the main manuscript and reformulate them directly in terms of $[\fhv^\top, \fhsv{^\top}]^\top$, rather than obtaining them by multiplying $\alphav$ by $[\K^\top, \Ks{^\top}]^\top$.
\subsection{Equivalent Formulations of Kernel Ridge Regression}
In Lemma \ref{thm:equ_f}, we present three different objective functions, that all render the same solution for $[\fhv^\top, \fhsv{^\top}]^\top$ as in Equation \ref{eq:krr_s_fs}. When expressing KRR in terms of $\alphav\in\R^n$, it is enough to include $\K\in\R^{n\times n}$ in the expression to capture the dynamics of the kernel. However, when expressing KRR in terms of $[\fhv^\top, \fhsv{^\top}]^\top\in \R^{n+\ns}$, we need to introduce the larger kernel matrix
$$\Kss:=\begin{bmatrix}\K(\X,\X) & \K(\X,\Xs)\\ \K(\Xs,\X) & \K(\Xs,\Xs)\end{bmatrix}
\in \R^{(n+\ns)\times(n+\ns)}$$
in order to let the kernel affect not only $\fv\in \R^n$, but also $\fsv\in \R^{\ns}$.
Furthermore, we need the extended response vector $[\yv^\top,\ytv^\top]^\top$, where $\ytv$ is a copy of $\fsv$, so that $\ytv-\fsv=\nv$.
\begin{lemma}
\label{thm:equ_f}
\begin{subequations}
\label{eq:equ_f}
\begin{align}
\begin{bmatrix}\fhv\\\fhsv\end{bmatrix}=&\argmin_{[\fv^\top,\fsv{^\top}]^\top\in \R^{n+\ns}}\frac12\left\|\yv-\fv\right\|_2^2+\frac\lambda2\left\|\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|^2_{(\Kss)^{-1}}\label{eq:equ_f1}\\
=&\argmin_{[\fv^\top,\fsv{^\top}]^\top\in \R^{n+\ns}}\frac12\left\|\begin{bmatrix}\yv\\\ytv\end{bmatrix}-\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|_2^2+\frac\lambda2\left\|\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|^2_{(\Kss)^{-1}}\label{eq:equ_f1a}\\
=&\argmin_{[\fv^\top,\fsv{^\top}]^\top\in \R^{n+\ns}}\frac12\left\|\begin{bmatrix}\yv\\\ytv\end{bmatrix}-\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|_{\Kss}^2+\frac\lambda2\left\|\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|^2_2\label{eq:equ_f2}\\
=&\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\K+\lambda\I\right)^{-1}\yv.\notag
\end{align}
\end{subequations}
\end{lemma}
\textbf{Remark 1:} We assert that $\fsv$ does not affect the reconstruction error by requiring that $\ytv-\fsv=\nv$, We may, however, not define $\ytv$ to equal $\fsv$, since we do not want $\ytv$ to be considered when differentiating Equations \ref{eq:equ_f1a} and \ref{eq:equ_f2} with respect to $\fsv$.\\
\textbf{Remark 2:} The term $\left\|\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|^2_{(\Kss)^{-1}}$, might appear a bit peculiar, including the inverse of the extended kernel matrix. However, it is very closely related to the expression for the reproducing kernel Hilbert space norm, $\|f\|^2_{\Hh_k}$, obtained when expressing the functions $f$ and $k$ in terms of the orthogonal basis given by Mercer's Theorem:
$f(\xv)=\sum_{i=1}^\infty f_i\phi_i(\xv)$ and $k(\xv,\xpv)=\sum_{i=1}^\infty k_i\phi_i(\xv)\phi_i(\xpv)$, with $\|f\|^2_{\Hh_k}:=\sum_{i=1}^\infty \frac{f_i^2}{k_i}$. Defining the vector $\bm{\tilde{f}}$ and the diagonal matrix $\bm{\tilde{K}}$ according to $\bm{\tilde{f}}_i=f_i$ and $\bm{\tilde{K}}_{ii}=k_i$, we obtain $\|f\|^2_{\Hh_k}=\sum_{i=1}^\infty \frac{f_i^2}{k_i}=\bm{\tilde{f}}^\top\bm{\tilde{K}}^{-1}\bm{\tilde{f}}=\|\bm{\tilde{f}}\|_{\bm{\tilde{K}}^{-1}}^2$.
\subsection{Kernel Gradient Descent and Kernel Gradient Flow}
The gradient of Equation \ref{eq:equ_f2} with respect to $[\fv^\top,\fsv{^\top}]^\top$ is
$$\begin{bmatrix}\K\\\Ks\end{bmatrix}(\fv-\yv)+\lambda\begin{bmatrix}\fv\\\fsv\end{bmatrix},$$
which coincides with Equation \ref{eq:kgd_grad_f}, as expected. Thus, the KGD update in $[\fhv^\top,\fhsv{^\top}]^\top$ is
\begin{equation*}
\label{eq:kgd_update_f}
\begin{bmatrix}\fhv_{k+1}\\\fhsv_{k+1}\end{bmatrix}=\begin{bmatrix}\fhv_k\\\fhsv_k\end{bmatrix}+\eta\cdot\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\yv-\fhv_k\right),\ \begin{bmatrix}\fhv_0\\\fhsv_0\end{bmatrix}=\nv,
\end{equation*}
with the corresponding differential equation
\begin{equation}
\label{eq:kgd_diff_eq_f}
\frac{\partial}{\partial t}\left(\begin{bmatrix}\fhv(t)\\\fhsv(t)\end{bmatrix}\right)=\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\yv-\fhv(t)\right),\ \begin{bmatrix}\fhv(0)\\\fhsv(0)\end{bmatrix}=\nv,
\end{equation}
whose solution is stated in Lemma \ref{thm:kgf_s_f}.
\begin{lemma}
\label{thm:kgf_s_f}~\\
The solution to the differential equation in Equation \ref{eq:kgd_diff_eq_f} is
\begin{equation*}
\label{eq:kgf_s_f}
\begin{bmatrix}\fhv(t)\\\fhsv(t)\end{bmatrix}=\begin{bmatrix}\I\\\Ks\K^{-1}\end{bmatrix}(\I-\exp(-t\K))\yv
\end{equation*}
\end{lemma}
\subsection{Kernel Regression with the \texorpdfstring{$\ell_\infty$}{linf} and \texorpdfstring{$\ell_1$}{l1} Norms}
%\label{sec:l1linf}
If we replace the squared $\ell_2$ norm in Equation \ref{eq:equ_f2} by the $\ell_\infty$ and $\ell_1$ norms, we obtain
\begin{equation}
\label{eq:finf}
\argmin_{[\fv^\top,\fsv{^\top}]^\top\in \R^{n+\ns}}\frac12\left\|\begin{bmatrix}\yv\\\ytv\end{bmatrix}-\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|_{\Kss}^2+\lambda\left\|\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|_\infty
\end{equation}
and
\begin{equation}
\label{eq:f1}
\argmin_{[\fv^\top,\fsv{^\top}]^\top\in \R^{n+\ns}}\frac12\left\|\begin{bmatrix}\yv\\\ytv\end{bmatrix}-\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|_{\Kss}^2+\lambda\left\|\begin{bmatrix}\fv\\\fsv\end{bmatrix}\right\|_1.
\end{equation}
In contrast to ridge regression, where the formulations in $\alphav$ and $[\fv^\top,\fsv{^\top}]^\top$ are equivalent in the sense that the latter can always be obtained from the first through multiplication with $[\K^\top, \Ks{^\top}]^\top$, the solutions of Equations \ref{eq:finf} and \ref{eq:f1} cannot be obtained from the solutions of Equations \ref{eq:alphainf} and \ref{eq:alpha1}.
Since $\ell_\infty$ regularization promotes a solution where the parameters have equal absolute values, many of the predictions in the solution to Equation \ref{eq:finf} are exactly equal.
For $\ell_1$ regularization, which promotes a sparse solution with the most significant parameters included first, in the solution to Equation \ref{eq:f1}, many of the predictions equal exactly zero, with only the most extreme predictions being non-zero.
In Figure \ref{fig:comp_f}, we exemplify this for explicit regularization for two different regularization strengths. Even though it is technically possible to use gradient-based optimization with early stopping for $[\fv^\top,\fsv{^\top}]^\top$, since the reconstruction error, and thus the gradient, is affected only by $\fv$ and not by $\fsv$, this does not work well for the sign gradient and coordinate descent algorithms.
\begin{figure}
\center
\includegraphics[width=\textwidth]{figures/compare_pen_f}
\caption{Comparisons of the effects of $\ell_2$, $\ell_\infty$, and $\ell_1$ regularization on $[\fv^\top,\fsv{^\top}]^\top$.\\
In the top panel, a larger regularization is used than in the bottom panel.\\
For $\ell_\infty$ regularization, the absolute values of many predictions are exactly equal. To obtain this, compared to the non-regularized solution, some predictions are shifted away from zero, while some predictions are shifted toward zero.
For $\ell_1$ regularization, many predictions are set exactly to zero, with peaks at some extreme observations.}
\label{fig:comp_f}
\end{figure}
Finally, in Proposition \ref{thm:sgn_inf_f}, we present the analogue of Proposition \ref{thm:sgn_inf} for $[\fv^\top,\fsv{^\top}]^\top$.
\begin{prop}~\\
\label{thm:sgn_inf_f}
For $\fpv:=[\fv^\top,\fsv{^\top}]^\top$ and $\ypv:=[\yv^\top,\ytv^\top]^\top$, where $\ytv-\fsv=\nv$, let $\fhpinfv(c)$ denote the solution to
$$\min_{\fpv\in\R^{n+\ns}}\left\|\ypv-\fpv\right\|_{\Kss}^2 \text{ s.t. } \|\fpv\|_\infty\leq c,$$
and let $\fhpsgfv(t)$ denote the solution to
$$\frac{\partial\fpv(t)}{\partial t}=\sgn\left(-\frac{\partial}{\partial\fpv(t)}\left(\left\|\ypv-\fpv(t)\right\|_{\Kss}^2\right)\right),\ \fpv(0)=\nv.$$
When $\Kss$ is diagonal, with elements $\{k_{ii}\}_{i=1}^{n+\ns}$, $k_{ii}>0$, the two solutions decompose element-wise and coincide for $c=t$:
$$\fhpinfi(t)=\fhpsgfi(t)=
\begin{cases}
\sgn(y_i)\cdot\min(t,|y_i|) &\text{if }i\leq n\\
0 &\text{if }i> n.
\end{cases}$$
\end{prop}
\section{Additional Experiments}
\label{sec:more_exps}
In this section, we extend the results of Tables \ref{tab:sgd} and \ref{tab:cd} by repeating the experiments for the four kernels in Table \ref{tab:kerns}. The results are presented in Tables \ref{tab:sgd2}, \ref{tab:sgd3}, \ref{tab:cd2}, and \ref{tab:cd3}, and agree with those of Section \ref{sec:exps}.
\begin{table}
\caption{Additional kernels used}
\center
\begin{tabular}{|l|l|}
\hline
Name & Equation \\
\hline
Matérn, $\nu=\frac12$ (Laplace) & $\exp\left(-\frac{\|\xv-\xpv\|_2}\sigma\right)$\\
Matérn, $\nu=\frac32$ & $\left(1+\frac{\sqrt{3}\cdot\|\xv-\xpv\|_2}\sigma\right)\cdot\exp\left(-\frac{\sqrt{3}\cdot\|\xv-\xpv\|_2}\sigma\right)$\\
Matérn, $\nu=\frac52$ & $\left(1+\frac{\sqrt{5}\cdot\|\xv-\xpv\|_2}\sigma+\frac{5\cdot\|\xv-\xpv\|^2_2}{3\cdot\sigma^2}\right)\cdot\exp\left(-\frac{\sqrt{5}\cdot\|\xv-\xpv\|_2}\sigma\right)$\\
Cauchy & $\left(1+\frac{\|\xv-\xpv\|_2^2}{\sigma^2}\right)^{-1}$\\
\hline
\end{tabular}
\label{tab:kerns}
\end{table}
\begin{table}
\caption{Median and first and third quartile of test $R^2$ and computation time in seconds for KSGD, K$\ell_\infty$R, KGD, and KRR. Occasions where the algorithm performs significantly worse in terms of $R^2$/computation time than KSGD (at significance level 0.05) are marked with bold. KSGD is significantly faster than K$\ell_\infty$R. K$\ell_\infty$R/KSGD are less sensitive to outliers than KRR/KGD are and thus perform significantly better. KGD is more sensitive to outliers than KRR is.}
\center
\begin{tabular}{|l|l|l|l|l|}
\hline
Kernel & Data & Method & $R^2$ & Computation Time [s]\\
\hline
\multirow{20}{*}{\makecell{Matérn,\\$\nu=1/2$\\(Laplace)}} & \multirow{4}{*}{Synthetic} & KSGD & $0.93,\ (0.91, 0.95)$ & $107,\ (87, 126)$ \\
& & K$\ell_\infty$R & $\bm{0.89,\ (0.83, 0.92)}$ & $\bm{1973,\ (1922, 2058)}$ \\
& & KGD & $\bm{0.07,\ (-1.52, 0.64)}$ & $\bm{1899,\ (801, 5293)}$ \\
& & KRR & $\bm{0.46,\ (0.11, 0.75)}$ & $10,\ (10, 10)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KSGD & $0.42,\ (0.20, 0.50)$ & $237,\ (192, 294)$ \\
& & K$\ell_\infty$R & $0.41,\ (0.17, 0.54)$ & $\bm{3551,\ (3352, 3857)}$ \\
& & KGD & $\bm{0.31,\ (0.06, 0.48)}$ & $\bm{2079,\ (666, 5314)}$ \\
& & KRR & $0.37,\ (0.09, 0.51)$ & $125,\ (124, 127)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{California\\Housing}} & KSGD & $0.46,\ (0.23, 0.57)$ & $2476,\ (1845, 3089)$ \\
& & K$\ell_\infty$R & $0.42,\ (0.21, 0.58)$ & $\bm{5576,\ (5483, 5722)}$ \\
& & KGD & $\bm{0.27,\ (0.03, 0.47)}$ & $\bm{4644,\ (3641, 6369)}$ \\
& & KRR & $\bm{0.32,\ (0.11, 0.51)}$ & $125,\ (124, 127)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KSGD & $0.11,\ (0.04, 0.16)$ & $661,\ (523, 918)$ \\
& & K$\ell_\infty$R & $0.10,\ (0.04, 0.17)$ & $\bm{4934,\ (4748, 5124)}$ \\
& & KGD & $\bm{0.00,\ (-0.20, 0.07)}$ & $\bm{3822,\ (2998, 5551)}$ \\
& & KRR & $\bm{0.03,\ (0.00, 0.08)}$ & $115,\ (107, 123)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KSGD & $0.16,\ (0.09, 0.24)$ & $3905,\ (3174, 4869)$ \\
& & K$\ell_\infty$R & $0.16,\ (0.05, 0.25)$ & $\bm{6206,\ (5759, 6334)}$ \\
& & KGD & $\bm{0.01,\ (-0.18, 0.11)}$ & $\bm{4582,\ (2334, 9255)}$ \\
& & KRR & $\bm{0.09,\ (0.01, 0.18)}$ & $115,\ (105, 122)$ \\
\hline
\multirow{20}{*}{\makecell{Matérn,\\$\nu=3/2$}} & \multirow{4}{*}{Synthetic} & KSGD & $0.95,\ (0.91, 0.97)$ & $96,\ (77, 127)$ \\
& & K$\ell_\infty$R & $\bm{0.92,\ (0.87, 0.95)}$ & $\bm{1985,\ (1925, 2050)}$ \\
& & KGD & $\bm{-0.12,\ (-2.38, 0.73)}$ & $\bm{1405,\ (449, 2886)}$ \\
& & KRR & $\bm{0.47,\ (-0.12, 0.80)}$ & $12,\ (12, 12)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KSGD & $0.38,\ (0.15, 0.49)$ & $218,\ (189, 265)$ \\
& & K$\ell_\infty$R & $0.40,\ (0.13, 0.51)$ & $\bm{3588,\ (3363, 3810)}$ \\
& & KGD & $\bm{0.28,\ (0.08, 0.44)}$ & $\bm{13090,\ (3957, 21178)}$ \\
& & KRR & $0.34,\ (0.08, 0.49)$ & $139,\ (132, 141)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{California\\Housing}} & KSGD & $0.43,\ (0.23, 0.57)$ & $2083,\ (1635, 2574)$ \\
& & K$\ell_\infty$R & $0.41,\ (0.18, 0.57)$ & $\bm{5542,\ (5468, 5728)}$ \\
& & KGD & $\bm{0.23,\ (0.01, 0.45)}$ & $\bm{4746,\ (3590, 6641)}$ \\
& & KRR & $\bm{0.33,\ (0.14, 0.51)}$ & $141,\ (140, 143)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KSGD & $0.10,\ (0.03, 0.15)$ & $553,\ (420, 770)$ \\
& & K$\ell_\infty$R & $0.10,\ (0.03, 0.17)$ & $\bm{4953,\ (4781, 5207)}$ \\
& & KGD & $\bm{0.00,\ (-0.32, 0.08)}$ & $\bm{4181,\ (3305, 6687)}$ \\
& & KRR & $\bm{0.04,\ (0.00, 0.09)}$ & $135,\ (119, 139)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KSGD & $0.15,\ (0.07, 0.22)$ & $3178,\ (2684, 4081)$ \\
& & K$\ell_\infty$R & $0.15,\ (0.04, 0.24)$ & $\bm{6248,\ (5679, 6380)}$ \\
& & KGD & $\bm{0.00,\ (-0.29, 0.12)}$ & $\bm{5245,\ (3036, 10291)}$ \\
& & KRR & $\bm{0.09,\ (0.00, 0.17)}$ & $126,\ (112, 138)$ \\
\hline
\end{tabular}
\label{tab:sgd2}
\end{table}
\begin{table}
\caption{Median and first and third quartile of test $R^2$ and computation time in seconds for KSGD, K$\ell_\infty$R, KGD, and KRR. Occasions where the algorithm performs significantly worse in terms of $R^2$/computation time than KSGD (at significance level 0.05) are marked with bold. KSGD is significantly faster than K$\ell_\infty$R. K$\ell_\infty$R/KSGD are less sensitive to outliers than KRR/KGD are and thus perform significantly better. KGD is more sensitive to outliers than KRR is.}
\center
\begin{tabular}{|l|l|l|l|l|}
\hline
Kernel & Data & Method & $R^2$ & Computation Time [s]\\
\hline
\multirow{20}{*}{\makecell{Matérn,\\$\nu=5/2$}} & \multirow{4}{*}{Synthetic} & KSGD & $0.95,\ (0.92, 0.97)$ & $87,\ (69, 105)$ \\
& & K$\ell_\infty$R & $\bm{0.93,\ (0.88, 0.95)}$ & $\bm{1987,\ (1917, 2055)}$ \\
& & KGD & $\bm{-0.15,\ (-2.52, 0.73)}$ & $\bm{1682,\ (505, 3472)}$ \\
& & KRR & $\bm{0.51,\ (-0.18, 0.81)}$ & $13,\ (13, 13)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KSGD & $0.37,\ (0.16, 0.48)$ & $222,\ (186, 261)$ \\
& & K$\ell_\infty$R & $0.38,\ (0.12, 0.49)$ & $\bm{3573,\ (3243, 3838)}$ \\
& & KGD & $\bm{0.28,\ (0.07, 0.43)}$ & $\bm{12770,\ (6819, 24139)}$ \\
& & KRR & $0.34,\ (0.08, 0.47)$ & $150,\ (132, 152)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{California\\Housing}} & KSGD & $0.42,\ (0.22, 0.57)$ & $1838,\ (1459, 2270)$ \\
& & K$\ell_\infty$R & $0.41,\ (0.18, 0.57)$ & $\bm{5569,\ (5484, 5693)}$ \\
& & KGD & $\bm{0.22,\ (0.01, 0.43)}$ & $\bm{5067,\ (3789, 7551)}$ \\
& & KRR & $\bm{0.32,\ (0.12, 0.50)}$ & $152,\ (150, 153)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KSGD & $0.10,\ (0.03, 0.15)$ & $520,\ (399, 715)$ \\
& & K$\ell_\infty$R & $0.10,\ (0.02, 0.17)$ & $\bm{4955,\ (4700, 5117)}$ \\
& & KGD & $\bm{0.00,\ (-0.30, 0.08)}$ & $\bm{4529,\ (3437, 6734)}$ \\
& & KRR & $\bm{0.03,\ (-0.01, 0.10)}$ & $137,\ (117, 148)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KSGD & $0.15,\ (0.07, 0.22)$ & $3048,\ (2461, 3806)$ \\
& & K$\ell_\infty$R & $0.15,\ (0.04, 0.24)$ & $\bm{6179,\ (5485, 6385)}$ \\
& & KGD & $\bm{0.00,\ (-0.39, 0.13)}$ & $\bm{5605,\ (3318, 10466)}$ \\
& & KRR & $\bm{0.08,\ (0.00, 0.18)}$ & $135,\ (117, 147)$ \\
\hline
\multirow{20}{*}{Cauchy} & \multirow{4}{*}{Synthetic} & KSGD & $0.95,\ (0.91, 0.97)$ & $88,\ (69, 102)$ \\
& & K$\ell_\infty$R & $\bm{0.92,\ (0.86, 0.94)}$ & $\bm{1945,\ (1899, 2020)}$ \\
& & KGD & $\bm{-0.12,\ (-2.26, 0.69)}$ & $\bm{1607,\ (381, 3240)}$ \\
& & KRR & $\bm{0.47,\ (-0.12, 0.84)}$ & $10,\ (10, 10)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KSGD & $0.38,\ (0.17, 0.48)$ & $256,\ (217, 315)$ \\
& & K$\ell_\infty$R & $0.39,\ (0.13, 0.50)$ & $\bm{3574,\ (3267, 3819)}$ \\
& & KGD & $\bm{0.25,\ (0.05, 0.43)}$ & $\bm{12929,\ (6800, 21247)}$ \\
& & KRR & $0.32,\ (0.08, 0.47)$ & $115,\ (103, 118)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{California\\Housing}} & KSGD & $0.46,\ (0.22, 0.57)$ & $2145,\ (1626, 2857)$ \\
& & K$\ell_\infty$R & $0.41,\ (0.18, 0.57)$ & $\bm{5575,\ (5469, 5727)}$ \\
& & KGD & $\bm{0.23,\ (0.00, 0.43)}$ & $\bm{4535,\ (3079, 7122)}$ \\
& & KRR & $\bm{0.32,\ (0.15, 0.50)}$ & $118,\ (117, 119)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KSGD & $0.10,\ (0.04, 0.15)$ & $607,\ (454, 891)$ \\
& & K$\ell_\infty$R & $0.10,\ (0.03, 0.16)$ & $\bm{4880,\ (4519, 5138)}$ \\
& & KGD & $\bm{0.00,\ (-0.21, 0.08)}$ & $\bm{4439,\ (2465, 7006)}$ \\
& & KRR & $\bm{0.05,\ (-0.01, 0.10)}$ & $106,\ (91, 115)$ \\
\cline{2-5}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KSGD & $0.16,\ (0.09, 0.22)$ & $3447,\ (2462, 4541)$ \\
& & K$\ell_\infty$R & $0.16,\ (0.04, 0.24)$ & $\bm{6216,\ (5447, 6428)}$ \\
& & KGD & $\bm{0.00,\ (-0.41, 0.14)}$ & $\bm{4904,\ (2424, 11329)}$ \\
& & KRR & $\bm{0.10,\ (0.00, 0.18)}$ & $105,\ (91, 115)$ \\
\hline
\end{tabular}
\label{tab:sgd3}
\end{table}
\begin{table}
\caption{Median and first and third quartile of test $R^2$, computation time in seconds, and sparsity for KCD, K$\ell_1$R, KGD, and KRR. Occasions where the algorithm performs significantly worse in terms of $R^2$/computation time/sparsity than KCD (at significance level 0.05) are marked with bold. In contrast to KRR/KGD, K$\ell_1$R/KCD/KEGD only includes a fraction of the observations in the model. It is not obvious which of the K$\ell_1$R and KCD algorithms performs better. KEGD generally outperforms K$\ell_1$R/KCD in terms of computation time.}
\center
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Kernel & Data & Method & $R^2$ & Computation Time [s] & Sparsity\\
\hline
\multirow{25}{*}{\makecell{Matérn,\\$\nu=1/2$\\(Laplace)}} & \multirow{4}{*}{Synthetic} & KCD & $0.88,\ (0.81, 0.91)$ & $342,\ (277, 419)$ & $0.08,\ (0.04, 0.11)$ \\
& & K$\ell_1$R & $0.89,\ (0.85, 0.91)$ & $\bm{430,\ (412, 453)}$ & $\bm{0.83,\ (0.51, 0.95)}$ \\
& & KGD & $\bm{0.85,\ (0.79, 0.88)}$ & $\bm{879,\ (294, 3613)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.86,\ (0.82, 0.88)$ & $10,\ (10, 10)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.86,\ (0.77, 0.91)}$ & $\mathit{103,\ (85, 125)}$ & $\mathit{0.09,\ (0.02, 0.19)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KCD & $0.05,\ (0.01, 0.08)$ & $309,\ (220, 440)$ & $0.01,\ (0.01, 0.02)$ \\
& & K$\ell_1$R & $\bm{0.02,\ (0.00, 0.06)}$ & $180,\ (95, 268)$ & $0.01,\ (0.00, 0.01)$ \\
& & KGD & $0.62,\ (0.56, 0.68)$ & $\bm{5503,\ (2381, 14392)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.63,\ (0.57, 0.68)$ & $124,\ (122, 125)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.05,\ (0.02, 0.08)}$ & $\mathit{77,\ (53, 124)}$ & $\mathit{0.01,\ (0.01, 0.02)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{California\\Housing}} & KCD & $0.44,\ (0.37, 0.50)$ & $55212,\ (48025, 58912)$ & $0.62,\ (0.54, 0.70)$ \\
& & K$\ell_1$R & $0.45,\ (0.40, 0.50)$ & $499,\ (477, 532)$ & $0.31,\ (0.29, 0.34)$ \\
& & KGD & $0.72,\ (0.67, 0.75)$ & $13669,\ (8550, 17961)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.72,\ (0.68, 0.76)$ & $113,\ (107, 119)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.49,\ (0.42, 0.55)}$ & $\mathit{481,\ (437, 511)}$ & $\mathit{0.82,\ (0.72, 0.89)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KCD & $0.00,\ (-0.03, 0.03)$ & $16344,\ (8759, 24215)$ & $0.12,\ (0.07, 0.19)$ \\
& & K$\ell_1$R & $0.32,\ (0.26, 0.35)$ & $1134,\ (1023, 1205)$ & $\bm{0.93,\ (0.93, 0.94)}$ \\
& & KGD & $0.33,\ (0.27, 0.40)$ & $14663,\ (7759, 22181)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.35,\ (0.27, 0.40)$ & $111,\ (98, 119)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.14,\ (0.10, 0.20)}$ & $\mathit{762,\ (524, 1040)}$ & $\mathit{0.67,\ (0.51, 0.77)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KCD & $0.05,\ (0.02, 0.08)$ & $6515,\ (3977, 8859)$ & $0.17,\ (0.14, 0.21)$ \\
& & K$\ell_1$R & $0.08,\ (0.04, 0.10)$ & $361,\ (309, 410)$ & $0.17,\ (0.15, 0.17)$ \\
& & KGD & $0.13,\ (0.08, 0.22)$ & $6342,\ (4535, 9949)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.17,\ (0.11, 0.23)$ & $119,\ (107, 124)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.09,\ (0.05, 0.13)}$ & $\mathit{240,\ (196, 283)}$ & $\mathit{0.28,\ (0.18, 0.50)}$ \\
\hline
\multirow{25}{*}{\makecell{Matérn,\\$\nu=3/2$}} & \multirow{4}{*}{Synthetic} & KCD & $0.91,\ (0.84, 0.95)$ & $292,\ (222, 361)$ & $0.04,\ (0.02, 0.09)$ \\
& & K$\ell_1$R & $0.91,\ (0.87, 0.93)$ & $\bm{445,\ (419, 457)}$ & $\bm{0.57,\ (0.36, 0.83)}$ \\
& & KGD & $\bm{0.84,\ (0.76, 0.88)}$ & $\bm{441,\ (241, 1438)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $\bm{0.86,\ (0.83, 0.89)}$ & $12,\ (12, 12)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.89,\ (0.81, 0.94)}$ & $\mathit{92,\ (74, 116)}$ & $\mathit{0.04,\ (0.01, 0.11)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KCD & $0.05,\ (0.01, 0.10)$ & $250,\ (163, 366)$ & $0.01,\ (0.01, 0.02)$ \\
& & K$\ell_1$R & $\bm{0.03,\ (0.00, 0.07)}$ & $182,\ (109, 263)$ & $0.01,\ (0.00, 0.01)$ \\
& & KGD & $0.62,\ (0.55, 0.67)$ & $\bm{27681,\ (18322, 38318)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.63,\ (0.56, 0.68)$ & $126,\ (123, 138)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.05,\ (0.01, 0.09)}$ & $\mathit{60,\ (43, 100)}$ & $\mathit{0.01,\ (0.01, 0.02)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{California\\Housing}} & KCD & $0.43,\ (0.36, 0.50)$ & $47819,\ (41571, 52410)$ & $0.50,\ (0.41, 0.57)$ \\
& & K$\ell_1$R & $0.47,\ (0.43, 0.53)$ & $505,\ (447, 539)$ & $0.28,\ (0.26, 0.31)$ \\
& & KGD & $0.71,\ (0.67, 0.75)$ & $10343,\ (8197, 14302)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.72,\ (0.68, 0.75)$ & $122,\ (108, 134)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.51,\ (0.43, 0.57)}$ & $\mathit{571,\ (509, 678)}$ & $\mathit{0.77,\ (0.69, 0.89)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KCD & $0.00,\ (-0.03, 0.02)$ & $10637,\ (5671, 17015)$ & $0.07,\ (0.04, 0.11)$ \\
& & K$\ell_1$R & $0.31,\ (0.25, 0.36)$ & $1214,\ (1081, 1276)$ & $\bm{0.93,\ (0.93, 0.94)}$ \\
& & KGD & $0.29,\ (0.22, 0.37)$ & $\bm{13293,\ (7937, 17890)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.31,\ (0.24, 0.37)$ & $131,\ (114, 140)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.13,\ (0.08, 0.17)}$ & $\mathit{800,\ (574, 1090)}$ & $\mathit{0.56,\ (0.42, 0.69)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KCD & $0.05,\ (0.01, 0.08)$ & $5306,\ (3101, 7467)$ & $0.14,\ (0.09, 0.18)$ \\
& & K$\ell_1$R & $0.09,\ (0.05, 0.12)$ & $375,\ (330, 415)$ & $\bm{0.16,\ (0.14, 0.17)}$ \\
& & KGD & $0.13,\ (0.09, 0.23)$ & $\bm{5741,\ (4487, 8062)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.17,\ (0.11, 0.23)$ & $134,\ (121, 139)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.08,\ (0.03, 0.14)}$ & $\mathit{251,\ (196, 312)}$ & $\mathit{0.28,\ (0.16, 0.46)}$ \\
\hline
\end{tabular}
\label{tab:cd2}
\end{table}
\begin{table}
\caption{Median and first and third quartile of test $R^2$, computation time in seconds, and sparsity for KCD, K$\ell_1$R, KGD, and KRR. Occasions where the algorithm performs significantly worse in terms of $R^2$/computation time/sparsity than KCD (at significance level 0.05) are marked with bold. In contrast to KRR/KGD, K$\ell_1$R/KCD/KEGD only includes a fraction of the observations in the model. It is not obvious which of the K$\ell_1$R and KCD algorithms performs better. KEGD generally outperforms K$\ell_1$R/KCD in terms of computation time.}
\center
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Kernel & Data & Method & $R^2$ & Computation Time [s] & Sparsity\\
\hline
\multirow{25}{*}{\makecell{Matérn,\\$\nu=5/2$}} & \multirow{4}{*}{Synthetic} & KCD & $0.92,\ (0.86, 0.95)$ & $288,\ (219, 364)$ & $0.04,\ (0.02, 0.07)$ \\
& & K$\ell_1$R & $0.91,\ (0.87, 0.93)$ & $\bm{442,\ (423, 458)}$ & $\bm{0.54,\ (0.33, 0.82)}$ \\
& & KGD & $\bm{0.83,\ (0.77, 0.88)}$ & $\bm{535,\ (284, 1387)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $\bm{0.86,\ (0.82, 0.89)}$ & $13,\ (13, 13)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.90,\ (0.81, 0.94)}$ & $\mathit{93,\ (74, 125)}$ & $\mathit{0.03,\ (0.01, 0.10)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KCD & $0.06,\ (0.02, 0.10)$ & $219,\ (146, 335)$ & $0.01,\ (0.01, 0.01)$ \\
& & K$\ell_1$R & $\bm{0.03,\ (0.00, 0.08)}$ & $203,\ (104, 270)$ & $0.01,\ (0.00, 0.01)$ \\
& & KGD & $0.60,\ (0.52, 0.65)$ & $\bm{30525,\ (20442, 41275)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.62,\ (0.55, 0.67)$ & $136,\ (130, 149)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.05,\ (0.02, 0.08)}$ & $\mathit{58,\ (42, 89)}$ & $\mathit{0.01,\ (0.01, 0.02)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{California\\Housing}} & KCD & $0.42,\ (0.36, 0.48)$ & $44890,\ (38451, 49298)$ & $0.45,\ (0.39, 0.52)$ \\
& & K$\ell_1$R & $0.48,\ (0.44, 0.54)$ & $485,\ (434, 540)$ & $0.28,\ (0.26, 0.30)$ \\
& & KGD & $0.70,\ (0.66, 0.74)$ & $13042,\ (8985, 16382)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.71,\ (0.67, 0.75)$ & $122,\ (108, 144)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.49,\ (0.44, 0.55)}$ & $\mathit{607,\ (535, 729)}$ & $\mathit{0.73,\ (0.60, 0.81)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KCD & $0.00,\ (-0.02, 0.02)$ & $8920,\ (4712, 13959)$ & $0.05,\ (0.02, 0.09)$ \\
& & K$\ell_1$R & $0.30,\ (0.25, 0.35)$ & $1189,\ (1105, 1264)$ & $\bm{0.93,\ (0.92, 0.94)}$ \\
& & KGD & $0.28,\ (0.22, 0.36)$ & $\bm{15025,\ (9936, 19893)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.31,\ (0.23, 0.37)$ & $136,\ (124, 147)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.12,\ (0.06, 0.16)}$ & $\mathit{725,\ (584, 1125)}$ & $\mathit{0.51,\ (0.31, 0.65)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KCD & $0.04,\ (0.01, 0.07)$ & $4811,\ (3082, 6991)$ & $0.13,\ (0.06, 0.17)$ \\
& & K$\ell_1$R & $0.09,\ (0.05, 0.12)$ & $379,\ (334, 417)$ & $\bm{0.16,\ (0.14, 0.17)}$ \\
& & KGD & $0.14,\ (0.08, 0.22)$ & $\bm{6069,\ (4422, 8783)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.17,\ (0.11, 0.23)$ & $139,\ (128, 151)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.07,\ (0.03, 0.13)}$ & $\mathit{246,\ (185, 315)}$ & $\mathit{0.24,\ (0.14, 0.39)}$ \\
\hline
\multirow{25}{*}{Cauchy} & \multirow{4}{*}{Synthetic} & KCD & $0.89,\ (0.82, 0.92)$ & $248,\ (206, 326)$ & $0.04,\ (0.02, 0.07)$ \\
& & K$\ell_1$R & $0.89,\ (0.85, 0.92)$ & $\bm{436,\ (419, 455)}$ & $\bm{0.66,\ (0.44, 0.89)}$ \\
& & KGD & $\bm{0.83,\ (0.78, 0.87)}$ & $\bm{409,\ (233, 929)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.86,\ (0.82, 0.88)$ & $10,\ (10, 11)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.87,\ (0.77, 0.92)}$ & $\mathit{88,\ (68, 111)}$ & $\mathit{0.05,\ (0.01, 0.11)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Aspen\\Fibres}} & KCD & $0.04,\ (0.01, 0.07)$ & $271,\ (194, 378)$ & $0.01,\ (0.01, 0.02)$ \\
& & K$\ell_1$R & $\bm{0.01,\ (0.00, 0.04)}$ & $174,\ (75, 235)$ & $0.01,\ (0.00, 0.01)$ \\
& & KGD & $0.58,\ (0.52, 0.65)$ & $\bm{26312,\ (18620, 37784)}$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.60,\ (0.53, 0.66)$ & $106,\ (101, 116)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.02,\ (0.00, 0.06)}$ & $\mathit{59,\ (40, 95)}$ & $\mathit{0.01,\ (0.01, 0.02)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{California\\Housing}} & KCD & $0.34,\ (0.29, 0.40)$ & $33838,\ (29575, 38592)$ & $0.58,\ (0.47, 0.69)$ \\
& & K$\ell_1$R & $0.47,\ (0.43, 0.53)$ & $475,\ (425, 509)$ & $0.29,\ (0.27, 0.32)$ \\
& & KGD & $0.70,\ (0.66, 0.74)$ & $11129,\ (7946, 14170)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.71,\ (0.68, 0.75)$ & $103,\ (91, 112)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.36,\ (0.30, 0.43)}$ & $\mathit{295,\ (270, 335)}$ & $\mathit{0.60,\ (0.48, 0.71)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{Protein\\Structure}} & KCD & $0.01,\ (-0.03, 0.03)$ & $25031,\ (16247, 35956)$ & $0.20,\ (0.09, 0.32)$ \\
& & K$\ell_1$R & $0.31,\ (0.25, 0.35)$ & $1084,\ (984, 1206)$ & $\bm{0.93,\ (0.92, 0.94)}$ \\
& & KGD & $0.29,\ (0.21, 0.37)$ & $13858,\ (8677, 18424)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.31,\ (0.24, 0.37)$ & $99,\ (89, 108)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.14,\ (0.10, 0.18)}$ & $\mathit{670,\ (504, 871)}$ & $\mathit{0.72,\ (0.59, 0.79)}$ \\
\cline{2-6}
& \multirow{4}{*}{\makecell{U.K.\ Black\\Smoke}} & KCD & $0.05,\ (0.02, 0.07)$ & $6812,\ (4676, 10323)$ & $0.16,\ (0.14, 0.20)$ \\
& & K$\ell_1$R & $0.08,\ (0.05, 0.12)$ & $332,\ (289, 379)$ & $0.16,\ (0.14, 0.17)$ \\
& & KGD & $0.13,\ (0.06, 0.22)$ & $6219,\ (4868, 9396)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & KRR & $0.17,\ (0.11, 0.23)$ & $103,\ (91, 114)$ & $\bm{1.00,\ (1.00, 1.00)}$ \\
& & \textit{KEGD} & $\mathit{0.08,\ (0.03, 0.12)}$ & $\mathit{223,\ (184, 252)}$ & $\mathit{0.29,\ (0.18, 0.46)}$ \\
\hline
\end{tabular}
\label{tab:cd3}
\end{table}
\section{Proofs}
\label{sec:proofs}
\begin{proof}[Proof of Lemma \ref{thm:equ_alpha}]~\\
Differentiating Equation \ref{eq:equ_alpha2} with respect to $\alphav$ and setting the gradient to $\nv$, we obtain
$$\nv=\frac{\partial}{\partial \alphahv}\left(\frac12\left\|\yv-\K\alphahv\right\|_{\K^{-1}}^2+\frac\lambda2\|\alphahv\|^2_2\right)=-\yv+\K\alphahv+\lambda\alphahv=-\yv+(\K+\lambda\I)\alphahv\iff \alphahv=(\K+\lambda\I)^{-1}\yv.$$
\end{proof}
\begin{proof}[Proof of Lemma \ref{thm:kgf_s}]~\\
With $\etahv(t)=\K\alphahv(t)-\yv$, we obtain $\frac{d\etahv(t)}{dt}=\K\frac{d\alphahv(t)}{dt}$ and Equation \ref{eq:kgd_diff_eq} can be written as
$$\frac{d\etahv(t)}{dt}=-\K\etahv(t)\iff \etahv(t)=\exp\left(-t\K\right)\etahnv.$$
Now,
$$\alphahv(0)=\bm{0}\implies\etahnv=\etahv(0)=-\yv\implies \etahv(t)=-\exp(-t\K)\yv.$$
Solving for $\alphahv(t)$, we obtain
$$\alphahv(t)=\K^{-1}\left(\I-\exp(-t\K)\right)\yv=\left(\I-\exp(-t\K)\right)\K^{-1}\yv.$$
\end{proof}
\begin{proof}[Proof of Remark 3 of Lemma \ref{thm:kgf_s}]~\\
The differential equation in Equation \ref{eq:kgd_diff_eq} can be obtained by writing
$$\alphahv_{k+1}=\alphahv_k+\eta\cdot\left(\yv-\K\alphahv_k\right)$$
as
$$\alphahv(t+\Delta t)=\alphahv(t)+\Delta t\cdot\left(\yv-\K\alphahv(t)\right),$$
rearranging and letting $\Delta t\to 0$, to obtain
$$\frac{d\alphahv(t)}{dt}=\lim_{\Delta t\to 0}\left(\frac{\alphahv(t+\Delta t)-\alphahv(t)}{\Delta t}\right)=\yv-\K\alphahv(t).$$
For momentum and Nesterov accelerated gradient, the update rule for $\alphahv$ in Equation \ref{eq:kgd_update} generalizes to
$$\alphahv(t+\Delta t)=\alphahv(t)+\gamma\cdot\left(\alphahv(t)-\alphahv(t-\Delta t)\right)+\Delta t\cdot\left(\yv-\K\left(\alphahv(t)+\mathbb{I}_{\text{NAG}}\cdot\gamma\cdot\left(\alphahv(t)-\alphahv(t-\Delta t)\right)\right)\right),$$
where $\gamma\in[0,1)$ is the strength of the momentum and $\mathbb{I}_{\text{NAG}} \in \{0,1\}$ decides whether to use Nesterov accelerated gradient or not. Rearranging, we obtain
$$\frac{\alphahv(t+\Delta t)-\alphahv(t)}{\Delta t}-\gamma\cdot\frac{\alphahv(t)-\alphahv(t-\Delta t)}{\Delta t}=\yv-\K\left(\alphahv(t)+\mathbb{I}_{\text{NAG}}\cdot\gamma\cdot\left(\alphahv(t)-\alphahv(t-\Delta t)\right)\right),$$
and, when $\Delta t \to 0$,
$$(1-\gamma)\cdot\frac{d\alphahv(t)}{d t}=\yv-\K\alphahv(t).$$
Solving the differential equations in the same way as in Lemma \ref{thm:kgf_s}, we obtain
$$\alphahv(t)=\left(\I-\exp\left(-\frac t{1-\gamma}\K\right)\right)\K^{-1}\yv.$$
\end{proof}
\begin{proof}[Proof of Lemma \ref{thm:rewrite_alpha_krr}]
\begin{equation*}
\begin{aligned}
\left(\K+\lambda\I\right)^{-1}&=\left(\K+\lambda\I\right)^{-1}\underbrace{(\K+\lambda\I-\lambda\I)\K^{-1}}_{=\I}=\left(\I-\lambda\left(\K+\lambda\I\right)^{-1}\right)\K^{-1}\\
&=\left(\I-\left(\I+1/\lambda\cdot\K\right)^{-1}\right)\K^{-1}.
\end{aligned}
\end{equation*}
\end{proof}
\begin{proof}[Proof of Proposition \ref{thm:kgf_krr_diff}]~\\
By using the fact that $\K$, $\K^{-1}$ and $\DK:=\left(\left(\I+t\K\right)^{-1}-\exp(-t\K)\right)$ commute, we obtain
\begin{equation*}
\begin{aligned}
\alphahfv(t)-\alphahrv(1/t)&=\left(\I-\exp(-t\K)\right)\K^{-1}\yv-\left(\I-\left(\I+t\K\right)^{-1}\right)\K^{-1}\yv\\
&=\underbrace{\left(\left(\I+t\K\right)^{-1}-\exp(-t\K)\right)}_{=:\DK}\K^{-1}\yv=\DK\K^{-1}\yv,\\
\fhfv(\X,t)-\fhrv(\X,1/t)&=\K\left(\alphahfv-\alphahrv\right)=\K\DK\K^{-1}\yv=\DK\K\K^{-1}\yv=\DK\yv,\\
\fhf(\xsv,t)-\fhr(\xsv,1/t)&=\kv(\xsv)^\top\left(\alphahfv-\alphahrv\right)=\kv(\xsv)^\top\DK\K^{-1}\yv=\kv(\xsv)^\top\K^{-1}\DK\yv,\\
\end{aligned}
\end{equation*}
We denote the singular value decomposition of the symmetric matrix $\K$ by $\K=\U\Ss\U^\top$, where $\Ss$ is a diagonal matrix with diagonal elements $\Ss_{ii}=s_i$. Then $\DK=\U\D\U^\top$, where $\D$ is a diagonal matrix with entries
$$\D_{ii}=d_i=\frac1{1+t\cdot s_i}-e^{t\cdot s_i}.$$
Since for any vector $\xv$ and any diagonal matrix $\D$,
\begin{equation}
\label{eq:move_middle}
\xv^\top\D\xv=\sum_i x_i^2\cdot d_i\leq\sum_ix_i^2\cdot\max_i d_i=\xv^\top\xv\cdot\max_i d_i,
\end{equation}
and since $\K^{-1}$ and $\DK$ are both symmetric,
\begin{equation*}
\begin{aligned}
\|\alphahfv(t)-\alphahrv(1/t)\|_2^2&=\yv^\top\K^{-1}\DK\DK\K^{-1}\yv=\yv^\top\K^{-1}\U\D^2\U^\top\K^{-1}\yv\\
&\leq\yv^\top\K^{-1}\underbrace{\U\U^\top}_{=\I}\K^{-1}\yv\cdot \max_i d_i^2=\|\K^{-1}\yv\|_2^2\cdot\max_i d_i^2,
\end{aligned}
\end{equation*}
and
\begin{equation*}
\|\fhfv(\X,t)-\fhrv(\X,1/t)\|_2^2=\yv^\top\DK\DK\yv=\yv^\top\U\D^2\U^\top\yv\leq\yv^\top\underbrace{\U\U^\top}_{=\I}\yv\cdot \max_i d_i^2=\|\yv\|_2^2\cdot\max_i d_i^2.
\end{equation*}
For out-of-sample predictions, calculations become different, since $\kv(\xsv)$ and $\K$ do not commute. Hence we need to take expectation over $\yv$. We first note that for $\yv=\K\alphanv+\epsv$, where $\alphanv\sim(\bm{0},\Sigmaalpha)$, and $\epsv\sim(\bm{0},\sigma_\varepsilon^2\I)$, and where $\K$ and $\Sigmaalpha$ commute,
$$\E_{\epsv,\alphanv}\left(\yv\yv^\top\right)=\K\E\left(\alphanv\alphanv^\top\right)\K+2\K\E(\alphanv)\E(\epsv)^\top+\E(\epsv\epsv^\top)=\Sigmaalpha\K^2+\sigma_\varepsilon^2\I.$$
Now,
\begin{equation*}
\begin{aligned}
&\E_{\epsv,\alphanv}\left(\left(\fhf(\xsv,t)-\fhr(\xsv,1/t)\right)^2\right)=\E_{\epsv,\alphanv}\left(\kv(\xsv)^\top\K^{-1}\DK\yv\yv^\top\DK\K^{-1}\kv(\xsv)\right)\\
&=\kv(\xsv)^\top\K^{-1}\DK\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\DK\K^{-1}\kv(\xsv)\\
&=\kv(\xsv)^\top\K^{-1}\Sigmaalpha^{1/2}\K\U\D^2\U^\top\K\Sigmaalpha^{1/2}\K^{-1}\kv(\xsv)+ \sigma_\varepsilon^2\cdot\kv(\xsv)^\top\K^{-1}\U\D^2\U^\top\K^{-1}\kv(\xsv)\\
&\leq\left(\kv(\xsv)^\top\K^{-1}\Sigmaalpha^{1/2}\K\underbrace{\U\U^\top}_{=\I}\K\Sigmaalpha^{1/2}\K^{-1}\kv(\xsv)+ \sigma_\varepsilon^2\cdot\kv(\xsv)^\top\K^{-1}\underbrace{\U\U^\top}_{=\I}\K^{-1}\kv(\xsv)\right)\cdot \max_i d_i^2\\
&=\kv(\xsv)^\top\K^{-1}\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\K^{-1}\kv(\xsv)\cdot \max_i d_i^2
=\E_{\epsv,\alphanv}\left(\kv(\xsv)^\top\K^{-1}\yv\yv^\top\K^{-1}\kv(\xsv)\right)\cdot \max_i d_i^2\\
&=\E_{\epsv,\alphanv}\left((\kv(\xsv)^\top\K^{-1}\yv)^2\right)\cdot \max_i d_i^2
=\E_{\epsv,\alphanv}\left(\fhn(\xsv)^2\right)\cdot \max_i d_i^2,\\
\end{aligned}
\end{equation*}
where we have again used Equation \ref{eq:move_middle}, and the fact that $\K$, $\Sigmaalpha^{1/2}$ and $\DK$ commute.
If we repeat the calculations with $\kv(\xsv)$ replaced by $\bm{e_i}\in \R^n$, where $(\bm{e_i})_i=1$ and remaining elements equal 0, so that $\hat{\alpha}_i=\bm{e_i}^\top\alphahv$, we obtain
\begin{equation*}
\E_{\epsv,\alphanv}\left(\left(\alphahfi(t)-\alphahri(1/t)\right)^2\right)x_i d_i^2
=\E_{\epsv,\alphanv}\left((\bm{e_i}^\top\K^{-1}\yv)^2\right)\cdot \max_i d_i^2
=\E_{\epsv,\alphanv}\left((\alphahni)^2\right)\cdot \max_i d_i^2.
\end{equation*}
What is left to do, is to calculate $\max_id_i^2$. Let $x=t\cdot s_i$. Then, for $x\geq0$ (which holds, since both $t$ and $s_i$ are non-negative), $d_i^2=\left(\frac1{1+x}-e^{-x}\right)^2$ is obviously larger than 0, and equals 0 for $x=0$ and $x=+\infty$. To find the maximum of the expression we differentiate and set the derivative to zero.
$$\frac{\partial}{\partial x}\left(\left(\frac1{1+x}-e^{-x}\right)^2\right)=2\left(e^{-x}-\frac1{(1+x)^2}\right)\left(\frac1{1+x}-e^{-x}\right).$$
For $i\in\{1,2\}$ we obtain
\begin{equation*}
\begin{aligned}
&e^{-x}-\frac1{(1+x)^i}=0 \iff -\frac{1+x}ie^{-\frac{1+x}i}=-\frac{e^{-\frac1i}}i\iff -\frac{1+x}i=W_k\left(-\frac{e^{-\frac1i}}i\right)\\
\iff& x=-1-i\cdot W_k\left(-\frac{e^{-\frac1i}}i\right)
\end{aligned}
\end{equation*}
where $W_k$ is the $k$-th branch of the Lambert W function, for $k\in\{-1,0\}$. The only combination of $i$ and $k$ that yields an $x\neq0$ is $i=2$, $k=-1$, for which we obtain $x\approx 2.513$ and $\left.\left(\frac1{1+x}-e^{-x}\right)^2\right|_{x=2.513}\approx0.0415.$
\end{proof}
\begin{proof}[Proof of Proposition \ref{thm:kgf_krr_y}]~\\
In all four parts, we will use the inequality $e^x\geq 1+x$ and the equlities $\max_{\vv\neq \nv}\frac{\|\A\vv\|_2}{\|\vv\|_2}=\|\A\|_2=\smax(\A)$, where $\smax(\A)$ denotes the largest singular value. We will denote the largest and smallest singular values of $\K$ by $\smax=\smax(\K)$ and $\smin=\smin(\K).$
For part \emph{(a)}, we have
\begin{equation*}
\begin{aligned}
&\max_{\yv\neq \nv}\left(\frac{\left\|\fhfv(\X,t)-\yv\right\|_2}{\left\|\yv\right\|_2}\right)
=\max_{\yv\neq \nv}\left(\frac{\left\|\K\K^{-1}(\I-\exp(-t\K))\yv-\yv\right\|_2}{\left\|\yv\right\|_2}\right)\\
&=\max_{\yv\neq \nv}\left(\frac{\left\|\exp(-t\K))\yv\right\|_2}{\left\|\yv\right\|_2}\right)
=\left\|\exp(-t\K))\right\|_2=e^{-t\smin},
\end{aligned}
\end{equation*}
where $\smin$ denotes the smallest singular value of $\K$.
Similarly,
\begin{equation*}
\begin{aligned}
&\max_{\yv\neq \nv}\left(\frac{\left\|\fhrv(\X,1/t)-\yv\right\|_2}{\left\|\yv\right\|_2}\right)
=\max_{\yv\neq \nv}\left(\frac{\left\|\K\K^{-1}(\I-(\I+t\K)^{-1})\yv-\yv\right\|_2}{\left\|\yv\right\|_2}\right)\\
&=\max_{\yv\neq \nv}\left(\frac{\left\|(\I+t\K)^{-1}\yv\right\|_2}{\left\|\yv\right\|_2}\right)
=\left\|(\I+t\K)^{-1}\right\|_2=\frac1{1+t\smin},
\end{aligned}
\end{equation*}
where $e^{-t\smin}\leq \frac1{1+t\smin}$, since $e^x\geq 1+x$.
~\\
For part \emph{(b)},
\begin{equation*}
\begin{aligned}
&\max_{\yv\neq \nv}\left(\frac{\left\|\fhrv(\X,1/t)\right\|_2}{\left\|\yv\right\|_2}\right)
=\max_{\yv\neq \nv}\left(\frac{\left\|\K\K^{-1}(\I-(\I+t\K)^{-1})\yv\right\|_2}{\left\|\yv\right\|_2}\right)\\
&=\max_{\yv\neq \nv}\left(\frac{\left\|(\I-(\I+t\K)^{-1})\yv\right\|_2}{\left\|\yv\right\|_2}\right)
=\left\|\I-(\I+t\K)^{-1}\right\|_2=1-\frac1{1+t\smax}
\end{aligned}
\end{equation*}
and
\begin{equation*}
\begin{aligned}
&\max_{\yv\neq \nv}\left(\frac{\left\|\fhfv(\X,t)\right\|_2}{\left\|\yv\right\|_2}\right)
=\max_{\yv\neq \nv}\left(\frac{\left\|\K\K^{-1}(\I-\exp(-t\K))\yv\right\|_2}{\left\|\yv\right\|_2}\right)\\
&=\max_{\yv\neq \nv}\left(\frac{\left\|(\I-\exp(-t\K)))\yv\right\|_2}{\left\|\yv\right\|_2}\right)
=\left\|\I-\exp(-t\K))\right\|_2=1-e^{-t\smax},
\end{aligned}
\end{equation*}
where $1-\frac1{1+t\smax}\leq 1-e^{-t\smax}$, since $e^x\geq 1+x$.
~\\
For parts \emph{(c)} and \emph{(d)}, we first note that for $\yv=\K\alphanv+\epsv$, where $\alphanv\sim(\bm{0},\Sigmaalpha)$, and $\epsv\sim(\bm{0},\sigma_\varepsilon^2\I)$, and where $\K$ and $\Sigmaalpha$ commute,
$$\E_{\epsv,\alphanv}\left(\yv\yv^\top\right)=\K\E\left(\alphanv\alphanv^\top\right)\K+2\K\E(\alphanv)\E(\epsv)^\top+\E(\epsv\epsv^\top)=\Sigmaalpha\K^2+\sigma_\varepsilon^2\I,$$
and that this quantity commutes with other functions of $\K$.
Furthermore, let $\bm{e_i}\in \R^n$, denote the i-th base vector, where $(\bm{e_i})_i=1$ and remaining elements equal 0, so that $\bm{e_i}^\top\vv=v_i$.
For part \emph{(c)},
\begin{equation*}
\begin{aligned}
&\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\fhf(\xvi,t)-y_i\right)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right)
=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top\exp(-t\K)\yv\right)^2\right)}{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top\yv\right)^2\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top\exp(-t\K)\yv\yv^\top\exp(-t\K)\bm{e_i}\right)}{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top\yv\yv^\top\bm{e_i}\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\bm{e_i}^\top\exp(-t\K)\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\exp(-t\K)\bm{e_i}}{\bm{e_i}^\top\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\bm{e_i}}\right)\\
&=\max_{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2\neq 0}\left(\frac{\left\|\exp(-t\K)\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}\right)^2
=e^{-2t\smin}
\end{aligned}
\end{equation*}
and
\begin{equation*}
\begin{aligned}
&\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\fhr(\xvi,1/t)-\yv\right)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right)
=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top(\I+t\K)^{-1}\yv\right)^2\right)}{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top\yv\right)^2\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top(\I+t\K)^{-1}\yv\yv^\top(\I+t\K)^{-1}\bm{e_i}\right)}{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top\yv\yv^\top\bm{e_i}\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\bm{e_i}^\top(\I+t\K)^{-1}\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)(\I+t\K)^{-1}\bm{e_i}}{\bm{e_i}^\top\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\bm{e_i}}\right)\\
&=\max_{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2\neq 0}\left(\frac{\left\|(\I+t\K)^{-1}\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}\right)^2
=\frac1{(1+t\smin)^2},
\end{aligned}
\end{equation*}
where $e^{-2t\smin}\leq \frac1{(1+t\smin)^2}$, since $e^x\geq 1+x$.
For part \emph{(d)},
\begin{equation*}
\begin{aligned}
&\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\fhr(\xvi,1/t)\right)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right)
=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top(\I-(\I+t\K)^{-1})\yv\right)^2\right)}{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top\yv\right)^2\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top((\I-\I+t\K)^{-1})\yv\yv^\top(\I-(\I+t\K)^{-1})\bm{e_i}\right)}{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top\yv\yv^\top\bm{e_i}\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\bm{e_i}^\top(\I-(\I+t\K)^{-1})\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)(\I-(\I+t\K)^{-1})\bm{e_i}}{\bm{e_i}^\top\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\bm{e_i}}\right)\\
&=\max_{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2\neq 0}\left(\frac{\left\|(\I-(\I+t\K)^{-1})\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}\right)^2
=\left(1-\frac1{(1+t\smax)}\right)^2,
\end{aligned}
\end{equation*}
and
\begin{equation*}
\begin{aligned}
&\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\fhf(\xvi,t)-\yv\right)^2\right)}{\E_{\epsv,\alphanv}\left(y_i^2\right)}\right)
=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top(\I-\exp(-t\K))\yv\right)^2\right)}{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top\yv\right)^2\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top(\I-\exp(-t\K))\yv\yv^\top(\I-\exp(-t\K))\bm{e_i}\right)}{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top\yv\yv^\top\bm{e_i}\right)}\right)\\
&=\max_{\E_{\epsv,\alphanv}(y_i^2)\neq 0}\left(\frac{\bm{e_i}^\top(\I-\exp(-t\K))\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)(\I-\exp(-t\K))\bm{e_i}}{\bm{e_i}^\top\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\bm{e_i}}\right)\\
&=\max_{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2\neq 0}\left(\frac{\left\|(\I-\exp(-t\K))\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}{\left\|\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2}\right)^2
=(1-e^{-t\smax})^2,
\end{aligned}
\end{equation*}
where $\left(1-\frac1{1+t\smax}\right)^2\leq \left(1-e^{-t\smax}\right)^2$, since $e^x\geq 1+x$.
\end{proof}
%\begin{proof}[Proof of Proposition \ref{thm:kgf_krr_y}]~\\
%We use the same technique to prove all four parts. We first show that for matrices $\A$, $\B$ and $\C$, where $\B$ is invertible,
%$$\frac{\|\A\C\|_2}{\|\B\C\|_2}\leq \|\A\|_2\|\B^{-1}\|_2:$$
%\begin{equation*}
%\begin{aligned}
%&\|\A\C\|_2\leq\|\A\|_2\cdot\|\C\|_2,\\
%&\|\C\|_2=\|\B^{-1}\B\C\|_2\leq\|\B^{-1}\|_2\cdot\|\B\C\|_2\iff\frac1{\|\B\C\|_2}\leq\frac{\|\B^{-1}\|_2}{\|\C\|_2}.
%\end{aligned}
%\end{equation*}
%Thus,
%$$\frac{\|\A\C\|_2}{\|\B\C\|_2}\leq\|\A\|_2\cdot\|\C\|_2\cdot\frac{\|\B^{-1}\|_2}{\|\C\|_2}=\|\A\|_2\cdot\|\B^{-1}\|_2.$$
%
%We will also use that applying the matrix function, $\bm{f}$, to the matrix $\K=\U\Ss\U^\top$ is equivalent to applying the corresponding scalar function, $f$, to the singular values of $\K$, i.e.\ $\bm{f}(\K)=\bm{f}(\U\Ss\U^\top)=\U f(\Ss)\U^\top$, where $f(\Ss)$ is applied element-wise on the singular values, and thus $\|\bm{f}(\K)\|_2 = f(s_M)$, where $s_M$ is the largest singular value of $\K$.
%
%For part \emph{(a)}, we obtain
%\begin{equation*}
%\begin{aligned}
%\frac{\|\fhfv(\X,t)-\yv\|_2}{\|\fhrv(\X,1/t)-\yv\|_2}
%&=\frac{\|\K\K^{-1}(\I-\exp(-t\K))\yv-\yv\|_2}{\|\K\K^{-1}\left(\I-(\I+t\K)^{-1}\right)\yv-\yv\|_2}
%=\frac{\|\exp(-t\K)\yv\|_2}{\|(\I+t\K)^{-1}\yv\|_2}\\
%&=\|\exp(-t\K)\|_2\cdot \|\I+t\K\|_2
%=e^{-ts_M}\cdot (1+ts_M).
%\end{aligned}
%\end{equation*}
%We now need to show that $e^{-x}\cdot (1+x)\leq 1$ for $x=ts_M$. Since
%$$\frac{d}{dx}\left(e^{-x}\cdot (1+x)\right)=-x\cdot e^{-x},$$
%the function obtains an optimum for $x=0$, for which $e^{0}\cdot (1+0)=1$. It is easy to verify that the optimum is indeed a maximum (e.g. by comparing to $x=\infty$), and thus
%$$\|\fhfv(\X,t)-\yv\|_2\leq\|\fhrv(\X,1/t)-\yv\|_2\cdot e^{-ts_M}\cdot (1+ts_M)\leq\|\fhrv(\X,1/t)-\yv\|_2.$$
%
%~\\
%For part \emph{(b)},
%\begin{equation*}
%\begin{aligned}
%\frac{\|\fhrv(\X,1/t)\|_2}{\|\fhfv(\X,t)\|_2}
%&=\frac{\|\K(\K+1/t\I)^{-1})\yv\|_2}{\|\K\K^{-1}(\I-\exp(-t\K))\yv\|_2}
%=\frac{\|\K(\K+1/t\I)^{-1})\yv\|_2}{\|(\I-\exp(-t\K))\yv\|_2}\\
%&=\left\|\K(\K+1/t\I)^{-1}\right\|_2\cdot\left\|(\I-\exp(-t\K))^{-1}\right\|_2
%=\frac{s_M}{s_M+1/t}\cdot\frac{1}{1-e^{-ts_M}}\\
%&=\frac{ts_M}{(ts_M+1)(1-e^{-ts_M})}
%\end{aligned}
%\end{equation*}
%By numerically maximizing $\frac{x}{(x+1)(1-e^{-x})}$ for $x=ts_M\geq 0$, we can show that the maximum is 1, and thus
%$$\|\fhrv(\X,1/t)\|_2\leq\|\fhfv(\X,t)\|_2\cdot\frac{ts_M}{(ts_M+1)(1-e^{-ts_M})}\leq\|\fhfv(\X,t)\|_2.$$
%
%~\\
%For parts \emph{(c)} and \emph{(d)}, we first note that for $\yv=\K\alphanv+\epsv$, where $\alphanv\sim(\bm{0},\Sigmaalpha)$, and $\epsv\sim(\bm{0},\sigma_\varepsilon^2\I)$, and where $\K$ and $\Sigmaalpha$ commute,
%$$\E_{\epsv,\alphanv}\left(\yv\yv^\top\right)=\K\E\left(\alphanv\alphanv^\top\right)\K+2\K\E(\alphanv)\E(\epsv)^\top+\E(\epsv\epsv^\top)=\Sigmaalpha\K^2+\sigma_\varepsilon^2\I,$$
%and that this quantity commutes with other functions of $\K$.
%
%For part \emph{(c)},
%\begin{equation*}
%\begin{aligned}
%&\frac{\E_{\epsv,\alphanv}\left(\left(\fhfv(\X,t)-\yv\right)_i^2\right)}{\E_{\epsv,\alphanv}\left(\left(\fhrv(\X,1/t)-\yv\right)_i^2\right)}
%=\frac{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top\exp(-t\K)\yv\right)^2\right)}{\E_{\epsv,\alphanv}\left(\left(\bm{e_i}^\top(\I+t\K)^{-1}\yv\right)^2\right)}\\
%&=\frac{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top\exp(-t\K)\yv\yv^\top\exp(-t\K)\bm{e_i}\right)}{\E_{\epsv,\alphanv}\left(\bm{e_i}^\top(\I+t\K)^{-1}\yv\yv^\top(\I+t\K)^{-1}\bm{e_i}\right)}
%=\frac{\bm{e_i}^\top\exp(-t\K)\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\exp(-t\K)\bm{e_i}}{\bm{e_i}^\top(\I+t\K)^{-1}\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)(\I+t\K)^{-1}\bm{e_i}}\\
%&=\frac{\bm{e_i}^\top\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\exp(-t\K)\exp(-t\K)\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}}{\bm{e_i}^\top\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}(\I+t\K)^{-1}(\I+t\K)^{-1}\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}}
%=\frac{\left\|\exp(-t\K)\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2^2}{\left\|(\I+t\K)^{-1}\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\bm{e_i}\right\|_2^2}\\
%&=\left\|\exp(-t\K)\right\|_2^2\cdot\left\|\I+t\K\right\|_2^2
%=\left(e^{-ts_M}\cdot(1+ts_M)\right)^2\\
%\end{aligned}
%\end{equation*}
%
%We now need to show that $(e^{-x}\cdot (1+x))^2\leq 1$ for $x=ts_M\geq 0$. Since
%$$\frac{d}{dx}\left((e^{-x}\cdot (1+x))^2\right)=-2x(1+x)\cdot e^{-2x},$$
%the function obtains an optimum for $x=0$ (and one for $x=-1$, which we ignore since we require $x\geq 0$), for which $(e^{0}\cdot (1+0))^2=1$. It is easy to verify that the optimum is indeed a maximum (e.g. by comparing to $x=\infty$), and thus
%\begin{equation*}
%\begin{aligned}
%\E_{\epsv,\alphanv}\left(\left(\fhfv(\X,t)-\yv\right)_i^2\right)&\leq\E_{\epsv,\alphanv}\left(\left(\fhrv(\X,1/t)-\yv\right)_i^2\right)\cdot \left(e^{-ts_M}\cdot (1+ts_M)\right)^2\\
%&\leq\E_{\epsv,\alphanv}\left(\left(\fhrv(\X,1/t)-\yv\right)_i^2\right).
%\end{aligned}
%\end{equation*}
%
%~\\
%
%For part \emph{(d)},
%\begin{equation*}
%\begin{aligned}
%&\frac{\E_{\epsv,\alphanv}\left(\fhr(\xsv,1/t)^2\right)}{\E_{\epsv,\alphanv}\left(\fhf(\xsv,t)^2\right)}
%=\frac{\E_{\epsv,\alphanv}\left(\kv(\xsv)^\top\left(\K+1/t\I\right)^{-1}\yv\yv^\top\left(\K+1/t\I\right)^{-1}\kv(\xsv)\right)}{\E_{\epsv,\alphanv}\left(\kv(\xsv)^\top\K^{-1}\left(\I-\exp(-t\K)\right)\yv\yv^\top\K^{-1}\left(\I-\exp(-t\K)\right)\kv(\xsv)\right)}\\
%&=\frac{\kv(\xsv)^\top\left(\K+1/t\I\right)^{-1}\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\left(\K+1/t\I\right)^{-1}\kv(\xsv)}{\kv(\xsv)^\top\K^{-1}\left(\I-\exp(-t\K)\right)\left(\Sigmaalpha\K^2+\sigma_\varepsilon^2\I\right)\K^{-1}\left(\I-\exp(-t\K)\right)\kv(\xsv)}\\
%&=\frac{\kv(\xsv)^\top\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\left(\K+1/t\I\right)^{-1}\left(\K+1/t\I\right)^{-1}\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\kv(\xsv)}{\kv(\xsv)^\top\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\K^{-1}\left(\I-\exp(-t\K)\right)\K^{-1}\left(\I-\exp(-t\K)\right)\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\kv(\xsv)}\\
%&=\frac{\left\|\left(\K+1/t\I\right)^{-1}\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\kv(\xsv)\right\|_2^2}{\left\|\K^{-1}\left(\I-\exp(-t\K)\right)\sqrt{\Sigmaalpha\K^2+\sigma_\varepsilon^2\I}\kv(\xsv)\right\|_2^2}\\
%&=\left\|\left(\K+1/t\I\right)^{-1}\right\|_2^2\cdot\left\|\K\left(\I-\exp(-t\K)^{-1}\right)\right\|_2^2
%=\left(\frac1{s_M+1/t}\cdot \frac{s_M}{1-e^{-t s_M}}\right)^2
%=\left(\frac{ts_M}{(ts_M+1)(1-e^{-t s_M})}\right)^2.
%\end{aligned}
%\end{equation*}
%By numerically maximizing $\left(\frac{x}{(x+1)(1-e^{-x})}\right)^2$ for $x=ts_M\geq 0$, we can show that the maximum is 1, and thus
%\begin{equation*}
%\begin{aligned}
%\E_{\epsv,\alphanv}\left(\fhr(\xsv,1/t)^2\right)&\leq\E_{\epsv,\alphanv}\left(\fhf(\xsv,t)^2\right)\cdot\left(\frac{ts_M}{(ts_M+1)(1-e^{-t s_M})}\right)^2
%&\leq\E_{\epsv,\alphanv}\left(\fhf(\xsv,t)^2\right).
%\end{aligned}
%\end{equation*}
%
%\end{proof}
\begin{proof}[Proof of Proposition \ref{thm:kgf_krr_risk}]~\\
The proof is an adaptation of the proofs of Theorems 1 and 2 by \citet{ali2019continuous}.
Unless otherwise stated, all expectations and covariances are with respect to the random variable $\epsv$. We let $\K=\U\Ss\U^\top$ denote the eigenvalue decomposition of $\K$.
Before starting the calculations, we show some intermediary results that will be needed:
\begin{equation*}
\begin{aligned}
\E(\alphahfv(t))-\alphanv=&\left(\I-\exp\left(-t\K\right)\right)\K^{-1}\E(\yv)-\alphanv=\left(\I-\exp\left(-t\K\right)\right)\alphanv-\alphanv=-\exp\left(-t\K\right)\alphanv.\\
\Cov(\alphahfv(t))=&\E\left(\alphahfv(t)\left(\alphahfv(t)\right)^\top\right)-\E(\alphahfv(t))\E(\alphahfv(t))^\top\\
=&\left(\I-\exp\left(-t\K\right)\right)\alphanv\alphanv^\top\left(\I-\exp\left(-t\K\right)\right) +\sigma_\varepsilon^2\K^{-2}\left(\I-\exp\left(-t\K\right)\right)^2\\
&-\left(\I-\exp\left(-t\K\right)\right)\alphanv\alphanv^\top\left(\I-\exp\left(-t\K\right)\right)\\
=&\sigma_\varepsilon^2\K^{-2}\left(\I-\exp\left(-t\K\right)\right)^2.\\
\E(\alphahrv(1/t))-\alphanv=&\left(\K+1/t\cdot\I\right)^{-1}\E(\yv)-\alphanv=\left(\K+1/t\cdot\I\right)^{-1}\K\alphanv-\alphanv\\
=&\left(\left(\K+1/t\cdot\I\right)^{-1}\K-\I\right)\alphanv=\left(\K+1/t\cdot\I\right)^{-1}\left(\K-\K-1/t\cdot\I\right)\alphanv\\
=&-1/t\cdot\left(\K+1/t\cdot\I\right)^{-1}\alphanv.\\
\Cov(\alphahrv(1/t))=&\E\left(\alphahrv(1/t)\left(\alphahrv(1/t)\right)^\top\right)-\E(\alphahrv(1/t))\E(\alphahrv(1/t))^\top\\
=&\left(\K+1/t\cdot\I\right)^{-1}\left(\K\alphanv\alphanv^\top\K+\sigma_\varepsilon^2\I\right)\left(\K+1/t\cdot\I\right)^{-1}\\
&-\left(\K+1/t\cdot\I\right)^{-1}\K\alphanv\alphanv^\top\K\left(\K+1/t\cdot\I\right)^{-1}\\
=&\sigma_\varepsilon^2\left(\K+1/t\cdot\I\right)^{-2}.
\end{aligned}
\end{equation*}
\begin{equation*}
\begin{aligned}
\E\left(\fhf(\xsv,t)\right)-f_0(\xsv)=&\kv(\xsv)^\top\left(\E(\alphahfv(t))-\alphanv\right)=-\kv(\xsv)^\top\exp\left(-t\K\right)\alphanv.\\
\E\left(\fhr(\xsv,1/t)\right)-f_0(\xsv)=&\kv(\xsv)^\top\left(\E(\alphahrv(1/t))-\alphanv\right)=-1/t\cdot\kv(\xsv)^\top\left(\K+1/t\cdot\I\right)^{-1}\alphanv.\\
\Var(\fhf(\xsv,t))=&\E\left(\fhf(\xsv,t)^2\right)-\E\left(\fhf(\xsv,t)\right)^2\\
=&\kv(\xsv)^\top\left(\I-\exp\left(-t\K\right)\right)\alphanv\alphanv^\top\left(\I-\exp\left(-t\K\right)\right)\kv(\xsv)\\
&+\sigma_\varepsilon^2\kv(\xsv)^\top\K^{-2}\left(\I-\exp\left(-t\K\right)\right)^2\kv(\xsv)\\
&-\kv(\xsv)^\top\left(\I-\exp\left(-t\K\right)\right)\alphanv\alphanv^\top\left(\I-\exp\left(-t\K\right)\right)\kv(\xsv)\\
=&\sigma_\varepsilon^2\kv(\xsv)^\top\K^{-2}\left(\I-\exp\left(-t\K\right)\right)^2\kv(\xsv).\\
\Var(\fhr(\xsv,1/t))=&\E\left(\fhr(\xsv,1/t)^2\right)-\E\left(\fhr(\xsv,1/t)\right)^2\\
=&\kv(\xsv)^\top\left(\K+1/t\cdot\I\right)^{-1}\left(\K\alphanv\alphanv^\top\K+\sigma_\varepsilon^2\I\right)\left(\K+1/t\cdot\I\right)^{-1}\kv(\xsv)\\
&-\kv(\xsv)^\top\left(\K+1/t\cdot\I\right)^{-1}\K\alphanv\alphanv^\top\K\left(\K+1/t\cdot\I\right)^{-1}\kv(\xsv)\\
=&\sigma_\varepsilon^2\kv(\xsv)^\top\left(\K+1/t\cdot\I\right)^{-2}\kv(\xsv).
\end{aligned}
\end{equation*}
According to the bias covariance decomposition, we can write the risk of an estimator $\bm{\hat{\theta}}$, estimating $\bm{\theta_0}$, as
\begin{equation*}
\label{eq:risk}
\text{Risk}(\thetahv;\thetanv)=\E\left(\|\thetahv-\thetanv\|_2^2\right)=\|\E(\thetahv)-\thetanv\|_2^2+\Tr(\Cov(\thetahv)).
\end{equation*}
Thus,
\begin{equation*}
\begin{aligned}
\text{Risk}\left(\alphahfv(t);\alphanv\right)&=\|\E(\alphahfv(t))-\alphanv\|_2^2+\Tr(\Cov(\alphahfv))\\
&=\|-\exp(-t\K)\alphanv\|_2^2+\Tr\left(\sigma_\varepsilon^2\K^{-2}\left(\I-\exp(-t\K)\right)^2\right)\\
&=\alphanv^\top\U e^{-t\Ss}\U^\top\U e^{-t\Ss}\U^\top\alphanv +\sigma_\varepsilon^2\Tr\left(\U\left(\frac{1-e^{-t\Ss}}{\Ss}\right)^2\U^\top\right)\\
&=\alphanv^\top\U e^{-2t\Ss}\U^\top\alphanv +\sigma_\varepsilon^2\Tr\left(\left(\frac{1-e^{-t\Ss}}{\Ss}\right)^2\right)\\
&=\sum_{i=1}^n((\U_{:,i})^\top\alphanv)^2 e^{-2ts_i}+\sigma_\varepsilon^2\sum_{i=1}^n\left(\frac{1-e^{-ts_i}}{s_i}\right)^2,
\end{aligned}
\end{equation*}
where in the third equality, we have used the cyclic property of the trace.
Equivalently, for the risk of $\alphahrv$, for $\lambda=1/t$,
\begin{equation*}
\begin{aligned}
\text{Risk}\left(\alphahrv(1/t);\alphanv\right)&=\|\E(\alphahrv(1/t))-\alphanv\|_2^2+\Tr(\Cov(\alphahrv))\\
&=\|-1/t\cdot\left(\K+1/t\cdot\I\right)^{-1}\alphanv\|_2^2+\Tr\left(\sigma_\varepsilon^2\left(\K+1/t\cdot\I\right)^{-2}\right)\\
&=\alphanv^\top\U \frac{1/t}{\Ss+1/t}\U^\top\U \frac{1/t}{\Ss+1/t}\U^\top\alphanv +\sigma_\varepsilon^2\Tr\left(\U\left(\frac1{\Ss+1/t}\right)^2\U^\top\right)\\
&=\alphanv^\top\U \frac1{(t\cdot\Ss+1)^2}\U^\top\alphanv +\sigma_\varepsilon^2\Tr\left(\frac{t^2}{(t\cdot\Ss+1)^2}\right)\\
&=\sum_{i=1}^n((\U_{:,i})^\top\alphanv)^2 \frac{1}{(ts_i+1)^2}+\sigma_\varepsilon^2\sum_{i=1}^n\frac{t^2}{(ts_i+1)^2}.
\end{aligned}
\end{equation*}
For $x\geq 0$, the following two inequalities are shown by \cite{ali2019continuous}:
$$e^{-x}\leq1/(x+1)\text{ and }1-e^{-x}\leq1.2985\cdot x(x+1).$$
Using these, we obtain
\begin{equation*}
\begin{aligned}
\text{Risk}\left(\alphahfv(t);\alphanv\right)=&\sum_{i=1}^n\left(((\U_{:,i})^\top\alphanv)^2 e^{-2ts_i}+\sigma_\varepsilon^2\left(\frac{1-e^{-ts_i}}{s_i}\right)^2\right)\\
\leq&\sum_{i=1}^n\left(((\U_{:,i})^\top\alphanv)^2 \frac1{(ts_i+1)^2}+\sigma_\varepsilon^2\left(1.2985\frac {ts_i}{ts_i+1}\right)^2\frac1{s_i^2}\right)\\
=&\sum_{i=1}^n\left(((\U_{:,i})^\top\alphanv)^2 \frac1{(ts_i+1)^2}+\sigma_\varepsilon^21.2985^2\frac {t^2}{(ts_i+1)^2}\right)\\
\leq&1.6862\sum_{i=1}^n\left(((\U_{:,i})^\top\alphanv)^2 \frac1{(ts_i+1)^2}+\sigma_\varepsilon^2\frac {t^2}{(ts_i+1)^2}\right)
=1.6862\cdot\text{Risk}\left(\alphahrv(1/t);\alphanv\right),
\end{aligned}
\end{equation*}
which proves part \emph{(a)}.
The calculations for part \emph{(b)} are basically identical to those for part \emph{(a)}. We obtain
\begin{equation*}
\begin{aligned}
&\text{Risk}\left(\fhfv(\X,t);\fnv(\X)\right)=\sum_{i=1}^n((\U_{:,i})^\top\alphanv)^2 s_i^2e^{-2ts_i}+\sigma_\varepsilon^2\sum_{i=1}^n\left(1-e^{-ts_i}\right)^2,\\
&\text{Risk}\left(\fhrv(\X,1/t);\fnv(\X)\right)=\sum_{i=1}^n((\U_{:,i})^\top\alphanv)^2 \frac{s_i^2}{(ts_i+1)^2}+\sigma_\varepsilon^2\sum_{i=1}^n\frac{t^2s_i^2}{(ts_i+1)^2},
\end{aligned}
\end{equation*}
and thus
\begin{equation*}
\begin{aligned}
\text{Risk}\left(\fhfv(\X,t);\fnv(\X)\right)=&\sum_{i=1}^n\left(((\U_{:,i})^\top\alphanv)^2 s_i^2e^{-2ts_i}+\sigma_\varepsilon^2\left(1-e^{-ts_i}\right)^2\right)\\
\leq&\sum_{i=1}^n\left(((\U_{:,i})^\top\alphanv)^2 \frac{s_i^2}{(ts_i+1)^2}+\sigma_\varepsilon^2\left(1.2985\frac {ts_i}{ts_i+1}\right)^2\right)\\
\leq&1.2985^2\sum_{i=1}^n\left(((\U_{:,i})^\top\alphanv)^2 \frac{s_i^2}{(ts_i+1)^2}+\sigma_\varepsilon^2\frac {t^2s_i^2}{(ts_i+1)^2}\right)\\
=&1.6862\cdot\text{Risk}\left(\fhrv(\X,1/t);\fnv(\X)\right),
\end{aligned}
\end{equation*}
which proves part \emph{(b)}.
For the out-of-sample prediction risk, since $\kv(\xsv)$ and $\K$ do not commute, calculations become slightly different. We now obtain
\begin{equation*}
\begin{aligned}
&\text{Risk}\left(\fhf(\xsv,t);f_0(\xsv,t)\right)=\left(\E(\fhf(\xsv,t))-f_0(\xsv,t)\right)^2+\Var(\fhf(\xsv,t))\\
=&\left(-\kv(\xsv)^\top\exp(-t\K)\alphanv\right)^2+\sigma_\varepsilon^2\kv(\xsv)^\top\K^{-2}\left(\I-\exp(-t\K)\right)^2\kv(\xsv)\\
=&\alphanv^\top\exp(-t\K)\kv(\xsv)\kv(\xsv)^\top\exp(-t\K)\alphanv\\
&+\sigma_\varepsilon^2\Tr\left(\K^{-2}\left(\I-\exp(-t\K)\right)^2\kv(\xsv)\kv(\xsv)^\top\right)\\
=&\Tr\left(\exp(-t\K)\alphanv\alphanv^\top\exp(-t\K)\kv(\xsv)\kv(\xsv)^\top\right)\\
&+\sigma_\varepsilon^2\Tr\left(\K^{-2}\left(\I-\exp(-t\K)\right)^2\kv(\xsv)\kv(\xsv)^\top\right),\\
\end{aligned}
\end{equation*}
where we have used the fact that the trace of a scalar is the scalar itself, and the cyclic property of the trace.
Analogously, for the risk of $\fhr(\xsv)$,
\begin{equation*}
\begin{aligned}
&\text{Risk}\left(\fhr(\xsv,1/t);f_0(\xsv)\right)=\left(\E(\fhr(\xsv,1/t))-f_0(\xsv)\right)^2+\Var(\fhr(\xsv,1/t))\\
=&\left(-1/t\cdot\kv(\xsv)^\top\left(\K+1/t\cdot\I\right)^{-1}\alphanv\right)^2+\sigma_\varepsilon^2\kv(\xsv)^\top\left(\K+1/t\cdot\I\right)^{-2}\kv(\xsv)\\
=&1/t^2\cdot\alphanv^\top\left(\K+1/t\cdot\I\right)^{-1}\kv(\xsv)\kv(\xsv)^\top\left(\K+1/t\cdot\I\right)^{-1}\alphanv\\
&+\sigma_\varepsilon^2\Tr\left(\left(\K+1/t\cdot\I\right)^{-2}\kv(\xsv)\kv(\xsv)^\top\right)\\
=&\Tr\left(1/t^2\cdot\left(\K+1/t\cdot\I\right)^{-1}\alphanv\alphanv^\top\left(\K+1/t\cdot\I\right)^{-1}\kv(\xsv)\kv(\xsv)^\top\right)\\
&+\sigma_\varepsilon^2\Tr\left(\left(\K+1/t\cdot\I\right)^{-2}\kv(\xsv)\kv(\xsv)^\top\right).
\end{aligned}
\end{equation*}
Taking expectation over $\alphanv$, for $\alphanv\sim(\bm{0},\Sigmaalpha)$, we obtain
\begin{equation*}
\begin{aligned}
\E_{\alphanv}&\left(\text{Risk}\left(\fhf(\xsv,t);f_0(\xsv)\right)\right)\\
&=\Tr\left(\left(\Sigmaalpha\exp(-t\K)^2+\sigma_\varepsilon^2\K^{-2}\left(\I-\exp(-t\K)\right)^2\right)\kv(\xsv)\kv(\xsv)^\top\right),\\
\E_{\alphanv}&\left(\text{Risk}\left(\fhr(\xsv,1/t);f_0(\xsv)\right)\right)\\
&=\Tr\left(\left(\Sigmaalpha1/t^2\cdot\left(\K+1/t\cdot\I\right)^{-2}+\sigma_\varepsilon^2\left(\K+1/t\cdot\I\right)^{-2}\right)\kv(\xsv)\kv(\xsv)^\top\right).
\end{aligned}
\end{equation*}
Finally comparing the risks, we obtain
\begin{equation*}
\begin{aligned}
&\E_{\alphanv}\left(\text{Risk}(\fhf(\xsv,t);f_0(\xsv))\right)\\
&=\Tr\left(\left(\Sigmaalpha\exp(-t\K)^2+\sigma_\varepsilon^2t^2(t\K)^{-2}\left(\I-\exp(-t\K)\right)^2\right)\kv(\xsv)\kv(\xsv)^\top\right)\\
&\leq \Tr\left(\left(\Sigmaalpha\left(\I+t\K\right)^{-2}+\sigma_\varepsilon^2t^2\left(1.2985 \left(\I+t\K\right)^{-1}\right)^2\right)\kv(\xsv)\kv(\xsv)^\top\right)\\
&= \Tr\left(\left(\Sigmaalpha\cdot1/t^2\cdot\left(1/t\cdot\I+\K\right)^{-2}+\sigma_\varepsilon^2 \cdot 1.2985^2 \left(1/t\cdot\I+\K\right)^{-2}\right)\kv(\xsv)\kv(\xsv)^\top\right)\\
&\leq 1.6862\cdot\Tr\left(\left(\Sigmaalpha\cdot1/t^2\cdot\left(1/t\cdot\I+\K\right)^{-2}+\sigma_\varepsilon^2 \left(1/t\cdot\I+\K\right)^{-2}\right)\kv(\xsv)\kv(\xsv)^\top\right)\\
&=1.6862\cdot\E_{\alphanv}\left(\text{Risk}(\fhr(\xsv,1/t);f_0(\xsv))\right),\\
\end{aligned}
\end{equation*}
which proves part \emph{(c)}.
\end{proof}
\begin{proof}[Proof of Proposition \ref{thm:sgn_inf}]~\\
The proofs of the two parts are very similar, differing only in the details.
For part \emph{(a)}, when $\X^\top\X$ is a diagonal matrix with elements $\{s_{ii}\}_{i=1}^p$,
$$\left\|\yv-\X\betav\right\|_2^2=\yv^\top\yv-2\yv^\top\X\betav+\betav^\top\X^\top\X\betav=\sum_{i=1}^p\left(y_i^2-2(\X^\top\yv)_i\beta_i+s_{ii}\beta_i^2\right).$$
The constraint $\|\betav\|_\infty=\max_i|\beta_i| \leq c$ is equivalent to $|\beta_i| \leq c$ for $i=1,2,\dots p$.
Thus, $$\left\|\yv-\X\betav\right\|_2^2 \text{ s.t. } \|\betav\|_\infty\leq c \iff\sum_{i=1}^p\left(y_i^2-2(\X^\top\yv)_i\beta_i+s_{ii}\beta_i^2\right) \text{ s.t. } |\beta_i|\leq c\ \forall i,$$
which decomposes element-wise. In the absence of the constraint, $\hat{\beta}_i=\left((\X^\top\X)^{-1}\X^\top\yv\right)_i=\left(\X^\top\yv\right)_i/s_{ii}$.
Assume $\left(\X^\top\yv\right)_i/s_{ii}\geq 0$. Then, the optimal value for $\beta_i$ is $\left(\X^\top\yv\right)_i/s_{ii}$, unless $\left(\X^\top\yv\right)_i/s_{ii}>c$, then, due to convexity, the optimal value is $c$, i.e.\ $\hat{\beta}_i=\min\left(\left(\X^\top\yv\right)_i/s_{ii},c\right)$. Accounting also for the case of $\left(\X^\top\yv\right)_i/s_{ii}< 0$, we obtain
$$\hat{\beta}_i(c)=\sgn\left(\left(\X^\top\yv\right)_i/s_{ii}\right)\cdot\min\left(\left|\left(\X^\top\yv\right)_i/s_{ii}\right|,c\right).$$
The sign gradient flow solution is calculated as follows:
\begin{equation*}
\begin{aligned}
\sgn\left(-\frac{\partial}{\partial\betav(t)}\left(\left\|\yv-\X\betav(t)\right\|_2^2\right)\right)&=\sgn\left(\X^\top\yv-\X^\top\X\betav(t)\right)=\sgn\left(\begin{bmatrix}\vrule\\(\X^\top\yv)_i-s_{ii}\beta_i(t)\\ \vrule\end{bmatrix}\right)\\
&=\begin{bmatrix}\vrule\\\sgn\left((\X^\top\yv)_i-s_{ii}\beta_i(t)\right)\\ \vrule\end{bmatrix}.
\end{aligned}
\end{equation*}
Since element $i$ in the vector only depends on $\betav$ through $\beta_i$,
\begin{equation*}
\begin{aligned}
&\frac{\partial\betav(t)}{\partial t}=\sgn\left(-\frac{\partial}{\partial\betav(t)}\left(\left\|\yv-\X\betav(t)\right\|_2^2\right)\right)\\
\iff&\frac{\partial\beta_i(t)}{\partial t}= \sgn((\X^\top \yv)_i-s_{ii}\beta_i(t))
=
\begin{cases}
1&\text{if }(\X^\top\yv)_i>s_{ii}\beta(t)\\
-1&\text{if }(\X^\top\yv)_i<s_{ii}\beta(t)\\
0&\text{if }(\X^\top\yv)_i=s_{ii}\beta(t)\\
\end{cases},
\ i=1,2,\dots p.
\end{aligned}
\end{equation*}
Thus, with $\beta_i(0)=0$, $\beta_i(t)=\pm t$, depending on the sign of $(\X^\top\yv)_i/s_{ii}$. Once $\beta_i=(\X^\top\yv)_i/s_{ii}$ (and $\sgn(s_{ii}\beta_i-(\X^\top\yv)_i)=0$), then $\beta_i=(\X^\top\yv)_i/s_{ii}$. That is,
$$\hat{\beta}_i(t)=\sgn\left((\X^\top\yv)_i/s_{ii}\right)\cdot\min\left(t,\left|(\X^\top\yv)_i/s_{ii}\right|\right).$$
For part \emph{(b)}, when $\K$ is a diagonal matrix with elements $\{k_{ii}\}_{i=1}^n$,
$$\left\|\yv-\K\alphav\right\|_{\K^{-1}}^2=\left\|\left(\sqrt{\K}\right)^{-1}\yv-\sqrt{\K}\alphav\right\|_2^2=\sum_{i=1}^n\left(\frac{y_i}{\sqrt{k_{ii}}}-\sqrt{k_{ii}}\alpha_i\right)^2.$$
The constraint $\|\alphav\|_\infty=\max_i|\alpha_i| \leq c$ is equivalent to $|\alpha_i| \leq c$, for $i=1,2,\dots n$.
Thus, $$\left\|\yv-\K\alphav\right\|_{\K^{-1}}^2 \text{ s.t. } \|\alphav\|_\infty\leq c \iff\sum_{i=1}^n\left(\frac{y_i}{\sqrt{k_{ii}}}-\sqrt{k_{ii}}\alpha_i\right)^2 \text{ s.t. } |\alpha_i|\leq c\ \forall i,$$
which decomposes element-wise. Assume $y_i/k_{ii}\geq 0$. Then, the optimal value for $\alpha_i$ is $y_i/k_{ii}$, unless $y_i/k_{ii}>c$, then, due to convexity, the optimal value is $c$, i.e.\ $\hat{\alpha}_i=\min(y_i/k_{ii},c)$. Accounting also for the case of $y_i/k_{ii}< 0$, we obtain
$$\hat{\alpha}_i(c)=\sgn(y_i/k_{ii})\cdot\min(|y_i/k_{ii}|,c).$$
The sign gradient flow solution is calculated as follows:
$$\sgn\left(-\frac{\partial}{\partial\alphav(t)}\left(\left\|\yv-\K\alphav(t)\right\|_{\K^{-1}}^2\right)\right)=\sgn\left(\yv-\K\alphav(t)\right)=\sgn\left(\begin{bmatrix}\vrule\\y_i-k_{ii}\alpha_i(t)\\ \vrule\end{bmatrix}\right)=\begin{bmatrix}\vrule\\\sgn(y_i-k_{ii}\alpha_i(t))\\ \vrule\end{bmatrix}.$$
Since element $i$ in the vector only depends on $\alphav$ through $\alpha_i$,
\begin{equation*}
\begin{aligned}
&\frac{\partial\alphav(t)}{\partial t}=\sgn\left(-\frac{\partial}{\partial\alphav(t)}\left(\left\|\yv-\K\alphav(t)\right\|_{\K^{-1}}^2\right)\right)\\
\iff&\frac{\partial\alpha_i(t)}{\partial t}= \sgn(y_i-k_{ii}\alpha_i(t))
=\begin{cases}
1&\text{if }y_i>k_{ii}\alpha_i(t)\\
-1&\text{if }y_i<k_{ii}\alpha_i(t)\\
0&\text{if }y_i=k_{ii}\alpha_i(t)\\
\end{cases},
\ i=1,2,\dots n.
\end{aligned}
\end{equation*}
Thus, with $\alpha_i(0)=0$, $\alpha_i(t)=\pm t$, depending on the sign of $y_i/k_{ii}$. Once $\alpha_i=y_i/k_{ii}$ (and $\sgn(y_i-k_{ii}\alpha_i)=0$), then $\alpha_i=y_i/k_{ii}$. That is,
$$\hat{\alpha}_i(t)=\sgn(y_i/k_{ii})\cdot\min(t,|y_i/k_{ii}|).$$
\end{proof}
\begin{proof}[Proof of Proposition \ref{thm:other_objs}]~\\
We first calculate the gradients of $\|\yv-\Ph\betav\|_2^2$, $\|\yv-\Ph\betav\|_1$ and $\|\yv-\Ph\betav\|_\infty$, respectively:
$$\frac{\partial}{\partial \betav}\left(\left\|\yv-\Ph\betav\right\|_2^2\right)=\frac{\partial}{\partial \betav}\left((\yv-\Ph\betav)^\top(\yv-\Ph\betav)\right)=-\Ph^\top(\yv-\Ph\betav).$$
Since $\|\vv\|_1=\sgn(\vv)^\top\vv$,
$$\frac{\partial}{\partial \betav}\left(\left\|\yv-\Ph\betav\right\|_1\right)=\frac{\partial}{\partial \betav}\left(\sgn(\yv-\Ph\betav)^\top(\yv-\Ph\betav)\right)=-\Ph^\top\sgn(\yv-\Ph\betav).$$
Let $\mathbb{I}_\infty(\vv)$ be a vector denoting the sign of the largest (absolute) value in $\vv$, such that
$$\mathbb{I}_\infty(\vv)_d=
\begin{cases}
\sgn(v_d)&\text{ if } d=\argmax_{d'}|v_{d'}|\\
0&\text{ else.}
\end{cases}$$
Then $\|\vv\|_\infty=\mathbb{I}_\infty(\vv)^\top\vv$, and
$$\frac{\partial}{\partial \betav}\left(\left\|\yv-\Ph\betav\right\|_\infty\right)=\frac{\partial}{\partial \betav}\left(\mathbb{I}_\infty(\yv-\Ph\betav)^\top(\yv-\Ph\betav)\right)=-\Ph^\top\mathbb{I}_\infty(\yv-\Ph\betav).$$
The three update rules for gradient descent in $\betav$ are thus
\begin{equation}
\label{eq:eq_gd_beta}
\begin{aligned}
\betahv_{k+1}&=\betahv_k+\eta\cdot\Ph^\top\left(\yv-\Ph\betahv_k\right), && \text{ for } \|\yv-\Ph\betav\|_2^2\\
\betahv_{k+1}&=\betahv_k+\eta\cdot\Ph^\top\sgn\left(\yv-\Ph\betahv_k\right), && \text{ for } \|\yv-\Ph\betav\|_1\\
\betahv_{k+1}&=\betahv_k+\eta\cdot\Ph^\top\mathbb{I}_\infty\left(\yv-\Ph\betahv_k\right), && \text{ for } \|\yv-\Ph\betav\|_\infty.
\end{aligned}
\end{equation}
Since the gradient of $\|\yv-\K\alphav\|_{\K^{-1}}^2$ is $-(\yv-\K\alphav)$, the update rules for gradient descent, sign gradient descent and coordinate descent are, respectively
\begin{equation}
\label{eq:eq_gd_alpha}
\begin{aligned}
\alphahv_{k+1}&=\alphahv_k+\eta\cdot\left(\yv-\K\alphahv_k\right)\\
\alphahv_{k+1}&=\alphahv_k+\eta\cdot\sgn\left(\yv-\K\alphahv_k\right)\\
\alphahv_{k+1}&=\alphahv_k+\eta\cdot\mathbb{I}_\infty\left(\yv-\K\alphahv_k\right).\\
\end{aligned}
\end{equation}
By multiplying each term in Equation \ref{eq:eq_gd_alpha} by $\Ph^\top$, and using $\betav=\Ph^\top\alphav$ and $\K=\Ph\Ph^\top$, and thus $\K\alphav=\Ph\betav$, we obtain Equation \ref{eq:eq_gd_beta}.
\end{proof}
\begin{proof}[Proof of Proposition \ref{thm:conv}]~\\
For KGD, let $L_2(\alphahv):=\|\yv-\K\alphahv\|_2^2$. Then $L_2'(\alphahv)=-\K(\yv-\K\alphav)$, $L_2''(\alphahv)=\K^2$ and higher order derivatives are zero. With $\dalphahv=\eta\cdot(\yv-\K\alphahv)$, according to Taylor's theorem
\begin{equation*}
\begin{aligned}
L_2(\alphahv_{k+1})-L_2(\alphahv_k)
&=L_2(\alphahv_k+\dalphahv_k)-L_2(\alphahv_k)
=L_2(\alphahv_k)+L_2'(\alphahv_k)^\top\dalphahv+\frac12\dalphahv^\top L_2''(\alphahv_k)\dalphahv-L_2(\alphahv_k)\\
&=-\eta\cdot(\yv-\K\alphav)^\top\K(\yv-\K\alphahv)+\frac{\eta^2}2(\yv-\K\alphahv_k)^\top\K^2(\yv-\K\alphahv)\\
&\leq-\eta\cdot(\yv-\K\alphav)^\top\K(\yv-\K\alphahv)+\frac{\eta^2}2\smax(\K)(\yv-\K\alphahv_k)^\top\K(\yv-\K\alphahv).
\end{aligned}
\end{equation*}
Factoring out $\eta\cdot(\yv-\K\alphav)^\top\K(\yv-\K\alphahv)$, and using that $(\yv-\K\alphav)^\top\K(\yv-\K\alphahv)>0$, since $\K$ is positive definite, we see that
$L_2(\alphahv_{k+1})-L_2(\alphahv_k)<0$ if $\eta<\frac2{\smax(\K)}$.
For KSGD, $L_1(\alphahv):=\|\yv-\K\alphahv\|_1=\sgn(\yv-\K\alphahv)^\top(\yv-\K\alphahv)$. Then $L_1'(\alphahv)=-\K\sgn(\yv-\K\alphav)$ and higher order derivatives are zero. With $\dalphahv=\eta\cdot\sgn(\yv-\K\alphahv)$, according to Taylor's theorem, for $\eta< \min_{1\leq i\leq n}\left(\left|\yv-\K\alphahv_k\right|_i\right)$
\begin{equation*}
\begin{aligned}
L_1(\alphahv_{k+1})-L_1(\alphahv_k)
&=L_1(\alphahv_k+\dalphahv_k)-L_1(\alphahv_k)
=L_1(\alphahv_k)+L'(\alphahv_k)^\top\dalphahv\\
&=-\eta\cdot\sgn(\yv-\K\alphav)^\top\K\sgn(\yv-\K\alphahv)<0.\\
\end{aligned}
\end{equation*}
For KCD, let $\mathbb{I}_\infty(\vv)$ be a vector denoting the sign of the largest (absolute) value in $\vv$, such that
$$\mathbb{I}_\infty(\vv)_d=
\begin{cases}
\sgn(v_d)&\text{ if } d=\argmax_{d'}|v_{d'}|\\
0&\text{ else,}
\end{cases}$$
so that $\|\vv\|_\infty=\mathbb{I}_\infty(\vv)^\top\vv$. Furthermore, let $L_\infty(\alphahv):=\|\yv-\K\alphahv\|_\infty=\mathbb{I}_\infty(\yv-\K\alphahv)^\top(\yv-\K\alphahv)$. Then $L_\infty'(\alphahv)=-\K\mathbb{I}_\infty(\yv-\K\alphav)$ and higher order derivatives are zero. With $\dalphahv=\eta\cdot\mathbb{I}_\infty(\yv-\K\alphahv)$, according to Taylor's theorem, for $\eta< \min_{1\leq i\leq n}\left(\left|\yv-\K\alphahv_k\right|_i\right)$
\begin{equation*}
\begin{aligned}
L_\infty(\alphahv_{k+1})-L_\infty(\alphahv_k)
&=L_\infty(\alphahv_k+\dalphahv_k)-L_\infty(\alphahv_k)
=L_\infty(\alphahv_k)+L'(\alphahv_k)^\top\dalphahv\\
&=-\eta\cdot\mathbb{I}_\infty(\yv-\K\alphav)^\top\K\mathbb{I}_\infty(\yv-\K\alphahv)<0.\\
\end{aligned}
\end{equation*}
\end{proof}
\begin{proof}[Proof of Lemma \ref{thm:equ_f}]~\\
Let $\fpv:=[\fhv^\top,\fhsv{^\top}]^\top$, $\ypv:=[\yv^\top,\ytv^\top]^\top$, and let
$$\Ih:=\begin{bmatrix}\I_{n\times n} & \nv_{n\times \ns}\end{bmatrix}\in \R^{n\times (n+\ns)}$$
denote the training data selection matrix, so that $\fv=\Ih\fpv$ and $\K=\Ih\cdot[\K^\top,\Ks{^\top}]^\top=\Ih\Kss\Ih^\top$.
Differentiating Equation \ref{eq:equ_f1} with respect $\fpv$ and setting the gradient to $\nv$, we obtain
\begin{equation*}
\begin{aligned}
\label{eq:fh_temp}
\nv&=\frac{\partial}{\partial \fhpv}\left(\frac12\left\|\yv-\Ih\fpv\right\|_2^2+\frac\lambda2\|\fpv\|^2_{(\Kss)^{-1}}\right)=\Ih^\top\left(\Ih\fhpv-\yv\right)+\lambda(\Kss)^{-1}\fhpv\\
\iff\fhpv&=\left(\Ih^\top\Ih+\lambda(\Kss)^{-1}\right)^{-1}\Ih^\top\yv.
\end{aligned}
\end{equation*}
According to the matrix inversion lemma,
$$(\D-\C\A^{-1}\B)^{-1}\C\A^{-1}=\D^{-1}\C(\A-\B\D^{-1}\C)^{-1}.$$
For $\A=\I$, $\B=\Ih$, $\C=\Ih^\top$ and $\D=\lambda(\Kss)^{-1}$, we obtain
$$\left(\lambda(\Kss)^{-1}+\Ih^\top\Ih\right)^{-1}\Ih^\top=1/\lambda\underbrace{\Kss\Ih^\top}_{=[\K^\top,\Ks{^\top}]^\top}\left(\I+1/\lambda\underbrace{\Ih\Kss\Ih^\top}_{=\K}\right)^{-1}=\begin{bmatrix}\K\\\Ks\end{bmatrix}(\lambda\I+\K)^{-1},$$
and thus
$$\fhpv=\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\K+\lambda\I\right)^{-1}\yv.$$
Before differentiating Equation \ref{eq:equ_f2} with respect to $\fpv$, we first note that according to the definition of $\ytv$,
$$\ypv-\fpv=\begin{bmatrix}\yv-\Ih\fpv\\\nv\end{bmatrix}=\begin{bmatrix}\I_{n\times n} \\ \nv_{\ns\times n} \end{bmatrix}(\yv-\Ih\fpv)=\Ih^\top\yv-\Ih^\top\Ih\fpv.$$
Now, differentiating Equation \ref{eq:equ_f2} with respect $\fpv$ and setting the gradient to $\nv$ we obtain
\begin{equation*}
\begin{aligned}
\nv&=\frac{\partial}{\partial \fhpv}\left(\frac12\left\|\ypv-\fhpv\right\|_{\Kss}^2+\frac\lambda2\|\fhpv\|^2_2\right)=-\Kss\ypv+\Kss\fhpv+\lambda\fhpv=-\Kss(\ypv-\fhpv)+\lambda\fhpv\\
&=-\Kss\left(\Ih^\top\yv-\Ih^\top\Ih\fhpv\right)+\lambda\fhpv=-\Kss\Ih^\top\yv+(\Kss\Ih^\top\Ih+\lambda\I)\fhpv\\
\iff\fhpv&=\left(\begin{bmatrix}\K\\\Ks\end{bmatrix}\Ih+\lambda\I\right)^{-1}\begin{bmatrix}\K\\\Ks\end{bmatrix}\yv,
\end{aligned}
\end{equation*}
Using the matrix inversion lemma,
$$(\D-\C\A^{-1}\B)^{-1}\C\A^{-1}=\D^{-1}\C(\A-\B\D^{-1}\C)^{-1},$$
with $\A=\I$, $\B=\Ih$, $\C=[\K^\top,\Ks{^\top}]^\top$ and $\D=\lambda\I$, we obtain
$$\left(\lambda\I+\begin{bmatrix}\K\\\Ks\end{bmatrix}\Ih\right)^{-1}\begin{bmatrix}\K\\\Ks\end{bmatrix}=1/\lambda\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\I+1/\lambda\underbrace{\Ih\begin{bmatrix}\K\\\Ks\end{bmatrix}}_{=\K}\right)^{-1}=\begin{bmatrix}\K\\\Ks\end{bmatrix}(\lambda\I+\K)^{-1},$$
and thus
$$\fhpv=\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\K+\lambda\I\right)^{-1}\yv.$$
\end{proof}
\begin{proof}[Proof of Lemma \ref{thm:kgf_s_f}]~\\
With $\etahv(t)=\fhv(t)-\yv$ we obtain $\frac{d\etahv(t)}{dt}=\frac{d\fhv(t)}{dt}$ and the first part of Equation \ref{eq:kgd_diff_eq_f} can be written as
$$\frac{d\etahv(t)}{dt}=-\K\etahv(t)\iff \etahv(t)=\exp\left(-t\K\right)\etahnv.$$
Now,
$$\fhv(0)=\bm{0}\implies\etahnv=\etahv(0)=-\yv\implies \etahv(t)=-\exp(-t\K)\yv$$
Solving for $\fhv(t)$, we obtain
$$\fhv(t)=\left(\I-\exp(-t\K)\right)\yv.$$
Finally,
$$\frac{d\fhsv(t)}{dt}=\Ks\left(\yv-\fhv(t)\right)=\Ks\exp(-t\K)\yv\implies \fhsv(t)=\bm{c}-\Ks\K^{-1}\exp(-t\K)\yv.$$
Now,
$$\fhsv(0)=\bm{0}\implies\bm{c}=\Ks\K^{-1}\yv\implies \fhsv(t)=\Ks\K^{-1}(\I-\exp(-t\K))\yv.$$
\end{proof}
\begin{proof}[Proof of Lemma \ref{thm:kgf_s_f} with momentum and Nesterov accelerated gradient]~\\
Analogously to for the case of $\alphahv$ in the proof of the remark of Lemma \ref{thm:kgf_s}, for momentum and Nesterov accelerated gradient, Equation \ref{eq:kgd_diff_eq_f} generalizes into
$$(1-\gamma)\cdot\begin{bmatrix}\frac{d\fhv(t)}{dt}\\\frac{d\fhv(t)}{dt}\end{bmatrix}=\begin{bmatrix}\K\\\Ks\end{bmatrix}\left(\yv-\fhv(t)\right).$$
Solving the differential equations in the same way as in Lemma \ref{thm:kgf_s_f}, we obtain
%$$\begin{bmatrix}\fhv(t)\\\fhsv(t)\end{bmatrix}=\begin{bmatrix}\left(\I-\exp\left(-\frac t{1-\gamma}\K\right)\right)\yv,\\\Ks\K^{-1}\left(\I-\exp\left(-\frac t{1-\gamma}\K\right)\right)\yv\end{bmatrix}.$$
$$\begin{bmatrix}\fhv(t)\\\fhsv(t)\end{bmatrix}=\begin{bmatrix}\I\\\Ks\K^{-1}\end{bmatrix}\left(\I-\exp\left(-\frac t{1-\gamma}\K\right)\right)\yv.$$
\end{proof}
\begin{proof}[Proof of Proposition \ref{thm:sgn_inf_f}]~\\
When $\Kss$ is a diagonal matrix with elements $\{k_{ii}\}_{i=1}^{n+\ns}$, $k_{ii}>0$,
$$\left\|\ypv-\fpv\right\|_{\Kss}^2=\sum_{i=1}^{n+\ns}\left(k_{ii}(\yp_i-\fp_i)\right)^2.$$
The constraint $\|\fpv\|_\infty=\max_i|\fp_i| \leq c$ is equivalent to $|\fp_i| \leq c\text{ for } i=1,2,\dots n+\ns$.\\
Thus, $$\left\|\ypv-\fpv\right\|_{\Kss}^2 \text{ s.t. } \|\fpv\|_\infty\leq c \iff\sum_{i=1}^{n+\ns}\left(k_{ii}(\yp_i-\fp_i)\right)^2 \text{ s.t. } |\fp_i|\leq c\ \forall i,$$
which decomposes element-wise to
\begin{equation}
\label{eq:fs_opt1}
\fhp_i=\argmin_{\fp_i\in \R}\left(k_{ii}(\yp_i-\fp_i)\right)^2 \text{ s.t. } |\fp_i|\leq c.
\end{equation}
We solve Equation \ref{eq:fs_opt1} separately for the two cases $i\leq n$ and $i>n$. For $i>n$, we have defined $\yp_i$ to be a copy of $\fp_i$, which means that the reconstruction error is always 0 and Equation \ref{eq:fs_opt1} reduces to
\begin{equation}
\label{eq:fs_opt2}
\fhp_i=\argmin_{\fp_i\in \R}\left(k_{ii}(\yp_i-\fp_i)\right)^2 \text{ s.t. } |\fp_i|\leq c =\argmin_{\fp_i\in \R}\left(k_{ii}(\yp_i-\fp_i)\right)^2 +\lambda_i|\fp_i|=\lambda_i|\fp_i|,
\end{equation}
for $\lambda_i=\max(0,k_{ii}(\yp_i-c))$, where the first equality is due to Lagrangian duality, and the second is due to the reconstruction error being 0.
First assume $\lambda_i=0$, in which case Equation \ref{eq:fs_opt2} reduces to the ill-posed problem $\fh_i=\argmin_{\fp_i\in \R} 0$. However, in this case, we can use Equation \ref{eq:krr_s_fs} with $\lambda=0$. Since a diagonal $\Kss$ implies $\Ks=\nv$, we obtain, for $i>n$, $\fhp_i=0$. If, on the other hand, $\lambda_i>0$, then $\fhp_i=\argmin_{\fp_i\in \R} \lambda_i|\fp_i|=0$. Thus, for $i>n$, regardless of $\lambda_i$, $\fhp_i=0$.
For $i\leq n$, we have $\yp_i=y_i$. Assume $y_i\geq 0$. Then, the optimal value for $\fp_i$ is $y_i$, unless $y_i>c$, then, due to convexity, the optimal value is $c$, i.e.\ $\fhp_i=\min(y_i,c)$. Accounting also for the case of $y_i< 0$, we obtain
$$\fhp_i(c)=
\begin{cases}
\sgn(y_i)\cdot\min(|y_i|,c) &\text{if }i\leq n\\
0 &\text{if }i> n.\\
\end{cases}
$$
The sign gradient flow solution is calculated as follows:
\begin{equation*}
\begin{aligned}
&\sgn\left(-\frac{\partial}{\partial\fpv(t)}\left(\left\|\ypv-\fpv(t)\right\|_{\Kss}^2\right)\right)=\sgn\left(\Kss(\ypv-\fpv(t))\right)\\
&=\sgn\left(\begin{bmatrix}\vrule\\k_{ii}(\yp_i-\fp_i(t))\\ \vrule\end{bmatrix}\right)
=\sgn\left(\begin{bmatrix}[k_{ii}(y_i-f_i(t))]_{i=1}^n \\\nv\end{bmatrix}\right)=
\begin{bmatrix}[\sgn(k_{ii}(y_i-f_i(t)))]_{i=1}^n\\\nv\end{bmatrix},
\end{aligned}
\end{equation*}
where $[\sgn(k_{ii}(y_i-f_i(t)))]_{i=1}^n\in \R^n$ and $\nv\in \R^{\ns}$.
Since element $i$ in the vector depends on $\fpv$ only through $\fp_i$, and since $k_{ii}>0$,
\begin{equation*}
\begin{aligned}
&\frac{\partial\fpv(t)}{\partial t}=\sgn\left(-\frac{\partial}{\partial\fpv(t)}\left(\left\|\ypv-\fpv(t)\right\|_{\Kss}^2\right)\right)\\
\iff&\frac{\partial \fp_i(t)}{\partial t}
= \begin{cases}
\sgn(k_{ii}(y_i-f_i(t))) &\text{if }i\leq n\\
0 &\text{if }i> n
\end{cases}
= \begin{cases}
1&\text{if }i\leq n \text{ and } y_i>\fp_i(t)\\
-1&\text{if }i\leq n \text{ and } y_i<\fp_i(t)\\
0&\text{if }i>n \text{ or } y_i=\fp_i(t).
\end{cases}
\end{aligned}
\end{equation*}
Thus, since $\fp_i(0)=0$, $\fp_i(t)=0$, for $i>n$. For $i\leq n$, $\fp_i(t)=\pm t$, depending on the sign of $y_i$. Once $\fp_i=y_i$ (and $\sgn(k_{ii}(y_i-\fp_i)=0$), then $\fp_i=y$:
$$\fhp_i(t)=
\begin{cases}
\sgn(y_i)\cdot\min(t,|y_i|) &\text{if }i\leq n\\
0 &\text{if }i> n.\\
\end{cases}
$$
\end{proof}
\end{document}
\begin{thebibliography}{}
\bibitem[Ali et~al., 2019]{ali2019continuous}
Ali, A., Kolter, J.~Z., and Tibshirani, R.~J. (2019).
\newblock A continuous-time view of early stopping for least squares
regression.
\newblock In {\em The 22nd International Conference on Artificial Intelligence
and Statistics}, pages 1370--1378. PMLR.
\bibitem[Ali et~al., 2020]{ali2020complete}
Ali, M., Prasad, R., Xiang, Y., and Yaseen, Z.~M. (2020).
\newblock Complete ensemble empirical mode decomposition hybridized with random
forest and kernel ridge regression model for monthly rainfall forecasts.
\newblock {\em Journal of Hydrology}, 584:124647.
\bibitem[Allerbo et~al., 2023]{allerbo2022elastic}
Allerbo, O., Jonasson, J., and Jörnsten, R. (2023).
\newblock Elastic gradient descent, an iterative optimization method
approximating the solution paths of the elastic net.
\newblock {\em Journal of Machine Learning Research}, 24(277):1--53.
\bibitem[Balles and Hennig, 2018]{balles2018dissecting}
Balles, L. and Hennig, P. (2018).
\newblock Dissecting adam: The sign, magnitude and variance of stochastic
gradients.
\newblock In {\em International Conference on Machine Learning}, pages
404--413. PMLR.
\bibitem[Bondell and Reich, 2008]{bondell2008simultaneous}
Bondell, H.~D. and Reich, B.~J. (2008).
\newblock Simultaneous regression shrinkage, variable selection, and supervised
clustering of predictors with oscar.
\newblock {\em Biometrics}, 64(1):115--123.
\bibitem[Chen and Leclair, 2021]{chen2021optimizing}
Chen, H. and Leclair, J. (2021).
\newblock Optimizing etching process recipe based on kernel ridge regression.
\newblock {\em Journal of Manufacturing Processes}, 61:454--460.
\bibitem[Chen et~al., 2022]{chen2022kernel}
Chen, Z., Hu, J., Qiu, X., and Jiang, W. (2022).
\newblock Kernel ridge regression-based tv regularization for motion correction
of dynamic mri.
\newblock {\em Signal Processing}, 197:108559.
\bibitem[Demontis et~al., 2017]{demontis2017infinity}
Demontis, A., Biggio, B., Fumera, G., Giacinto, G., Roli, F., et~al. (2017).
\newblock Infinity-norm support vector machines against adversarial label
contamination.
\newblock In {\em CEUR Workshop Proceedings}, volume 1816, pages 106--115.
CEUR-WS.
\bibitem[Efron et~al., 2004]{efron2004least}
Efron, B., Hastie, T., Johnstone, I., Tibshirani, R., et~al. (2004).
\newblock Least angle regression.
\newblock {\em Annals of Statistics}, 32(2):407--499.
\bibitem[Fan et~al., 2021]{fan2021well}
Fan, P., Deng, R., Qiu, J., Zhao, Z., and Wu, S. (2021).
\newblock Well logging curve reconstruction based on kernel ridge regression.
\newblock {\em Arabian Journal of Geosciences}, 14(16):1--10.
\bibitem[Feng et~al., 2016]{feng2016kernelized}
Feng, Y., Lv, S.-G., Hang, H., and Suykens, J.~A. (2016).
\newblock Kernelized elastic net regularization: generalization bounds, and
sparse recovery.
\newblock {\em Neural Computation}, 28(3):525--562.
\bibitem[Friedman and Popescu, 2004]{friedman2004gradient}
Friedman, J. and Popescu, B.~E. (2004).
\newblock Gradient directed regularization.
\newblock {\em Unpublished manuscript, http://www-stat.stanford.edu/\~{}
jhf/ftp/pathlite.pdf}.
\bibitem[Guigue et~al., 2005]{guigue2005kernel}
Guigue, V., Rakotomamonjy, A., and Canu, S. (2005).
\newblock Kernel basis pursuit.
\newblock In {\em ECML}, pages 146--157. Springer.
\bibitem[Hastie et~al., 2007]{hastie2007forward}
Hastie, T., Taylor, J., Tibshirani, R., Walther, G., et~al. (2007).
\newblock Forward stagewise regression and the monotone lasso.
\newblock {\em Electronic Journal of Statistics}, 1:1--29.
\bibitem[Hinton et~al., 2012]{hinton2012neural}
Hinton, G., Srivastava, N., and Swersky, K. (2012).
\newblock Neural networks for machine learning lecture 6a overview of
mini-batch gradient descent.
\bibitem[Kingma and Ba, 2014]{kingma2014adam}
Kingma, D.~P. and Ba, J. (2014).
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}.
\bibitem[Krige, 1951]{krige1951statistical}
Krige, D.~G. (1951).
\newblock A statistical approach to some basic mine valuation problems on the
witwatersrand.
\newblock {\em Journal of the Southern African Institute of Mining and
Metallurgy}, 52(6):119--139.
\bibitem[Le et~al., 2021]{le2021fingerprinting}
Le, Y., Jin, S., Zhang, H., Shi, W., and Yao, H. (2021).
\newblock Fingerprinting indoor positioning method based on kernel ridge
regression with feature reduction.
\newblock {\em Wireless Communications and Mobile Computing}, 2021.
\bibitem[Ma and Belkin, 2017]{ma2017diving}
Ma, S. and Belkin, M. (2017).
\newblock Diving into the shallows: a computational perspective on large-scale
shallow learning.
\newblock {\em Advances in Neural Information Processing Systems}, 30.
\bibitem[Mason et~al., 1999]{mason1999boosting}
Mason, L., Baxter, J., Bartlett, P., and Frean, M. (1999).
\newblock Boosting algorithms as gradient descent.
\newblock {\em Advances in Neural Information Processing Systems}, 12.
\bibitem[Matheron, 1963]{matheron1963principles}
Matheron, G. (1963).
\newblock Principles of geostatistics.
\newblock {\em Economic Geology}, 58(8):1246--1266.
\bibitem[Mercer, 1909]{mercer1909xvi}
Mercer, J. (1909).
\newblock Xvi. functions of positive and negative type, and their connection
the theory of integral equations.
\newblock {\em Philosophical Transactions of the Royal Society of London.
Series A, Containing Papers of a Mathematical or Physical Character},
209(441-458):415--446.
\bibitem[Nesterov, 1983]{nesterov1983method}
Nesterov, Y. (1983).
\newblock A method for unconstrained convex minimization problem with the rate
of convergence o (1/k\^{} 2).
\newblock In {\em Doklady an USSR}, volume 269, pages 543--547.
\bibitem[Pace and Barry, 1997]{pace1997sparse}
Pace, R.~K. and Barry, R. (1997).
\newblock Sparse spatial autoregressions.
\newblock {\em Statistics \& Probability Letters}, 33(3):291--297.
\bibitem[Polyak, 1964]{polyak1964some}
Polyak, B.~T. (1964).
\newblock Some methods of speeding up the convergence of iteration methods.
\newblock {\em USSR Computational Mathematics and Mathematical Physics},
4(5):1--17.
\bibitem[Raskutti et~al., 2014]{raskutti2014early}
Raskutti, G., Wainwright, M.~J., and Yu, B. (2014).
\newblock Early stopping and non-parametric regression: an optimal
data-dependent stopping rule.
\newblock {\em Journal of Machine Learning Research}, 15(1):335--366.
\bibitem[Rockafellar, 1976]{rockafellar1976monotone}
Rockafellar, R.~T. (1976).
\newblock Monotone operators and the proximal point algorithm.
\newblock {\em SIAM Journal on Control and Optimization}, 14(5):877--898.
\bibitem[Roth, 2004]{roth2004generalized}
Roth, V. (2004).
\newblock The generalized lasso.
\newblock {\em IEEE Transactions on Neural Networks}, 15(1):16--28.
\bibitem[Russu et~al., 2016]{russu2016secure}
Russu, P., Demontis, A., Biggio, B., Fumera, G., and Roli, F. (2016).
\newblock Secure kernel machines against evasion attacks.
\newblock In {\em Proceedings of the 2016 ACM Workshop on Artificial
Intelligence and Security}, pages 59--69.
\bibitem[Safari and Rahimzadeh~Arashloo, 2021]{safari2021kernel}
Safari, M. J.~S. and Rahimzadeh~Arashloo, S. (2021).
\newblock Kernel ridge regression model for sediment transport in open channel
flow.
\newblock {\em Neural Computing and Applications}, 33(17):11255--11271.
\bibitem[Shahsavar et~al., 2021]{shahsavar2021experimental}
Shahsavar, A., Jamei, M., and Karbasi, M. (2021).
\newblock Experimental evaluation and development of predictive models for
rheological behavior of aqueous fe3o4 ferrofluid in the presence of an
external magnetic field by introducing a novel grid optimization based-kernel
ridge regression supported by sensitivity analysis.
\newblock {\em Powder Technology}, 393:1--11.
\bibitem[Singh~Alvarado et~al., 2021]{singh2021neural}
Singh~Alvarado, J., Goffinet, J., Michael, V., Liberti, W., Hatfield, J.,
Gardner, T., Pearson, J., and Mooney, R. (2021).
\newblock Neural dynamics underlying birdsong practice and performance.
\newblock {\em Nature}, 599(7886):635--639.
\bibitem[Tibshirani, 1996]{tibshirani1996regression}
Tibshirani, R. (1996).
\newblock Regression shrinkage and selection via the lasso.
\newblock {\em Journal of the Royal Statistical Society: Series B
(Methodological)}, 58(1):267--288.
\bibitem[Tibshirani, 2015]{tibshirani2015general}
Tibshirani, R.~J. (2015).
\newblock A general framework for fast stagewise algorithms.
\newblock {\em Journal of Machine Learning Research}, 16(1):2543--2588.
\bibitem[Wood et~al., 2017]{wood2017generalized}
Wood, S.~N., Li, Z., Shaddick, G., and Augustin, N.~H. (2017).
\newblock Generalized additive models for gigadata: Modeling the uk black smoke
network daily data.
\newblock {\em Journal of the American Statistical Association},
112(519):1199--1210.
\bibitem[Wu et~al., 2021]{wu2021increasing}
Wu, Y., Prezhdo, N., and Chu, W. (2021).
\newblock Increasing efficiency of nonadiabatic molecular dynamics by
hamiltonian interpolation with kernel ridge regression.
\newblock {\em The Journal of Physical Chemistry A}, 125(41):9191--9200.
\bibitem[Yao et~al., 2007]{yao2007early}
Yao, Y., Rosasco, L., and Caponnetto, A. (2007).
\newblock On early stopping in gradient descent learning.
\newblock {\em Constructive Approximation}, 26(2):289--315.
\bibitem[Yuan and Lin, 2006]{yuan2006model}
Yuan, M. and Lin, Y. (2006).
\newblock Model selection and estimation in regression with grouped variables.
\newblock {\em Journal of the Royal Statistical Society: Series B (Statistical
Methodology)}, 68(1):49--67.
\bibitem[Zahrt et~al., 2019]{zahrt2019prediction}
Zahrt, A.~F., Henle, J.~J., Rose, B.~T., Wang, Y., Darrow, W.~T., and Denmark,
S.~E. (2019).
\newblock Prediction of higher-selectivity catalysts by computer-driven
workflow and machine learning.
\newblock {\em Science}, 363(6424):eaau5631.
\bibitem[Zhou et~al., 2010]{zhou2010exclusive}
Zhou, Y., Jin, R., and Hoi, S. C.-H. (2010).
\newblock Exclusive lasso for multi-task feature selection.
\newblock In {\em Proceedings of the Thirteenth International Conference on
Artificial Intelligence and Statistics}, pages 988--995.
\bibitem[Zou, 2006]{zou2006adaptive}
Zou, H. (2006).
\newblock The adaptive lasso and its oracle properties.
\newblock {\em Journal of the American Statistical Association},
101(476):1418--1429.
\end{thebibliography}