Sharper Regret Bounds for Time-Varying Gaussian Process Bandits with Constant Exploration

Many sequential decision problems require optimizing an unknown objective through a limited number of costly evaluations. Examples include tuning the hyperparameters of machine learning models (20), adapting control policies in robotics (8), and selecting treatment strategies in clinical trials (24). In these settings, each evaluation reveals only imperfect information about the objective, forcing a fundamental trade-off between exploration and exploitation.

Bayesian optimization (BO) provides a framework for sequentially optimizing an unknown objective by combining a probabilistic surrogate model with an acquisition rule. In the standard GP bandit formulation, the objective is a fixed function $f:\mathcal{S}\to\mathbb{R}$. At each round $t$, the learner selects $x_{t}\in\mathcal{S}$ and observes $y_{t}=f(x_{t})+z_{t}$, where $z_{t}$ is observation noise. Gaussian processes (GPs) are commonly used as surrogate models because they provide posterior mean and uncertainty estimates in closed form.

When decisions affect performance throughout the learning process, performance is typically measured by cumulative regret, $R_{T}=\sum_{t=1}^{T}(\max_{x\in\mathcal{S}}f(x)-f(x_{t}))$. This criterion originates in the multi-armed bandit literature (17; 16; 2). Algorithms such as Gaussian Process Upper Confidence Bound (GP-UCB) (21) and GP-Thompson sampling (18; 9) extend bandit methods to continuous domains and achieve sublinear regret under suitable kernel regularity assumptions, with guarantees typically expressed through the maximum information gain (21; 23; 19).

However, many applications are inherently dynamic. System performance may drift due to environmental changes or evolving user behavior, necessitating a time-dependent reward model $f_{t}:\mathcal{S}\to\mathbb{R}$. When the objective changes over time, sublinear cumulative regret is generally unachievable. The relevant theoretical goal shifts to bounding the average regret, $R_{T}/T$, as a function of the environmental drift rate.

A principled probabilistic model for this setting was introduced by 5. In their time-varying Gaussian process (TV-GP) model, the latent function evolves according to a Markov recursion driven by GP innovations, which induces a separable spatio-temporal kernel. Under this model, the learner updates a temporally discounted GP posterior and selects actions using a modified GP-UCB rule. Their analysis shows that near-optimal regret rates are achievable up to logarithmic factors. Subsequent work has studied related models of nonstationarity, including change-point structure (6), irregular evaluation times (13), and bounded variation in RKHS norms (10). A separate frequentist line studies nonstationary kernelized bandits under sup-norm variation, including algorithms (25; 12; 14) and lower bounds (7). Other extensions include kernel-agnostic adaptation (26) and stale-data removal in dynamic Bayesian optimization (3; 4).

A recurring theoretical and practical limitation in these analyses is the exploration schedule. Existing regret guarantees typically require the GP-UCB exploration parameter, $\beta_{t}$, to grow with time $t$. This leads to deteriorating performance on long time horizons and introduces schedule tuning parameters. In practice, such schedules are difficult to calibrate and induce unnecessary over-exploration.

This paper revisits the TV-GP setting of 5, where the temporal evolution is governed by a drift parameter $\epsilon>0$, and shows that GP-UCB can be run with a constant exploration parameter while retaining rigorous performance guarantees. Our analysis uses local confidence events at each round rather than a single event that holds uniformly over the horizon. For any fixed drift level, Theorem 1 gives an expected regret bound with linear dependence on $T$ and a coefficient controlled by $\tilde{\gamma}_{T}/T$. In the persistent-drift regime, this yields a coefficient that decreases as the drift rate decreases. This differs from standard deterministic uniform-confidence GP-UCB analyses in static environments, where the confidence parameter generally depends on the horizon or time.

The analysis also gives a sharper dependence on the drift rate through the time-varying maximum information gain. If the static information gain satisfies $\gamma_{N}=\widetilde{\mathcal{O}}(N^{q})$, then in the persistent-drift regime

For the squared exponential kernel, the latter becomes $\widetilde{\mathcal{O}}(\epsilon^{1/4})$. Our analysis also isolates how the bound-suggested constant exploration parameter depends on the drift rate. In the regret bounds of 5, the prescribed exploration schedule is independent of $\epsilon$. In contrast, balancing the upper-bound scaling suggests that the bound-suggested constant $\beta$ increases as $\epsilon$ decreases. This does not by itself imply uniformly more exploration, since the UCB rule depends on the product of $\sqrt{\beta}$ and the posterior standard deviation. Rather, when drift is slower, past observations remain informative for longer and the posterior variance is typically smaller. A larger $\beta$ compensates for this reduced variance by assigning more value to reducing the remaining uncertainty, which is also more consequential because information acquired now remains useful over a longer effective horizon.

The dependence of the bound-suggested exploration parameter on the temporal dynamics also gives a simple rule for changes in sampling frequency. If observations are collected every $\Delta$ time units, larger $\Delta$ corresponds to greater effective drift between successive evaluations. Temporal correlations then decay faster, past observations become obsolete more quickly, and the analysis predicts a smaller bound-suggested exploration parameter.

The remainder of the paper is organized as follows. Section 2 introduces the time-varying GP model, the domain regularity assumptions, and the temporally discounted GP-UCB policy. Section 3 presents the expected-regret bound and the post-hoc and prescribed-confidence guarantees for realized regret. Section 4 derives the drift-dependent information-gain and regret rates and analyzes the bound-suggested exploration parameter. Section 5 provides a supporting empirical simulation study. Section 6 provides the conclusion. Additional proofs and technical results are deferred to the appendices.

We start by introducing the dynamic regret setting. Let $\mathcal{S}\subseteq\mathbb{R}^{d}$ be compact. At each round $t=1,2,\ldots$, the algorithm selects a query point $x_{t}\in\mathcal{S}$ and observes

where $f_{t}:\mathcal{S}\to\mathbb{R}$ is an unknown reward function that changes with time. Let

denote a maximizer at round $t$. The instantaneous regret is

and the cumulative regret over $T$ rounds is

In the non-stationary setting, sublinear cumulative regret is not expected in general, so the main objective is to control the linear growth rate and its dependence on the temporal properties of $f_{t}$. We model the function sequence $f_{t}$ using the time-varying Gaussian process construction of 5. Let $(g_{t})_{t\geq 1}$ be i.i.d. draws from $\mathcal{GP}(0,k_{S})$, where $k_{S}$ is a spatial kernel on $\mathcal{S}$. The latent process is defined by

with drift parameter $\epsilon\in(0,1]$. For each fixed $t$, the marginal law remains $\mathcal{GP}(0,k_{S})$, while the temporal dependence decays geometrically. Equivalently, $\{f_{t}(x)\}_{(x,t)}$ is a Gaussian process on $\mathcal{S}\times\mathbb{N}$ with separable kernel

We refer to (5) as the time-decayed kernel, since its temporal factor discounts correlations geometrically with temporal separation. We assume throughout that the prior variance is uniformly bounded,

Given past data

Gaussian process regression under the kernel (5) yields a Gaussian posterior for $f_{t}(x)$ with mean and variance

where

These expressions are obtained by evaluating the spatio-temporal kernel (5) at the observed pairs $(x_{i},i)$ and the prediction point $(x,t)$. The factors $(\sqrt{1-\epsilon})^{|t-i|}$ account for the fact that older observations are less informative about the current function $f_{t}$.

As in standard GP bandit analysis, we measure cumulative uncertainty by the time-varying maximum information gain (5), which controls sums of posterior variances in regret bounds (21, Lemma 5.4). For a design set

let $f_{A}=(f_{1}(x_{1}),\ldots,f_{T}(x_{T}))^{\top}$ and $y_{A}=(y_{1},\ldots,y_{T})^{\top}$. Since $y_{A}=f_{A}+z_{A}$ with independent Gaussian noise,

where $\tilde{K}_{A}$ is the covariance matrix of $f_{A}$ induced by (5). We define the time-varying maximum information gain as

To control the discretization error incurred on a continuous domain, we impose the following standard GP-UCB regularity condition (21; 5).

This condition holds for many commonly used kernels, including the squared exponential kernel and Matérn kernels with smoothness parameter $\nu>2$, under appropriate differentiability conditions on $k_{S}$; see 11. The TV-GP-UCB selection rule is given in Algorithm 1. At round $t$, the algorithm selects

The main result of this paper shows that under the time-varying model (4), one can take $\beta_{t}\equiv\beta$ constant and obtain a regret guarantee with improved long-horizon scaling. This contrasts with earlier analyses that rely on time-increasing exploration schedules.

Static GP-UCB analyses usually prove a single confidence event that holds simultaneously over all rounds and all points in a discretization. To keep this event at fixed probability over a horizon $T$, the per-round failure probabilities must be summable, which forces the exploration parameter $\beta_{t}$ to grow with $t$. Existing analyses of the time-varying GP model (4), in particular 5, use the same time-increasing exploration schedule. This introduces horizon-dependent tuning, leads to average-regret bounds that grow with $T$, and obscures the dependence of exploration on the drift rate $\epsilon$.

We avoid this global-in-time confidence event. When $\epsilon>0$, older observations are progressively discounted, so posterior uncertainty need not decrease monotonically. This built-in forgetting prevents the algorithm from becoming permanently overconfident at a fixed point and removes the need for a time-increasing exploration schedule in the analysis. We prove that Algorithm 1 with a constant exploration parameter attains an $\mathcal{O}(T)$ expected cumulative regret bound for every fixed $\epsilon>0$. The proof combines a per-round UCB regret bound on a local confidence event with an algorithm-agnostic bound on the rare complement event.

Theorem 1 gives a constant-exploration regret bound whose leading term is controlled by $\tilde{\gamma}_{T}/T$. We first combine it with the time-varying information-gain bound in Appendix B to derive explicit drift-dependent rates, and then examine the corresponding bound-suggested choice of $\beta$.

We empirically evaluate constant-$\beta$ GP-UCB under the TV-GP model on the spatial domain $\mathcal{S}=[0,1]^{2}$. The domain is discretized using a $20\times 20$ grid, giving a state dimension of $N=400$. The latent function uses a radial basis function (RBF, squared exponential) kernel with lengthscale $l=0.1$ and prior variance $1.0$, while the observation noise variance is fixed at $\sigma^{2}=0.01$. We simulate $T=10{,}000$ rounds for each parameter pair $(\epsilon,\beta)$ and record the time-averaged regret $R_{T}/T$ over 100 independent trials. This horizon is used as a long-run stress test rather than as an application-specific benchmark; it is made feasible by the Kalman implementation described below. The simulation setup and compute resources are reported in Appendix E, and the accompanying code archive contains the scripts used to reproduce Figure 1.

Existing theoretical guarantees for Gaussian process bandits in time-varying environments typically require exploration schedules that grow with time. We showed that, under the TV-GP drift model, a constant exploration parameter can instead be analyzed using local confidence events at each round rather than a single horizon-wide confidence event. The resulting expected-regret bound has a drift-dependent leading term controlled by the time-varying information gain, and the same local-confidence analysis yields realized-regret guarantees with $\beta$ constant in $t$ and $T$.

We also derived a sharper bound on the time-varying maximum information gain. If $\gamma_{N}=\widetilde{\mathcal{O}}(N^{q})$, the resulting persistent-drift average-regret rate is $\widetilde{\mathcal{O}}(\epsilon^{(1-q)/(4-q)})$; for the squared exponential kernel this gives $\widetilde{\mathcal{O}}(\epsilon^{1/4})$. These rates apply when the optimized block length fits within the horizon and are not fixed-$T$, $\epsilon\to 0$ statements. Separately, balancing the remaining terms in the regret bound suggests a bound-suggested constant exploration parameter with $\beta_{\mathrm{opt}}(\epsilon)\propto\log(1/\epsilon)$. This provides a simple tuning principle when changing the sampling frequency changes the effective drift between observations. The same Markov structure also implies exponential covariance decay for the oracle-value process, so long-run average performance can be compared against a deterministic benchmark.

Several extensions remain open. Most importantly, the present analysis relies on the geometric temporal kernel induced by the Markov drift model. Extending constant-exploration guarantees to more general temporal kernels would clarify which forms of nonstationarity are sufficient to prevent posterior overconfidence. Another important direction is to remove the assumption that the drift parameter is known and fixed, allowing $\beta$ to be adapted online when the rate of temporal variation is unknown or changes over time. Finally, sharper concentration tools for the realized regret of TV-GP-UCB could improve the current high-probability bounds and better reflect the empirical stability observed in the simulations.

The first author gratefully acknowledges support from the NWO Spinoza Prize awarded to Aad van der Vaart.