Non-Stationary Lipschitz Bandits
Extracted source text · arXiv 2505.18871
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
%\usepackage{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
\usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{subcaption}
\usepackage{alias}
%\usepackage{aistats2025}
%\usepackage{alias}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage[T1]{fontenc}
\usepackage{tikz}
\usepackage{framed}
\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}
\newtheorem{fact}{Fact}
\newtheorem{corollary}{Corollary}
\newtheorem{proposition}{Proposition}
\newtheorem{remark}{Remark}
\newtheorem{definition}{Definition}
\newtheorem{example}{Example}
\usepackage{newtxtext}
\bibliographystyle{plainnat}
\usepackage{natbib}
\title{Non-Stationary Lipschitz Bandits}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{Nicolas Nguyen \\
University of Tübingen \\
\texttt{nicolas.nguyen@uni-tuebingen.de} \\
% examples of more authors
\And
Solenne Gaucher \\
École Polytechnique, CMAP \\
\texttt{solenne.gaucher@polytechnique.edu} \\
\AND
Claire Vernade \\
University of Tübingen \\
\texttt{claire.vernade@uni-tuebingen.de} \\
}
\begin{document}
\maketitle
\begin{abstract}
We study the problem of non-stationary Lipschitz bandits, where the number of actions is infinite and the reward function, satisfying a Lipschitz assumption, can change arbitrarily over time. We design an algorithm that adaptively tracks the recently introduced notion of significant shifts, defined by large deviations of the cumulative reward function. To detect such reward changes, our algorithm leverages a hierarchical discretization of the action space. Without requiring any prior knowledge of the non-stationarity, our algorithm
achieves a minimax-optimal dynamic regret bound of $\mathcal{\widetilde{O}}(\Tilde{L}^{1/3}T^{2/3})$, where $\Tilde{L}$ is the number of significant shifts and $T$ the horizon. This result provides the first optimal guarantee in this setting.
\end{abstract}
\input{Introduction}
\input{Setting}
\input{Algorithm}
\input{MainResults}
\input{Proof_Sketch}
\input{Conclusion}
\input{broader_impact}
\input{acknowledgements}
\newpage
\bibliography{references}
\newpage
\input{Checklist}
\newpage
\appendix
\input{Appendix}
\end{document}
\section{Introduction}\label{sec:introduction}
%
%Bandits algorithms \citep{slivkins2019introduction,lattimore2020bandit}.
Many practical applications involve decision-making over a continuous space of actions (or \emph{arms}) where the environment evolves over time. For example, in dynamic pricing, the space of possible prices is inherently continuous, and market dynamics may shift abruptly. In such a setting, it is both natural and crucial to leverage smoothness in the reward function to generalize across similar actions, while also remaining adaptive to temporal changes. However, most existing works on non-stationary bandits focus on discrete action spaces and overlook the structure present in continuous domains.
The non-stationary bandit problem extends the classical stochastic multi-armed bandit setting \citep{slivkins2019introduction, lattimore2020bandit} by allowing the mean reward functions to change over time. Over the past two decades, several frameworks have been proposed to quantify non-stationarity. Common approaches quantify the number of changes $L_T$ of any arm’s mean reward over $T$ rounds \citep{auer2002nonstochastic, garivier2011upper}, or their total variation $V_T$ in the reward functions \citep{besbes2014stochastic, cheung2019learning}. Algorithms that are built upon these approaches achieve dynamic regret bounds of order $\sqrt{L_TT}$ or $V_T^{1/3}T^{2/3}$ respectively. Importantly, they all assume prior knowledge of the degree of non-stationarity $L_T$ or $V_T$. It remained an open question whether such rates could be achieved without knowing $L_T$ or $V_T$, until a line of work answered this affirmatively \citep{auer2019achieving, auer2019adaptively, chen2019new, wei2021non}.
Recent works have introduced more refined measures of non-stationarity. In fact, the standard metrics $L_T$ and $V_T$ can be overly pessimistic: both of them may be arbitrarily large even when the best arm remains unchanged. To address this, \citet{abbasi2023new} designed an algorithm that tracks the number of times the best arm changes over $T$ rounds, $S_T$, and showed that it achieves $\sqrt{S_TT}$ for the dynamic regret, \emph{adaptively}\footnote{In all our work, "adaptively" should be understood the same as "without knowledge of the non-stationarity".}. The important work of \citet{suk2022tracking} proposed a finer notion of non-stationarity called \emph{significant shifts}, which identifies only changes that meaningfully affect regret through large aggregate differences in reward. In the $K$-armed bandit setting, their algorithm achieves the optimal rate of $\sqrt{\Tilde{L}_TT}$ where $\Tilde{L}_T$ is the number of significant shifts over $T$ rounds, with $\Tilde{L}_T\ll L_T$.
In this work, we extend the ideas of \citet{suk2022tracking} to the setting of Lipschitz bandits, where the arm space is continuous (typically the interval $[0,1]$) and the reward functions satisfy a Lipschitz condition at each round. Despite the maturity of research in both non-stationary and Lipschitz bandits individually, their intersection remains unexplored. Our work is the first one to tackle the problem of non-stationary Lipschitz bandits.
\subsection{Outline and contributions}
In this work, we answer the following question positively: \emph{Can we design an algorithm that adaptively achieves the minimax-optimal rate in non-stationary Lipschitz bandits?} We argue that non-stationary Lipschitz bandits pose unique and fundamental challenges that go beyond a simple combination of non-stationarity and Lipschitz continuity. We highlight two central challenges addressed in this paper.
\textbf{Discretization level.} In the stationary Lipschitz bandit setting, a common approach is to discretize the continuous arm space into a finite set of \emph{bins}, thereby allowing the application of standard multi-armed bandit algorithms \citep{kleinberg2004nearly}. When the time horizon $T$ is known, the discretization level can be carefully calibrated, typically using $\mathcal{O}(T^{1/3})$ bins. This choice reflects the optimal balance between discretization error and the regret incurred by running a bandit algorithm over the discretized arms, leading to the minimax-optimal regret of $\mathcal{O}(T^{2/3})$.
In the non-stationary setting, however, the durations of the time intervals during which the mean reward remains stable are unknown and not rigorously defined. Thus, direct horizon-dependent tuning becomes infeasible. This raises a key question: how to adaptively adjust the discretization level to match the unknown degree of non-stationarity in the environment?
\textbf{Forced exploration via restarts and replays.} In non-stationary bandits, detecting changes in the reward distribution requires frequent exploration of suboptimal arms. In the $K$-armed bandit setting, this is often handled through periodic restarts, where all $K$ arms are re-explored to detect shifts of the reward function. In the continuous setting, this strategy is significantly harder: not only must we decide when to restart, but also at what resolution to discretize the arm space during such restarts. Finer discretization enables more precise detection of changes but significantly increases the cost of exploration. Balancing this trade-off is a central technical challenge tackled in our work.
This paper is organized as follows. In \cref{sec:problem_setting}, we formally introduce the problem setting, and present our algorithm, \algo, in \cref{sec:algorithm}. Our main results are stated in \cref{sec:main_results}, followed by a high-level proof sketch of its regret analysis in \cref{sec:proof_sketch}. For completeness, the full pseudocode of \algo is provided in \cref{app:sec:pseudo_code}, and all detailed regret proofs can be found in \cref{app:sec:proof_th_upper_bound}.
\subsection{Related Work}\label{subsec:related_works}
\textbf{Non-stationary bandits beyond $K$-armed.} A growing body of work has extended non-stationary bandit models beyond the classical $K$-armed setting. These include linear and generalized linear bandits \citep{russac2019weighted, faury2021optimal}, contextual bandits \citep{suk2021self}, convex bandits \citep{zhao2021bandit, wang2023adaptivity,liu2025non}, kernelized bandits \citep{hong2023optimization, iwazaki2024near, cai2024lower}, and more structured problems \citep{seznec2020single, vernade2020non, azizi2022non}. Some works further explore temporal regularity assumptions, such as smooth or slowly varying reward functions \citep{slivkins2008adapting, krishnamurthy2021slowly, jia2023smooth}. Across these settings, non-stationarity is typically quantified using either the number of global changes $L_T$ or the total variation $V_T$.
\textbf{Bandits with infinite arms.} In Lipschitz bandits \citep{agrawal1995continuum}, the infinite arm space is tackled via discretization strategies, either fixed or adaptive, that exploit the Lipschitz continuity of the reward function \citep{kleinberg2004nearly, kleinberg2008multi, bubeck2011lipschitz, bubeck2011x, magureanu2014lipschitz}. More recently, adversarial formulations have been studied, where the goal is to minimize the \emph{regret in hindsight} \citep{podimata2021adaptive, kang2023robust}. These differ fundamentally from our focus on dynamic regret in a stochastic setting. A separate line of work investigates infinite-armed bandits in the so-called \emph{reservoir model}, where arm means are drawn independently from a fixed distribution \citep{wang2008algorithms, carpentier2015simple, de2021bandits, kim2022rotting, kim2024adaptive}. Unlike Lipschitz bandits, these models assume no structural relationship between arms and therefore do not exploit similarity across them. Hence, direct comparisons to our setting do not apply. The only work we are aware of that addresses the problem of non-stationary Lipschitz bandits is \citet{kang2023online}, who propose a zooming-like algorithm to tune a generalized bandit framework. However, their formulation of non-stationary Lipschitz bandits is tailored to their specific setting and does not achieve optimal theoretical guarantees.
\textbf{Tracking significant shifts.} The idea of tracking \emph{significant shifts} was first introduced in the $K$-armed setting by \citet{suk2022tracking}, and has since been extended to contextual bandits \citep{suk2023tracking}, preference-based settings \citep{suk2023can, buening2023anaconda, suk2024non}, and smooth non-stationary models with temporal structure \citep{suk2024adaptive}.
The works closest to ours are \citet{suk2023tracking} and \citet{suk2025tracking}, both extending the framework of tracking significant shifts beyond the classical $K$-armed setting. The former considers contextual bandits where rewards are Lipschitz in the context space, but with a finite number of arms. After discretizing the context space, their algorithm effectively reduces to a finite-arm bandit problem at each context, allowing the use of techniques from the $K$-armed setting \citep{suk2022tracking} without requiring discretization of the arm space. The latter studies infinite-armed bandits under a reservoir model, where arms are independently drawn from a fixed distribution. While this shares the challenge of choosing from infinitely many arms, their setting assumes no regularity across arms. Therefore, their techniques and guarantees are not applicable to our setting.
\subsection{Notations}
For any integer $n$, we write $[\![n]\!] = \{1,\dots, n\}$. For a filtration $(\cF_t)_t$, we denote the conditional expectation by $\E{t}{\cdot} = \E{}{\cdot\condi\cF_{t}}$ (the underlying filtration will be clear from context). For real numbers $a$ and $b$, we write $a \vee b = \max\{a, b\}$ and $a \wedge b = \min\{a, b\}$.
\section{Problem setting}\label{sec:problem_setting}
Without loss of generality, we assume that the arm space is the unit interval $[0,1]$.\footnote{Extensions to arbitrary metric spaces and other Lipschitz constants are discussed in \cref{app:sec:extension_arbitrary_spaces}.} An \emph{oblivious adversary} selects a sequence of $T$ mean reward functions $(\mu_t)_{t=1}^T$, where each $\mu_t : [0,1] \to [0,1]$ satisfies the following Lipschitz condition:
\begin{assumption}[\textbf{Lipschitz mean rewards $\bm{\mu_t}$}]\label{assumption:lipschitz}
$\forall x,x'\in[0, 1], \, \left|\mu_t(x)-\mu_t(x') \right| \leq |x-x'| $.
\end{assumption}
At each round $t \in [T]$, an algorithm $\pi$ selects an arm $x_t \in [0,1]$ and observes a stochastic reward $Y_t(x_t) \in [0,1]$ independent from the past (arms and observations) and satisfying $\E{}{Y_t(x_t)\,|\,x_t=x} = \mu_t(x)$. The objective is
% to maximize the cumulative reward, or equivalently
to minimize the \emph{dynamic regret}
\begin{align*}
R(\pi, T) = \sum_{t=1}^T \sup_{x\in \cX}\mu_t(x) - \sum_{t=1}^T \mu_t(x_t) \,,
\end{align*}
defined as the difference between the cumulative gain of the algorithm and that of the dynamic benchmark oracle, which, at each round, knows the reward function $\mu_t$. In this paper, we design an algorithm and provide a bound on its \emph{expected dynamic regret} $\mathbb{E}[R(\pi,T)]$, where the expectation is taken with respect to the randomness of the interactions between the algorithm and the environment. We denote the instantaneous gap between two arms $x'$ and $x$ as $\delta_t(x',x) = \mu_t(x') - \mu_t(x)$, and the instantaneous regret of arm $x$ as $\delta_t(x) = \max_{x'\in[0, 1]}\mu_t(x') - \mu_t(x)$, so that $R(\pi, T) =\sum_{t=1}^T \delta_t(x_t)$.
\subsection{Significant regret and significant shifts}\label{subsec:sig_regret_and_sig_shifts}
We now introduce the notion of \emph{significant regret} for an arm $x\in[0, 1]$.
\begin{definition}[\textbf{Significant regret of an arm} $\bm{x}$]\label{def:significant_regret_arm}
An arm $x\in\cX$ incurs significant regret on interval $[s_1, s_2]$ if its cumulative regret on this interval is lower bounded as
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(x)\geq \log(T)(s_2 - s_1)^{2/3}\,.
\end{align*}
We call such arm an \textbf{unsafe} arm on interval $[s_1, s_2]$, and otherwise we call it \textbf{safe} on this interval.
\end{definition}
The right hand side of \cref{def:significant_regret_arm} corresponds to the minimax-optimal regret for a stationary Lipschitz bandit over horizon $s_2 - s_1$, and thus captures the level of regret that is statistically detectable.
Conversely, if no such interval exists where arm $x$ incurs significant regret, then its cumulative regret over the entire horizon must remain small.
A \emph{significant shift} occurs when \emph{all} arms become unsafe, \emph{i.e.} when each arm has incurred significant regret over some interval.
\begin{definition}[\textbf{Significant shift, significant phase}]\label{def:significant_shift}
Let $\tau_0 = 1$. For $i\geq1$, we define the $i^{\text{th}}$ significant shift as the smallest round $\tau_i\in]\tau_{i-1},T]$ such that for all arm $x\in\cX$, there exists an interval $[s_1, s_2]\subseteq[\tau_{i-1}, \tau_i]$ on which arm $x$ incurs significant regret (\cref{def:significant_regret_arm}).
For all $i$, we call $[\tau_i, \tau_{i+1}[$ a \textbf{significant phase}, and we denote by $\Tilde{L}_T$ the (unknown) \textbf{number of significant phases}.
\end{definition}
This definition implies that tracking significant shifts amounts to detecting intervals where the regret exceeds what would be expected in a stationary Lipschitz environment.
\subsection{Comparison with other non-stationary metrics.}
The notion of significant shifts provides a refined perspective compared to classical metrics such as the \emph{number of global changes} $L_T = \sum_{t=1}^{T-1} \mathbb{I}\left\{ \exists x \in [0,1] : \mu_{t+1}(x) \neq \mu_t(x) \right\}$, the \emph{total variation} $V_T = \sum_{t=1}^{T-1} \max_{x \in [0,1]} |\mu_{t+1}(x) - \mu_t(x)|$, and the \emph{number of best-arm changes} over $T$ rounds, $S_T = \sum_{t=1}^{T-1} \mathbb{I}\left\{ \argmax_{x\in[0, 1]} \mu_{t+1}(x) \neq \argmax_{x\in[0, 1]} \mu_t(x) \right\}.$
Unlike these metrics, $\tilde{L}_T$ is robust to benign transformations. For instance, if each $\mu_t$ is shifted by the same constant across all $x$, then $L_T$ and $V_T$ may scale linearly with $T$, while $\tilde{L}_T = 0$. Similarly, $S_T$ may count frequent changes in the optimal arm due to small fluctuations, even if the overall impact is negligible. In contrast, $\tilde{L}_T$ captures only statistically meaningful shifts and satisfies $\tilde{L}_T \leq S_T \leq L_T$.
For example, consider a recommendation system with a continuous pool of content (\emph{e.g.} movies), indexed by $x \in [0,1]$, where nearby values of $x$ correspond to similar content types. Suppose there exist two regions of high and comparable user preference, centred around $x_1 = 0.3$ and $x_2 = 0.7$. Imagine a scenario where preferences near $x_1$ remain stable over time (\emph{e.g.} a timeless classic), while preferences near $x_2$ undergo very frequent changes (\emph{e.g.} a trending topic that evolves daily). In this case, an algorithm that consistently recommends content near $x_1$ would incur little to no regret, even though the underlying reward function changes frequently in other regions. From a global perspective, the number of changes or the total variation in mean reward could be as large as $L_T = V_T= \mathcal{O}(T)$. An algorithm that relies solely on such metrics would unnecessarily restart its estimates too frequently, as it would overestimate the effective difficulty of the problem. This leads to both theoretical sub-optimality and practical inefficiency. However, such changes do \emph{not} constitute a significant shift, as the latter captures only changes that are \emph{statistically consequential} to learning, rather than indiscriminately counting all shifts in the environment.
\section{Algorithm \algo: Multi-Depth Bin Elimination}\label{sec:algorithm}
We now present the key ideas leading to the design of our algorithm, before detailing each one of them\footnote{The full pseudo-code of Multi-Depth Bin Elimination (\algo) can be found in \cref{app:sec:pseudo_code}.}. At a high level, our goal is to detect significant changes in the aggregate gaps $\sum_{t=s_1}^{s_2} \delta_t(x', x)$ between pairs of arms over time intervals. This quantity lower bounds the dynamic regret $\sum_{t=s_1}^{s_2} \delta_t(x)$ over this interval, meaning that large changes in these aggregate gaps are indicative of large dynamic regret. However, directly monitoring such changes for all possible pairs of arms is infeasible in a continuous action space.
To address this, we adopt a \textbf{discretization} strategy and instead monitor aggregate gaps between pairs of \emph{bins} $B,B'$ (\emph{i.e.}, subintervals of $[0, 1]$) in the form $\sum_{t=s_1}^{s_2} \delta_t(B', B)$. %By carefully choosing the bin width (\emph{i.e.}, the discretization level), we can control the discretization error, and estimate the gap between the actions contained within them.\\
\smallskip
Our algorithm operates in stable \textbf{episodes}, each split into \textbf{blocks} of doubling length with varying discretization levels. Within each block, a variant of the \textit{Successive Elimination} algorithm removes empirically suboptimal regions. Since discarded bins may become optimal later, we use the replay mechanism of \citet{suk2022tracking} to revisit them and avoid missing promising regions. A key challenge is selecting the right discretization level for each replay. Our \textbf{sampling procedure} guarantees sufficient exploration at different scales, while a carefully designed \textbf{bin eviction mechanism} progressively eliminates regions that appear suboptimal based on empirical evidence.
%Within each block, the algorithm selects a bin $B_t$ at a chosen depth and samples an action uniformly within it, \emph{i.e.}, $x_t \sim \mathcal{U}(B_t)$. This \textbf{sampling procedure} guarantees sufficient exploration across all currently active bins, while a carefully designed \textbf{bin eviction mechanism} progressively eliminates regions that appear suboptimal based on empirical evidence.
\textbf{Discretization.} To estimate the reward function at multiple discretization levels, we use a recursive dyadic partitioning of the space, described below.
%\subsection{Discretization scheme}
%We now introduce a hierarchical representation of the bins, which will allow us to effectively exploit the Lipschitz continuity of the reward functions.
\begin{definition}[\textbf{Depth and Dyadic tree}]\label{def:dyadic_tree}
$\cT_d$ denotes the partition of $[0, 1]$ into $2^d$ bins of size $1/2^d$ each. We say that $d$ is the corresponding \textbf{depth} of this discretization. We define the \textbf{dyadic tree} $\cT = \{\cT_d\}_{d\in\Nat}$ as the hierarchy of nested partitions of $[0, 1]$ at all possible depth $d\in\Nat$. For any bin $B\in\cT_d$, we denote by $\mathrm{Children}(B,d')$ the \textbf{set of children} of bin $B$ at depth $d'>d$, and $\Par(B, d'')$ the unique \textbf{parent bin} of $B$ at depth $d''<d$. In particular we have $d''<d<d'\implies\forall B'\in \mathrm{Children}(B,d')\,,\,B'\subset B \subset \Par(B, d'')$.
\end{definition}
For any bin $B\in\cT_d$, we define the mean reward of bin $B$ at round $t\in\llbracket T \rrbracket$ as
\begin{align*}
\mu_t(B) =\frac{1}{|B|}\int_{x\in B}\mu_t(x)\dint x\,,
\end{align*}
where $|B|=1/2^d$ denotes the width of bin $B\in\cT_d$. We define the instantaneous relative regret between two bins $B, B' \in \cT_d$ as $\delta_t(B', B) = \mu_t(B') - \mu_t(B)$, and introduce its empirical estimate, to be formalized later, as $\hat\delta_t(B', B) = \hat\mu_t(B') - \hat\mu_t(B)$. We also define the instantaneous regret of a bin $B\in\cT_d$ as $\delta_t(B)=\max_{B'\in\cT_d}\mu_t(B')-\mu_t(B)$. For simplicity, we use the same notation $\mu_t(B), \delta_t(B)$ and $\mu_t(x), \delta_t(x)$ to denote the mean and regret for bins and arms, respectively. The distinction should be clear from context.
%\subsection{\algo: \textbf{M}ulti- \textbf{D}epth \textbf{B}in \textbf{E}limination}\label{subsec:algorithm_MDBE}
%Our algorithm operates a hierarchical discretization over the dyadic tree $\cT$ to address non-stationarity at multiple scales.
%It is structured around \textbf{episodes}, further divided into \textbf{blocks}, where the action space is discretized at varying resolutions.
%At the core of the algorithm are \textbf{replays} \citep{chen2019new, wei2021non,suk2022tracking}, which allow the algorithm to detect and respond to shifts in reward by playing actions that have been evicted.
\textbf{Episodes and blocks.}
The algorithm proceeds in \emph{episodes}, indexed by $l \in \mathbb{N}$, with episode $l$ starting at round $t_l$ (with $t_0 = 1$). Each episode is divided into \emph{blocks}, where the $m^{\text{th}}$ block lasts for $8^m$ rounds. Within the $m^{\text{th}}$ block, the action space $[0,1]$ is discretized into $2^m = (8^m)^{1/3}$ bins, corresponding to the $m^{\text{th}}$ depth $\cT_m$. We denote the block interval by $[\tau_{l,m}, \tau_{l,m+1}[$, and maintain a set $\cBM \subseteq \cT_m$ of \emph{safe bins} at depth $m$ (referred to as the MASTER set).
A \emph{significant shift} is detected when $\cBM = \emptyset$, meaning all bins at depth $m$ have been evicted. In such cases, the block (and thus the episode) ends prematurely. Otherwise, a block ends naturally after $8^m$ rounds.
\begin{algorithm}
\caption{Routine procedure for one block}
\KwIn{Starting round of the block $\tau_{l,m}$.}
$\cBM\gets\cT_m$\tcp*{Initialize MASTER set with all bins at depth $m$}
Schedule replays for $t=\tau_{l,m}+1,\dots,\tau_{l,m}+8^m-1$ and $d\in\llbracket m-1\rrbracket$ according to \eqref{eq:proba_of_replay};\\
\For{$t=\tau_{l,m},\dots,\tau_{l,m}+8^m-1$}{
\If{Enter replay at depth $d$}{
$\cD_t\gets\cD_t\cup\{d\}$\tcp*{Activate depth $d$ for replay}
}
\If{Exit replay at depth $d$}{
$\cD_t\gets\cD_t\setminus\{d\}$\tcp*{Deactivate depth $d$ after replay ends}
}
\textcolor{white}{x}Choose $x_t$ using \textbf{Sampling scheme} (\cref{alg:sampling_scheme});\\
Update mean estimates and evict bins using \eqref{eq:star};\\
\If{$\cBM=\emptyset$}{
\textbf{Break}; terminate episode $l\gets l+1$, restart from block $m\gets 1$ \tcp*{Shift detected}}
}
Change block: $m\gets m+1$, $\tau_{l,m+1}\gets\tau_{l,m}+8^m$\tcp*{If $\cBM\neq\emptyset$}
\label{alg:routine}
\end{algorithm}
At each round $t \in \llbracket T \rrbracket$, \algo maintains a set of \emph{active depths} $\cD_t \subset \mathbb{N}$. For each active depth $d\in\cD_t$, the algorithm maintains a set of \emph{active bins} $\cB_t(d) \subseteq \cT_d$, which are those not yet evicted based on their observed performance. If $d$ is not active, \emph{i.e.} $d \notin \cD_t$, then $\cB_t(d) = \emptyset$.
\textbf{Replays.} To detect non-stationarity at different scales, the algorithm performs \emph{replays} at various depths. Each block $[\tau_{l,m}, \tau_{l,m+1}[$ always initiates a replay at depth $m$ by setting $\cBM \gets \mathcal{T}_m$. Additionally, replays may be triggered for all rounds $s=\tau_{l,m} + 1,\dots,\tau_{l,m}+8^m-1$ at coarser depths $d < m$ with probability
\begin{align}\label{eq:proba_of_replay}
p_{s, d} = \sqrt{\frac{8^d}{s-\tau_{l,m}}}\I{s - \tau_{l,m}\equiv 0[8^d]}\,.
\end{align}
Each replay at depth $d$ runs for $8^d$ rounds, unless it is interrupted by the end of the current block. At the beginning of such a replay, the set of \emph{active bins} at each active depth $d \in \cD_t$ is reset to contain \emph{all} bins at that depth: $\cB_t(d) \gets \cT_d$ for all $d \in \cD_t$.
%This particularly implies that active bins at the finest depth $m$ are also reset, \emph{i.e.} $\cB_t(m) \gets \cT_m$. However, the set $\cBM$ \emph{is not} reset and continues to reflect the outcome of eliminations from previous rounds within the block.
When the replay at depth $d$ concludes, depth $d$ is simply removed from the set of active depths $\cD_t$, but any other active depths may continue their replays independently. Thanks to this design, multiple replays at different depths can overlap in time, allowing the algorithm to monitor for potential significant shifts at multiple scales simultaneously (\cref{alg:routine}).
\textbf{Sampling scheme.}
At each round, the algorithm selects an action using a hierarchical, top-down sampling procedure from the \emph{minimum active depth} $\dmin = \min\cD_t$ down to the deepest active depth that contains an active bin (\cref{alg:sampling_scheme}). Remark that if $\cD_t = \{m\}$, then $\cB_t(m) = \cBM$. In this case, the sampling procedure reduces to selecting a bin uniformly from $\cBM$, and subsequently sampling an action from the chosen bin (see \cref{fig:algo_explanation}).
\begin{algorithm}
\caption{Sampling scheme}
\KwIn{Round $t$, active depths $\cD_t$, active bins $(\cB_t(d))_{d\in\cD_t}$.}
Compute $\dmin = \min\cD_t$ \tcp*{Identify the minimum active depth}
$B_{\mathrm{parent}}\sim\mathcal{U}(\cB_t(\dmin))$\tcp*{Sample a parent bin at depth $\dmin$}
\For{$d\in\mathrm{Sort}(\cD_t)\setminus\{\dmin\}$}{
\If{$\mathrm{Children}(B_{\mathrm{parent}}, d)\cap\cB_t(d)=\emptyset$\tcp*{No active child bin found}}{
$x_t\sim\mathcal{U}(B_{\mathrm{parent}})$\tcp*{Sample uniformly from current bin}
\textbf{Return} $x_t$;
}
\Else{
$B_{\mathrm{child}}\sim\mathcal{U}(\mathrm{Children}(B_{\mathrm{parent}}, d)\cap \cB_t(d))$\tcp*{Sample an active child bin}
$B_{\mathrm{parent}}\gets B_{\mathrm{child}}$\tcp*{Continue to the selected node}
}
}
\label{alg:sampling_scheme}
\end{algorithm}
\textbf{Bin eviction criterion.} Our algorithm aims to identify and eliminate arms that incur significant regret, as defined in \cref{def:significant_regret_arm}. However, since the algorithm only relies on bin estimates $\delta_t(B)$, we introduce a notion of \emph{significant regret for bins} to make localized decisions.
\begin{definition}[\textbf{Significant regret for a bin}]\label{def:significant_regret_bin}
We say a bin $B\in\cT_d$ incurs significant regret on interval $[s_1, s_2]$ if
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(B) \geq 3\log(T)\sqrt{(s_2 - s_1)2^d}\,.
\end{align*}
We call such bin an \textbf{unsafe bin} on this interval, and otherwise we call it \textbf{safe} on this interval.
\end{definition}
The right-hand side of the inequality corresponds to the rate of the minimax regret of a $2^d$-armed bandit over interval $[s_1, s_2]$. Intuitively, this threshold captures whether the cumulative sub-optimality of the bin is statistically significant over $[s_1, s_2]$. Importantly, we will later show that if a bin meets this threshold, then all arms within the bin also suffer significant regret in the sense of \cref{def:significant_regret_arm} (see \cref{prop:from_bin_to_arm}). We emphasize that the notion of significant shifts (\cref{def:significant_shift}) is defined \emph{independently of any depth} (and thus independently of the depth used by the algorithm).\\
To estimate the mean reward of a bin $B$, we use the following importance-weighted estimator:
\begin{align}\label{eq:IPS_mean_estimate}
\forall B\in\cT_d,\quad\hat\mu_t(B) = \frac{Y_t(x_t)}{\mathbb{P}(x_t \in B \condi \mathcal{F}_{t-1})}\I{x_t \in B},
\end{align}
where $\cF_{t}$ is the natural filtration generated by the algorithm. Under uniform sampling within bin $B$, $\hat\mu_t(B)$ is an unbiased estimate of $\mu_t(B)$. In particular, during a replay at depth $d$, the algorithm leverages these estimates to potentially evict bins across active depths $d' \geq d$ that appear suboptimal: over some $[s_1, s_2]$ of length at most $8^d$ ($d$ is active), bin $B$ is evicted if
\begin{align}\tag{\textcolor{red}{$\star$}}\label{eq:star}
\max_{B' \in \cB_{[s_1, s_2]}(d)} \sum_{t=s_1}^{s_2} \hat{\delta}_t(B', B)
> c_0 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d} + \frac{4(s_2 - s_1)}{2^d},
\end{align}
where $c_0$ is a positive universal constant\footnote{An exact value of $c_0$ can be derived from the analysis.}, and $\cB_{[s_1, s_2]}(d)$ denotes the set of bins that remain active throughout interval $[s_1, s_2]$. Importantly, when a bin $B \in \cT_d$ is evicted based on this criterion, it is immediately removed from the current set of active bins $\cB_t$, and also permanently \textbf{removed from the global MASTER set} $\cBM$. In addition, all children of $B$ in the dyadic tree are evicted as well. As a result, no future arms will be sampled from the entire region corresponding to $B$ and its descendants in $\cT$ until a new replay is scheduled.
The next proposition establishes that the eviction rule \eqref{eq:star} is sound: evicted bins indeed correspond to areas where all arms suffer significant regret.
\begin{proposition}[\textbf{Significant regret of a bin implies significant regret of an action}]\label{prop:from_bin_to_arm}
If a bin $B \in \mathcal{T}_d$ incurs significant regret on an interval $[s_1, s_2]$ with $s_2 - s_1 \leq 8^d$, then every point $x \in B$ also incurs significant regret over $[s_1, s_2]$.
\end{proposition}
\begin{figure}[h]
\centering
\includegraphics[width=0.325\textwidth]{Figures/t1.png}
\includegraphics[width=0.325\textwidth]{Figures/t2.png}
\includegraphics[width=0.325\textwidth]{Figures/t3.png}
\caption{Example of sampling with $m=4$; active bins are in blue. \textit{Left}: At time $t_1$, depths $3$ and $m$ are active. A sample path may select bin $B_{3,3}$ uniformly at random (\emph{u.a.r.}) at depth $3$, then $B_{m,6}$ \emph{u.a.r.} among its active children, then arm $x_t$ \emph{u.a.r.} in $B_{m,6}$ (red path).
\textit{Center}: At $t_1 + 1$, a replay starts at depth $1$. A path may go through $B_{1,1} \rightarrow B_{3,2} \rightarrow B_{m,4}$, selecting $x_t$ in $B_{m,4}$ (red path). Alternatively, $B_{1,2}$ could be chosen; with no active children, $x_t$ is sampled directly from it (green choice).
\textit{Right}: At $t_1 + 9$, depth $1$ exits replay. Bin has been $B_{3,2}$ eliminated during the replay, and a path may select $B_{3,1} \rightarrow B_{m,1}$, then $x_t$ in $B_{m,1}$ (red path).}
\label{fig:algo_explanation}
\end{figure}
\section{Main results}\label{sec:main_results}
We establish the fundamental limits for the non-stationary Lipschitz bandit problem, and prove that our algorithm \algo achieves minimax optimal dynamic regret. We first derive a lower bound on the dynamic regret, expressed in terms of both the number of significant shifts $\tilde{L}_T$ and the total variation $V_T$. To the best of our knowledge, this is the first lower bound for non-stationary Lipschitz bandits.
\begin{theorem}[\textbf{Lower bound on the dynamic regret}]\label{th:lower_bound}
Let $\mathcal{E}_{\mathrm{Lip}}(T, \tilde{L})$ denote the class of non-stationary Lipschitz bandit environments over the arm space $[0,1]$ with at most $\tilde{L}_T\leq \tilde{L}$ significant shifts. There exists a numerical constant $\ubar{c}_1>0$ such that for any algorithm $\pi$,
\begin{align*}
\sup_{\bm{\mu}\in\mathcal{E}_{\mathrm{Lip}}(T,\tilde{L})}\E{\bm{\mu}}{R(\pi,T)}\geq \ubar{c}_1 \Tilde{L}^{1/3}T^{2/3} \,.
\end{align*}
% Let $\mathcal{E}_{\mathrm{Lip}}(T, V)$ denote the class of non-stationary Lipschitz bandit environments over the arm space $[0,1]$ with at most $V_T\leq V$ total variation. There exists a numerical constant $\ubar{c}_2>0$ such that for any algorithm $\pi$,
Similarly, for the class $\mathcal{E}_{\mathrm{Lip}}(T, V)$ with at most $V_T\leq V$ total variation, there exists a numerical constant $\ubar{c}_2>0$ such that for any algorithm $\pi$,
\begin{align*}
\sup_{\bm{\mu}\in\mathcal{E}_{\mathrm{Lip}}(T, V)}\E{\bm{\mu}}{R(\pi,T)}\geq \ubar{c}_2\left(T^{2/3}+ V^{1/4}T^{3/4}\right) \,.
\end{align*}
\end{theorem}
To give intuition whether the optimal rate $\mathcal{O}(\tilde{L}^{1/3} T^{2/3})$ is achievable, consider an \emph{oracle algorithm} $\pi_{\rm{oracle}}$ with access to the exact times of the significant shifts $\{\tau_i\}_{i=0}^{\tilde{L}_T}$. The optimal algorithm partitions the horizon into $\tilde{L}_T$ phases and, within each phase, runs a minimax-optimal bandit algorithm (\emph{e.g.} \texttt{UCB} \citep{auer2002finite}) using a discretization of $(\tau_{i+1} - \tau_i)^{1/3}$ arms.
This oracle algorithm suffers regret at most $c_{\rm{oracle}}(\tau_{i+1} - \tau_i)^{2/3}$ per phase, where $c_{\rm{oracle}} > 0$ is a numerical constant, yielding
\begin{align*}
\E{}{R(\pi_{\rm{oracle}}, T)} \leq c_{\rm{oracle}} \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3} \leq c_{\rm{oracle}} \tilde{L}_T^{1/3} T^{2/3} \,.
\end{align*}
However, this oracle relies on complete knowledge of $\tau_i$'s. In contrast, our algorithm \algo is \emph{fully adaptive}. \cref{th:upper_bound} proves that it achieves minimax optimal regret, up to poly-logarithmic factors.
\begin{theorem}[\textbf{Adaptive upper bound on dynamic regret}]\label{th:upper_bound}
Let $\bm{\mu}\in\mathcal{E}_{\mathrm{Lip}}(T, \tilde{L}_T)$ be a Lipschitz bandit environment with $\tilde{L}_T$ unknown significant shifts $\{\tau_i(\bm{\mu})\}_{i=0}^{\tilde{L}_T}$. There exists a numerical constant $\bar{c}_1>0$ such that the expected dynamic regret of \algo satisfies
\begin{align*}
\E{\bm{\mu}}{R(\pi_{\algo}, T)}\leq \bar{c}_1\log^2(T)\sum_{i=1}^{\tilde{L}_T}\left(\tau_i(\bm{\mu})-\tau_{i-1}(\bm{\mu})\right)^{2/3}\,.
\end{align*}
In particular, this yields the worst-case upper bound
\begin{align*}
\E{\bm{\mu}}{R(\pi_{\algo}, T)}\leq \tilde{O}(\tilde{L}_T^{1/3}T^{2/3})\,.
\end{align*}
Therefore, \algo is minimax optimal for the number significant shifts $\tilde{L}_T$ up to poly-log factors.
\end{theorem}
Since $\tilde{L}_T \leq S_T \leq L_T$, \cref{th:upper_bound} achieves minimax optimal rates of $\tilde{\mathcal{O}}(S_T^{1/3} T^{2/3})$ and $\tilde{\mathcal{O}}(L_T^{1/3} T^{2/3})$, without requiring prior knowledge of $S_T$ or $L_T$, respectively. The bound provided by \cref{th:upper_bound} can be significantly sharper when $\tilde{L}_T \ll S_T$. The following result further shows that \cref{th:upper_bound} also recovers the optimal rate in terms of the total variation $V_T$. The proof is deferred to \cref{app:sec:proof_th_upper_bound}.
\begin{remark}[Beyond $1$-Lipschitz bandits]
Minimax optimality of \cref{th:upper_bound} extends beyond Lipschitz bandits with Lipschitz constant $1$.
In particular, for $(\kappa,\beta)$-Hölder bandits we can show that the minimax optimal rate (up to polylogarithmic factors) can be adapted as
$\E{}{R(\pi_{\texttt{MBDE}}, T)}\leq \widetilde{\mathcal{O}}\Big(T^{\frac{1+\beta}{1+2\beta}}\Tilde{L}_T^{\frac{\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}}\Big)$, where the minimax optimality is with respect to $T$, $\tilde{L}_T$ and Hölder constants $\kappa$ and $\beta)$ jointly.
The necessary modifications to our setting and algorithm are detailed in \cref{app:sec:extension_arbitrary_spaces}.
\end{remark}
\begin{corollary}[\textbf{Regret bound in terms of total variation }$V_T$]\label{cor:upper_bound_total_variation}
Let $\bm{\mu}\in\mathcal{E}_{\mathrm{Lip}}(T, V_T)$ be any Lipschitz bandit environment with total variation $V_T$. There exists a numerical constant $\bar{c}_2>0$ such that the expected dynamic regret of \algo satisfies:
\begin{align*}
\E{\bm{\mu}}{R(\pi_{\algo}, T)}\leq \bar{c}_2\log^2(T)\left( T^{2/3} + V_T^{1/4}T^{3/4}\right)\,.
\end{align*}
Therefore, \algo is minimax optimal for the total variation $V_T$ up to poly-log factors.
\end{corollary}
\section{Proof intuition of \texorpdfstring{\cref{th:upper_bound}}{Lg}}\label{sec:proof_sketch}
%We now outline the key ideas behind the proof of \cref{th:upper_bound}.
A natural question is whether one could directly apply the \texttt{MASTER} procedure of \citet{wei2021non} using a discretization of the arm space as the base algorithm. However, such a direct application would not be possible in our setting, for the following reasons. The \texttt{MASTER} procedure is specifically designed to detect and adapt to changes in the \emph{mean reward functions} $\mu_t$, whereas our goal is to handle \emph{significant shifts}, defined in terms of \emph{cumulative reward gaps} $\sum_t \delta_t$. This distinction is crucial: the quantity $\sum_t \delta_t$ is inherently more stable than the instantaneous mean rewards, as it captures \emph{suboptimality} over time, even when the underlying reward functions change smoothly or gradually. As a result, the detection problem in our setting is more subtle and requires finer control over the accumulation of regret. Therefore, even when discretizing the arm space with replays at different depths, the analysis of \citet{wei2021non} would not achieve the desired dependence on $\tilde{L}_T$.
We further emphasize that our analysis goes beyond a straightforward adaptation of the techniques developed for the $K$-armed setting by \citet{suk2022tracking}. In their setting, the algorithm maintains a set of \emph{safe arms} and applies a variant of \emph{Successive Elimination} \citep{even2006action} by uniformly sampling from the current set and progressively eliminating arms identified as suboptimal. Roughly speaking, to detect changes of magnitude $\Delta$, they schedule replays of duration $d = K/\Delta^2$, during which all $K$ arms are re-sampled and their estimates updated. However, such a replay scheme is infeasible in our infinite-arm setting. If we were to discretize the arm space into $2^m$ bins at each replay, reliably detecting a shift of magnitude $\Delta$ would require roughly $2^m/\Delta^2$ rounds, resulting in a prohibitive regret cost. To circumvent this issue, our algorithm schedules replays at different discretization scales. By leveraging the hierarchical structure of the dyadic tree $\cT$, we update estimates of the gaps at multiple depths \emph{simultaneously}. We exploit the key observation that each sample $x_t$ provides information not only for its bin at depth $d$, but also for all its child bins at depths $d' \geq d$. Through systematic aggregation of information across these nested bins, our multi-scale estimation strategy allows the algorithm to detect significant shifts efficiently at different resolutions, while avoiding the prohibitive sample complexity associated with naive discretization.
We begin by highlighting the technical challenges that arise when estimating these cumulative gaps, in particular how they differ from estimating changes in the mean reward itself (\cref{subsec:proof_sketch_estimating_gaps}). We then present the high-level ideas for bounding the dynamic regret in two scenarios: (\emph{i}) when the process is in a \emph{stable block}, and no significant shift is detected, we show that the replays do not increases the regret excessively (\cref{subsec:proof_sketch_stable_block}); and (\emph{ii}) when a \emph{significant shift} does occur, we show that the algorithm responds appropriately (\cref{subsec:proof_sketch_unstable_block}). The proofs are provided in \cref{app:sec:proof_th_upper_bound}.
\subsection{Estimating the reward gaps}\label{subsec:proof_sketch_estimating_gaps}
The continuous arm space setting presents key technical challenges. These difficulties are specific to our framework and are not addressed in existing work on non-stationary bandits, and we believe these challenges are of independent interest and may be relevant beyond our specific setting.\\
The main challenge in this setting lies in estimating a continuous, non-stationary mean reward function without prior knowledge of the magnitude of distributional shifts, or, equivalently, the discretization level and time scale at which such shifts become detectable. Coarse discretization is essential to rapidly detect \emph{large} shifts, while finer discretization is required to capture \emph{smaller yet statistically significant changes}. This motivates the multi-scale replay framework of \texttt{MDBE} in which each scale contributes adaptively based on the magnitude of the underlying shift.
\textbf{Bias in mean reward estimation.} Our algorithm actively prunes bins at different depths, meaning that for a given bin $B \in \cT_d$, its sub-bins $B' \subset B$ at deeper depths $d' > d$ may be evicted. As a result, the empirical mean reward $\hat\mu_t(B)$ can be biased, since arms $x_t$ are only drawn from non-evicted sub-bins. Leveraging \cref{assumption:lipschitz}, we control this bias, which decays as
\begin{align}\label{eq:main_bias_control}
\forall t \in \llbracket T\rrbracket,\,\forall d\in\cD_t,\ \forall B, B'\in\cB_t(d),\quad \left|\E{t-1}{\hat\delta_t(B',B)} - \delta_t(B', B)\right| \leq \frac{4}{2^d}.
\end{align}
Thus, the bias in estimating the gap $\sum_{t=s_1}^{s_2} \hat\delta_t(B', B)$ of bins at depth $d$ scales as $(s_2 - s_1)/2^d$.
\textbf{Concentration of mean reward estimates.} Due to the martingale property of $(\hat\delta_t(B', B))_t$, we can tightly control its deviation from its conditional expectation. Recall that our \emph{hierarchical sampling scheme} (\cref{alg:sampling_scheme}) ensures that the algorithm selects an active bin at the minimum active depth $\dmin$ and recursively samples from child bins at finer active depths. This choice guarantees that
\begin{align*}
\forall t \in \llbracket T\rrbracket,\, \forall d \in \cD_t, \,\forall B \in \cB_t(d), \quad \mathbb{P}(x_t \in B' \condi \cF_{t-1}) \geq \frac{1}{2^d}.
\end{align*}
This ensures that all active bins at any depth $d$ are sampled with at least uniform probability, allowing us to control the variance of the inverse weighted estimates at all scales and to derive the following high-probability bound.
\begin{proposition}[\textbf{Concentration event}]\label{prop:concentration}
Let $\mathcal{E}_1$ be the following event: for all intervals $[s_1,s_2]$, depths $d \in \bigcap_{t=s_1}^{s_2} \cD_t$, and bins $B, B' \in \cB_{[s_1,s_2]}(d)$, we have
\begin{align}\label{main:concentration_control}
\left| \sum_{t=s_1}^{s_2}\hat\delta_t(B', B) - \sum_{t=s_1}^{s_2}\E{t-1}{\hat\delta_t(B', B)} \right|\leq c_1\log(T)\sqrt{(s_2 - s_1)2^{d}\vee4^d} \,,
\end{align}
where $c_1$ is a positive numerical constant. Then $\mathcal{E}_1$ holds with probability at least $1-1/T^3$.
\end{proposition}
Unlike in classical finite-armed bandits \citep{suk2022tracking, suk2023tracking, suk2023can}, uniform sampling only holds strictly at depth $\dmin$. This subtle distinction plays a critical role in the analysis. Thanks to both the bias control \eqref{eq:main_bias_control} and the concentration control \eqref{main:concentration_control}, we can ensure that bins are only evicted when truly unsafe, despite the complexity introduced by the hierarchical sampling.
\subsection{Regret within a block without significant shift}\label{subsec:proof_sketch_stable_block}
Assume, for the sake of clarity, that no significant shift occurs during the block $[\tau_{l,m},\tau_{l,m+1}[$. When only the replay at depth $m$ is active, \emph{i.e.} $\cD_t = \{m\}$, the algorithm effectively runs a \emph{Successive Elimination} strategy at depth $m$: at each round, it selects a bin uniformly at random from the set of safe bins $\cBM$, samples an arm $x_t$ from this bin, and eliminates bins that are deemed \emph{unsafe} at this depth. In this regime, since the discretization level at depth $m$ is tuned to the block length $\tau_{l,m+1} - \tau_{l,m}$, classical results indicate that the regret incurred within the block is of order $(8^m)^{2/3} = 4^m$ (up to logarithmic factors).
It remains to control the regret from replays across different scales. A key difficulty is that such replays may overlap, making it delicate to control the cumulative regret contribution at each depth simultaneously. A crucial property leveraged in the analysis is that our importance-weighted estimator \eqref{eq:IPS_mean_estimate} allows each sampled arm $x_t$ to update the mean estimates $\hat\mu_t(B)$ for \emph{all} bins $B$ containing $x_t$. This multi-scale update property ensures that replays at any given depth $d$ contribute information \emph{not only at that level} but also propagate to finer depths $d'>d$. In particular, we leverage \cref{prop:from_bin_to_arm}, which guarantees that if a bin is identified as unsafe at \emph{any} depth $d$, then \emph{all} arms it contains are suboptimal.
Our proof strategy is to quantify the contribution of each replay at depth $d$ \emph{independently}. Since each such replay lasts at most $8^d$ rounds, and arms are always selected from the set of safe bins $\cB_t(d)$ at that depth (otherwise they would have been evicted under the concentration event $\mathcal{E}_1$), the regret contribution of each replay can be upper bounded by $(8^d)^{2/3} = 4^d$. Thanks to the well-calibrated replay scheduling probability \eqref{eq:proba_of_replay}, there are roughly $\sqrt{8^{m-d}}$ replays at depth $d$ that are scheduled during a block, so overall we can show that the aggregate regret from all replays across depths $d<m$ remains bounded by $4^m$, thus matching the regret incurred by the main replay at depth $m$.
\subsection{Detecting significant shifts}\label{subsec:proof_sketch_unstable_block}
When a significant shift $\tau_i$ occurs within a block $[\tau_{l,m},\tau_{l,m+1}[$, the primary challenge is that the shift is initially \emph{undetected}, potentially causing the algorithm to suffer large regret if it continues exploiting outdated information. In particular, such regret can accumulate if a bin that was previously evicted (due to being deemed unsafe) becomes optimal after the shift, while the algorithm keeps ignoring it. More precisely, detecting a shift of magnitude $\Delta$ at depth $d$ requires a replay of length approximately $2^d/\Delta^2$. Undetected shifts of this magnitude are tolerable at this resolution over a period of duration $D$ as long as $\Delta D \leq 4^d$. By carefully calibrating the probability of scheduling replays at each depth (see \eqref{eq:proba_of_replay}), we guarantee that \emph{replays at depth $d$ occur frequently enough} to ensure, with high probability, the timely detection of shifts of size $\Delta$ before their contribution to regret becomes significant.
\section{Conclusion}\label{sec:conclusion}
We studied the previously unexplored problem of non-stationary Lipschitz bandits. We introduced \algo, a novel algorithm that leverages a hierarchical discretization of the action space to exploit the Lipschitz structure of the evolving reward function. We established a lower bound of $\mathcal{O}(\tilde{L}_T^{1/3}T^{2/3})$ on the expected dynamic regret and showed that \algo achieves this rate \emph{adaptively}. \\
While our work is mainly theoretical, we also analyze the computational worst-case complexity of \algo in \cref{app:sec:computational_complexity} and show its empirical performance on a synthetic example in \cref{app:sec:simulations}. Developing computationally efficient algorithms with strong empirical performance that do not require prior knowledge of the non-stationarity \citep{gerogiannis2025dal,gerogiannis2410prior} remains an open problem. Future work could extend the significant shift detection framework to other structured bandit models such as convex or linear bandits.
\section*{Broader impacts}
This work is mainly theoretical and contributes to the analysis of algorithms for sequential-decision making under uncertainty. Our generic setting and algorithms have broad potential use; practitioners will therefore need to
specifically address possible social impacts with respect to the relevant application.
\subsection*{Acknowledgements}
The authors gratefully acknowledge Joe Suk for his insightful discussions and for sharing his code, which served as inspiration for our numerical experiments.\\
N.Nguyen and C.Vernade are funded by the Deutsche Forschungsgemeinschaft (DFG) under both the project 468806714
of the Emmy Noether Programme and under Germany’s Excellence Strategy–EXC number 2064/1–Project number
390727645. Both also thank the international Max Planck Research School for Intelligent Systems (IMPRS-IS).
\section*{NeurIPS Paper Checklist}
\begin{enumerate}
\item {\bf Claims}
\item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our algorithm is provided in \cref{sec:algorithm} and its corresponding bounds are provided in \cref{sec:main_results}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the abstract and introduction do not include the claims made in the paper.
\item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
\item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
\item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
\item {\bf Limitations}
\item[] Question: Does the paper discuss the limitations of the work performed by the authors?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We acknowledge in \cref{sec:conclusion} that our work is mainly theoretical and does not consider numerical performances in depth.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
\item The authors are encouraged to create a separate "Limitations" section in their paper.
\item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
\item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
\item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
\item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
\item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
\item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
\item {\bf Theory assumptions and proofs}
\item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Theoretical results are stated in \cref{sec:main_results}, and all complete proofs can be found in Appendix.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include theoretical results.
\item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
\item All assumptions should be clearly stated or referenced in the statement of any theorems.
\item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
\item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
\item Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
\item {\bf Experimental result reproducibility}
\item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The code is provided in a Github repository.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
\item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
\item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
\item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
\item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
\item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
\item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
\item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
\item {\bf Open access to data and code}
\item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The code is available at \url{https://github.com/nguyenicolas/NS_Lipschitz_Bandits}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that paper does not include experiments requiring code.
\item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
\item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
\item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
\item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
\item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
\item {\bf Experimental setting/details}
\item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The details are available in \cref{app:sec:simulations}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
\item The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
\item {\bf Experiment statistical significance}
\item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Error bars can be found in \cref{app:sec:simulations}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
\item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
\item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
\item The assumptions made should be given (e.g., Normally distributed errors).
\item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
\item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
\item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
\item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
\item {\bf Experiments compute resources}
\item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: No particular computer resource is needed to run the experiments.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
\item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
\item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
\item {\bf Code of ethics}
\item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We have reviewed the NeurIPS Code of Ethics and confirm that our paper adheres to its principles.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
\item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
\item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
\item {\bf Broader impacts}
\item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Societal impacts of our work are discussed in the broader impact section.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that there is no societal impact of the work performed.
\item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
\item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
\item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
\item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
\item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
\item {\bf Safeguards}
\item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our paper poses no such risks.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper poses no such risks.
\item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
\item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
\item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
\item {\bf Licenses for existing assets}
\item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our paper does not use existing assets.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not use existing assets.
\item The authors should cite the original paper that produced the code package or dataset.
\item The authors should state which version of the asset is used and, if possible, include a URL.
\item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
\item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
\item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
\item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
\item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
\item {\bf New assets}
\item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our paper does not release new assets.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not release new assets.
\item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
\item The paper should discuss whether and how consent was obtained from people whose asset is used.
\item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
\item {\bf Crowdsourcing and research with human subjects}
\item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our paper does not involve crowdsourcing nor research with human subjects.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
\item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
\item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
\item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our paper does not involve crowdsourcing nor research with human subjects.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
\item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
\item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
\item {\bf Declaration of LLM usage}
\item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
%this research?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We only use LLM for writing purposes.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
\item Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
\end{itemize}
\end{enumerate}
\section*{Appendix}
\begin{comment}
The Supplementary is organized as follows.
\begin{itemize}
\item \cref{app:sec:notations} provides a summary of the notations used throughout the paper and the proofs.
\item \cref{app:sec:pseudo_code} presents the complete pseudocode of our algorithm, \algo.
\item \cref{app:sec:proof_of_prop_bin_to_arm} contains the proof of \cref{prop:from_bin_to_arm}.
\item \cref{app:sec:proof_th_upper_bound} establishes the proof of our main upper bound result stated in \cref{th:upper_bound}.
\item \cref{app:sec:proof_corollary_V_T} derives the proof of the regret bound in terms of total variation $V_T$ (\cref{cor:upper_bound_total_variation}).
\item \cref{app:sec:proof_lower_bound} provides the proofs of the lower bound results stated in \cref{th:lower_bound}.
\item \cref{app:sec:extension_arbitrary_spaces} discusses extensions of our results to multi-dimensions and to Hölder bandits.
\item \cref{app:sec:computational_complexity} discusses the computational complexity of our algorithm.
\item \cref{app:sec:simulations} presents a numerical simulation on synthetic data.
\end{itemize}
\end{comment}
\section{Notations}\label{app:sec:notations}
We present in \cref{tab:notations} a compilation of used notations in our paper.
\begin{table}[h!]
\centering
\begin{tabular}{@{}ll@{}}
\toprule
\textbf{Notation} & \textbf{Description} \\ \midrule
$T$ & Time horizon. \\
$L_T$ & Number of arm changes in $T$ rounds: $L_T = \sum_{t=1}^{T-1}\I{\exists x \in [0,1],\, \mu_{t+1}(x) \neq \mu_t(x)}$. \\
$S_T$ & Number of best-arm changes in $T$ rounds: $S_T = \sum_{t=1}^{T-1}\I{\argmax_{x} \mu_{t+1}(x) \neq \argmax_{x} \mu_t(x)}$. \\
$V_T$ & Total variation in $T$ rounds: $V_T = \sum_{t=1}^{T-1} \max_{x \in [0,1]} |\mu_{t+1}(x) - \mu_t(x)|$. \\
$\tilde{L}_T$ & Number of significant shifts in $T$ rounds (see \cref{def:significant_shift}). \\
$\cT$ & Dyadic tree, \emph{i.e.} hierarchical partition of $[0,1]$ (see \cref{def:dyadic_tree}). \\
$\cT_d$ & Partition of $[0,1]$ into $2^d$ bins of size $1/2^d$ each. \\
$\Par(B, d)$ & Unique parent of bin $B$ in $\cT_d$. \\
$\mathrm{Children}(B, d)$ & Children (set of bins) of bin $B$ at depth $d$. \\
$[t_l, t_{l+1}[$ & Episode $l$. An episode terminates when a significant shift is detected. \\
$[\tau_{l,m}, \tau_{l,m+1}[$ & Block $m$ within episode $l$; if uninterrupted, its duration is $\tau_{l,m+1}-\tau_{l,m}= 8^m$. \\
$M_l$ & Maximum number of blocks during one episode: $M_l = \lceil \log_8(7(t_{l+1} - t_l) + 1) - 1 \rceil$.\\
$\cD_t$ & Set of active depths at round $t$. \\
$\dmin$ & Minimum active depth at round $t$: $\dmin = \min \cD_t$. \\
$\cB_t(d)$ & Set of active bins at depth $d$ at time $t$. \\
$\cB_{[s_1, s_2]}(d)$ & Bins at depth $d$ active at all time in $[s_1, s_2]$. \\
$\cBM$ & MASTER set of block $m$: $\cBM \subseteq \cT_m$. \\
$R_{t, d}$ & Replay trigger variable: $R_{t,d}=1$ indicates a replay at depth $d$ starts at time $t$. \\
$\textsc{Replay}(t, d)$ & Replay initiated at round $t$ at depth $d$ (see \cref{def:replay_s_d}).\\
$\{B_d^{(1)},\dots,B_d^{(2^d)}\}$ & Ordered bins in $\cT_d$ (see \cref{def:bin_ordering}). \\
$\mathcal{E}_1$ & Concentration event (see \cref{prop:concentration}). \\
$\mathcal{E}_2(\tau_{l,m})$ & Event defined in \cref{prop:a_replay_occurs}. \\
$x_t$ & Arm selected at round $t$ by the algorithm. \\
$x_i^\sharp$ & Last safe arm in phase $i$ (see \cref{def:x_sharp}). \\
$x_t^\sharp$ & $x_t^\sharp = x_i^\sharp$ for $t\in[\tau_{i},\tau_{i+1}[$. \\
$\delta_t(x)$ & Instantaneous regret at round $t$ of arm $x$: $\delta_t(x) = \max_{x\in[0, 1]}\mu_t(x)-\mu_t(x)$. \\
$B_{t,d}$ & Bin selected at depth $d$: $x_t \in B_{t,d}$. \\
$B^\sharp_{i, d}$ & Bin at depth $d$ containing $x_i^\sharp$ (see \cref{def:Bsharp}). \\
$B^{\sharp}_{t, d}$ & $B_{t,d}^\sharp = B_{i,d}^\sharp$ for $t\in[\tau_i,\tau_{i+1}[$. \\
$\Blast_{l,m}$ & Last bin of block $[\tau_{l,m}, \tau_{l,m+1}[$ in $\cBM$ (see \cref{def:Blast}). \\
$\delta_t(B)$ & Instantaneous regret at round $t$ of bin $B$: if $B\in\cT_d$, $\delta_t(B) = \max_{B'\in\cT_d}\mu_t(B)-\mu_t(B)$. \\
$[s_{i,j}^m(B), s_{i,j+1}^m(B)]$ & $j^\text{th}$ bad segment of bin $B$ (see \cref{def:bad_segment}). \\
$\tilde{s}_{i,j}^m(B)$ & Approximate midpoint of bad segment $[s_{i,j}^m(B), s_{i,j+1}^m(B)]$ (see \cref{def:bad_segment}). \\
$s_{l,m}(B)$ & Bad round of bin $B$ (see \cref{def:bad_round}). \\
$M(s, d, B)$ & Last round where bin $B$ is active during $\textsc{Replay}(s, d)$ (see \cref{def:eviction_time}). \\
$\KL(\mathbb{P}, \mathbb{Q})$ & Kullback-Leibler divergence between two distributions $\mathbb{P}$ and $\mathbb{Q}$.\\
$kl(p, q)$ & Kullback-Leibler divergence between two Bernoulli with parameters $p$ and $q$.\\
\bottomrule
\end{tabular}
\caption{Summary of notations.}
\label{tab:notations}
\end{table}
\section{Full pseudo-code of \algo}\label{app:sec:pseudo_code}
\vspace{-1cm}
\begin{algorithm}[H]
\caption*{\textbf{Algorithm: }\algo: Multi-Depth Bin Elimination}
\textbf{Input.} horizon $T$, Lipschitz constant. \\
\stepcounter{linenumber} \textbf{Init.} $l\gets 1$, $t\gets 1$\tcp*{Global initialization}
\While{$t\leq T$}{
$\textcolor{blue}{(\bigstar)}$ $m\gets 1$, $\tau_{l,m}\gets t$\tcp*{Init first block of episode $l$}
\If(\tcp*[f]{Check if current block has ended}){$t=\tau_{l,m}+8^m$}{
$m\gets m + 1$ \tcp*{Change block}
$\cBM\gets\cT_m$\tcp*{Reset MASTER set to full bin set at depth $m$}
$\cD_t\gets\{m\}$\tcp*{Activate only the current depth $m$}
$\tau_{l,m}\gets t$\tcp*{Update block start time}
$\mathrm{StoreActive}[m][t_{\mathrm{start}}]\gets\tau_{l,m}$\tcp*{Record start time of this replay}
\For(\tcp*[f]{Schedule replays in this block}){$s=\tau_{l,m}+1,\dots,\tau_{l,m}+8^m$}{
\For{$d=0,\dots,m-1$}{
\If{$s-\tau_{l,m}\equiv0[8^d]$}{Sample $R_{s,d}\sim\mathrm{Ber}\big(\sqrt{8^d/(s-\tau_{l,m})}\big)$;}
\Else{$R_{s,d}=0$\tcp*{No replay triggered at depth $d$ in round $s$}}
}
}
\For{$d=0,\dots,m-1$}{
$\mathrm{StoreActive}[d][t_{\mathrm{start}}]\gets\emptyset,\mathrm{StoreActive}[d][t_{\mathrm{end}}]\gets\emptyset\,$\tcp*{Initialize variable}
$\cB_t(d)\gets\emptyset$ \tcp*{Initialize set of active bins}
}
}
\For(\tcp*[f]{Check which replays are triggered in this round}){$d=0,\dots,m-1$}{
\If(\tcp*[f]{If a replay at depth $d$ is triggered}){$R_{s,d}=1$}{
$\cD_t\gets \cD_t\cup\{d\}$\tcp*{Add this depth to current active depths}
$\mathrm{StoreActive}[d][t_\mathrm{start}]\gets t$\tcp*{Store replay start time}
$\mathrm{StoreActive}[d][t_\mathrm{end}]\gets t+8^d$\tcp*{Store replay end time}
$\cB_t(d)\gets \cT_d$\tcp*{Activate all bins at this new active depth}
}
}
\For(\tcp*[f]{Check if any active replays are ending}){$d\in\cD_t$}{
\If{$\mathrm{StoreActive}[d][t_\mathrm{end}]=t$}{
$\cD_t\gets\cD_t\setminus\{d\}$\tcp*{Deactivate this depth}
$\cB_t(d)\gets\emptyset$\tcp*{Deactivate its bins}
}
}
$\dmin\gets\min\cD_t$\tcp*{Identify the minimum active depth}
$B_{\rm{parent}}\sim\mathcal{U}(\cB_t(\dmin))$\tcp*{Sample a bin from active bins at depth $\dmin$}
\For(\tcp*[f]{Go trough the active depths, from shallowest to deepest}){$d\in\mathrm{Sort}(\cD_t\setminus\{\dmin\})$}{
\If(\tcp*[f]{No active children at this depth}){$\mathrm{Children}(B_{\mathrm{parent}},d)\cap \cB_t(d)=\emptyset$}{
$x_t\sim\mathcal{U}(B_{\mathrm{parent}})$\tcp*{Sample arm uniformly from current bin}
}
\Else{
$B_{\mathrm{child}}\sim\mathcal{U}(\mathrm{Children}(B_{\mathrm{parent}},d)\cap \cB_t(d))$ \tcp*{Sample active child bin}
$B_{\mathrm{parent}}\gets B_{\mathrm{child}}$\tcp*{Move to child bin}
}
}
\For{$d\in\cD_t$}{
\For{$B\in\cB_t(d)$}{
\If{$\exists [s_1,s_2]\subseteq [\mathrm{StoreActive[d][t_{\mathrm{start}}]},t]\text{ \rm{such that} }\eqref{eq:star}$ \text{\rm{holds} for some active bin }$B'\in\cB_t(d)$}{
\For{$d'\in [\![d,m]\!]\cap \cD_t$}{$\cB_t(d')\gets\cB_t(d')\setminus\bigcup_{B'\in\cT_{d'}:B'\subseteq B}\{B'\}$ \tcp*{Evict bins at active depths}}
}
}
}
$\cBM\gets\cBM\cap\cB_t(m)$\tcp*{Update MASTER set}
\If(\tcp*[f]{A significant shift is detected}){$\cBM = \emptyset$}{
$\tau_{l,m+1}\gets t+1$;\\
$l\gets l+1$ and Restart from $\textcolor{blue}{(\bigstar)}$\tcp*{Change episode}
}
$t\gets t+1$;}
\end{algorithm}
\section{Proof of \texorpdfstring{\cref{prop:from_bin_to_arm}}{Lg}}\label{app:sec:proof_of_prop_bin_to_arm}
Let $d$ be any fixed depth, and let $B \in \cT_d$ be a bin that incurs significant regret over an interval $[s_1, s_2]$. Let $x \in B$ be any action within this bin. For each round $t$, define $B_{t,d}^* = \argmax_{B' \in \cT_d} \mu_t(B')$ as the best bin at depth $d$ at round $t$, and let $x^*_{t,d} \in B_{t,d}^*$ be any action contained in it.
To relate the regret of bin $B$ with respect to $B_{t,d}^*$ to the regret of any action $x\in B$ with respect to $x^*_{t,d}$, we discretize the bin-level regret at depth $d$ as follows:
\begin{align*}
\delta_t(B^*_{t, d}, B) &= \frac{1}{|B^*_{t, d}|}\int_{u\in B^*_{t, d}}\mu_t(u)\dint u - \frac{1}{|B|}\int_{u\in B}\mu_t(u)\dint u\\
&= \frac{1}{|B^*_{t, d}|}\int_{u\in B^*_{t, d}}(\mu_t(u) - \mu_t(x^*_{t, d}))\dint u + \mu_t(x^*_{t, d}) \\
&- \frac{1}{|B|}\int_{u\in B}(\mu_t(u)-\mu_t(x))\dint u - \mu_t(x)\\
&\leq \frac{1}{|B^*_{t, d}|}\int_{u\in B^*_{t, d}}|\mu_t(u) - \mu_t(x^*_{t, d})|\dint u + \mu_t(x^*_{t, d}) \\
&+ \frac{1}{|B|}\int_{u\in B}|\mu_t(u)-\mu_t(x)|\dint u - \mu_t(x)\\
&\leq \frac{2}{2^d}+\delta_t(x^*_{t, d}, x)\quad \text{by \cref{assumption:lipschitz}}\,.
\end{align*}
Summing over the interval $[s_1, s_2]$, we obtain:
\begin{align}
\sum_{t=s_1}^{s_2}\delta_t(B) = \sum_{t=s_1}^{s_2}\delta_t(B_{t,d}^*, B) \leq \sum_{t=s_1}^{s_2}\delta_t(x^*_{t,d}, x) + \frac{2}{2^d}(s_2 - s_1). \label{eq:discretization_trick}
\end{align}
By assumption, bin $B$ incurs significant regret over $[s_1, s_2]$; that is, from \cref{def:significant_regret_bin},
\begin{align}
\sum_{t=s_1}^{s_2}\delta_t(B) \geq 3\log(T)\sqrt{(s_2 - s_1) 2^d}. \label{eq:temp_bin}
\end{align}
Substituting \eqref{eq:temp_bin} into \eqref{eq:discretization_trick}, we get a lower bound on the cumulative regret of action $x$ with respect to $x_{d,t}^*$:
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(x^*_{t,d}, x) &\geq 3 \log(T)\sqrt{(s_2 - s_1) 2^d} - \frac{2}{2^d}(s_2 - s_1) \\
&= (s_2 - s_1)^{2/3} \left(3\log(T)\sqrt{\frac{2^d}{(s_2 - s_1)^{1/3}}} - 2\frac{(s_2 - s_1)^{1/3}}{2^d} \right) \\
&\geq (s_2 - s_1)^{2/3} \left(3\log(T)\sqrt{\frac{2^d}{(s_2 - s_1)^{1/3}}} - 2\log(T)\frac{(s_2 - s_1)^{1/3}}{2^d} \right)\,,
\end{align*}
for horizon $T$ satisfying $\log(T)\geq 1$.
Now, since $s_2 - s_1 \leq 8^d$, it follows that
\[
\frac{2^d}{(s_2 - s_1)^{1/3}} > 1,
\]
which implies the above expression is at least:
\[
\log(T) (s_2 - s_1)^{2/3}.
\]
Therefore
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(B) \geq 3\log(T)\sqrt{(s_2 - s_1) 2^d} \implies \log(T)(s_2 - s_1)^{2/3} \leq \sum_{t=s_1}^{s_2}\delta_t(x^*_{t,d}, x) \leq \sum_{t=s_1}^{s_2}\delta_t(x),
\end{align*}
\emph{i.e.} action $x$ incurs significant regret on interval $[s_1, s_2]$, as per \cref{def:significant_regret_arm}.
\section{Proof of the upper bound of \algo (\texorpdfstring{\cref{th:upper_bound}}{Lg})}\label{app:sec:proof_th_upper_bound}
Before giving the proof, we introduce a few definitions and notations that will be useful. A full summary of notations used in the proof can be found in \cref{app:sec:notations}.
\begin{definition}[\textbf{Bin ordering}]\label{def:bin_ordering}
For each depth $d\in\Nat$, we partition the bins of $\cT_d$ by ordering them as $\cT_d =\{B_d^{(1)},\dots,B_d^{(2^d)}\}$.
\end{definition}
\begin{definition}[$\texttt{Replay}(s,d)$]\label{def:replay_s_d}
$\texttt{Replay}(s,d)$ denotes a replay at depth $d$ starting at round $s$.
\end{definition}
\begin{remark}[\textbf{Eviction time}]
A bin $B\in\cT_d$ is said to be evicted at round $t$ if it is evicted at the \emph{end} of round $t$, in the sense that $B\in\cB_t(d)$ and $B\notin\cB_{t+1}(d)$.
\end{remark}
\subsection{Bias of the mean estimates}\label{app:subsec:bias}
One challenge arising from our discretization scheme is that the importance-weighted estimate defined in \eqref{eq:IPS_mean_estimate} is inherently biased. This bias comes from the fact that our algorithm selectively samples arms $x_t$ only from active bins, while evicting bins at different depths over time.
\begin{proposition}[\textbf{Sampling bias}]\label{prop:sampling_bias}
For all round $t$, any active depth $d \in \cD_t$, and any active bin $B \in \cB_t(d)$ at this depth, the bias in the mean estimate for bin $B$ is bounded as
\begin{align*}
\left|\E{t-1}{\hat\mu_t(B)} - \mu_t(B)\right| \leq \frac{2}{2^d}\,.
\end{align*}
Moreover, the cumulative bias in the gap estimates between any pair of \textbf{active} bins at depth $d$ over an interval is bounded as:
\begin{small}
\begin{align*}
\forall [s_1, s_2],\,\forall d \in \bigcap_{t=s_1}^{s_2} \cD_t,\,\forall B, B' \in \cB_{[s_1, s_2]},\quad \left|\sum_{t=s_1}^{s_2} \E{t-1}{\hat\delta_t(B', B)} - \sum_{t=s_1}^{s_2} \delta_t(B', B)\right| \leq \frac{4(s_2 - s_1)}{2^d}\,.
\end{align*}
\end{small}
\end{proposition}
\begin{proof}[Proof of \cref{prop:sampling_bias}]
We begin with the first claim. Let $d \in \cD_t$ be an active depth, and let $B \in \cB_t(d)$ be an active bin at this depth. For all round $t$, by the definition of the estimator, we have for all $B \in \cT_d$,
\begin{align*}
\hat\mu_t(B) = \frac{Y_t(x_t)}{\mathbb{P}(x_t \in B \mid \cF_{t-1})} \I{x_t \in B} \,.
\end{align*}
Taking the conditional expectation with respect to $\cF_{t-1}$ yields:
\begin{align*}
\E{t-1}{\hat\mu_t(B)} = \int_{u \in B} \mu_t(u) f_{t-1}(u)\, \mathrm{d}u\,,
\end{align*}
where $f_{t-1}$ denotes the density of $x_t$ given $\cF_{t-1}$ and $x_t \in B$.
From \cref{assumption:lipschitz}, for all bin $B \in \cT_d$ and any $x \in B$, we have:
\begin{align*}
|\mu_t(B) - \mu_t(x)| = \left| \frac{1}{|B|} \int_{u \in B} \mu_t(u) - \mu_t(x) \, \mathrm{d}u \right| \leq \frac{1}{|B|} \int_{u \in B} |\mu_t(u) - \mu_t(x)| \, \mathrm{d}u \leq |B| = \frac{1}{2^d}\,.
\end{align*}
Using this, we can bound the estimation bias:
\begin{align*}
\left|\E{t-1}{\hat\mu_t(B)} - \mu_t(B)\right| &= \left|\int_{u \in B} \mu_t(u) f_{t-1}(u)\, \mathrm{d}u - \mu_t(B)\right|\\
&\leq \int_{u \in B} |\mu_t(u) - \mu_t(B)| f_{t-1}(u) \, \mathrm{d}u\\
&\leq \frac{1}{2^d} \int_{u \in B} f_{t-1}(u)\, \mathrm{d}u = \frac{1}{2^d}\,.
\end{align*}
For the second statement, let $B'$ also belong to $\cB_t(d)$. Applying the triangle inequality and the previous bound, we can control the bias in the gap estimate:
\begin{align*}
\left|\E{t-1}{\hat\delta_t(B', B)} - \delta_t(B', B)\right| &= \left| \E{t-1}{\mu_t(B')} - \mu_t(B') - \E{t-1}{\mu_t(B)} + \mu_t(B) \right|\\
&\leq \left|\E{t-1}{\mu_t(B')} - \mu_t(B')\right| + \left|\E{t-1}{\mu_t(B)} - \mu_t(B)\right|\\
&\leq 2 \frac{1}{2^d} + 2 \frac{1}{2^d} = \frac{4}{2^d}\,.
\end{align*}
Summing this inequality over the interval $[s_1, s_2]$ establishes the second claim.
\end{proof}
\subsection{Concentration of the mean estimates}
Our sampling scheme (\cref{alg:sampling_scheme}) is carefully constructed to ensure that the estimated gap between any two active bins at any active depth concentrates around its conditional expectation, especially by \emph{controlling its cumulative variance}. Our result relies on this following inequality.
\begin{lemma}[\citet{beygelzimer2011contextual}]\label{lemma:freedman}
Let $(X_t)_t$ be a real-valued martingale difference sequence adapted to the natural filtration $\cF_t = \sigma(X_1,\dots, X_t)$, such that $\E{}{X_t \condi \cF_{t-1}} = 0$. Suppose that $X_t \leq R$ almost surely and that $\sum_{i=1}^t \E{t-1}{X_i^2} \leq V_t$. Then for all $\delta \in (0,1)$, with probability at least $1 - \delta$,
\begin{align*}
\sum_{i=1}^t X_i \leq (e - 1)\left( \sqrt{V_t \log\left(\frac{1}{\delta}\right)} + R \log\left(\frac{1}{\delta} \right) \right).
\end{align*}
\end{lemma}
\begin{reproposition2}[\textbf{Concentration Event}]
Let $\mathcal{E}_1$ be the following event: for all intervals $[s_1,s_2]$, depths $d \in \bigcap_{t=s_1}^{s_2} \cD_t$, and bins $B, B' \in \cB_{[s_1,s_2]}(d)$, we have
\begin{align}
\left| \sum_{t=s_1}^{s_2}\hat\delta_t(B', B) - \sum_{t=s_1}^{s_2}\E{t-1}{\hat\delta_t(B', B)} \right|\leq c_1\log(T)\sqrt{(s_2 - s_1)2^{d}\vee4^d} \,,
\end{align}
where $c_1$ is a positive numerical constant. Then $\mathcal{E}_1$ holds with probability at least $1-1/T^3$.
\end{reproposition2}
\begin{proof}[Proof of \cref{prop:concentration}]
For all depth $d$ and pair of bins $B, B' \in \cT_d$, define the martingale difference sequence $(M_t)_t = \left( \hat\delta_t(B', B) - \E{t-1}{\hat\delta_t(B', B)} \right)_t$, adapted to the natural filtration $(\cF_t)_{t\geq 1} = \left(\sigma(x_t, Y_t(x_t))\right)_{t\geq 1}$.
\textbf{Lower Bound on Sampling Probability.}
Let $B_d \in \cB_t(d)$ be any active bin at depth $d$ at round $t$. We distinguish two cases:
\textbf{Case 1: $d = \dmin$.} By construction, arms are sampled uniformly at depth $\dmin$, implying
\begin{align*}
\mathbb{P}(x_t \in B_d \condi \cF_{t-1}) = \frac{1}{|\cB_t(\dmin)|} \geq \frac{1}{2^d}.
\end{align*}
\textbf{Case 2: $d \neq \dmin$.} The replay scheduling ensures that at round $t$, a replay at depth $d$ can only be triggered if $t - \tau_{l,m} \equiv 0 [8^d]$. This implies that no replay
at shallower depth scheduled before $t$ can still be active at round $t$. Thus, when a replay is scheduled at round $t$ at depth $d$, either $d=\dmin$, or all replays at depth $d'<d$ have also been scheduled at time $t$. No bin has been evicted at these depths before round $t$, so all bins at depth $d$ are children of active bins at any depth $d'<d$ such that $d'\in\cD_t$. This ensures there exists a path of active bins from any bin $B$ up to depth $\dmin$. \\
Using induction, we now show that
\begin{align}\label{eq:lower_bound_proba}
\forall d' \in \cD_t,\, \mathbb{P}(x_t \in B_{d'} \condi \cF_{t-1}) \geq 1/2^{d'}\,.
\end{align}
The base case at $\dmin$ is already established. Assuming the claim holds at depth $d$. For a deeper depth $d' > d$ and any $B' \in \cB_t(d')$ with $B' \subset B_d$, we have
\begin{align*}
\mathbb{P}(x_t \in B' \condi \cF_{t-1}) &= \mathbb{P}(x_t \in B' \condi x_t \in B_d, \cF_{t-1}) \mathbb{P}(x_t \in B_d \condi \cF_{t-1}) \\
&\geq \frac{1}{|\cB_t(d') \cap \mathrm{Children}(B_d, d')|} \frac{1}{2^d} \\
&\geq \frac{2^d}{2^{d'}} \frac{1}{2^d} \\
&= \frac{1}{2^{d'}}\,.
\end{align*}
\textbf{Bound on each $M_t$.}
Recall the definition of $\hat\delta_t(B', B)$,
\begin{align*}
\hat\delta_t(B', B) = \frac{Y_t(x_t)}{\mathbb{P}(x_t \in B' \condi \cF_{t-1})} \I{x_t \in B'} - \frac{Y_t(x_t)}{\mathbb{P}(x_t \in B \condi \cF_{t-1})} \I{x_t \in B}.
\end{align*}
Using the lower bound \eqref{eq:lower_bound_proba} and the fact that rewards lie in $[0,1]$, we have:
\begin{align*}
\left|\hat\delta_t(B', B) - \E{t-1}{\hat\delta_t(B', B)}\right| \leq 2^{d+1}.
\end{align*}
\textbf{Bound on the variance.}
First, remark that
\begin{align*}
\E{t-1}{\left( \hat\delta_t(B', B) - \E{t-1}{\hat\delta_t(B', B)} \right)^2} \leq \E{t-1}{\hat\delta_t^2(B', B)}.
\end{align*}
Applying the definition and the lower bound \eqref{eq:lower_bound_proba}, we have
\begin{align*}
\E{t-1}{\hat\delta_t^2(B', B)} \leq \frac{2}{(1/2^d)^2}\cdot \frac{1}{2^d} = 2^{d+1},\,
\end{align*}
where we use the fact that the probability that $x_t$ belong to $B_d$ or $B_d'$ is upper bounded by $1/2^d$.
\textbf{Final concentration bound.}
Applying \cref{lemma:freedman} over any interval $[s_1, s_2]$ for a fixed depth $d$ and for any pair of bins $B, B'\in\cT_d$, we get that with probability at least $1 - \delta$,
\begin{align*}
\left| \sum_{t=s_1}^{s_2} \hat\delta_t(B', B) - \sum_{t=s_1}^{s_2} \E{t-1}{\hat\delta_t(B', B)} \right| \leq (e - 1) \left( \sqrt{(s_2 - s_1) 2^{d+1} \log\left(\frac{1}{\delta}\right)} + 2^{d+1} \log\left(\frac{1}{\delta}\right) \right).
\end{align*}
Taking an union bound over all possible choices of rounds $s_1, s_2$ (there are $T^2$ choices), all possible choices of depths (there are at most $m$ depths, where $m \leq \log(T)$), and all possible choices of pair of bins $\mathcal{B}$ (there are at most $\sum_{m=1}^{\log_8(T)} (2^m)^2 \leq T$ choices of pairing), we have with probability at least $1 - \delta T^3 \log(T)$,
\begin{align*}
&\forall [s_1, s_2],\ \forall d \in \bigcap_{t=s_1}^{s_2} \mathcal{D}_t,\ \forall B', B \in \mathcal{B}_t(d),\\
&
\left| \sum_{t=s_1}^{s_2} \hat{\delta}_t(B', B) - \sum_{t=s_1}^{s_2} \mathbb{E}_{t-1}\left[\hat{\delta}_t(B', B)\right] \right|
\leq (e - 1) \left( \sqrt{(s_2 - s_1) 2^{d+1} \log\left(\frac{1}{\delta}\right)} + 2^{d+1} \log\left(\frac{1}{\delta}\right) \right).
\end{align*}
Choosing $\delta = 1/T^7$ and $\textcolor{Maroon}{c_1 = 7(e-1)\sqrt{2}}$ concludes the proof.
\end{proof}
We summarize the bias and concentration control of mean estimates in the following statement.
\begin{corollary}[\textbf{Bias and Concentration Control}]\label{cor:concentration_and_bias}
On event $\mathcal{E}_1$, for all intervals $[s_1, s_2]$, all depths $d \in \bigcap_{t=s_1}^{s_2} \cD_t$, and all $B, B' \in \cB_{[s_1, s_2]}(d)$,
\begin{align*}
\left| \sum_{t=s_1}^{s_2} \hat\delta_t(B', B) - \sum_{t=s_1}^{s_2} \delta_t(B', B) \right|
\leq \underbrace{c_1 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d}}_{\text{Concentration}}
+ \underbrace{4(s_2 - s_1)/2^d}_{\text{Bias}}.
\end{align*}
\end{corollary}
\begin{proof}[Proof of \cref{cor:concentration_and_bias}]
This follows immediately by combining the concentration event (\cref{prop:concentration}) with the bias control result from \cref{prop:sampling_bias}.
\end{proof}
\subsection{Properties of the eviction scheme of \algo}
We show that, under the concentration event $\mathcal{E}_1$, if a bin is \emph{evicted} by \algo, then all arms within that bin must have incurred significant regret.
\begin{proposition}\label{prop:eviction_implies_sig_regret}
On event $\mathcal{E}_1$, if a bin $B\in\cT_d$ is evicted at some round $t\geq \tau_{l,m}$, then for all arm $x\in B$, there exists an interval $[s_1, s_2]$ with $s_1\geq \tau_{l,m}$ and $s_2=t$ such that $x$ has incurred significant regret on this interval.
\end{proposition}
\begin{proof}[Proof of \cref{prop:eviction_implies_sig_regret}]
Assume $B\in\cT_d$ has been evicted at a round $t$. By design of \algo, there exists an interval $[s_1,s_2]$ with $s_1\geq\tau_{l,m}$ and $s_2=t$, such that
\begin{align*}
\exists d'\in \bigcap_{t=s_1}^{s_2}\cD_t,\, \exists \BParent\in\cB_{[s_1,s_2]}(d') \text{ s.t. }B\subseteq \BParent \text{ and } \BParent \text{ is evicted at round }t\,,
\end{align*}
which means that there exists a bin $B'\in\cB_{[s_1, s_2]}(d')$ such that
\begin{align*}
\sum_{t=s_1}^{s_2}\hat\delta_t(B', \BParent)> c_0\log(T)\sqrt{(s_2-s_1)2^{d'}\vee4^{d'}} + \frac{4(s_2-s_1)}{2^{d'}}\,.
\end{align*}
On the concentration event $\mathcal{E}_1$, we have by \cref{cor:concentration_and_bias}, on interval $[s_1, s_2]$,
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(B', \BParent)\geq\sum_{t=s_1}^{s_2}\hat\delta_t(B',\BParent) - c_1\log(T)\left(\sqrt{(s_2 - s_1)2^{d'}\vee4^{d'}}\right) - \frac{4(s_2-s_1)}{2^{d'}}\,.
\end{align*}
and therefore by eviction criteria \eqref{eq:star} we have
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(B', \BParent)&\geq c_0\log(T)\left(\sqrt{(s_2 - s_1)2^{d'}\vee4^{d'}}\right) + \frac{4(s_2-s_1)}{2^{d'}} \\
&- c_1\log(T)\left(\sqrt{(s_2 - s_1)2^{d'}\vee4^{d'}}\right) - \frac{4(s_2-s_1)}{2^{d'}}\\
&\geq 3\log(T)\sqrt{(s_2-s_1)2^{d'}\vee 4^{d'}}\,,
\end{align*}
for $c_0$ satisfying $c_0\geq 3 + c_1$. Since $\sum_{t=s_1}^{s_2} \delta_t(B', \BParent) \leq \sum_{t=s_1}^{s_2} \delta_t(\BParent)$, this implies that $\BParent$ incurs significant regret on $[s_1, s_2]$ (\cref{def:significant_regret_bin}).
Since $x\in\BParent$, it also implies that $x$ incurs significant regret (\cref{def:significant_regret_arm}) by \cref{prop:from_bin_to_arm}.
\end{proof}
We now present a complementary result showing that bins which are not evicted must exhibit low relative cumulative regret.
\begin{proposition}[\textbf{Safe bins have low relative regret}]\label{cor:useful_corollary}
On concentration event $\mathcal{E}_1$, $\forall\,[s_1,s_2]\,,\forall d\in\bigcap_{t=s_1}^{s_2}\cD_t\,,\, \forall B, B'\in\cB_{[s_1, s_2]}(d)$, if both $B$ and $B'$ are not evicted during this interval, then
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(B, B') \leq (c_0+c_1)\log(T)\sqrt{(s_2-s_1)2^d\vee4^d}+\frac{8(s_2-s_1)}{2^d}\,,
\end{align*}
where $c_0$ and $c_1$ are the positive numerical constants defined respectively in \eqref{eq:star} and \cref{cor:concentration_and_bias}.
\end{proposition}
\begin{proof}[Proof of \cref{cor:useful_corollary}]
If $B$ and $B'$ are not evicted on $[s_1, s_2]$, then for all $B'' \in \cB_{[s_1, s_2]}(d)$, the eviction condition does not hold, \emph{i.e.} we have
\begin{align*}
\sum_{t=s_1}^{s_2} \hat\delta_t(B'', B')
\leq c_0 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d} + \frac{4(s_2 - s_1)}{2^d}.
\end{align*}
In particular, for $B'' = B$, we get
\begin{align*}
\sum_{t=s_1}^{s_2} \hat\delta_t(B, B')
\leq c_0 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d} + \frac{4(s_2 - s_1)}{2^d}.
\end{align*}
By \cref{cor:concentration_and_bias}, the true cumulative gap satisfies
\begin{align*}
\sum_{t=s_1}^{s_2} \delta_t(B, B')
&\leq \sum_{t=s_1}^{s_2} \hat\delta_t(B, B')
+ c_1 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d}
+ \frac{4(s_2 - s_1)}{2^d} \\
&\leq (c_0 + c_1) \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d} + \frac{8(s_2 - s_1)}{2^d}.
\end{align*}
\end{proof}
\subsection{Relating episode and significant shifts}
We show that, with high probability, a new episode begins only if a significant shift occurs within it. This ensures a correspondence between \emph{episodes} $[t_l, t_{l+1}[$ and significant phases $[\tau_i, \tau_{i+1}[$. In particular, each significant phase overlaps with at most two episodes.
\begin{proposition}\label{prop:relating_phase_and_episode}
On concentration event $\mathcal{E}_1$, for each episode $[t_l, t_{l+1}[$, there exists at least one significant shift $\tau_i\in[t_l, t_{l+1}[$.
\end{proposition}
\begin{proof}[Proof of \cref{prop:relating_phase_and_episode}]
Let $[\tau_{l,m}, \tau_{l,m+1}[$ be the block during which episode $[t_l, t_{l+1}[$ terminates. By the design of \algo, the episode ends whenever $\cBM = \emptyset$ at the end of this block. Thus, by the eviction condition \eqref{eq:star}, we have:
\begin{align*}
\forall x \in \cX,\; \exists [s_1, s_2] \subseteq [\tau_{l,m}, \tau_{l,m+1}[,\; \exists d \in \bigcap_{s = s_1}^{s_2} \cD_s,\; \exists B \in \cB_{[s_1, s_2]}(d) \text{ with } x \in B,\; \exists B' \in \cB_{[s_1, s_2]}(d)\; \text{s.t.}
\end{align*}
\begin{align*}
\sum_{t = s_1}^{s_2} \hat\delta_t(B', B)
> c_0 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d} + \frac{4(s_2 - s_1)}{2^d}.
\end{align*}
Since we are on event $\mathcal{E}_1$, by \cref{cor:concentration_and_bias},
\begin{align*}
\sum_{t = s_1}^{s_2} \hat\delta_t(B', B)
&\leq \sum_{t = s_1}^{s_2} \delta_t(B', B)
+ c_1 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d} + \frac{4(s_2 - s_1)}{2^d} \\
\implies \sum_{t = s_1}^{s_2} \delta_t(B', B)
&\geq \sum_{t = s_1}^{s_2} \hat\delta_t(B', B)
- c_1 \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d} - \frac{4(s_2 - s_1)}{2^d} \\
\implies \sum_{t = s_1}^{s_2} \delta_t(B', B(x))
&\geq (c_0 - c_1) \log(T) \sqrt{(s_2 - s_1) 2^d \vee 4^d}
\geq 3 \log(T) \sqrt{(s_2 - s_1) 2^d},
\end{align*}
where we used $B_d(x)$ to denote the bin at depth $d$ containing arm $x$.
This implies that bin $B_d(x)$ incurs significant regret on interval $[s_1, s_2]$ (by \cref{def:significant_regret_bin}). Then, by \cref{prop:from_bin_to_arm}, the arm $x \in B$ also incurs significant regret on $[s_1, s_2]$ (\cref{def:significant_regret_arm}).
Thus, every episode contains at least one significant shift.
\end{proof}
\subsection{Regret decomposition and discretion within one block}
We first introduce important definitions used throughout the regret analysis.
\begin{definition}[\textbf{Last safe arm} $\bm{x_i^\sharp}$]\label{def:x_sharp}
For each significant phase $[\tau_i, \tau_{i+1}[$, we define $x_i^\sharp$ as the last safe arm (\cref{def:significant_regret_arm}), with ties broken arbitrarily. For all $t \in [\tau_i, \tau_{i+1}[$, we define $x_t^\sharp = x_i^\sharp$.
\end{definition}
\begin{definition}[\textbf{Last safe bin} $\bm{B^\sharp_{i,d}}$]\label{def:Bsharp}
We denote by $B^\sharp_{i,d} \in \cT_d$ the unique bin at depth $d$ containing $x_i^\sharp$. For all $t \in [\tau_i, \tau_{i+1}[$, we define $B_{t,d}^\sharp = B^\sharp_{i,d}$.
\end{definition}
\begin{definition}[\textbf{Bin }$\bm{B_{t,d}}$]\label{def:B_t_d}
For each round $t$, let $B_{t,d} \in \cT_d$ denote the unique bin at depth $d$ containing the arm $x_t$ played at time $t$: $x_t \in B_{t,d}$.
\end{definition}
Note that $B_{t,d}$ may not belong to the active set $\cB_t(d)$.
\textbf{Decomposing dynamic regret into relative regrets.}
We begin from the definition of the expected dynamic regret:
\begin{align*}
\E{}{R(\pi_{\algo}, T)} = \sum_{t=1}^T \sup_{x\in \cX}\mu_t(x) - \E{}{\sum_{t=1}^T \mu_t(x_t)} = \E{}{\sum_{t=1}^T \delta_t(x_t)}\,,
\end{align*}
which decomposes as
\begin{align*}
\E{}{\sum_{t=1}^T \delta_t(x_t)} &= \sum_{t=1}^T \delta_t(x_t^\sharp) + \E{}{\sum_{t=1}^T \delta_t(x_t^\sharp,x_t)}\,,
\end{align*}
where $x_t^\sharp$ is deterministic. The first term can be bounded using the definition of $x_t^\sharp$:
\begin{align*}
\sum_{t=1}^T \delta_t(x_t^\sharp) = \sum_{i=0}^{\tilde{L}_T}\sum_{t=\tau_i}^{\tau_{i+1}} \delta_t(x_i^\sharp) \leq \sum_{i=0}^{\tilde{L}_T}\log(T)\left(\tau_{i+1}-\tau_i\right)^{2/3}\,.
\end{align*}
The difficulty lies in upper bounding the second sum $\E{}{\sum_{t} \delta_t(x_t^\sharp,x_t)}$. Without loss of generality, there are $T$ total episodes, and by convention we set $t_l = T+1$ if only $l-1$ episodes occur by round $t$. Summing over episodes gives
\begin{align*}
\E{}{\sum_{t=1}^T \delta_t(x_t^\sharp,x_t)}\leq \sum_{l=1}^T\E{}{\sum_{t=t_l}^{t_{l+1}}\delta_t(x_t^\sharp,x_t)}\,.
\end{align*}
We further decompose each episode $[t_l, t_{l+1}[$ into blocks. Let $M_l = \lceil \log_8(7(t_{l+1} - t_l) + 1) - 1 \rceil$ be the maximum number of block within this episode. For notational convenience, we can extend $\tau_{l,m} = t_{l+1} - 1$ for $m > M_l$. Then,
\begin{align*}
\sum_{l=1}^T \E{}{\sum_{t=t_l}^{t_{l+1}}\delta_t(x_t^\sharp, x_t)} &\leq \sum_{l=1}^T \E{}{\sum_{m=0}^{M_l}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(x_t^\sharp, x_t)} \,.
\end{align*}
\textbf{Discretization trick within each block.}
Within each block $[\tau_{l,m}, \tau_{l,m+1}[$, we relate the regret $\delta_t(x_t^\sharp, x_t)$ to the regret of their corresponding parent bins at depth $m$, introducing a discretization bias of order $1/2^m$:
\begin{align*}
\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(x_t^\sharp, x_t) &= \sumblock \mu_t(x_t^\sharp) - \mu_t(x_t)\\
&=\sumblock \mu_t(x_t^\sharp) - \mu_t(B_{t,m}^\sharp) + \sumblock \mu_t(B_{t,m}^\sharp)-\mu_t(x_t)
\end{align*}
The first term captures the \emph{cumulative discretization error} over one block $[\tau_{l,m},\tau_{l,m+1}[$ at depth $m$, and by \cref{assumption:lipschitz}, it is upper bounded as
\begin{align*}
\sumblock \mu_t(x_t^\sharp) - \mu_t(B_{t,m}^\sharp)\leq \sumblock |\mu_t(x_t^\sharp) - \mu_t(B_{t,m}^\sharp)| \leq \frac{1}{2^m}(\tau_{l,m+1}-\tau_{l,m})\leq 4^m\,.
\end{align*}
For the second term, by design of our sampling scheme (\cref{alg:sampling_scheme}), if $B_{t,m}$ is active, then conditionally on $B_{t,m}=B$ we have $x_t\sim\mathcal{U}(B)$. Otherwise, $x_t$ is sampled uniformly from one of its active parent at higher depth (that has no active child), and conditionally on $B_{t,m}=B$ we also have $x_t\sim\mathcal{U}(B)$, and therefore in both cases, $\forall t,\,\E{}{\mu_t(x_t)\condi B_{t,m}=B}=\mu_t(B)$. This yields
\begin{align*}
\E{}{\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1} \mu_t(B^\sharp_{t,m}) - \mu_t(x_t)}&= \E{}{\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1} \mu_t(B^\sharp_{t,m}) - \mu_t(B_{t,m})}\\
&=\E{}{\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1} \delta_t(B^\sharp_{t,m},B_{t,m})}\,.
\end{align*}
Putting this together,
\begin{align}\label{eq:goal_regret_one_block}
\E{}{\sum_{m=0}^{M_l}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(x_t^\sharp, x_t)}
&\leq \E{}{\sum_{m=0}^{M_l} 4^m} + \E{}{\sum_{m=0}^{M_l} \sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B^\sharp_{t,m}, B_{t,m})}\,.
\end{align}
\textbf{Upper bounding the bias.} Using the definition of $M_l$, we upper bound the bias over one episode:
\begin{align*}
\E{}{\sum_{m=0}^{M_l} 4^m}
= \frac{4^{M_l+1} - 1}{3}
\leq 6\E{}{(t_{l+1} - t_l)^{2/3}}.
\end{align*}
Summing over all episodes yields:
\begin{align}\label{eq:bias_term}
\sum_{l=1}^T \E{}{\sum_{m=0}^{M_l} 4^m}
\leq 6 \sum_{l=1}^T \E{}{(t_{l+1} - t_l)^{2/3}}.
\end{align}
\textbf{Summary of regret decomposition.}
So far, we have shown that expected dynamic regret is upper bounded as
\begin{align*}
\E{}{R(\pi_{\algo}, T)} &\leq \log(T) \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3}
+ 6 \sum_{l=1}^T \E{}{(t_{l+1} - t_l)^{2/3}} \\
&+ \sum_{l=1}^T \E{}{\sum_{m=0}^{M_l} \sumblock \delta_t(B_{t,m}^\sharp, B_{t,m})}
\end{align*}
Using the fact that event $\mathcal{E}_1$ holds with probability at least $1-1/T^3$ (\cref{prop:concentration}), and the fact that rewards are bounded in $[0, 1]$, we get
\begin{align}\label{eq:summary}
\E{}{R(\pi_{\algo}, T)} &\leq \log(T) \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3}
+ 6 \sum_{l=1}^T \E{}{\I{\mathcal{E}_1}(t_{l+1} - t_l)^{2/3}} + 6\frac{T^2}{T^3}\nonumber\\
&+ \sum_{l=1}^T \E{}{\I{\mathcal{E}_1}\sum_{m=0}^{M_l} \sumblock \delta_t(B_{t,m}^\sharp, B_{t,m})} + \frac{T^2}{T^3}\nonumber\\
&= \log(T) \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3}
+ 6 \sum_{l=1}^T \E{}{\I{\mathcal{E}_1}(t_{l+1} - t_l)^{2/3}} \nonumber\\
&+ \sum_{l=1}^T \E{}{\I{\mathcal{E}_1}\sum_{m=0}^{M_l} \sumblock \delta_t(B_{t,m}^\sharp, B_{t,m})} +\frac{7}{T}\,.
\end{align}
It remains to bound the terms
\begin{align*}
\sum_{l=1}^T \E{}{\I{\mathcal{E}_1}(t_{l+1} - t_l)^{2/3}}
\end{align*}
and
\begin{align*}
\sum_{l=1}^T \E{}{\I{\mathcal{E}_1}\sum_{m=0}^{M_l} \sumblock \delta_t(B_{t,m}^\sharp, B_{t,m})}\,.
\end{align*}
We first focus on the latter term. We introduce the following useful definition.
\begin{definition}[\textbf{Last bin in }$\bm{\cBM}$: $\bm{\Blast_{l,m}}$]\label{def:Blast}
We denote by $\Blast_{l,m} \in \cT_m$ the last bin (with ties broken arbitrarily) at depth $m$ within block $[\tau_{l,m}, \tau_{l,m+1}[$ that belongs to $\cBM$.
\end{definition}
Then we decompose the last sum of \eqref{eq:summary}, and condition on $\cF_{\tau_{l,m}}$ using a tower rule,
\begin{align}\label{eq:reconditionning}
&\E{}{\sum_{m=0}^{M_l}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B^\sharp_{t,m}, B_{t,m})} \nonumber\\
&= \E{}{\sum_{m=0}^{M_l}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(\Blast_{l,m}, B_{t,m})} + \E{}{\sum_{m=0}^{M_l}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})}\nonumber\\
&=\E{}{\sum_{m=0}^{M_l}\E{}{\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(\Blast_{l,m}, B_{t,m})\,\Big|\,\cF_{\tau_{l,m}}}} + \E{}{\sum_{m=0}^{M_l}\E{}{\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\,\Big|\,\cF_{\tau_{l,m}}} }\,,
\end{align}
where we recall we defined the filtration $(\cF_t)_{t\geq 1}$ as $\cF_{t} = \sigma(\{x_s, Y_s(x_s)\}_{s=1}^{t-1})$, with by convention $\cF_{1}$ being the trivial sigma algebra. For the next two subsections, we focus on the conditional expectations
\begin{align*}
\textcolor{blue}{(A)}=\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(\Blast_{l,m}, B_{t,m})\,\Big|\,\cF_{\tau_{l,m}}}
\end{align*}
and
\begin{align*}
\textcolor{blue}{(B)}=\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\,\Big|\,\cF_{\tau_{l,m}}}\,,
\end{align*}
and our goal is to show that these terms are upper-bounded almost surely by a term of order of $4^m$.
We conclude this subsection by noting two important observations:
\begin{itemize}
\item At the start of each block $\tau_{l,m}$, all previous observations are discarded.
\item Replays within block $[\tau_{l,m}, \tau_{l,m+1}[$ are scheduled at $\tau_{l,m}$, \emph{independently of observations prior to that round}. Thus, for $t\in[\tau_{l,m},\tau_{l,m+1}[$ and $d<m$, $R_{t,d}$ is $\cF_{\tau_{l,m}}$-measurable.
\end{itemize}
\subsection{Upper bounding \textcolor{blue}{(A)}}
The term ${\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(\Blast_{l,m}, B_{t,m})}$ captures the cumulative regret between any bin that is deemed safe by the \emph{algorithm} and the bin $B_{t,m}$ selected by the algorithm at each round $t$. Notably, $B_{t,m}$ is either
\begin{itemize}
\item A bin at depth $m$ deemed safe and \emph{retained} in the active set $\cBM$, or
\item A bin that has been \emph{evicted} (or whose parent was evicted at a shallower depth $d < m$) and is being re-explored as part of a \emph{replay} initiated at any active depth $d<m$.
\end{itemize}
Importantly, evicted bins can reappear during replays, but only through replays initiated at depths strictly less than $m$. We aim to show that \textcolor{blue}{(A)} is upper-bounded almost surely by a quantity of order $4^m$.
\begin{proposition}[\textbf{Upper bound of \textcolor{blue}{(A)}}]\label{prop:upper_bound_term_A}
There exists a positive numerical constant $c_A > 0$ such that
\begin{align*}
\textcolor{blue}{(A)} \leq c_A\log^2(T) 4^m\,.
\end{align*}
\end{proposition}
Our approach begins by upper bounding \textcolor{blue}{(A)} through a decomposition at depth $\dmin$ and leverage the properties of our sampling scheme (\cref{alg:sampling_scheme}) . In particular, it leverages the fact that at each round $t$, we choose \emph{uniformly} an active bin at depth $\dmin$.
\begin{lemma}\label{prop:alternative_form_A}
We have that \textcolor{blue}{(A)} is upper bounded as
\begin{align*}
\textcolor{blue}{(A)}\leq \E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\left(\sum_{B\in\cB_t(\dmin)}\frac{\delta_t\left( \Par(\Blast_{l,m}, \dmin), B\right)}{|\cB_t(\dmin)|}+\frac{4}{2^{\dmin}}\right)\condicFgros}\,.
\end{align*}
\end{lemma}
\begin{proof}[Proof of \cref{prop:alternative_form_A}]
For all $t\in[\tau_{l,m}, \tau_{l,m+1}[$, we first relate the relative regret of $B_{t,m}$ to $\Blast_{l,m}$ to the relative regret of their parent at depth $\dmin$,
\begin{align*}
\delta_t(\Blast_{l,m}, B_{t,m}) &= \mu_t(\Blast_{l,m}) - \mu_t(B_{t,m})\\
&= \mu_t(\Blast_{l,m})- \mu_t\left(\Par(\Blast_{l,m}, \dmin)\right) \\
&+ \mu_t\left(\Par(\Blast_{l,m}, \dmin)\right) - \mu_t(\Par(B_{t,m}, \dmin)) \\
&+ \mu_t(\Par(B_{t,m}, \dmin)) - \mu_t(B_{t,m})\\
&\leq \left|\mu_t(\Blast_{l,m})- \mu_t\left(\Par(\Blast_{l,m}, \dmin)\right) \right| \\
&+ \mu_t\left(\Par(\Blast_{l,m}, \dmin)\right) - \mu_t(\Par(B_{t,m}, \dmin)) \\
&+ \left|\mu_t(\Par(B_{t,m}, \dmin)) - \mu_t(B_{t,m})\right|\\
&\leq \delta_t\left( \Par(\Blast_{l,m}, \dmin), \Par(B_{t,m}, \dmin) \right) + \frac{4}{2^{\dmin}}\,,
\end{align*}
where in the last inequality we used the fact that $B_{t,m}\subseteq \Par(B_{t,m},\dmin)$, $\Blast_{l,m}\subseteq\Par(\Blast_{l,m}, \dmin)$ and \cref{assumption:lipschitz}. Note that $\Par(B_{t,m}, \dmin)$ is necessarily active at depth $\dmin$.
Summing this inequality over the block $[\tau_{l,m}, \tau_{l,m+1}[$,
\begin{align*}
\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t\left(\Blast_{l,m}, B_{t,m}\right) &\leq \underbrace{\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t\left( \Par(\Blast_{l,m}, \dmin), \Par(B_{t,m}, \dmin) \right)}_{\text{Regret of parents at minimum active depth}} \\
&+ \underbrace{ \sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\frac{4}{2^{\dmin}}}_{\text{cumulative bias}}\,.
\end{align*}
By tower rule, since $\cF_{\tau_{l,m}}\subseteq\cF_{t-1}$ we have for the first sum, for any bin $B\in\cT_m$
\begin{align*}
&\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t\left( \Par(B, \dmin), \Par(B_{t,m}, \dmin) \right)\,\Big|\,\cF_{\tau_{l,m}}} \\
&=\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}} \E{}{\delta_t\left( \Par(B, \dmin), \Par(B_{t,m}, \dmin) \right)\,\bigg|\,\cF_{t-1}}\,\Big|\,\cF_{\tau_{l,m}}}\,.
\end{align*}
Since $\Par(B_{t,m},\dmin)$ is an active bin at depth $\dmin$ and since we choose \emph{uniformly} an active bin at depth $\dmin$, we can write for any bin $B\in\cT_m$,
\begin{align*}
&\E{}{\delta_t\left( \Par(B, \dmin), \Par(B_{t,m}, \dmin) \right)\,\bigg|\, \cF_{t-1}} \\
&= \sum_{B\in\cB_t(\dmin)}\delta_t\left( \Par(B, \dmin), B \right)\mathbb{P}\left(\Par(B_{t,m}, \dmin)=B\,\big|\, \cF_{t-1}\right)\\
&=\sum_{B\in\cB_t(\dmin)}\delta_t\left( \Par(B, \dmin), B \right)\frac{1}{|\cB_t(\dmin)|}\,,
\end{align*}
and therefore for all $B\in\cT_m$,
\begin{align*}
&\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t\left( \Par(B, \dmin), \Par(B_{t,m}, \dmin) \right)\,\Big|\,\cF_{\tau_{l,m}}}\\
&=\E{}{\I{\mathcal{E}_1}\sumblock \sum_{B\in\cB_t(\dmin)}\delta_t\left( \Par(B, \dmin), B \right)\frac{1}{|\cB_t(\dmin)|}\,\Big|\,\cF_{\tau_{l,m}}}
\end{align*}
Choosing $B = \Blast_{l,m}\in\cT_m$ in particular, the result above implies
\begin{align*}
&\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t\left( \Par(\Blast_{l,m}, \dmin), \Par(B_{t,m}, \dmin) \right)\,\Big|\,\cF_{\tau_{l,m}}}\\
&=\E{}{\I{\mathcal{E}_1}\sumblock \sum_{B\in\cB_t(\dmin)}\delta_t\left( \Par(\Blast_{l,m}, \dmin), B \right)\frac{1}{|\cB_t(\dmin)|}\,\Big|\,\cF_{\tau_{l,m}}}\,,
\end{align*}
and therefore
\begin{align*}
&\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t\left(\Blast_{l,m}, B_{t,m}\right)\condicFgros} \\
&\leq \E{}{\I{\mathcal{E}_1}\sumblock\left( \sum_{B\in\cB_t(\dmin)}\delta_t\left( \Par(\Blast_{l,m}, \dmin), B \right)\frac{1}{|\cB_t(\dmin)|}+\frac{4}{2^{\dmin}}\right)\,\Big|\,\cF_{\tau_{l,m}}}
\end{align*}
\end{proof}
We are now ready to prove \cref{prop:upper_bound_term_A}. The rest of this subsection is dedicated to this purpose.
Thanks to \cref{prop:alternative_form_A}, it now suffices to bound both the parent's regret contribution at depth $\dmin$
\begin{align*}
\textcolor{blue}{(A.1)} = \E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{B\in\cB_t(\dmin)}\frac{\delta_t\left( \Par(\Blast_{l,m}, \dmin), B\right)}{|\cB_t(\dmin)|}\condicFgros}
\end{align*}
and the cumulative bias term
\begin{align*}
\textcolor{blue}{(A.2)} = \E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\frac{4}{2^{\dmin}}\condicFgros}\,.
\end{align*}
\textbf{Upper bounding \textcolor{blue}{(A.1.)}.}
Considering all possible value of $\dmin$, \textcolor{blue}{(A.1)} can be rewritten using our ordering introduced in \cref{def:bin_ordering}: denoting $B_d^{(1)}$, $B_d^{(1)}\dots,B_d^{(2^d)}$ the bins at depth $d$, \begin{align}\label{eq:A1_two_sums}
&\textcolor{blue}{(A.1)}\nonumber\\
&=\E{}{\I{\mathcal{E}_1} \sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1} \sum_{d=0}^m \sum_{B\in\cB_t(d)}\I{\dmin=d}\frac{\delta_t\left(\Par(\Blast_{l,m}, d), B\right)}{|\cB_t(d)|}\condicFgros}\nonumber \\
&= \E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{d=0}^m \sum_{k=1}^{2^d}\I{\dmin = d}\frac{\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)}{|\cB_t(d)|}\I{B_d^{(k)}\in\cB_t(d)}\condicFgros} \nonumber \\
&= \underbrace{\E{}{\I{\mathcal{E}_1} \sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{k=1}^{2^m}\I{\dmin = m}\frac{\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right)}{|\cB_t(m)|}\I{B_m^{(k)}\in\cB_t(m)}\condicFgros}}_{\text{Regret contribution at depth }m}\nonumber \\
&+\underbrace{\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}\I{\dmin = d}\frac{\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)}{|\cB_t(d)|}\I{B_d^{(k)}\in\cB_t(d)}\condicFgros}}_{\text{Regret contribution of replays of depth }d<m}\,.
\end{align}
We introduce the \emph{eviction time} of a bin $B$, which will be convenient to quantify the regret contribution of each bin.
\begin{definition}[\textbf{Eviction time within a replay} $\bm{M(s,d,B)}$]\label{def:eviction_time}
For all bin $B$, $M(s,d,B)$ denotes the last round where bin $B$ is active within a replay at depth $d$ starting at round $s$, $M(s,d,B)\in[s,s+8^d]$. If bin $B$ is not evicted during this replay, we define $M(s,d,B)=s+8^d$. By abuse of notations, we define $M(\tau_{l,m},m,B)$ as the last round where bin $B$ is retained in $\cBM$, and define $M(\tau_{l,m},m,B)=\tau_{l,m}+8^m-1=\tau_{l,m+1}-1$ if it is not evicted during the block $[\tau_{l,m},\tau_{l,m+1}[$.
\end{definition}
\textbf{Regret contribution at depth $m$.}
Since $\cBM$ is initialized at round $t=\tau_{l,m}$, we have for all bins $B_m^{(k)}$, on event $\mathcal{E}_1$,
\begin{align*}
&\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{k=1}^{2^m}\I{\dmin = m}\frac{1}{|\cB_t(m)|}\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right)\I{B_m^{(k)}\in\cB_t(m)}\\
&=\sum_{k=1}^{2^m}\sum_{t=\tau_{l,m}}^{M(\tau_{l,m},m,B_m^{(k)})}\I{\dmin = m}\frac{1}{|\cB_t(m)|}\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right) \,.
\end{align*}
Without any loss of generality, we assume that $\{B_m^{(1)},\cdots,B_m^{(2^m)}\}$ is the ordering according to the \emph{eviction time} of these bins in the block $[\tau_{l,m},\tau_{l,m+1}[$ (otherwise we can always re-index the bins), that is,
\begin{align*}
\tau_{l,m}\leq M\big(\tau_{l,m},d,B_m^{(1)}\big)\leq M\big(\tau_{l,m},m,B_m^{(2)}\big) \leq \dots \leq \underbrace{M\big(\tau_{l,m}, m, B_d^{(2^m)}\big)}_{=\Blast_{l,m}}\leq \tau_{l,m+1}-1\,.
\end{align*}
Thanks to this ordering, we can lower bound the active bin set at depth $d$ as
\begin{align*}
\min_{t\in[\tau_{l,m},M(\tau_{l,m},m,B_m^{(k)})]}|\cB_t(m)|\geq 2^m +1 - k\,,
\end{align*}
and therefore
\begin{align*}
&\sum_{k=1}^{2^m}\sum_{t=\tau_{l,m}}^{M(\tau_{l,m},m,B_m^{(k)})}\I{\dmin = m}\frac{1}{|\cB_t(m)|}\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right) \\
&\leq \sum_{k=1}^{2^m}\frac{1}{2^m+1-k}\sum_{t=\tau_{l,m}}^{M(\tau_{l,m},m,B_m^{(k)})}\I{\dmin = m}\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right)\,.
\end{align*}
Moreover, since $\Blast_{l,m}$ and $B_m^{(k)}$ are by definition not evicted on interval $[\tau_{l,m}, M(\tau_{l,m},m,B_m^{(k)})]$, we have by \cref{cor:useful_corollary} on event $\mathcal{E}_1$
\begin{align*}
&\sum_{t=\tau_{l,m}}^{M(\tau_{l,m},m,B_m^{(k)})}\I{\dmin = m}\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right)\\
&\leq (c_0+c_1)\log(T)\sqrt{\left(M(\tau_{l,m},m,B_m^{(k)})-\tau_{l,m}\right)2^m \vee 4^m}+\frac{8\left(M(\tau_{l,m},m,B_m^{(k)})-\tau_{l,m}\right)}{2^m}\\
&\leq (c_0+c_1)\log(T)4^m+8\times 4^m\\
&\leq (c_0 + c_1 + 8)\log(T)4^m\\
&\leq c_2\log(T)4^m\,,
\end{align*}
assuming again horizon $T$ satisfies $\log(T)\geq 1$ and setting $\textcolor{Maroon}{c_2=c_0+c_1+8}$. Therefore, the regret contribution of replay at depth $m$ is upper bounded as
\begin{align*}
\sum_{k=1}^{2^m}\sum_{t=\tau_{l,m}}^{M(\tau_{l,m},m,B_m^{(k)})}\I{\dmin = m}\frac{\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right)}{|\cB_t(m)|} &\leq c_2\log(T)4^m \sum_{k=1}^{2^m}\frac{1}{2^m+1-k} \nonumber \\
&\leq (c_0 + c_1 + 8)\log(T)4^m (\log(2^m)+1) \nonumber\\
&\leq (c_2+1)\log^2(T)4^m\,,
\end{align*}
where we used the fact that $m\leq \log(T)$.
Therefore, the regret contribution of the replay at depth $m$ is upper bounded as
\begin{align}\label{eq:regret_contrib_replay_m}
&\E{}{\I{\mathcal{E}_1} \sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{k=1}^{2^m}\I{\dmin = m}\frac{\delta_t\left(\Blast_{l,m}, B_m^{(k)}\right)}{|\cB_t(m)|}\I{B_m^{(k)}\in\cB_t(m)}\condicFgros}\nonumber\\
&\leq (c_2+1)\log^2(T)4^m\,.
\end{align}
\textbf{Regret contribution of replays at depth $d<m$.}
Fix a depth $d \in [\![0, m-1]\!]$ and a bin $B_d^{(k)}$ at this depth. We consider the set of rounds $t > \tau_{l,m}$ for which $\dmin = d$ and $B_d^{(k)} \in \cB_t(d)$, and partition this set into contiguous intervals (represented as blue intervals in \cref{fig:partition}).
Due to the structure of our replay schedule, each such interval must correspond to a single replay at depth $d$, which begins at some round $s$ satisfying $R_{s,d} = 1$. That is, each interval is initiated at the start of a replay and includes only the rounds during which $B_d^{(k)}$ remains active in that replay.
Each interval concludes either when $B_d^{(k)}$ is evicted from the active set $\cB_t(d)$, or when the replay itself ends at time $s + 8^d$. Therefore, each interval spans the range $[s, M(s,d,B_d^{(k)})]$, where $M(s,d,B_d^{(k)})$ is the final round during which $B_d^{(k)}$ is active within that specific replay (see \cref{def:eviction_time}).
The key idea of our analysis is to treat these replays at depth $d$ \emph{independently}, and quantify the regret incurred due to playing the bin $B_d^{(k)}$ during each such replay window. This allows us to control the regret by bounding its contribution separately within each replay interval.
\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{Figures/partition_proof.pdf}
\caption{For a given depth $d$, partition of the rounds of a block where $\dmin=d$. Any of these blue intervals should be initialized by the start of a replay at this depth, \emph{i.e.} $R_{s,d}=1$.}
\label{fig:partition}
\end{figure}
Following these remarks, we can upper bound the second sum of \cref{eq:A1_two_sums} on event $\mathcal{E}_1$ as
\begin{small}
\begin{align*}
&\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}\I{\dmin = d}\frac{\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)}{|\cB_t(d)|}\I{B_d^{(k)}\in\cB_t(d)}\\
&\leq \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}R_{s,d}\left(\sum_{t=s}^{M(s,d,B_d^{(k)})}\frac{\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)}{|\cB_t(d)|}\right)_{+} \\
&\leq \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}R_{s,d}\frac{1}{\min_{t\in[s,M(s,d,B_d^{(k)})]}|\cB_t(d)|}\left(\sum_{t=s}^{M(s,d,B_d^{(k)})}\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)\right)_+\,.
\end{align*}
\end{small}
Again, without any loss of generality, we assume that $\{B_d^{(1)},\cdots,B_d^{(2^d)}\}$ is the ordering corresponding to eviction time of the replay at depth $d$ starting at round $s$,
\begin{align*}
s\leq M\big(s,d,B_d^{(1)}\big)\leq M\big(s,d,B_d^{(2)}\big) \leq \dots \leq M\big(s, d, B_d^{(2^d)}\big)\leq s+8^d\,.
\end{align*}
Thanks to this ordering, we can lower bound the size of the active bin set at depth $d$ as
\begin{align*}
\min_{t\in[s,M(s,d,B_d^{(k)})]}|\cB_t(d)|\geq 2^d +1 - k\,,
\end{align*}
and therefore,
\begin{align*}
& \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}R_{s,d}\frac{1}{\min_{t\in[s,M(s,d,B_d^{(k)})]}|\cB_t(d)|}\left(\sum_{t=s}^{M(s,d,B_d^{(k)})}\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)\right)_+\\
&\leq \sum_{d=0}^{m-1} \sum_{k=1}^{2^d}\frac{1}{2^d - k + 1}\sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d}\left(\sum_{t=s}^{M(s,d,B_d^{(k)})}\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)\right)_+ \,.
\end{align*}
We now proceed to bound the cumulative relative regret incurred by a bin $B_d^{(k)}$ with respect to its parent $\Par(\Blast_{l,m}, d)$. By design of $\Blast_{l,m}$ (it is a safe bin over the entire block), it follows that its parents $\{\Par(\Blast_{l,m}, d)\}_{d \in \cD_t}$ must also be safe. Indeed, if a parent $\Par(\Blast_{l,m}, d)$ were deemed unsafe at some active depth $d \in \cD_t$, it would necessarily be evicted. Due to the eviction mechanism, this would result in the eviction of all of its descendants, including $\Blast_{l,m}$ itself, contradicting the assumption that $\Blast_{l,m}$ remains safe.
Now, since the bin $B_d^{(k)}$ is active over the interval $[s, M(s,d,B_d^{(k)})]$, we can apply \cref{cor:useful_corollary} to obtain the desired bound on the event $\mathcal{E}_1$.
\begin{align*}
\sum_{t=s}^{M(s,d,B_d^{(k)})}\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)} \right) &\leq (c_0+c_1)\log(T)\sqrt{2^d\left(
M(s,d,B_d^{(k)}) - s \right)\vee 4^d} \\
&+ \frac{8\left(
M(s,d,B_d^{(k)})-s \right)}{2^d}\\
&\leq (c_0+c_1)\log(T)\sqrt{2^d8^d\vee 4^d} + \frac{8\times8^d}{2^d}\\
&\leq (c_0+c_1)\log(T)4^d + 8\times4^d \\
&\leq (c_0+c_1+8)\log(T)4^d\\
&\leq c_2\log(T)4^d\,,
\end{align*}
assuming horizon $T$ satisfies $\log(T)\geq 1$, and where the second inequality follows from the fact that the length of a replay at depth $d$ stating at round $s$ is at most $8^d$. We used the same definition of $c_2$ as above. Therefore, we have
\begin{align*}
& \E{}{\I{\mathcal{E}_1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}\frac{1}{2^d - k + 1}\sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d}\left(\sum_{t=s}^{M(s,d,B_d^{(k)})}\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)\right)_+ \condicFgros}\\
&\leq c_2 \log(T)\E{}{\I{\mathcal{E}_1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}\frac{1}{2^d - k + 1}\sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d}4^d \condicFgros} \\
&\leq c_2 \log(T)\E{}{\I{\mathcal{E}_1}\sum_{d=0}^{m-1} \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d} (\log(2^d)+1) 4^d \condicFgros} \\
&\leq c_2\log^2(T)\E{}{\I{\mathcal{E}_1}\sum_{d=0}^{m-1} \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d} 4^d \condicFgros} \\
&\leq c_2\log^2(T)\E{}{\sum_{d=0}^{m-1} \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d} 4^d \condicFgros}\,,
\end{align*}
where we used the fact that $m\leq \log(T)$.
We now leverage the fact that \emph{relatively few replays occur in expectation}, and the fact that the Bernoulli random variables $R_{s,d}$ are sampled \emph{independently of the observations collected during the episode}. More precisely, all Bernoulli variables are drawn \emph{i.i.d.} at the beginning of the block, \emph{i.e.} at round $t = \tau_{l,m}$. This allows us to treat the scheduling decisions as fixed ahead of time. Furthermore, by the design of the replay scheduling mechanism in \cref{alg:sampling_scheme},
\begin{align}\label{eq:bound_mixture_proba}
\E{}{\sum_{d=0}^{m-1} \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d} 4^d\condicFgros}
&\leq \E{}{\sum_{d=0}^{m-1} \sum_{s=\tau_{l,m}+1}^{\tau_{l,m+1}-1}\frac{\I{s-\tau_{l,m}\equiv 0[8^d]}}{\sqrt{s-\tau_{l,m}}} \sqrt{8^d}4^d\condicFgros}\nonumber \\
&\leq \sum_{d=0}^{m-1} \sum_{i=1}^{8^{m-d}+1}\frac{1}{\sqrt{i8^d}} \sqrt{8^d}4^d\nonumber \\
&\leq 2\sqrt{8}\sum_{d=0}^{m-1}4^d\sqrt{8^{m-d}} \nonumber\\
&\leq 2\sqrt{8}\sqrt{8^m}\frac{\sqrt{2^m}}{\sqrt{2}-1} \leq 6\sqrt{8}\times 4^m\,.
\end{align}
Therefore, the regret contribution of replays at depth $d\in[\![0,m-1]\!]$ is upper bounded as
\begin{align}\label{eq:temp1_A_1}
&\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\sum_{d=0}^{m-1} \sum_{k=1}^{2^d}\I{\dmin = d}\frac{\delta_t\left(\Par(\Blast_{l,m}, d), B_d^{(k)}\right)}{|\cB_t(d)|}\I{B_d^{(k)}\in\cB_t(d)}\condicFgros}\nonumber \\
&\leq 6\sqrt{8}c_2\log^2(T)4^m\,.
\end{align}
From \cref{eq:A1_two_sums}, combining \cref{eq:regret_contrib_replay_m,eq:temp1_A_1} gives the desired rate for \textcolor{blue}{(A.1)},
\begin{align}\label{eq:bound_A_1}
\textcolor{blue}{(A.1)} \leq 18(c_2+1)\log^2(T) 4^m\,.
\end{align}
\textbf{Upper bounding \textcolor{blue}{(A.2)}.}
To bound this term, we apply the same reasoning previously used to control \textcolor{blue}{(A.1)}, and in particular the partition argument of \cref{fig:partition}. By observing that each replay at depth $\dmin=d$ is initiated by a round $s$ such that $R_{s,d}=1$, and by using the fact that each replay at depth $d$ lasts at most $8^d$ rounds, we distinguish the cumulative discretization bias incurred at depth $m$ and the one incurred at other depths $d<m$,
\begin{align*}
\textcolor{blue}{(A.2)} &=\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\frac{4}{2^{\dmin}}\condicFgros}\\
&\leq \E{}{\sumblock\frac{4}{2^m}\I{\dmin=m}\condicFgros}+ \E{}{\sum_{d=0}^{m-1}\sumblock\frac{4}{2^d}\I{\dmin=d}\condicFgros}\\
&\leq 4^{m+1} + 4\E{}{\sum_{d=0}^{m-1} \sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\frac{1}{2^d}\I{\dmin = d}\condicFgros} \\
&\leq 4^{m+1} + 4\E{}{\sum_{d=0}^{m-1} \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d}\frac{1}{2^d}8^d \condicFgros} \\
&= 4^{m+1} + 4\E{}{\sum_{d=0}^{m-1} \sum_{s=\tau_{l,m}}^{\tau_{l,m+1}-1}R_{s,d} 4^d\condicFgros } \,,
\end{align*}
which is exactly the same term we bounded in the proof of \textcolor{blue}{(A.1)}. By \eqref{eq:bound_mixture_proba} we thus have the upper bound
\begin{align}\label{eq:temp_final_bound_A_2}
\textcolor{blue}{(A.2)} \leq 4^{m+1} + 24\sqrt{8}\times4^m = 28\sqrt{8}\times 4^m\,.
\end{align}
\textbf{Conclusion.}
Combining \eqref{eq:bound_A_1} and \eqref{eq:temp_final_bound_A_2} gives the desired rate of $4^m$ for term \textcolor{blue}{(A)}, as
\begin{align*}
\textcolor{blue}{(A)} &=\E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(\Blast_{l,m}, B_{t,m})\condicFgros} \\
&\leq \left(18(c_2+1)\log^2(T)+28\sqrt{8}\right) 4^m
\end{align*}
Choosing $\textcolor{Maroon}{c_A=18(c_2+2)}$ concludes the proof.
\subsection{Upper bounding \textcolor{blue}{(B)}}
We recall that term $\textcolor{blue}{(B)}$ writes
\begin{align*}
\textcolor{blue}{(B)} = \E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\condicFgros}\,.
\end{align*}
As discussed in \cref{subsec:proof_sketch_unstable_block}, the main challenge arises in scenarios where a significant shift occurs (as defined in \cref{def:significant_regret_arm}), causing the identity of the last safe arm $x_i^\sharp$ (and consequently the bin $B_{t,m}^\sharp$) to change. In order to avoid incurring large cumulative regret before this shift is detected, the algorithm must initiate a replay at the \emph{appropriate depth} to identify the change in environment. We claim that $\textcolor{blue}{(B)}$ is also of order $4^m$ almost surely.
\begin{proposition}[\textbf{Upper bound of \textcolor{blue}{(B)}}]\label{prop:upper_bound_term_B}
There exists a positive numerical constant $c_B$ such that
\begin{align*}
\textcolor{blue}{(B)}\leq c_{B}\log^2(T)4^m\,.
\end{align*}
\end{proposition}
To prove \cref{prop:upper_bound_term_B}, we analyse the cumulative regret incurred by $\Blast_{l,m}$ with respect to $B_{l,m}^\sharp$ by decomposing the rounds of the blocks into \emph{bad segments}, that is, time intervals during which $\Blast_{l,m}$ incurs significant regret relative to the safe bin $B_{t,m}^\sharp$. Importantly, by construction, $\Blast_{l,m}$ is never evicted during the block $[\tau_{l,m}, \tau_{l,m+1}-1]$, and thus remains active at depth $m$ throughout. However, since $\Blast_{l,m}$ is only revealed at the \emph{end} of the block, we must define bad segments relative to all possible bins at depth $m$.
\begin{definition}[\textbf{Bad segment and midpoint of a bad segment}]\label{def:bad_segment}
Let $[\tau_{l,m}, \tau_{l,m+1}[$ a block, and let $[\tau_i, \tau_{i+1}[$ be any phase intersecting this block. For all bin $B\in\cT_m$, define rounds $(s^m_{i,j}(B))_{j}$ with $s^m_{i,j}(B)\in[\tau_{l,m}\vee\tau_i,\tau_{i+1}\wedge \tau_{l,m}+8^m[$ recursively as follows:
\begin{itemize}
\item $s^m_{i,0}(B) = \tau_{l,m}\vee\tau_i$.
\item $s^m_{i,j}(B)$ is the smallest round in $]s^m_{i,j-1}(B),\tau_{i+1}\wedge \tau_{l,m+1}[$ such that
\begin{align*}
\sum_{t=s^m_{i,j-1}(B)}^{s^m_{i,j}(B)}\delta_t(B_{t,m}^\sharp, B)\geq c_3\log(T) \sqrt{2^m\left(s^m_{i,j}(B) - s^m_{i,j-1}(B)\right)}\,,
\end{align*}
\end{itemize}
where $c_3$ is a fixed positive numerical constant.
For all bin $B\in \cT_m$, we define the \textbf{midpoint of a bad segment} $[s^m_{i,j}(B),s^m_{i,j+1}(B)[$ as the round
\begin{align*}
\tilde{s}^m_{i,j}(B) = \left\lfloor\frac{s^m_{i,j}(B) + s^m_{i,j+1}(B)}{2}\right\rfloor\,.
\end{align*}
\end{definition}
Now, our goal is to analyse the relative regret of any bin $B\in\cT_m$ with respect to $B_{t,m}^\sharp$ on these \emph{bad segments}. To build intuition, our objective is to prevent bad segments from accumulating for any bin $B\in\cT_m$. Fortunately, the design of our replay mechanism ensures that if a \emph{well-timed} replay at a \emph{suitable depth} is triggered at the start of a bad segment, it can promptly detect the shift in rewards and evict $B$, thereby limiting the incurred regret.
\begin{proposition}[\textbf{Properties on bad segments}]\label{prop:bad_segments}
Let $B\in\cT_m$ be a bin and let $[s^m_{i,j}(B),s^m_{i,j+1}(B)[$ be a bad segment. Then, on $\mathcal{E}_1$, any replay starting at round $t_{\mathrm{start}}\in[s^m_{i,j}(B),s^m_{i,j+1}(B)[$ never evicts $\Par(B_{t,m}^\sharp,d)$ for $d\in\bigcap_{t=t_{\mathrm{start}}}^{s^m_{i,j+1}(B)}\cD_t$. Moreover, if a replays at depth $d_{i,j}$ satisfying
$$8^{d_{i,j}+1}\leq s^m_{i,j+1}(B)-s^m_{i,j}(B)\leq 8^{d_{i,j}+2}$$
starts at $t_{\mathrm{start}}\in[s^m_{i,j}(B),\tilde{s}^m_{i,j}(B)[\,$,
then $B$ is evicted by round $s^m_{i,j+1}(B)$.
\end{proposition}
\begin{proof}[Proof of \cref{prop:bad_segments}]
Assume a replay evicts $\Par(B_{t,m}^\sharp,d)$ with $d\in\bigcap_{t=t_{\mathrm{start}}}^{s^m_{i,j+1}(B)}\cD_t$ and $t_{\mathrm{start}}\geq s_{i,j}^m(B)$ before round $s^m_{i,j+1}(B)$. Then, by \cref{prop:from_bin_to_arm}, it means that, on event $\mathcal{E}_1$, every arm $x\in B_{t,m}^\sharp$ incurs significant regret (\cref{def:significant_regret_arm}). This contradicts the definition of $B^\sharp_{l,m}$. Indeed, by definition, $x_{t}^\sharp$ does not change on $[t_{\mathrm{start}}, s_{i,j+1}^m]\subset [\tau_i,\tau_{i+1}]$, so neither does $B_{t,m}^\sharp$.
Therefore, such replay never evicts $\Par(B_{t,m}^\sharp, d)$.
Now, for all $B\in\cT_m$, we have
\begin{small}
\begin{align*}
\sum_{t=\tilde{s}^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\delta_t(B_{t,m}^\sharp, B) &= \sum_{s^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\delta_t(B_{t,m}^\sharp, B) - \sum_{s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)-1}\delta_t(B_{t,m}^\sharp, B)\,.
\end{align*}
\end{small}
By definition of a bad segment (\cref{def:bad_segment}),
\begin{align*}
\sum_{s^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\delta_t(B_{t,m}^\sharp, B) \geq c_3\log(T)\sqrt{2^m\left(s^m_{i,j+1}(B)-s^m_{i,j}(B)\right)}\,.
\end{align*}
and since $s^m_{i,j+1}(B)$ is defined as the \emph{first} round that satisfies the above inequality,
\begin{align*}
\sum_{s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}\delta_t(B_{t,m}^\sharp, B) \leq c_3\log(T)\sqrt{2^m\left( \tilde{s}^m_{i,j}(B) - s^m_{i,j}(B)\right)}\,.
\end{align*}
By combining these two inequalities above,
\begin{align*}
\sum_{t=\tilde{s}^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\delta_t(B_{t,m}^\sharp, B)&\geq c_3\log(T)\sqrt{2^m}\left(\sqrt{s^m_{i,j+1}(B) - s^m_{i,j}(B)}-\sqrt{\tilde{s}^m_{i,j}(B) - s^m_{i,j}(B)}\right)\\
&\geq \frac{c_3}{4}\log(T)\sqrt{2^m \left(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B)\right)}\,,
\end{align*}
where we used the inequality $\sqrt{a+b}-\sqrt{a}\geq \frac{\sqrt{b}}{4}$.
Since $B^\sharp_{l,m}$ is active during the whole of the bad segment, on $\mathcal{E}_1$, we have by \cref{cor:concentration_and_bias}, on event $\mathcal{E}_1$,
\begin{align*}
&\sum_{t=\tilde{s}^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\hat\delta_t(B_{t,m}^\sharp, B) \\
&\geq \sum_{t=\tilde{s}^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\delta_t(B_{t,m}^\sharp, B) - c_1\log(T)\sqrt{(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B))2^m} -\frac{4(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B))}{2^m}\\
&\geq \left(\frac{c_3}{4}-c_1\right)\log(T)\sqrt{ \left(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B)\right)2^m} - \frac{4(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B))}{2^m}\,.
\end{align*}
Using the fact that $s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B)\leq 8^{d_{i,j}+2}\leq 8^{m+1}$, we have
\begin{align*}
\frac{s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B)}{2^m}&\leq 8\sqrt{s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B)}\frac{(2\sqrt{2})^m}{2^m}\\
&\leq 8\sqrt{2^m(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B))}\,,
\end{align*}
and therefore
\begin{align*}
\sum_{t=\tilde{s}^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\hat\delta_t(B_{t,m}^\sharp, B)\geq \left(\frac{c_3}{4}-c_1-\frac{32}{\log(T)}\right)\log(T)\sqrt{ \left(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B)\right)2^m} \,.
\end{align*}
Assuming horizon large enough, \emph{e.g.} $\log(T)\geq 1$, setting $\textcolor{Maroon}{c_3 = 140 + 4c_1}$ implies
\begin{align*}
\sum_{t=\tilde{s}^m_{i,j}(B)}^{s^m_{i,j+1}(B)}\hat\delta_t(B_{t,m}^\sharp, B)\geq 3\log(T)\sqrt{ \left(s^m_{i,j+1}(B)-\tilde{s}^m_{i,j}(B)\right)2^m}\,,
\end{align*}
and hence by \eqref{eq:star}, on event $\mathcal{E}_1$, $B$ is evicted.
\end{proof}
Having established that a well-timed replay at the appropriate depth can evict a bin $B$, it remains to demonstrate that such a replay is indeed \emph{triggered} before the cumulative regret accrued over the bad segments is too large. To formally define the \emph{earliest} round at which this cumulative regret becomes significant, we introduce the notion of a \emph{bad round}.
\begin{definition}[\textbf{Bad round with respect to bin $\bm{B}$}]\label{def:bad_round}
For all block $[\tau_{l,m},\tau_{l,m+1}[\,$, for all bin $B\in\cT_m$, we define the bad round with respect to $B$ as
\begin{align*}
s_{l,m}(B) = &\inf\left\{s>\tau_{l,m}\,:\,\sum_{(i,j)\in\mathcal{P}(B,s)}\sqrt{2^m (s^m_{i,j+1}(B) - s^m_{i,j}(B))}>c_{4}\log(T)\sqrt{2^m(s - \tau_{l,m})}\right\}\\
&\wedge (\tau_{l,m}+8^m)\,,
\end{align*}
where $c_4$ is a fixed positive numerical constant, and we define the set $\mathcal{P}(B,s)$ as
\begin{align*}
\mathcal{P}(B,s) =\big\{ (i,j)\,:\, i,j\in\Nat\,,\text{ such that }[s^m_{i,j}(B),s^m_{i,j+1}(B)]\text{ is a bad segment and }s^m_{i,j+1}(B)<s \big\}\,.
\end{align*}
\end{definition}
Now, let $[\tau_{l,m}, \tau_{l,m+1})$, be a block, and let $B \in \cT_m$. We distinguish two cases for this bin.
\textbf{\underline{Case 1:} the bad round happens after the end of the block $(s_{l,m}(B) = \tau_{l,m}+8^m)$.}
By applying the definition of $s_{l,m}(B)$ (\cref{{def:bad_round}}) directly, we have
\begin{align*}
\sum_{(i,j)\in\mathcal{P}(B,s_{l,m}(B))}\sqrt{2^m(s_{i,j+1}^m (B)-s_{i,j}^m(B))}&\leq c_4\log(T)\sqrt{8^m2^m}= c_4\log(T)4^m\,.
\end{align*}
\textbf{\underline{Case 2:} the bad round happens before the end of the block $(s_{l,m}(B) < \tau_{l,m}+8^m)$.} In this case, we claim the following result.
\begin{proposition}[\textbf{A well-timed replay at a suitable depth is triggered}]\label{prop:a_replay_occurs}
Let $[\tau_{l,m},\tau_{l,m+1}[$ be a block. We define $\mathcal{E}_2(\tau_{l,m})$ as the following event:
\begin{align*}
&\forall B \in \cT_m \text{ such that } s_{l,m}(B) < \tau_{l,m+1}, \text{ there exists a bad segment } [s_{i,j}^m(B), s_{i,j+1}^m(B)] \text{ such that } \\
&s_{i,j+1}^m(B) < s_{l,m}(B) \text{ and a } \underbrace{\texttt{Replay}(t_{\mathrm{start}}, d_{i,j})}_{\text{(\cref{def:replay_s_d})}} \text{ such that }t_{\mathrm{start}}\text{ and }d_{i,j}\text{ satisfy} \\
&t_{\mathrm{start}} \in [s_{i,j}^m(B), s_{i,j+1}^m(B)] \text{ and }8^{d_{i,j}+1} \leq s_{i,j+1}^m(B) - s_{i,j}^m(B) \leq 8^{d_{i,j}+2}\,.
\end{align*}
Then, $\mathcal{E}_2(\tau_{l,m})$ holds with probability at least $1-1/T^2$.
\end{proposition}
\begin{proof}[Proof of \cref{prop:a_replay_occurs}]
Let $[\tau_{l,m},\tau_{l,m+1}[$ be a block, and let $B\in\cT_m$ such that $s_{l,m}(B)<\tau_{l,m}+8^m$. Let $d_{i,j}$ the integer satisfying
\begin{align*}
8^{d_{i,j}+1}\leq s_{i,j+1}^m(B) - s_{i,j}^m(B)\leq 8^{d_{i,j}+2}\,.
\end{align*}
First, we remark that $R_{t,d_{i,j}}$, $s_{i,j}^m(B)$, $\tilde{s}_{i,j}^m(B)$ and $s_{l,m}(B)$ only depend on the fixed bin $B$, the starting round of the block $\tau_{l,m}$ $B_{t,m}^\sharp$ (which is deterministic), and the randomness of scheduling of the replays. Using Chernoff's bound over randomness of the algorithm conditionally on $\cF_{\tau_{l,m}}$ gives
\begin{align}\label{eq:temp_constante_8}
&\mathbb{P}\left( \sum_{(i, j)\in\cP(B,s_{l,m}(B))} \sum_{t=s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}R_{t,d_{ij}}\leq \frac{1}{2}\E{}{\sum_{(i,j)\in\cP(B,s_{l,m}(B))} \sum_{t=s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}R_{t,d_{ij}}\,\bigg|\, \cF_{\tau_{l,m}}} \,\bigg|\, \cF_{\tau_{l,m}} \right)\nonumber\\
&\leq \exp\left(-\frac{1}{8}\E{}{\sum_{(i,j)\in\cP(B,s_{l,m}(B))} \sum_{t=s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}R_{t,d_{ij}}\,\bigg|\, \cF_{\tau_{l,m}}}\right)\,.
\end{align}
Then, we lower bound the expectation of \eqref{eq:temp_constante_8},
\begin{align*}
&\E{}{\sum_{(i,j)\in\cP(B,s_{l,m}(B))} \sum_{t=s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}R_{t,d_{ij}}\,\bigg|\, \cF_{\tau_{l,m}}} \\
&= \E{}{ \sum_{(i,j)\in\cP(B,s_{l,m}(B))} \sum_{t=s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}\frac{\I{t-\tau_{l,m}\equiv 0[8^{d_{i,j}}]}}{\sqrt{t-\tau_{l,m}}}\sqrt{8^{d_{i,j}}} \,\bigg|\, \cF_{\tau_{l,m}}}\\
&\geq \frac{1}{8}\E{}{ \sum_{(i,j)\in\cP(B)} \frac{1}{\sqrt{\tilde{s}^m_{i,j}(B)-\tau_{l,m}}}\sqrt{8^{d_{i,j}+2}} \,\bigg|\, \cF_{\tau_{l,m}}}\\
&\geq \frac{1}{8}\E{}{ \sum_{(i,j)\in\cP(B)} \sqrt{\frac{s^m_{i,j+1}(B)-s^m_{i,j}(B)}{s_{l,m}(B)-\tau_{l,m}}} \,\bigg|\, \cF_{\tau_{l,m}}}\\
&\geq\frac{c_4}{8}\log(T)\,,
\end{align*}
where we use the fact that there is at least one round $t\in[s_{i,j}^m(B),\tilde{s}_{i,j}^m(B)]$ such that $t-\tau_{l,m}\equiv 0[8^{d_{ij}}]$, \emph{i.e.} that at least a replay at depth $d_{i,j}$ that can be scheduled in $[s_{i,j}^m(B),s_{i,j+1}(B)]$, and where we used \cref{def:bad_round} in the last inequality. Setting $\textcolor{Maroon}{c_4=192}$, we have
\begin{align*}
&\mathbb{P}\left( \sum_{(i, j)\in\cP(B,s_{l,m}(B))} \sum_{t=s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}R_{t,d_{ij}}\leq \frac{1}{2}\E{}{\sum_{(i,j)\in\cP(B,s_{l,m}(B))} \sum_{t=s^m_{i,j}(B)}^{\tilde{s}^m_{i,j}(B)}R_{t,d_{ij}}\,\bigg|\, \cF_{\tau_{l,m}}} \,\bigg|\, \cF_{\tau_{l,m}} \right)\\
&\leq 1/T^3\,.
\end{align*}
Taking a union bound with respect to all possible bins $B\in\cT_m$ (there are $2^m\leq T$ choices) concludes the proof.
\end{proof}
Therefore, using \cref{prop:a_replay_occurs}, for any bin $B\in\cT_m$, on event $\mathcal{E}_1\cap\mathcal{E}_2(\tau_{l,m})$, we have that there exists a bad segment $[s_{i,j_0}^m(B), s_{i,j_0+1}^m(B)]$ such that $s_{i,j_0+1}^m(B)<s_{l,m}(B)<\tau_{l,m+1}$, and there exists a $\texttt{Replay}(t_{\mathrm{start}},d_{i,j_0})$ starting within this segment, $t_{\mathrm{start}}\in[s_{i,j_0}^m(B), s_{i,j_0+1}^m(B)]$ and $d_{i,j_0}$ satisfying $8^{d_{i,j_0}}\leq s_{i,j_0+1}^m(B) -s_{i,j_0}^m(B)\leq 8^{d_{i,j_0}+2}$. By \cref{prop:bad_segments}, this implies that bin $B$ is evicted by round $s_{i,j_0+1}^m(B)$. Therefore, on $\mathcal{E}_1\cap\mathcal{E}_2(\tau_{l,m})$, using the definition of the bad round $s_{l,m}(B)<\tau_{l,m+1}$,
\begin{align*}
\sum_{(i,j)\in\mathcal{P}(B, s_{l,m}(B))}\sqrt{2^m (s^m_{i,j+1}(B) - s^m_{i,j}(B))}&\leq c_4\log(T)\sqrt{2^m(s_{i,j_0} - \tau_{l,m})}\\
&\leq c_4\log(T) 4^m\,.
\end{align*}
Since \cref{prop:a_replay_occurs} holds uniformly over all bins $B\in\cT_m$, it also holds for the particular choice $B = \Blast_{l,m}$, and hence in both cases $s_{l,m}(B) = \tau_{l,m}+8^m$ and $s_{l,m}(B)<\tau_{l,m}+8^m$, we have
\begin{align*}
\sum_{(i,j)\in\mathcal{P}(\Blast_{l,m}, s_{l,m}(\Blast_{l,m}))}\sqrt{2^m (s^m_{i,j+1}(\Blast_{l,m}) - s^m_{i,j}(\Blast_{l,m}))} \leq c_4\log(T) 4^m\,.
\end{align*}
Now recall by \cref{def:bad_segment} that $s_{i,j_0+1}^m(\Blast_{l,m})$ is the \emph{earliest} round satisfying the lower bound on the cumulative regret
\begin{align*}
\sum_{t=s^m_{i,j_0}(\Blast_{l,m})}^{s^m_{i,j_0+1}(\Blast_{l,m})}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\geq c_3\log(T) \sqrt{2^m\left(s^m_{i,j_0}(\Blast_{l,m}) - s^m_{i,j_0-1}(\Blast_{l,m})\right)}\,,
\end{align*}
Therefore, up to round $s^m_{i,j_0+1}(\Blast_{l,m})-1$ included, the cumulative regret can be \emph{upper bounded} as
\begin{align*}
\sum_{t=s^m_{i,j_0}(\Blast_{l,m})}^{s^m_{i,j_0+1}(\Blast_{l,m})-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\leq c_3\log(T) \sqrt{2^m\left(s^m_{i,j_0}(\Blast_{l,m}) - s^m_{i,j_0-1}(\Blast_{l,m})-1\right)}\,.
\end{align*}
Since on event $\mathcal{E}_1\cap\mathcal{E}_2(\tau_{l,m})$, $\Blast_{l,m}$ is evicted by round $s_{i,j_0+1}(\Blast_{l,m})$, we can upper bound the cumulative regret contribution of $\Blast_{l,m}$ with respect to $B_{t,m}^\sharp$ over all bad segments of $\Blast_{l,m}$,
\begin{align}\label{eq:regret_bound_over_bad_segments}
\sum_{(i,j)\in\mathcal{P}(\Blast_{l,m}, s_{l,m}(\Blast_{l,m}))}\sum_{t=s_{i,j}^m(\Blast_{l,m})}^{s_{i,j+1}^m(\Blast_{l,m})}\delta_t(B_{t, m}^\sharp,\Blast_{l,m})\leq c_3 c_4\log^2(T)4^m\,.
\end{align}
It remains to bound the regret incurred over the \emph{non-bad segments}. To this end, we observe from \cref{def:bad_segment} that each phase contains \emph{at most one such segment}. Therefore, there exists a positive numerical constant $c_5$ such that
\begin{align}\label{eq:regret_contribution_over_non_bad_segments}
\sum_{I\subseteq [\tau_{l,m},\tau_{l,m+1}[\text{ s.t. }I\text{ is non-bad segment}}\,\sum_{t\in I}\,\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\leq c_5\log(T)4^m\,.
\end{align}
Summing \cref{eq:regret_bound_over_bad_segments,eq:regret_contribution_over_non_bad_segments} gives the total relative regret of $\Blast_{l,m}$ with respect to $B_{t,m}^\sharp$ over the block on event $\mathcal{E}_1\cap\mathcal{E}_2(\tau_{l,m})$,
\begin{align*}
\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m}) \leq \overbrace{\left(c_3 c_4 + c_5\right)}^{\textcolor{Maroon}{=c_6}}\log^2(T)4^m\,.
\end{align*}
Finally, we use the fact that event $\mathcal{E}_2(\tau_{l,m})$ holds with probability at least $1-1/T^2$ (\cref{prop:a_replay_occurs}),
\begin{align*}
\textcolor{blue}{(B)} &= \E{}{\I{\mathcal{E}_1}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\condicFgros}\\
&=\E{}{\I{\mathcal{E}_1\cap\mathcal{E}_2(\tau_{l,m})}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\condicFgros}\\
&+ \E{}{\I{\mathcal{E}_1\cap\mathcal{E}^c_2(\tau_{l,m})}\sum_{t=\tau_{l,m}}^{\tau_{l,m+1}-1}\delta_t(B_{t,m}^\sharp, \Blast_{l,m})\condicFgros}\\
&\leq c_6\log^2(T) 4^m +\frac{8^m}{T^2}\\
&\leq c_6\log^2(T) 4^m + \frac{1}{T}\,,
\end{align*}
where we used the fact that $\I{\mathcal{E}_1\cap\mathcal{E}^c_2(\tau_{l,m})}\leq \I{\mathcal{E}_2^c}$.
Choosing $\textcolor{Maroon}{c_B = c_6 + 1}$ concludes the proof.
\subsection{Conclusion of the proof}
Recall from \cref{eq:summary} that we have
\begin{align*}
\E{}{R(\pi_{\algo}, T)}&\leq \log(T) \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3} +\frac{7}{T}
\nonumber\\
&+ 6 \sum_{l=1}^T \E{}{\I{\mathcal{E}_1}(t_{l+1} - t_l)^{2/3}}+ \sum_{l=1}^T \E{}{\I{\mathcal{E}_1}\sum_{m=0}^{M_l} \sumblock \delta_t(B_{t,m}^\sharp, B_{t,m})} \,.
\end{align*}
\cref{prop:upper_bound_term_A,prop:upper_bound_term_B} prove that the last sum is upper bounded as
\begin{align*}
\E{}{\I{\mathcal{E}_1}\sum_{m=0}^{M_l}\sumblock \delta_t(B_{t,m}^\sharp,B_{t,m})}&\leq (c_A+c_B)\log^2(T)\E{}{\sum_{m=0}^{M_l}4^m}\\
&\leq 6(c_A+c_B)\log^2(T)\E{}{(t_{l+1}-t_l)^{2/3}}\\
&\leq 6(c_A+c_B)\log^2(T)\E{}{\I{\mathcal{E}_1}(t_{l+1}-t_l)^{2/3}} + \frac{c_A+c_B}{T^2} \,.
\end{align*}
where we used \cref{eq:bias_term} for the second inequality, and where we reconditioned on event $\mathcal{E}_1$ in the last line.
Therefore,
\begin{align}\label{eq:final_equation}
\E{}{R(\pi_{\algo}, T)}&\leq \log(T) \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3} +\frac{7+ c_A + c_B}{T}\nonumber\\
&+ 6\E{}{\I{\mathcal{E}_1}\sum_{l=1}^T (t_{l+1}-t_l)^{2/3}}+ 6(c_A+c_B)\log^2(T) \E{}{\I{\mathcal{E}_1}\sum_{l=1}^T(t_{l+1}-t_l)^{2/3}}\,.
\end{align}
We conclude the proof by relating episodes and phases, which will permit to bound the final term
\begin{align*}
\E{}{\I{\mathcal{E}_1}\sum_{l=1}^T(t_{l+1}-t_l)^{2/3}}\,.
\end{align*}
\begin{definition}[\textbf{Phases intersecting episode $l$}]
We define the phases intersecting episode $[t_l,t_{l+1}[$ as
\begin{align*}
\Phases = \left\{ i \in [\tilde{L}_T] \;:\; [\tau_i, \tau_{i+1}[ \,\cap\, [t_l, t_{l+1}[ \,\neq\, \emptyset \right\} \,.
\end{align*}
\end{definition}
With this definition, we can rewrite the upper bound on cumulative regret over each episode as
\begin{align*}
\E{}{\I{\mathcal{E}_1}\sum_{l=1}^T(t_{l+1}-t_l)^{2/3} } \leq \E{}{\I{\mathcal{E}_1}\sum_{l=1}^T \sum_{i\in\Phases}(\tau_{i+1}-\tau_i)^{2/3}}\,.
\end{align*}
Recall that \cref{prop:relating_phase_and_episode} shows that on $\mathcal{E}_1$, each phase $[\tau_i,\tau_{i+1}[$ intersects at most two episodes. Therefore,
\begin{align*}
\E{}{\I{\mathcal{E}_1}\sum_{l=1}^T \sum_{i\in\Phases}(\tau_{i+1}-\tau_i)^{2/3}} \leq 2 \sum_{i=1}^{\tilde{L}_T}(\tau_{i+1}-\tau_i)^{2/3}\,,
\end{align*}
and finally we can conclude the proof by plugging the bound above into \eqref{eq:final_equation},
\begin{align*}
\E{}{R(\pi_{\algo}, T)}&\leq \log(T) \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3} +\frac{7+ c_A + c_B}{T}\nonumber\\
&+ 12\sum_{i=1}^{\tilde{L}_T}(\tau_{i+1}-\tau_i)^{2/3}+ 12(c_A+c_B)\log^2(T) \sum_{i=1}^{\tilde{L}_T}(\tau_{i+1}-\tau_i)^{2/3}\\
&\leq \left(\log(T)+12+12(c_A+c_B)\log^2(T)\right)\sum_{i=1}^{\tilde{L}_T}(\tau_{i+1}-\tau_i)^{2/3}+\frac{7+c_A+c_B}{T}\\
&\leq \bar{c_1}\log^2(T)\sum_{i=1}^{\tilde{L}_T}(\tau_{i+1}-\tau_i)^{2/3}\,,
\end{align*}
where $\textcolor{Maroon}{\bar{c_1} = 12(c_A+c_B)+2}$. This completes the proof. We summarize in \cref{tab:pure_constants} the exact values of the numerical constants used in the proof.
\begin{table}[h]
\centering
\begin{tabular}{c|c}
\toprule
\textbf{Constant} & \textbf{Numerical value} \\ \midrule
$c_0$ & $3 + 7(e-1)\sqrt{2}$ \\
$c_1$ & $7(e-1)\sqrt{2}$\\
$c_2$ & $c_0+c_1+8$ \\
$c_3$ & $140+4c_1$\\
$c_4$ & $192$\\
$c_5$ & \cref{eq:regret_contribution_over_non_bad_segments}\\
$c_6$ & $c_3 c_4+c_5$\\
$c_A$ & $18(c_0+c_1+10)$\\
$c_B$ & $c_6+1$\\
$\Bar{c_1}$ & $12(c_A+c_B)+2$
\end{tabular}
\caption{Summary of numerical constants used in the proof}
\label{tab:pure_constants}
\end{table}
\section{Proof of \texorpdfstring{\cref{cor:upper_bound_total_variation}}{Lg}}\label{app:sec:proof_corollary_V_T}
Recall the definition of the total variation over the horizon $T$:
\begin{align*}
V_T = \sum_{t=1}^{T-1} \max_{x \in [0, 1]} |\mu_{t+1}(x) - \mu_t(x)|\,.
\end{align*}
Fix a phase $[\tau_i, \tau_{i+1}[$, where $\tau_{i+1} \leq T+1$. Define the total variation within this phase as
\begin{align*}
V_{[\tau_i, \tau_{i+1}[} = \sum_{t=\tau_i}^{\tau_{i+1}-1} \max_{x \in [0, 1]} |\mu_{t+1}(x) - \mu_t(x)|\,.
\end{align*}
Let $x_i = \argmax_{x \in [0, 1]} \mu_{\tau_{i+1}}(x)$ denote the best arm at the end of phase $i$. By definition, there exists an interval $[s_1, s_2] \subset [\tau_i, \tau_{i+1}[$ over which $x_i$ suffers significant regret:
\begin{align*}
\sum_{t=s_1}^{s_2} \delta_t(x_i) \geq \log(T)\, (s_2 - s_1)^{2/3}.
\end{align*}
Now observe that
\begin{align*}
(s_2 - s_1)^{2/3} = \sum_{t=s_1}^{s_2} \frac{1}{(s_2 - s_1)^{1/3}} \geq \sum_{t=s_1}^{s_2} \frac{1}{(\tau_{i+1} - \tau_i)^{1/3}}\,,
\end{align*}
which implies
\begin{align*}
\sum_{t=s_1}^{s_2} \delta_t(x_i) \geq \sum_{t=s_1}^{s_2} \frac{\log(T)}{(\tau_{i+1} - \tau_i)^{1/3}}\,.
\end{align*}
Hence, there exists at least one round $t_i \in [s_1, s_2]$ such that:
\begin{align*}
\delta_{t_i}(x_i) \geq \frac{\log(T)}{(\tau_{i+1} - \tau_i)^{1/3}}\,.
\end{align*}
Define $\tilde{x}_i = \argmax_{x \in [0, 1]} \mu_{t_i}(x)$. Then:
\begin{align*}
\delta_{t_i}(x_i) &\leq \mu_{t_i}(\tilde{x}_i) - \mu_{t_i}(x_i) + \underbrace{\mu_{\tau_{i+1}}(x_i) - \mu_{\tau_{i+1}}(\tilde{x}_i)}_{\geq 0} \\
&\leq |\mu_{\tau_{i+1}}(x_i) - \mu_{t_i}(x_i)| + |\mu_{t_i}(\tilde{x}_i) - \mu_{\tau_{i+1}}(\tilde{x}_i)| \\
&\leq 2 V_{[\tau_i, \tau_{i+1})}\,.
\end{align*}
Therefore, assuming $\log(T) \geq 1$, we conclude that
\begin{align}\label{eq:relating_phase_and_TV}
\frac{1}{(\tau_{i+1} - \tau_i)^{1/3}} \leq 2 V_{[\tau_i, \tau_{i+1})}\,.
\end{align}
We now recall the upper bound from \cref{th:upper_bound}:
\begin{align*}
\mathbb{E}\left[R(\pi_{\algo}, T)\right] \leq \bar{c} \log^2(T) \sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3}\,.
\end{align*}
We decompose the sum as:
\begin{align*}
\sum_{i=0}^{\tilde{L}_T} (\tau_{i+1} - \tau_i)^{2/3} &= (T+1 - \tau_{\tilde{L}_T})^{2/3} + \sum_{i=0}^{\tilde{L}_T - 1} (\tau_{i+1} - \tau_i)^{2/3} \leq T^{2/3} + \sum_{i=0}^{\tilde{L}_T - 1} (\tau_{i+1} - \tau_i)^{2/3}\,.
\end{align*}
Applying Hölder’s inequality gives
\begin{align*}
\sum_{i=0}^{\tilde{L}_T - 1} (\tau_{i+1} - \tau_i)^{2/3}
&= \sum_{i=0}^{\tilde{L}_T - 1} \frac{1}{(\tau_{i+1} - \tau_i)^{1/12}} \, (\tau_{i+1} - \tau_i)^{3/4} \\
&\leq \left( \sum_{i=0}^{\tilde{L}_T - 1} \frac{1}{(\tau_{i+1} - \tau_i)^{1/3}} \right)^{1/4} \left( \sum_{i=0}^{\tilde{L}_T - 1} (\tau_{i+1} - \tau_i) \right)^{3/4}\,.
\end{align*}
Finally, by inequality \eqref{eq:relating_phase_and_TV}, we have
\begin{align*}
\sum_{i=0}^{\tilde{L}_T - 1} \frac{1}{(\tau_{i+1} - \tau_i)^{1/3}} \leq 2 \sum_{i=0}^{\tilde{L}_T - 1} V_{[\tau_i, \tau_{i+1})} \leq 2 V_T\,.
\end{align*}
Therefore,
\begin{align*}
\sum_{i=0}^{\tilde{L}_T - 1} (\tau_{i+1} - \tau_i)^{2/3}
\leq (2 V_T)^{1/4} \times T^{3/4} = 2^{1/4} V_T^{1/4} T^{3/4}\,.
\end{align*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proof of the lower bound (\texorpdfstring{\cref{th:lower_bound}}{Lg})}\label{app:sec:proof_lower_bound}
\textbf{Proof sketch.}
The proof of \cref{th:lower_bound} relies on the following ideas. First, note that the lower bound $T^{2/3}$ follows from classical results of Lipschitz bandits \citep{kleinberg2004nearly}. To prove the lower bound either with respect to total variation $V_T$, or with respect to the total number of changes $L_T$, we construct $T/\tau$ stationary sub-problems with horizon $\tau$ to be chosen later. For each problem, we design $K = \tau^{1/3}$ possible Lipschitz-continuous rewards function. Each of these functions is almost flat, with the exception of one bump of size $\epsilon \approx \tau^{-1/3}$ hidden among $K$ possible bins. Classical arguments show that any bandit algorithm will misidentify the optimal bin with constant probability in some problem instance, thus suffering a regret $\tau^{2/3}$. Then, we create a dynamic problem instance by concatenating $T/\tau$ such problems, where the reward function corresponding to each problem of horizon $\tau$ is chosen independently. The total regret of any algorithm must then be
\begin{align*}
\frac{T}{\tau} \tau^{2/3} = T\tau^{-1/3}\,.
\end{align*}
To obtain a lower bound depending on the number of reward changes $L_T$, we set $\tau = T/L$, so that there are exactly $T/\tau$ stationary sub-problems. This yields a regret of the order $L_T^{1/3}T^{2/3}$. To obtain a lower bound depending on the total variation $V_T$, we set
$\tau = (T/V_T)^{3/4}$. Then, we see that there are $T/\tau$ changes of magnitude $\tau^{-1/3}$ each, so that the total variation is indeed $T/\tau \times \tau^{-1/3} = V_T$; moreover the cumulative regret over all problems is $T\tau^{-1/3} = T^{3/4}V^{1/4}$.
In the following, we assume that, conditionally on $x_t$, the reward is a Bernoulli random variable with mean $\mu_t(x_t)$.
\textbf{Notations.} For some integer $K\geq 2$ to be specified later, we set $\tau = K^3$ and $\epsilon = 1/(2K)$ (note that $\epsilon \leq 1/4$). We define $L = \lfloor T/\tau \rfloor$, and for $l\in \llbracket L\rrbracket$, we define the stationary phase $\mathcal{P}^l = \left\{(l-1)\tau + 1,\dots, l\tau - 1\right\}$. We also define the last stationary phase $\mathcal{P} = \left\{\tau L, \dots, T\right\}$. For $k < K$, define the intervals $I_k = [\frac{k-1}{K}, \frac{k}{K}[$, and we define $I_K = [\frac{K-1}{K}, 1]$. Finally, for a given sequence of rewards $\mu$, we denote by $\mathbb{E}_\mu$ and $\mathbb{P}_{\mu}$ the expectation and probability when the reward sequence is $\mu$. Note that at each round $t$, the algorithm $\pi$ only depends on an internal randomisation and on $\mathcal{F}_{t-1}$-measurable events. Thus, for all $l \leq L$ and all $\mathcal{F}_{\tau l}$-measurable random variable $X$ and event $E$, $\E{\mu}{X}$ and $\mathbb{P}_{\mu}(E)$ only depend on the sequence of rewards up to phase $l$, \emph{i.e.} on $(\mu_t)_{t \leq \tau l}$. In the following, we therefore abuse notation, and define for reward sequences $\mu$ of length $\tau l$, the expectation $\mathbb{E}_\mu$ and probability $\mathbb{P}_{\mu}$ pertaining to the first $l$ phases, \emph{i.e.} to $\mathcal{F}_{\tau l}$-measurable random variables and events.
\textbf{Reward function for a stationary phase.}
We define the possible choice of reward function during one stationary phase $\mathcal{P}^l$. For each $k\in \{2, ..., K\}$, define the function $m^k : [0,1]\mapsto[0,1]$ as
$$m^k(x) = \frac{1 - \epsilon}{2} + \frac{\epsilon - \left\vert x - \frac{1}{2}\right\vert}{2}\mathds{1}\left\{x \in I_1\right\} + \left(\epsilon - \left\vert x - \frac{2k-1}{2K}\right\vert\right)\mathds{1}\left\{x \in I_k\right\}.$$
We also define
$$m^1(x) = \frac{1 - \epsilon}{2} + \frac{\epsilon - \left\vert x - \frac{1}{2}\right\vert}{2}\mathds{1}\left\{x \in I_1\right\}.$$
Note that for $k \in \llbracket K \rrbracket$, when the mean reward function is $m^k$ the optimal action is $\frac{2k-1}{K}$; moreover any action chosen outside of interval $I_k$ will have an instantaneous regret at least $\frac{\epsilon}{2}$. For $l\in \llbracket L \rrbracket$ and $k\in \llbracket K \rrbracket$, we denote $N_k^l = \sum_{t \in \mathcal{P}^l} \mathds{1}\left\{x_t \in I_k\right\}$ the number of actions chosen in interval $I_k$ during the stationary phase $\mathcal{P}^l$. Examples of possible reward functions for $K = 5$ are given in Figure \ref{fig:reward_functions}.
\begin{figure}[!ht]
\centering
\begin{subfigure}{.45\textwidth}
\centering
\includegraphics[width=\linewidth]{Figures/Lipschitz1.png}
\label{fig:exp1_power}
\end{subfigure}%
\hfill
\begin{subfigure}{.45\textwidth}
\centering
\includegraphics[width=\linewidth]{Figures/Lipschitz3.png}
\label{fig:exp2_power}
\end{subfigure}\\
\caption{Reward functions $m^1$ (left) and $m^3$ (right) when $K = 5$.}
\vspace{-3mm}
\label{fig:reward_functions}
\end{figure}
\textbf{Sequential construction of the non-stationary reward function.}
For a given algorithm $\pi$, we build a difficult non-stationary reward function sequentially. We show by induction that for all $l \leq L$, there exists a non-stationary sequence of reward functions $(\mu_t)_{t \leq l\tau}$ such that the expected cumulative regret of algorithm $\pi$ over the stationary phases $\mathcal{P}_1$, ..., $\mathcal{P}_{l}$ is at least $0.01K^2 l$ for all $l \in \llbracket L \rrbracket$. We abuse notation and define $\mathcal{P}_0 = \emptyset$, so that the base case of the recursion (corresponding to $l = 0$) becomes trivial.
We then proceed to prove this statement by induction. Assume that the statement holds for some $l$ such that $0 \leq l < L$ and some sequence of reward functions $(\mu_t)_{t \leq l\tau}$. For $k \leq K$, we define $\mu^k$ as the extended sequence of rewards such that $\mu^k_t = \mu_t \mathds{1}_{t \leq l\tau} + m^k\mathds{1}_{l\tau < t \leq (l+1)\tau}$. Then, let $k^*$ be defined as
$$k^* = \argmin_{k}\mathbb{E}_{\mu^1}[N_k^{l+1}].$$
It holds that $\mathbb{E}_{\mu^1}[N_{k^*}^{l+1}] \leq \frac{\tau}{K}$.
Now, if $k^* = 1$, it means that under $\mu^1$, the algorithm $\pi$ chooses in expectation at least $\frac{(K-1)\tau}{K}$ actions outside of interval $I_1$ in phase $l+1$. Then, the regret in this phase must be at least $\frac{(K-1)\tau}{K}\times \frac{\epsilon}{4} \geq \frac{\tau \epsilon}{8} = K^{2}/16$. In this case, using the assumption that the expected cumulated regret of algorithm $\pi$ over the stationary phases $\mathcal{P}_1$, ..., $\mathcal{P}_{l}$ is at least $0.01 K^2 l$, we find that the cumulative rewards up to phase $\mathcal{P}_{l+1}$ is at least $0.01K^2 (l+1)$, thus completing the induction step. We therefore assume henceforth that $k^* \neq 1$. Now, denoting $\KL\left(\mathbb{P}, \mathbb{Q}\right)$ the Kullback-Leibler divergence between probabilities $\mathbb{P}$ and $\mathbb{Q}$, and by $kl(p,q)$ the Kullback-Leibler divergence between Bernoulli distributions with parameters $p$ and $q$, classical arguments (see, \emph{e.g.}, \cite[Chapter 15.1]{lattimore2020bandit}) show that
\begin{align*}
\KL\left(\mathbb{P}_{\mu^1}, \mathbb{P}_{\mu^{k^*}}\right) &= \sum_{\tau l < t\leq\tau(l+1)} \mathbb{E}_{\mu^1}\left[kl\left(m^1\left(x_t\right), m^{k^*}\left(x_t\right)\right)\right].
\end{align*}
Now, for $x_t \notin I_{k^*}$, $kl\left(m^1\left(x_t\right), m^{k^*}\left(x_t\right)\right)=0$. Moreover, for all $x_t \in [0,1]$, $kl\left(m^1\left(x_t\right), m^{k^*}\left(x_t\right)\right)\leq kl\left(m^1\left(\frac{2k^*-1}{K}\right), m^{k^*}\left(\frac{2k^*-1}{K}\right)\right)$. Thus,
\begin{align*}
\KL\left(\mathbb{P}_{\mu^1}, \mathbb{P}_{\mu^{k^*}}\right) &\leq \sum_{\tau l < t\leq\tau(l+1)} \mathbb{E}_{\mu^1}\left[\mathds{1}\{x_t \in I_{k^*}\}kl\left(m^1\left(\frac{2k^*-1}{K}\right), m^{k^*}\left(\frac{2k^*-1}{K}\right)\right)\right]\\
&\leq \mathbb{E}_{\mu^1}\left[N_{k^*}^{l+1}\right]kl\left(\frac{1-\epsilon}{2}, \frac{1+\epsilon}{2}\right)\\
&\leq \frac{\tau}{K} 4\epsilon^2\\
&\leq\frac{4K^3}{K\times 4K^2} = 1
\end{align*}
where we have used the definition of $k^*$ and of $\tau$ and $\epsilon$. Now, we apply Bretagnolle-Huber Inequality (see, \emph{e.g.}, \cite[Chapter 14]{lattimore2020bandit}) to the $\mathcal{F}_{\tau(l+1)}$-measurable event $A = \{ N_1^{l+1} \leq \frac{\tau}{2}\}$. This yields
\begin{align*}
\mathbb{P}_{\mu_1}\left(A\right) + \mathbb{P}_{\mu_{k^*}}\left(A^c\right) &\geq \frac{1}{2}\exp\left(-\KL\left(\mathbb{P}_{\mu^1}, \mathbb{P}_{\mu^{k^*}}\right)\right)\\
&\geq \frac{1}{2}\exp\left(-1\right).
\end{align*}
Using $\frac{1}{2}\exp\left(-1\right) \geq 0.18$ and $2\max(a, b) \geq a+b$, we find that
\begin{align*}
\max\left\{\mathbb{P}_{\mu_1}\left(A\right), \mathbb{P}_{\mu_{k^*}}\left(A^c\right)\right\} &\geq 0.09.
\end{align*}
Assume that $\mathbb{P}_{\mu_1}\left(A\right) \geq 0.09$. On the event $A$, the regret on phase $l$ under reward sequence $\mu^1$ is at least $\nicefrac{\tau}{2}\times \nicefrac{\epsilon}{2} = K^{2}/8$. Thus, the expected regret on phase $l$ under reward sequence $\mu^1$ is at least $0.01K^2$. Similar arguments show that if $\mathbb{P}_{\mu_{k^*}}\left(A^c\right) \geq 0.09$, the expected regret on phase $l$ under reward sequence $\mu^{k^*}$ is at least $0.01K^2$. Using the induction assumption, we find that the cumulative rewards up to phase $\mathcal{P}_{l+1}$ is at least $0.01K^2 (l+1)$, thus completing the induction step.
\textbf{Conclusion.}
The induction above shows that for all $K\geq 2$, there exists a sequence of reward functions $\mu_t$ with a most $\lfloor T/K^3 \rfloor$ shifts such that the total cumulative regret of algorithm $\pi$ is at least $0.01\lfloor T/K^{3} \rfloor K^2$. Assuming moreover that $K < (T/2)^{1/3}$, we see that $\lfloor T/K^3 \rfloor \geq T/(2K^3)$, so that the regret is at least $0.005T/K$. We now consider the two lower bounds separately.
\textbf{Lower bound depending on $L_T$.}
For $L_T \in \{0, 1, 2\}$, the bound follows from classical results on Lipschitz-continuous bandits (see, \emph{e.g.}, \cite{kleinberg2004nearly}), by noticing that since there exists a stationary problem and a constant $c>0$ on which any bandit algorithm must have regret at least $cT^{2/3}$, there exists a problem with 0 shifts (\emph{i.e.}, at most $L_T$) such that the regret is at least $cL^{1/3}T^{2/3}/2$. Therefore, we assume that $L_T > 2$. Then, setting $K = \lfloor (T/L_T)^{1/3} \rfloor$, we see that $K^3\leq T/2$. Thus, the non-stationary reward function designed above has regret at least $0.005T/K \geq 0.005T^{2/3}L_T^{1/3}$. Moreover, this reward function has at most $L_T$ changes.
\textbf{Lower bound depending on $V_T$.}
For $V_T \leq 2T^{-1/3}$, we have $V_T^{1/4}T^{3/4} \leq 2T^{2/3}$, the result follows from classical results on Lipschitz-continuous bandits (see, \emph{e.g.}, \cite{kleinberg2004nearly}) by noticing that there exists a stationary environment and a constant $c>0$ on which any bandit algorithm must have regret at least $cT^{2/3}$. Therefore, we assume henceforth that $V_T > T^{-1/3}$. Then, setting $K = \lfloor (T/V_T)^{1/4} \rfloor$, we see that $K^3\leq T/2$. Thus, the non-stationary reward function designed above has regret at least $0.005T/K \geq 0.005T^{3/4}V_T^{1/4}$. Moreover, the cumulative variation of this reward function is at most $V_T$.
\section{Extensions}\label{app:sec:extension_arbitrary_spaces}
\subsection{Extension to multi-dimensional action space}
In this subsection, we discuss how to extend our algorithm and results to multi-dimensional spaces.
\textbf{Problem setting in dimension $p$.}
We assume that at each round, the player chooses an action in the set $[0,1]^p$, where $p \in \mathbb{N}_*$ is the dimension. We assume that each mean reward function $\mu_t : [0, 1]^p\to [0, 1]$ satisfies the following Lipschitz condition:
$$\forall x, x'\in[0, 1]^p,\, |\mu_t(x') - \mu_t(x)|\leq \frac{1}{\sqrt{p}}\lVert x - x'\rVert_p,$$
where $\lVert \cdot \rVert_p$ denotes the Euclidean norm in dimension $p$.
In dimension $p$, classical results show that when the horizon is $T$ and the problem is stationary, in the worst case a minimax-optimal algorithm can learn the reward function up to an error of order $\log(T)T^{-\frac{1}{p+2}}$, and incurs a regret of order $\log(T)T^{\frac{p+1}{p+2}}$. This justifies extending the notion of significant regret to $p$-dimensional action space as follows. We say that an arm $x\in [0, 1]^p$ incurs significant regret on interval $[s_1, s_2]$ if its cumulative regret on this interval is lower bounded as
\begin{align}\label{eq:significant_p}
\sum_{t=s_1}^{s_2}\delta_t(x)\geq \log(T)(s_2-s_1)^{\frac{p+1}{p+2}}\,.
\end{align}
Then, based on this definition, we can define the \emph{significant shifts} and \emph{significant phases} the same as in \cref{def:significant_shift}.
\textbf{Adapting the algorithm.}
To estimate the mean reward function in dimension $p$, we rely on a recursive $p$-adic partitioning of the action space. More precisely, we can define \textbf{\textit{p}-adic tree} $\cT = \{\cT_d\}_{d\in\Nat}$ as the hierarchy of nested partitions of $[0, 1]^p$ at all possible depth $d\in\Nat$, where $\cT_d$ denotes the partition of $[0, 1]^p$ into bins (\emph{i.e.} hypercubes) with side length $1/2^d$.
We also adapt our doubling trick, considering \textbf{blocks} of duration $\tau_{l,m+1} - \tau_{l, m} = 2^{m(p+2)}$. Then, on the $m$-th block of the $l$-th episode, we consider as the \textsc{MASTER} set the bins at depths $m$. If the rewards are stable enough during this round, we expect the regret of the block to be of order $2^{m(p+1)}$.
We can use the same \textbf{sampling strategy} as presented in Section \ref{sec:algorithm}: given a set of active depth at time $t$, we sample a bin uniformly at the shallowest active depth, and then proceed by sampling uniformly bins among its active children at the next depth. An analysis similar to that of Proposition \ref{prop:concentration} reveals that the probability of sampling a bin at depth $d$ (given that this bin is active) is lower bounded by $2^{-pd}$. Then, the error for estimating the cumulative gaps between two bins at depth $d$ active between rounds $s_1$ and $s_2$ must be of order $\sqrt{(s_2-s_1)2^{pd} \vee 4^{pd}} + \frac{s_2-s_1}{2^d}$.
This motivates our considering an new \textbf{eviction criteria} of the form
\begin{align*}
\max_{B' \in \cB_{[s_1, s_2]}(d)} \sum_{t=s_1}^{s_2} \hat{\delta}_t(B', B)
> c'_0 \left(\log(T) \sqrt{(s_2 - s_1) 2^{pd} \vee 4^{pd}} + \frac{(s_2 - s_1)}{2^d}\right)
\end{align*}
for some well-chosen positive numerical constant $c_0'$.
As previously, we choose as duration of a replay at depth $d$ the length of the $d$-th block of an episode. In the $p$-dimensional setting, this implies that we should schedule \textbf{replays of duration $2^{d(p+2)}$ at depth $d$}. Note that such a replay allows to estimate the cumulative gaps of two bins up to a precision $2^{d(p+1)}$.
It remains to choose the \textbf{probability of scheduling a replay} at depth $d$. At all times $s \in [\tau_{l,m}, \tau_{l,+1}[$ such that $t - \tau_{l,m} \equiv 0 \ [2^{d(p+2)}]$, we schedule a replay of duration $d<$ with probability $p_{s,d}$ given by
$$p_{s,d} = \sqrt{\frac{2^{d(p+2)}}{s-\tau_{l,m}}}.$$
\textbf{Regret analysis.}
Simple computations show that there are on average $\sqrt{2^{m(p+2)}/2^{d(p+2)}}$ replays of length $2^{d(p+2)}$ during a block of length $2^{m(p+2)}$, each one with regret of order $2^{d(p+1)}$. Thus, the regret due to replays at depth $d$ over this block is of order $\sqrt{2^{m(p+2)}/2^{d(p+2)}}2^{d(p+1)} = \sqrt{2^{m(p+2)}2^{dp}}$. Summing over the depths $d<m$, we find that the total contribution of the replays over a block of length $2^{m(p+2)}$ is of order $2^{m(p+1)}$, \emph{i.e.} of the same order as the minimax-optimal regret over this phase in the stationary setting.
To conclude, we argue that this choice of replay probability allows to detect significant shifts fast enough. Assume that the mean reward of an optimal arm undergoes a shift of magnitude $2^{-dp}$, so that it becomes sub-optimal. The algorithm should identify it as unsafe in less than $D$ rounds, where $D$ is such that $D2^{-dp} \leq 2^{m(p+1)}$, \emph{i.e.}, $D \leq 2^{m(p+1)+dp}$. Now, our choice of $p_{s,d}$ ensures that there are approximately $\sqrt{2^{m(p+2)}/2^{d(p+2)}}$ replays at depth $d$ scheduled during the block, so that on average, a replay at depth $d$ is scheduled every $\sqrt{2^{m(p+2)}2^{d(p+2)}}$ rounds. Noticing that $\sqrt{2^{m(p+2)}2^{d(p+2)}} \leq 2^{m(p+1)+dp}$, we see that enough replays of the adequate length are schedule, so to ensure that a significant shift does not go undetected for too long.
Thus, the algorithm has almost \emph{minimax-optimal} regret over stable phases, and detects quickly significant shifts. Conducting the same analysis as in the 1-dimensional case, we see that in dimension $p$ the regret of our algorithm should be bounded as
\begin{align*}
\mathbb{E}[R(\pi_{\algo}, T)] \leq c\log(T)^2\sum_{i = 1}^{\tilde{L}_T} (\tau_{i+1}(\mathbf{\mu}) - \tau_i(\mathbf{\mu}))^{\frac{p+1}{p+2}}\,,
\end{align*}
which yields the \emph{worst-case regret}
\begin{align*}
\mathbb{E}[R(\pi_{\algo}, T)] \leq c'\log(T)^2\tilde{L}_T^{\frac{1}{p+2}}T^{\frac{p+1}{p+2}}\,,
\end{align*}
for some positive numerical constant $c'$. We emphasize here that both the phases $\tau_i(\mu)$ and number of phases $\tilde{L}_T$ are based on the definition of a significant shift for $p$-dimensional actions, given in Equation \eqref{eq:significant_p}.
\subsection{Extension to Hölder bandits}
In this subsection, we discuss how to extend our algorithm and results to Hölder bandits.
\begin{assumption}[$(\kappa, \beta)$\textbf{-Hölder mean reward functions}]\label{assump:holder}
Each mean reward function $\mu_t$ satisfies $(\beta\leq1)$
\begin{align*}
\forall x, x'\in[0, 1],\,\quad \lvert \mu_t(x)-\mu_t(x') \rvert\leq \kappa\lvert x-x' \rvert^\beta\,.
\end{align*}
\end{assumption}
\textbf{Problem setting for Hölder bandits.} We start by defining a significant regret in this setting. Arm $x\in[0, 1]$ incurs significant regret on interval $[s_1, s_2]$ if
\begin{align*}
\sum_{t=s_1}^{s_2}\delta_t(x)\geq\log(T)(s_2-s_1)^{\frac{1+\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}}\,.
\end{align*}
To give intuition, an oracle policy $\pi_{\mathrm{oracle}}$ that knows when the significant shifts $\tau_i$'s discretizes the space into $K_i=(\tau_{i+1}-\tau_i)^{\frac{1}{1+2\beta}}\kappa^{\frac{2}{1+2\beta}}$ bins at each phase $[\tau_i, \tau_{i+1}[$, and incurs a regret of $\Tilde{\mathcal{O}}\left((\tau_{i+1}-\tau_i)^{\frac{1+\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}}\right)$ (by directly adapting results from \citet{kleinberg2004nearly}), where $\Tilde{\mathcal{O}}$ only hides logarithmic factors and numerical constants that does not depend on $\kappa$ or $\beta$. Thus, the dynamic regret of the oracle is upper bounded by (up to polylog factors)
\begin{align*}
\E{}{R(\pi_{\mathrm{oracle}}, T)}&\leq \sum_{i=1}^{\Tilde{L}_T}(\tau_{i+1}-\tau_i)^{\frac{1+\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}} \\
&\leq T^{\frac{1+\beta}{1+2\beta}}\Tilde{L}_T^{\frac{\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}}\,.
\end{align*}
We now show how to adapt \algo so that it achieves this rate \emph{adaptively}.
\textbf{Adapting the algorithm.} We consider blocks $[\tau_{l,m}, \tau_{l,m+1}[$ of size
\begin{align*}
\tau_{l,m+1}-\tau_{l,m} = 2^{m(1+2\beta)}/\kappa^2\,.
\end{align*}
Thus, on each block, we discretize the space into \begin{align*}
(\tau_{l,m+1}-\tau_{l,m})^{\frac{1}{1+2\beta}}\kappa^{\frac{2}{1+2\beta}}=2^m
\end{align*}
bins, allowing us to consider the MASTER set as the bins at depth $m$ (that is, the set of bins $\mathcal{T}_m$ of the dyadic tree). If the rewards are stable enough during this block, we expect the regret of the block to be of order
\begin{align*}
R(m)=(\tau_{l,m+1}-\tau_{l,m})^{\frac{1+\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}} = 2^{m(1+\beta)}\,.
\end{align*}
For the \textbf{replays}, we choose as duration of a replay at depth $d$ the length of the $d$-th block of an episode: this implies that we should schedule replays of duration \begin{align*}
\ell(d)=2^{d(1+2\beta)}/\kappa^2
\end{align*}
at depth $d$, and discretize $\ell(d)^{\frac{1}{1+2\beta}}\kappa^{\frac{2}{1+2\beta}}=2^d$ bins. Such replay allows to estimate the cumulative gaps of two bins up to precision $2^{d(1+\beta)}$.
Since we use exactly the same dyadic tree $(\mathcal{T}_d)_{1\leq d\leq m}$ as in the $1$-Lipschitz case, \textbf{the sampling strategy is exactly the same}, as well as the concentration error \cref{prop:concentration}. Only the bias for estimating the cumulative reward of two bins at depth $d$ changes, and will be exactly equals to $2(s_2-s_1)\kappa/2^{d\beta}$ over an interval $[s_1, s_2]$. This motivates a new eviction criteria depending on the Hölder constants $(\kappa, \beta)$, of the form
\begin{align*}
\max_{B'\in\mathcal{B}_{[s_1, s_2]}(d)}\sum_{t=s_1}^{s_2}\hat\delta_t(B', B)>c_0\log(T)\sqrt{(s_2-s_1)2^d\vee4^d}+4\frac{(s_2-s_1)\kappa}{2^{d\beta}}\,,
\end{align*}
where $c_0 = 7(e-1)\sqrt{2}$ is a numerical constant.
It remains to choose the \textbf{probability of scheduling a replay} at depth $d$, at a given round. At a round $s\in[\tau_{l,m},\tau_{l,m+1}[$ such that $s-\tau_{l,m}\equiv 0[\ell(d)]$, we schedule a replay of depth $d<m$ with probability $p_{s, d}$ given by
\begin{align*}
p_{s, d} = \frac{1}{\kappa} \frac{2^{\frac{d(1+2\beta)}{2}}}{\sqrt{s-\tau_{l,m}}} \,\quad\, (\text{we always have }p_{s,d}< 1\text{ for any }d< m)\,.
\end{align*}
\textbf{Regret analysis.} Simple computations show that there are on average $2^{\frac{(1+2\beta)(m-d)}{2}}$ replays of length $\ell(d)$ during one block $[\tau_{l,m},\tau_{l,m+1}[$. Thus, the regret due to replays at depth $d$ over this block is of order (ignoring multiplicative numerical constants that does not depend on $\kappa$ or $\beta$)
\begin{align*}
R(d)2^{\frac{(1+2\beta)(m-d)}{2}} = 2^{\frac{(1+2\beta)(m-d)+2d(1+\beta)}{2}}\,.
\end{align*}
Summing over the depths $d<m$, we find that the total contribution of the replays over a block $m$ is exactly of order $2^{m(1+\beta)}$ (up to numerical constants), \emph{i.e.} of the same order as the minimax-optimal regret over this phase in the non-stationary setting.
Moreover, this choice of replay ensures that there are approximatively $2^{\frac{(1+2\beta)(m-d)}{2}}$ replays at depth $d$ scheduled during block $m$, so that on average, a replay at depth $d$ is scheduled every $2^{\frac{(1+2\beta)(m+d)}{2}}$ rounds. Noticing that $2^{\frac{(1+2\beta)(m+d)}{2}}\leq 2^{m(1+\beta)}\times 2^{d\beta}$, we see that enough replays of the adequate length are scheduled so to ensure that a shift of magnitude $2^{-d\beta}$ does not go undetected for too long.
Since in each episode $[t_{l+1}-t_l[$ there are at most $M_l = c\frac{\log_2((t_{l+1}-t_l)\kappa)}{1+2\beta}$, where $c$ is a numerical constant, the regret over one episode is upper bounded as
\begin{align*}
\sum_{m=1}^{M_l} 2^{m(1+\beta)}\leq (t_{l+1}-t_l)^{\frac{1+\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}}\,.
\end{align*}
Applying exactly \cref{prop:relating_phase_and_episode} and applying Hölder's inequality, we have exactly the upper bound of
\begin{align*}
\E{}{R(\pi_{\algo}, T)}\leq \widetilde{\mathcal{O}}\left(T^{\frac{1+\beta}{1+2\beta}}\Tilde{L}_T^{\frac{\beta}{1+2\beta}}\kappa^{\frac{1}{1+2\beta}}\right)\,,
\end{align*}
where $\widetilde{\mathcal{O}}$ hides polylog factors and numerical constants that does not depend on $\kappa$ or $\beta$. We next prove that this bound is in fact minimax-optimal with respect to $T, \tilde{L}_T$, $\kappa$ and $\beta$ simultaneously.
\textbf{Lower Bound for non-stationary Hölder bandits.} We adapt the proof of \cref{th:lower_bound} to this setting. Any policy interacting with a $(\kappa, \beta)$-Hölder bandit environment suffers regret at least $T^{\frac{1+\beta}{1+2\beta}} \kappa^{\frac{1}{1+2\beta}}$ in the stationary setting \citep{kleinberg2004nearly}. We divide the horizon into $T/\tau$ blocks of length $\tau$. In each, we define an amount of $K = \tau^{\frac{1}{1+2\beta}} \kappa^{\frac{2}{1+2\beta}}$ $(\kappa,\beta)$ mean reward functions satisfying \cref{assump:holder} with a single bump of size $\varepsilon = \sqrt{K/\tau} = \tau^{-\frac{\beta}{1+2\beta}} \kappa^{\frac{1}{1+2\beta}}$,
hidden uniformly at random. Standard arguments show that any algorithm misidentifies the optimal region in some instance with constant probability, implying per-block regret
\begin{align*}
R(\pi, \tau) \geq \tau \varepsilon = \tau^{\frac{1+\beta}{1+2\beta}} \kappa^{\frac{1}{1+2\beta}}\,.
\end{align*}
Concatenating $T/\tau$ such blocks (with reward functions chosen independently) gives total regret of
\begin{align*}
(T/\tau) \times R(\pi, \tau) \geq T \tau^{-\frac{\beta}{1+2\beta}} \kappa^{\frac{1}{1+2\beta}}\,.
\end{align*}
To derive a lower bound that depends on $\Tilde{L}_T$, we first set $\tau = T / L_T$. This yields $
R(\pi, T) \geq T^{\frac{1+\beta}{1+2\beta}} L_T^{\frac{\beta}{1+2\beta}} \kappa^{\frac{1}{1+2\beta}}$, and we observe that we always have $L_T\geq \Tilde{L}_T$. For the dependency on $V_T$, we set \begin{align*}
\tau = \kappa^{\frac{1}{1+3\beta}} T^{\frac{1+2\beta}{1+3\beta}}V_T^{-\frac{1+2\beta}{1+3\beta}}\,.
\end{align*}
It is easy to verify that the total variation over $T$ is exactly $V_T$. So for any algorithm $\pi$, there exist an environment such that
\begin{align*}
R(\pi, T) \geq T^{\frac{1+\beta}{1+2\beta}} \widetilde{L}_T^{\frac{\beta}{1+2\beta}} \kappa^{\frac{1}{1+2\beta}}\wedge T^{\frac{1+\beta}{1+2\beta}} + T^{\frac{1+2\beta}{1+3\beta}} V_T^{\frac{\beta}{1+3\beta}}\kappa^{\frac{1}{1+3\beta}}\,.
\end{align*}
We recover our results for $\kappa=1$ and $\beta=1$. As the smoothness of the reward functions increases ($\beta \to 1$ and/or $\kappa \to 0$), the problem becomes easier, as the lower bound decreases.
\section{Computational complexity of \algo}\label{app:sec:computational_complexity}
While our algorithm is feasible on small-scale problems (as showed in our experiments in \cref{app:sec:simulations}), we acknowledge that its computational complexity may limit scalability in large-scale applications. Below, we provide a more precise characterization of the computational cost.
At each round within a block of length $\tau_{l,m+1}-\tau_{l,m} = 8^m$, the algorithm maintains estimates for all bins across all discretization depths $d = 1, \ldots, m$. The total number of bins at depth $d$ is $2^d$, and across all depths, we maintain a total of
\begin{align*}
\text{NumberBins}_m = \sum_{d=1}^m 2^d = \mathcal{O}(2^m)\,.
\end{align*}
For each bin, we compute an importance-weighted mean estimate, resulting in a per-round cost of $\mathcal{O}(2^m)$ for estimation alone. In addition, at each round $t$ of this block, the algorithm performs a statistical test $\eqref{eq:star}$ over \emph{all} pairs of bins $(B_1, B_2)$ at each depth $d \in [1, m]$. There are $\binom{2^d}{2} = \mathcal{O}(4^d)$ such pairs per depth, and summing over all depths yields \begin{align*}
\text{TotalPairs}_m = \sum_{d=1}^m \mathcal{O}(4^d) = \mathcal{O}(4^m)
\end{align*}
per block. For each pair $(B_1,B_2)$, we must consider all possible intervals $[s_1, t] \subseteq [\tau_{l,m}, t]$ during the block. The number of such intervals is $\mathcal{O}(8^m)$ in the worst case.
Putting it all together, the worst-case time complexity for each round is
\begin{align*}
\mathcal{O}(2^m) + \mathcal{O}(4^m \cdot 8^m) = \mathcal{O}(32^m)\,,
\end{align*}
and the memory complexity is $\mathcal{O}(8^m)$. Since the number of block is upper bounded as $m\leq\log(T)$, we conclude that the worst-case time computational complexity of our algorithm is $\mathcal{O}(T^6)$ and its worst-case memory complexity is $\mathcal{O}(T^4)$, which are both \textbf{polynomial in $T$.}
While the computational cost per block is manageable for small values of $m$ (e.g., $m \leq 4$), it becomes quickly intractable as $m$ increases. Developing efficient \textbf{adaptive algorithms} that enjoy minimax optimal regret is therefore an important direction for future work, even in the $K$-armed setting.
\section{Numerical simulations}\label{app:sec:simulations}
In this section, we illustrate some numerical experiments to show the empirical performances of \algo on a synthetic dataset. The code for these implementations is available at \url{https://github.com/nguyenicolas/NS_Lipschitz_Bandits}.
\textbf{Environment.} We simulate a $1$-Lipschitz, piecewise-linear reward function defined over the action space $[0,1]$, with a single peak shifting smoothly from \( x=0.3 \) to \( x=0.7 \) every \(10^5\) rounds. This setup induces \(\tilde{L}_T = 10\) significant shifts over a time horizon \(T = 10^6\). Thus, the mean reward changes every round, but only ten of these changes are significant under our framework.
\textbf{Benchmarks.} We compare our method against two baselines: \texttt{BinningUCB (naive)} and \texttt{BinningUCB (oracle)}. The first baseline assumes knowledge only of the total time horizon $T$ and naively discretizes the action space into $K(T) \propto T^{1/3}$ actions, as if operating in a stationary environment of length $T$. It then runs a standard \texttt{UCB} algorithm without resetting its statistical estimates. The \texttt{oracle} baseline, by contrast, has access to the \emph{exact} times of the significant shifts $\tau_i$'s and resets its estimates at each significant phase, using the optimal per-phase discretization $K_i \propto (\tau_{i+1}-\tau_i)^{1/3}$.
\textbf{Results.} We report in \cref{fig:regret_plot} the cumulative dynamic regret of the three methods, averaged over $100$ independent runs. Our method (\algo) significantly outperforms \texttt{BinningUCB (naive)} by adapting to non-stationarities through replay mechanisms.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figures/regret_plot.jpg}
\caption{Cumulative dynamic regret of \algo, \texttt{BinningUCB (naive)}, and \texttt{BinningUCB (oracle)} over a total horizon of $T = 10^6$ rounds with $10$ significant shifts. Results are averaged over $100$ independent runs, with $95\%$ confidence intervals of the mean dynamic regret shown.}
\label{fig:regret_plot}
\end{figure}
\begin{thebibliography}{52}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abbasi-Yadkori et~al.(2023)Abbasi-Yadkori, Gy{\"o}rgy, and Lazi{\'c}]{abbasi2023new}
Yasin Abbasi-Yadkori, Andr{\'a}s Gy{\"o}rgy, and Nevena Lazi{\'c}.
\newblock A new look at dynamic regret for non-stationary stochastic bandits.
\newblock \emph{Journal of Machine Learning Research}, 24\penalty0 (288):\penalty0 1--37, 2023.
\bibitem[Agrawal(1995)]{agrawal1995continuum}
Rajeev Agrawal.
\newblock The continuum-armed bandit problem.
\newblock \emph{SIAM journal on control and optimization}, 33\penalty0 (6):\penalty0 1926--1951, 1995.
\bibitem[Auer et~al.(2002{\natexlab{a}})Auer, Cesa-Bianchi, and Fischer]{auer2002finite}
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer.
\newblock Finite-time analysis of the multiarmed bandit problem.
\newblock \emph{Machine learning}, 47:\penalty0 235--256, 2002{\natexlab{a}}.
\bibitem[Auer et~al.(2002{\natexlab{b}})Auer, Cesa-Bianchi, Freund, and Schapire]{auer2002nonstochastic}
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert~E Schapire.
\newblock The nonstochastic multiarmed bandit problem.
\newblock \emph{SIAM journal on computing}, 32\penalty0 (1):\penalty0 48--77, 2002{\natexlab{b}}.
\bibitem[Auer et~al.(2019{\natexlab{a}})Auer, Chen, Gajane, Lee, Luo, Ortner, and Wei]{auer2019achieving}
Peter Auer, Yifang Chen, Pratik Gajane, Chung-Wei Lee, Haipeng Luo, Ronald Ortner, and Chen-Yu Wei.
\newblock Achieving optimal dynamic regret for non-stationary bandits without prior information.
\newblock In \emph{Conference on Learning Theory}, pages 159--163. PMLR, 2019{\natexlab{a}}.
\bibitem[Auer et~al.(2019{\natexlab{b}})Auer, Gajane, and Ortner]{auer2019adaptively}
Peter Auer, Pratik Gajane, and Ronald Ortner.
\newblock Adaptively tracking the best bandit arm with an unknown number of distribution changes.
\newblock In \emph{Conference on Learning Theory}, pages 138--158. PMLR, 2019{\natexlab{b}}.
\bibitem[Azizi et~al.(2022)Azizi, Duong, Abbasi-Yadkori, Gy{\"o}rgy, Vernade, and Ghavamzadeh]{azizi2022non}
MohammadJavad Azizi, Thang Duong, Yasin Abbasi-Yadkori, Andr{\'a}s Gy{\"o}rgy, Claire Vernade, and Mohammad Ghavamzadeh.
\newblock Non-stationary bandits and meta-learning with a small set of optimal arms.
\newblock \emph{Reinforcement Learning Journal}, 2022.
\bibitem[Besbes et~al.(2014)Besbes, Gur, and Zeevi]{besbes2014stochastic}
Omar Besbes, Yonatan Gur, and Assaf Zeevi.
\newblock Stochastic multi-armed-bandit problem with non-stationary rewards.
\newblock \emph{Advances in neural information processing systems}, 27, 2014.
\bibitem[Beygelzimer et~al.(2011)Beygelzimer, Langford, Li, Reyzin, and Schapire]{beygelzimer2011contextual}
Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire.
\newblock Contextual bandit algorithms with supervised learning guarantees.
\newblock In \emph{Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics}, pages 19--26. JMLR Workshop and Conference Proceedings, 2011.
\bibitem[Bubeck et~al.(2011{\natexlab{a}})Bubeck, Munos, Stoltz, and Szepesv{\'a}ri]{bubeck2011x}
S{\'e}bastien Bubeck, R{\'e}mi Munos, Gilles Stoltz, and Csaba Szepesv{\'a}ri.
\newblock X-armed bandits.
\newblock \emph{Journal of Machine Learning Research}, 12\penalty0 (5), 2011{\natexlab{a}}.
\bibitem[Bubeck et~al.(2011{\natexlab{b}})Bubeck, Stoltz, and Yu]{bubeck2011lipschitz}
S{\'e}bastien Bubeck, Gilles Stoltz, and Jia~Yuan Yu.
\newblock Lipschitz bandits without the lipschitz constant.
\newblock In \emph{Algorithmic Learning Theory: 22nd International Conference, ALT 2011, Espoo, Finland, October 5-7, 2011. Proceedings 22}, pages 144--158. Springer, 2011{\natexlab{b}}.
\bibitem[Buening and Saha(2023)]{buening2023anaconda}
Thomas~Kleine Buening and Aadirupa Saha.
\newblock Anaconda: An improved dynamic regret algorithm for adaptive non-stationary dueling bandits.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, pages 3854--3878. PMLR, 2023.
\bibitem[Cai and Scarlett(2025)]{cai2024lower}
Xu~Cai and Jonathan Scarlett.
\newblock Lower bounds for time-varying kernelized bandits.
\newblock \emph{International Conference on Artificial Intelligence and Statistics}, 2025.
\bibitem[Carpentier and Valko(2015)]{carpentier2015simple}
Alexandra Carpentier and Michal Valko.
\newblock Simple regret for infinitely many armed bandits.
\newblock In \emph{International Conference on Machine Learning}, pages 1133--1141. PMLR, 2015.
\bibitem[Chen et~al.(2019)Chen, Lee, Luo, and Wei]{chen2019new}
Yifang Chen, Chung-Wei Lee, Haipeng Luo, and Chen-Yu Wei.
\newblock A new algorithm for non-stationary contextual bandits: Efficient, optimal and parameter-free.
\newblock In \emph{Conference on Learning Theory}, pages 696--726. PMLR, 2019.
\bibitem[Cheung et~al.(2019)Cheung, Simchi-Levi, and Zhu]{cheung2019learning}
Wang~Chi Cheung, David Simchi-Levi, and Ruihao Zhu.
\newblock Learning to optimize under non-stationarity.
\newblock In \emph{The 22nd International Conference on Artificial Intelligence and Statistics}, pages 1079--1087. PMLR, 2019.
\bibitem[De~Heide et~al.(2021)De~Heide, Cheshire, M{\'e}nard, and Carpentier]{de2021bandits}
Rianne De~Heide, James Cheshire, Pierre M{\'e}nard, and Alexandra Carpentier.
\newblock Bandits with many optimal arms.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 22457--22469, 2021.
\bibitem[Even-Dar et~al.(2006)Even-Dar, Mannor, Mansour, and Mahadevan]{even2006action}
Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan.
\newblock Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems.
\newblock \emph{Journal of machine learning research}, 7\penalty0 (6), 2006.
\bibitem[Faury et~al.(2021)Faury, Russac, Abeille, and Calauz{\`e}nes]{faury2021optimal}
Louis Faury, Yoan Russac, Marc Abeille, and Cl{\'e}ment Calauz{\`e}nes.
\newblock Optimal regret bounds for generalized linear bandits under parameter drift.
\newblock \emph{Proceedings of Machine Learning Research vol}, 132:\penalty0 1--37, 2021.
\bibitem[Garivier and Moulines(2011)]{garivier2011upper}
Aur{\'e}lien Garivier and Eric Moulines.
\newblock On upper-confidence bound policies for switching bandit problems.
\newblock In \emph{International conference on algorithmic learning theory}, pages 174--188. Springer, 2011.
\bibitem[Gerogiannis et~al.(2025{\natexlab{a}})Gerogiannis, Huang, Bose, and Veeravalli]{gerogiannis2025dal}
Argyrios Gerogiannis, Yu-Han Huang, Subhonmesh Bose, and Venugopal~V Veeravalli.
\newblock Dal: A practical prior-free black-box framework for non-stationary bandit environments.
\newblock \emph{arXiv preprint arXiv:2501.19401}, 2025{\natexlab{a}}.
\bibitem[Gerogiannis et~al.(2025{\natexlab{b}})Gerogiannis, Huang, and Veeravalli]{gerogiannis2410prior}
Argyrios Gerogiannis, Yu-Han Huang, and Venugopal~V Veeravalli.
\newblock Is prior-free black-box non-stationary reinforcement learning feasible?
\newblock \emph{International Conference on Artificial Intelligence and Statistics}, 2025{\natexlab{b}}.
\bibitem[Hong et~al.(2023)Hong, Li, and Tewari]{hong2023optimization}
Kihyuk Hong, Yuhang Li, and Ambuj Tewari.
\newblock An optimization-based algorithm for non-stationary kernel bandits without prior knowledge.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, pages 3048--3085. PMLR, 2023.
\bibitem[Iwazaki and Takeno(2025)]{iwazaki2024near}
Shogo Iwazaki and Shion Takeno.
\newblock Near-optimal algorithm for non-stationary kernelized bandits.
\newblock \emph{International Conference on Artificial Intelligence and Statistics}, 2025.
\bibitem[Jia et~al.(2023)Jia, Xie, Kallus, and Frazier]{jia2023smooth}
Su~Jia, Qian Xie, Nathan Kallus, and Peter~I Frazier.
\newblock Smooth non-stationary bandits.
\newblock In \emph{International Conference on Machine Learning}, pages 14930--14944. PMLR, 2023.
\bibitem[Kang et~al.(2023{\natexlab{a}})Kang, Hsieh, and Lee]{kang2023online}
Yue Kang, Cho-Jui Hsieh, and Thomas Lee.
\newblock Online continuous hyperparameter optimization for generalized linear contextual bandits.
\newblock \emph{Transactions on Machine Learning Research}, 2023{\natexlab{a}}.
\bibitem[Kang et~al.(2023{\natexlab{b}})Kang, Hsieh, and Lee]{kang2023robust}
Yue Kang, Cho-Jui Hsieh, and Thomas Chun~Man Lee.
\newblock Robust lipschitz bandits to adversarial corruptions.
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 10897--10908, 2023{\natexlab{b}}.
\bibitem[Kim et~al.(2022)Kim, Vojnovic, and Yun]{kim2022rotting}
Jung-Hun Kim, Milan Vojnovic, and Se-Young Yun.
\newblock Rotting infinitely many-armed bandits.
\newblock In \emph{International Conference on Machine Learning}, pages 11229--11254. PMLR, 2022.
\bibitem[Kim et~al.(2024)Kim, Vojnovic, and Yun]{kim2024adaptive}
Jung-hun Kim, Milan Vojnovic, and Se-Young Yun.
\newblock An adaptive approach for infinitely many-armed bandits under generalized rotting constraints.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 8785--8833, 2024.
\bibitem[Kleinberg(2004)]{kleinberg2004nearly}
Robert Kleinberg.
\newblock Nearly tight bounds for the continuum-armed bandit problem.
\newblock \emph{Advances in Neural Information Processing Systems}, 17, 2004.
\bibitem[Kleinberg et~al.(2008)Kleinberg, Slivkins, and Upfal]{kleinberg2008multi}
Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal.
\newblock Multi-armed bandits in metric spaces.
\newblock In \emph{Proceedings of the fortieth annual ACM symposium on Theory of computing}, pages 681--690, 2008.
\bibitem[Krishnamurthy and Gopalan(2021)]{krishnamurthy2021slowly}
Ramakrishnan Krishnamurthy and Aditya Gopalan.
\newblock On slowly-varying non-stationary bandits.
\newblock \emph{The second Reinforcement Learning Conference}, 2021.
\bibitem[Lattimore and Szepesv{\'a}ri(2020)]{lattimore2020bandit}
Tor Lattimore and Csaba Szepesv{\'a}ri.
\newblock \emph{Bandit algorithms}.
\newblock Cambridge University Press, 2020.
\bibitem[Liu et~al.(2025)Liu, Baudry, Zimmert, Rebeschini, and Akhavan]{liu2025non}
Xiaoqi Liu, Dorian Baudry, Julian Zimmert, Patrick Rebeschini, and Arya Akhavan.
\newblock Non-stationary bandit convex optimization: A comprehensive study.
\newblock \emph{arXiv preprint arXiv:2506.02980}, 2025.
\bibitem[Magureanu et~al.(2014)Magureanu, Combes, and Proutiere]{magureanu2014lipschitz}
Stefan Magureanu, Richard Combes, and Alexandre Proutiere.
\newblock Lipschitz bandits: Regret lower bound and optimal algorithms.
\newblock In \emph{Conference on Learning Theory}, pages 975--999. PMLR, 2014.
\bibitem[Podimata and Slivkins(2021)]{podimata2021adaptive}
Chara Podimata and Alex Slivkins.
\newblock Adaptive discretization for adversarial lipschitz bandits.
\newblock In \emph{Conference on Learning Theory}, pages 3788--3805. PMLR, 2021.
\bibitem[Russac et~al.(2019)Russac, Vernade, and Capp{\'e}]{russac2019weighted}
Yoan Russac, Claire Vernade, and Olivier Capp{\'e}.
\newblock Weighted linear bandits for non-stationary environments.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Seznec et~al.(2020)Seznec, Menard, Lazaric, and Valko]{seznec2020single}
Julien Seznec, Pierre Menard, Alessandro Lazaric, and Michal Valko.
\newblock A single algorithm for both restless and rested rotting bandits.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, pages 3784--3794. PMLR, 2020.
\bibitem[Slivkins and Upfal(2008)]{slivkins2008adapting}
Aleksandrs Slivkins and Eli Upfal.
\newblock Adapting to a changing environment: the brownian restless bandits.
\newblock In \emph{COLT}, pages 343--354, 2008.
\bibitem[Slivkins et~al.(2019)]{slivkins2019introduction}
Aleksandrs Slivkins et~al.
\newblock Introduction to multi-armed bandits.
\newblock \emph{Foundations and Trends{\textregistered} in Machine Learning}, 12\penalty0 (1-2):\penalty0 1--286, 2019.
\bibitem[Suk(2024)]{suk2024adaptive}
Joe Suk.
\newblock Adaptive smooth non-stationary bandits.
\newblock \emph{arXiv preprint arXiv:2407.08654}, 2024.
\bibitem[Suk and Agarwal(2023)]{suk2023can}
Joe Suk and Arpit Agarwal.
\newblock When can we track significant preference shifts in dueling bandits?
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 38347--38369, 2023.
\bibitem[Suk and Agarwal(2024)]{suk2024non}
Joe Suk and Arpit Agarwal.
\newblock Non-stationary dueling bandits under a weighted borda criterion.
\newblock \emph{arXiv preprint arXiv:2403.12950}, 2024.
\bibitem[Suk and Kim(2025)]{suk2025tracking}
Joe Suk and Jung-hun Kim.
\newblock Tracking most significant shifts in infinite-armed bandits.
\newblock \emph{International Conference on Machine Learning}, 2025.
\bibitem[Suk and Kpotufe(2022)]{suk2022tracking}
Joe Suk and Samory Kpotufe.
\newblock Tracking most significant arm switches in bandits.
\newblock In \emph{Conference on Learning Theory}, pages 2160--2182. PMLR, 2022.
\bibitem[Suk and Kpotufe(2023)]{suk2023tracking}
Joe Suk and Samory Kpotufe.
\newblock Tracking most significant shifts in nonparametric contextual bandits.
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 6202--6241, 2023.
\bibitem[Suk and Kpotufe(2021)]{suk2021self}
Joseph Suk and Samory Kpotufe.
\newblock Self-tuning bandits over unknown covariate-shifts.
\newblock In \emph{Algorithmic Learning Theory}, pages 1114--1156. PMLR, 2021.
\bibitem[Vernade et~al.(2020)Vernade, Gyorgy, and Mann]{vernade2020non}
Claire Vernade, Andras Gyorgy, and Timothy Mann.
\newblock Non-stationary delayed bandits with intermediate observations.
\newblock In \emph{International Conference on Machine Learning}, pages 9722--9732. PMLR, 2020.
\bibitem[Wang(2023)]{wang2023adaptivity}
Yining Wang.
\newblock On adaptivity in nonstationary stochastic optimization with bandit feedback.
\newblock \emph{Operations Research}, 2023.
\bibitem[Wang et~al.(2008)Wang, Audibert, and Munos]{wang2008algorithms}
Yizao Wang, Jean-Yves Audibert, and R{\'e}mi Munos.
\newblock Algorithms for infinitely many-armed bandits.
\newblock \emph{Advances in Neural Information Processing Systems}, 21, 2008.
\bibitem[Wei and Luo(2021)]{wei2021non}
Chen-Yu Wei and Haipeng Luo.
\newblock Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach.
\newblock In \emph{Conference on learning theory}, pages 4300--4354. PMLR, 2021.
\bibitem[Zhao et~al.(2021)Zhao, Wang, Zhang, and Zhou]{zhao2021bandit}
Peng Zhao, Guanghui Wang, Lijun Zhang, and Zhi-Hua Zhou.
\newblock Bandit convex optimization in non-stationary environments.
\newblock \emph{Journal of Machine Learning Research}, 22\penalty0 (125):\penalty0 1--45, 2021.
\end{thebibliography}