Monte-Carlo Tree Search by Best Arm Identification
Extracted source text ยท arXiv 1706.02986
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading nips_2017
%
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{nips_2017}
%\usepackage{nips_2017}
\usepackage{macrosArticle}
% to compile a camera-ready version, add the [final] option, e.g.:
%\usepackage[final]{nips_2017}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{todonotes}
\usepackage{rotating}
% specific macros for this paper
\def\UCB{\mathrm{U}}
\def\LCB{\mathrm{L}}
\def\Par{\cP}
\def\Anc{\mathrm{Anc}}
\def\U{\mathrm{U}}
\def\L{\mathrm{L}}
\def \Alt{\mathrm{Alt}}
\def \w{\bm w}
\def \vmu{\bm\mu}
% Making tree search Banditable
% Searching games trees with bandits
% Passing Confidences up the Tree
% Putting Confidence in Trees
% When Tree Search meets Best Arm Identification
% Best Arm Identification methods for MCTS
% Best Arm Identification Meets Principal Variation
% A Best Arm Identification Approach to Monte-Carlo Tree Search
% Running Best Arm at the Root
% A best-arm root with confidence sub-trees
% New bandit tools and sample complexity results for Monte-Carlo Tree Search
\title{
Monte-Carlo Tree Search by Best Arm Identification
}
% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.
\author{Emilie Kaufmann$^1$ and Wouter M.Koolen$^2$}
\date{
$^1$ CNRS \& Univ. Lille, UMR 9189 (CRIStAL), Inria SequeL, Lille, France \\
$^2$ Centrum Wiskunde \& Informatica, Science Park 123, 1098 XG Amsterdam, The Netherlands
}
\begin{document}
\maketitle
\begin{abstract}
Recent advances in bandit tools and techniques for sequential learning are steadily enabling new applications and are promising the resolution of a range of challenging related problems. We study the game tree search problem, where the goal is to quickly identify the optimal move in a given game tree by sequentially sampling its stochastic payoffs. We develop new algorithms for trees of arbitrary depth, that operate by summarizing all deeper levels of the tree into confidence intervals at depth one, and applying a best arm identification procedure at the root. We prove new sample complexity guarantees with a refined dependence on the problem instance. We show experimentally that our algorithms outperform existing elimination-based algorithms and match previous special-purpose methods for depth-two trees.
\end{abstract}
\section{Introduction}
We consider two-player zero-sum turn-based interactions, in which the sequence of possible successive moves is represented by a maximin game tree $\cT$. This tree models the possible actions sequences by a collection of MAX nodes, that correspond to states in the game in which player A should take action, MIN nodes, for states in the game in which player B should take action, and leaves which specify the payoff for player A. The goal is to determine the best action at the root for player A. For deterministic payoffs this search problem is primarily algorithmic, with several powerful pruning strategies available \cite{PLAAT1996255}. We look at problems with stochastic payoffs, which in addition present a major statistical challenge.
Sequential identification questions in game trees with stochastic payoffs arise naturally as robust versions of bandit problems. They are also a core component of Monte Carlo tree search (MCTS) approaches for solving intractably large deterministic tree search problems, where an entire sub-tree is represented by a stochastic leaf in which randomized play-out and/or evaluations are performed \cite{SurveyMCTS12}. A play-out consists in finishing the game with some simple, typically random, policy and observing the outcome for player A.
%
% The goal is to determine the best action at the root for player A, by repeatedly choosing a sequence of actions for the two players from the root to a leaf. Starting from this leaf, which correspond to some position in the game, a \emph{play-out} is performed, that consists of finishing the game by either playing at random or using some play-out strategy. An alternative to this play-out is to produce a (possibly random) evaluation of the current position.
For example, MCTS is used within the AlphaGo system \cite{deep.go}, and the evaluation of a leaf position combines supervised learning and (smart) play-outs.
While MCTS algorithms for Go have now reached expert human level, such algorithms remain very costly, in that many (expensive) leaf evaluations or play-outs are necessary to output the next action to be taken by the player. In this paper, we focus on the \emph{sample complexity} of Monte-Carlo Tree Search methods, about which very little is known. For this purpose, we work under a simplified model for MCTS already studied by \cite{Teraoka14MCTS}, and that generalizes the depth-two framework of \cite{GKK16}.
\subsection{A simple model for Monte-Carlo Tree Search}\label{subsec:model}
We start by fixing a game tree $\cT$, in which the root is a MAX node. %This tree might be obtained as a sub-tree of a larger game tree, and we do not assume it to be balanced.
Letting $\cL$ be the set of leaves of this tree, for each $\ell\in \cL$ we introduce a stochastic oracle $\cO_\ell$ that represents the leaf evaluation or play-out performed when this leaf is reached by an MCTS algorithm. In this model, we do not try to optimize the evaluation or play-out strategy, but we rather assume that the oracle $\cO_\ell$ produces i.i.d.\ samples from an unknown distribution whose mean $\mu_\ell$ is the value of the position $\ell$. To ease the presentation, we focus on binary oracles (indicating the win or loss of a play-out), in which the oracle $\cO_\ell$ is a Bernoulli distribution with unknown mean $\mu_\ell$ (the probability of player A winning the game in the corresponding state). Our algorithms can be used without modification in case the oracle is a distribution bounded in $[0,1]$.
For each node $s$ in the tree, we denote by $\cC(s)$ the set of its children and by $\cP(s)$ its parent. The root is denoted by $s_0$. The \emph{value} (for player A) of any node $s$ is recursively defined by $V_\ell = \mu_\ell$ if $\ell \in \cL$ and
\[V_s = \left\{
\begin{array}{ll}
%\mu_s & \text{if s } \in \cL \\
\max_{c \in \cC(s)} V_c & \text{if s is a MAX node}, \\
\min_{c \in \cC(s)} V_c & \text{if s is a MIN node}.
\end{array}\right. \]
% Assuming that player $B$ is strategic,
The best move is the action at the root with highest value, \[s^* = \argmax{s \in \cC(s_0)} \ V_s.\]
% so that $V(s_0)=V(s^*)$.
To identify $s^*$ (or an $\epsilon$-close move), an MCTS algorithm sequentially selects paths in the game tree and calls the corresponding leaf oracle. At round $t$, a leaf $L_t \in \cL$ is chosen by this adaptive \emph{sampling rule}, after which a sample $X_t \sim \cO_{L_t}$ is collected. We consider here the same PAC learning framework as \cite{Teraoka14MCTS,GKK16}, in which the strategy also requires a \emph{stopping rule}, after which leaves are no longer evaluated, and a \emph{recommendation rule} that outputs upon stopping a guess $\hat{s}_\tau \in \cC(s_0)$ for the best move of player A.
Given a risk level $\delta$ and some accuracy parameter $\epsilon \geq 0$ our goal is have a recommendation $\hat{s}_\tau \in \cC(s_0)$ whose value is within $\epsilon$ of the value of the best move, with probability larger than $1-\delta$, that is
\[\bP\left(V(s_0) - V(\hat{s}_\tau) \leq \epsilon \right) \geq 1 - \delta.\]
An algorithm satisfying this property is called $(\epsilon,\delta)$-correct. The main challenge is \emph{to design $(\epsilon,\delta)$-correct algorithms that use as few leaf evaluations $\tau$ as possible.} %In particular, we seek strategies minimizing the \emph{sample complexity}, that is the expected number of play-outs $\bE[\tau]$.
\paragraph{Related work}
The model we introduce for Monte-Carlo Tree Search is very reminiscent of a stochastic bandit model. In those, an agent repeatedly selects one out of several probability distributions, called arms, and draws a sample from the chosen distribution. Bandits models have been studied since the 1930s \cite{Thompson33}, mostly with a focus on \emph{regret minimization}, where the agent aims to maximize the sum of the samples collected, which are viewed as rewards \cite{LaiRobbins85bandits}. In the context of MCTS, a sample corresponds to a win or a loss in one play-out, and maximizing the number of successful play-outs (that correspond to simulated games) may be at odds with identifying quickly the next best action to take at the root. In that, our best action identification problem is closer to a so-called \emph{Best Arm Identification} (BAI) problem.
%Best arm identification have been studied quite a lot recently (see \cite{KG17ESAIM} for a survey)
The goal in the standard BAI problem is to find quickly and accurately the arm with highest mean. The BAI problem in the fixed-confidence setting \cite{EvenDaral06} is the special case of our simple model for a tree of depth one. For deeper trees, rather than finding the best arm (i.e.\ leaf), we are interested in finding the best action at the root. As the best root action is a function of the means of all leaves, this is a more structured problem.
Bandit algorithms, and more recently BAI algorithms have been successfully adapted to tree search. Building on the UCB algorithm \cite{Aueral02}, a regret minimizing algorithm, variants of the UCT algorithm \cite{KocsisBBMCP06} have been used for MCTS in growing trees, leading to successful AIs for games. However, there are only very weak theoretical guarantees for UCT. Moreover, observing that maximizing the number of successful play-outs is not the target, recent work rather tried to leverage tools from the BAI literature. In \cite{Pepels14HybridMCTS,Cazenave15SHOT} Sequential Halving \cite{Karnin:al13} is used for exploring game trees. The latter algorithm is a state-of-the-art algorithm for the fixed-budget BAI problem \cite{Bubeck10BestArm}, in which the goal is to identify the best arm with the smallest probability of error based on a given budget of draws. The proposed SHOT (Sequential Halving applied tO Trees) algorithm \cite{Cazenave15SHOT} is compared empirically to the UCT approach of \cite{KocsisBBMCP06}, showing improvements in some cases. A hybrid approach mixing SHOT and UCT is also studied \cite{Pepels14HybridMCTS}, still without sample complexity guarantees.
In the fixed-confidence setting, \cite{Teraoka14MCTS} develop the first sample complexity guarantees in the model we consider. The proposed algorithm, $\mathrm{FindTopWinner}$ is based on \emph{uniform sampling and eliminations}, an approach that may be related to the Successive Eliminations algorithm \cite{EvenDaral06} for fixed-confidence BAI in bandit models. $\mathrm{FindTopWinner}$ proceeds in rounds, in which the leaves that have not been eliminated are sampled repeatedly until the precision of their estimates doubled. Then the tree is pruned of every node whose estimated value differs significantly from the estimated value of its parent, which leads to the possible elimination of several leaves. For depth-two trees, \cite{GKK16} propose an elimination procedure that is not round-based. In this simpler setting, an algorithm that exploits \emph{confidence intervals} is also developed, inspired by the LUCB algorithm for fixed-confidence BAI \cite{Shivaramal12}. Some variants of the proposed M-LUCB algorithm appear to perform better in simulations than elimination based algorithms. We now investigate this trend further in deeper trees, both in theory and in practice.
\paragraph{Our Contribution.}
In this paper, we propose a generic architecture, called BAI-MCTS, that builds on a Best Arm Identification (BAI) algorithm and on confidence intervals on the node values in order to solve the best action identification problem in a tree of arbitrary depth. In particular, we study two specific instances, UGapE-MCTS and LUCB-MCTS, that rely on \emph{confidence-based} BAI algorithms \cite{Gabillon:al12,Shivaramal12}. We prove that these are ($\epsilon,\delta$)-correct and give a high-probability upper bound on their sample complexity. Both our theoretical and empirical results improve over the elimination-based state-of-the-art algorithm, $\mathrm{FindTopWinner}$ \cite{Teraoka14MCTS}.
\section{BAI-MCTS algorithms}
We present a generic class of algorithms, called BAI-MCTS, that combines a BAI algorithm with an exploration of the tree based on confidence intervals on the node values. Before introducing the algorithm and two particular instances, we first explain how to build such confidence intervals, and also introduce the central notion of \emph{representative child} and \emph{representative leaf}.
\subsection{Confidence intervals and representative nodes\label{sec:CI}}
For each leaf $\ell\in\cL$, using the past observations from this leaf we may build a confidence interval
\[\cI_\ell(t) = [\LCB_\ell(t), \UCB_\ell(t)],\]
where $\UCB_\ell(t)$ (resp.\ $\LCB_\ell(t)$) is an Upper Confidence Bound (resp.\ a Lower Confidence Bound) on the value $V(\ell) =\mu_{\ell}$. The specific confidence interval we shall use will be discussed later.
These confidence intervals are then propagated upwards in the tree using the following construction. For each internal node $s$, we recursively define $\cI_s(t) = [\LCB_s(t),\UCB_s(t)]$ with
\begin{minipage}{0.5\textwidth}
\[\LCB_s(t) = \left\{\begin{array}{cl}
\max_{c\in\cC(s)} \LCB_c(t)& \text{for a MAX node } s, \\
\min_{c\in\cC(s)} \LCB_c(t)& \text{for a MIN node } s,
\end{array}
\right. \]
\end{minipage}
\begin{minipage}{0.49\textwidth}
\[\UCB_s(t) = \left\{\begin{array}{cl}
\max_{c\in\cC(s)} \UCB_c(t)& \text{for a MAX node } s, \\
\min_{c\in\cC(s)} \UCB_c(t)& \text{for a MIN node } s.
\end{array}
\right. \]
\end{minipage}
\medskip
Note that these intervals are the tightest possible on the parent under the sole assumption that the child confidence intervals are all valid.
A similar construction was used in the OMS algorithm of \cite{OMS14} in a different context.
It is easy to convince oneself (or prove by induction, see Appendix~\ref{proof:CorrectCI}) that the accuracy of the confidence intervals is preserved under this construction, as stated below.
\begin{proposition}\label{prop:CorrectCI} Let $t\in \N$. One has $\bigcap_{\ell \in \cL}\left(\mu_\ell \in \cI_\ell(t)\right) \ \ \Rightarrow \ \ \bigcap_{s \in \cT}\left(V_s \in \cI_s(t)\right)$.
\end{proposition}
We now define the \emph{representative child} $c_s(t)$ of an internal node $s$ as
\begin{eqnarray*}
c_s(t) &=& \left\{\begin{array}{cl} \text{argmax}_{c \in \cC(s)} \ \UCB_c(t) & \text{if } s \text{ is a MAX node},\\
\text{argmin}_{c \in \cC(s)} \ \LCB_c(t) & \text{if } s \text{ is a MIN node},\end{array}\right.
\end{eqnarray*}
and the \emph{representative leaf} $\ell_s(t)$ of a node $s\in\cT$, which is the leaf obtained when going down the tree by always selecting the representative child:
\[\ell_s(t) =
s \ \ \text{if } s \in \cL, \ \ \ \ \ \ell_s(t) = \ell_{c_s(t)}(t) \ \ \text{otherwise}.\]
%\left\{\begin{array}{cl} s & \text{if } s \in \cL \\ \ell_{c_s(t)}(t) & \text{otherwise}. \end{array}\right.\]
The confidence intervals in the tree represent the statistically plausible values in each node, hence the representative child can be interpreted as an ``optimistic move'' in a MAX node and a ``pessimistic move'' in a MIN node (assuming we play against the best possible adversary). This is reminiscent of the behavior of the UCT algorithm \cite{KocsisBBMCP06}. The construction of the confidence intervals and associated representative children are illustrated in Figure~\ref{fig:IC}.
\begin{figure}[h]
\centering
\parbox{.4\textwidth}{%
\centering
\newcommand{\drawarma}[5]{
\node (#1m) at (#2, #3) [inner sep=2pt,circle,draw,fill]{};
\node (#1U) [above= #4 of #1m, inner xsep=2pt, inner ysep=0pt] {}; \draw[thick,-|] (#1m) -- (#1U);
\node (#1L) [below= #5 of #1m, inner xsep=2pt, inner ysep=0pt] {}; \draw[thick,-|] (#1m) -- (#1L);
}
\newcommand{\drawarm}[4]{
\drawarma{#1}{#2}{#3}{#4}{#4}
}
\scalebox{.45}{
\begin{tikzpicture}[very thick]
\drawarm{c1}{0}{0.95+.4}{1}
\drawarm{c2}{1}{2.1+.4}{2}
\begin{scope}[red]
\drawarm{c3}{2}{3.5}{2}
\end{scope}
\drawarm{c4}{3}{2.5}{1}
\drawarm{c5}{4}{3.1}{.5}
\drawarm{c6}{5}{2.0}{1.5}
% \draw[dashed] (6,0) -- (6,6);
\drawarma{p}{9}{3.45}{2.0}{0.9}
\draw [help lines, dashed, draw, ->] (c3U) -- (pU);
\draw [help lines, dashed, draw, ->] (c5L) -- (pL);
\draw [help lines, dashed, draw, ->] (c3m) -- (pm);
\node at (3,0) [below] {(a) Children};
\node at (9,0) [below] {(b) Parent};
\end{tikzpicture}
}
\caption{\label{fig:IC}Construction of confidence interval and representative child (in red) for a MAX node.}
}%
\qquad
\begin{minipage}{.54\textwidth}%
\begin{algorithm}[H]
\textbf{Input}: a BAI algorithm \\
\textbf{Initialization}: $t=0$. \\
\While{\textbf{not} $\mathrm{BAIStop}\left(\left\{s \in \cC(s_0) \right\} \right)$}{
{$R_{t+1}=\mathrm{BAIStep}\left(\left\{s \in \cC(s_0) \right\} \right)$ \\
Sample the representative leaf $L_{t+1} = \ell_{R_{t+1}}(t)$ \\
Update the information about the arms. $t = t+1$.
}
}
\textbf{Output}: $\mathrm{BAIReco}\left(\left\{s \in \cC(s_0) \right\} \right)$ \\
\end{algorithm}
\caption{The BAI-MCTS architecture \label{algo:BAIMCTS}}
\end{minipage}%
\end{figure}
% \begin{remark} Starting from confidence intervals for the leaves, an alternative construction is possible:
% \[\cI^{\text{Rep}}_s(t) = \left[\LCB^{\text{Rep}}_{c_s(t)}(t),\UCB^{\text{Rep}}_{c_s(t)}(t)\right]\]
% where the representative child of a MAX node (resp. MIN node) is now defined as
% \[c_s(t) = \argmax{c \in \cC(s)} \ \UCB^{\text{Rep}}_{c}(t) \ \ \ \big(\text{resp.} \ \argmin{c \in \cC(s)} \ \LCB^{\text{Rep}}_{c}(t)\big).\]
%
% In particular, a confidence interval $\cI^{\text{Rep}}_s(t)$ is exactly the confidence interval of some representative leaf, whereas $\cI_s(t)$ is a ``hybrid'' confidence interval that may not correspond to any leaf in particular, but is expected to be smaller. Indeed it is not hard to prove by induction that starting from the same leaf intervals, $\cI_s(t) \subseteq \cI_s^{\text{Rep}}(t)$ for all $s\in \cT$ hence Proposition~\ref{prop:CorrectCI} is still satisfied for the confidence intervals $\cI^{\text{Rep}}_s(t)$.
% \end{remark}
\subsection{The BAI-MCTS architecture}
In this section we present the generic BAI-MCTS algorithm, whose sampling rule combines two ingredients: a best arm identification step which selects an action at the root, followed by a \emph{confidence based exploration step}, that goes down the tree starting from this depth-one node in order to select the \emph{representative leaf} for evaluation.
The structure of a BAI-MCTS algorithm is presented in Figure~\ref{algo:BAIMCTS}. The algorithm depends on a Best Arm Identification (BAI) algorithm, and uses the three components of this algorithm:
\begin{itemize}
\item the sampling rule $\mathrm{BAIStep}(\cS)$ selects an arm in the set $\cS$
\item the stopping rule $\mathrm{BAIStop}(\cS)$ returns $\texttt{True}$ if the algorithm decides to stop
\item the recommendation rule $\mathrm{BAIReco}(\cS)$ selects an arm as a candidate for the best arm
\end{itemize}
%All the decisions are made based on the information available for the arms in $\cS$, and possibly the number of rounds $t$ as well.
In BAI-MCTS, the arms are the depth-one nodes, hence the information needed by the BAI algorithm to make a decision (e.g. \texttt{BAIStep} for choosing an arm, or \texttt{BAIStop} for stopping) is information about depth-one nodes, that has to be updated at the end of each round (last line in the \texttt{while} loop). Different BAI algorithms may require different information, and we now present two instances that rely on confidence intervals (and empirical estimates) for the value of the depth-one nodes.
\subsection{UGapE-MCTS and LUCB-MCTS}
Several Best Arm Identification algorithms may be used within BAI-MCTS, and we now present two variants, that are respectively based on the UGapE \cite{Gabillon:al12} and the LUCB \cite{Shivaramal12} algorithms. These two algorithms are very similar in that they exploit confidence intervals and use the same stopping rule, however the LUCB algorithm additionally uses the empirical means of the arms, which within BAI-MCTS requires defining an estimate $\hat{V}_s(t)$ of the value of the depth-one nodes.
%One could also try to use the Track-and-Stop algorithm of \cite{GK16}. However, this algorithm does not exploit confidence intervals, it rather relies on the empirical means and the \emph{number of draws} of each arm. In our MCTS scenario, it is not clear what the counterpart of the ``number of draws'' of a depth-one node should be, as the estimate $\hat{V}_s(t)$ is no longer based on i.i.d. samples of mean $V_s$.
The generic structure of the two algorithms is similar. At round $t+1$ two promising depth-one nodes are computed, that we denote by $\underline{b}_t$ and $\underline{c}_t$. Among these two candidates, the node whose confidence interval is the largest (that is, the most uncertain node) is selected:
\[R_{t+1} = \argmax{i \in \{\underline{b}_t,\underline{c}_t\}} \ \left[U_i(t) - L_i(t)\right].\]
Then, following the BAI-MCTS architecture, the representative leaf of $R_{t+1}$ (computed by going down the tree) is sampled: $L_{t+1} = \ell_{R_{t+1}}(t)$. The algorithm stops whenever the confidence intervals of the two promising arms overlap by less than $\epsilon$:
\[\tau = \inf \left\{t \in \N : U_{\underline{c}_t}(t) - L_{\underline{b}_t}(t) < \epsilon \right\},\]
and it recommends $\hat{s}_\tau = \underline{b}_\tau$.
In both algorithms that we detail below $\underline{b}_t$ represents a guess for the best depth-one node, while $\underline{c}_t$ is an ``optimistic'' challenger, that has the maximal possible value among the other depth-one nodes. Both nodes need to be explored enough in order to discover the best depth-one action quickly.
\paragraph{UGapE-MCTS.} In UGapE-MCTS, introducing for each depth-one node the index
\[
B_s(t) = \max_{s' \in \cC(s_0) \backslash\{s\}} U_{s'}(t) - L_s(t),
\]
the promising depth-one nodes are defined as
\[\underline{b}_t = \argmin{a \in \cC(s_0)} \ B_a(t) \ \ \ \text{and} \ \ \ \underline{c}_t = \argmax{b \in \cC(s_0) \backslash \{\underline{b}_t\}} \ U_b(t).\]
\paragraph{LUCB-MCTS.} In LUCB-MCTS, the promising depth-one nodes are defined as
\[\underline{b}_t = \argmax {a \in \cC(s_0)} \ \hat{V}_a(t) \ \ \ \text{and} \ \ \ \underline{c}_t = \argmax{b \in \cC(s_0) \backslash \{\underline{b}_t\}} \ U_b(t),\]
where $\hat{V}_s(t) = \hat{\mu}_{\ell_s(t)}(t)$ is the empirical mean of the reprentative leaf of node $s$. Note that several alternative definitions of $\hat{V}_s(t)$ may be proposed (such as the middle of the confidence interval $\cI_s(t)$, or $\max_{a \in \cC(s)} \hat{V}_a(t)$), but our choice is crucial for the analysis of LUCB-MCTS, given in Appendix~\ref{sec:AnalysisLUCB}.
\section{Analysis of UGapE-MCTS}
In this section we first prove that UGapE-MCTS and LUCB-MCTS are both ($\epsilon,\delta$)-correct. Then we give in Theorem~\ref{thm:MCTSSC} a high-probability upper bound on the number of samples used by UGapE-MCTS. A similar upper bound is obtained for LUCB-MCTS in Theorem~\ref{thm:MCTSSCLUCB}, stated in Appendix~\ref{sec:AnalysisLUCB}.
\subsection{Choosing the Confidence Intervals}
From now on, we assume that the confidence intervals on the leaves are of the form
\begin{eqnarray}\label{eq:cis}
\LCB_\ell(t)&=& \hat{\mu}_\ell(t) - \sqrt{\frac{\beta(N_\ell(t),\delta)}{2N_\ell(t)}} \ \ \ \text{and} \ \ \ \UCB_\ell(t) = \hat{\mu}_\ell(t) + \sqrt{\frac{\beta(N_\ell(t),\delta)}{2N_\ell(t)}}.
\end{eqnarray}
$\beta(s,\delta)$ is some \emph{exploration function}, that can be tuned to have a $\delta$-PAC algorithm, as expressed in the following lemma, whose proof can be found in Appendix~\ref{proof:PAC}
\begin{lemma}\label{lem:PAC} If $\delta \leq \max(0.1 |\cL|,1)$, for the choice
\begin{equation}\beta(s,\delta) = \ln (|\cL|/\delta) + 3 \ln\ln(|\cL|/\delta) + (3/2) \ln (\ln s+1)\label{def:ExploRate}\end{equation}
both UGapE-MCTS and LUCB-MCTS satisfy $\bP(V(s^*) - V(\hat{s}_\tau) \leq \epsilon ) \geq 1-\delta$.
\end{lemma}
An interesting practical feature of these confidence intervals is that they only depend on the local number of draws $N_\ell(t)$, whereas most of the BAI algorithms use exploration functions that depend on the number of rounds $t$. Hence the only confidence intervals that need to be updated at round $t$ are those of the ancestors of the selected leaf, which can be done recursively.
Moreover, $\beta(s,\delta)$ scales with $\ln(\ln(s))$, and not $\ln(s)$, leveraging some tools recently introduced to obtain tighter confidence intervals \cite{Jamiesonal14LILUCB,JMLR15}.
The union bound over $\cL$ (that may be an artifact of our current analysis) however makes the exploration function of Lemma~\ref{lem:PAC} still a bit over-conservative and in practice, we recommend the use of $\beta(s,\delta) = \ln\left({\ln(e s)}/{\delta}\right)$.
Finally, similar correctness results (with slightly larger exploration functions) may be obtained for confidence intervals based on the Kullback-Leibler divergence (see \cite{KLUCBJournal}), which are known to lead to better performance in standard best arm identification problems \cite{COLT13} and also depth-two tree search problems \cite{GKK16}. However, the sample complexity analysis is much more intricate, hence we stick to the above Hoeffding-based confidence intervals for the next section.
% WMK: Done
%\todo[inline]{Remove this last paragraph, may allude to it in the experimental section}
\subsection{Complexity term and sample complexity guarantees}
We first introduce some notation. Recall that $s^*$ is the optimal action at the root, identified with the depth-one node satisfying $V(s^*)=V(s_0)$, and define the second-best depth-one node as $s^*_2 = \text{argmax}_{s \in \cC(s_0) \backslash \{s^*\}} \ V_s$. Recall $\Par(s)$ denotes the parent of a node $s$ different from the root. Introducing furthermore the set $\Anc(s)$ of all the ancestors of a node $s$, we define the complexity term by
\begin{equation}H^*_\epsilon(\bm\mu) := \sum_{\ell \in \cL} \frac{1}{\Delta_\ell^2\vee \Delta_*^2 \vee \epsilon^2}, \ \ \ \text{where} \ \ \ \begin{array}{ccl} \Delta_* & : = & V(s^*) - V(s^*_2) \\\Delta_\ell &:=& \max_{s \in \Anc(\ell) \backslash \{s_0 \}} |V_s - V(\Par(s))|\end{array}
\label{def:Delta}\end{equation}
The intuition behind these squared terms in the denominator is the following. We will sample a leaf $\ell$ until we either prune it (by determining that it or one of its ancestors is a bad move), prune everyone else (this happens for leaves below the optimal arm) or reach the required precision $\epsilon$.
% For depth-two tree, the complexity term becomes
% \[\sum_{j=1}^L \frac{1}{(\mu_{1,1} - \mu_{2,1})^2\vee (\mu_{1,j}-\mu_{1,1})^2 \vee \epsilon^2} + \sum_{i=2}^K\sum_{j=1}^L \frac{1}{(\mu_{1,1} - \mu_{2,1})^2\vee (\mu_{i,j} - \mu_{i,1})^2 \vee \epsilon^2},\]
% which improves over the work of \cite{GKK16}.
\begin{theorem}\label{thm:MCTSSC} Let $\delta \leq \min(1,0.1|\cL|)$. UGapE-MCTS using the exploration function~\eqref{def:ExploRate} is such that, with probability larger than $1-\delta$, $\left(V(s^*) - V(\hat{s}_\tau) < \epsilon\right)$ and, letting $\overline{\Delta}_{\ell,\epsilon}=\Delta_\ell \vee \Delta_* \vee \epsilon$,
\begin{align*}
\tau
~\le~ &
8 H^*_\epsilon(\bm\mu) \ln \frac{|\mathcal L|}{\delta} +\sum_\ell \frac{16}{\overline{\Delta}_{\ell,\epsilon}^2} \ln\ln \frac{1}{\overline{\Delta}_{\ell,\epsilon}^2} \\
& + \ \ 8H^*_\epsilon(\bm\mu) \left[3\ln \ln \frac{|\mathcal L|}{\delta} +
2 \ln \ln \left(8 e \ln \frac{|\mathcal L|}{\delta} + 24e \ln \ln \frac{|\mathcal L|}{\delta}\right)\right] + 1.
\end{align*}
\end{theorem}
%
% \begin{theorem}\label{thm:MCTSSC} Let $\delta \leq \min(1,0.1|\cL|)$. UGapE-MCTS using the exploration function~\eqref{def:ExploRate} is such that, with probability larger than $1-\delta$, $\left(V(s^*) - V(\hat{s}_\tau) < \epsilon\right)$ and
% \[\tau \leq 20H_\epsilon^*(\bm\mu) \left[\ln\left(\frac{20H_\epsilon^*(\bm\mu)|\cL|}{\delta}\right) + \ln\ln\left(\frac{20H_\epsilon^*(\bm\mu)|\cL|}{\delta}\right) + 1\right].\]
% \end{theorem}
\begin{remark}
If $\beta(N_a(t),\delta)$ is changed to $\beta(t,\delta)$, one can still prove $(\epsilon,\delta)$ correctness and furthermore upper bound the expectation of $\tau$. However the algorithm becomes less efficient to implement, since after each leaf observation, ALL the confidence intervals have to be updated. In practice, this change lowers the probability of error but does not effect significantly the number of play-outs used. %It may be interesting to check in practice which type of CI yield the best performance.
\end{remark}
%\todo[inline]{Should we also generalize our elimination-based algorithm? It is still a bit different from Teraoka's paper... FUTURE WORK }
\subsection{Comparison with previous work}\label{sec:comparison}
To the best of our knowledge\footnote{In a recent paper, \cite{HuangASM17} independently proposed the LUCBMinMax algorithm, that differs from UGapE-MCTS and LUCB-MCTS only by the way the best guess $\underline{b}_t$ is picked. The analysis is very similar to ours, but features some refined complexity measure, in which $\Delta_\ell$ (that is the maximal distance between \emph{consecutive} ancestors of the leaf, see~\eqref{def:Delta}) is replaced by the maximal distance between \emph{any} ancestors of that leaf. Similar results could be obtained for our two algorithms following the same lines. %Using similar technique, we can improve our analysis to feature this new complexity measure.
}, the $\mathrm{FindTopWinner}$ algorithm \cite{Teraoka14MCTS} is the only algorithm from the literature designed to solve the best action identification problem in any-depth trees. The number of play-outs of this algorithm is upper bounded with high probability by
\[\sum_{\ell : \Delta_\ell > 2\epsilon}\left(\frac{32}{\Delta_\ell^2}\ln\frac{16|\cL|}{\Delta_\ell\delta} + 1\right) + \sum_{\ell : \Delta_\ell \leq 2\epsilon}\left(\frac{8}{\epsilon^2}\ln\frac{8|\cL|}{\epsilon\delta} + 1\right)\]
One can first note the improvement in the constant in front of the leading term in $\ln(1/\delta)$, as well as the presence of the $\ln\ln (1/\overline{\Delta_{\ell,\epsilon^2}})$ second order term, that is unavoidable in a regime in which the gaps are small \cite{Jamiesonal14LILUCB}. The most interesting improvement is in the control of the number of draws of $2\epsilon$-optimal leaves (such that $\Delta_\ell \leq 2\epsilon$). In UGapE-MCTS, the number of draws of such leaves is at most of order $(\epsilon \vee \Delta_*^2)^{-1} \ln(1/\delta)$, which may be significantly smaller than $\epsilon^{-1}\ln(1/\delta)$ if there is a gap in the best and second best value. Moreover, unlike $\mathrm{FindTopWinner}$ and M-LUCB \cite{GKK16} in the depth two case, UGapE-MCTS can also be used when $\epsilon=0$, with provable guarantees.
Regarding the algorithms themselves, one can note that M-LUCB, an extension of LUCB suited for depth-two tree, does \emph{not} belong to the class of BAI-MCTS algorithms. Indeed, it has a ``reversed'' structure, first computing the representative leaf for each depth-one node: $\forall s \in \cC(s_0), R_{s,t} = \ell_s(t)$ and then performing a BAI step over the representative leaves: $\tilde{L}_{t+1} = \mathrm{BAIStep}({R_{s,t}, s \in \cC(s_0)})$. This alternative architecture can also be generalized to deeper trees, and was found to have empirical performance similar to BAI-MCTS. M-LUCB, which will be used as a benchmark in Section~\ref{sec:Experiments}, also distinguish itself from LUCB-MCTS by the fact that it uses an exploration rate that depends on the global time $\beta(t,\delta)$ and that $\underline{b}_t$ is the empirical maximin arm (which can be different from the arm maximizing $\hat{V}_s$). This alternative choice is not yet supported by theoretical guarantees in deeper trees.
Finally, the exploration step of BAI-MCTS algorithm bears some similarity with the UCT algorithm \cite{KocsisBBMCP06}, as it goes down the tree choosing alternatively the move that yields the highest UCB or the lowest LCB. However, the behavior of BAI-MCTS is very different at the root, where the first move is selected using a BAI algorithm. Another key difference is that BAI-MCTS relies on \emph{exact} confidence intervals: each interval $\cI_s(t)$ is shown to contain with high probability the corresponding value $V_s$, whereas UCT uses more heuristic confidence intervals, based on the number of visits of the parent node, and aggregating all the samples from descendant nodes. Using UCT in our setting is not obvious as it would require to define a suitable stopping rule, hence we don't include a comparison with this algorithm in Section~\ref{sec:Experiments}. A hybrid comparison between UCT and $\mathrm{FindTopWinner}$ is proposed in \cite{Teraoka14MCTS}, providing UCT with the random number of samples used by the the fixed-confidence algorithm. It is shown that $\mathrm{FindTopWinner}$ has the advantage for hard trees that require many samples. Our experiments show that our algorithms in turn always dominate $\mathrm{FindTopWinner}$.
\subsection{Proof of Theorem~\ref{thm:MCTSSC}.}
Letting $\cE_t = \bigcap_{\ell \in \cL} \left(\mu_\ell \in \cI_\ell(t)\right) \ \ \text{and} \ \ \cE = \bigcap_{t\in\N} \cE_t,$
we upper bound $\tau$ assuming the event $\cE$ holds, using the following key result, which is proved in Appendix~\ref{proof:UltraKey}.
\begin{lemma}\label{lem:UltraKey}Let $t \in \N$. $\cE_t \cap (\tau > t) \cap (L_{t+1} = \ell) \ \ \ \Rightarrow \ \ \ N_{\ell}(t) \leq \frac{8\beta(N_\ell(t),\delta)}{\Delta_\ell^2\vee \Delta_*^2 \vee \epsilon^2}$.\end{lemma}
An intuition behind this result is the following. First, using that the selected leaf $\ell$ is a representative leaf, it can be seen that the confidence intervals from $s_D=\ell$ to $s_0$ are nested (Lemma~\ref{lem:Rep1}). Hence if $\cE_t$ holds, $V(s_k) \in \cI_\ell(t)$ for all $k=1,\dots,D$, which permits to lower bound the width of this interval (and thus upper bound $N_\ell(t)$) as a function of the $V(s_k)$ (Lemma~\ref{lem:Rep2}). Then Lemma~\ref{lem:CsqceBAI} exploits the mechanism of UGapE to further relate this width to $\Delta_*$ and $\epsilon$.
Another useful tool is the following lemma, that will allow to leverage the particular form of the exploration function $\beta$ to obtain an explicit upper bound on $N_\ell(\tau)$.
\begin{lemma}\label{lem:loglog} Let $\beta(s) = C + \frac{3}{2}\ln(1+\ln(s))$ and define $S = \sup \{ s \geq 1 : a \beta(s) \geq s \}$. Then
\[S \leq aC + {2}a \ln(1+\ln(aC)).\]
\end{lemma}
This result is a consequence of Theorem~\ref{thm:loglog} stated in Appendix~\ref{proof:InvertingBounds}, that uses the fact that for $C \ge -\ln (0.1)$ and $a \ge 8$, it holds that
\[
\frac{3}{2}
\frac{C (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}
~\le~
1.7995564
~\le~
2
.
\]
On the event $\cE$, letting $\tau_\ell$ be the last instant before $\tau$ at which the leaf $\ell$ has been played before stopping, one has $N_\ell(\tau -1)=N_\ell(\tau_\ell)$ that satisfies by Lemma~\ref{lem:UltraKey}
\[N_{\ell}(\tau_\ell) \leq \frac{8\beta(N_\ell(\tau_\ell),\delta)}{\Delta_\ell^2\vee \Delta_*^2 \vee \epsilon^2}.\]
Applying Lemma~\ref{lem:loglog} with $a=a_\ell=\frac{8}{\Delta_\ell^2\vee \Delta_*^2 \vee \epsilon^2}$ and $C ~=~ \ln \frac{|\mathcal L|}{\delta} + 3 \ln \ln \frac{|\mathcal L|}{\delta}
$ leads to
\[
N_\ell(\tau-1) ~\le~
a_\ell \left( C +
2 \ln (1+\ln (a_\ell C))
\right).
\]
Letting $\overline{\Delta}_{\ell,\epsilon} = \Delta_\ell \vee \Delta_* \vee \epsilon$ and summing over arms, we find
\begin{align*}
\tau
&~=~
1 + \sum_\ell N_\ell(\tau-1) \\
&~\le~
1 + \sum_\ell \frac{8}{\overline{\Delta}_{\ell,\epsilon}^2} \left( \ln \frac{|\mathcal L|}{\delta} + 3 \ln \ln \frac{|\mathcal L|}{\delta} +
2 \ln \ln \left(8 e \frac{\ln \frac{|\mathcal L|}{\delta} + 3 \ln \ln \frac{|\mathcal L|}{\delta}}{\overline{\Delta}_{\ell,\epsilon}^2}\right)
\right)
\\
&~=~ 1 + \sum_\ell \frac{8}{\overline{\Delta}_{\ell,\epsilon}^2} \left( \ln \frac{|\mathcal L|}{\delta} + 2 \ln\ln \frac{1}{\overline{\Delta}_{\ell,\epsilon}^2}
\right) + 8H^*_\epsilon(\bm\mu) \left[3\ln \ln \frac{|\mathcal L|}{\delta} +
2 \ln \ln \left(8 e \ln \frac{|\mathcal L|}{\delta} + 24e \ln \ln \frac{|\mathcal L|}{\delta}\right)\right].
% \\
% &~\le~
% 1 + 8 H^*_\epsilon(\bm\mu) \left( \ln \frac{|\mathcal L|}{\delta} + 3 \ln \ln \frac{|\mathcal L|}{\delta} +
% 2 \ln \ln \left(8 e H^*_\epsilon(\bm\mu) (\ln \frac{|\mathcal L|}{\delta} + 3 \ln \ln \frac{|\mathcal L|}{\delta})\right)
% \right)
% .
\end{align*}
To conclude the proof, we remark that from the proof of Lemma~\ref{lem:PAC} (see Appendix~\ref{proof:PAC}) it follows that on $\cE$, $V(s^*) - V(\hat{s}_\tau) < \epsilon$ and that $\cE$ holds with probability larger than $1-\delta$.
%%%%%%% OLD BELOW
%
%
% Now let $T$ be a deterministic time. Using Lemma~\ref{lem:UltraKey} and the definition of the algorithm, one has
% \begin{eqnarray*}
% \min(\tau ,T) &=& \sum_{t=1}^T \ind_{(\tau > t)} \leq \sum_{t=1}^T \ind_{\left(N_{L_{t+1}} \leq \frac{8\beta(t,\delta)}{\Delta_{L_{t+1}}^2\vee \Delta_*^2 \vee \epsilon^2}\right)} = \sum_{\ell \in \cL} \sum_{t=1}^T\ind_{(L_{t+1}=\ell)}\ind_{\left(N_{\ell} \leq \frac{8\beta(T,\delta)}{\Delta_\ell^2\vee \Delta_*^2 \vee \epsilon^2}\right)} \\
% & \leq & \sum_{\ell \in \cL} \frac{8\beta(T,\delta)}{\Delta_\ell^2\vee \Delta_*^2 \vee \epsilon^2} = 8 H^*_\epsilon(\bm\mu)\beta(T,\delta).
% \end{eqnarray*}
% For any $T$ such that
% $8H_\epsilon^*(\bm\mu) \beta(T,\delta) < T$, one has $\min(\tau, T)< T$, which implies $\tau < T$. Therefore
% \[\tau \leq \inf \{T \in \N : \forall t \geq T, \ 8 H^*_\epsilon(\bm\mu) \beta(t,\delta) \leq t \}.\]
% Then, using the expression of the exploration function~\eqref{def:ExploRate}, it can be checked (using that $\ln(x) \leq x/e$ for all $x>0$) that $\beta(s,\delta) \leq ({5}/{2})\ln\left({|\cL|t}/{\delta}\right)$ for all $s,\delta$. Hence, if $\cE$ holds, $\tau$ is upper bounded by
% \[N^*(\bm\mu) = \inf\left\{ T \in \N : \forall t \geq T, 20 H^*_\epsilon(\bm\mu) \ln\left({t|\cL|}/{\delta}\right)\leq t \right\}.\]
% Simple arguments (see Lemma~\ref{lem:TechnicalCalculation} in Appendix~\ref{proof:TechnicalCalculation}) can then be used to further upper bound
% \[N^*(\bm\mu) \leq 20H_\epsilon^*(\bm\mu) \left[\ln\left(\frac{20H_\epsilon^*(\bm\mu)|\cL|}{\delta}\right) + \ln\ln\left(\frac{20H_\epsilon^*(\bm\mu)|\cL|}{\delta}\right) + 1\right].\]
% To conclude the proof, we remark that from the proof of Lemma~\ref{lem:PAC} (see Appendix~\ref{proof:PAC}) it follows that on $\cE$, $V(s^*) - V(\hat{s}_\tau) < \epsilon$ and that $\cE$ holds with probability larger than $1-\delta$.
\section{Experimental Validation}\label{sec:Experiments}
In this section we evaluate the performance of our algorithms in three experiments. We evaluate on the depth-two benchmark tree from \cite{GKK16}, a new depth-three tree and the random tree ensemble from \cite{Teraoka14MCTS}.
%
We compare to the $\mathrm{FindTopWinner}$ algorithm from \cite{Teraoka14MCTS} in all experiments, and in the depth-two experiment we include the M-LUCB algorithm from \cite{GKK16}. Its relation to BAI-MCTS is discussed in Section~\ref{sec:comparison}.
%
For our BAI-MCTS algorithms and for M-LUCB we use the exploration rate $\beta(s, \delta) = \ln \frac{|\cL|}{\delta} + \ln(\ln(s)+1)$ (a stylized version of Lemma~\ref{lem:PAC} that works well in practice), and we use the KL refinement of the confidence intervals \eqref{eq:cis}.
%
To replicate the experiment from \cite{Teraoka14MCTS}, we supply all algorithms with $\delta=0.1$ and $\epsilon = 0.01$. For comparing with \cite{GKK16} we run all algorithms with $\epsilon = 0$ and $\delta = 0.1 |\cL|$ (undoing the conservative union bound over leaves. This excessive choice, which might even exceed one, does not cause a problem, as the algorithms depend on $\frac{\delta}{|\cL|} = 0.1$). In none of our experiments the observed error rate exceeds $0.1$.
Figure~\ref{fig:cmp.GKK16} shows the benchmark tree from \cite[Section~5]{GKK16} and the performance of four algorithms on it. We see that the special-purpose depth-two M-LUCB performs best, very closely followed by both our new arbitrary-depth LUCB-MCTS and UGapE-MCTS methods. All three use significantly fewer samples than $\mathrm{FindTopWinner}$. Figure~\ref{fig:newtree} (displayed in Appendix~\ref{sec:ExtraFigure} for the sake of readability) shows a full 3-way tree of depth 3 with leafs drawn uniformly from $[0,1]$. Again our algorithms outperform the previous state of the art by an order of magnitude. Finally, we replicate the experiment from \cite[Section 4]{Teraoka14MCTS}. To make the comparison as fair as possible, we use the proven exploration rate from \eqref{def:ExploRate}. On 10K full 10-ary trees of depth 3 with Bernoulli leaf parameters drawn uniformly at random from $[0,1]$ the average numbers of samples are: LUCB-MCTS 141811, UGapE-MCTS 142953 and FindTopWinner 2254560. To closely follow the original experiment, we do apply the union bound over leaves to all algorithms, which are run with $\epsilon=0.01$ and $\delta=0.1$. We did not observe any error from any algorithm (even though we allow 10\%). Our BAI-MCTS algorithms deliver an impressive 15-fold reduction in samples.
\begin{figure}[h]
\centering
\includegraphics[width=.9\textwidth]{GKK16plusMLUCB}
\caption{The $3 \times 3$ tree of depth 2 that is the benchmark in \cite{GKK16}. Shown below the leaves are the average numbers of pulls for 4 algorithms: LUCB-MCTS (0.89\% errors, 2460 samples), UGapE-MCTS (0.94\%, 2419), $\mathrm{FindTopWinner}$ (0\%, 17097) and M-LUCB (0.14\%, 2399). All counts are averages over 10K repetitions with $\epsilon=0$ and $\delta=0.1 \cdot 9$.
% \todo[inline]{Is the multi-level $w=0$ pattern apparent?}
}\label{fig:cmp.GKK16}
\end{figure}
\section{Lower bounds and discussion}
Given a tree $\cT$, a MCTS model is parameterized by the leaf values, $\bm\mu := (\mu_{\ell})_{\ell \in \cL}$, which determine the best root action: $s^*=s^*(\bm\mu)$. For $\bm \mu \in [0,1]^{|\cL|}$, We define $\mathrm{Alt}(\bm\mu) = \{ \bm \lambda \in [0,1]^{|\cL|} : s^*(\bm\lambda) \neq s^*(\bm\mu)\}$. Using the same technique as \cite{GK16} for the classic best arm identification problem, one can establish the following (non explicit) lower bound. The proof is given in Appendix~\ref{sec:proofLB}.
\begin{theorem}\label{thm:LBgene} Assume $\epsilon=0$. Any $\delta$-correct algorithm satisfies
\begin{equation}\bE_{\bm\mu}[\tau] \geq T^*(\bm\mu) d(\delta,1-\delta), \ \ \text{where} \ \ \ T^*(\bm\mu)^{-1} := \sup_{\bm w \in \Sigma_{|\cL|}}\inf_{\bm \lambda \in \mathrm{Alt}(\bm\mu)} \sum_{\ell \in \cL} w_\ell d\left(\mu_\ell,\lambda_{\ell}\right)\label{def:supremumT}\end{equation}
with $\Sigma_k = \{ \bm w \in [0,1]^i : \sum_{i=1}^k w_i = 1\}$ and $d(x,y) = x\ln(x/y) + (1-x) \ln((1-x)/(1-y))$ is the binary Kullback-Leibler divergence.
\end{theorem}
This result is however not directly amenable for comparison with our upper bounds, as the optimization problem defined in Lemma~\ref{thm:LBgene} is not easy to solve. Note that $d(\delta,1-\delta) \geq \ln(1/(2.4\delta))$ \cite{JMLR15}, thus our upper bounds have the right dependency in $\delta$. For depth-two trees with $K$ (resp. $M$) actions for player A (resp. B), we can moreover prove the following result, that suggests an intriguing behavior.
\begin{lemma} \label{thm:LBdepth2}Assume $\epsilon=0$ and consider a tree of depth two with $\bm\mu =(\mu_{i,j})_{1 \leq i \leq K, 1 \leq j \leq M}$ such that $\forall (i,j) , \mu_{1,1} > \mu_{i,1}, \ \mu_{i,1} < \mu_{i,j}$. The supremum in the definition of $T^*(\bm \mu)^{-1}$ can be restricted to %the set
\[\tilde{\Sigma}_{K,M} := \{ \bm w \in \Sigma_{K\times M} : w_{i,j} = 0 \ \text{if} \ \ i\geq 2 \ \text{and} \ j \geq 2\}\]
\vspace{-0.4cm}
and
\vspace{-0.6cm}
\[T^*(\bm\mu)^{-1}\! =\!\!\max_{\bm w \in \tilde{\Sigma}_{K,M}} \!\min_{\substack{i=2,\dots,K \\a=1,\dots,M}} \left[w_{1,a} d \left(\mu_{1,a},\frac{w_{1,a}\mu_{1,a} + w_{i,1}\mu_{i,1}}{w_{1,a} + w_{i,1}} \right) \! + \! w_{i,1}d\left(\mu_{i,1},\frac{w_{1,a}\mu_{1,a} + w_{i,1}\mu_{i,1}}{w_{1,a} + w_{i,1}}\right)\right].\]
\end{lemma}
It can be extracted from the proof of Theorem~\ref{thm:LBgene} (see Appendix~\ref{sec:proofLB}) that the vector $\bm w^*(\bm\mu)$ that attains the supremum in \eqref{def:supremumT} represents the average proportions of selections of leaves by any algorithm matching the lower bound. Hence, the sparsity pattern of Lemma~\ref{thm:LBdepth2} suggests that matching algorithms should draw many of the leaves \emph{much less than $O(\ln(1/\delta))$ times}. This hints at the exciting prospect of optimal stochastic pruning, at least in the asymptotic regime $\delta \to 0$.
As an example, we numerically solve the lower bound optimization problem (which is a concave maximization problem) for $\bm \mu$ corresponding to the benchmark tree displayed in Figure~\ref{fig:cmp.GKK16} to obtain
\[
T^*(\bm \mu) = 259.9
\qquad
\text{and}
\qquad
\bm{w}^* ~=~ (0.3633, 0.1057, 0.0532), (0.3738, \bm 0, \bm 0), (0.1040, \bm 0, \bm 0)
.
\]
With $\delta = 0.1$ we find $\kl(\delta,1-\delta) = 1.76$ and the lower bound is $\bE_{\bm\mu}[\tau] \ge 456.9$. We see that there is a potential improvement of at least a factor $4$.
\paragraph{Future directions}
An (asymptotically) optimal algorithm for BAI called Track-and-Stop was developed by \cite{GK16}. It maintains the empirical proportions of draws close to $\bm w^*(\hat{\bm \mu})$, adding forced exploration to ensure $\hat{\bm \mu} \to \bm \mu$. We believe that developing this line of ideas for MCTS would result in a major advance in the quality of tree search algorithms. The main challenge is developing efficient solvers for the general optimization problem \eqref{def:supremumT}. For now, even the sparsity pattern revealed by Lemma~\ref{thm:LBdepth2} for depth two does not give rise to efficient solvers. We also do not know how this sparsity pattern evolves for deeper trees, let alone how to compute $\bm w^*(\bm \mu)$.
\paragraph{Acknowledgments.} Emilie Kaufmann acknowledges the support of the French Agence Nationale de la Recherche (ANR), under grants ANR-16-CE40-0002 (project BADASS) and ANR-13-BS01-0005 (project SPADRO). Wouter Koolen acknowledges support from the Netherlands Organization for Scientific Research (NWO) under Veni grant 639.021.439.
\bibliography{biblioBandits}
\appendix
\section{Numerical Results for a Depth Three Tree}\label{sec:ExtraFigure}
The results for our experiments on a depth-three tree are displayed in Figure~\ref{fig:newtree}.
\begin{sidewaysfigure}[hp]
\centering
\includegraphics[width=\textwidth]{pulls}
\caption{Our benchmark 3-way tree of depth 3. Shown below the leaves are the numbers of pulls of 3 algorithms: LUCB-MCTS (0.72\% errors, 1551 samples), UGapE-MCTS (0.75\%, 1584), and $\mathrm{FindTopWinner}$ (0\%, 20730). Numbers are averages over 10K repetitions with $\epsilon=0$ and $\delta=0.1 \cdot 27$.
% \todo[inline]{Is the multi-level $w=0$ pattern apparent?}
}\label{fig:newtree}
\end{sidewaysfigure}
\section{Confidence Intervals}
\subsection{Proof of Proposition~\ref{prop:CorrectCI}}\label{proof:CorrectCI} The proof proceeds by induction. Let the inductive hypothesis be $\cH_d$=``for all the nodes $s$ at (graph) distance $d$ from a leaf, $V_s \in \cI_s(t)$''.
$\cH_0$ clearly holds by definition of $\cE_t$. Now let $d$ such that $\cH_d$ holds and let $s$ be at distance $d+1$ of a leaf. Then all $s' \in \cC(s)$ are at distance at most $d$ from a leaf and using the inductive hypothesis,
\[\forall c \in \cC(s), \ \ \ \LCB_{c}(t) \leq V_c \leq \UCB_c(t).\]
Assume that $s$ is a MAX node. Using that $\UCB_s(t) = \max_{c\in\cC(s)} \UCB_c(t)$, one has $c\in \cC(s)$, $V_c \leq \UCB_c(t) \leq \UCB_s(t)$. By definition, $V_s = \max_{c \in \cC(s)} V_c$, thus it follows that $V_s \leq \UCB_s(t)$. Still by definition of $V_s$, it holds that $\forall c \in \cC(s), \LCB_c(t) \leq V_c \leq V_s$ and finally, as $\LCB_s(t) = \max_{c \in \cC(s)} \LCB_c(t)$, $\LCB_s(t) \leq V_s \leq \UCB_s(t)$. A similar reasoning yields the same conclusion if $s$ is a MIN node, thus $\cH_{d+1}$ holds.
As the tree $\cT$ is finite, we conclude by induction that $\forall s \in \cT, V_s \in \cI_s(t)$. %For the alternative construction described above, denoting by $\cI^{\mathrm{Alt}}(t)$ the resulting confidence intervals, it is easily proved by induction that $\cI_s(t) \subseteq \cI_{s}^{\mathrm{Alt}}(t)$, thus the result follows.
\subsection{Proof of Lemma~\ref{lem:PAC}} \label{proof:PAC}
Let \[\cE_t = \bigcap_{\ell \in \cL} \left(\mu_\ell \in \cI_\ell(t)\right) \ \ \text{and} \ \ \cE = \bigcap_{t\in\N} \cE_t.\] Using Proposition~\ref{prop:CorrectCI}, on $\cE_t$, for all $s\in \cT$, $V_s \in \cI_s(t)$. If the algorithm stops at some time $t$, as
$\LCB_{\underline{b}_t}(t) > \UCB_{\underline{c}_t}(t) - \epsilon,$
the outputted action, $\hat{s}_\tau=\underline{b}_t$, satisfies
$\LCB_{\hat{s}_\tau}(t) > \UCB_{s'}(t) - \epsilon,$
for all $s' \neq \hat{s}_\tau$. As $\cE$ holds, one obtains
\[V(\hat{s}_\tau) \geq \max_{s'\neq \hat{s}_\tau} V(s') - \epsilon,\]
and $\hat{s}_\tau$ is an $\epsilon$-maximin action. Hence, the algorithm is correct on $\cE$. The error probability is thus upper bounded by
\begin{eqnarray*}
\bP\left(\cE^c\right) & \leq & \bP\left(\exists \ell \in \cL, \exists t \in \N : |\hat{\mu}_{\ell}(t) - \mu_\ell | > \sqrt{{\beta(N_\ell(t),\delta)}/({2N_\ell(t)})} \right) \\
& \leq & \sum_{\ell \in \cL} \bP\left(\exists s \in \N : 2s(\hat{\mu}_{\ell,s} - \mu_\ell)^2 > \beta(s,\delta)\right)\\
& \leq & 2 |\cL| \bP\left(\exists s \in \N : S_s > \sqrt{2\sigma^2 s \beta(s,\delta)}\right),
\end{eqnarray*}
where $S_s = X_1 + \dots + X_s$ is a martingale with $\sigma^2$-subgaussian increments, with $\sigma^2 = 1/4$. It was shown in \cite{JMLR15} that for $\delta \leq 0.1$, if
\[\beta(t,\delta) = \ln\left(\frac{1}{\delta}\right) + 3\ln \ln \left(\frac{1}{\delta}\right) + (3/2)\ln (\ln (s) +1),\]
one has $\bP\left(\exists s \in \N : S_s > \sqrt{2\sigma^2 s \beta(s,\delta)}\right)\leq \delta$, which concludes the proof.
\section{Sample complexity analysis of LUCB-MCTS} \label{sec:AnalysisLUCB}
We provide an analysis of a slight variant of LUCB-MCTS that may stop at even rounds only and for $t\in 2\N$ draws the representative leaf of the two promising depth-one nodes:
\begin{equation}L_{t+1} = \ell_{\underline{b}_{t}}(t) \ \ \ \text{and} \ \ \ L_{t+2} = \ell_{\underline{c}_{t}}(t).\label{2armsLUCB}\end{equation}
The stopping rule is then $\tau = \inf \left\{t \in 2\N^* : U_{\underline{c}_t}(t) - L_{\underline{b}_t}(t) < \epsilon \right\}$.
For this algorithm, our sample complexity guarantee features a slightly different complexity term. For a leaf $\ell = s_0 s_1 \dots s_D$, we first introduce
\[\tilde{\Delta}_\ell = \max_{s \in \Anc(\ell) \backslash \{s_0,s_1\}} |V(s) - V(\bP(s))|,\]
a quantity that differs from $\Delta_\ell$ only by the fact that the maximum doesn't take into account the gap between the root and the depth-one ancestor of $\ell$. Then $\tilde{H}^*_\epsilon(\bm\mu)$ is defined similarly as ${H}^*_\epsilon(\bm\mu)$ by
\[\tilde{H}^*_\epsilon(\bm\mu) = \sum_{\ell \in \cL} \frac{1}{\tilde{\Delta}^2_\ell \vee \Delta_*^2 \vee \epsilon^2}.\]
\begin{theorem}\label{thm:MCTSSCLUCB} Let $\delta \leq \min(1,0.1|\cL|)$. LUCB-MCTS using the exploration function~\eqref{def:ExploRate} and selecting the two promising leaves at each round is such that, with probability larger than $1-\delta$, $\left(V(s^*) - V(\hat{s}_\tau) < \epsilon\right)$ and
\[\tau \leq 16\tilde{H}^*_\epsilon(\bm\mu)\left[ \ln\frac{|\cL|}{\delta} + 3 \ln\ln\frac{|\cL|}{\delta} + 2 \ln\ln\left(16e\tilde{H}^*_\epsilon(\bm\mu) \left(\ln\frac{|\cL|}{\delta}+3\ln\ln\frac{|\cL|}{\delta}\right)\right)\right]. \]
\end{theorem}
\paragraph{Proof.} The analysis follows the same lines as that of UGapE-MCTS, yet it relies on a slightly different key result, proved in the next section. Letting $\cE_t = \cap_{\ell \in \cL} (\mu_\ell \in \cI_\ell(t))$ as in the proof of Theorem~\ref{thm:MCTSSC} and defining $\cE = \cap_{t\in 2\N^*} \cE_t$, one can state the following.
\begin{lemma}\label{lem:UltraKeyLUCB}Let $t \in 2\N$. \[\cE_t \cap (\tau > t) \ \ \ \Rightarrow \ \ \exists \ell \in \{L_{t+1},L_{t+2}\}: \ \ N_{\ell}(t) \leq \frac{8\beta(t,\delta)}{\tilde{\Delta}_\ell^2\vee\Delta_*^2\vee\epsilon^2}.\]\end{lemma}
Let $T$ be a deterministic time. We upper bound $\tau$ assuming the event $\cE$ holds. Using Lemma~\ref{lem:UltraKeyLUCB} and the fact that for every even $t$, $(\tau_\delta > t)=(\tau_\delta > t+1)$ by definition of the algorithm, one has
\begin{eqnarray*}
\min(\tau,T) &=& \sum_{t=0}^T \ind_{(\tau > t)} =2\sum_{\substack{t \in 2\N\\ t \leq T}} \ind_{(\tau_\delta > t)} =2 \sum_{\substack{t \in 2\N\\ t \leq T}} \ind_{\left(\exists \ell \in \{L_{t+1},L_{t+2}\} : N_\ell(t) \leq 2 \beta(t,\delta)/(\tilde{\Delta}_\ell^2\vee\Delta_*^2\epsilon^2)\right)} \\
& \leq & 2 \sum_{\substack{t \in 2\N\\ t \leq T}} \sum_{\ell \in \cL} \ind_{(L_{t+1}=\ell)\cup(L_{t+2}=\ell)}\ind_{(N_\ell(t) \leq 8\beta(T,\delta)/(\tilde{\Delta}_\ell^2\vee\Delta_*^2\epsilon^2))} \\
& \leq & 16\sum_{\ell \in \cL}\frac{1}{\tilde{\Delta}_\ell^2\vee\Delta_*^2\vee\epsilon^2} \beta(T,\delta) = 16 \tilde{H}^*_\epsilon(\bm\mu) \beta(T,\delta).
\end{eqnarray*}
For any $T$ such that
$16 \tilde{H}^*_\epsilon(\bm\mu) \beta(T,\delta) < T$, one has $\min(\tau, T)< T$, which implies $\tau < T$. Therefore
\[\tau \leq \sup \{t \in \N : 16 \tilde{H}^*_\epsilon(\bm\mu) \beta(t,\delta) \geq t \}.\]
Just like in the analysis of UGapE-MCTS, the conclusion now follows from Lemma~\ref{lem:loglog}, applied with $a=16\tilde{H}_\epsilon^*(\bm\mu)$ and $C=\ln(|\cL|/\delta)+3\ln\ln(|\cL|/\delta)$, which yields
\[\tau \leq 16\tilde{H}^*_\epsilon(\bm\mu)\left[ \ln\frac{|\cL|}{\delta} + 3 \ln\ln\frac{|\cL|}{\delta} + 2 \ln\ln\left(16e\tilde{H}^*_\epsilon(\bm\mu) \left(\ln\frac{|\cL|}{\delta}+3\ln\ln\frac{|\cL|}{\delta}\right)\right)\right]. \]
Using that $\cP(\cE) \geq 1 - \delta$ and that the algorithm is correct on $\cE$ yields the conclusion.
\section{Proof of Lemma~\ref{lem:UltraKey} and Lemma~\ref{lem:UltraKeyLUCB}} \label{proof:UltraKey}
We first state Lemma~\ref{lem:Rep2}, that holds for both UGapE and LUCB-MCTS and is a consequence of the definition of the exploration procedure. This result builds on the following lemma, that expresses the fact that along a path from the root to a representative leaf, the confidence intervals are nested.
\begin{lemma}\label{lem:Rep1} Let $t\in\N$ and $s_0,s_1, \dots ,s_D$ be a path from the root down to a leaf $\ell=s_D$.
\[(\ell_{s_1}(t)= s_D) \Rightarrow \left(\forall k = 2 ,\dots , D, \ \ \ \ \cI_{s_{k-1}}(t) \subseteq \cI_{s_k}(t)\right)\]
\end{lemma}
\begin{lemma}\label{lem:Rep2} Let $t\in\N$ and $s_0,s_1, \dots ,s_D$ be a path from the root down to a leaf $\ell=s_D$. If $\cE_t$ holds and $\ell$ is selected at round $t+1$ (UGapE) or if $t$ is even and $\ell \in \{L_{t+1},L_{t+2}\}$ (LUCB), then
\[\sqrt{\frac{2\beta(N_\ell(t),\delta)}{N_\ell(t)}} \geq \max_{k=2 \dots D} |V(s_k) - V(s_{k-1})|.\]
\end{lemma}
\paragraph{UGapE-MCTS: proof of Lemma~\ref{lem:UltraKey}.} The following lemma is specific to UGapE-MCTS. We let $s_0,s_1,\dots,s_D$ be a path down to a leaf $\ell = s_{D}$.
\begin{lemma}\label{lem:CsqceBAI} Let $t \in \N$. If $\cE_t$ holds and UGapE-MCTS has not stopped after $t$ observations, that is $(\tau > t)$,
\[(L_{t+1} = \ell) \Rightarrow \left(\sqrt{\frac{8\beta(N_\ell(t),\delta)}{N_\ell(t)}} \geq \max\left(\Delta_* , V(s_0)-V(s_1), \epsilon \right)\right)\]
\end{lemma}
Putting together Lemma~\ref{lem:Rep2} and Lemma~\ref{lem:CsqceBAI} and using that
\[\Delta_\ell = \max\left(V(s_0)-V(s_1),\max_{k=2\dots D} |V(s_k) - V(s_{k-1})|\right)\]
one obtains
\[\cE_t \cap (\tau > t) \cap (L_{t+1} = \ell) \ \Rightarrow \ \left(\sqrt{\frac{8\beta(N_\ell(t),\delta)}{N_\ell(t)}} \geq \max\left( \Delta_\ell , \Delta_*, \epsilon\right)\right),\]
which yields the proof of Lemma~\ref{lem:UltraKey} by inverting the bound.
\paragraph{LUCB-MCTS: proof of Lemma~\ref{lem:UltraKeyLUCB}.} The following lemma is specific to the LUCB-MCTS algorithm. It can be viewed as a generalization of Lemma 2 in \cite{Shivaramal12}.
\begin{lemma}\label{lem:CsqceBAILUCB} Let $t \in 2\N$ and let $\gamma \in [V(s_2^*),V(s^*)]$. If $\cE_t$ holds and LUCB-MCTS has not stopped after $t$ observations, that is $(\tau > t)$, then
\[ \exists \ell \in \{L_{t+1},L_{t+2}\}: (\gamma \in \cI_\ell(t)) \cap \left( \sqrt{\frac{2\beta(t,\delta)}{N_\ell(t)}} \geq \epsilon\right).\]
\end{lemma}
Choosing $\gamma = \frac{V(s^*) + V(s^*_2)}{2}$ and letting $s_\ell$ be the depth-one ancestor of $\ell$, on $\cE_t$ it holds that $V(s_\ell) \in \cI_\ell(t)$ (by Lemma~\ref{lem:Rep1}) and
\begin{eqnarray*}
(\gamma \in \cI_\ell(t)) & \Rightarrow & \left(\sqrt{\frac{2\beta(t,\delta)}{N_\ell(t)}}\geq |V(s_\ell) - \gamma |\right) \\
& \Rightarrow & \left(\sqrt{\frac{2\beta(t,\delta)}{N_\ell(t)}}\geq \frac{V(s^*) - V(s_2^*)}{2}\right).
\end{eqnarray*}
Recall $\Delta_* = V(s^*) - V(s_2^*)$. By Lemma~\ref{lem:CsqceBAILUCB}, on $\cE_t \cap (\tau >t)$, there exists $\ell \in \{L_{t+1},L_{t+2}\}$ such that
\begin{equation}\sqrt{\frac{8\beta(t,\delta)}{N_\ell(t)}}\geq \max({\Delta_*},\epsilon).\label{eq:Int1}\end{equation}
Moreover, noting that for a leaf $\ell = s_0,s_1,\dots,s_D$,
\[\tilde{\Delta}_\ell = \max_{k = 2,\dots,D} |V(s_k) - V(s_{k-1})|\]
a consequence of Lemma~\ref{lem:Rep2} is that for $\ell \in \{L_{t+1},L_{t+2}\}$,
\begin{equation}\sqrt{\frac{2\beta(t,\delta)}{N_\ell(t)}} \geq \tilde{\Delta}_\ell.\label{eq:Int2}\end{equation}
Combining \eqref{eq:Int1} and \eqref{eq:Int2} yields
\[\cE_t \cap (\tau > t) \ \Rightarrow \ \exists \ell \in \{L_{t+1},L_{t+2}\}: \ \ \sqrt{\frac{8\beta(t,\delta)}{N_\ell(t)}} \geq \max\left( \tilde{\Delta}_\ell , \Delta_*, \epsilon\right),\]
which yields the proof of Lemma~\ref{lem:UltraKeyLUCB} by inverting the bound.
\subsection{Proof of Lemma~\ref{lem:Rep1}.}
The leaf $\ell$ is the representative of the depth 1 node $s_1$, therefore the path $s_1,\dots,s_D$ is such that $c_{s_{k-1}}(t) = s_k$ for all $k=2,\dots,D$. Using the way the representative are build, we now show that
\[\forall k \in \{2,\dots,D\}, \ \ \ \cI_{s_{k-1}}(t) \subseteq \cI_{s_{k}}(t).\]
If $s_{k-1}$ is a MAX node, $\U_{s_{k-1}}(t) = \U_{s_k}(t)$ by definition and $\L_{s_{k-1}}(t) = \max_{s \in \cC(s_{k-1})} \L_{s}(t) \geq \L_{s_k}(t)$. Similarly, if $s_{k-1}$ is a MIN node, $\L_{s_{k-1}}(t) = \L_{s_k}(t)$ by definition and $\U_{s_{k-1}}(t) = \min_{s \in \cC(s_{k-1})} \U_{s}(t) \leq \U_{s_k}(t)$, so that in both cases $\cI_{s_{k-1}}(t) \subseteq \cI_{s_{k}}(t)$.
\subsection{Proof of Lemma~\ref{lem:Rep2}.}
Let $\ell \in \cL$ be a leaf that is sampled based on the information available at round $t$. In particular, as $\ell$ is a representative leaf of the depth 1 node $s_1$, the path $s_1,\dots,s_D$ is such that $c_{s_{k-1}}(t) = s_k$ for all $k=2,\dots,D$. Let $k =2,\dots,D$. If $s_{k-1} \in \{2,\dots,D\}$ is a MAX node, it holds by definition of the representative children that, for all $s' \in \cC(s_{k-1})$,
\[\UCB_{s_{k}}(t) \geq \UCB_{s'}(t).\]
Now, from Lemma~\ref{lem:Rep1} one has $\U_\ell(t) \geq \U_{s_k}(t)$ and from Proposition~\ref{prop:CorrectCI} as $\cE_t$ holds, one has
\begin{equation}
\forall s \in \cT, \ \ V_s \in \cI_s(t). \label{Useful:Correctness}
\end{equation}
Using these two ingredients yields
\begin{eqnarray*}
\UCB_{\ell}(t) & \geq & V(s') \\
\LCB_{\ell}(t) + 2\sqrt{\frac{\beta(N_\ell(t),\delta)}{2N_\ell(t)}} & \geq & V(s') \\
\LCB_{s_{k}}(t) + 2\sqrt{\frac{\beta(N_\ell(t),\delta)}{2N_\ell(t)}} & \geq & V(s') \\
V(s_k) + 2\sqrt{\frac{\beta(N_\ell(t),\delta)}{2N_\ell(t)}} & \geq & V(s').
\end{eqnarray*}
Thus
\begin{eqnarray*}
\sqrt{\frac{2\beta(N_\ell(t),\delta)}{N_\ell(t)}} &\geq& \max_{s' \in \cC(s_{k-1})} V(s') - V(s_k) = V(s_{k-1}) - V(s_k) \geq 0.
\end{eqnarray*}
If $s_{k-1}$ is a MIN node, a similar reasoning show that
\begin{eqnarray*}
\sqrt{\frac{2\beta(N_\ell(t),\delta)}{N_\ell(t)}} &\geq& V(s_{k}) - V(s_{k-1}) \geq 0.
\end{eqnarray*}
Putting everything together yields
\begin{eqnarray*}
\sqrt{\frac{2\beta(N_\ell(t),\delta)}{N_\ell(t)}} &\geq& \max_{k=2,\dots,D} |V(s_{k}) - V(s_{k-1})|.
\end{eqnarray*}
\subsection{Proof of Lemma~\ref{lem:CsqceBAI}.}
We first prove the following intermediate result, that generalizes Lemma 4 in \cite{Gabillon:al12}.
\begin{lemma}\label{lem:IntermediateLemma} For all $t \in \N^*$, the following holds
\begin{eqnarray*}
\text{if} \ R_{t+1} = \underline{b}_t & \text{then} & U_{\underline{c}_t}(t) \leq U_{\underline{b}_t}(t) \\
\text{if} \ R_{t+1} = \underline{c}_t & \text{then} & L_{\underline{c}_t}(t) \leq L_{\underline{b}_t}(t) \\
\end{eqnarray*}
\end{lemma}
%\todo[inline]{NB: this is THE lemma that is specific to UGapE and does NOT apply for LUCB}
\begin{proof}
Assume $\underline{c}_t$ is selected (i.e. $R_{t+1}=\underline{c}_t$) and $\L_{\underline{c}_t}(t) > \L_{\underline{b}_t}(t)$. As the confidence interval on $V(\underline{c}_t)$ is larger than the confidence intervals on $V(\underline{b}_t)$ ($\underline{c}_t$ is selected), this also yields $\U_{\underline{c}_t}(t) > \U_{\underline{b}_t}(t)$. Hence
\[B_{\underline{b}_t}(t) = \U_{\underline{c}_t}(t) - \L_{\underline{b}_t}(t) > \U_{\underline{b}_t}(t) - \L_{\underline{c}_t}(t).\]
Also, by definition of $\underline{c}_t$, $\U_{\underline{c}_t}(t) \geq \U_{b}(t)$. Hence
\[B_{\underline{b}_t}(t) > \max_{b \neq \underline{c}_t} \U_b(t) - \L_{\underline{c}_t}(t)= B_{\underline{c}_t}(t),\]
which contradicts the definition of $\underline{b}_t$. Thus, we proved by contradiction that $\L_{\underline{c}_t}(t) \geq \L_{\underline{b}_t}(t)$.
A similar reasoning can be used to prove that $R_{t+1} = \underline{b}_t \ \Rightarrow \ U_{\underline{c}_t}(t) \leq U_{\underline{b}_t}(t)$.
%\todo[inline]{Do it?}
\end{proof}
A simple consequence of Lemma~\ref{lem:IntermediateLemma} is the fact that, on $\cE_t \cap (\tau >t)$,
\begin{equation}
(L_{t+1} = \ell) \ \Rightarrow \ \left(\sqrt{\frac{2\beta(N_\ell(t),\delta)}{N_\ell(t)}} > \epsilon\right).
\label{target:EpsilonPart}
\end{equation}
Indeed, as the algorithm doesn't stop after $t$ rounds, it holds that $U_{\underline{c}_t}(t) - L_{\underline{b}_t}(t) >\epsilon$. If $\ell$ is the arm selected at round $t+1$, $\ell = \ell_{R_{t+1}}(t)$ and one can prove using Lemma~\ref{lem:IntermediateLemma} that $U_{{R}_{t+1}}(t) - L_{R_{t+1}}(t) >\epsilon$ (by distinguishing two cases). Finally, as $\cE_t$ holds, by Lemma~\ref{lem:Rep1}, $\cI_{R_{t+1}}(t) \subseteq \cI_{\ell}(t)$. Hence $U_{\ell}(t) - L_{\ell}(t) >\epsilon$, and \eqref{target:EpsilonPart} follows using the particular form of the confidence intervals.
To complete the proof, we now show that
\begin{equation}
(L_{t+1} = \ell) \ \Rightarrow \ \left(\sqrt{\frac{8\beta(N_\ell(t),\delta)}{N_\ell(t)}} > \max(\Delta_*,V(s_0)-V(s_1)\right).
\label{target:GapPart}
\end{equation}
by distinguishing several cases.
\paragraph{Case 1: $s^* \notin \Anc (\ell)$ and $R_{t+1} = \underline{c}_t$.} Using that the algorithm doesn't stop yields
\[\L_{\underline{c}_t}(t) - \U_{\underline{b}_t}(t) + 2 (\U_{\underline{c}_t} (t) - L_{\underline{c}_t} (t)) > \epsilon. \]
As $\cE_t$ holds, $L_{\underline{c}_t}(t) \leq V(\underline{c}_t) = V(s_1)$ and $U_{\underline{b}_t}(t) \geq V(\underline{b}_t)$.
Therefore, if $\underline{b}_t = s^*$ it holds that
\[2 (\U_{\underline{c}_t} (t) - \L_{\underline{c}_t} (t)) > V(s^*) - V(s_1) + \epsilon.\]
If $\underline{b}_t \neq s^*$, by definition of $\underline{c}_t$ one has
\[\U_{\underline{c}_t}(t) \geq \U_{s^*}(t) \geq V(s^*),\]
hence
\begin{eqnarray*}
\L_{\underline{c}_t}(t) + (\U_{\underline{c}_t}(t)-\L_{\underline{c}_t}(t)) & \geq & V(s^*) \\
V(s_1) + (\U_{\underline{c}_t}(t)-\L_{\underline{c}_t}(t)) & \geq & V(s^*).
\end{eqnarray*}
Thus, recalling that $V(s_0)=V(s^*)$, whatever the value of $\underline{b}_t$, one obtains
\[2(\U_{\underline{c}_t}(t)-\L_{\underline{c}_t}(t)) \geq V(s_0) - V(s_1).\]
From Lemma~\ref{lem:Rep1} the width of $\cI_{\underline{c}_t}(t)$ is upper bounded by the width of $\cI_{\ell}(t)$, hence
\begin{equation}2\left(\U_\ell(t) - \L_\ell(t)\right) \geq V(s_0) - V(s_1).\label{Case1}\end{equation}
\paragraph{Case 2: $s^* \notin \Anc (\ell)$ and $R_{t+1} = \underline{b}_t$.} As $s^* \neq \underline{b}_t$, by definition of $\underline{c}_t$ one has
\[\U_{\underline{c}_t}(t) \geq \U_{s^*}(t) \geq V(s^*).\]
Hence, using Lemma~\ref{lem:IntermediateLemma},
\[\U_{\underline{b}_t}(t) - \L_{\underline{b}_t}(t) \geq \U_{\underline{c}_t}(t) - \L_{\underline{b}_t}(t) \geq V(s^*) - V(s_1),\]
as $\cE_t$ holds. Finally, by Lemma~\ref{lem:Rep1},
\begin{equation}\U_{\ell}(t) - \L_{\ell}(t) \geq V(s_0) - V(s_1).\label{Case2}\end{equation}
\paragraph{Case 3: $s^* \in \Anc (\ell)$ and $R_{t+1} = \underline{b}_t$.} One has $\underline{b}_t = s^*$. Using that the algorithm doesn't stop yields
\[\L_{\underline{c}_t}(t) - \U_{\underline{b}_t}(t) + 2 (\U_{\underline{b}_t} (t) - \L_{\underline{b}_t} (t)) > \epsilon. \]
As $\cE_t$ holds, $\U_{\underline{b}_t}(t) \leq V(s^*)$ and $\L_{\underline{c}_t}(t) \leq V(\underline{c}_t) \leq V(s_2^*)$. Therefore, if $\underline{b}_t = s^*$ it holds that
\[2 (U_{\underline{b}_t} (t) - L_{\underline{b}_t} (t)) > V(s^*) - V(s_2^*) + \epsilon.\]
and by Lemma~\ref{lem:Rep1}
\begin{equation}2\left(\U_\ell(t) - \L_\ell(t)\right) \geq V(s^*) - V(s_2^*).\label{Case3}\end{equation}
\paragraph{Case 4: $s^* \in \Anc (\ell)$ and $R_{t+1} = \underline{c}_t$.} One has $\underline{c}_t = s^*$. Using Lemma~\ref{lem:IntermediateLemma} yields
\[\U_{\underline{c}_t}(t) - \L_{\underline{c}_t}(t) \geq \U_{\underline{c}_t}(t) - \L_{\underline{b}_t}(t) \geq V(s^*) - V(s_2^*),\]
as $\cE_t$ holds and $V(\underline{b}_t) \leq V(s_2^*)$. Finally, by Lemma~\ref{lem:Rep1},
\begin{equation}\U_{\ell}(t) - \L_{\ell}(t) \geq V(s^*) - V(s_2^*).\label{Case4}\end{equation}
Combining \eqref{Case1}-\eqref{Case4}, we see that in all four cases \[2(\U_{\ell}(t) - \L_{\ell}(t)) \geq \max(V(s^*) - V(s_2^*),V(s_0)-V(s_1)),\]
as for $s^* \notin \Anc(\ell)$, $V(s_0) - V(s_1) = V(s^*) - V(s_1) \geq V(s^*)-V(s_2^*)$, and for $s^* \in \Anc(\ell)$, $V(s_0)-V(s_1)=0$.
Using the expression of the confidence intervals and recalling that $\Delta_* = V(s^*) - V(s^*_2)$, one obtains
\[4\sqrt{\frac{\beta(N_\ell(t),\delta)}{2N_\ell(t)}} \geq \max(\Delta_*,V(s_0)-V(s_1))\]
which proves \eqref{target:GapPart}.
\subsection{Proof of Lemma~\ref{lem:CsqceBAILUCB}.}
Fix $\gamma \in [V(s_2^*),V(s^*)]$ and assume $\cE_t \cap (\tau > t)$ holds. We assume (by contradiction) that $\gamma$ doesn't belong to $\cI_{L_{t+1}}(t)$ nor to $\cI_{L_{t+2}}(t)$. There are four possibilities:
\begin{itemize}
\item $\L_{L_{t+1}}(t) > \gamma$ and $\L_{L_{t+2}}(t) > \gamma$. As $\cE_t$ holds and $L_{t+1}$ and $L_{t+2}$ are representative, it yields that there exists two nodes $s\in \cC(s_0)$ such that $V_s > \gamma$, which contradicts the definition of $\gamma$.
\item $\U_{L_{t+1}}(t) < \gamma$ and $\U_{L_{t+2}}(t) < \gamma$. From the definition of $\underline{c}_{t+1}$, it yields that for all $s \in \cC(s_0)$, $\U_{s}(t) < \gamma$ and as $\cE_t$ holds one obtains $V_s < \gamma$ for all $s\in \cC(s_0)$, which contradicts the definition of $\gamma$.
\item $\L_{L_{t+1}}(t) > \gamma$ and $\gamma > \U_{L_{t+2}}(t)$. This implies that $\L_{L_{t+1}}(t) > \U_{L_{t+2}}(t)$ and that $\L_{\underline{b}_{t}}(t) > \U_{\underline{c}_t}(t)$ (by Lemma~\ref{lem:Rep1} and the fact that $L_{t+1}$ and $L_{t+2}$ are representative leaves). This yields $(\tau \leq t)$ and a contradiction.
\item $\U_{L_{t+1}}(t) < \gamma$ and $\gamma < \L_{L_{t+2}}(t)$. This implies in particular that $\hat{\mu}_{L_{t+1}}(t) < \hat{\mu}_{L_{t+2}}(t)$. Thus $\hat{V}(\underline{b}_{t},t) < \hat{V}(\underline{c}_{t},t)$, which contradicts the definition of $\underline{b}_t$.
\end{itemize}
Hence, we just proved by contradiction that there exists $\ell \in \{L_{t+1},L_{t+2}\}$ such that $\gamma \in \cI_\ell(t)$. To prove Lemma~\ref{lem:CsqceBAILUCB}, it remains to establish the following three statements.
\begin{enumerate}
\item $\left(\gamma \in \cI_{L_{t+1}}(t)\right) \cap \left(\gamma \in \cI_{L_{t+2}}(t)\right) \ \Rightarrow \ \ \left(\exists \ell \in \{L_{t+1},L_{t+2}\} : \sqrt{\frac{2\beta(t,\delta)}{N_\ell(t)}} > \epsilon\right)$
\item $\left(\gamma \in \cI_{L_{t+1}}(t)\right) \cap \left(\gamma \notin \cI_{L_{t+2}}(t)\right) \ \Rightarrow \ \ \left( \sqrt{\frac{2\beta(t,\delta)}{N_{L_{t+1}}(t)}} > \epsilon\right)$
\item $\left(\gamma \notin \cI_{L_{t+1}}(t)\right) \cap \left(\gamma \in \cI_{L_{t+2}}(t)\right) \ \Rightarrow \ \ \left( \sqrt{\frac{2\beta(t,\delta)}{N_{L_{t+2}}(t)}} > \epsilon\right)$
\end{enumerate}
\textit{Statement 1}. As the algorithm doesn't stop, $\U_{\underline{c}_t}(t) - \L_{\underline{b}_t}(t)>\epsilon$. Hence
\begin{eqnarray*}
\U_{L_{t+2}}(t) - \L_{L_{t+1}}(t) &>& \epsilon \\
\hat{\mu}_{L_{t+2}}(t) + \sqrt{\frac{\beta(N_{L_{t+2}}(t),\delta)}{2N_{L_{t+2}}(t)}} - \hat{\mu}_{L_{t+1}}(t) + \sqrt{\frac{\beta(N_{L_{t+1}}(t),\delta)}{2N_{L_{t+1}}(t)}} &>& \epsilon \\
\sqrt{\frac{\beta(N_{L_{t+2}}(t),\delta)}{2N_{L_{t+2}}(t)}} + \sqrt{\frac{\beta(N_{L_{t+1}}(t),\delta)}{2N_{L_{t+1}}(t)}} &>& \epsilon
\end{eqnarray*}
using that by definition of $\underline{b}_t$, $\hat{\mu}_{L_{t+2}}(t)<\hat{\mu}_{L_{t+1}}(t)$. Hence, there exists $\ell \in \{L_{t+1},L_{t+2}\}$ such that
\begin{eqnarray*}
\sqrt{\frac{\beta(N_{\ell}(t),\delta)}{2N_{\ell}(t)}} > \frac{\epsilon}{2} \ \Rightarrow \ \sqrt{\frac{2\beta(t,\delta)}{N_{\ell}(t)}} >{\epsilon}.
\end{eqnarray*}
\textit{Statement 2}. We consider two cases and first assume that $\gamma \in \cI_{L_{t+1}}(t)$ and $\gamma \geq \U_{L_{t+2}}(t)$. Using the fact that the algorithm doesn't stop at round $t$, the following events hold
\begin{eqnarray*}
&& \left(\U_{L_{t+2}}(t) - \L_{L_{t+1}}(t) > \epsilon \right)\cap \left(\U_{L_{t+1}}(t) > \gamma\right) \cap \left(\U_{L_{t+2}}(t) \leq \gamma\right) \\
& \Rightarrow & \left(\U_{L_{t+2}}(t) - \L_{L_{t+1}}(t) > \epsilon \right) \cap \left(\U_{L_{t+1}}(t) > \gamma\right) \cap \left(\U_{L_{t+2}}(t) - \L_{L_{t+1}}(t) + \L_{L_{t+1}}(t) \leq \gamma\right) \\
& \Rightarrow & \left(\U_{L_{t+1}}(t) > \gamma\right) \cap \left(\L_{L_{t+1}}(t) \leq \gamma - \epsilon\right) \\
& \Rightarrow & \left(\U_{L_{t+1}}(t) - \L_{L_{t+1}}(t) > \epsilon\right) \\
& \Rightarrow & \left( \sqrt{\frac{2\beta(t,\delta)}{N_{L_{t+1}}(t)}} > \epsilon\right).
\end{eqnarray*}
The second case is $\gamma \in \cI_{L_{t+1}}(t)$ and $\gamma \leq \L_{L_{t+2}}(t)$. Then the following holds
\begin{eqnarray*}
&& \left(\U_{L_{t+2}}(t) - \L_{L_{t+1}}(t) > \epsilon \right)\cap \left(\L_{L_{t+1}}(t) \leq \gamma\right) \cap \left(\L_{L_{t+2}}(t) \geq \gamma\right) \\
& \Rightarrow & \left(\L_{L_{t+1}}(t) \leq \gamma\right) \cap \left(\U_{L_{t+2}}(t) + \L_{L_{t+2}}(t) - \L_{L_{t+1}}(t) > \gamma + \epsilon \right)\\
& \Rightarrow & \left(\L_{L_{t+1}}(t) \leq \gamma\right) \cap \left(2\hat{\mu}_{L_{t+2}}(t) -\hat{\mu}_{L_{t+1}}(t) + \sqrt{\frac{2\beta(N_{L_{t+1}}(t),\delta)}{2N_{L_{t+1}}(t)}} > \gamma + \epsilon \right)\\
& \Rightarrow & \left(\L_{L_{t+1}}(t) \leq \gamma\right) \cap \left(\hat{\mu}_{L_{t+1}}(t) + \sqrt{\frac{2\beta(N_{L_{t+1}}(t),\delta)}{2N_{L_{t+1}}(t)}} > \gamma + \epsilon \right)\\
& \Rightarrow & \left(\U_{L_{t+1}}(t) - \L_{L_{t+1}}(t) > \epsilon\right),
\end{eqnarray*}
where the third implication uses the fact that $\hat{\mu}_{L_{t+2}}(t) \leq \hat{\mu}_{L_{t+1}}(t)$.
\textit{Statement 3}. We consider two cases and first assume that $\gamma \in \cI_{L_{t+2}}(t)$ and $\gamma \leq \L_{L_{t+1}}(t)$. Using the fact that the algorithm doesn't stop at round $t$, the following events hold
\begin{eqnarray*}
&& \left(\U_{L_{t+2}}(t) - \L_{L_{t+1}}(t) > \epsilon \right)\cap \left(\L_{L_{t+1}}(t) \geq \gamma\right) \cap \left(\L_{L_{t+2}}(t) \leq \gamma\right) \\
& \Rightarrow & \left(\U_{L_{t+2}}(t) > \gamma + \epsilon \right)\cap \left(\L_{L_{t+2}}(t) \leq \gamma\right) \\
& \Rightarrow & \left(\U_{L_{t+2}}(t) - \L_{L_{t+2}}(t) > \epsilon\right) \\
& \Rightarrow & \left( \sqrt{\frac{2\beta(t,\delta)}{N_{L_{t+2}}(t)}} > \epsilon\right).
\end{eqnarray*}
The second case is $\gamma \in \cI_{L_{t+2}}(t)$ and $\gamma \geq \U_{L_{t+1}}(t)$. Then the following holds
\begin{eqnarray*}
&& \left(\U_{L_{t+2}}(t) - \L_{L_{t+1}}(t) > \epsilon \right)\cap \left(\U_{L_{t+2}}(t) \geq \gamma\right) \cap \left(\gamma \geq \U_{L_{t+1}}(t)\right) \\
& \Rightarrow & \left(\U_{L_{t+2}}(t) - (\L_{L_{t+1}}(t) + \U_{L_{t+1}}(t)) > \epsilon - \gamma \right)\cap \left(\U_{L_{t+2}}(t) \geq \gamma\right) \\
& \Rightarrow & \left(\hat{\mu}_{L_{t+2}}(t) + \sqrt{\frac{2\beta(N_{L_{t+2}}(t),\delta)}{2N_{L_{t+2}}(t)}}- 2 \hat{\mu}_{L_{t+1}}(t) > \epsilon - \gamma \right)\cap \left(\U_{L_{t+2}}(t) \geq \gamma\right)\\
& \Rightarrow & \left( - \hat{\mu}_{L_{t+2}}(t) + \sqrt{\frac{2\beta(N_{L_{t+2}}(t),\delta)}{2N_{L_{t+2}}(t)}} > \epsilon - \gamma \right)\cap \left(\U_{L_{t+2}}(t) \geq \gamma\right)
\\
& \Rightarrow & \left(\U_{L_{t+2}}(t) - \L_{L_{t+2}}(t) > \epsilon\right),
\end{eqnarray*}
where the third implication uses the fact that $\hat{\mu}_{L_{t+2}}(t) \leq \hat{\mu}_{L_{t+1}}(t)$.
\section{Proof of the lower bounds} \label{sec:proofLB}
\subsection{Proof of Theorem~\ref{thm:LBgene}}
Theorem~\ref{thm:LBgene} follows from considering the best possible change of distribution $\bm \lambda \in \mathrm{Alt}(\bm\mu)$. The expected log-likelihood ratio of the observations until $\tau$ under a model parameterized by $\bm\mu$ and a model parameterized by $\bm\lambda$ is
\[\bE_{\bm\mu}[L_\tau(\bm\mu,\bm\lambda)] = \sum_{\ell \in \cL} \bE_{\bm\mu}[N_\ell(\tau)] d(\mu_\ell,\lambda_\ell),\]
where $N_\ell(t)$ is the number of draws of the leaf $\ell$ until round $t$. Using Lemma 1 of \cite{JMLR15}, for any event $\cE$ in the filtration generated by $\tau$,
\[\bE_{\bm\mu}[L_\tau(\bm\mu,\bm\lambda)] \geq d(\bP_{\bm\mu}(\cE), \bP_{\bm\lambda}(\cE)).\]
As the strategy is $\delta$-correct, letting $\cE = (\hat{s}_\tau = s^*(\bm\mu))$ one has $\bP_{\bm\mu}(\cE) \geq 1 - \delta$ and $\bP_{\bm\lambda}(\cE) \leq \delta$ (under this model, $s^*(\bm\mu)$ is not the best action at the root under the model parameterized by $\bm\lambda$). Using monotonicity properties of the Bernoulli KL-divergence, one obtains, for any $\bm\lambda \in \mathrm{Alt}(\bm\mu)$,
\[\sum_{\ell \in \cL} \bE_{\bm\mu}[N_\ell(\tau)] d(\mu_\ell,\lambda_\ell) \geq d(1-\delta, \delta).\]
Then, one can write
\begin{eqnarray*}
\inf_{\bm\lambda \in \mathrm{Alt}(\bm\mu)} \sum_{\ell \in \cL} \bE_{\bm\mu}[N_\ell(\tau)] d(\mu_\ell,\lambda_\ell) &\geq & d(1-\delta, \delta) \\
\bE_{\bm\mu}[\tau] \inf_{\bm\lambda \in \mathrm{Alt}(\bm\mu)} \sum_{\ell \in \cL} \frac{\bE_{\bm\mu}[N_\ell(\tau)]}{\bE_{\bm\mu}[\tau]} d(\mu_\ell,\lambda_\ell) &\geq & d(1-\delta, \delta) \\
\bE_{\bm\mu}[\tau] \left(\sup_{\bm w \in \Sigma_{|\cL|}} \inf_{\bm\lambda \in \mathrm{Alt}(\bm\mu)} \sum_{\ell \in \cL} \frac{\bE_{\bm\mu}[N_\ell(\tau)]}{\bE_{\bm\mu}[\tau]} d(\mu_\ell,\lambda_\ell)\right) & \geq &d(1-\delta, \delta),
\end{eqnarray*}
using that $\sum_{\ell \in \cL} \frac{\bE_{\bm\mu}[N_\ell(\tau)]}{\bE_{\bm\mu}[\tau]} = 1$. This concludes the proof.
One can also note that for an algorithm to match the lower bound, all the inequalities above should be equalities. In particular one would need $w^*_{\ell}(\bm\mu) \simeq \frac{\bE_{\bm\mu}[N_\ell(\tau)]}{\bE_{\bm\mu}[\tau]}$, where $w^*_{\ell}(\bm\mu)$ is a maximizer in the definition of $T^*(\bm\mu)^{-1}$ in \eqref{def:supremumT}.
\subsection{Proof of Lemma~\ref{thm:LBdepth2}} In the particular case of a depth-two tree with $K$ actions for player A and $M$ actions for player B,
\[T^*(\bm\mu)^{-1} = \sup_{\bm w \in \Sigma_{K\times M}} \inf_{\bm\lambda \in \Alt(\bm\mu)}\sum_{k=1}^K \sum_{m=1}^K w_{k,m} d(\mu_{k,m},\lambda_{k,m}).\]
From the particular structure of $\bm\mu$, the best action at the root is action $i=1$. Hence
\[\Alt(\bm\mu) = \{\bm \lambda : \exists a \in \{1, \dots, M\}, \exists i \in \{2,\dots,K\} : \forall j \in \{1,\dots,M\}, \lambda_{1,a} < \lambda_{i,j}\}.\]
It follows that
\begin{eqnarray}T^*(\bm\mu)^{-1} & = & \sup_{\bm w \in \Sigma_{K\times M}} \min_{\substack{a \in \{1,\dots,M\} \\ i \in \{2,\dots,K\}}}\inf_{\bm\lambda : \forall j, \lambda_{1,a} < \lambda_{i,j}}\sum_{k=1}^K \sum_{m=1}^M w_{k,m} d(\mu_{k,m},\lambda_{k,m}) \nonumber\\
& = & \sup_{\bm w \in \Sigma_{K\times M}} \min_{\substack{a \in \{1,\dots,M\} \\ i \in \{2,\dots,K\}}}\underbrace{\inf_{\bm\lambda : \forall j, \lambda_{1,a} < \lambda_{i,j}}\left[w_{1,a}d(\mu_{1,a},\lambda_{1,a}) + \sum_{j=1}^M w_{i,j} d(\mu_{i,j},\lambda_{i,j})\right]}_{:=F_{a,i}(\bm\mu,\w)}.
\label{LB:ToBound}\end{eqnarray}
Indeed, in the rightmost constrained minimization problem, all the $\lambda_{k,m}$ on which no constraint lie can be set to $\mu_{k,m}$ to minimize the corresponding term in the sum. Using tools from constrained optimization, one can prove that
\[F_{a,i}(\bm\mu,\bm w) = \inf_{(\lambda_{1,a},(\lambda_{i,j})_j) \in \cC} \left[w_{1,a}d(\mu_{1,a},\lambda_{1,a}) + \sum_{j=1}^M w_{i,j} d(\mu_{i,j},\lambda_{i,j})\right],\]
where $\cC$ is the set
\[\cC = \{(\mu',\vmu') \in [0,1]^{M+1} : \exists j_0 \in \{1,\dots,K\}, \exists c \in [\mu_{i,j_0},\mu_{1,a}] : \mu'=\vmu'_{1} = \dots = \vmu'_{j_0}=c \ \ \text{and} \ \ \vmu'_{j} = \mu_{i,j} \ \ \text{for} \ j> j_0\}.\]
Letting $H(\mu',\bm \mu',w,\w) = w_{1,a} d(\mu_{1,a},\mu') + \sum_{j=1}^M w_{i,j} d(\mu_{i,j},\vmu'_j)$ one can easily show that for all $(\mu',\bm \mu') \in \cC$,
\[H(\mu',\vmu',w,\w) \leq H(\mu',\vmu',w,\tilde{\w}),\]
where $\tilde{\bm w}$ is constructed from $\bm w$ by putting all the weight on the smallest arm:
\[\tilde{w}_{i,1} = \sum_{j\geq 1}w_{1,j} \ \ \text{and} \ \ \tilde{w}_{i,j}=0 \ \ \text{for} \ \ j\geq 2.\]
This is because the largest $d(\mu_{i,j},c)$ is $d(\mu_{i,1},c)$ as $\mu_{i,1} \leq \mu_{i,j} \leq c$ for $j \leq j_0$. Hence taking the infimum, one obtains
\[F_{a,i}(\bm\mu,\bm w) \leq F_{a,i}(\bm\mu,\tilde{\w}).\]
Repeating this argument for all $i$, one can construct $\tilde{\bm w}$ such that
\[\forall i \geq 2, \tilde{w}_{i,1} = \sum_{j \geq 1} w_{i,j} \ \ \text{and} \ \ \tilde{w}_{i,j}=0 \ \ \text{for} \ j\geq 2\]
and $F_{a,i}(\bm \mu,\bm w) \leq F_{a,i}(\bm\mu,\tilde{\bm w})$ for all $a,i$. Thus, the supremum in \ref{LB:ToBound} is necessarily attained for $\bm w$ in the set $\tilde{\Sigma}_{K\times M} =\{\bm w \in \Sigma_{K,M} : w_{i,j} = 0 \ \text{for} \ i\geq 2, j\geq 2\}$. It follows that
\begin{eqnarray*}T^*(\bm\mu)^{-1} &=& \sup_{\bm w \in \tilde{\Sigma}_{M+K-1}} \min_{\substack{a \in \{1,\dots,M\} \\ i \in \{2,\dots,K\}}}\inf_{\bm\lambda : \forall j, \lambda_{1,a} < \lambda_{i,1}}\left[w_{1,a}d(\mu_{1,a},\lambda_{1,a}) + w_{i,1} d(\mu_{i,1},\lambda_{i,1})\right]\\
& = & \sup_{\bm w \in \tilde{\Sigma}_{K+M-1}} \min_{\substack{a=1,\dots,M \\ i=2,\dots,K}} \left[w_{1,a} d \left(\mu_{1,a},\frac{w_{1,a}\mu_{1,a} + w_{i,1}\mu_{i,1}}{w_{1,a} + w_{i,1}} \right) + w_{i,1}d\left(\mu_{i,1},\frac{w_{1,a}\mu_{1,a} + w_{i,1}\mu_{i,1}}{w_{1,a} + w_{i,1}}\right)\right],
\end{eqnarray*}
which concludes the proof.
\section{Inverting Bounds} \label{proof:InvertingBounds}
Consider the exploration rate
\[
\beta(s) ~=~ C + \frac{3}{2} \ln \left(1 + \ln s\right)
\qquad
\text{where}
\qquad
C ~=~ \ln \frac{|\mathcal L|}{\delta} + 3 \ln \ln \frac{|\mathcal L|}{\delta}
\]
where we assume $C \ge 1$, so that $\beta(1) = C \ge 1$. Now fix some $a \ge 1$, and let us define
\[
S ~=~ \sup \left\{s \ge 1 \middle\mid a \beta(s) \ge s \right\}
.
\]
The goal is to get a tight upper bound on $S$. We claim that
\begin{theorem}\label{thm:loglog}
\[
a C + \frac{3}{2} a \ln (1+\ln (a C))
~\le~
S
~\le~
a C +
\frac{3}{2} a \ln (1+\ln (a C))
\underbrace{
\frac{C (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}
}_{\to 1}
,
\]
where the upper bound is only non-trivial if
$
C \left(1+\ln (a C)\right) > \frac{3}{2}
$, which, for example, is implied by $C > 1.23696$.
\end{theorem}
\begin{proof}
The requirement $a \beta(s) \ge s$ is monotone in $s$, in that it holds for small $s$ (including $s=1$) and fails for large $s$. So to show the theorem it suffices to show that $a \beta(s) \ge s$ holds at $s$ given by the left-hand-side, while it fails for $s$ equal to the right-hand side.
First, we need to establish that $a \beta(s) \ge s$ for $s$ equal to the left-hand side expression of the theorem. That is, we need to show
\[
a \left(C + \frac{3}{2} \ln \left(1 + \ln \left(a \left( C+\frac{3}{2} \ln (1+\ln (a C))\right)\right)\right)\right)
~\ge~
a \left( C+\frac{3}{2} \ln (1+\ln (a C))\right)
\]
which is equivalent (the simplification is entirely mechanical) to
\[
a C
~\ge~
1
\]
which holds by assumption. Then we need to plug in the right hand side. Here we need to show that
\begin{multline*}
a \left(C + \frac{3}{2} \ln \left(1 + \ln \left(a C +
\frac{3}{2} a \ln (1+\ln (a C)) \frac{C (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}\right)\right)
\right)
\\
~\le~
a C +
\frac{3}{2} a \ln (1+\ln (a C)) \frac{C (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}},
\end{multline*}
that is
\[
\ln \left(1 + \ln \left(a C +
\frac{3}{2} a \ln (1+\ln (a C)) \frac{C (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}\right)\right)
~\le~
\ln (1+\ln (a C)) \frac{C (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}
.
\]
To show this, we use $\ln(1+x) \le x$ twice to show
\begin{align*}
& \ln \left(1 + \ln \left(a C +
\frac{3}{2} a \ln (1+\ln (a C)) \frac{C (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}\right)\right)
\\
&~=~
\ln \left(1 + \ln (a C) + \ln \left(1 +
\frac{3}{2} \ln (1+\ln (a C)) \frac{(1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}\right)\right)
\\
&~\le~
\ln \left(1 + \ln (a C) + \frac{3}{2} \ln (1+\ln (a C)) \frac{(1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}\right)
\\
&~=~
\ln \left(1 + \ln (a C)\right)
+
\ln \left(1 + \frac{\frac{3}{2} \ln (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}\right)
\\
&~\le~
\ln \left(1 + \ln (a C)\right)
+
\frac{\frac{3}{2} \ln (1+\ln (a C))}{C \left(1+\ln (a C)\right)-\frac{3}{2}}
\\
&~=~
\frac{C \left(1+\ln (a C)\right)}{C \left(1+\ln (a C)\right)-\frac{3}{2}} \ln \left(1 + \ln (a C)\right)
\end{align*}
as desired.
\end{proof}
\end{document}
\begin{thebibliography}{10}
\bibitem{Bubeck10BestArm}
J-Y. Audibert, S.~Bubeck, and R.~Munos.
\newblock {Best Arm Identification in Multi-armed Bandits}.
\newblock In {\em {Proceedings of the 23rd Conference on Learning Theory}},
2010.
\bibitem{Aueral02}
P.~Auer, N.~Cesa-Bianchi, and P.~Fischer.
\newblock {Finite-time analysis of the multiarmed bandit problem}.
\newblock {\em Machine Learning}, 47(2):235--256, 2002.
\bibitem{OMS14}
L.~Borsoniu, R.~Munos, and E.~P{\'a}ll.
\newblock An analysis of optimistic, best-first search for minimax sequential
decision making.
\newblock In {\em ADPRL14}, 2014.
\bibitem{SurveyMCTS12}
C.~Browne, E.~Powley, D.~Whitehouse, S.~Lucas, P.~Cowling, P.~Rohlfshagen,
S.~Tavener, D.~Perez, S.~Samothrakis, and S.~Colton.
\newblock A survey of monte carlo tree search methods.
\newblock {\em IEEE Transactions on Computational Intelligence and AI in
games,}, 4(1):1--49, 2012.
\bibitem{KLUCBJournal}
O.~Capp{\'e}, A.~Garivier, O-A. Maillard, R.~Munos, and G.~Stoltz.
\newblock {{K}ullback-{L}eibler upper confidence bounds for optimal sequential
allocation}.
\newblock {\em Annals of Statistics}, 41(3):1516--1541, 2013.
\bibitem{Cazenave15SHOT}
T.~Cazenave.
\newblock Sequential halving applied to trees.
\newblock {\em IEEE Transactions on Computational Intelligence and AI in
Games}, 7(1):102--105, 2015.
\bibitem{EvenDaral06}
E.~Even-Dar, S.~Mannor, and Y.~Mansour.
\newblock {Action Elimination and Stopping Conditions for the Multi-Armed
Bandit and Reinforcement Learning Problems}.
\newblock {\em Journal of Machine Learning Research}, 7:1079--1105, 2006.
\bibitem{Gabillon:al12}
V.~Gabillon, M.~Ghavamzadeh, and A.~Lazaric.
\newblock {Best Arm Identification: A Unified Approach to Fixed Budget and
Fixed Confidence}.
\newblock In {\em {Advances in Neural Information Processing Systems}}, 2012.
\bibitem{GK16}
A.~Garivier and E.~Kaufmann.
\newblock Optimal best arm identification with fixed confidence.
\newblock In {\em Proceedings of the 29th Conference On Learning Theory
(COLT)}, 2016.
\bibitem{GKK16}
A.~Garivier, E.~Kaufmann, and W.M. Koolen.
\newblock Maximin action identification: A new bandit framework for games.
\newblock In {\em Proceedings of the 29th Conference On Learning Theory}, 2016.
\bibitem{HuangASM17}
Ruitong Huang, Mohammad~M. Ajallooeian, Csaba Szepesv{\'{a}}ri, and Martin
M{\"{u}}ller.
\newblock Structured best arm identification with fixed confidence.
\newblock In {\em 28th International Conference on Algorithmic Learning Theory
(ALT)}, 2017.
\bibitem{Jamiesonal14LILUCB}
K.~Jamieson, M.~Malloy, R.~Nowak, and S.~Bubeck.
\newblock {lil'{UCB}: an Optimal Exploration Algorithm for Multi-Armed
Bandits}.
\newblock In {\em {Proceedings of the 27th Conference on Learning Theory}},
2014.
\bibitem{Shivaramal12}
S.~Kalyanakrishnan, A.~Tewari, P.~Auer, and P.~Stone.
\newblock {{PAC} subset selection in stochastic multi-armed bandits}.
\newblock In {\em {International Conference on Machine Learning (ICML)}}, 2012.
\bibitem{Karnin:al13}
Z.~Karnin, T.~Koren, and O.~Somekh.
\newblock {Almost optimal Exploration in multi-armed bandits}.
\newblock In {\em {International Conference on Machine Learning (ICML)}}, 2013.
\bibitem{JMLR15}
E.~Kaufmann, O.~Capp{\'e}, and A.~Garivier.
\newblock {On the Complexity of Best Arm Identification in Multi-Armed Bandit
Models}.
\newblock {\em Journal of Machine Learning Research}, 17(1):1--42, 2016.
\bibitem{COLT13}
E.~Kaufmann and S.~Kalyanakrishnan.
\newblock {Information complexity in bandit subset selection}.
\newblock In {\em {Proceeding of the 26th Conference On Learning Theory.}},
2013.
\bibitem{KocsisBBMCP06}
L.~Kocsis and C.~Szepesv\'{a}ri.
\newblock Bandit based monte-carlo planning.
\newblock In {\em Proceedings of the 17th European Conference on Machine
Learning}, ECML'06, pages 282--293, Berlin, Heidelberg, 2006.
Springer-Verlag.
\bibitem{LaiRobbins85bandits}
T.L. Lai and H.~Robbins.
\newblock {Asymptotically efficient adaptive allocation rules}.
\newblock {\em Advances in Applied Mathematics}, 6(1):4--22, 1985.
\bibitem{Pepels14HybridMCTS}
T.~Pepels, T.~Cazenave, M.~Winands, and M.~Lanctot.
\newblock Minimizing simple and cumulative regret in monte-carlo tree search.
\newblock In {\em Computer Games Workshop, ECAI}, 2014.
\bibitem{PLAAT1996255}
Aske Plaat, Jonathan Schaeffer, Wim Pijls, and Arie de~Bruin.
\newblock Best-first fixed-depth minimax algorithms.
\newblock {\em Artificial Intelligence}, 87(1):255 -- 293, 1996.
\bibitem{deep.go}
David Silver, Aja Huang, Chris~J. Maddison, Arthur Guez, Laurent Sifre, George
van~den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda
Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal
Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray
Kavukcuoglu, Thore Graepel, and Demis Hassabis.
\newblock Mastering the game of go with deep neural networks and tree search.
\newblock {\em Nature}, 529:484--489, 2016.
\bibitem{Teraoka14MCTS}
K.~Teraoka, K.~Hatano, and E.~Takimoto.
\newblock Efficient sampling method for monte carlo tree search problem.
\newblock {\em IEICE Transactions on Infomation and Systems}, pages 392--398,
2014.
\bibitem{Thompson33}
W.R. Thompson.
\newblock {On the likelihood that one unknown probability exceeds another in
view of the evidence of two samples}.
\newblock {\em Biometrika}, 25:285--294, 1933.
\end{thebibliography}