Optimal Stopping for Sequential Bayesian Experimental Design

Extracted source text · arXiv 2509.21734

\documentclass{article}
\usepackage{arxiv}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
% \usepackage{hyperref}       % hyperlinks
\usepackage[dvipsnames]{xcolor}
\definecolor{mycolor}{RGB}{14,3,123}
\usepackage[
    colorlinks=true,
    citecolor=blue,      % Citations
    linkcolor=red,       % Internal links (\ref, \eqref, etc.)
    urlcolor=magenta,    % URLs
    filecolor=cyan,      % File links
    menucolor=red,       % Menu links
    runcolor=filecolor,  % Run links
    allcolors=mycolor      % Override all colors to black
]{hyperref}
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{amsmath}
\usepackage{cleveref}       % smart cross-referencing
\usepackage{lipsum}         % Can be removed after putting your text content
\usepackage{graphicx}
\usepackage{natbib}
\usepackage{doi}


\usepackage{enumitem}
\usepackage{amsfonts}
\usepackage{amsthm}
\usepackage{bm}
\usepackage{subfig}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{etoc}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\CN}{\mathcal{N}}
\newcommand{\CF}{\mathcal{F}}
\newcommand{\CL}{\mathcal{L}}
\newcommand{\CS}{\mathcal{S}}
\newcommand{\CT}{\mathcal{T}}
\newcommand{\DKL}{D_{\mathrm{KL}}}
\newcommand{\Hist}{I}
\newtheorem{theorem}{Theorem}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\allowdisplaybreaks[4] % break whenever needed

\title{Optimal Stopping for Sequential Bayesian\\Experimental Design}

% Here you can change the date presented in the paper title
%\date{September 9, 1985}
% Or remove it
%\date{}

\newif\ifuniqueAffiliation
% Comment to use multiple affiliations variant of author block 
\uniqueAffiliationtrue

\ifuniqueAffiliation % Standard variant of author block
\author{ \href{https://orcid.org/0000-0001-9773-2993}{\includegraphics[scale=0.06]{orcid.pdf}\hspace{1mm}Chen Cheng} \\
	Department of Mechanical Engineering\\
	University of Michigan\\
	Ann Arbor, MI 48109 \\
	\texttt{chech@umich.edu} \\
	%% examples of more authors
	\And
	\href{https://orcid.org/0000-0001-6544-2764}{\includegraphics[scale=0.06]{orcid.pdf}\hspace{1mm}Xun Huan} \\
	Department of Mechanical Engineering\\
	University of Michigan\\
	Ann Arbor, MI 48109 \\
	\texttt{xhuan@umich.edu} \\
	%% \AND
	%% Coauthor \\
	%% Affiliation \\
	%% Address \\
	%% \texttt{email} \\
	%% \And
	%% Coauthor \\
	%% Affiliation \\
	%% Address \\
	%% \texttt{email} \\
	%% \And
	%% Coauthor \\
	%% Affiliation \\
	%% Address \\
	%% \texttt{email} \\
}
\else

% Multiple affiliations variant of author block
\usepackage{authblk}
\renewcommand\Authfont{\bfseries}
\setlength{\affilsep}{0em}
% box is needed for correct spacing with authblk
\newbox{\orcid}\sbox{\orcid}{\includegraphics[scale=0.06]{orcid.pdf}} 
\author[1]{%
	\href{https://orcid.org/0000-0001-9773-2993}{\usebox{\orcid}\hspace{1mm}Chen Cheng\thanks{\texttt{chech@umich.edu}}}%
}
\author[1,2]{%
	\href{https://orcid.org/0000-0000-0000-0000}{\usebox{\orcid}\hspace{1mm}Xun Huan\thanks{\texttt{xhuan@umich.edu}}}%
}
\affil[1]{Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109}
\affil[2]{Department of Electrical Engineering, Mount-Sheikh University, Santa Narimana, Levand}
\fi

% Uncomment to override  the `A preprint' in the header
%\renewcommand{\headeright}{Technical Report}
%\renewcommand{\undertitle}{Technical Report}
\renewcommand{\shorttitle}{Optimal Stopping for Sequential Bayesian Experimental Design}

%%% Add PDF metadata to help others organize their library
%%% Once the PDF is generated, you can check the metadata with
%%% $ pdfinfo template.pdf
\hypersetup{
pdftitle={A template for the arxiv style},
pdfsubject={q-bio.NC, q-bio.QM},
pdfauthor={David S.~Hippocampus, Elias D.~Striatum},
pdfkeywords={First keyword, Second keyword, More},
}



\begin{document}
\date{}
\maketitle

\begin{abstract}
	In sequential Bayesian experimental design, the number of experiments is usually fixed in advance. In practice, however, campaigns may terminate early, raising the fundamental question: \emph{when should one stop}? Threshold-based rules are simple to implement but inherently myopic, as they trigger termination based on a fixed criterion while ignoring the expected future information gain that additional experiments might provide. We develop a principled Bayesian framework for optimal stopping in sequential experimental design, formulated as a Markov decision process where stopping and design policies are jointly optimized. We prove that the optimal rule is to stop precisely when the immediate terminal reward outweighs the expected continuation value. To learn such policies, we introduce a policy gradient method, but show that na\"{i}ve joint optimization suffers from circular dependencies that destabilize training. We resolve this with a curriculum learning strategy that gradually transitions from forced continuation to adaptive stopping. Numerical studies on a linear-Gaussian benchmark and a contaminant source detection problem demonstrate that curriculum learning achieves stable convergence and outperforms vanilla methods, particularly in settings with strong sequential dependencies.
\end{abstract}


% keywords can be removed
\keywords{Bayesian experimental design \and Sequential experimental design \and Optimal stopping \and Markov decision process \and Policy gradient}


\section{Introduction}
Sequential experimentation plays a central role in science and engineering, from clinical trials~\citep{murphy2003optimal} and materials discovery~\citep{lookman2019active} to environmental monitoring~\citep{krause2008near}. In these settings, practitioners adaptively choose new experiments based on outcomes of earlier ones. A fundamental yet often overlooked question is: \emph{when should experimentation stop}?

The prevailing practice is to fix the number of experiments in advance. However, real-world campaigns frequently face uncertain budgets, limited resources, or diminishing returns that make early termination desirable. Simple threshold-based stopping rules are often used in practice, halting experimentation once a predefined criterion (e.g., posterior variance, remaining budget) falls below a cutoff. While convenient, such rules are inherently myopic: they depend only on the current state and ignore the expected long-term trade-off between the value of additional information and the cost of further experiments. As a result, thresholds can terminate too early, missing valuable insights, or too late, wasting resources (see Figure~\ref{fig:threshold_example}).

\begin{figure}[htbp]
    \centering
    \includegraphics[width=.55\linewidth]{figures/threshold_example.pdf}
    \caption{Accumulated reward trajectories over six experiments, illustrating the limitations of threshold-based stopping. Here, a fixed threshold of 1.8 (dashed line) yields very different outcomes across trajectories: the orange path is stopped prematurely after one experiment, missing the higher rewards achievable at stages 2--3; the blue path never crosses the threshold, leading to unnecessary continuation past its peak; the green path happens to cross the threshold near its maximum, but only by chance. Threshold rules are thus highly outcome-dependent, either stopping too early or too late, and fail to account for the expected value of future experiments.
    }\label{fig:threshold_example}
\end{figure}

A more principled approach is to treat stopping as part of the sequential decision-making process, naturally leading to a \emph{Bayesian optimal stopping formulation} that weighs the immediate reward of terminating against the expected value of continuing. While optimal stopping theory is well developed in stochastic processes (e.g.,~\citealt{peskir2006optimal}), its integration with Bayesian experimental design remains largely unexplored. Existing sequential design methods 
typically assumes fixed horizons (see Section~\ref{sec:related} for a review),
leaving open a fundamental gap: \emph{no general framework jointly optimizes design and stopping in sequential Bayesian experimental design}.

In this work, we close this gap by developing a rigorous framework for optimal stopping in sequential Bayesian experimental design. Our contributions are:
\begin{itemize}[leftmargin=*]
\setlength\itemsep{0em}
\item We cast sequential design with stopping as a Markov decision process (MDP), jointly optimizing design and stopping policies.
\item We prove that the optimal stopping rule is to terminate exactly when the terminal reward exceeds the expected continuation value. 
\item We introduce an actor-critic  
policy gradient algorithm for jointly learning design and stopping policies. 
\item We identify a circular dependency in joint optimization that destabilizes training, and resolve it via curriculum learning that gradually transitions from forced continuation to adaptive stopping.
\item Through a linear-Gaussian benchmark and a contaminant source detection problem, we demonstrate that our policy gradient method with curriculum learning achieves stable convergence and outperforms vanilla variants in settings with strong sequential dependencies.
\end{itemize}
We bridge optimal stopping with Bayesian experimental design, advancing the development of autonomous and resource-efficient experimentation. 


\section{Related Work}\label{sec:related}

Bayesian experimental design~\citep{Chaloner1995,Ryan2016,Alexanderian2021,Rainforth2024,Huan2024} provides a principled framework for selecting experiments by maximizing expected information gain~\citep{Lindley1956}. 
Modern sequential extensions~\citep{foster2021deep,ivanova2021implicit,blau2022optimizing,shen2023bayesian,Shen2025} adapt each experiment to previous data, but almost always assume a fixed horizon: the total number of experiments is set in advance, with no explicit stopping policy.

When early termination {is} considered, it is usually via simple thresholds, such as halting when posterior variance drops below a cutoff or when budget is exhausted.
Similar rules appear in active learning~\citep{zhu2010confidence,pullar2024hitting},
multi-armed bandits~\citep{audibert2010best},
and online A/B testing~\citep{daskalakis2017optimal}.
These methods are convenient but \emph{myopic}, ignoring the expected long-term value of further experimentation (Figure~\ref{fig:threshold_example}).

Optimal stopping theory is well studied in stochastic processes and dynamic programming~\citep{peskir2006optimal, shiryaev2008optimal,haggstrom1966optimal,bertsekas2012dynamic,puterman2014markov}. 
Reinforcement learning can learn stopping rules but assume a fixed data-acquisition process and do not co-optimize experimental design~\citep{fathan2021deep, li2023deltav}. 
Some Bayesian designs include \textit{ad hoc} stopping; for example, \citet{Berry2002} introduced a ``terminator'' action in adaptive drug trials using low-dimensional Gaussian approximations and backward induction over small discrete state and action spaces. Their method is specialized and non-information-theoretic, unlike our general MDP formulation with explicit stopping theory and policy gradient training.

Despite progress in sequential design and stopping separately, to our knowledge no prior work provides a general framework for jointly optimizing design and stopping in information-theoretic sequential Bayesian experimental design. Our work closes this gap with a principled MDP formulation, an explicit optimal stopping rule, and a stable learning algorithm.


\section{Problem Formulation}\label{sec:formulation}

\subsection{Preliminaries}
We consider a finite sequence of $N$ experiments indexed by $k=0, 1, \ldots, N-1$. Let $\theta\in \RR^{N_{\theta}}$ denote model parameters, $\xi_k \in \Xi_k\subseteq \RR^{N_\xi}$ the design for experiment $k$, and $y_k \in \RR^{N_y}$ the corresponding observation. The information history at stage $k$ is
$\Hist_k=\{ \xi_0,y_0,\dots,\xi_{k-1},y_{k-1} \}$ with $\Hist_0=\emptyset$.
The stage-$k$ belief (the prior for experiment $k$) is $p(\theta | \Hist_k)$.
Observations are assumed to follow
\begin{align}
    y_k = G_k(\theta, \xi_k; \Hist_k) + \epsilon_k,
\end{align}
where $G_k$ is the forward map (which may depend on $\Hist_k$), and $\epsilon_k$ is an addictive observation noise with known density $p_{\epsilon}$. We assume $\epsilon_k$ is conditionally independent across stages given $(\theta,\xi_k,\Hist_k)$.

Upon observing $y_k$, the belief is updated via Bayes' rule:
\begin{align}
    \label{eq:bayes_rule}
    p(\theta | y_k,\xi_k,\Hist_k) = \frac{p(y_k | \theta,\xi_k,\Hist_k)\, p(\theta | \Hist_k)}{p(y_k | \xi_k,\Hist_k)},
\end{align}
where $p(y_k | \theta,\xi_k,\Hist_k)$ is the likelihood induced by $(G_k,p_{\epsilon})$, and $p(y_k | \xi_k,\Hist_k)=\int p(y_k | \theta,\xi_k,\Hist_k) p(\theta|\Hist_k)\,\mathrm{d}\theta$ is the marginal likelihood. The posterior $ p(\theta | y_k,\xi_k,\Hist_k)$ then becomes the prior for the next stage, i.e., $p(\theta | \Hist_{k+1})$. This recursive update defines the belief-state dynamics that will underlie our MDP formulation in section~\ref{ss:MDP}.

\subsection{Markov Decision Process for Optimal Stopping}\label{ss:MDP}

We formulate optimal stopping for sequential Bayesian experimental design as an MDP (see Figure~\ref{fig:flowchart}). This extends existing MDP-based sequential design framework (e.g., \citealp{shen2023bayesian})
by introducing an explicit stopping action that enables early termination of the experimental sequence. 

\begin{figure*}[htbp]
  \centering
  \includegraphics[width=\linewidth]{figures/flowchart.pdf}
    \caption{Flowchart of the MDP for sequential Bayesian experimental design with optimal stopping. At each stage, the agent chooses to continue or stop. If continuing, the design policy selects an experiment, yielding an observation and reward, and the state is updated via Bayes' rule. If stopping, a terminal reward is collected. The process repeats until termination or all experiments are exhausted.
    }
  \label{fig:flowchart}
\end{figure*}

\textbf{States.} At stage $k$, the state $s_k \in \CS$ is represented as $s_k = \{s_k^b, s_k^p\}$. The belief state $s_k^b$ summarizes the uncertainty about $\theta$, which is determined from $\Hist_k$ (for this paper, we can effectively view $s_k=\Hist_k$). The physical state $s_k^p$ captures any additional deterministic variables relevant to design (e.g., sensor positions). An additional special terminal state $T$ indicates that the experimental sequence has stopped and no future experiments will be conducted.
    
\textbf{Actions and policies.} At each stage, the agent either (i) terminates the design process, or (ii) chooses a design $\xi_k \in \Xi_k$ to perform an experiment. The stopping policy is a collection of binary decision rules, $\psi = \{\varphi_k: \CS \to \{0, 1\}, k=0,\ldots,N-1\}$, where $\varphi_k(s_k)=1$ indicates stopping at stage $k$. The design policy is a collection of mappings $\pi = \{\mu_k : \CS \to \Xi_k, k=0,\ldots,N-1\}$, which determine designs via $\xi_k=\mu_k(s_k)$.
    
\textbf{State transitions.} When an experiment is performed, the state evolves according to $s_{k+1}=\CF_k(s_k,\xi_k,y_k)$, which encodes the Bayesian update in (\ref{eq:bayes_rule}) given observation $y_k$. If the termination action is selected, or if all $N$ experiments have been exhausted, the state transitions to the terminal state $T$, where it remains thereafter. Formally,
    \begin{align}
        s_{k+1} = 
        \begin{cases}
            \CF_k(s_k,\xi_k,y_k), & s_k \neq T \text{ and } \varphi_k(s_k)=0,\\
            T, & \text{otherwise}.
        \end{cases}
    \end{align}
    
\textbf{Rewards.} The reward encodes information gain and experimental cost. Let $r_k(s_k,\xi_k,y_k)$ denote the immediate reward at stage $k$, and $r_T(s_k)$ the terminal reward collected when the experimental sequence ends.
Experiment $k$ incurs cost $c_k(\xi_k)$. Information gain is measured using the Kullback--Leibler (KL) divergence between distributions. We consider two equivalent reward formulations:
\begin{itemize}[leftmargin=*]
\setlength\itemsep{0em}
\item \emph{Terminal formulation} (all rewards are accumulated at termination):
    \begin{align}
    r_k(s_k, \xi_k, y_k) &= 0, \label{eq:terminal1}\\
    r_T(s_k) &=
        \begin{cases}
            \DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) + \sum\limits_{i=0}^{k-1}c_i(\xi_i), & s_k \neq T \text{ and } \varphi_k(s_k)=1,\\
            0, & \text{otherwise}.
        \end{cases}
        \label{eq:terminal2}
    \end{align}
\item \emph{Incremental formulation} (rewards distributed across stages):
      \begin{align}
        r_k(s_k, \xi_k, y_k) &= 
        \begin{cases}
        \DKL( p_{\theta | \Hist_{k+1}}\,||\,p_{\theta | \Hist_k} ) + c_k(\xi_k),  & s_k \neq T \text{ and } \varphi_k(s_k)=0, \\
        0, & \text{otherwise},
        \end{cases} 
        \label{eq:incremental1} \\
        r_T(s_k) &= 0. \label{eq:incremental2}
    \end{align}
\end{itemize}

\textbf{Problem statement.} 
The goal is to find optimal design and stopping policies:
\begin{align}
    \label{eq:optimization_problem}
    \pi^*, &\psi^* = \argmax_{\pi, \psi} U(\pi, \psi)  \\
   \text{subject to} \hspace{0.4em}
    &\qquad \varphi_k(s_k) \in \{0, 1\}, \quad \xi_k = \mu_k(s_k) \in \Xi_k, \nonumber \\
    &\qquad s_{k+1} = 
    \begin{cases}
        \CF_k(s_k,\xi_k,y_k), & s_k \neq T \text{ and } \varphi_k(s_k)=0,\\
        T, & \text{otherwise}.
    \end{cases}
    \nonumber
\end{align}
The objective (expected utility) is the expected total reward:
\begin{align}
    U(\pi, \psi) = \mathbb{E}_{y_{0 : N-1} \mid \pi, \psi, s_0} \left[ 
    \sum_{k=0}^{N-1} \left( r_k(s_k, \xi_k, y_k) + r_T(s_k) \right) 
    + r_T(s_N)
    \right]. \label{eq:objective_soed}
\end{align}
Although the terminal and incremental formulations differ in reward structure, they induce equivalent optimal policies when maximizing expected total reward. This MDP formulation thus unifies design and stopping, providing a principled framework for resource-efficient experimentation.


\section{Solution Method}\label{sec:method}

\subsection{Optimal stopping policy}
Before presenting the optimal stopping rule, we introduce the value function (V-function). The V-function at stage $k$ ($k = 0, \ldots, N-1$) under design policy $\pi$ and stopping policy $\psi$ is:
\begin{align}
V^{\pi,\psi}_k(s_k) &= \mathbb{E}_{y_{k : N-1} \mid \pi,\psi,s_k}\Bigg[
    \sum_{l=k}^{N-1} \left( r_l(s_l,\mu_l(s_l),y_l) + r_T(s_l) \right) 
    + r_T(s_N)
\Bigg],
\end{align}
which represents the expected cumulative reward from state $s_k$ onward when following $\pi$ and $\psi$. 

Let us further define 
$r^S_T(s_k)$ to be $r_T(s_k)$ when $s_k \neq T$ and $\varphi(s_k) = 1$---i.e., the terminal reward obtained by stopping at state $s_k$; and 
$r^C_k(s_k, \mu_k(s_k), y_k)$ to be $r_k(s_k, \mu_k(s_k), y_k)$ when $s_k \neq T$ and $\varphi(s_k) = 0$---i.e., the immediate reward obtained by continuing.


\begin{theorem}[Optimal Stopping Policy]
    For any design policy $\pi = \{\mu_0, \ldots, \mu_{N-1}\}$, the optimal stopping policy is $\psi = \{\varphi_0, \ldots, \varphi_{N-1}\}$ with 
    \begin{align}
         \varphi_k(s_k) = \bm{1}_{s_k\in \CT_k}, \quad k=0,\ldots,N-1,
    \end{align}
    where the stopping set $\CT_k \subseteq \mathcal{S}$ 
    %for non-terminal states 
    is defined as:
    \begin{align}
    \CT_k = \left\{ s_k \, \Big| \,
    r^S_T(s_k) \geq \mathbb{E}_{y_k \mid s_k, \pi} \left[ 
    r^C_k(s_k, \mu_k(s_k), y_k) + V^{\pi, \psi}_{k+1} \big( \CF_k(s_k, \mu_k(s_k), y_k) \big)
    \right] \right\}.
\label{eq:stopping_sets_V}
\end{align}
\label{thm:optimal_stopping_rule}
\end{theorem}
We provide a proof in Appendix~\ref{app:optimal_stopping_rule_proof}. 
Intuitively, the theorem states that the optimal decision at each stage is to compare the terminal reward from stopping with the expected continuation value. Stopping is optimal whenever the former dominates, balancing the value of ending experimentation now against the potential information gain from further experiments.

To formalize this comparison, we define a specialized action-value function (Q-function) that evaluates the return when the next action is to continue---\emph{the value of continuation}---after which $\pi$ and $\psi$ are followed:
\begin{align}
Q^{\pi,\psi}_k(s_k, \xi_k) = \mathbb{E}_{y_k \mid s_k, \xi_k} \left[
    r^C_k(s_k,\xi_k,y_k) + V^{\pi,\psi}_{k+1}(s_{k+1})
\right],
\end{align}
where $s_{k+1} = \CF_k(s_k, \xi_k, y_k)$. The stopping set can be expressed equivalently and succinctly as:
\begin{align}
        \CT_k=\left\{s_k \,\Big|\, r^S_T(s_k) \geq Q_k^{\pi,\psi}\big(s_k, \mu_k(s_k)\big)\right\}.
    \label{eq:stopping_sets_Q}
\end{align}
Thus the Q-function provides a direct way to evaluate whether stopping and taking the terminal reward is better than the value of continuation. 

\begin{theorem}[Terminal-Incremental Equivalence]\label{thm:equiv_terminal_incremental}
    Let $U_T(\pi, \psi)$ denote the expected utility under the terminal formulation, and
    $U_I(\pi, \psi)$ the expected utility under the incremental formulation.
    Then for any policies $(\pi,\psi)$, $U_T(\pi, \psi) = U_I(\pi, \psi)$.
\end{theorem}
We provide a proof in Appendix~\ref{app:equiv_terminal_incremental}. 
Hence both reward structures yield the same optimization problem, and one may choose whichever is more convenient for computation.

\subsection{Policy gradient for optimal stopping}
Theorem~\ref{thm:optimal_stopping_rule} shows that the optimal stopping policy is implicitly determined by the design policy $\pi$. This suggests that the joint optimization problem can be reformulated as a single optimization over the design policy alone. We therefore parameterize the design policy, derive the gradient of the expected utility with respect to its parameters, and employ gradient-based optimization. This leads to a policy gradient (PG) method~\citep{pmlr-v32-silver14,Wang2020Neural}.

\subsubsection{Policy gradient derivation}
We parameterize each design function $\mu_k$ with parameters $w_k$, and write $\mu_{k,w_k}$. The full design policy $\pi$ is parameterized by  $w = \{w_k\}_{k=0}^{N-1}$, denoted $\pi_w$.
The corresponding stopping policy is implicitly determined by $w$ as $\psi_w$, with 
\begin{align}
    \varphi_{k,w}(s_k) = \bm{1}_{s_k\in \CT_{k,w}},
\end{align}
where the stopping set $\CT_{k,w}$ is defined by:
\begin{align}
\CT_{k,w} &= \left\{ s_k \, \Big|\,
r^S_T(s_k) \geq \mathbb{E}_{y_k \mid \pi_w, s_k} \Big[
    r^C_k(s_k, \mu_{k,w_k}(s_k), y_k) + V^{\pi_w, \psi_w}_{k+1} \big( \CF_k(s_k, \mu_{k,w_k}(s_k), y_k) \big)
\Big] \right\} \nonumber\\
        &=\left\{s_k \,\Big|\, r^S_T(s_k) \geq Q_k^{\pi_w,\psi_w}(s_k, \mu_{k,w_k}(s_k))\right\}.
\end{align}
The expected utility (\ref{eq:optimization_problem}) becomes:
\begin{align}
U(w) &= \mathbb{E}_{y_{0 : N-1} \mid \pi_w, \psi_w, s_0} \Bigg[\sum_{k=0}^{N-1} \left( r_k(s_k, \mu_{k, w_k}(s_k), y_k) + r_T(s_k) \right) + r_T(s_N)
\Bigg].
\label{eq:expected_utility_w}
\end{align}

\begin{theorem}[Policy Gradient]
\label{thm:PG}
The gradient of the expected utility with respect to the design policy parameters is:
\begin{align}
\nabla_w U(w) 
= \sum_{k=0}^{N-1} \mathbb{E}_{s_k \mid \pi_w,\psi_w,s_0} \left[
    \bm{1}_{k < \tau_w} \nabla_w \mu_{k,w_k}(s_k) \nabla_{\xi_k} Q^{\pi_w,\psi_w}_k(s_k,\xi_k) \Big|_{\xi_k=\mu_{k,w_k}(s_k)}
\right],
\label{eq:pg_theorem}
\end{align}
where $\tau_w = \inf\{k: \varphi_{k,w}(s_k) = 1\}$ is the stage when the state first enters the stopping set.
\end{theorem}
We provide a proof in Appendix~\ref{app:policy_gradient_expression}. 

\subsubsection{Numerical estimation}
The gradient in Theorem~\ref{thm:PG} involves nested expectations over stochastic trajectories and stopping decisions, which are intractable in closed form. We therefore use a Monte Carlo (MC) estimator:
\begin{align}
\label{eq:policy_grad_MC}
\nabla_w U(w) \approx &\frac{1}{M} \sum_{m=1}^M \sum_{k=0}^{N-1}
\bm{1}_{k<\tau^{(m)}} \nabla_w \mu_{k,w_k}\left(s^{(m)}_k\right) 
\nabla_{\xi^{(m)}_k} 
Q^{\pi_w, \psi_w}_k\left(s^{(m)}_k, \xi^{(m)}_k\right)
\Big|_{\xi^{(m)}_k = \mu_{k,w_k}\left(s^{(m)}_k\right)},
\end{align}
where superscript indicates the $m$th sampled trajectory. 

Two challenges arise: (1) policy gradients $\nabla_w \mu_{k,w_k}$ must be computed efficiently, and (2) Q-function gradients $\nabla_{\xi} Q^{\pi_w, \psi_w}_k$ cannot be estimated reliably from nested MC sampling. We address both using neural network approximations in an actor-critic framework~\citep{sutton2018reinforcement}.


\subsubsection{Actor-Critic Implementation}

\textbf{Policy network.}
The design policy is parameterized by a single deep neural network $\mu_w(k, s_k)$. Gradients $\nabla_w \mu_{k,w_k}\left(s^{(m)}_k\right) = \nabla_w \mu_{w}\left(k, s^{(m)}_k\right)$ are computed via standard backpropagation.

\textbf{Q-network.}
To approximate the continuation value, we train a neural network $Q_{\eta}^{\pi_w, \psi_w}(k, s_k, \xi_k)$ with parameters $\eta$. This avoids costly inner MC sampling for directly estimating $Q$ and provides differentiable Q-function estimates. The Q-network is trained by minimizing:
\begin{align}
\label{eq:value_loss}
\CL(\eta) = &\frac{1}{M} \sum_{m=1}^M \sum_{k=0}^{N-1} \Bigg[
    Q^{\pi_w, \psi_w}_{\eta}\left(k, s^{(m)}_k, \xi^{(m)}_k\right) - \left( r_k\left(s^{(m)}_k, \xi^{(m)}_k, y^{(m)}_k\right)
    + Q^{\pi_w, \psi_w}_{k+1}\left(s^{(m)}_{k+1}, \xi^{(m)}_{k+1}\right) \right)
\Bigg]^2 \bm{1}_{k<\tau^{(m)}},
\end{align}
with $\xi^{(m)}_k = \mu_{w}\left(k,s^{(m)}_k\right)$ 
and $Q^{\pi_w, \psi_w}_N\left(s^{(m)}_N,\cdot\right) = r_T\left(s^{(m)}_N\right)$. 
This loss enforces the Bellman equation and ensures accurate return predictions.

\textbf{Training.} We follow a standard actor-critic loop in Algorithm~\ref{alg:PG-sOED}. During training, a parameter sample  $\theta^{(m)}\sim p(\theta)$ generates an episode, and exploration is encouraged by injecting Gaussian noise to the design:
\begin{align}
    \xi_k = \mu_k(s_k) + \epsilon_{\text{explore}}, \quad \epsilon_{\text{explore}} \sim \mathcal{N}(0, \mathbb{I}_{N_{\xi}}\sigma^2_{\text{explore}}).
\end{align}
Stopping is determined when $r_T^S(s_k)$ exceeds the continuation value $Q_k^{\pi, \psi}(s_k, \mu_k(s_k))$ in (\ref{eq:stopping_sets_Q}). 
Policy parameters $w$ are updated via gradient ascent using the MC gradient estimator in (\ref{eq:policy_grad_MC}), while Q-network parameters $\eta$ are updated via stochastic gradient descent on the loss in (\ref{eq:value_loss}). See Appendix~\ref{app:pseudocode} for more implementation details.

Algorithm~\ref{alg:PG-sOED} presents the overall pseudocode of our method.

\begin{algorithm}
\caption{Actor-critic PG for optimal stopping.}
\label{alg:PG-sOED}
\begin{algorithmic}[1]
\STATE{Set initial state $s_0$, policy updates $L$, MC sample size $M$, policy and Q-network architectures, learning rate $\alpha$ for policy update, exploration scale $\sigma_{\rm{explore}}$;
}
\STATE{Initialize policy and Q-network parameters $w$ and $\eta$;}
\FOR{$\ell=1,\dots,L$}
\STATE{Simulate $M$ episodes with path $m = 1, \ldots, M$ following steps 5--10 below;}
\STATE{Initialize $s_0^{(m)} = s_0$ and sample $\theta^{(m)} \sim s_{0,b}$;}
\FOR{$k = 0, \ldots, N-1$}
\STATE{Sample design $\xi_k^{(m)} = \mu_{w}(k, s_k^{(m)}) + \epsilon_{\text{explore}}$ and $y_k^{(m)}\sim p(\cdot \big|\theta^{(m)},\xi_k^{(m)},I_k^{(m)})$;}\STATE{Update state $s_{k+1}^{(m)}$, and compare $r_T^S(s^{(m)}_{k+1})$ with $Q_{\eta}^{\pi, \psi}(k+1, s^{(m)}_{k+1}, \mu_w(k+1, s^{(m)}_{k+1}))$ for stopping decision;}
\STATE{If $s_{k+1}^{(m)} \in \mathcal{T}^{(m)}_{k+1, w}$ or $k+1 = N$, set $\tau^{(m)} = k+1$ and exit the loop;}
\ENDFOR
\STATE{Store the full information history from all episodes $\{I_{\tau^{(m)}}^{(m)}\}_{m=1}^M$};
\STATE{Compute and store immediate and terminal rewards for all episodes $\{r_k^{(m)}, r_T^{(m)}\}_{m=1}^M$;}
\STATE{Update $\eta$ by minimizing the loss in (\ref{eq:value_loss});}
\STATE{Update $w$ by gradient ascent: $w = w + \alpha \nabla_w U(w)$, where $\nabla_w U(w)$ is estimated via (\ref{eq:policy_grad_MC});}
\ENDFOR
\STATE{Return optimized design policy $\pi_{w}$ and stopping policy $\psi_{w, \eta}$.}
\end{algorithmic}
\end{algorithm}

\subsection{Training instabilities and curriculum learning}
Although the PG method provides a principled framework, implementing it in practice reveals training instabilities arising from the problem's inherent circular dependency. The optimal stopping set $\mathcal{T}_k$ in (\ref{eq:stopping_sets_Q}) depends on the Q-function $Q_k^{\pi,\psi}(s_k, \mu_k(s_k))$, which itself depends on both the design policy $\pi$ and stopping policy $\psi$. Thus, the stopping policy requires knowledge of the Q-function computed under that very same policy, leading to a fixed-point relationship $\psi^* = f(\pi, \psi^*)$. 

In practice, this circularity manifests during joint training of the policy network (parameters $w$) and the Q-network (parameters $\eta$). Early in training, poorly initialized design policies generate weak experiments with little information gain. The Q-network then learns pessimistic continuation values, which in turn induce premature stopping that prevents policy improvement and entering suboptimal designs. This issue is particularly pronounced in sequential design problems with strong sequential dependencies, where early stopping disrupts the learning of effective long-term strategies.

To overcome this challenge, we introduce a \emph{curriculum learning} scheme based on a stopping probability schedule $p_{\text{stop}}(\ell)$, where $\ell$ indexes training iterations.
When the stopping condition $s_k \in \mathcal{T}_k$ is satisfied, the algorithm follows the optimal stopping rule with probability $p_{\text{stop}}(\ell)$, and overrides it (forcing continuation) with probability $(1 - p_{\text{stop}}(\ell))$. 

This strategy deliberately relaxes stopping early in training, producing longer trajectories that allow both the policy and Q-networks to improve before stopping decisions dominate. As training progresses, $p_{\text{stop}}(\ell)$ is gradually increased, so that stopping behavior converges to the optimal fixed point. In this way, curriculum learning breaks the self-reinforcing cycle of premature stopping and stabilizes training, particularly in settings with strong sequential dependencies.

\section{Numerical Experiments}\label{sec:results}
We demonstrate our approach on two numerical examples: a linear-Gaussian benchmark for validation and a sensor movement problem for contaminant source detection that highlights the benefit of 
curriculum learning. 
The curriculum learning schedule is provided in Appendix~\ref{app:stopping_schedule}.

\subsection{Linear-Gaussian Benchmark}\label{sec:case_linear_Gaussian}
We first validate our PG algorithm on a canonical linear-Gaussian benchmark where analytical solutions are available. The forward model is linear in $\theta$ with additive Gaussian noise:
\begin{align}
    y_k = G(\theta,\xi_k) + \epsilon_k = \theta \xi_k + \epsilon_k, \quad \epsilon_k \sim \CN(0,1^2),
\end{align}
with prior $\theta\sim\CN(0, 3^2)$. Experiments are constrained to designs $\xi_k \in [0.1,3]$. Conjugacy allows closed-form solutions for optimal policies (see Appendix~\ref{app:linear_gaussian_analytic_solution}).

\begin{figure}[htbp]
  \centering
    \includegraphics[width=0.325\linewidth]{figures/LinearGaussian-T=3-Cost=0-Reward-both.pdf}
    \includegraphics[width=0.325\linewidth]{figures/LinearGaussian-T=3-Cost=-0.5-Reward-both.pdf}
     \includegraphics[width=0.325\linewidth]{figures/LinearGaussian-T=4-Cost=-0.25-Reward-both.pdf}\\
    \includegraphics[width=0.325\linewidth]{figures/LinearGaussian-T=3-Cost=0-tau-both.pdf}
   \includegraphics[width=0.325\linewidth]{figures/LinearGaussian-T=3-Cost=-0.5-tau-both.pdf}
    \includegraphics[width=0.325\linewidth]{figures/LinearGaussian-T=4-Cost=-0.25-tau-both.pdf}
  \caption{Training convergence of the linear-Gaussian benchmark. Top: average reward; bottom: average stopping stage. Columns correspond to different horizons $N$ and experimental costs.}
  \label{fig:LinGau_Cnvg_T3_no_cost}
\end{figure}

For zero cost ($c_k=0$), the analytical solution shows that stopping is always optimal at the horizon $N$, since the KL divergence reward is non-decreasing. Figure~\ref{fig:LinGau_Cnvg_T3_no_cost} (left column) shows that our method converges to the maximum stage $N= 3$, achieving the analytical optimum, while na\"{i}ve threshold policies stop prematurely and underperform. 

For negative costs, Figure~\ref{fig:LinGau_Cnvg_T3_no_cost} (middle and right columns) illustrate two cases: 
\begin{itemize}[leftmargin=*]
\setlength\itemsep{0em}
\item An $N=3$ problem with $c_k = -0.5$, where both vanilla and curriculum methods converge to the optimal early stopping at stage 1, consistent with analytical results.
\item An $N=4$ problem with cost $c_k = -0.25$, where both methods achieve near-optimal performance but with distinct convergence patterns. The vanilla approach often underestimates the optimal stopping stage, becoming trapped below the analytical solution. In contrast, the curriculum method converges more reliably, exploring longer trajectories before settling at the correct stage.
\end{itemize}
These results illustrate that curriculum learning promotes more stable convergence by mitigating premature stopping during training. The small discrepancy from theory in the $N=4$ case reflects the flat reward landscape, where stopping at stages 1--3 yields nearly indistinguishable rewards. Even so, the PG method successfully navigates these subtle trade-offs, validating both the algorithm and the role of curriculum learning.


\subsection{Contaminant Source Detection in Convection-Diffusion Field}
We next consider a sensor-movement problem for contaminant source detection, which exhibits strong sequential dependencies where curriculum learning is essential. Mobile sensors must be strategically repositioned to locate an unknown pollution source, with each placement decision depending critically on previous locations and measurements. Contaminant transport is modeled by a convection-diffusion partial differential equation (PDE) in a two-dimensional square domain with a Gaussian source term (see Appendix~\ref{app:contaminant_setup} for details). We present constant-cost experiments below; design-dependent cost cases are given in Appendix~\ref{app:additional_results}.

\begin{figure}[htbp]
  \centering
   \includegraphics[width=0.325\linewidth]{figures/ConvectionDiffusion-T=4-Cost=0-Reward-both.pdf}
  \includegraphics[width=0.325\linewidth]{figures/ConvectionDiffusion-T=4-Cost=-0.5-Reward-both.pdf}
  \includegraphics[width=0.325\linewidth]{figures/ConvectionDiffusion-T=4-Cost=-0.8-Reward-both.pdf}\\
   \includegraphics[width=0.325\linewidth]{figures/ConvectionDiffusion-T=4-Cost=0-tau-both.pdf}
  \includegraphics[width=0.325\linewidth]
  {figures/ConvectionDiffusion-T=4-Cost=-0.5-tau-both.pdf}
  \includegraphics[width=0.325\linewidth]{figures/ConvectionDiffusion-T=4-Cost=-0.8-tau-both.pdf}
  \caption{Training convergence of the convection-diffusion 
  %source detection
  problem. Top: average reward; bottom: average stopping stage. Columns correspond to different horizons $N$ and experimental costs.}
  \label{fig:ConDiff_Cnvg_T4_negative_cost}
\end{figure}

For zero cost ($c_k=0$), Figure~\ref{fig:ConDiff_Cnvg_T4_negative_cost} (left column) shows that both vanilla and curriculum learning converge successfully. The vanilla PG approach quickly settles onto the maximum stage of $N= 4$, while curriculum PG achieves slightly higher rewards. In this regime, continuation rewards dominate termination rewards, even under poor initial designs, so both methods are naturally incentivized to continue and eventually converge. 

For negative costs, the training challenges identified in our theoretical analysis become evident. With moderate cost ($c_k = -0.5$, middle column), vanilla PG fails to outperform a fixed stage-4 stopping policy and prematurely convergence to stage 3. In contrast, curriculum learning sustains exploration longer, achieving higher rewards and more appropriate stopping behavior that balances information gain with cost. At higher cost ($c_k = -0.8$, right column), vanilla method degrades further, stopping too early, whereas curriculum learning maintains stable convergence and superior reward. 

These results show that sequential dependence and cost-sensitive stopping can destabilize joint optimization, trapping vanilla PG in suboptimal policies. Curriculum learning overcomes this by gradually relaxing stopping constraints, enabling robust convergence from exploration to exploitation.



\section{Conclusion}\label{sec:conclusion}
This work tackles the fundamental challenge of deciding when to terminate sequential experiments in Bayesian experimental design. We developed a principled framework for optimal stopping that accounts for the expected future value of continuation, showing that the optimal rule is simple: continue only when the expected continuation reward exceeds the immediate terminal reward. 

From a computational perspective, the coupling between design and stopping policies creates circular dependencies that destabilize training. We addressed this with a PG approach augmented with curriculum learning, which gradually transitions from forced continuation to adaptive stopping. 
%
Our experiments demonstrate that,
%while independent design problems can be solved by standard optimization, 
especially for tasks with strong sequential dependencies, curriculum learning achieves stable convergence. The convection-diffusion source detection problem illustrates this setting, where curriculum learning yields substantial gains over fixed stopping rules by breaking the coupling during training. 

While our framework offers a principled solution, several limitations point to future work. First, evaluating terminal rewards at every decision stage can be expensive with costly posterior updates or high-dimensional reward structures. Second, alternative formulations that directly parameterize the stopping policy without relying on the optimal stopping theorem (Theorem~\ref{thm:optimal_stopping_rule}) might avoid circular dependencies and simplify training. Such approaches may converge more slowly or sacrifice theoretical optimality, but could broaden the range of practical applications---an avenue that remains open for investigation.

Overall, this framework advances autonomous experimental systems by enabling intelligent, resource-aware stopping decisions, improving efficiency of sequential Bayesian experimental design.


\bibliography{references}
\bibliographystyle{unsrtnat}


\newpage
\appendix
\section{Appendix}
\localtableofcontents

\subsection{Proof of Theorem~\ref{thm:optimal_stopping_rule}}
\label{app:optimal_stopping_rule_proof}

\begin{proof}
[Proof of Theorem~\ref{thm:optimal_stopping_rule}] For a given design policy $\pi$, consider the V-function under any stopping policy $\psi'$ different from $\psi$:
\begin{align}
    &V^{\pi,\psi'}_N\left(s_N\right)= r_T(s_N) \notag \\
    &V^{\pi,\psi'}_k\left(s_k\right)=  
    \begin{cases}
        r^S_T\left(s_k\right), & \varphi'_k(s_k) = 1, \\
        \EE_{y_k \mid \pi, s_k}\left[r^C_k\left(s_k, \mu_k(s_k), y_k\right)+V^{\pi, \psi'}_{k+1}\left(\CF_k\left(s_k, \mu_k(s_k), y_k\right)\right)\right], &\text{ otherwise}.
    \end{cases}
\end{align}
Note that at the terminal stage, $V^{\pi, \psi}_N(s_N) = r_T(s_N) = V^{\pi, \psi'}_N(s_N)$. V-function under the optimal stopping policy $\psi$ satisfies:
\begin{align}
    V^{\pi, \psi}_k(s_k) = \max \left\{r^S_T(s_k), \EE_{y_k \mid \pi, s_k}\left[r^C_k(s_k, \mu_k(s_k), y_k)+V^{\pi, \psi}_{k+1}(\CF_k(s_k, \mu_k(s_k), y_k))\right]\right\}.
\end{align}

\textbf{Base case ($k = N-1$).}
If $\varphi'_{N-1}(s_{N-1}) = 1$, then 
\begin{align}
    V^{\pi,\psi'}_{N-1}(s_{N-1}) &= r^S_T(s_{N-1}) \nonumber \\
    &\leq \max \Big\{r^S_T(s_{N-1}), \EE_{y_{N-1} \mid \pi, s_{N-1}}\big[r^C_{N-1}(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1})\nonumber\\
    &\hspace{14em}+V^{\pi, \psi}_{N}(\CF_{N-1}(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1}))\big]\Big\} \nonumber \\
    &= V^{\pi,\psi}_{N-1}(s_{N-1}).
\end{align}
If $\varphi'_{N-1}(s_{N-1}) = 0$, then
\begin{align}
    V^{\pi,\psi'}_{N-1}(s_{N-1}) &= \EE_{y_{N-1} \mid \pi, s_{N-1}}\Big[r^C_{N-1}\left(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1}\right)+V^{\pi, \psi'}_{N}\left(\CF_{N-1}\left(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1}\right)\right)\Big] \nonumber \\
    &= \EE_{y_{N-1} \mid \pi, s_{N-1}}\Big[r^C_{N-1}\left(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1}\right)+V^{\pi, \psi}_{N}\left(\CF_{N-1}\left(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1}\right)\right)\Big]\nonumber \\
    &\leq \max \Big\{r^S_T(s_{N-1}), \EE_{y_{N-1} \mid \pi, s_{N-1}}\big[r^C_{N-1}(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1})\nonumber\\
    &\hspace{14em}+V^{\pi, \psi}_{N}(\CF_{N-1}(s_{N-1}, \mu_{N-1}(s_{N-1}), y_{N-1}))\big]\Big\}   \nonumber \\
    &= V^{\pi,\psi}_{N-1}(s_{N-1}).
\end{align}
Hence, $V^{\pi,\psi}_{N-1}(s_{N-1}) \geq V^{\pi,\psi'}_{N-1}(s_{N-1})$. 

\textbf{Inductive step.}
Suppose $V^{\pi,\psi}_{k+1}(s_{k+1}) \geq V^{\pi,\psi'}_{k+1}(s_{k+1})$. Then
\begin{align}
    V^{\pi,\psi}_k(s_k) &=\max \left\{r^S_T(s_k), \EE_{y_k \mid \pi, s_k}\left[r^C_k\left(s_k, \mu_k(s_k), y_k\right)+V^{\pi, \psi}_{k+1}\left(\CF_k\left(s_k, \mu_k(s_k), y_k\right)\right)\right]\right\}  \nonumber \\
    &\geq \max \left\{r^S_T(s_k), \EE_{y_k \mid \pi, s_k}\left[r^C_k\left(s_k, \mu_k(s_k), y_k\right)+V^{\pi, \psi'}_{k+1}\left(\CF_k\left(s_k, \mu_k(s_k), y_k\right)\right)\right]\right\}  \nonumber \\
    &\geq V^{\pi,\psi'}_k(s_k).
\end{align}

By induction, $V^{\pi,\psi}_k(s_k) \geq V^{\pi,\psi'}_k(s_k)$ for all $k = 0,\ldots,N-1$. In particular, $V^{\pi,\psi}_0(s_0) \geq V^{\pi,\psi'}_0(s_0)$, which proves that the stopping policy in Theorem~\ref{thm:optimal_stopping_rule} is optimal.
\end{proof}


\subsection{Proof of Theorem~\ref{thm:equiv_terminal_incremental}}\label{app:equiv_terminal_incremental}

%In this section, we provide the proof of the equivalence of incremental and terminal reward formulations in OS-sOED. 
\begin{proof}
[Proof of Theorem~\ref{thm:equiv_terminal_incremental}] We first prove the stopping sets are equivalent under two formulations. For convenience, we omit the notation of design policy $\pi$ and stopping policy $\psi$, but distinguish them using upperscript of \emph{incr} and \emph{term} respectively. The policy dependence in the expectations is also omitted for notational consistency. We first prove the following relationship between value functions:
\begin{align}
    V^{incr}_k(s_k) = V^{term}_k(s_k) - \left[\DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{k-1}c_i(\xi_i)\right], k=0, \ldots, N.
\label{eq:V-func-equiv} 
\end{align}
\textbf{Base case ($k = N$).} For the terminal formulation,
\begin{align}
    V^{term}_N(s_N) = r_T(s_N) = \DKL( p_{\theta | \Hist_N}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{N-1} c_i(\xi_i);
\end{align}
for the incremental formulation,
\begin{align}
    V^{incr}_N(s_N) = r_T(s_N) = 0.
\end{align}
Therefore
\begin{align}
    V^{incr}_N(s_N) = V^{term}_N(s_N) - \left[\DKL( p_{\theta | \Hist_N}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{N-1} c_i(\xi_i)\right].
\end{align}
\textbf{Inductive step.} Suppose
\begin{align}
    V^{incr}_{k+1}(s_{k+1}) = V^{term}_{k+1}(s_{k+1}) - \left[\DKL( p_{\theta | \Hist_{k+1}}\,||\,p_{\theta | \Hist_0} )) + \sum_{i=0}^{k} c_i(\xi_i)\right].
\end{align}
Then at stage $k$, for the terminal formulation,
\begin{align}
    V^{term}_k(s_k) = \max\left\{\DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{k-1} c_i(\xi_i), \EE_{y_k}\left[V^{term}_{k+1}(s_{k+1})\right]\right\};
\end{align}
for the incremental formulation,
\begin{align}
    &\qquad V^{incr}_k(s_k) \nonumber \\
    &= \max\left\{0, \,\, \EE_{y_k}\left[\DKL( p_{\theta | \Hist_{k+1}}\,||\,p_{\theta | \Hist_k} ) + c_k(\xi_k) + V^{incr}_{k+1}(s_{k+1})\right]\right\} \nonumber \\
    &= \max\Bigg\{0, \,\, \EE_{y_k}\Bigg[\DKL( p_{\theta | \Hist_{k+1}}\,||\,p_{\theta | \Hist_k} ) + c_k(\xi_k) + V^{term}_{k+1}(s_{k+1}) - \DKL( p_{\theta | \Hist_{k+1}}\,||\,p_{\theta | \Hist_0} ) - \sum_{i=0}^k c_i(\xi_i)\Bigg]\Bigg\} \nonumber \\
    &= \max\left\{0, \,\, \EE_{y_k}\left[V^{term}_{k+1}(s_{k+1}) - \DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) - \sum_{i=0}^{k-1} c_i(\xi_i)\right]\right\} \nonumber \\
    &= \max\left\{\DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{k-1} c_i(\xi_i),\,\, \EE_{y_k}\left[V^{term}_{k+1}(s_{k+1})\right]\right\} - \left[\DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{k-1} c_i(\xi_i)\right] \nonumber \\
    &= V^{term}_k(s_k) - \left[\DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{k-1} c_i(\xi_i)\right].
\end{align}
By induction, (\ref{eq:V-func-equiv}) is proved. Using this relationship, we examine the stopping sets. The terminal formulation stopping sets are:
\begin{align}
    \CT^{term}_k = \left\{s_k \,\Bigg|\, \DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) + \sum_{i=0}^{k-1} c_i(\xi_i) \geq \EE_{y_k}\left[V^{term}_{k+1}(s_{k+1})\right]\right\};
\end{align}
the incremental stopping sets are:
\begin{align}
    \CT^{incr}_k &= \left\{s_k \mid 0 \geq \EE_{y_k}\left[r^C_k(s_k,\xi_k,y_k) + V^{incr}_{k+1}(s_{k+1})\right]\right\} \nonumber \\
    &= \left\{s_k \mid 0 \geq \EE_{y_k}\left[\DKL( p_{\theta | \Hist_{k+1}}\,||\,p_{\theta | \Hist_k} ) + c_k(\xi_k) + V^{incr}_{k+1}(s_{k+1})\right]\right\} \nonumber \\
    &= \left\{s_k \,\Bigg|\, 0 \geq \EE_{y_k}\left[V^{term}_{k+1}(s_{k+1})\right] - \DKL( p_{\theta | \Hist_k}\,||\,p_{\theta | \Hist_0} ) - \sum_{i=0}^{k-1} c_i(\xi_i)\right\} \nonumber \\
    &= \CT^{term}_k.
\end{align}
Therefore, the stopping sets are equivalent in both formulations. Especially, the optimization objective is 
\begin{align}
    U(\pi, \psi) = V_0^{incr}(s_0) = V_0^{term}(s_0),
\end{align}
which further proves the equivalence of the optimization problem.
\end{proof}


\subsection{Proof of Theorem~\ref{thm:PG}}\label{app:policy_gradient_expression}

To avoid notation congestion, below we will omit the subscript on $w$ and shorten $\mu_{k, w_k}(s_k)$ to $\mu_{k, w}(s_k)$, with the understanding that $w$ takes the same subscript as the $\mu$ function. We also omit explicit conditioning in expectations to save formula space, with the understanding that expectations are taken with respect to the appropriate policies and states.

\begin{proof}[Proof of Theorem~\ref{thm:PG}]

The gradient of the expected utility can be written using the V-function:
\begin{equation}
\nabla_w U(w)=\nabla_w V_0^{\pi_w, \psi_w}\left(s_0\right).
\end{equation}
By denoting the V-function as 
\begin{align}
    V_k^{\pi_w, \psi_w}(s_k) &= \bm{1}_{s_k\in \CT_{k,w}}r^S_T(s_k)  + (1-\bm{1}_{s_k\in \CT_{k,w}})\EE_{y_k}\left[r^C_k(s_k, \mu_{k, w}(s_k), y_k)+V^{\pi_w, \psi_w}_{k+1}(\CF_k(s_k, \mu_{k, w}(s_k), y_k))\right],
\end{align}
the gradient of $V_k^{\pi_w, \psi_w}(s_k)$ is
{\small
\begin{align}
    \quad \nabla_w V^{\pi_w, \psi_w}_k(s_k) 
    &= \nabla_w\bm{1}_{s_k\in \CT_{k,w}}r^S_T(s_k) - \nabla_w \bm{1}_{s_k\in \CT_{k,w}}\EE_{y_k}\left[r^C_k(s_k, \mu_{k, w}(s_k), y_k) + V^{\pi_w, \psi_w}_{k+1}(\CF_k(s_k, \mu_{k, w}(s_k), y_k))\right]\nonumber \\
    &\quad + (1-\bm{1}_{s_k\in \CT_{k,w}})\nabla_w\EE_{y_k}\left[r^C_k(s_k, \mu_{k, w}(s_k), y_k)+V^{\pi_w, \psi_w}_{k+1}(\CF_k(s_k, \mu_{k, w}(s_k), y_k))\right]\nonumber \\
    &= \nabla_w\bm{1}_{s_k\in \CT_{k,w}}\left(r^S_T(s_k) - \EE_{y_k}\left[r^C_k(s_k, \mu_{k, w}(s_k), y_k) + V^{\pi_w, \psi_w}_{k+1}(\CF_k(s_k, \mu_{k, w}(s_k), y_k))\right]\right)\nonumber \\
    &\quad + \bm{1}_{s_k\notin \CT_{k,w}}\nabla_w\EE_{y_k}\left[r^C_k(s_k, \mu_{k, w}(s_k), y_k)+V^{\pi_w, \psi_w}_{k+1}(\CF_k(s_k, \mu_{k, w}(s_k), y_k))\right].\label{eq:gradient_with_indicator}
\end{align}
}
The first term in (\ref{eq:gradient_with_indicator}) is related to the gradient of an indicator function. To compute this, we define the boundary of the stopping set $\CT_{k,w}$ as
\begin{align}
    h(s_k, w) = r^S_T(s_k) - \EE_{y_k}\left[r^C_k(s_k, \mu_{k, w}(s_k), y_k) + V^{\pi_w, \psi_w}_{k+1}(\CF_k(s_k, \mu_{k, w}(s_k), y_k))\right] = 0.
\end{align}
The indicator function changes value only on this boundary. Therefore, the gradient can be expressed as a Dirac delta function centered on the boundary:
\begin{align}
\nabla_w \bm{1}_{s_k \in \CT_{k,w}} = \delta(h(s_k, w)) \nabla_w h(s_k, w).
\end{align}
The first term in (\ref{eq:gradient_with_indicator}) is then reduced to $h(s_k, w)\delta(h(s_k, w)) \nabla_w h(s_k, w)$, which is always 0 with the property of the Dirac Delta function $x\delta(x) \equiv 0$. For the second term in (\ref{eq:gradient_with_indicator}), 
{\small
\begin{align}
    &\quad \nabla_w\EE_{y_k}\left[r^C_k(s_k, \mu_{k, w}(s_k), y_k)+V^{\pi_w, \psi_w}_{k+1}(\CF_k(s_k, \mu_{k, w}(s_k), y_k))\right] \nonumber \\ 
    &= \nabla_w \Bigg[ \int_{y_k} p(y_k|s_k,\mu_{k,w}(s_k))r^C_k(s_k,\mu_{k,w}(s_k),y_k)\,dy_k + \int_{s_{k+1}} p(s_{k+1}|s_k,\mu_{k,w}(s_k)) V^{\pi_w, \psi_w}_{k+1}(s_{k+1}) \,ds_{k+1} \Bigg] \nonumber\\
    &= \nabla_w \int_{y_k} p(y_k|s_k,\mu_{k,w}(s_k))r^C_k(s_k,\mu_{k,w}(s_k),y_k)\,dy_k + \nabla_w \int_{s_{k+1}} p(s_{k+1}|s_k,\mu_{k,w}(s_k)) V^{\pi_w, \psi_w}_{k+1}(s_{k+1}) \,ds_{k+1} \nonumber\\
    &= \int_{y_k} \nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} \left[p(y_k|s_k,\xi_k) r^C_k(s_k,\xi_k,y_k)\right] \Big|_{\xi_k=\mu_{k,w}(s_k)} \,dy_k \nonumber \\
    &\quad + \int_{s_{k+1}} \Bigg[ p(s_{k+1}|s_k,\mu_{k,w}(s_k)) \nabla_w V^{\pi_w, \psi_w}_{k+1}(s_{k+1}) + \nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} p(s_{k+1}|s_k,\xi_k)\Big|_{\xi_k=\mu_{k,w}(s_k)} V^{\pi_w, \psi_w}_{k+1}(s_{k+1}) \Bigg] \,ds_{k+1} \nonumber\\
    &= \nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} \Bigg[ \int_{y_k} p(y_k|s_k,\xi_k) r^C_k(s_k,\xi_k,y_k) \,dy_k + \int_{s_{k+1}}  p(s_{k+1}|s_k,\xi_k)V^{\pi_w, \psi_w}_{k+1}(s_{k+1})ds_{k+1} \Bigg]\Bigg\vert_{\xi_k=\mu_{k,w}(s_k)} \nonumber\\
    &\quad + \int_{s_{k+1}} p(s_{k+1}|s_k,\mu_{k,w}(s_k)) \nabla_w V^{\pi_w, \psi_w}_{k+1}(s_{k+1}) \,ds_{k+1} \nonumber\\
    &= \nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} Q^{\pi_w, \psi_w}_{k}(s_k,\xi_k)\Big|_{\xi_k=\mu_{k,w}(s_k)} + \int_{s_{k+1}} p(s_k \rightarrow s_{k+1}|\pi_w) \nabla_w V^{\pi_w, \psi_w}_{k+1}(s_{k+1}) \,ds_{k+1}. \nonumber
\end{align}
}
Combining the two terms, we have
\begin{align}
    \nabla_w V^{\pi_w, \psi_w}_k(s_k) 
    &= \bm{1}_{s_k\notin \CT_{k,w}}\nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} Q^{\pi_w, \psi_w}_{k}(s_k,\xi_k)\Big|_{\xi_k=\mu_{k,w}(s_k)} 
    \label{e:gradV_recursive}\\
    &\quad + \bm{1}_{s_k\notin \CT_{k,w}}\int_{s_{k+1}} p(s_k \rightarrow s_{k+1}|\pi_w, \psi_w) \nabla_w V^{\pi_w, \psi_w}_{k+1}(s_{k+1}) \,ds_{k+1}. \nonumber
\end{align}
Applying the recursive (\ref{e:gradV_recursive}) to itself repeatedly, we obtain
{\small
\begin{align}
    &\quad \nabla_w V^{\pi_w, \psi_w}_k(s_k) \nonumber\\
    &= \bm{1}_{s_k\notin \CT_{k,w}}\nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} Q^{\pi_w, \psi_w}_k(s_k,\xi_k)\Big|_{\xi_k=\mu_{k,w}(s_k)} \nonumber\\
    &\quad + \bm{1}_{s_k\notin \CT_{k,w}}\int_{s_{k+1}} p(s_k \rightarrow s_{k+1}|\pi_w, \psi_w) \nonumber \\
    &\hspace{7em}\cdot \bm{1}_{s_{k+1}\notin \CT_{k+1,w}}\nabla_w \mu_{k+1,w}(s_{k+1}) \nabla_{\xi_{k+1}} Q^{\pi_w, \psi_w}_{k+1}(s_{k+1},\xi_{k+1})\Big|_{\xi_{k+1}=\mu_{k+1,w}(s_{k+1})} \,ds_{k+1} \nonumber\\
    &\quad + \bm{1}_{s_k\notin \CT_{k,w}}\int_{s_{k+1}} p(s_k \rightarrow s_{k+1}|\pi_w, \psi_w) \bm{1}_{s_{k+1}\notin \CT_{k+1,w}} \int_{s_{k+2}} p(s_{k+1} \rightarrow s_{k+2}|\pi_w, \psi_w) \nabla_w V^{\pi_w, \psi_w}_{k+2}(s_{k+2}) \,ds_{k+2} \,ds_{k+1} \nonumber\\
    &= \prod_{j=k}^{k} \bm{1}_{s_j\notin \CT_{j,w}}\nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} Q^{\pi_w, \psi_w}_k(s_k,\xi_k)\Big|_{\xi_k=\mu_{k,w}(s_k)} \nonumber\\
    &\quad + \int_{s_{k+1}} p(s_k \rightarrow s_{k+1}|\pi_w, \psi_w) \prod_{j=k}^{k+1} \bm{1}_{s_j \notin \CT_{j,w}} \nabla_w \mu_{k+1,w}(s_{k+1}) \nabla_{\xi_{k+1}} Q^{\pi_w, \psi_w}_{k+1}(s_{k+1},\xi_{k+1})\Big|_{\xi_{k+1}=\mu_{k+1,w}(s_{k+1})} \,ds_{k+1} \nonumber\\
    &\quad + \int_{s_{k+2}} p(s_{k} \rightarrow s_{k+2}|\pi_w, \psi_w) \prod_{j=k}^{k+1} \bm{1}_{s_j \notin \CT_{j,w}}\nabla_w V^{\pi_w, \psi_w}_{k+2}(s_{k+2}) \,ds_{k+2} \nonumber\\
    &= \prod_{j=k}^{k} \bm{1}_{s_j\notin \CT_{j,w}}\nabla_w \mu_{k,w}(s_k) \nabla_{\xi_k} Q^{\pi_w, \psi_w}_k(s_k,\xi_k)\Big|_{\xi_k=\mu_{k,w}(s_k)} \nonumber\\
    &\quad + \int_{s_{k+1}} p(s_k \rightarrow s_{k+1}|\pi_w, \psi_w) \prod_{j=k}^{k+1} \bm{1}_{s_j \notin \CT_{j,w}}\nabla_w \mu_{k+1,w}(s_{k+1}) \nabla_{\xi_{k+1}} Q^{\pi_w, \psi_w}_{k+1}(s_{k+1},\xi_{k+1})\Big|_{\xi_{k+1}=\mu_{k+1,w}(s_{k+1})} \,ds_{k+1} \nonumber\\
    &\quad + \int_{s_{k+2}} p(s_k \rightarrow s_{k+2}|\pi_w, \psi_w) \prod_{j=k}^{k+2} \bm{1}_{s_j \notin \CT_{j,w}}\nabla_w \mu_{k+2,w}(s_{k+2}) \nabla_{\xi_{k+2}} Q^{\pi_w, \psi_w}_{k+2}(s_{k+2},\xi_{k+2})\Big|_{\xi_{k+2}=\mu_{k+2,w}(s_{k+2})} \,ds_{k+2}\nonumber\\
    &\hspace{2.5em}\vdots \nonumber\\
    &\quad + \int_{s_{N}} p(s_{k} \rightarrow s_{N}|\pi_w, \psi_w) \prod_{j=k}^{N} \bm{1}_{s_j \notin \CT_{j,w}}\nabla_w V^{\pi_w, \psi_w}_N(s_{N})\,ds_{N} \nonumber\\
    &= \sum_{l=k}^{N-1} \int_{s_l} p(s_k \rightarrow s_l|\pi_w, \psi_w) \prod_{j=k}^{l} \bm{1}_{s_j \notin \CT_{j,w}}\nabla_w \mu_{l,w}(s_l) \nabla_{\xi_l} Q^{\pi_w, \psi_w}_l(s_l,\xi_l)\Big|_{\xi_l=\mu_{l,w}(s_l)} \,ds_l\nonumber\\
    &= \sum_{l=k}^{N-1} \EE_{s_l| \pi_w, \psi_w, s_k} \left[\prod_{j=k}^{l} \bm{1}_{s_j \notin \CT_{j,w}}\nabla_w \mu_{l,w}(s_l) \nabla_{\xi_l} Q^{\pi_w, \psi_w}_l(s_l,\xi_l)\Big|_{\xi_l=\mu_{l,w}(s_l)}\right] \,ds_l\label{e:gradV_final}
\end{align}
}
At last, we obtain the policy gradient expression:
\begin{align}
    \nabla_w U(w) &= \nabla_w V^{\pi_w, \psi_w}_0(s_0) \nonumber\\
    &= \sum_{l=0}^{N-1} \EE_{s_l|\pi_w, \psi_w, s_0} \left[\prod_{j=0}^{l} (1-\bm{1}_{s_j \in \CT_{j,w}}) \nabla_w \mu_{l,w}(s_l) \nabla_{\xi_l} Q^{\pi_w, \psi_w}_l(s_l,\xi_l)\Big|_{\xi_l=\mu_{l,w}(s_l)} \right] \nonumber\\
    &= \sum_{l=0}^{N-1} \EE_{s_l|\pi_w, \psi_w, s_0} \left[\bm{1}_{l<\tau_w} \nabla_w \mu_{l,w}(s_l) \nabla_{\xi_l} Q^{\pi_w, \psi_w}_l(s_l,\xi_l) \Big|_{\xi_l=\mu_{l,w}(s_l)} \right]. 
    \end{align}
\end{proof}

\subsection{Implementation details}\label{app:pseudocode}

This section addresses several practical challenges including neural network design for variable-length histories and efficient computation of information-theoretic rewards.

\paragraph{Policy Network Architecture} The policy network $\mu_w(k, s_k)$ takes as input both the current stage index and the experimental history. To handle the temporal nature of sequential experimentation, we design the input representation as follows: For the stage information, we employ one-hot encoding to capture the discrete temporal structure:
\begin{align}
    k \qquad \longrightarrow \qquad e_{k+1}=[0,\dots,0,\underbrace{1}_{(k+1)\rm{th}},0,\dots,0]^T.
\end{align}
for the second component, i.e., the state $s_k$ (including both $s_{k,b}$ and $s_{k,p}$), we represent it in a nonparametric manner
\begin{align}
s_k \qquad \longrightarrow \qquad \Hist_k=\{\xi_0,y_0,\dots,\xi_{k-1},y_{k-1}\},
\end{align}
which poses a challenge due to its variable length across different stages. We address this by using a fixed-size representation that accommodates the maximum possible history length. For experiments up to stage $(N-1)$, we allocate a total dimension of $(N-1)(N_{\xi}+N_y)$ and pad incomplete histories with zeros. The complete input vector for stage $k$ takes the form:
\begin{align}
    \label{eq:NN_input}
    [ \underbrace{e_{k+1}}_{N},\overbrace{\xi_0}^{N_{\xi}},\dots,\xi_{k-1},\underbrace{0,\dots,0}_{N_{\xi}(N-1-k)},\overbrace{y_0}^{N_y},\dots,y_{k-1},\underbrace{0,\dots,0}_{N_y(N-1-k)} ] ^ T.
\end{align}
This representation ensures consistent input dimensionality across all stages while preserving the sequential structure of the experimental process. The total input dimension scales linearly as $N + (N-1)(N_{\xi}+N_y)$. The output layer is an $N_{\xi}$-dimensional vector representing $\xi_k$, and the network architecture can be chosen by the user. 
\paragraph{Q-Network Architecture}The Q-network follows a similar input design but includes an additional component for the candidate design $\xi_k$, resulting in input dimension $N+(N-1)(N_{\xi}+N_y)+N_{\xi}$. The network outputs a scalar value representing the expected future return.

\paragraph{Information Gain Computation}
The terminal reward calculation requires computing KL divergences between posterior distributions. For problems without analytical posteriors, we approximate these quantities by discretizing the parameter space $\theta$ on a regular grid and estimating posterior densities pointwise.

\subsection{Analytical solution to the linear-Gaussian problem}\label{app:linear_gaussian_analytic_solution}
We derive the analytical solution to the linear-Gaussian problem described in section~\ref{sec:case_linear_Gaussian}, under the terminal reward formulation. The conjugate Gaussian structure enables us to obtain closed-form expressions for the optimal stopping and design policies. Due to this conjugacy, both the prior and posterior distributions remain Gaussian throughout the sequential process, with log-density functions given by:
\begin{align}
\ln p(\theta \mid I_0) &= -\frac{1}{2} \ln (2 \pi \sigma_0^2)-\frac{(m_0-\theta)^2}{2 \sigma_0^2} \\
\ln p(\theta \mid I_k) &= -\frac{1}{2} \ln (2 \pi \sigma_k^2)-\frac{(m_k-\theta)^2}{2 \sigma_k^2},
\end{align}
where $m_k$ and $\sigma_k^2$ denote the posterior mean and variance after performing $k$ experiments. These posterior parameters can be updated iteratively as:
\begin{align}
(m_k, \sigma_k^2) = \left(
\frac{
\frac{y_{k-1} / \xi_{k-1}}{\sigma_{\epsilon}^2 / \xi_{k-1}^2} + \frac{m_{k-1}}{\sigma_{k-1}^2}
}
{
\frac{1}{\sigma_{\epsilon}^2 / \xi_{k-1}^2} + \frac{1}{\sigma_{k-1}^2}
}, 
\frac{1}{\frac{1}{\sigma_{\epsilon}^2 / \xi_{k-1}^2} + \frac{1}{\sigma_{k-1}^2}} \right).
\end{align}
Alternatively, they can be computed directly from the complete experimental history:
\begin{align}
    \sigma_{k}^2 &= \frac{\sigma_0^2 \sigma_{\epsilon}^2}{\sigma_{\epsilon}^2 + \sigma_0^2\sum_{i=0}^{k-1}\xi_i^2 }\nonumber \\
    m_k &= \sigma_{k}^2 \left(\frac{m_0}{\sigma_0^2} + \frac{1}{\sigma_{\epsilon}^2}\sum_{i=0}^{k-1}y_i\xi_i \right).
\end{align}
For Gaussian distributions, the KL divergence has a closed form expression:
\begin{align}
    D_{\mathrm{KL}}(\mathcal{N}(m_a, \sigma_a^2) \,||\, \mathcal{N}(m_b, \sigma_b^2)) = \frac{1}{2}\left[\frac{\sigma_a^2}{\sigma_b^2} + \frac{(m_a-m_b)^2}{\sigma_b^2} + \ln\frac{\sigma_b^2}{\sigma_a^2} - 1\right].
\end{align}
This closed form allows us to derive explicit expressions for the expected information gain. Since $\sigma_{k+1}^2$ is updated independently of the observation $y_k$ for all $k$, the expected information gain from performing experiment $k$ is given by:
\begin{align}
    &\quad \EE_{y_k \mid s_k, \xi_k} \left[D_{\mathrm{KL}}\left(p_{\theta \mid \Hist_{k+1}} \,||\, p_{\theta \mid \Hist_k}\right)\right] \nonumber \\
    &= \EE_{y_k \mid s_k, \xi_k} \left[\frac{1}{2}\left\{\frac{\sigma_{k+1}^2}{\sigma_k^2} + \frac{(m_{k+1}-m_k)^2}{\sigma_k^2} + \ln\frac{\sigma_k^2}{\sigma_{k+1}^2} - 1\right\}\right] \nonumber \\
    &= \frac{1}{2}\EE_{y_k \mid s_k, \xi_k} \left[\frac{(m_{k+1}-m_k)^2}{\sigma_k^2}\right] + \frac{1}{2}\left(\frac{\sigma_{k+1}^2}{\sigma_k^2} + \ln\frac{\sigma_k^2}{\sigma_{k+1}^2} - 1\right) \nonumber \\
    &= \frac{1}{2}\EE_{y_k \mid s_k, \xi_k} \left[\frac{\sigma_k^2(y_k/\xi_k - m_k)^2}{(\sigma_k^2 + \sigma_\epsilon^2 / \xi_k^2)^2}\right] + \frac{1}{2}\left(\frac{\sigma_{k+1}^2}{\sigma_k^2} + \ln\frac{\sigma_k^2}{\sigma_{k+1}^2} - 1\right) \nonumber \\
    &= \frac{1}{2} \frac{\sigma_k^2 \xi_k^2}{\sigma_k^2 \xi_k^2 + \sigma_\epsilon^2} + \frac{1}{2}\left(\frac{\sigma_{k+1}^2}{\sigma_k^2} + \ln\frac{\sigma_k^2}{\sigma_{k+1}^2} - 1\right) \nonumber \\
    &= \frac{1}{2} \left(1 - \frac{\sigma_{k+1}^2}{\sigma_k^2}\right) + \frac{1}{2}\left(\frac{\sigma_{k+1}^2}{\sigma_k^2} + \ln\frac{\sigma_k^2}{\sigma_{k+1}^2} - 1\right) \nonumber \\
    &= \frac{1}{2} \ln\frac{\sigma_k^2}{\sigma_{k+1}^2}
\label{e:information_gain}
\end{align}
in which computing the expectations utilizes the identities
\begin{align}
\EE_{y_k \mid m_k, \sigma_k^2, \xi_k} \left[y_k\right] & =\int_{-\infty}^{+\infty} y_k p(y_k \mid m_k, \sigma_k^2, \xi_k) d y_k \nonumber \\
& =\int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} y_k p(y_k \mid \theta, m_k, \sigma_k^2, \xi_k) p(\theta \mid m_k, \sigma_k^2, \xi_k) d y_k d \theta \nonumber\\
& =\int_{-\infty}^{+\infty} \xi_k \theta p(\theta \mid m_k, \sigma_k^2, \xi_k) d \theta \nonumber\\
& =\xi_k m_k,
\label{e:Ey}
\end{align}
and
\begin{align}
\mathbb{E}_{y_k \mid m_k, \sigma_k^2, \xi_k} \left[y_k^2\right] & =\int_{-\infty}^{+\infty} y_k^2 p\left(y_k \mid m_k, \sigma_k^2, \xi_k\right) d y_k \nonumber\\
& =\int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} y_k^2 p\left(y_k \mid \theta, m_k, \sigma_k^2, \xi_k\right) p\left(\theta \mid m_k, \sigma_k^2, \xi_k\right) d y_k d \theta \nonumber\\
& =\int_{-\infty}^{+\infty}\left(\sigma_\epsilon^2+\xi_k^2 \theta^2\right) p\left(\theta \mid m_k, \sigma_k^2, \xi_k\right)  d \theta \nonumber\\
& =\sigma_\epsilon^2+\xi_k^2\left(\sigma_k^2+m_k^2\right).
\label{e:Ey2}
\end{align}
Using (\ref{e:information_gain}) we now present the first two steps in deriving the optimal sequential designs for the Linear-Gaussian case. For the last design,
\begin{align}
    \xi_{N-1}^* 
    &= \argmax_{\xi_{N-1}} Q^{\pi, \psi}_{N-1}(s_{N-1}, \xi_{N-1}) \nonumber \\
    &= \argmax_{\xi_{N-1}} \EE_{y_{N-1} \mid \xi_{N-1}} \left[ V^{\pi, \psi}_{N}(s_N) \right] \nonumber \\
    &= \argmax_{\xi_{N-1}} \left\{\EE_{y_{N-1} \mid \xi_{N-1}} \left[  D_{\mathrm{KL}} (p_{\theta \mid \Hist_N} \,||\, p_{\theta \mid \Hist_0})\right] + \sum_{i=0}^{N-1}c_i(\xi_i) \right\} \nonumber \\
    &= \argmax_{\xi_{N-1}} \Bigg\{ D_{\mathrm{KL}} (p_{\theta \mid I_{N-1}} \,||\, p_{\theta \mid \Hist_0}) + \sum_{i=0}^{N-2}c_i(\xi_i) + \EE_{y_{N-1} \mid \xi_{N-1}} \left[  D_{\mathrm{KL}} (p_{\theta \mid \Hist_N} \,||\, p_{\theta \mid \Hist_{N-1}})\right] + c_{N-1}(\xi_{N-1}) \Bigg\} \nonumber \\
    &= \argmax_{\xi_{N-1}} \left\{ \EE_{y_{N-1} \mid \xi_{N-1}} \left[  D_{\mathrm{KL}} (p_{\theta \mid \Hist_N} \,||\, p_{\theta \mid \Hist_{N-1}})\right] + c_{N-1}(\xi_{N-1}) \right\} \nonumber \\
    &= \argmax_{\xi_{N-1}} \left\{ \frac{1}{2} \ln\frac{\sigma_{N-1}^2}{\sigma_N^2} + c_{N-1}(\xi_{N-1}) \right\} \nonumber \\
    &= \argmax_{\xi_{N-1}} \left\{ \frac{1}{2} \ln \left[\left( \frac{1}{\sigma^2_{N-1}} + \frac{\xi_{N-1}^2}{\sigma_\epsilon^2}\right)\sigma^2_{N-1}\right] + c_{N-1}(\xi_{N-1}) \right\}
\end{align}
which achieves optimality at the maximum value $\xi_{N-1}^* = 3$ due to the monotone increasing property and design-independent costs. The corresponding maximum Q value is
\begin{align}
    Q_{N-1}^{\pi^*, \psi^*}(s_{N-1}, \xi^*_{N-1}) = D_{\mathrm{KL}} (p_{\theta \mid \Hist_{N-1}} \,||\, p_{\theta \mid \Hist_0}) + \frac{1}{2} \ln \frac{\sigma_{N-1}^2}{\sigma_{N}^{*2}} + c_{N-1}(\xi^*_{N-1}).
\end{align}
The stopping set for this stage is then
\begin{align}
    \CT_{N-1}^{\pi^*,\psi^*}
    &= \left\{s_{N-1} \mid r^S_T(s_{N-1}) \geq  Q_{N-1}^{\pi^*, \psi^*}(s_{N-1}, \xi^*_{N-1})\right\} \nonumber \\
    &= \left\{s_{N-1} \mid 0 \geq \frac{1}{2} \ln \frac{\sigma_{N-1}^2}{\sigma_{N}^{*2}} + c_{N-1}(\xi^*_{N-1})\right\},
\end{align}
which is independent of observations. Thus for stage $N-2$, 
\begin{align}
    Q^{\pi, \psi}_{N-2}(s_{N-2}, \xi_{N-2}) 
    &= \EE_{y_{N-2} \mid \xi_{N-2}} \max \left\{r_T^S(s_{N-1}),  Q_{N-1}^{\pi^*, \psi^*}(s_{N-1}, \xi^*_{N-1})\right\} \nonumber \\
    &= \max\left\{\EE_{y_{N-2} \mid \xi_{N-2}}\left[r_T^S(s_{N-1})\right],  \EE_{y_{N-2} \mid \xi_{N-2}}\left[Q_{N-1}^{\pi^*, \psi^*}(s_{N-1}, \xi^*_{N-1})\right]\right\}.
\end{align}
For the first case,
\begin{align}
    \xi_{N-2}^* 
    &= \argmax Q^{\pi, \psi}_{N-2}(s_{N-2}, \xi_{N-2}) \nonumber \\
    &= \argmax_{\xi_{N-2}} \left\{ \EE_{y_{N-2} \mid \xi_{N-2}} \left[  D_{\mathrm{KL}} (p_{\theta \mid \Hist_{N-1}} \,||\, p_{\theta \mid \Hist_0})\right] + \sum_{i=0}^{N-2}c_i(\xi_i) \right\} \nonumber \\
    &= \argmax_{\xi_{N-2}} \left\{\EE_{y_{N-2} \mid \xi_{N-2}} \left[  D_{\mathrm{KL}} (p_{\theta \mid \Hist_{N-1}} \,||\, p_{\theta \mid \Hist_{N-2}})\right] + c_{N-2}(\xi_{N-2})\right\} \nonumber \\
    &= \argmax_{\xi_{N-2}} \left\{\frac{1}{2} \ln \left[\left( \frac{1}{\sigma_{N-2}^2} + \frac{\xi_{N-2}^2}{\sigma_\epsilon^2}\right)\sigma_{N-2}^2\right] + c_{N-2}(\xi_{N-2})\right\}; 
\end{align}
for the second case,
\begin{align}
    \xi_{N-2}^* 
     &= \argmax Q^{\pi, \psi}_{N-2}(s_{N-2}, \xi_{N-2}) \nonumber \\
    &= \argmax_{\xi_{N-2}} \left\{\EE_{y_{N-2} \mid \xi_{N-2}} \left[  D_{\mathrm{KL}} (p_{\theta \mid \Hist_{N-1}} \,||\, p_{\theta \mid \Hist_0})\right] + \frac{1}{2} \ln \left(\frac{\sigma_{N-1}^2}{\sigma_N^{*2}}\right) + c_{N-1}(\xi^*_{N-1})\right\}  \nonumber \\
    &= \argmax_{\xi_{N-2}} \left\{ \frac{1}{2} \ln \frac{\sigma_{N-2}^2}{\sigma_{N-1}^2} + \frac{1}{2} \ln \left(\frac{\sigma_{N-1}^2}{\sigma_N^{*2}}\right) + c_{N-2}(\xi_{N-2}) + c_{N-1}(\xi^*_{N-1}) \right\} \nonumber \\
    &= \argmax_{\xi_{N-2}} \left\{ \frac{1}{2} \ln \left[\left( \frac{1}{\sigma_{N-2}^2} + \frac{\xi_{N-2}^2}{\sigma_\epsilon^2} + \frac{\xi_{N-1}^{*2}}{\sigma_\epsilon^2}\right)\sigma_{N-2}^2\right] + c_{N-2}(\xi_{N-2}) \right\}.
\end{align}
Both cases are equivalent in achieving the optimality at the maximum value $\xi_{N-2}^* = 3$, due to the  monotone increasing property and design-independent costs. Iteratively we have 
\begin{align}
    \xi_k^*
     = \argmax_{\xi_k} \frac{1}{2} \ln \left[\left( \frac{1}{\sigma_k^2} + \frac{\xi_k^2}{\sigma_\epsilon^2}\right)\sigma_k^2\right] = 3, \qquad \text{ for } k = 0, \ldots, N-1.
\end{align}
The stopping sets for each stage are
\begin{align}
    \CT_k^{\pi^*,\psi^*}
    &= \left\{s_k \mid 0 \geq \frac{1}{2} \ln \frac{\sigma_k^{*2}}{\sigma_{k+1}^{*2}} + c_{k}(\xi^*_{k})\right\} \nonumber \\
    &= \left\{s_k \mid 0 \geq \frac{1}{2} \ln \left(1 + \frac{\sigma_k^{*2}\sigma_0^2}{\sigma_\epsilon^{2} + \sigma_0^2\sum_{i=0}^{k-1}\xi_i^{*2}}\right)  + c_{k}(\xi^*_{k})\right\}
\end{align}
Finally the optimal utility for performing $N$ experiments is 
\begin{align}
    Q^*_0(s_0) &= \frac{1}{2} \ln \left[\left(\frac{1}{\sigma_0^2} + \frac{\xi_0^2}{\sigma_\epsilon^2} + \frac{\xi_1^{*2}}{\sigma_\epsilon^2} + \ldots + \frac{\xi_{N-1}^{*2}}{\sigma_\epsilon^2}\right) \sigma_0^2\right] + \sum_{i=0}^{N-1}c_i(\xi_i^*) \nonumber \\
    &= \frac{1}{2}\ln \left(\sigma_\epsilon^2 + \sigma_0^2\sum_{i=0}^{N-1}\xi_i^{*2}\right) - \frac{1}{2}\ln \sigma_\epsilon^2 + \sum_{i=0}^{N-1}c_i(\xi_i^*).
\end{align}
We list the optimal utilities for varying number of experiments in Table~\ref{tab: analytic_result}. The optimal stopping stages are highlighted in red.
\begin{table}[h]
\centering
\caption{Optimal Utility for varying experimental horizons.}
\label{tab: analytic_result}
\begin{tabular}{lllll}
 \hline
  $N$ &  1  &  2  & 3  & 4  
 \\
 \hline
  $U(\Xi_N)(c_k=0)$ & 2.203 & 2.547  & 2.749 & \color{red}2.892 
 \\
 \hline
 $U(\Xi_N)(c_k=-0.5)$ & \color{red}1.703 & 1.547  & 1.249 & 0.892 
 \\
 \hline
 $U(\Xi_N)(c_k=-0.25)$ & 1.953 & \color{red}2.047  & 1.999 & 1.892 
 \\
 \hline
\end{tabular}
\end{table}

\subsection{Numerical experiments details}
\subsubsection{Stopping probability schedule}\label{app:stopping_schedule}
We use an adaptive sigmoid-like stopping probability schedule, with $p_{\text{stop}} > 0.999$ for the final 30 training iterations to guarantee convergence to the true optimal stopping policy. Figure~\ref{fig:stopping_probability} illustrates this schedule over 300 training iterations.
\begin{figure}[!htb]
  \centering
  \includegraphics[width=.55\linewidth]{figures/stopping_probability1.pdf}
    \caption{Stopping probability schedule used in the curriculum learning approach over 300 training iterations.}
  \label{fig:stopping_probability}
\end{figure}

\subsubsection{Contaminant source detection problem setup}\label{app:contaminant_setup}
The contaminant concentration $G$ at spatial location $z=[z_x,z_y]$ and time $t$ follows convection-diffusion PDE:
\begin{align}
\frac{\partial G(z,t;\theta)}{\partial t} 
&= \nabla^2 G - u(t) \cdot \nabla G + S(z,t;\theta), \nonumber \\
&\qquad \qquad z \in [z_L, z_R]^2, \quad t > 0
\label{e:PDE}
\end{align}
where $\theta = [\theta_x,\theta_y,\theta_h,\theta_s] \in \RR^4$ parameterizes the Gaussian source function
\begin{align}
    S(z,t;\theta) = \frac{\theta_s}{2\pi \theta_h^2} \exp( -\frac{(\theta_x-z_x)^2 + (\theta_y-z_y)^2}{2 \theta_h^2} )
\end{align}
with location $(\theta_x,\theta_y)$, width $\theta_h$, and strength $\theta_s$. ${u}=[u_x,u_y] \in \RR^2$ is a time-dependent convection velocity. 

A mobile sensor measures noisy concentrations
\begin{align}
    y_k = G(z=s_{k+1,p}, t_k; \theta) + \epsilon_k
\end{align}
where $\epsilon_k \sim \mathcal{N}(0, \sigma_\epsilon^2)$. The design variable $\xi_k$ represents sensor displacement: 
\begin{align}
    s_{k+1, p} = s_{k, p} + \xi_k.
\end{align}
We consider $N$ measurement opportunities at times $t_k = k \cdot \Delta t$ for $k = 0, \ldots, N-1$, with $\Delta t = 5.0 \times 10^{-4}$. We focus on source localization with  $\theta_h = 0.05$, $\theta_s = 2$, $u_x = u_y = 10t/0.2$ and prior $\theta_x, \theta_y \sim \mathcal{U}(0, 1)$. 

Solving the forward model (\ref{e:PDE}) using finite volume is still computationally expensive. We use DNNs to construct surrogate models of $G(z, t_k; \theta)$ for $k = 0, \ldots, N-1$, to accelerate the computation. Each DNN uses a 4-dimensional input layer taking $z$ and $\theta$, 5 hidden layers with 40, 80, 40, 20, and 10 nodes, and a scalar output $G$. A dataset is generated by solving for $G$ on 2000 samples of $\theta$ drawn from its prior distribution. These concentration values are then first restricted to only the domain that is reachable by the vehicle (due to the design constraint), then shuffled across $\theta$ and split $80\%$ for training and $20\%$ for testing. 

\subsubsection{Additional results}\label{app:additional_results}
We present distributional analysis and representative episode comparisons to provide deeper insights into the learned policies.

The distributional analysis in Figure~\ref{fig:LinGau_Hist_T3_no_cost} for the linear-Gaussian benchmark shows that the trained policy predominantly terminates at stage 3 with designs concentrated near the upper bound ($\xi = 3$), consistent with analytical solutions.  The stopping stage histogram (left) confirms that the learned policy correctly identifies stage 3 as optimal when experimental costs are absent. The design histogram (right) reveals that optimal designs cluster near the constraint boundary, maximizing the signal-to-noise ratio by selecting the largest feasible design values. This concentration pattern validates that the policy gradient approach successfully learns the theoretically optimal behavior in this benchmark case.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.49\linewidth]{figures/LinearGaussian-T=3-Cost=0-StoppingHist.pdf}
  \includegraphics[width=0.49\linewidth]{figures/LinearGaussian-T=3-Cost=0-DesignHist.pdf}
  \caption{Distributional analysis of stopping stages (left) and experimental designs (right) for the linear-Gaussian benchmark with zero experimental cost ($c_k = 0$).}
  \label{fig:LinGau_Hist_T3_no_cost}
\end{figure}

Figure~\ref{fig:ConDiff_Posterior_T4_negative_cost} presents representative episodes from both methods in the contaminant source detection problem ($c_k = -0.8$). Vanilla policy terminates after two experiments with a total reward of 1.33, while curriculum-trained policy continues for three experiments, achieving a higher total reward of 1.99. The curriculum approach achieves superior expected performance and more stable convergence, which is precisely what the optimization objective seeks to maximize.

\begin{figure}[!htb]
  \centering
  \subfloat[Vanilla, $\theta=(0.61, 0.73)$, \textbf{total reward = 1.33} (information gain = 2.93, experimental cost = -1.6)]
  {\includegraphics[width=\linewidth]{figures/CD-posterior-T=4-Cost=-0.8-id=0-vanilla.pdf}}\\
  \subfloat[Curriculum, $\theta=(0.61, 0.73)$, \textbf{total reward = 1.99} (information gain = 4.39, experimental cost = -2.4)]
  {\includegraphics[width=\linewidth]{figures/CD-posterior-T=4-Cost=-0.8-id=0-curriculum.pdf}}
  \caption{Episode instances for vanilla (upper) and curriculum (lower) policies in the $c_k=-0.8$ scenario. Purple star: true source location; red dots: sensor positions; red lines: sensor movements; contours: posterior PDF.}
\label{fig:ConDiff_Posterior_T4_negative_cost}
\end{figure}

Finally, we explore a more complex cost structure in the contaminant source detection problem with design-dependent experimental costs $c_k = -\|\xi_k\|^2$, where sensor movements to distant locations incur higher penalties. Figure~\ref{fig:ConDiff_Cnvg_T4_varying_cost} shows training convergence under this quadratic cost structure. Curriculum learning again outperforms the vanilla method, achieving significant higher average rewards and converging to an optimal stopping stage around 3. This scenario demonstrates that the curriculum approach maintains its effectiveness even when cost structures become more complex and spatially dependent, suggesting broader applicability to realistic experimental design problems where costs vary with design choices.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.49\linewidth]{figures/ConvectionDiffusion-T=4-Cost=1quadratic-Reward-both.pdf}
  \includegraphics[width=0.49\linewidth]{figures/ConvectionDiffusion-T=4-Cost=1quadratic-tau-both.pdf}
  \caption{Training convergence of the convection-diffusion 
  problem with design-dependent costs $c_k = -\|\xi_k\|^2$. Left: average reward; right: average stopping stage.}
  \label{fig:ConDiff_Cnvg_T4_varying_cost}
\end{figure}


\end{document}
\begin{thebibliography}{29}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Murphy(2003)]{murphy2003optimal}
Susan~A Murphy.
\newblock Optimal dynamic treatment regimes.
\newblock \emph{Journal of the Royal Statistical Society Series B: Statistical
  Methodology}, 65\penalty0 (2):\penalty0 331--355, 2003.

\bibitem[Lookman et~al.(2019)Lookman, Balachandran, Xue, and
  Yuan]{lookman2019active}
Turab Lookman, Prasanna~V Balachandran, Dezhen Xue, and Ruihao Yuan.
\newblock Active learning in materials science with emphasis on adaptive
  sampling using uncertainties for targeted design.
\newblock \emph{npj Computational Materials}, 5\penalty0 (1):\penalty0 21,
  2019.

\bibitem[Krause et~al.(2008)Krause, Singh, and Guestrin]{krause2008near}
Andreas Krause, Ajit Singh, and Carlos Guestrin.
\newblock Near-optimal sensor placements in gaussian processes: Theory,
  efficient algorithms and empirical studies.
\newblock \emph{Journal of Machine Learning Research}, 9\penalty0 (2), 2008.

\bibitem[Peskir and Shiryaev(2006)]{peskir2006optimal}
Goran Peskir and Albert Shiryaev.
\newblock \emph{Optimal Stopping and Free-boundary Problems}.
\newblock Springer, 2006.

\bibitem[Chaloner and Verdinelli(1995)]{Chaloner1995}
Kathryn Chaloner and Isabella Verdinelli.
\newblock {Bayesian experimental design: A review}.
\newblock \emph{Statistical Science}, 10\penalty0 (3):\penalty0 273--304, 1995.

\bibitem[Ryan et~al.(2016)Ryan, Drovandi, Mcgree, and Pettitt]{Ryan2016}
Elizabeth~G. Ryan, Christopher~C. Drovandi, James~M. Mcgree, and Anthony~N.
  Pettitt.
\newblock {A review of modern computational algorithms for Bayesian optimal
  design}.
\newblock \emph{International Statistical Review}, 84\penalty0 (1):\penalty0
  128--154, 2016.

\bibitem[Alexanderian(2021)]{Alexanderian2021}
Alen Alexanderian.
\newblock {Optimal experimental design for infinite-dimensional Bayesian
  inverse problems governed by PDEs: A review}.
\newblock \emph{Inverse Problems}, 37\penalty0 (4):\penalty0 043001, 2021.

\bibitem[Rainforth et~al.(2024)Rainforth, Foster, Ivanova, and
  Smith]{Rainforth2024}
Tom Rainforth, Adam Foster, Desi~R Ivanova, and Freddie~Bickford Smith.
\newblock {Modern Bayesian experimental design}.
\newblock \emph{Statistical Science}, 39\penalty0 (1):\penalty0 100--114, 2024.

\bibitem[Huan et~al.(2024)Huan, Jagalur, and Marzouk]{Huan2024}
Xun Huan, Jayanth Jagalur, and Youssef Marzouk.
\newblock {Optimal experimental design: Formulations and computations}.
\newblock \emph{Acta Numerica}, 33:\penalty0 715--840, 2024.

\bibitem[Lindley(1956)]{Lindley1956}
Dennis~V. Lindley.
\newblock {On a Measure of the Information Provided by an Experiment}.
\newblock \emph{The Annals of Mathematical Statistics}, 27\penalty0
  (4):\penalty0 986--1005, 1956.

\bibitem[Foster et~al.(2021)Foster, Ivanova, Malik, and
  Rainforth]{foster2021deep}
Adam Foster, Desi~R. Ivanova, Ilyas Malik, and Tom Rainforth.
\newblock {Deep adaptive design: Amortizing sequential Bayesian experimental
  design}.
\newblock In \emph{Proceedings of the 38th International Conference on Machine
  Learning (ICML 2021)}, volume 139, pages 3384--3395, 2021.

\bibitem[Ivanova et~al.(2021)Ivanova, Foster, Kleinegesse, Gutmann, and
  Rainforth]{ivanova2021implicit}
Desi~R. Ivanova, Adam Foster, Steven Kleinegesse, Michael~U. Gutmann, and Tom
  Rainforth.
\newblock {Implicit deep adaptive design: Policy-based experimental design
  without likelihoods}.
\newblock In \emph{Advances in Neural Information Processing Systems 34}, pages
  25785--25798, 2021.

\bibitem[Blau et~al.(2022)Blau, Bonilla, Chades, and
  Dezfouli]{blau2022optimizing}
Tom Blau, Edwin~V. Bonilla, Iadine Chades, and Amir Dezfouli.
\newblock {Optimizing sequential experimental design with deep reinforcement
  learning}.
\newblock In \emph{Proceedings of the 39th International Conference on Machine
  Learning (ICML 2022)}, volume 162, pages 2107--2128, 2022.

\bibitem[Shen and Huan(2023)]{shen2023bayesian}
Wanggang Shen and Xun Huan.
\newblock Bayesian sequential optimal experimental design for nonlinear models
  using policy gradient reinforcement learning.
\newblock \emph{Computer Methods in Applied Mechanics and Engineering},
  416:\penalty0 116304, 2023.

\bibitem[Shen et~al.(2025)Shen, Dong, and Huan]{Shen2025}
Wanggang Shen, Jiayuan Dong, and Xun Huan.
\newblock Variational sequential optimal experimental design using
  reinforcement learning.
\newblock \emph{Computer Methods in Applied Mechanics and Engineering},
  444:\penalty0 118068, 2025.

\bibitem[Zhu et~al.(2010)Zhu, Wang, Hovy, and Ma]{zhu2010confidence}
Jingbo Zhu, Huizhen Wang, Eduard Hovy, and Matthew Ma.
\newblock Confidence-based stopping criteria for active learning for data
  annotation.
\newblock \emph{ACM Transactions on Speech and Language Processing (TSLP)},
  6\penalty0 (3):\penalty0 1--24, 2010.

\bibitem[Pullar-Strecker et~al.(2024)Pullar-Strecker, Dost, Frank, and
  Wicker]{pullar2024hitting}
Zac Pullar-Strecker, Katharina Dost, Eibe Frank, and J{\"o}rg Wicker.
\newblock Hitting the target: Stopping active learning at the cost-based
  optimum.
\newblock \emph{Machine Learning}, 113\penalty0 (4):\penalty0 1529--1547, 2024.

\bibitem[Audibert and Bubeck(2010)]{audibert2010best}
Jean-Yves Audibert and S{\'e}bastien Bubeck.
\newblock Best arm identification in multi-armed bandits.
\newblock In \emph{COLT-23th Conference on Learning Theory}, pages 41--53,
  2010.

\bibitem[Daskalakis and Kawase(2017)]{daskalakis2017optimal}
Constantinos Daskalakis and Yasushi Kawase.
\newblock Optimal stopping rules for sequential hypothesis testing.
\newblock In \emph{25th Annual European Symposium on Algorithms (ESA)}, 2017.

\bibitem[Shiryaev(2008)]{shiryaev2008optimal}
Albert~N Shiryaev.
\newblock \emph{Optimal Stopping Rules}.
\newblock Springer, 2008.

\bibitem[Haggstrom(1966)]{haggstrom1966optimal}
Gus~W Haggstrom.
\newblock Optimal stopping and experimental design.
\newblock \emph{The Annals of Mathematical Statistics}, pages 7--29, 1966.

\bibitem[Bertsekas(2012)]{bertsekas2012dynamic}
Dimitri Bertsekas.
\newblock \emph{Dynamic Programming and Optimal Control: Volume I}.
\newblock Athena scientific, 2012.

\bibitem[Puterman(2014)]{puterman2014markov}
Martin~L Puterman.
\newblock \emph{Markov Decision Processes: Discrete Stochastic Dynamic
  Programming}.
\newblock John Wiley \& Sons, 2014.

\bibitem[Fathan and Delage(2021)]{fathan2021deep}
Abderrahim Fathan and Erick Delage.
\newblock Deep reinforcement learning for optimal stopping with application in
  financial engineering.
\newblock \emph{arXiv preprint arXiv:2105.08877}, 2021.

\bibitem[Li and Lee(2023)]{li2023deltav}
Xiying Li and Chi-Guhn Lee.
\newblock $\delta$v-learning: an adaptive reinforcement learning algorithm for
  the optimal stopping problem.
\newblock \emph{Expert Systems with Applications}, 231:\penalty0 120702, 2023.

\bibitem[Berry et~al.(2002)Berry, M\"{u}ller, Grieve, Smith, Parke, Blazek,
  Mitchard, and Krams]{Berry2002}
Donald~A. Berry, Peter M\"{u}ller, Andy~P. Grieve, Michael Smith, Tom Parke,
  Richard Blazek, Neil Mitchard, and Michael Krams.
\newblock \emph{Adaptive Bayesian Designs for Dose-Ranging Drug Trials}, page
  99–181.
\newblock Springer New York, 2002.

\bibitem[Silver et~al.(2014)Silver, Lever, Heess, Degris, Wierstra, and
  Riedmiller]{pmlr-v32-silver14}
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and
  Martin Riedmiller.
\newblock Deterministic policy gradient algorithms.
\newblock In \emph{Proceedings of the 31st International Conference on Machine
  Learning}, pages 387--395, 2014.

\bibitem[Wang et~al.(2020)Wang, Cai, Yang, and Wang]{Wang2020Neural}
Lingxiao Wang, Qi~Cai, Zhuoran Yang, and Zhaoran Wang.
\newblock Neural policy gradient methods: Global optimality and rates of
  convergence.
\newblock In \emph{International Conference on Learning Representations}, 2020.

\bibitem[Sutton and Barto(2018)]{sutton2018reinforcement}
Richard~S Sutton and Andrew~G Barto.
\newblock \emph{Reinforcement Learning: An Introduction}.
\newblock MIT press, 2018.

\end{thebibliography}