Reinforcement Learning with Random Time Horizons

Extracted source text · arXiv 2506.00962

\documentclass{article}

\usepackage[accepted]{icml2025}

\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} 
\usepackage[noend]{algpseudocode}
\usepackage[hidelinks]{hyperref}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{bbm}
\usepackage[capitalize,noabbrev]{cleveref}

\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}

\newcommand{\theHalgorithm}{\arabic{algorithm}}
\setlength{\tabcolsep}{0.4em}
\newcommand{\R}{\mathbb{R}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\f}{\mu}
\newcommand{\Z}{Z}
\newcommand{\X}{X}
\newcommand{\cX}{\mathcal{X}}
\newcommand{\cY}{\mathcal{Y}}
\newcommand{\cZ}{\mathcal{Z}}
\newcommand{\cm}{m}
\newcommand{\tr}{\operatorname{Tr}}
\newcommand{\E}{\mathbbm{E}}
\renewcommand{\P}{\mathbbm{P}}
\renewcommand{\S}{\mathcal{S}}
\newcommand{\A}{\mathcal{A}}
\newcommand{\Q}{\mathbbm{Q}}
\newcommand{\V}{\operatorname{Var}}
\newcommand{\Var}{\operatorname{Var}}
\newcommand{\KL}{\operatorname{KL}}
\newcommand{\LV}{\operatorname{LV}}
\renewcommand{\div}{\operatorname{div}}
\DeclareMathOperator*{\argmin}{arg\,min}

\icmltitlerunning{Reinforcement Learning with Random Time Horizons}

\begin{document}

\twocolumn[
\icmltitle{Reinforcement Learning with Random Time Horizons}

\icmlsetsymbol{equal}{*}

\begin{icmlauthorlist}
\icmlauthor{Enric Ribera Borrell}{equal,zib,fub}
\icmlauthor{Lorenz Richter}{equal,zib,dida}
\icmlauthor{Christof Schütte}{zib,fub}
\end{icmlauthorlist}

\icmlaffiliation{zib}{Zuse Institute Berlin, 14195 Berlin, Germany}
\icmlaffiliation{fub}{Institute of Mathematics, Free University Berlin, 14195 Berlin, Germany}
\icmlaffiliation{dida}{dida Datenschmiede GmbH, 10827 Berlin, Germany}

\icmlcorrespondingauthor{Enric Ribera Borrell}{ribera.borrell@zib.de}
\icmlcorrespondingauthor{Lorenz Richter}{richter@zib.de}

\icmlkeywords{Machine Learning, ICML}

\vskip 0.3in
]

\printAffiliationsAndNotice{\icmlEqualContribution}

\begin{abstract}
We extend the standard reinforcement learning framework to random time horizons. While the classical setting typically assumes finite and deterministic or infinite runtimes of trajectories, we argue that multiple real-world applications naturally exhibit random (potentially trajectory-dependent) stopping times. Since those stopping times typically depend on the policy, their randomness has an effect on policy gradient formulas, which we (mostly for the first time) derive rigorously in this work both for stochastic and deterministic policies. We present two complementary perspectives, trajectory or state-space based, and establish connections to optimal control theory. Our numerical experiments demonstrate that using the proposed formulas can significantly improve optimization convergence compared to traditional approaches.
\end{abstract}

%\icmlkeywords{reinforcement learning, policy gradient theorem, random stopping times, optimal control}

\section{Introduction}

The goal in reinforcement learning is to identify policies $\pi$ that maximize the expected return
\begin{equation}
    J(\pi) = \E_{\pi}\left[\sum_{n=0}^{N} r_n(S_n, A_n) \right],
\end{equation}
where the states $S_n$ and the actions $A_n$ run until a time $N$. While the classical framework considers the case where $N$ is either finite and deterministic or infinite, runtimes can as well be random. To name two examples, one could consider robots that aim to reach a target in a noisy environment or one could play a game whose terminating state depends on reaching a certain level. Furthermore, considering stochastic policies (as commonly done in practice in order to balance exploration and exploitation) leads to the fact that runtimes are random even in deterministic environments. In all those cases the termination depends on previous (random) choices and is therefore random itself. However, the common literature has so far largely ignored the fact that runtimes can be random. In particular, policy gradient theorems typically assume that they are either deterministic or infinite \cite{sutton2018reinforcement}. Nonetheless, it is intuitively evident that considering random times should have an effect on optimization formulas since the runtimes mostly depend on the current policy such that in principle one should be careful with taking gradients (cf. \citet{nota2019policy}). 

In this work, we close this theoretical gap and systematically investigate the effect of random time horizons in reinforcement learning problems. Our contributions can be summarized as follows:
    \begin{itemize}
        \item We extend the typical reinforcement learning framework to random time horizons, including (deterministic) finite and infinite runtimes as special cases.
        \item We state corresponding (random time) policy gradient theorems both for deterministic and stochastic policies.
        \item This allows us for the first time to rigorously derive a trajectory-based gradient estimator for random time horizons (for stochastic as well as deterministic policies) in discrete time.
        \item For deterministic policies we derive a novel model-based policy gradient formula, which does not rely on learning the $Q$-value function as in actor-critic approaches.
        \item In multiple  numerical experiments, we systematically investigate the effect of incorporating the randomness of the time horizon in the gradient computation. For most cases we can see significant performance improvements of our gradient formulas compared to the standard ones, in particular in terms of convergence speed.
    \end{itemize}


\subsection{Related work}

Gradient based on-policy optimization in reinforcement learning has been pioneered by \citet{sutton1999policy}, where a policy gradient theorem for infinite time horizons and discrete state spaces has been suggested. In \citet{silver2014deterministic} it has been extended to deterministic policies and continuous state-spaces, still in infinite time. In the seminal monograph from \citet{sutton2018reinforcement} random runtimes are mentioned, however, not studied. To the best of our knowledge, the only work that studies trajectory-dependent random runtimes is from \citet{bojun2020steady}. However, it solely focuses on discrete state and action spaces and considers only stochastic policies, mostly relying on Markov chain theory for proving the statements. Our work, in contrast, operates in continuous spaces and adds formulas for deterministic policies. For further work on random time horizons in reinforcement learning, we refer to \citet{mandal_online_2023} and \citet{chen2024finite}, who, however, only study trajectory-independent stopping times.

We further refer to \citet{nota2019policy}, who show that taking heuristic versions of policy gradients (e.g. dropping discount factors without motivation) might lead to wrong expressions and that in general one should be careful with employing algorithms that are not backed by theory. Also, we refer to \citet{white_unifying_2017}, who unifies finite and infinite time horizons by introducing generalized (transition-based) discount factors, which might in principle be enhanced to random time problems.

Finally, we want to mention that random time horizons are more popular in stochastic optimal control settings, which consider deterministic policies and are typically stated in continuous time \cite{pham2009continuous}. We want to highlight \citet{ribera2024improving}, who derive trajectory-based gradient estimators for random time optimal control problems. In fact, for discrete times, we can generalize their result to non-Gaussian transition densities. We refer to \citet{lie2021fr}, who analyzes convexity properties of cost functionals with random stopping times, and to \citet{zhou2021actor}, who suggest an algorithm valid for random stopping times in optimal control that is inspired by the actor-critic method. Lastly, we refer to \citet{schutte2023overcoming} for random stopping time problems and related optimization methods in molecular dynamics and to \citet{quer_connecting_2024} who approach optimal control problems appearing in molecular dynamics applications with reinforcement learning techniques.


\subsection{Outline of the article}

In \Cref{sec: reinforcement learning with random time horizons} we formally state the reinforcement learning setting allowing for random runtimes. \Cref{sec: trajectory vs. state-space perspective} comments on taking either trajectory or state-space perspectives on this setting. Using both perspectives, we then derive policy gradient theorems incorporating random time horizons in \Cref{sec: optimization and policy gradient theorems}. In \Cref{sec: numerical experiments} we compare our novel gradient formulas to classical formulas in numerical experiments. Finally, \Cref{sec: conclusion} concludes and provides an outlook for future research.

\section{Reinforcement learning with random time horizons}
\label{sec: reinforcement learning with random time horizons}

In the following we will formally introduce the reinforcement learning setting we consider in this article. Further details on the notation can be found in \Cref{app: notation}. We consider the continuous state-space $\mathcal{S} \subset \mathbb{R}^{d_s}$ and the continuous action space $\mathcal{A} \subset \mathbb{R}^{d_a}$ and define a time-discrete \textit{Markov decision process} $\mathcal{M} = (\mathcal{S}, \mathcal{A}, r_n, \rho_0, p)$ on the probability space $(\Omega, \mathcal{F}, \mathbb{P})$. Here the reward function $r_n: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ provides the signal that is received after being in state $S_n \in \mathcal{S}$ and having taken action $A_n \in \mathcal{A}$ at time\footnote{In principle, the reward function can be explicitly time-dependent, however, for random time horizons one typically chooses it to be time-independent.} $n \in \mathbb{N}$. The function $\rho_0: \mathcal{S} \rightarrow \mathbb{R}_{\ge 0}$ represents the initial probability density of the states and $p: \mathcal{S} \times \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}_{\ge 0}$ is the time-homogeneous (state-action) transition probability density. To be more precise, let $\Lambda \subset \mathcal{S}$, then the probability of starting in a set $\Lambda$ is given by
\begin{equation}
\label{eq: initial density}
\mathbb{P}(S_0 \in \Lambda) = \int_{\Lambda} \rho_0(s)\mathrm{d} s
\end{equation}
and the probability of transitioning into the set $\Lambda$ conditioned on being in state $s \in \mathcal{S}$ and having chosen action $a \in \mathcal{A}$ at time $n \in \mathbb{N}$ is given by
\begin{equation}
\label{eq: state-action transition density}
\mathbb{P}(S_{n+1} \in \Lambda \mid S_n=s, \, A_n=a ) = \int_{\Lambda} p(s', s, a) \mathrm{d} s'.
\end{equation}

A \emph{policy} is a decision rule determining which action $A_n \in \mathcal{A}$ to take when being in state $S_n \in \mathcal{S}$ at time $n \in \mathbb{N}$. In this work, we consider stationary Markovian policies, i.e., for every time step the decision rule is the same and the choice of the action at time $n$ does not depend on the previous states $S_0, \dots, S_{n-1}$ of the trajectory. We distinguish between two types of policies:
\begin{itemize}
\item A \emph{deterministic policy} $\mu: \mathcal{S} \rightarrow \mathcal{A}$ is a function that directly maps each state to an action.
\item A \emph{stochastic policy} $\pi: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}_{\ge 0}$ is the probability density of taking an action in a given state. To be more precise, let $M \subset \mathcal{A}$, then the probability to choose an action in $M$ when being in state $s \in \mathcal{S}$ is given by
\begin{equation}
\label{eq: stat stoch policy}
\mathbb{P}(A_n \in M \mid S_n = s) = \int_M \pi(s, a) \mathrm{d}a .
\end{equation}
\end{itemize}



\begin{remark}[Deterministic as stochastic policy]
\label{rem: deterministic as stochastic policy}
Formally, a deterministic policy $\mu$ can be expressed as a stochastic policy via $\pi(s, a) = \delta(\mu(s) - a)$, where $\delta$ is the Dirac delta distribution. In the sequel, we will therefore often only refer to the stochastic policy $\pi$ and only mention the deterministic policy $\mu$ explicitly when necessary.
\end{remark}

The goal in reinforcement learning is to identify policies $\pi$ that maximize the expected return (i.e. cumulative reward)
\begin{equation}
\label{eq: expected return}
    J(\pi) = \E_{\pi}\left[\sum_{n=0}^{N} r_n(S_n, A_n) \right].
\end{equation}


We note that the randomness of the return depends on the policy $\pi$ and the densities $\rho_0, p$ defined before. For notational convenience, only the learnable $\pi$ is included in the subscript of the expectation operator (see \Cref{app: notation} for more details). Typically, one either considers $N$ to be fixed and deterministic\footnote{Markov decision processes with finite runtimes are often called \textit{episodic}, whereas infinite time processes are called \textit{continuing}. For the episodic case, some works include random, wheres some assume deterministic runtimes. However, computational consequences for random runtimes are typically not studied.} or $N=\infty$. In the latter case one has to discount the reward function, e.g. via $r_n(s, a) = \gamma^n r(s, a)$, where $\gamma \in (0, 1)$ is a discount factor and $r : \S \times \A \to \R$ is not explicitly time-dependent. In this paper we generalize this setting to time horizons $N$ that can be random. In practice, $N$ often depends on the (random) trajectory and is defined as the first time reaching a predefined set of terminal states $\mathcal{T} \subset \S$, i.e.
\begin{equation}
    N := \min \{ n \in \mathbbm{N} : S_n \in \mathcal{T} \}.
\end{equation}
We note that, due to the dependency on the trajectory, $N$ also depends on the policy $\pi$. Throughout this paper, we assume that $N$ is almost surely finite, i.e. $\P(N < \infty) = 1$.

Interestingly, as noted already in \citet{derman1970finite} (see also Chapter 5.3 in \citet{puterman2014markov}), the discounted infinite horizon case can be considered as a random time horizon problem\footnote{We note, however, that due to the time-dependent reward function, the starting time matters for infinite time horizon problems in this setting.}. For convenience, we state the proof in \Cref{app: proofs}.

\begin{lemma}[Random time perspective on discounted infinite time problem.]
\label{lem: discounted infinite as random time problem}
    Let $N_\gamma$ be a geometrically distributed random variable with success probability $1 - \gamma$, where $\gamma \in (0, 1)$. Then, choosing  $N = N_\gamma$ and $r_n = r$ in objective \eqref{eq: expected return} is equivalent to choosing $N = \infty$ and $r_n = \gamma^n r$.
\end{lemma}

A notable conclusion is that infinite time horizon problems can in fact be seen as problems that are finite with probability one. To be precise, they can be interpreted as random time horizon problems where the trajectory can be arbitrarily long, however, (almost surely) not infinitely long. We note, however, that the equivalence only holds in expectation and numerical implications might have to be studied further, cf. \citet{paternain2018stochastic}, \citet{zhang2020global} and \citet{mandal_online_2023}.



\subsection{Trajectory vs. state-space perspective}
\label{sec: trajectory vs. state-space perspective}

As commonly done, let us in the following assume the reward function to be not explicitly time-dependent, i.e. we set $r_n = r$. For the reinforcement learning problem defined above, we can consider two complementary perspectives, leading to different formulas and different implementations. We can either average the reward function \eqref{eq: expected return} over trajectories, such that the expectation is taken w.r.t. \textit{discrete path measures}, or we can view the problem on a state-space level, not inherently incorporating any dynamics. Let us elaborate in the following.


\textbf{Trajectory perspective.} This viewpoint is the one described above, i.e., we consider trajectories $(S_n, A_n)_{n=0}^N$ that evolve over time, given transition densities for the states and policies for the actions. 


\textbf{State-space perspective.} In this viewpoint we define the state-space density $\rho^\pi$ (sometimes also called \textit{on-policy distribution}) as the density of the process $S$ under policy $\pi$. Recall that the density is independent of time since we consider stationary problems. It can be defined as follows. First, we define the function $\rho^\pi_n$ s.t. for all $\Lambda \subset \mathcal{S}$ it holds
\begin{equation}
\label{def: n-step function}
    \int_\Lambda \rho^\pi_n(s) \mathrm ds = \P_\pi(S_n \in \Lambda),
\end{equation}
so it measures the probability of being in a state at a given time\footnote{We assume that the process $S$ stops after $N$ steps.} (to be even more precise, the above probability means $\P_\pi(S_n \in \Lambda) = \P_\pi(S_n \in \Lambda, n \le N)$). We can now define the state-space density as the probability of being in a certain state irrespective of the time via
\begin{equation}
\label{eq: definition state-space density}
    \rho^\pi := \frac{\eta^\pi}{Z^\pi}, \quad \eta^\pi := \sum_{n = 0}^\infty \rho^\pi_n, \quad Z^\pi := \int_{\mathcal{S}} \eta^\pi(s) \mathrm ds.
\end{equation}

In other words, $\rho^\pi$ measures the frequency of visiting a certain region at some point during the evolution of the trajectory and thus does not contain time information anymore. To make this observation more precise, the following lemma shows that the (unnormalized) function $\eta^\pi$ defined in \eqref{eq: definition state-space density} can be interpreted as counting how often trajectories stay in certain regions of the space, see \Cref{app: proofs} for the proof.

\begin{lemma}
\label{lem: eta region counting}
For a set $\Lambda \subset \mathcal{S}$ it holds
\begin{equation}
\int_\Lambda \eta^\pi(s)\mathrm ds = \E_\pi\left[\sum_{n=0}^N \mathbbm{1}_\Lambda(S_n) \right]
\end{equation}
and in particular
\begin{equation}
Z^\pi = \int_S \eta^\pi(s)\mathrm ds = \E_\pi\left[N +1 \right].
\end{equation}
\end{lemma}

The state-space perspective is particularly prominent in reinforcement learning since it readily allows for off-policy learning strategies, see, e.g., \citet{degris2012off, mnih2013playing, lillicrap2015continuous}. We refer to \citet{agarwal_reinforcement_2022} for additional context on the state-space perspective in terms of so-called occupational measures and refer to \Cref{fig: trajectory vs. state-space perspective} for an illustration of trajectory and state-space perspectives.

\begin{figure}
\centering
\includegraphics[width=\linewidth]{img/dw2d_traj-vs-state-space-prespective.pdf}
\caption{We illustrate trajectory and state-space perspectives on the reinforcement learning problem. In the left panel, we plot three trajectories that start at the black crosses, respectively, and run until hitting the target set displayed in red under the dynamics that is governed by a multi-well potential, see \Cref{sec: double well numerics} for details. The right-hand side displays the corresponding state-space density $\rho^\pi$, as defined in \eqref{eq: definition state-space density}. In the first row we choose the initial (deterministic) policy that only returns zero actions and in the second row we consider the (learned) optimal policy according to the problem defined in \Cref{sec: double well numerics}.
}
\label{fig: trajectory vs. state-space perspective}
\end{figure}

We can now state the state-space version of the expected return defined in \eqref{eq: expected return}, which has been derived in Theorem 4 in \citet{bojun2020steady} in a discrete space setting.

\begin{proposition}[State-space expected return] 
\label{prop: costs state-space perspective}
    Assuming $\P(N < \infty) = 1$, the expected return \eqref{eq: expected return} can be written as
\begin{equation}
\label{eq: state-space expected return}
    J(\pi) = \E_\pi[N + 1] \, \E_{\substack{s \sim \rho^\pi \\ a \sim \pi(s, \cdot)}} \left[ r(s, a) \right],
\end{equation}
where $\rho^\pi$ is the state-space density defined in \eqref{eq: definition state-space density}.
\end{proposition}

Contrary to classical results, we note the appearance of the expected runtime in \eqref{eq: state-space expected return}, acting as a scaling factor. Loosely speaking, with the change from the trajectory to the state-space perspective one omits the dynamics and therefore the runtime information, and \Cref{prop: costs state-space perspective} shows the correct way of integrating this information back into the state-space perspective. We refer to \Cref{app: proofs} for the proof and to \Cref{rem: alternative derivation of state-space expected return} for an alternative derivation, which might also bring some further intuition.

For infinite time horizons, often the identity 
\begin{equation}
    \bar{J}(\pi) =  \E_{\substack{s \sim \widetilde{\eta}^\pi \\ a \sim \pi(s, \cdot)}} \left[ r(s, a) \right]
\end{equation}
is stated, where, in analogy to \eqref{eq: definition state-space density}, $\widetilde{\eta}^\pi := \sum_{n = 0}^\infty \gamma^{n} \rho^\pi_n$ is called \textit{discounted state distribution}, see e.g. \citet{silver2014deterministic}. We note, however, that $\widetilde{\eta}^\pi$ is not a density since it is not normalized, and the correct expression for the infinite time case would be
\begin{equation}
    J(\pi) = \frac{1}{1 - \gamma}  \E_{\substack{s \sim \widetilde{\rho}^\pi \\ a \sim \pi(s, \cdot)}} \left[ r(s, a) \right],
\end{equation}
where $\widetilde{\rho}^\pi$ is the normalized version of $\widetilde{\eta}^\pi$, thus aligning with \Cref{lem: discounted infinite as random time problem}.

\vspace{0.5cm}

\begin{remark}[Sampling from $\rho^\pi$]
    While we call expectations w.r.t. $\rho^\pi$ state-space-based, we note that typically, in order to approximate the expectations numerically, one still needs to simulate trajectories. However, at the same time, as mentioned before, the state-space perspective allows to readily design off-policy learning strategies and to use replay buffers (cf. \citet{degris2012off}). 
\end{remark}

\vspace{0.2cm}

\subsection{Optimization and policy gradient theorems}
\label{sec: optimization and policy gradient theorems}

The notorious question in reinforcement learning is how to optimize the expected return \eqref{eq: expected return} w.r.t. the policy $\pi$ (for stochastic policies) or w.r.t. the function $\mu$ (for deterministic policies). In practice, we typically assume that $\pi$ or $\mu$ are parametrized by the parameter vector $\theta \in \R^p$, and we may write $\pi_\theta$ or $\mu_\theta$, respectively. For ease of notation, we often omit the parameter dependence, however.

A quantity that often appears in the context of optimization is the so-called \textit{$Q$-value function} defined as the expected return conditioned on a state-action pair,
\begin{equation}
\label{eq: def Q-value function}
    Q^\pi(s, a) := \E_\pi \left[ \sum_{n=0}^N r(S_n, A_n) \Big| S_0 = s, A_0 = a \right].
\end{equation}

We note that starting at $n = 0$ holds without loss of generality and one could also start at any other time, since the problem is time-autonomous. Further, one can define the  \textit{value function} (or \textit{return-to-go}) as $V^\pi(s) = \E_{a \sim \pi (s, \cdot)}\left[Q^\pi(s, a) \right]$.
We note that for the expected return \eqref{eq: expected return} it holds $J(\pi) = \E_{s \sim \rho_0}\left[V^\pi(s) \right]$ and refer to \Cref{app: background on RL} for further details.

\vspace{0.2cm}

The following statement shows how one can compute gradients w.r.t. stochastic policies of expected returns that incorporate random stopping times. A proof can be found in \Cref{app: proofs}.

\vspace{0.5cm}

{\begin{proposition}[Policy gradient]
\label{prop: stochastic policy gradient}
For the gradient of the expected return \eqref{eq: expected return} it holds
\begin{align}
\label{eq: policy gradient trajectory perspective}
&\nabla_\theta J(\pi_\theta) = \E_\pi \left[ \sum_{n=0}^N \nabla_\theta \log \pi_\theta(S_n, A_n) Q^{\pi}(S_n, A_n) \right] \\
\label{eq: policy gradient state-space perspective}
&\quad = \E_\pi[N + 1] \, \E_{\substack{s \sim \rho^\pi \\ a \sim \pi(s, \cdot)}} \left[ \nabla_\theta \log \pi_\theta(s, a) Q^{\pi}(s, a) \right],
\end{align} 
where $\rho^\pi$ is the state-space density defined in \eqref{eq: definition state-space density}.
\end{proposition}}

We note that -- analogous to \Cref{sec: trajectory vs. state-space perspective} -- \Cref{prop: stochastic policy gradient} provides formulas following either a trajectory perspective, stated in \eqref{eq: policy gradient trajectory perspective}, or a state-space perspective, as in \eqref{eq: policy gradient state-space perspective}. For the former we note that a rigorous derivation for random time horizons has to the best of our knowledge not been conducted before\footnote{In fact, assuming $N$ to be deterministic yields the same trajectory-based formula as in \eqref{eq: policy gradient trajectory perspective} and it is remarkable that the dependency of $N$ on $\pi$ in the random stopping time case does not add any extra term.}. Further, we note that, in contrast to deterministic stopping times, one cannot simply apply automatic differentiation of the expected return w.r.t. the parameter $\theta$ for optimization due to the policy-dependency of the runtime. However, our formula \eqref{eq: policy gradient trajectory perspective} suggests the alternative objective
\begin{equation}
J_\mathrm{eff}(\pi_\theta, \pi_\vartheta) := \E_{\pi_\vartheta} \left[\sum_{n=0}^N \log \pi_\theta(S_n, A_n) Q^{\pi_\vartheta}(S_n, A_n) \right],
\end{equation}
which can be (auto-)differentiated w.r.t. $\theta$, yielding $\nabla_\theta J(\pi_\theta)$ when setting $\vartheta = \theta$ after differentiation, i.e.
\begin{equation}
\nabla_\theta J_\mathrm{eff}(\pi_\theta, \pi_\vartheta)\big|_{\vartheta = \theta} = \nabla_\theta J(\pi_\theta)
\end{equation}
(in practice we can simply detach $S_n$ and $A_n$ from the computational graph; for the analog reasoning in the state-space perspective see \Cref{app: computational details}). Further, we note that the state-space-based formula \eqref{eq: policy gradient state-space perspective} in fact also relies on trajectories due to the definition of the $Q$-value function in \eqref{eq: def Q-value function} -- it has already been stated in \citet{bojun2020steady} in a discrete space setting. To the best of our knowledge, the state-space formula in a continuous state setting is novel.

\begin{remark}[Interpretation as effective learning rate]
\label{rem: interpretation as effective learning rate}
As for the expected return in \Cref{prop: costs state-space perspective}, we note that the expected stopping time appears as a scaling factor in the state-space-based policy gradient formula \eqref{eq: policy gradient state-space perspective}. While one could be tempted to argue that this acts only as a constant multiplicative factor that can be absorbed into the learning rate in gradient based optimization (as, e.g., argued in \citet{sutton2018reinforcement}), we note that $\E_\pi[N + 1]$ depends on the current policy and can therefore substantially vary during the course of optimization. Conversely, omitting the factor leads to a different \textit{effective learning rate} in stochastic gradient ascent. To be precise, let $\{\gamma^{(k)}\}_k$ be a sequence of learning rates, then we may update the parameters via 
\begin{equation}
    \theta^{(k+1)} = \theta^{(k)} + \gamma^{(k)} \nabla_\theta J(\pi_{\theta^{(k)}}).
\end{equation}
If we instead update 
\begin{subequations}
\begin{align}
\theta^{(k+1)} &= \theta^{(k)} + \gamma^{(k)}   \nabla_{\theta} \widetilde{J}(\pi_{\theta^{(k)}}) \\
 &= \theta^{(k)} + \widetilde{\gamma}^{(k)}   \nabla_\theta J(\pi_{\theta^{(k)}}),
\end{align}
\end{subequations}
where 
\begin{equation}
\label{eq: scaled state-space gradient}
\nabla_\theta \widetilde{J} := \nabla_\theta J / \E_{\pi_{\theta^{(k)}}}[N + 1]
\end{equation}
is the (wrong) gradient with omitted scaling factor, then $\widetilde{\gamma}^{(k)} = \gamma^{(k)} / \E_{\pi_{\theta^{(k)}}}[N + 1]$ is the effective learning rate relating to gradient ascent relying on the correct gradient. We note that this learning rate may substantially vary with changing expected runtimes in the course of the optimization and refer to \Cref{sec: numerical experiments} for experiments investigating its effect on the optimization performance. Importantly, we note that previous policy gradient theorems have either stated the version without expected stopping time, i.e. have considered \eqref{eq: scaled state-space gradient} for finite time scenarios, or only dealt with the infinite time horizon case \cite{sutton1999policy}. Both cases lead to incorrect formulas when being applied to random time horizon problems. 
\end{remark}


For the trajectory-based policy gradient estimator \eqref{eq: policy gradient trajectory perspective} we can additionally derive the following alternative versions, which might be advantageous from a computational perspective and are proved in \Cref{app: proofs}.

\begin{corollary}[Alternative trajectory-based versions of the policy gradient]
\label{cor: alternative trajectory policy gradient}
    For the gradient of the expected return \eqref{eq: expected return} it holds
    \begin{align}
    \label{eq: alternative trajectory policy gradient m=0}
&\nabla_\theta J(\pi) = \E_\pi \left[ \sum_{n=0}^N \nabla_\theta \log \pi_\theta(S_n, A_n) \sum_{m=0}^N r(S_m,A_m) \right] \\
    \label{eq: alternative trajectory policy gradient m=n}
 & \quad = \E_\pi \left[ \sum_{n=0}^N \nabla_\theta \log \pi_\theta(S_n, A_n) \sum_{m=n}^N r(S_m,A_m) \right] \\
    \label{eq: alternative trajectory policy gradient baseline} 
  & \quad = \E_\pi \left[ \sum_{n=0}^N \nabla_\theta \log \pi_\theta(S_n, A_n) \left(Q^\pi(S_n, A_n) - b(S_n) \right) \right],
\end{align} 
where $b : \mathcal{S} \to \R$ is an arbitrary function (sometimes called \emph{baseline}).
\end{corollary}


For deterministic policies, the policy gradient takes a slightly different form, which we state in the following. We note that for infinite time horizons it has already been derived in \citet{silver2014deterministic}, however, we have not seen a formula for random time horizons before.

\begin{proposition}[Policy gradient for deterministic policies]
\label{prop: deterministic policy gradient}
For the gradient of the expected return \eqref{eq: expected return} it holds\footnote{We denote with $\nabla_\theta \mu_\theta$ the Jacobian matrix of $\mu_\theta$.}
\begin{align}
\label{eq: deterministic policy gradient trajectory perspective}
&\nabla_\theta J(\mu_\theta) = \E_{\mu} \left[ \sum_{n=0}^N \nabla_\theta \mu_\theta(S_n)^\top \nabla_{a} Q^{\mu_\theta}(S_n,a) \Big|_{a=\mu_{\theta}(S_n)} \right] \\
\label{eq: deterministic policy gradient state-space perspective}
&\quad =\E_\mu[N + 1] \, \E_{s \sim \rho^{\mu}} \left[ \nabla_\theta \mu_\theta(s)^\top \nabla_{a} Q^{\mu_\theta}(s,a) \Big|_{a=\mu_{\theta}(s)} \right],
\end{align} where $\rho^\mu$ is the state-space density defined in \eqref{eq: definition state-space density} (with deterministic instead of stochastic policy).
\end{proposition}

For the deterministic policy gradient we can further derive a version in the model-based setting, where we assume the knowledge of the transition density $p$ defined in \eqref{eq: state-action transition density}. 

\begin{corollary}[Model-based policy gradient for deterministic policies]
\label{cor: model-based deterministic policy gradient}
For the gradient of the expected return \eqref{eq: expected return} it holds
\begin{align}
\begin{split}
\label{eq: model-based deterministic policy gradient trajectory}
&\nabla_\theta J(\mu_\theta)
= \E_{\mu} \Bigg[\sum_{n=0}^{N} \nabla_\theta \mu_\theta(S_n)^\top  \Big( \nabla_a r(S_n, a)  \\
&\quad+  V^\mu(S_{n+1}) \nabla_a \log p(S_{n+1}, S_n, a) \Big)\Big|_{a=\mu_\theta(S_n)} \Bigg]
\end{split} \\
\begin{split}
\label{eq: model-based deterministic policy gradient state}
&= \E_\mu[N + 1] \, \E_{\substack{s \sim \rho^\mu, \\ s' \sim p^\mu(\cdot, s)}} \Bigg[ \nabla_\theta \mu_\theta(s)^\top  \Big( \nabla_a r(s, a)  \\ 
&\quad \qquad\qquad + V^\mu(s') \nabla_a \log p(s', s, a)  \Big) \Big|_{a=\mu_\theta(s)} \Bigg].
\end{split}
\end{align}
\end{corollary}

We note that the alternative versions stated in \eqref{eq: alternative trajectory policy gradient m=0} and \eqref{eq: alternative trajectory policy gradient m=n} for stochastic policies hold for deterministic policies only in the model-based and not in the model-free case, see \Cref{cor: alternative trajectory dpg} in the appendix.

\begin{remark}[Relation to stochastic optimal control]
The trajectory-based formula \eqref{eq: model-based deterministic policy gradient trajectory} is equivalent to the control cost gradient in stochastic optimal control problems, e.g. stated in \citet{ribera2024improving}, when using certain Gaussian transition densities, see also \Cref{rem: connection to stochastic optimal control} in \Cref{app: proofs}.
We also note that in \citet{quer_connecting_2024} a version of \eqref{eq: model-based deterministic policy gradient trajectory} is heuristically derived after assuming $N$ to be deterministic. To the best of our knowledge, the presented model-based policy gradient formula with random time horizons has not been proved in the discrete time setting yet.
\end{remark}

\section{Numerical experiments}
\label{sec: numerical experiments}

In this section we compare the previously discussed gradient estimators on numerical examples. In particular, we compare trajectory and state-space based formulas and investigate the effect of neglecting the factor $\E_\pi[N + 1]$ in the policy gradient (PG) computations. To be precise, for all our experiments we compute the gradient either with the trajectory-based expression \eqref{eq: policy gradient trajectory perspective} (\textit{trajectory PG}), by the state-space expression \eqref{eq: policy gradient state-space perspective} (\textit{state-space PG}), or by the modified state-space expression \eqref{eq: scaled state-space gradient}, which omits the scaling factor and is therefore, strictly speaking, an incorrect gradient (\textit{state-space PG (biased)}). We refer to \Cref{rem: interpretation as effective learning rate} for interpreting this incorrect formula as an effective learning rate that changes over the course of optimization. Crucially, note that this effective learning rate is highly problem-specific and can not be controlled easily -- this is an obvious downside of the gradient \eqref{eq: scaled state-space gradient}. In order to assure fair comparisons, we first search for the optimal learning rate for each gradient approach in all of our experiments and use standard stochastic gradient ascent for optimization (in order to not have interacting effects with sophisticated optimization methods). We refer to Algorithms \ref{alg: alg1}-\ref{alg: alg4} in \Cref{app: computational details} for further computational details. The code can be found at \url{https://github.com/riberaborrell/rl-random-times}.


\subsection{Modified continuous mountain car problem}
\label{sec: mountain car}


\begin{figure*}[h]
\centering
\includegraphics[width=0.245\textwidth]{img/mountaincar-cont_traj-vs-state-pg_sgd_lr-objective.pdf}
\includegraphics[width=0.245\textwidth]{img/mountaincar-cont_traj-vs-state-pg_sgd_ct-eff-lr.pdf}
\includegraphics[width=0.245\textwidth]{img/mountaincar-cont_traj-vs-state-pg_sgd_grad-it-objective.pdf}
\includegraphics[width=0.245\textwidth]{img/mountaincar-cont_traj-vs-state-pg_sgd_ct-objective.pdf}
\caption{We display the performance of the three different policy gradients (PG) on the mountain car problem described in \Cref{sec: mountain car}. In the left plot, the negative expected return is investigated for different learning rates, the different transparency values indicate different runs. For the performance plots, we always choose the best respective learning rate. The second plot shows the effective learning rates, see \Cref{rem: interpretation as effective learning rate} for an explanation (we always display a moving average above the raw data). We see that for the biased methods, the effective learning rate increases significantly over the course of the optimization. The two plots on the right-hand side display the negative expected return -- once per gradient step and once per computational time (in minutes). We see that the PG formulas incorporating the expected hitting time perform significantly better. For the biased formula, the algorithm stops at some point due to increased trajectory lengths and related memory issues.}
\label{fig: mountain car continuous}
\end{figure*}

The mountain car problem is a classical benchmark in reinforcement learning, where the goal is to reach the top of a mountain by leveraging gravitational energy additional to the car's acceleration \cite{singh1996reinforcement}. We consider the state space $\mathcal{S} = \mathcal{D}_{\mathrm{pos}} \times \mathcal{D}_{\mathrm{vel}} \subset \mathbb{R}^2$, where $\mathcal{D}_{\mathrm{pos}} = [-1.2, 0.6]$, $\mathcal{D}_{\mathrm{vel}} = [-0.07, 0.07]$ and the action space $\A = [-1, 1] \subset \R$. The target set is defined as $\mathcal{T} = [0.45, \infty) \times \mathbb{R}$ and the deterministic dynamics\footnote{We can think of the transition density defined in \eqref{eq: state-action transition density} as a Dirac delta distribution $p(s', s, a) = \delta(s' - h(s, a)).$} is described via the function $h:\S \times \A \to \S$, given by
\begin{align}
\begin{split}
v' &= v + 0.0015 \, a - 0.0025 \cos(3 x), \\  
x' &= x + v', 
\end{split}
\end{align}
where the state variable $s = (x, v)^\top$ has a position and a velocity part. Compared to the typical problem, we consider trajectories that only stop when reaching the target set and make the problem slightly harder as we aim for reaching this goal quickly by integrating the runtime into the reward function,
\begin{equation}
\label{eq: reward function mountain car}
r(s, a) \coloneqq \left\{
\begin{array}{cl}
-1 - 0.1 a^2 & \text{if} \quad s \notin \mathcal{T}, \\
0 & \text{if} \quad s \in \mathcal{T}.
\end{array}
\right.
\end{equation}

We consider a Gaussian stochastic policy whose mean and covariance are given by a feed-forward neural network with $3$ layers and $32$ hidden units per layer. We refer to \Cref{app: mountain car details} for further details. In \Cref{fig: mountain car continuous} we compare the three different policy gradient formulas and see that the ones incorporating the expected stopping time perform significantly better. This can be explained with the effective learning rate of the biased method, which increases significantly during the optimization due to the change of the expected hitting times (from $N \approx 10^4$ to $N \approx 10^2$). We highlight that this behavior is problem-specific and cannot be known a priori.



\subsection{Two-joint robot arm (reacher)}
\label{sec: reacher}

\begin{figure*}[h]
\centering
\includegraphics[width=0.3\textwidth]{img/reacher_traj-vs-state-pg_sgd_lr-objective.pdf}
\includegraphics[width=0.3\textwidth]{img/reacher_traj-vs-state-pg_sgd_ct-eff-lr.pdf}
\includegraphics[width=0.3\textwidth]{img/reacher_traj-vs-state-pg_sgd_ct-objective.pdf}
\caption{We display the performance of the three different policy gradients (PG) on the reacher problem described in \Cref{sec: reacher}. As before, the left plot shows the performance w.r.t. to different learning rates, from which we choose the best learning rate for each method. In the second plot we can see that the effective learning rate for the biased state-space PG, which ignores the expected hitting time, is rather small at the beginning and increases over the course of the optimization. This then results in slower convergence to the optimal performance, as can be seen in the plot on the right-hand side.}
\label{fig: episodic reacher}
\end{figure*}

The reacher environment contains a two-joint robot arm whose goal is to move its fingertip close to the target. It operates on continuous state and action spaces $\mathcal{S} = [-1, 1]^4 \times \mathbb{R}^6 \subset \mathbb{R}^{10}$, $\mathcal{A} = [-1, 1]^2 \subset \mathbb{R}^2$. The dynamics is deterministic and the initial state is sampled, see \citet{towers_gymnasium_2024} for details. Since the original reacher environment is posed for infinite time horizons, we define the target set such that trajectories terminate if the positions $s_9, s_{10}$ of the fingertip are near the target and the angular velocities $s_7, s_8$ of the two arms are low enough,
\begin{equation}
\label{eq: reacher target set}
\mathcal{T} = \{ s \in \mathcal{S} : \| (s_7, s_8) \| \leq 2, \| (s_9, s_{10}) \| \leq 0.05 \}.
\end{equation}
We consider the (slightly modified) reward function
\begin{equation}
\label{eq: reward function reacher}
r(s, a) \coloneqq \left\{
\begin{array}{cl}
-1 - \omega_{\mathrm{control}} \|a\|^2 & \text{if} \quad s \notin \mathcal{T}, \\
0 & \text{if} \quad s \in \mathcal{T},
\end{array}
\right.
\end{equation}
where $\omega_{\mathrm{control}} = 0.1$, again including a penalty for long runtimes as to motivate the robot to operate quickly, however, with small actions. We again consider a Gaussian stochastic policy, see \Cref{app: reacher details} for details. In \Cref{fig: episodic reacher} we can see that our PG formulas lead to faster convergence compared to the classical (biased) formula.




\subsection{Importance sampling of hitting times in molecular dynamics}
\label{sec: double well numerics}

\begin{figure*}[h]
\centering
\includegraphics[width=0.3\textwidth]{img/dw20d-asym_traj-vs-state-dpg_sgd_lr-objective.pdf}
\includegraphics[width=0.3\textwidth]{img/dw20d-asym_traj-vs-state-dpg_sgd_ct-eff-lr.pdf}
\includegraphics[width=0.3\textwidth]{img/dw20d-asym_traj-vs-state-dpg_sgd_ct-objective.pdf}
\caption{We consider importance sampling of hitting times in molecular dynamics as described in \Cref{sec: double well numerics} and compare the three different deterministic policy gradients (DPG). We display the performance depending on the learning rate as well as the effective learning rate and the negative expected return over the course of the optimization. We see that biased state-spaced PG converges more slowly compared to our DPG methods.}
\label{fig: dw20d asym det}
\end{figure*}


Finally, we consider a problem that is relevant for the estimation of rare events in the context of molecular dynamics \cite{hartmann2017variational}. The goal is to identify a deterministic policy that allows for the effective simulation of reaching a target set $\mathcal{T} \subset \mathcal{S}$ by shortening trajectories and reducing the variance of corresponding Monte Carlo estimators \cite{hartmann2012efficient}. To this end, we consider
$d$-dimensional state and action spaces $\mathcal{S} = \mathcal{A} = \mathbb{R}^d$, a Gaussian transition density
defined by
\begin{align}
\label{eq: ol transition density}
p(\cdot, s, a) = \mathcal{N}\left(s + \left(a -\nabla U(s)\right)\Delta t, \sigma^2 \Delta t \operatorname{Id} \right),
\end{align}
where $U : \R^d \to \R$ is a so-called potential function that is given by the specific problem and $\Delta t > 0$ is a step size. The reward function is given by
\begin{equation}
\label{eq: reward function is mgf}
r(s, a) = \left\{
\begin{array}{ll}
- \Delta t - \frac{1}{2} \|a\|^2 \Delta t  & \text{if} \quad s \notin \mathcal{T}, \\
0 & \text{if} \quad s \in \mathcal{T} .
\end{array}
\right.
\end{equation}
Intuitively, \eqref{eq: reward function is mgf} can be interpreted as aiming to minimize the runtime $N$, while keeping the actions small. This problem has already been studied with reinforcement learning methods in \citet{quer_connecting_2024}. In our experiments we consider $U(s) = \sum_{i=1}^d \alpha_i(s_i^2 - 1)^2$, where $\alpha_i$ quantifies the amount of metastability in the $i$-th dimension, as well as $\sigma = \sqrt{2}$ and $\Delta t = 10^{-2}$. For the gradient computations we rely on the model-based formulas stated in \Cref{cor: model-based deterministic policy gradient}.

We consider $d=20$ and choose $\alpha_1 = 5$, $\alpha_2 = 2$, $\alpha_i = 0.5$, for $i=\{3, \dots, 20\}$ as well as $\mathcal{T} = \widetilde{\mathcal{T}} \times \mathbb{R}^{d-2}$, where
\begin{equation}
\label{eq: def target set double well 2d}
    \widetilde{\mathcal{T}}  = \{ (s_1, s_2) \in \mathcal{S} : s_1, s_2 > 0, U(s) \leq 0.25 \},
\end{equation}
see also \Cref{fig: trajectory vs. state-space perspective} for an illustration. As described above, we first search for a suitable learning rate for each approach. In the first panel in \Cref{fig: dw20d asym det} we plot the performance dependence on the learning rate. As expected, the trajectory and state-space based gradient computations yield similar results -- they only differ due to statistical effects. The scaled state-space gradient, on the other hand, yields very different results. For each approach, we choose the optimal learning rate. As noted in \Cref{rem: interpretation as effective learning rate}, the scaled state-space approach has an effective learning rate that depends on the current expected stopping time. In the second panel we plot this quantity over the course of the optimization. Since the stopping times get shorter during the optimization, the effective learning rate increases. Consequently, this leads to slower optimization, since especially in the beginning the effective learning rate is comparatively small, see the third panel. In fact, the formulas that incorporate the stopping time (and thus provide the correct gradient) converge roughly twice as fast.


\section{Conclusion and outlook}
\label{sec: conclusion}

In this work, we have suggested a theoretical framework for reinforcement learning problems with random time horizons. In our experiments, we have seen that the related (mostly novel) policy gradient formulas can lead to improved and accelerated performance. Crucially, we can explain those performance improvements with the formulas we derived. We anticipate that our framework will allow to design advanced optimization algorithms that cleverly integrate the random time dependency, potentially combined with off-policy algorithms, leading to further improvements on state-of-the-art problems in the future. In particular, we envision integrations of our corrections for random time horizons into methods such as \textit{trust region policy optimization} \cite{schulman2015trust} and \textit{proximal policy optimization} \cite{schulman2017proximal}. Furthermore, we think that the connection of reinforcement learning to (stochastic) optimal control problems -- studying e.g. continuous-time perspectives \cite{wang2020reinforcement}, path space measures \cite{nusken2021solving} or diffusion models \cite{berner2022optimal} -- will be fruitful both for further theoretical insights and novel numerical algorithms.

\newpage

\section*{Acknowledgments}
The first author would like to thank A. Sikorski for his helpful insights on understanding the infinite horizon setting as a random time problem and previous fruitful discussions. 
This research has been partially funded by Deutsche Forschungsgemeinschaft~(DFG) through grant~CRC~1114 (Project~No.~235221301) and under Germany’s Excellence Strategy MATH+: Berlin~Mathematics Research Center (EXC~2046/1, Project~No.~390685689).

\section*{Impact statement}
The goal of this work is to advance the theoretical understanding of the field of reinforcement learning, leading to improvements in applications as well. While there are potential societal consequences of our work in principle, we do not see any concrete issues and thus believe that we do not specifically need to highlight any.

\bibliography{references}
\bibliographystyle{icml2025}

\newpage
\appendix
\onecolumn

\section{Notation and setting}
\label{app: notation}

As stated in \Cref{sec: reinforcement learning with random time horizons}, we denote with $\mathcal{M} = (\mathcal{S}, \mathcal{A}, r_n, \rho_0, p)$ a time-discrete \textit{Markov decision process}, where $\mathcal{S} \subset \mathbb{R}^{d_s}$ is the state space, $\mathcal{A} \subset \mathbb{R}^{d_a}$ is the action space, $r_n$ is the reward function, $\rho_0$ is the initial probability density of the states (defined in \eqref{eq: initial density}) and $p$ is the time-homogeneous (state-action) transition density (defined in \eqref{eq: state-action transition density}). Further, we consider the stochastic policy $\pi$ or the deterministic policy $\mu$. The dynamics of the Markov decision process is defined via $S_0 \sim \rho_0$ as well as $A_n \sim \pi(S_n, \cdot)$  and $S_{n+1} \sim p(\cdot, S_{n}, A_{n})$ for each $n \in \{0, \dots, N-1\}$, and we call the resulting set $\{S_0, A_0, S_1, A_1, \dots, S_N\}$ a state-action trajectory and $\{S_0, \dots, S_N\}$ a (state) trajectory. 

For the expectation operator we write $\E_\pi$ if the random variables in the expectation depend on the policy $\pi$, however note that the randomness not only depends on $\pi$, but also on the initial density $\rho_0$ and the transition density $p$, i.e. for a function $\varphi : (\S \times A)^N \times \S \to \S$ and random variables $S_0, A_0, \dots, S_N$, and for fixed $N$, we write
\begin{subequations}
\begin{align}
    &\E_{\pi}\left[ \varphi(S_0, A_0, \dots, S_N) \right] = \E_{\rho_0, p, \pi}\left[ \varphi(S_0, A_0,  \dots, S_N) \right] \\
    &\qquad \qquad = \int_{(\S \times \A)^{N} \times \S} \rho_0(s_0) \left(\prod_{n=0}^{N-1}\pi(s_n, a_n) p(s_{n+1}, s_n, a_n) \right) \mathrm d s_0 \mathrm d a_0 \dots \mathrm d s_{N-1} \mathrm d a_{N-1} \mathrm d s_N.
\end{align}
\end{subequations}

In our work, the stopping $N$ is typically considered to be random, so the expectation is also over $N$, i.e.

\begin{subequations}
\begin{align}
    &\E_{\pi}\left[ \varphi(S_0, A_0, \dots, S_N) \right] = \E_{\rho_0, p, \pi,N}\left[ \varphi(S_0, A_0,  \dots, S_N) \right] \\
    &\qquad = \sum_{m=0}^\infty \P(N = m) \int_{(\S \times \A)^{m} \times \S} \rho_0(s_0) \left(\prod_{n=0}^{m-1}\pi(s_n, a_n) p(s_{n+1}, s_n, a_n) \right) \mathrm d s_0 \mathrm d a_0 \dots \mathrm d s_{m-1} \mathrm d a_{m-1} \mathrm d s_m,
\end{align}
\end{subequations}

where $\P(N = m)$ is the probability of $N$ having the value $m$ 
and we use the convention $\prod_{n=0}^{-1} (\cdot)_n = 1$. This probability is typically dependent on the dynamics, and therefore the policy, and is thus unknown. For deterministic policies we write $\E_\mu$ (even though $\mu : \S \to \A$ is a deterministic function). Also, note that for state-space expectations stated e.g. in \eqref{eq: state-space expected return} we write $\E_{\substack{s \sim \rho^\pi \\ a \sim \pi(s, \cdot)}}[\cdot]$ to denote $\E_{s \sim \rho^\pi} [ \E_{a \sim \pi(s, \cdot)}[\cdot]]$. 

Further, we assume that $\pi$ or $\mu$ are parametrized by the parameter vector $\theta \in \R^p$, and we may interchangeably write $\pi = \pi_\theta$ or $\mu = \mu_\theta$ for ease of notation. We write $\nabla_\theta$ to denote the gradient w.r.t. the parameter vector $\theta$ and for any $s \in \S$ we write $\nabla_\theta \mu_\theta(s)$ to denote the Jacobian matrix of $\mu_{\theta}(s)$ w.r.t. $\theta$.





\section{Additional background on reinforcement learning}
\label{app: background on RL}

Let us consider the (state-action) \emph{reward function}\footnote{In principle, the reward function can also depend on the following next state.} $r_n: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ and recall that it provides the signal that is received after being in state $S_n \in \mathcal{S}$ and having taken action $A_n \in \mathcal{A}$ at time $n \in \mathbb{N}$. In our work we consider a time-independent reward function and we denote it by $r$. We define the \emph{return (from step $n$ onward)} by
\begin{equation}
    G_n = \sum_{m=n}^{N} r(S_m, A_m)
\end{equation}
and denote $G_0$ by the \emph{(initial) return}. We define the \emph{expected (initial) return} (sometimes also called objective function) as
\begin{equation}
    J(\pi) = \E_{\pi}\left[\sum_{n=0}^{N} r(S_n, A_n) \right].
\end{equation}
We note that starting at $n = 0$ holds without loss of generality and one could also start at any other time, since the problem is time-autonomous. Further, we can define the \textit{value function} (or \textit{return-to-go}) as the expected return conditioned on starting at state $s \in \mathcal{S}$, 
\begin{equation}
    V^{\pi}(s) := \E_\pi \left[ \sum_{n=0}^N r(S_n, A_n) \Big| S_0 = s \right].
\end{equation}
 We can further condition on applying the action $a \in \mathcal{A}$, and define the so-called \textit{Q-value function} as
\begin{equation}
    Q^{\pi}(s, a) := \E_\pi \left[ \sum_{n=0}^N r(S_n, A_n) \Big| S_0 = s, A_0 = a \right].
\end{equation}
We note that it holds $J(\pi) = \E_{s \sim \rho_0}\left[V^{\pi}(s) \right]$ and $V^{\pi}(s) = \E_{a \sim \pi (s, \cdot)}\left[Q^{\pi}(s, a) \right]$. For all expressions we have the Bellman equation\footnote{Note that for the infinite time horizon case the value function is scaled by the discount factor $\gamma$.}, e.g.
\begin{subequations}
\begin{align}
    J(\pi) &= \E_\pi\left[ r(S_0, A_0) + \sum_{n=1}^N r(S_n, A_n) \right] \\
    &= \E_\pi\left[ r(S_0, A_0) + \E_\pi\left[\sum_{n=1}^N r(S_n, A_n) \Big| S_1 \right]\right]\\
    &= \E_\pi\left[ r(S_0, A_0) + V^{\pi}(S_1) \right].
\end{align}
\end{subequations}


Analogously, it holds for all $s \in \mathcal{T}^c$
\begin{subequations}
\label{eq: Bellman equation V stochastic policy}
\begin{align}
V^{\pi}(s) &= \E_\pi\left[ r(S_0, A_0) + V^{\pi}(S_1) | S_0 = s \right], \\
&= \int_\A \pi(s, a) \left( r(s, a) + \int_\mathcal{S} p(s', s, a) V^{\pi}(s') \mathrm{d}s' \right) \mathrm{d}a, 
\end{align}
\end{subequations}
and 
\begin{subequations}
\label{eq: Bellman equation Q stochastic policy}
\begin{align}
Q^{\pi}(s, a) &=  r(s, a) + \E_\pi\left[Q^{\pi}(S_1, A_1) | S_0 = s, A_0 = a \right] \\
&= r(s, a) + \int_\mathcal{S} p(s', s, a) \int_\A \pi(s', a') Q^{\pi}(s', a') \mathrm{d}a' \mathrm{d}s' . 
\end{align}
\end{subequations}



For deterministic policies the above equations reduce to 
\begin{subequations}
\label{eq: Bellman equation V deterministic policy}
\begin{align}
V^{\mu}(s) &= r(s, \mu(s)) + \E_\mu\left[V^{\mu}(S_1) | S_0 = s \right] \\
&= r(s, \mu(s)) + \int_\mathcal{S} p(s', s, \mu(s)) V^{\mu}(s') \mathrm{d}s', 
\end{align}
\end{subequations}
and
\begin{subequations}
\label{eq: Bellman equation Q deterministic policy}
\begin{align}
Q^{\mu}(s, a) &= r(s, a)  + \E_\mu\left[ Q^{\mu}(S_1, \mu(S_1)) | S_0 = s, A_0 = a \right] \\
&= r(s, a) +  \int_\mathcal{S} p(s', s, a) Q^{\mu}(s', \mu(s')) \mathrm{d}s'. \\
&= r(s, a) +  \int_\mathcal{S} p(s', s, a) V^{\mu}(s') \mathrm{d}s'.
\end{align}
\end{subequations}
Further, note that in control theory the \emph{value function} denotes the \emph{optimal cost-to-go}, i.e. the optimal value of the cost functional w.r.t. all possible controls. In contrast, in reinforcement learning the term \textit{value function} is used for any arbitrary policy. If the policy is optimal it is called $\emph{optimal value function}$.

In practice, one often chooses $\pi = \mathcal{N}(\mu, \sigma^2 \operatorname{Id})$, for which $\mu$ and $\sigma$ are learnable functions. We note that with $\sigma$ approaching the zero function $x \mapsto 0$, i.e. with reducing the stochasticity of the stochastic policy, $\pi$ approaches a Dirac distribution and therefore a deterministic policy, see also \Cref{rem: deterministic as stochastic policy} in the main text and Theorem 2 in \citet{silver2014deterministic}.




\section{Proofs and additional statements}
\label{app: proofs}

\begin{proof}[Proof of \Cref{lem: discounted infinite as random time problem}]

First note the following re-ordering of sums, containing the sequences $\{a_i\}_{i}$ and $\{b_j\}_j$. It holds that
\begin{equation}
\label{eq: sum re-ordering}
\sum_{i=0}^\infty a_i \sum_{j=0}^{i} b_j = \lim_{l \to \infty} \sum_{i=0}^l a_i \sum_{j=0}^{i} b_j = \lim_{l \to \infty} \sum_{j=0}^{l} b_j \sum_{i=j}^l a_i = \sum_{j=0}^\infty b_j \sum_{i=j}^\infty a_i .
\end{equation}

Assuming that $N_\gamma \sim \operatorname{Geom}(1- \gamma)$ with $\gamma \in (0, 1)$, we have that $\P(N_\gamma = m) = \gamma^m(1-\gamma)$ and can compute
\begin{subequations}
\begin{align}
    \E_\pi \left[ \sum_{n=0}^{N_\gamma} r(S_n, A_n) \right] &= \E_{N_\gamma}\left[\E_\pi \left[ \sum_{n=0}^{N_\gamma} r(S_n, A_n) \right] \right] \\
    \label{eq: independent random time}
    &= \sum_{m=0}^\infty \P(N_\gamma = m) \E_\pi \left[ \sum_{n=0}^m r(S_n, A_n) \right]  \\
    \label{eq: sum re-ordering execution}
    &= \E_\pi \left[ \sum_{n=0}^\infty  r(S_n, A_n) (1-\gamma) \sum_{m=n}^\infty \gamma^m \right]  \\
    &= \E_\pi \left[ \sum_{n=0}^\infty \gamma^n r(S_n, A_n) \right],
\end{align}
\end{subequations}
where we used the tower property in the first line, the fact that $N_\gamma$ does not depend on the policy in \eqref{eq: independent random time} and identity \eqref{eq: sum re-ordering} in \eqref{eq: sum re-ordering execution}. We can further check that
\begin{equation}
\P(N_\gamma = \infty) = \lim_{m \to \infty} \P(N_\gamma = m) = \lim_{m \to \infty} \gamma^m(1-\gamma) = 0.
\end{equation}

\end{proof}

\begin{proof}[Proof of \Cref{lem: eta region counting}] Recalling the definitions \eqref{def: n-step function} and \eqref{eq: definition state-space density}, 
\begin{equation}
    \eta^\pi := \sum_{n = 0}^\infty \rho^\pi_n, \qquad \int_\Lambda \rho^\pi_n(s) \mathrm ds = \P_\pi(S_n \in \Lambda),
\end{equation}
we compute
\begin{align}
\int_\Lambda \eta^\pi(s) \mathrm{d}s
&= \sum_{n=0}^\infty  \int_\Lambda \rho^\pi_n(s) \mathrm{d}s = \sum_{n=0}^\infty \mathbb{P}_\pi(S_n \in \Lambda) = \sum_{n=0}^\infty \mathbb{E}_\pi \left[\mathbbm{1}_{\Lambda}(S_n) \right] = \mathbb{E}_\pi \left[ \sum_{n=0}^\infty \mathbbm{1}_{\Lambda}(S_n) \right] = \mathbb{E}_\pi \left[ \sum_{n=0}^N \mathbbm{1}_{\Lambda}(S_n) \right].
\end{align}

Further, we note that choosing $\Lambda = \mathcal{S}$ yields

\begin{equation}
Z^\pi = \int_\mathcal{S} \eta^\pi(s) \mathrm{d}s = \mathbb{E}_\pi \left[ N \right] + 1.
\end{equation}
\end{proof}

\begin{proof}[Proof of \Cref{prop: costs state-space perspective}]



Let us first define the $n$-step transition function $p_n^\pi(s', s)$ via

\begin{equation}
\label{eq: def n-step transition probability}
    \int_{\Lambda} p_n^\pi(s', s) \mathrm d s' = \P_\pi \left(S_n \in \Lambda | S_0 = s \right). 
\end{equation}
Note that we have 
\begin{equation}
\label{eq: n-step transition probability}
    \int_{\S} p_n^\pi(s', s) \mathrm d s' = \P_\pi\left(S_n \in \S | S_0 = s \right)= \P_\pi\left(S_n \in \S, n \le N | S_0 = s \right) = \P_\pi\left(n \le N | S_0 = s \right),
\end{equation}
so it is only a density conditioned on the fact that $n \le N$. We can now compute
\begin{subequations}
\begin{align}
    \E_\pi\left[\sum_{n=0}^N r(S_n, A_n) \right] &= \E_\pi \left[\sum_{n=0}^\infty \mathbbm{1}_{[n, \infty)}(N) r(S_n, A_n) \right] \\ 
    \label{eq: costs finite to infinite sum}
&=  \int_{\S} \int_\S \int_\A \sum_{n=0}^\infty r(s, a)\pi(s, a) p_n^\pi(s, \bar{s}) \rho_0(\bar{s}) \, \mathrm d \bar{s} \, \mathrm d s \, \mathrm d a \\
    &= \int_{\S} \int_\A  r(s, a)\pi(s, a) \sum_{n=0}^\infty \int_\S  p_n^\pi(s, \bar{s}) \rho_0(\bar{s}) \, \mathrm d \bar{s} \, \mathrm d s \, \mathrm d a  \\
    &= \int_{\S} \int_\A  r(s, a)\pi(s, a) \sum_{n=0}^\infty  \rho_n^\pi(s) \, \mathrm d s \, \mathrm d a \\
    &= \int_{\S} \int_\A  r(s, a)\pi(s, a) \eta^\pi(s) \, \mathrm d s \, \mathrm d a  \\
    &= \E_\pi[N + 1] \E_{\substack{s \sim \rho^\pi \\ a \sim \pi(s, \cdot)}} \left[ r(s, a) \right],
\end{align}
\end{subequations}

where in the last line we used
\begin{equation}
    \eta^\pi = \rho^\pi  \int_{\mathcal{S}} \eta^\pi(s) \mathrm ds =  \rho^\pi \, \E_{\pi}[N + 1]
\end{equation}
via definition \eqref{eq: definition state-space density} and \Cref{lem: eta region counting}. Note that the sum in \eqref{eq: costs finite to infinite sum} switches from a sum until the (potentially random) time $N$ to an infinite sum since the transition probability $p_n^\pi$ defined in \eqref{eq: n-step transition probability} encodes not only the states, but also a potential stopping of corresponding trajectories and thus encodes the potential randomness of $N$.
\end{proof}

\begin{remark}[Alternative derivation of state-space expected return]
\label{rem: alternative derivation of state-space expected return}
    The state-space based formula for the expected return stated in \Cref{prop: costs state-space perspective} can also be derived as follows. The trajectory-based formula
\begin{equation}
    J(\pi) = \E_{\pi}\left[\sum_{n=0}^{N} r_n(S_n, A_n) \right],
\end{equation}
stated in \eqref{eq: expected return}, can be approximated by $K$ trajectories via
\begin{equation}
\label{eq: sample estimator trajectory return}
    \frac{1}{K}\sum_{k=1}^K \sum_{n=0}^{N^{(k)}} r(S_n^{(k)}, A_n^{(k)}),
\end{equation}
noting that each trajectory has a different length, namely $N^{(k)} + 1$ steps in the $k$-th trajectory. We note that we can consider the particles
\begin{equation}
    B := \bigcup_{k=1}^K \bigcup_{n=0}^{N^{(k)}} \left\{ S_n^{(k)}, A_n^{(k)} \right\} 
\end{equation}
as an unbiased sample from the action-state-space density $\rho^\pi(s) \pi(s, a)$, defined in \eqref{eq: definition state-space density}. Let us merge the two indices of the elements of $B$ and write 
\begin{equation}
    B = \bigcup_{m=1}^M  \left\{ \widetilde{S}^{(m)}, \widetilde{A}^{(m)} \right\},
\end{equation}
noting that $M = \sum_{k=1}^K (N^{(k)} + 1)$. We can now  write the sample estimator of the trajectory-based expected return \eqref{eq: sample estimator trajectory return} as
\begin{equation}
\label{eq: sample estimator state-space return}
    \frac{1}{K}\sum_{k=1}^K \sum_{n=0}^{N^{(k)}} r(S_n^{(k)}, A_n^{(k)}) = \frac{1}{K} \sum_{m=1}^M r(\widetilde{S}^{(m)}, \widetilde{A}^{(m)}) = \frac{\frac{M}{K}}{M} \sum_{m=1}^M r(\widetilde{S}^{(m)}, \widetilde{A}^{(m)}).
\end{equation}
Now, noting that $M / K$ converges to $\E_\pi[N + 1]$ for $K \to \infty$ by the law of large numbers, we conclude that \eqref{eq: sample estimator state-space return} converges to the state-space expected return 
\begin{equation}
    J(\pi) = \E_\pi[N + 1] \, \E_{\substack{s \sim \rho^\pi \\ a \sim \pi(s, \cdot)}} \left[ r(s, a) \right],
\end{equation}
as stated in \eqref{eq: state-space expected return}.
\end{remark}



\begin{lemma}[Unrolling for stochastic policies]
\label{lem: unrolling lemma}
For any $l \geq 1$ it holds that
\begin{equation}
\label{eq: unrolling lemma}
\nabla_\theta V^\pi(s)
= \sum_{n=0}^{l-1} \int_\mathcal{S} p_n^\pi(s', s) \int_{\mathcal{A}} \nabla_\theta \pi_\theta(s', a) Q^\pi(s', a) \mathrm{d}a \, \mathrm{d}s' + \int_\mathcal{S} p_l^\pi(s', s) \nabla_\theta V^\pi(s') \mathrm{d}s'.
\end{equation}
\end{lemma}

\begin{proof}

For the sake of notational convenience, let us define the shorthand notation
\begin{equation}
h_\theta(s, a) := \nabla_\theta \pi_\theta(s, a) Q^\pi(s, a). 
\end{equation}
We prove \Cref{lem: unrolling lemma} via induction over $l$. Let us start by the initial case $l=1$. Recalling that $V^{\pi}(s) = \E_{a \sim \pi (s, \cdot)}\left[Q^{\pi}(s, a) \right]$, we can compute
\begin{subequations}
\label{eq: grad value function}
\begin{align}
\nabla_\theta V^{\pi}(s)
&= \int_\mathcal{A} \Bigl( \nabla_\theta \pi_\theta(s, a) Q^\pi(s, a) +  \pi_\theta(s, a) \nabla_\theta Q^\pi(s, a) \Bigr) \mathrm{d}a \\
&= \int_\mathcal{A} h_\theta(s, a) \mathrm{d} a+ \int_\mathcal{A} \pi_\theta(s, a) \nabla_\theta \biggl( r(s, a) +  \int_\mathcal{S} p(s', s, a) V^\pi(s') \mathrm{d}s' \biggr) \mathrm{d} a \\
&= \int_\mathcal{A} h_\theta(s, a) \mathrm{d} a + \int_\mathcal{A} \pi_\theta(s, a)  \int_\mathcal{S} p(s', s, a) \nabla_\theta V^\pi(s') \mathrm{d}s'\mathrm{d}a \\
&= \int_\mathcal{A} h_\theta(s, a) \mathrm{d} a + \int_\mathcal{S} {p}_1^\pi(s', s) \nabla_\theta V^\pi(s') \mathrm{d} s',
\end{align}
\end{subequations}
where we use the Bellman equation stated in \eqref{eq: Bellman equation Q stochastic policy} and where $p_1^\pi$ is defined in \eqref{eq: def n-step transition probability}. The expression \eqref{eq: grad value function} coincides with \eqref{eq: unrolling lemma} for $l=1$. Let us now assume that \eqref{eq: unrolling lemma} holds for $l-1$. We can compute
\begin{subequations}
\begin{align}
\nabla_\theta V^\pi(s)
&= \sum_{n=0}^{l-2} \int_\mathcal{S} p_n^\pi(s', s) \int_\mathcal{A} h_\theta(s', a) \mathrm{d}a \, \mathrm{d}s' + \int_\mathcal{S} p_{l-1}^\pi(s', s) \nabla_\theta V^\pi(s') \mathrm{d}s' \label{eq: foo} \\ 
&= \sum_{n=0}^{l-2} \int_\mathcal{S} p_n^\pi(s', s) \int_\mathcal{A} h_\theta(s', a) \mathrm{d}a \, \mathrm{d}s' + \int_\mathcal{S} p_{l-1}^\pi(s', s) \Biggl( \int_\mathcal{A} h_\theta(s', a) \mathrm{d} a + \int_\mathcal{S} p_1^\pi(s'', s') \nabla_\theta V^\pi(s'') \mathrm{d} s'' \Biggr) \mathrm{d}s' \\
&= \sum_{n=0}^{l-1} \int_\mathcal{S} p_n^\pi(s', s) \int_\mathcal{A} h_\theta(s', a) \mathrm{d}a \, \mathrm{d}s' + \int_\mathcal{S} \int_\mathcal{S} p_{l-1}^\pi(s', s) p_1^\pi(s'', s') \nabla_\theta V^\pi(s'') \mathrm{d}s' \mathrm{d}s'' \\
\label{eq: unrolling lemma induction}
&= \sum_{n=0}^{l-1} \int_\mathcal{S} p_n^\pi(s', s) \int_\mathcal{A} h_\theta(s', a) \mathrm{d}a \, \mathrm{d}s'  + \int_\mathcal{S} p_{l}^\pi(s'', s) \nabla_\theta V^\pi(s'') \mathrm{d}s'',
\end{align}
\end{subequations}
where we have used \eqref{eq: grad value function} in the second line. Since \eqref{eq: unrolling lemma induction} conincides with \eqref{eq: unrolling lemma} we have proved the statement by induction.
\end{proof}


\begin{proof}[Proof of \Cref{prop: stochastic policy gradient}]

We assume that the gradient of the value function with respect to the policy parameters is bounded, i.e., for any $\theta \in \mathbb{R}^p$ there exists an $L_\theta > 0$ such that
\begin{equation}
\sup\limits_{s \in \mathcal{S}} || \nabla_\theta V^\pi(s)|| \leq L_\theta. 
\end{equation}
Then it holds
\begin{equation}
\lim_{l \rightarrow \infty} \int_\mathcal{S} p_l^\pi(s', s) || \nabla_\theta V^\pi(s')|| \mathrm{d}s'
\le \lim_{l \rightarrow \infty} L_\theta \, \mathbb{P}_\pi(l \le N \mid S_0 = s) = 0,
\end{equation}
by assumption of the (potentially random, but almost surely finite) time $N$. In the limit $l \to \infty$, the expression from \Cref{lem: unrolling lemma} therefore turns into
\begin{equation}
\nabla_\theta V^\pi(s)
= \sum_{n=0}^\infty \int_\mathcal{S} p_n^\pi(s', s) \int_{\mathcal{A}} \nabla_\theta \pi_\theta(s', a) Q^\pi(s', a) \mathrm{d}a \, \mathrm{d}s'.
\end{equation}

Further, we can compute
\begin{subequations}
\label{eq: stochastic policy gradient computation}
\begin{align}
\nabla_\theta J(\pi_\theta)
&= \int_\mathcal{S} \rho_0(s_0) \nabla_\theta V^\pi(s) \mathrm{d} s_0 = \int_\mathcal{S} \rho_0(s_0)  \sum_{n=0}^\infty \int_\mathcal{S} p_n^\pi(s, s_0) \int_{\mathcal{A}} \nabla_\theta \pi_\theta(s, a) Q^\pi(s, a) \mathrm{d}a \, \mathrm{d}s \, \mathrm{d} s_0 \\
&= \int_\mathcal{S} \eta^\pi(s) \int_\mathcal{A}  \nabla_\theta \pi_\theta(s, a) Q^\pi(s, a) \mathrm{d}s \, \mathrm{d}a \\
&= \E[N + 1] \, \int_\mathcal{S} \rho^\pi(s) \int_\mathcal{A} \pi_\theta(s, a) \nabla_\theta \log \pi_\theta(s, a) Q^\pi(s, a) \mathrm{d}s \, \mathrm{d}a  \\
& =\E[N + 1]  \, \mathbb{E}_{\substack{s \sim \rho^\pi \\ a \sim \pi_\theta(s, \cdot)}} \left[ \nabla_\theta \log \pi_\theta(s, a) Q^\pi(s, a) \right], 
\end{align}
\end{subequations}
where we have again used
\begin{equation}
    \eta^\pi = \rho^\pi  \int_{\mathcal{S}} \eta^\pi(s) \mathrm ds =  \rho^\pi \, \E_{\pi}[N + 1]
\end{equation}
via definition \eqref{eq: definition state-space density} and \Cref{lem: eta region counting}.
Finally, by choosing\footnote{Note that \Cref{prop: costs state-space perspective} holds for any arbitrary function $r\colon \S \times \A \rightarrow \mathbb{R}$.} $r(s, a) = \nabla_\theta \log \pi_\theta(s, a) Q^\pi(s, a)$ in \Cref{prop: costs state-space perspective} and using identity \eqref{eq: stochastic policy gradient computation}, we readily get
\begin{equation}
    \nabla_\theta J(\pi_\theta) = \E_\pi \left[ \sum_{n=0}^N \nabla_\theta \log \pi_\theta(S_n, A_n) Q^\pi(S_n, A_n) \right].
\end{equation}
\end{proof}

\begin{proof}[Proof of \Cref{cor: alternative trajectory policy gradient}]
We first note that for an arbitrary function $\varphi :(\S\times \A)^{n+1} \to \R$ it holds
\begin{subequations}
\begin{align}
    \E_\pi\left[\varphi(S_0, A_0, \dots, A_n, S_n) Q^\pi(S_n, A_n) \right] &= \E_\pi\left[\varphi(S_0, A_0, \dots, A_n, S_n) \E_\pi\left[ \sum_{m=n}^N r(S_m, A_m) \Bigg| S_n, A_n  \right] \right] \\ 
    &= \E_\pi\left[\E_\pi\left[\varphi(S_0, A_0, \dots, A_n, S_n)  \sum_{m=n}^N r(S_m, A_m) \right] \right] \\
    &= \E_\pi\left[\varphi(S_0, A_0, \dots, A_n, S_n)  \sum_{m=n}^N r(S_m, A_m) \right],
\end{align}
\end{subequations}
where we used the tower property, the fact that $\varphi(S_0, A_0, \dots, S_n, A_n)$ is $\mathcal{F}_n$-measurable and the Markov property. Equality \eqref{eq: alternative trajectory policy gradient m=n} now follows from choosing $\varphi(S_0, A_0 \dots, S_n, A_n) = \nabla_\theta \log \pi (S_n, A_n)$ and summing over $n$. For this choice we can even show that
\begin{equation}
\E_\pi\left[\nabla_\theta \log \pi (S_n, A_n) Q^\pi(S_n, A_n) \right] = \E_\pi\left[\nabla_\theta \log \pi (S_n, A_n)  \sum_{m=0}^N r(S_m, A_m) \right],
\end{equation}
which implies  \eqref{eq: alternative trajectory policy gradient m=0}. This can be seen by defining the $n$-step transition function $p_n^\pi(s', s, a)$, in analogy to \eqref{eq: def n-step transition probability}, via
\begin{equation}
    \int_{\Lambda} p_n^\pi(s', s, a) \mathrm d s' = \P_\pi\left(S_n \in \Lambda | S_0 = s, A_0 = a \right),
\end{equation}
and noting that for $m < n$ it holds
\begin{subequations}
\begin{align}
    &\E \left[\nabla_\theta \log \pi(S_n, A_n) r(S_m, A_m) \right]\\
    &\qquad =  \int_\S \int_\A  \int_\S \int_\A \rho_m (s_m) \pi(s_m, a_m) r(s_m, a_m) p_{n-m}^\pi(s_n, s_m, a_m) \nabla_\theta \log \pi (s_n, a_n) \pi(s_n, a_n) \mathrm d s_m \mathrm d a_m \mathrm d s_n \mathrm d a_n \\
    &\qquad =  \int_\S \int_\A  \int_\S \rho_m (s_m) \pi(s_m, a_m) r(s_m, a_m) p_{n-m}^\pi(s_n, s_m, a_m) \left( \int_\A  \nabla_\theta \log \pi (s_n, a_n) \pi(s_n, a_n)  \mathrm d a_n \right) \mathrm d s_m \mathrm d a_m \mathrm d s_n \\
    &\qquad  = 0, \nonumber
\end{align}
\end{subequations}
since
\begin{equation}
    \int_\A  \nabla_\theta \log \pi (s_n, a_n) \pi(s_n, a_n)  \mathrm d a_n = \nabla_\theta \int_\A  \pi (s_n, a_n) \mathrm d a_n  = 0.
\end{equation}

We can readily prove \eqref{eq: alternative trajectory policy gradient baseline} in the state-space perspective by noting that
\begin{equation}
\nabla_\theta J(\pi_\theta)
= \int_\mathcal{S} \eta^\pi(s) \int_\mathcal{A}  \nabla_\theta \pi_\theta(s, a) Q^\pi(s, a) \mathrm{d}s \, \mathrm{d}a = \int_\mathcal{S} \eta^\pi(s) \int_\mathcal{A}  \nabla_\theta \pi_\theta(s, a) (Q^\pi(s, a) - b(s)) \mathrm{d}s \, \mathrm{d}a
\end{equation}
since 
\begin{equation}
\int_\mathcal{A} b(s) \nabla_\theta \pi_\theta(s, a) \mathrm{d} a = b(s) \nabla_\theta \int_\mathcal{A} \pi_\theta(s, a) \mathrm{d} a = 0 .
\end{equation}
Then, by choosing $r(s, a) = \nabla_\theta \log \pi_\theta(s, a) (Q^\pi(s, a) - V^\pi(s))$ in \Cref{prop: costs state-space perspective}, we arrive at \eqref{eq: alternative trajectory policy gradient baseline} in the trajectory-based perspective.
\end{proof}


\begin{lemma}[Unrolling for deterministic policies]
\label{lem: unrolling dpg lemma}
For any $l \geq 1$ it holds that
\begin{equation}
\label{eq: unrolling lemma dpg}
\nabla_\theta V^\mu(s)
= \sum_{n=0}^{l-1} \int_\mathcal{S} {p}_n^\mu(s', s) \nabla_\theta \mu_\theta(s')^\top \nabla_a Q^\mu(s', a) \Big|_{a=\mu_\theta(s')} \mathrm{d}s' + \int_\mathcal{S} {p}_l^\mu(s', s) \nabla_\theta V^\mu(s') \mathrm{d}s'.
\end{equation}
\end{lemma}
\begin{proof}
The proof of the unrolling lemma for deterministic policies follows the same idea as for stochastic policies stated in \Cref{lem: unrolling lemma}. In analogy to \eqref{eq: def n-step transition probability}, let us define the $n$-step transition function $p_n^\mu(s', s)$ via
\begin{equation}
\int_{\Lambda} p_n^\mu(s', s) \mathrm d s' = \P_\mu \left(S_n \in \Lambda | S_0 = s \right). 
\end{equation}

As before, let us first consider $l=1$. Recalling the Bellman equation, $V^{\mu}(s) = r(s, \mu(s)) + \E_\mu\left[V^{\mu}(S_1) | S_0 = s \right]$, stated already in \eqref{eq: Bellman equation V deterministic policy}, we can compute
\begin{subequations}
\label{eq: deterministic policy gradient l=1}
\begin{align}
\label{eq: partial value function}
\nabla_\theta V^\mu(s)
&= \nabla_\theta \Bigl( r(s, \mu_\theta(s)) + \int_\mathcal{S} p(s', s, \mu_\theta(s)) V^\mu(s') \mathrm{d}s' \Bigr) \\
\begin{split}
&= \nabla_\theta \mu_\theta(s)^\top \nabla_a r(s, a) \Big |_{a=\mu_\theta(s)} \\ 
& \qquad\qquad +  \int_\mathcal{S} \left( p(s', s, \mu_\theta(s)) \nabla_\theta V^\mu(s') + \nabla_\theta \mu_\theta(s)^\top \nabla_a p(s', s, a) \Big|_{a=\mu_\theta(s)}  V^\mu(s') \right) \mathrm{d}s' 
\end{split} \\
\begin{split}
&= \nabla_\theta \mu_\theta(s)^\top \nabla_a \left( r(s, a) + \int_\mathcal{S} p(s', s, a) V^\mu(s') \mathrm{d}s' \right) \Big |_{a=\mu_\theta(s)} \\
& \qquad\qquad + \int_\mathcal{S} p(s', s, \mu_\theta(s)) \nabla_\theta V^\mu(s') \mathrm{d}s'
\end{split} \\
&= \nabla_\theta \mu_\theta(s)^\top \nabla_a  Q^\mu(s, a) \Big |_{a=\mu_\theta(s)} + \int_\mathcal{S} {p}_1^\mu(s', s) \nabla_\theta V^\mu(s') \mathrm{d}s'.
\end{align}
\end{subequations}

For notational convenience, let us define
\begin{equation}
\varphi_\theta(s) \coloneqq \nabla_\theta \mu_\theta(s)^\top \nabla_a Q^\mu(s, a)\Big|_{a=\mu_\theta(s)}.
\end{equation}

For the induction step, we can then compute
\begin{subequations}
\begin{align}
\nabla_\theta V^\mu(s)
&= \sum_{n=0}^{l-2} \int_\mathcal{S} {p}_n^\mu(s', s) \varphi_\theta(s') \mathrm{d}s' + \int_\mathcal{S} {p}_{l-1}^\mu(s', s) \left(\varphi_\theta(s') + \int_\mathcal{S} {p}_1^\mu(s'', s') \nabla_\theta V^\mu(s'') \mathrm{d}s'' \right) \mathrm{d} s' \\
&= \sum_{n=0}^{l-1} \int_\mathcal{S} {p}_n^\mu(s', s) \varphi_\theta(s') \mathrm{d}s' + \int_\mathcal{S} \int_\mathcal{S} {p}_{l-1}^\mu(s', s) p_1^\mu(s'', s') \nabla_\theta V^\mu(s'') \mathrm{d}s'' \mathrm{d}s' \\
&= \sum_{n=0}^{l-1} \int_\mathcal{S} {p}_n^\mu(s', s) \varphi_\theta(s') \mathrm{d}s' + \int_\mathcal{S} {p}_l^\mu(s', s) \nabla_\theta V^\mu(s') \mathrm{d} s',
\end{align}
\end{subequations}
where we used \eqref{eq: deterministic policy gradient l=1} in the first line.
\end{proof}


\begin{proof}[Proof of \Cref{prop: deterministic policy gradient}]
The proof is analog to the one of \Cref{prop: stochastic policy gradient}. First, we see that in the limit of $l \rightarrow \infty$, the second summand of \eqref{eq: unrolling lemma dpg} vanishes. We can then compute
\begin{subequations}
\begin{align}
\nabla_\theta J(\mu_\theta)
&= \int_\mathcal{S} \rho_0(s_0) \nabla_\theta V^\mu(s_0) \mathrm{d} s_0 \\
&= \int_\mathcal{S} \rho_0(s_0) \left( \sum_{n=0}^\infty \int_\mathcal{S} {p}_n^\mu(s,  s_0) \nabla_\theta \mu_\theta(s)^\top \nabla_a Q^\mu(s, a) \Big|_{a=\mu_\theta(s)} \mathrm{d} s \right) \mathrm{d} s_0 \\
&= \int_\mathcal{S} \eta^\mu(s) \nabla_\theta \mu_\theta(s)^\top \nabla_a Q^\mu(s, a) \Big|_{a=\mu_\theta(s)} \mathrm{d}s \\
\label{eq: deterministic policy gradient computation}
&= \E_{\mu}[N + 1] \, \E_{s \sim \rho^{\mu}} \left[ \nabla_\theta \mu_\theta(s)^\top \nabla_{a} Q^{\mu}(s,a) \Big|_{a=\mu_{\theta}(s)} \right],
\end{align}
\end{subequations}

where we have used
\begin{equation}
    \eta^\mu = \rho^\mu  \int_{\mathcal{S}} \eta^\mu(s) \mathrm ds =  \rho^\mu \, \E_{\mu}[N + 1]
\end{equation}
via definition \eqref{eq: definition state-space density} and \Cref{lem: eta region counting} (however, replacing the stochastic policy $\pi$ with the deterministic policy $\mu$). Finally, by choosing $r(s,a) = \nabla_\theta \mu_\theta(s)^\top \nabla_{a} Q^{\mu}(s,a)_{|a=\mu_{\theta}(s)}$ in the deterministic policy version of \Cref{prop: costs state-space perspective} and using identity \eqref{eq: deterministic policy gradient computation}, we readily get
\begin{equation}
    \E_{\mu} \left[ \sum_{n=0}^N \nabla_\theta \mu_\theta(S_n)^\top \nabla_{a} Q^{\mu_\theta}(S_n,a) \Big|_{a=\mu_{\theta}(S_n)} \right].
\end{equation}

\end{proof}

\begin{proof}[Proof of \Cref{cor: model-based deterministic policy gradient}]
Using the Bellman equation \eqref{eq: Bellman equation Q deterministic policy}, we can compute
\begin{subequations}
\label{eq: grad q-value wrt a}
\begin{align}
\nabla_a Q^\mu(s, a) 
&= \nabla_a r(s, a) +  \int_\mathcal{S} \nabla_a p(s', s, a) V^\mu(s') \mathrm{d}s' \\
&= \nabla_a r(s, a) +  \int_\mathcal{S} \nabla_a \log p(s', s, a) p(s', s, a) V^\mu(s') \mathrm{d}s' \\
&= \nabla_a r(s, a) +  \mathbb{E}_{s' \sim p(\cdot, s, a)} \left[ \nabla_a \log p(s', s, a) V^\mu(s')\right] .
\end{align}
\end{subequations}

Using \Cref{prop: deterministic policy gradient}, we get
\begin{subequations}
\begin{align}
    \nabla_\theta J(\mu_\theta) &= \E_{\mu} \left[ \sum_{n=0}^N \nabla_\theta \mu_\theta(S_n)^\top \nabla_{a} Q^{\mu_\theta}(S_n,a) \Big|_{a=\mu_{\theta}(S_n)} \right] \\
    &= \E_{\mu} \left[ \sum_{n=0}^N \nabla_\theta \mu_\theta(S_n)^\top \left( \nabla_a r(S_n, a) +  \mathbb{E}_{s' \sim p(\cdot, S_n, a)} \left[ \nabla_a \log p(s', S_n, a) V^\mu(s')\right] \right) \Big|_{a=\mu_{\theta}(S_n)}  \right] \\
    &= \E_{\mu} \left[ \sum_{n=0}^N \nabla_\theta \mu_\theta(S_n)^\top \left( \nabla_a r(S_n, a) +  \mathbb{E}_{\mu_\theta} \left[ \nabla_a \log p(S_{n+1}, S_n, a) V^\mu(S_{n+1})\right] \right) \Big|_{a=\mu_{\theta}(S_n)}  \right] \\
    &= \E_{\mu} \left[ \sum_{n=0}^N \nabla_\theta \mu_\theta(S_n)^\top \left( \nabla_a r(S_n, a) +   \nabla_a \log p(S_{n+1}, S_n, a) V^\mu(S_{n+1})\right) \Big|_{a=\mu_{\theta}(S_n)}  \right],
\end{align} 
\end{subequations}
where we used the tower property in the last line. The second equality follows from plugging in \eqref{eq: grad q-value wrt a} into the second equation of \Cref{prop: deterministic policy gradient}.
\end{proof}

\begin{remark}[Connection to stochastic optimal control gradient]
\label{rem: connection to stochastic optimal control}

From \Cref{cor: model-based deterministic policy gradient} we can recover the gradient estimator of stochastic optimal control problems with random stopping times, as for instance stated in Corollary 3.3 in \citet{ribera2024improving}. To this end, we realize that the continuous time SDE
\begin{equation}
    \mathrm d X_s^\mu = \left(b + \sigma \mu \right)(X_s^\mu)\, \mathrm dt + \sigma(X_s^\mu)\, \mathrm d W_s,
\end{equation}
where $W$ is standard Brownian motion, $b : \R^d \to \R^d$ an arbitrary drift and $\sigma : \R^{d} \to \R^{d \times d}$ is the diffusion matrix, can be discretized via the Euler-Maruyama scheme
\begin{equation}
    S_{n+1} = S_n + \left(b + \sigma \mu\right)(S_n)\Delta t + \sigma(S_n)\xi_{n+1} \sqrt{\Delta t},
\end{equation}
with step size $\Delta t > 0$ and Gaussian increment $\xi_{n+1} \sim \mathcal{N}(0, \operatorname{Id})$. Calling the terminal time $T=N\Delta t$, the continuous control costs
\begin{equation}
    J(\mu) = \E\left[\int_0^T \left( f + \frac{1}{2}\|\mu\|^2 \right)(X_s^\mu) \, \mathrm dt + g(X_T) \ \right]
\end{equation}
correspond to $r(s, a) = -\left(f(s) + \frac{1}{2}\| a\|^2\right) \Delta t \, \mathbbm{1}_{\mathcal{T}^c}(s)  - g(s) \, \mathbbm{1}_{\mathcal{T}}(s)$ in the discrete setting \eqref{eq: expected return} and we can thus compute
\begin{align}
\nabla_a r(s, a) &= - a \Delta t \, \mathbbm{1}_{\mathcal{T}^c}(s), \\
\nabla_a \log p(S_{n+1}, S_n, a) &= \sigma^{-1}(S_n)\left(S_{n+1} - \left( S_n + \left(b + \sigma a\right)(S_n)\Delta t \right) \right) = \xi_{n+1} \sqrt{\Delta t}.
\end{align}
We can therefore see that \eqref{eq: model-based deterministic policy gradient trajectory} from \Cref{cor: model-based deterministic policy gradient} corresponds to the time-discretized version of  Corollary 3.3 in \citet{ribera2024improving}. We highlight that a derivation in discrete time including random stopping times has not been rigorously done before (cf. \citet{hartmann2012efficient}, where it has been conjectured that the formula is inexact due to the random time).
\end{remark}

\begin{corollary}[Alternative trajectory-based versions of the model-based policy gradient for deterministic policies]
\label{cor: alternative trajectory dpg}
For the gradient of the expected return \eqref{eq: expected return} it holds
\begin{align}
\label{eq: alternative trajectory dpg m=0}
\nabla_\theta J(\mu_\theta) 
&= \E_{\mu} \Bigg[\sum_{n=0}^{N} \nabla_\theta \mu_\theta(S_n)^\top  \Big( \nabla_a r(S_n, a)
+  \sum_{m=0}^N r(S_m, \mu_\theta(S_m)) \nabla_a \log p(S_{n+1}, S_n, a) \Big)\Big|_{a=\mu_\theta(S_n)} \Bigg] \\ 
\label{eq: alternative trajectory dpg m=n}
&= \E_{\mu} \Bigg[\sum_{n=0}^{N} \nabla_\theta \mu_\theta(S_n)^\top  \Big( \nabla_a r(S_n, a)
+  \sum_{m=n+1}^N r(S_m, \mu_\theta(S_m)) \nabla_a \log p(S_{n+1}, S_n, a) \Big)\Big|_{a=\mu_\theta(S_n)} \Bigg] \\  
\label{eq: alternative trajectory dpg baseline} 
& = \E_{\mu} \left[ \sum_{n=0}^N \nabla_\theta \mu_\theta(S_n)^\top \nabla_{a} \Big(Q^{\mu_\theta}(S_n,a) \Big|_{a=\mu_{\theta}(S_n)} - b(S_n)\Big)\right].
\end{align} 
where $b : \mathcal{S} \to \R$ is an arbitrary function (sometimes called \emph{baseline}).
\end{corollary}

\begin{proof}[Proof of \Cref{cor: alternative trajectory dpg}]
The proof of the alternative versions of the deterministic policy gradient follows the same idea as for stochastic policies stated in \Cref{cor: alternative trajectory policy gradient}. 

We first note that for an arbitrary function $\varphi : \S^{n+2} \to \R$ it holds
\begin{subequations}
\begin{align}
\E_\mu\left[\varphi(S_0, \dots, S_{n+1}) V^\mu(S_{n+1}) \right] 
&= \E_\mu\left[\varphi(S_0, \dots, S_{n+1}) \E_\mu \left[ \sum_{m=n+1}^N r(S_m, \mu(S_m)) \Bigg| S_n, A_n  \right] \right] \\ 
&= \E_\mu \left[\E_\mu \left[\varphi(S_0, \dots, S_{n+1}) \sum_{m=n+1}^N r(S_m, \mu(S_m)) \right] \right] \\
&= \E_\mu \left[\varphi(S_0, \dots, S_{n+1}) \sum_{m=n+1}^N r(S_m, \mu(S_m)) \right],
\end{align}
\end{subequations}
where we used the tower property, the fact that $\varphi(S_0, \dots, S_{n+1})$ is $\mathcal{F}_{n+1}$-measurable and the Markov property. Equality \eqref{eq: alternative trajectory dpg m=n} now follows from choosing $\varphi(S_0, \dots, S_{n+1}) = \nabla_\theta \mu_\theta (S_n)^\top \nabla_a \log p(S_{n+1}, S_n, a) \Big)\Big|_{a=\mu_\theta(S_n)}$.
For this choice we can even show that
\begin{equation}
\begin{split}
& \E_\mu \left[\nabla_\theta \mu_\theta (S_n)^\top V^\pi(S_{n+1}) \nabla_a \log p(S_{n+1}, S_n, a) \big|_{a=\mu_\theta(S_n)} \right] \\
& \qquad = \E_\mu \left[ \nabla_\theta \mu_\theta (S_n)^\top \sum_{m=0}^N r(S_m, \mu_\theta(S_m)) \nabla_a \log p(S_{n+1}, S_n, a) \big|_{a=\mu_\theta(S_n)}\right],
\end{split}
\end{equation}
which implies  \eqref{eq: alternative trajectory dpg m=0}. This can be seen by using the $n$-step transition function $p_n^\mu(s', s, \mu(s))$, which is stated in \Cref{cor: alternative trajectory policy gradient} for stochastic policies, here denoted by $p_n^\mu(s', s)$,
as well as by noting that for $m < n+1$ it holds
\begin{subequations}
\begin{align}
& \E_\mu \left[ \nabla_\theta \mu_\theta (S_n)^\top r(S_m, \mu_\theta(S_m)) \nabla_a \log p(S_{n+1}, S_n, a) \big|_{a=\mu_\theta(S_n)} \right] \\
& =  \int_\S \int_\S \int_\S \rho_m (s_m) r(s_m, \mu(s_m)) p_{n-m}^\mu(s_n, s_m) \nabla_\theta \mu(s_n)^\top p_1^\mu(s_{n+1}, s_n) \nabla_a \log p(s_{n+1}, s_n, a) \big|_{a=\mu(s_n)} \mathrm d s_m \mathrm d s_n \mathrm d s_{n+1} \\
& =  \int_\S \int_\S \rho_m (s_m) r(s_m, \mu(s_m)) p_{n-m}^\mu(s_n, s_m) \nabla_\theta \mu(s_n)^\top \left(\int_\S p_1^\mu(s_{n+1}, s_n) \nabla_a \log p(s_{n+1}, s_n, a) \big|_{a=\mu(s_n)} \mathrm d s_{n+1} \right) \mathrm d s_m \mathrm d s_n \\
&  = 0, \nonumber
\end{align}
\end{subequations}
since for any tuple $(s', s, a) \in \S \times \S \times \A$ it holds that
\begin{equation}
\int_\S  \nabla_a \log p (s', s, a) p(s', s, a)  \mathrm d s' = \int_\S \nabla_a  p(s', s, a) \mathrm d s'  = \nabla_a \int_\S  p(s', s, a) \mathrm d s'  = 0.
\end{equation}

We can readily prove \eqref{eq: alternative trajectory dpg baseline} in the state-space perspective by noting that
\begin{equation}
\nabla_\theta J(\mu_\theta)
= \int_\mathcal{S} \eta^\mu(s) \nabla_\theta \mu_\theta(s)^\top \nabla_a Q^\pi(s, a) \big|_{a=\mu_\theta(s)} \mathrm{d}s
= \int_\mathcal{S} \eta^\mu(s) \nabla_\theta \mu_\theta(s)^\top \nabla_a  (Q^\pi(s, a) - b(s)) \big|_{a=\mu_\theta(s)} \mathrm{d}s .
\end{equation}
By choosing $r(s, a) = r(s, \mu_\theta(s)) = \nabla_\theta \mu_\theta(s) \nabla_a (Q^\mu(s, a) - b(s))\big|_{a=\mu_\theta(s)}$ in \Cref{prop: costs state-space perspective} we arrive at \eqref{eq: alternative trajectory dpg baseline} in the trajectory-based perspective.
\end{proof}

\section{Computational details}
\label{app: computational details}

In this section we provide computational details that are necessary for the numerical approximation of the gradients discussed in the main part.

For policy gradients of stochastic policies we consider non-actor-critic approaches where the $Q$-value function is estimated, but not learned. The alternative objectives corresponding to the trajectory PG stated in \eqref{eq: alternative trajectory policy gradient m=0} and to the state-space PG stated in \eqref{eq: policy gradient state-space perspective} are given by
\begin{align}
\label{eq: alternative objective trajectory stochastic policy}
J_\mathrm{eff}^\mathrm{traj}(\pi_\theta, \pi_\vartheta) &:= \E_{\pi_\vartheta} \left[\sum_{n=0}^N \log \pi_\theta(S_n, A_n) \sum_{m=0}^{N} r(S_m, A_m) \right], \\
\label{eq: alternative objective state stochastic policy}
J_\mathrm{eff}^\mathrm{state}(\pi_\theta, \pi_\vartheta) &:= \E_{\pi_\vartheta} \left[ N + 1 \right] \E_{\substack{s \sim \rho^{\pi_\vartheta} \\ a \sim \pi_\vartheta(s, \cdot)}} \left[ \log \pi_\theta(s, a) Q^{\pi_\vartheta}(s, a) \right].
\end{align}
We refer to \Cref{alg: alg1,alg: alg2} for implementational details.

For policy gradients of deterministic policies we consider non-actor-critic approaches as well. Recall that this is possible due to the model-based formulas provided in \eqref{eq: model-based deterministic policy gradient trajectory} and \eqref{eq: alternative trajectory dpg m=0}. Moreover, we have assumed that the state-action probability density is Gaussian and is motivated by a controlled SDE (see \Cref{rem: connection to stochastic optimal control}). The corresponding alternative objectives are given by
\begin{align}
\label{eq: alternative objective trajectory deterministic policy}
J_\mathrm{eff}^\mathrm{traj}(\mu_\theta, \mu_\vartheta)
&:= \E_{\mu_\vartheta} \Bigg[\sum_{n=0}^{N} \mu_\theta(S_n)^\top \Big( \nabla_a r(S_n, a)\big|_{a=\mu_\vartheta(S_n)} +  \sum_{m=0}^N r(S_m, \mu_\vartheta(S_m)) \xi_{n+1} \sqrt{\Delta t} \Big) \Bigg], \\
\label{eq: alternative objective state deterministic policy}
J_\mathrm{eff}^\mathrm{state}(\mu_\theta, \mu_\vartheta) 
&:= \E_{\mu_\vartheta} \left[ N + 1 \right] \, \mathbb{E}_{\substack{s \sim \rho^{\mu_\vartheta}, \\ s' \sim p^{\mu_\vartheta}(\cdot, s)}} \left[ \mu_\theta(s)^\top \Bigl( \nabla_a r(s, a)\big|_{a=\mu_\vartheta(s)} + V^{\mu_\vartheta}(s') \xi_{n+1} \sqrt{\Delta t}  \Bigr) \right].
\end{align}
We refer to \Cref{alg: alg3,alg: alg4} for implementational details and note that algorithm \Cref{alg: alg3} is already sketched in \citet{quer_connecting_2024}.

\begin{algorithm}
\caption{Trajectory Policy Gradient (REINFORCE with random time horizon).} \label{alg: alg1}
\begin{algorithmic} [1]
\State Initialize stochastic policy $\pi_\theta$ with random parameters.
\State Choose a gradient based optimization algorithm, a learning rate $\lambda$, a sample size $K$, and a stopping criterion.
\Repeat
\State Simulate $K$ samples of trajectories following the policy $\pi_\theta$, where the $k$-th trajectory has runtime $N^{(k)}$.
\State Estimate alternative objective stated in \eqref{eq: alternative objective trajectory stochastic policy} and compute the gradient via automatic differentiation
\[
\widehat{J}_\mathrm{eff}^\mathrm{traj}
= \frac{1}{K}\sum_{k=1}^K \sum_{n=0}^{N^{(k)}} \log \pi_\theta(S_n^{(k)}, A_n^{(k)}) \, \sum_{m=0}^{N^{(k)}} r(S_m^{(k)}, A_m^{(k)}).
\]
\State Update policy network parameters based on the optimization algorithm.
\Until{stopping criterion is fulfilled.}
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
\caption{State-space Policy Gradient.} \label{alg: alg2}
\begin{algorithmic} [1]
\State Initialize stochastic policy $\pi_\theta$ with random parameters.
\State Choose a gradient based optimization algorithm, a learning rate $\lambda$, a sample size $K$ for the trajectories, a sample size $M$ for the experiences and a stopping criterion.
\Repeat
\State Simulate $K$ samples of trajectories following the policy $\pi_\theta$, where the $k$-th trajectory has runtime $N^{(k)}$.
\State Compute estimate $\widehat{Z}^\pi = \frac{1}{K}\sum_{k=1}^{K} N^{(k)} + 1 $.
\For{each time step $n$}
\State Compute return
$G_n^{(k)} = \sum_{m=n}^{N^{(k)}} r(S_m^{(k)}, A_m^{(k)})$ and store the tuple $(S_n^{(k)}, A_n^{(k)}, G_n^{(k)})$ in memory.
\EndFor
\State Sample $M$ experiences from memory $\{\widetilde{S}^{(m)}, \widetilde{A}^{(m)}, \widetilde{G}^{(m)} \}_{m=1}^M$, where $\widetilde{S}^{(m)}, \widetilde{A}^{(m)}$ are unbiased samples from $\rho^\pi$ and $\pi(\widetilde{S}^{(m)}, \cdot)$, respectively, and $\widetilde{G}^{(m)}$ estimates $Q^{\pi}(\widetilde{S}^{(m)}, \widetilde{A}^{(m)})$.  
\State Estimate alternative objective stated in \eqref{eq: alternative objective state stochastic policy} and compute the gradient via automatic differentiation
\[
\widehat{J}_\mathrm{eff}^\mathrm{state}
= \widehat{Z}^\pi \frac{1}{M}\sum_{m=1}^{M} \log \pi_\theta(\widetilde{S}^{(m)}, \widetilde{A}^{(m)}) \widetilde{G}^{(m)} .
\]
\State Update policy network parameters based on the optimization algorithm.
\State Empty memory.
\Until{stopping criterion is fulfilled.}
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
\caption{Trajectory and model-based Deterministic Policy Gradient.}
\label{alg: alg3}
\begin{algorithmic} [1]
\State Initialize deterministic policy $\mu_\theta$ with random parameters.
\State Choose a gradient based optimization algorithm, a learning rate $\lambda$, a sample size $K$, and a stopping criterion.
\Repeat
\State Simulate $K$ samples of trajectories following the policy $\mu_\theta$.
\State Estimate alternative objective stated in \eqref{eq: alternative objective trajectory deterministic policy} and compute the gradient via automatic differentiation
\begin{equation*}
\widehat{J}_\mathrm{eff}^\mathrm{traj}
= \frac{1}{K}\sum_{k=1}^K  \Biggl( \sum_{n=0}^{N^{(k)}} \mu_\theta(S_n^{(k)})^\top \Biggl(\nabla_a r(S_n^{(k)}, a)\big|_{a=\mu_\vartheta(S_n^{(k)})} + \Biggl(\sum_{m=0}^{N^{(k)}} r(S_m^{(k)}, \mu_\vartheta(S_m^{(k)})) \Biggr) \xi_{n+1}^{(k)} \sqrt{\Delta t} \Biggr) \Biggr).
\end{equation*}
\State Update the parameters $\theta$ based on the optimization algorithm.
\Until{stopping criterion is fulfilled.}
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
\caption{State-space and model-based Deterministic Policy Gradient.} \label{alg: alg4}
\begin{algorithmic} [1]
\State Initialize deterministic policy $\mu_\theta$ with random parameters.
\State Choose a gradient based optimization algorithm, a learning rate $\lambda$, a sample size $K$ for the trajectories, a sample size $M$ for the experiences and a stopping criterion.
\Repeat
\State Simulate $K$ trajectories following the policy $\mu_\theta$ and store them in memory.
\State Compute estimate $\widehat{Z}^\mu = \frac{1}{K}\sum_{k=1}^{K} N^{(k)} + 1 $.
\For{each time step $n$}
\State Compute return 
$G_{n+1}^{(k)} = \sum_{m=n+1}^N r(S_m^{(k)}, \mu_\theta(S_m^{(k)}))$ and store the tuple $(S_n^{(k)}, S_{n+1}^{(k)}, \xi_{n+1}^{(k)}, G_{n+1}^{(k)})$ in memory.
\EndFor
\State Sample $M$ experiences from memory $\{\widetilde{S}^{(m)}, \widetilde{S}'^{(m)} \widetilde{\xi}^{(m)}, \widetilde{G}^{(m)}\}_{m=1}^M$ where $\widetilde{S}^{(m)}$ is an unbiased sample from $\rho^\mu$, $\widetilde{\xi}^{(m)} \sim \mathcal{N}(0, \operatorname{Id})$, and $\widetilde{G}^{(m)}$ estimates $V^{\mu}(\widetilde{S}'^{(m)})$.
\State Estimate alternative objective $J_\mathrm{eff}$ given in \eqref{eq: alternative objective state deterministic policy} and compute the gradient via automatic differentiation
\[
\widehat{J}_\mathrm{eff}^\mathrm{state}
= \widehat{Z}^\mu  \frac{1}{M}\sum_{m=1}^M  \mu_\theta(\widetilde{S}^{(m)})^\top \Biggl(\nabla_a r(\widetilde{S}^{(m)}, a)\big|_{a=\mu_\vartheta(\widetilde{S}^{(m)})} + \widetilde{G}^{(m)} \widetilde{\xi}^{(m)} \sqrt{\Delta t} \Biggr)
\]
\State Update policy network parameters based on the optimization algorithm.
\State Empty memory.
\Until{stopping criterion is fulfilled.}
\end{algorithmic}
\end{algorithm}

\section{Experimental details and additional experiments}

In this section we provide further details on the numerical examples presented in \Cref{sec: numerical experiments}.

\subsection{Architecture of the neural networks}
\label{app: neural network architectures}
Let $d_\mathrm{in}, d_\mathrm{out} \in \mathbb{N}^+$ be the input and output dimensions of the feed-forward network $\varphi_\theta \colon \mathbb{R}^{d_\mathrm{in}} \rightarrow \mathbb{R}^{d_\mathrm{out}}$ defined by
\begin{equation}
\label{eq: feed-foreward nn}
\varphi_\theta(x) = \rho_\mathrm{out}(A_L \rho(A_{L-1} \rho(\cdots  \rho(A_1 x + b_ 1) \cdots) + b_{L-1}) + b_L),
\end{equation}
where $L$ is the number of layers $d_0 = d_\mathrm{in}, d_L = d_\mathrm{out}$, $A_l \in \mathbb{R}^{d_{l} \times d_{l-1}}$ and $b_l \in \mathbb{R}^{d_l}, 1 \le l \le L$ are the weights and the bias vectors for each layer and $\rho, \rho_\mathrm{out}: \mathbb{R} \to \mathbb{R}$ are the inner and outer nonlinear activation functions applied componentwise. The collection of matrices $A_l$ and vectors $b_l$ contains the learnable parameters $\theta \in \mathbb{R}^p$. For all the experiments we choose the inner and outer activation functions $\rho = \tanh$ and $\rho_\mathrm{out} = \operatorname{Id}$ if not otherwise stated.

For the stochastic policy experiments described in \Cref{sec: mountain car,sec: reacher} we consider a Gaussian stochastic policy $\pi(s, \cdot) = \mathcal{N} (\mu(s), \sigma^2(s) \operatorname{Id})$, for which $\mu$ and $\sigma$ are learnable functions represented by two $L=3$ layer feed-forward neural networks sharing the first $L=2$ layers (a so-called two-head neural network). To guarantee that $\sigma^2 \operatorname{Id}$ is positive definite, the final activation function of the corresponding neural network is chosen as $\rho_\mathrm{out}^{\sigma} = x + \sqrt{x^2 + 1}$. 

To ensure that the initial output of the networks is close to zero, the final layer weights and biases are initialized by sampling from the uniform distribution $\mathcal{U}(-5 \times 10^{-3}, 5 \times 10^{-3})$. We also note that each experiment requires only one CPU core, and the maximum value of allocated memory is set to 64 GB.

\subsection{Modified continuous mountain car problem}
\label{app: mountain car details}

For the experiment described in \Cref{sec: mountain car} we consider a Gaussian stochastic policy for which $\mu$ and $\sigma$ are represented by a two-head neural network (see details in \Cref{app: neural network architectures}) with $L=3$ layers and $d_1 = d_2 = 32$ units. We compare the three different policy gradient formulas by implementing \Cref{alg: alg1} (\emph{trajectory PG}), \Cref{alg: alg2} (\emph{state-space PG}) and \Cref{alg: alg2} without estimating the $Z^\pi$-factor (\emph{state-space PG unbiased}) for a batch of $K=100$ trajectories, a batch of experiences containing all the information in the memory ($M= 100 \%$ of the memory size), and we stop the optimization algorithm after $I = 5 \times 10^4$ gradient iterations. The best performing learning rates for each gradient approach are $\lambda_\mathrm{traj} = \lambda_\mathrm{state} = 10^{-4}$, $\lambda_\mathrm{state}^\mathrm{biased} = 5 \times 10^{-2}$, respectively. Note that for $\lambda_\mathrm{state}^\mathrm{biased} \geq 10^{-1}$ the optimization algorithm fails due to high instabilities which lead to long trajectories and hence exceed the allowed allocated memory.  

\subsection{Two-joint robot arm (reacher)}
\label{app: reacher details}
For the experiment described in \Cref{sec: reacher} we consider a Gaussian stochastic policy with $L=3$ layers and $d_1 = d_2 = 32$ units and compare the performance of \Cref{alg: alg1} (\emph{trajectory PG}), \Cref{alg: alg2} (\emph{state-space PG}), and \Cref{alg: alg2} without estimating the $Z^\pi$-factor (\emph{state-space PG unbiased}) for $K=100$ trajectories, $M= 100 \%$ of experiences in memory, and $I = 10^4$ gradient iterations. The best performing learning rates for each gradient approach are $\lambda_\mathrm{traj} = \lambda_\mathrm{state} = 5 \times 10^{-4}$, $\lambda_\mathrm{state}^\mathrm{biased} = 5 \times 10^{-2}$, respectively. 


\subsection{Importance sampling of hitting times in molecular dynamics}
For the experiment described in \Cref{sec: double well numerics} we consider a deterministic policy represented by a neural network  with $L=2$ layers and $d_1 = 32$. We compare the three different model-based policy gradient formulas for deterministic policies by implementing \Cref{alg: alg3} (\emph{trajectory DPG}), \Cref{alg: alg4} (\emph{state-space DPG}) and \Cref{alg: alg4} without estimating the $Z^\mu$-factor (\emph{state-space DPG unbiased}) for $K=500$ trajectories, $M= 100 \%$ of experiences in memory, and $I = 5 \times 10^4$ gradient iterations. The best performing learning rates for each gradient approach are $\lambda_\mathrm{traj} = \lambda_\mathrm{state} = 2 \times 10^{-3}$, $\lambda_\mathrm{state}^\mathrm{biased} = 5 \times 10^{-1}$, respectively.
\end{document}
\begin{thebibliography}{31}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Agarwal et~al.(2022)Agarwal, Jiang, Kakade, and
  Sun]{agarwal_reinforcement_2022}
Agarwal, A., Jiang, N., Kakade, S.~M., and Sun, W.
\newblock Reinforcement {Learning}: {Theory} and {Algorithms}.
\newblock 2022.
\newblock URL \url{https://rltheorybook.github.io/}.

\bibitem[Berner et~al.(2024)Berner, Richter, and Ullrich]{berner2022optimal}
Berner, J., Richter, L., and Ullrich, K.
\newblock An optimal control perspective on diffusion-based generative
  modeling.
\newblock \emph{Transactions on Machine Learning Research}, 2024.

\bibitem[Bojun(2020)]{bojun2020steady}
Bojun, H.
\newblock Steady state analysis of episodic reinforcement learning.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}},
  volume~33, pp.\  9335--9345, 2020.

\bibitem[Chen et~al.(2024)Chen, Ross, and Youssef]{chen2024finite}
Chen, S.-W., Ross, K., and Youssef, P.
\newblock Finite-sample analysis of the {M}onte {C}arlo exploring starts
  algorithm for reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2410.02994}, 2024.

\bibitem[Degris et~al.(2012)Degris, White, and Sutton]{degris2012off}
Degris, T., White, M., and Sutton, R.~S.
\newblock Off-policy actor-critic.
\newblock In \emph{International Conference on Machine Learning}. PMLR, 2012.

\bibitem[Derman(1970)]{derman1970finite}
Derman, C.
\newblock \emph{Finite state Markovian decision processes}.
\newblock Academic Press, Inc., 1970.

\bibitem[Hartmann \& Sch{\"u}tte(2012)Hartmann and
  Sch{\"u}tte]{hartmann2012efficient}
Hartmann, C. and Sch{\"u}tte, C.
\newblock Efficient rare event simulation by optimal nonequilibrium forcing.
\newblock \emph{Journal of Statistical Mechanics: Theory and Experiment},
  2012\penalty0 (11):\penalty0 P11004, 2012.

\bibitem[Hartmann et~al.(2017)Hartmann, Richter, Sch{\"u}tte, and
  Zhang]{hartmann2017variational}
Hartmann, C., Richter, L., Sch{\"u}tte, C., and Zhang, W.
\newblock Variational characterization of free energy: Theory and algorithms.
\newblock \emph{Entropy}, 19\penalty0 (11):\penalty0 626, 2017.

\bibitem[Lie(2021)]{lie2021fr}
Lie, H.~C.
\newblock Fr\'echet derivatives of expected functionals of solutions to
  stochastic differential equations.
\newblock \emph{arXiv preprint arXiv:2106.09149}, 2021.

\bibitem[Lillicrap(2016)]{lillicrap2015continuous}
Lillicrap, T.
\newblock Continuous control with deep reinforcement learning.
\newblock In \emph{International Conference on Learning Representations}, 2016.

\bibitem[Mandal et~al.(2023)Mandal, Radanovic, Gan, Singla, and
  Majumdar]{mandal_online_2023}
Mandal, D., Radanovic, G., Gan, J., Singla, A., and Majumdar, R.
\newblock Online reinforcement learning with uncertain episode lengths.
\newblock \emph{Proceedings of the AAAI Conference on Artificial Intelligence},
  37\penalty0 (7):\penalty0 9064--9071, 2023.

\bibitem[Mnih et~al.(2013)Mnih, Kavukcuoglu, Silver, Graves, Antonoglou,
  Wierstra, and Riedmiller]{mnih2013playing}
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra,
  D., and Riedmiller, M.
\newblock Playing {A}tari with deep reinforcement learning.
\newblock \emph{arXiv preprint arXiv:1312.5602}, 2013.

\bibitem[Nota \& Thomas(2020)Nota and Thomas]{nota2019policy}
Nota, C. and Thomas, P.~S.
\newblock Is the policy gradient a gradient?
\newblock In \emph{Proceedings of the 19th International Conference on
  Autonomous Agents and MultiAgent Systems}, pp.\  939--947, 2020.

\bibitem[N{\"u}sken \& Richter(2021)N{\"u}sken and Richter]{nusken2021solving}
N{\"u}sken, N. and Richter, L.
\newblock Solving high-dimensional {H}amilton--{J}acobi--{B}ellman {PDE}s using
  neural networks: perspectives from the theory of controlled diffusions and
  measures on path space.
\newblock \emph{Partial differential equations and applications}, 2\penalty0
  (4):\penalty0 48, 2021.

\bibitem[Paternain(2018)]{paternain2018stochastic}
Paternain, S.
\newblock \emph{Stochastic control foundations of autonomous behavior}.
\newblock University of Pennsylvania, 2018.

\bibitem[Pham(2009)]{pham2009continuous}
Pham, H.
\newblock \emph{Continuous-time stochastic control and optimization with
  financial applications}, volume~61.
\newblock Springer Science \& Business Media, 2009.

\bibitem[Puterman(2014)]{puterman2014markov}
Puterman, M.~L.
\newblock \emph{Markov decision processes: discrete stochastic dynamic
  programming}.
\newblock John Wiley \& Sons, 2014.

\bibitem[Quer \& Ribera~Borrell(2024)Quer and
  Ribera~Borrell]{quer_connecting_2024}
Quer, J. and Ribera~Borrell, E.
\newblock Connecting stochastic optimal control and reinforcement learning.
\newblock \emph{Journal of Mathematical Physics}, 65\penalty0 (8):\penalty0
  083512, 2024.

\bibitem[Ribera~Borrell et~al.(2024)Ribera~Borrell, Quer, Richter, and
  Sch{\"u}tte]{ribera2024improving}
Ribera~Borrell, E., Quer, J., Richter, L., and Sch{\"u}tte, C.
\newblock Improving control based importance sampling strategies for metastable
  diffusions via adapted metadynamics.
\newblock \emph{SIAM Journal on Scientific Computing}, 46\penalty0
  (2):\penalty0 S298--S323, 2024.

\bibitem[Schulman et~al.(2015)Schulman, Levine, Abbeel, Jordan, and
  Moritz]{schulman2015trust}
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P.
\newblock Trust region policy optimization.
\newblock In \emph{International Conference on Machine Learning}, pp.\
  1889--1897. PMLR, 2015.

\bibitem[Schulman et~al.(2017)Schulman, Wolski, Dhariwal, Radford, and
  Klimov]{schulman2017proximal}
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.
\newblock Proximal policy optimization algorithms.
\newblock \emph{arXiv preprint arXiv:1707.06347}, 2017.

\bibitem[Sch{\"u}tte et~al.(2023)Sch{\"u}tte, Klus, and
  Hartmann]{schutte2023overcoming}
Sch{\"u}tte, C., Klus, S., and Hartmann, C.
\newblock Overcoming the timescale barrier in molecular dynamics: Transfer
  operators, variational principles and machine learning.
\newblock \emph{Acta Numerica}, 32:\penalty0 517--673, 2023.

\bibitem[Silver et~al.(2014)Silver, Lever, Heess, Degris, Wierstra, and
  Riedmiller]{silver2014deterministic}
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M.
\newblock Deterministic policy gradient algorithms.
\newblock In \emph{Proceedings of the 31st {International} {Conference} on
  {Machine} {Learning}}, pp.\  387--395. PMLR, 2014.

\bibitem[Singh \& Sutton(1996)Singh and Sutton]{singh1996reinforcement}
Singh, S.~P. and Sutton, R.~S.
\newblock Reinforcement learning with replacing eligibility traces.
\newblock \emph{Machine Learning}, 22\penalty0 (1):\penalty0 123--158, 1996.

\bibitem[Sutton \& Barto(2018)Sutton and Barto]{sutton2018reinforcement}
Sutton, R.~S. and Barto, A.~G.
\newblock Reinforcement learning: An introduction.
\newblock \emph{A Bradford Book}, 2018.

\bibitem[Sutton et~al.(1999)Sutton, McAllester, Singh, and
  Mansour]{sutton1999policy}
Sutton, R.~S., McAllester, D., Singh, S., and Mansour, Y.
\newblock Policy gradient methods for reinforcement learning with function
  approximation.
\newblock \emph{Advances in neural information processing systems}, 12, 1999.

\bibitem[Towers et~al.(2024)Towers, Kwiatkowski, Terry, Balis, De~Cola, Deleu,
  Goulão, Kallinteris, Krimmel, KG, Perez-Vicente, Pierré, Schulhoff, Tai,
  Tan, and Younis]{towers_gymnasium_2024}
Towers, M., Kwiatkowski, A., Terry, J., Balis, J.~U., De~Cola, G., Deleu, T.,
  Goulão, M., Kallinteris, A., Krimmel, M., KG, A., Perez-Vicente, R.,
  Pierré, A., Schulhoff, S., Tai, J.~J., Tan, H., and Younis, O.~G.
\newblock Gymnasium: {A} standard interface for reinforcement learning
  environments, 2024.

\bibitem[Wang et~al.(2020)Wang, Zariphopoulou, and Zhou]{wang2020reinforcement}
Wang, H., Zariphopoulou, T., and Zhou, X.~Y.
\newblock Reinforcement learning in continuous time and space: A stochastic
  control approach.
\newblock \emph{Journal of Machine Learning Research}, 21\penalty0
  (198):\penalty0 1--34, 2020.

\bibitem[White(2017)]{white_unifying_2017}
White, M.
\newblock Unifying task specification in reinforcement learning.
\newblock In \emph{Proceedings of the 34th {International} {Conference} on
  {Machine} {Learning}}, volume~70, pp.\  3742--3750. PMLR, 2017.

\bibitem[Zhang et~al.(2020)Zhang, Koppel, Zhu, and Basar]{zhang2020global}
Zhang, K., Koppel, A., Zhu, H., and Basar, T.
\newblock Global convergence of policy gradient methods to (almost) locally
  optimal policies.
\newblock \emph{SIAM Journal on Control and Optimization}, 58\penalty0
  (6):\penalty0 3586--3612, 2020.

\bibitem[Zhou et~al.(2021)Zhou, Han, and Lu]{zhou2021actor}
Zhou, M., Han, J., and Lu, J.
\newblock Actor-critic method for high dimensional static
  {H}amilton--{J}acobi--{B}ellman partial differential equations based on
  neural networks.
\newblock \emph{SIAM Journal on Scientific Computing}, 43\penalty0
  (6):\penalty0 A4043--A4066, 2021.

\end{thebibliography}