Explicit and Effectively Symmetric Schemes for Neural SDEs on Lie Groups
Extracted source text · arXiv 2509.20599
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% ---- arXiv toggle (leave commented for ICLR submission) ----
\def\isarxiv{1} % <-- UNCOMMENT this line *only* for the arXiv build
\makeatletter
\@ifundefined{isarxiv}{}{\iclrfinalcopy} % final mode => names shown, no line-number ruler
\makeatother
% ------------------------------------------------------------
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% \usepackage[nocompress]{cite}
\usepackage{lipsum}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{epstopdf}
%\usepackage{algorithmic}
\ifpdf
\DeclareGraphicsExtensions{.eps,.pdf,.png,.jpg}
\else
\DeclareGraphicsExtensions{.eps}
\fi
\usepackage{subcaption}
\usepackage{dirtytalk}
\usepackage{multirow}
\usepackage{array}
\newcolumntype{?}{!{\vrule width 1pt}}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage[dvipsnames]{xcolor}
\usepackage{tikz}
\usetikzlibrary{decorations.pathreplacing}
\usetikzlibrary{arrows}
\setlength{\parindent}{0pt}
\usepackage{stmaryrd}
\usepackage{mathrsfs}
\usepackage{enumitem}
\setenumerate{noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt}
\setlength{\textfloatsep}{5pt}
\usepackage{wrapfig}
\usepackage{csquotes}
\DeclareFontEncoding{LS1}{}{}
\DeclareFontSubstitution{LS1}{stix}{m}{n}
\DeclareSymbolFont{stixletters}{LS1}{stix}{m}{it}
\DeclareMathAccent{\backvec}{\mathord}{stixletters}{"91}
\usepackage{amsmath}
\usepackage{amsthm}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\newtheorem{notation}{Notation}[section]
\newtheorem{example}{Example}[section]
\theoremstyle{remark}
\newtheorem*{remark}{Remark}
\newtheorem{proposition}{Proposition}[section]
\newtheorem{theorem}{Theorem}[section]
\input{trees}
\title{Explicit and Effectively Symmetric Schemes for Neural SDEs}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{%
Daniil Shmelev\\
Department of Mathematics\\
Imperial College London\\
\texttt{daniil.shmelev23@imperial.ac.uk} \\
\And
Cristopher Salvi\\
Department of Mathematics\\
Imperial College London\\
\texttt{c.salvi@imperial.ac.uk} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\newcommand{\TODO}[1]{{\color{red}\textbf{{#1}}}}
% \iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
% ---- remove ICLR header/footer branding for arXiv build ----
\makeatletter
\@ifundefined{isarxiv}{}{%
\thispagestyle{plain}
\pagestyle{plain}
\fancyhead{} % blank out fancyhdr
\fancyfoot{}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
}
\makeatother
% ------------------------------------------------------------
\begin{abstract}
Backpropagation through (neural) SDE solvers is traditionally approached in two ways: discretise-then-optimise, which offers accurate gradients but incurs prohibitive memory costs due to storing the full computational graph (even when mitigated by checkpointing); and optimise-then-discretise, which achieves constant memory cost by solving an auxiliary backward SDE, but suffers from slower evaluation and gradient approximation errors. Algebraically reversible solvers promise both memory efficiency and gradient accuracy, yet existing methods such as the Reversible Heun scheme are often unstable under complex models and large step sizes. We address these limitations by introducing a novel class of stable, near-reversible Runge–Kutta schemes for neural SDEs. These \emph{Explicit and Effectively Symmetric (EES)} schemes retain the benefits of reversible solvers while overcoming their instability, enabling memory-efficient training without severe restrictions on step size or model complexity. Through numerical experiments, we demonstrate the superior stability and reliability of our schemes, establishing them as a practical foundation for scalable and accurate training of neural SDEs.
\end{abstract}
\section{Introduction}
\emph{Neural stochastic differential equations (SDEs)} have recently emerged as a flexible tool for modelling stochastic dynamics, with training typically cast as a distribution-matching problem between generated and observed trajectories. Several approaches have been proposed in the literature, differing mainly in the choice of discriminating divergence. SDE-GANs~\citep{kidger2021neural} use the 1-Wasserstein distance, while Latent SDEs~\citep{li2020scalable} optimize with respect to the KL divergence via variational inference, and can be viewed as variational autoencoders. Another alternative proposed by~\cite{issa2023non} trains neural SDEs non-adversarially using maximum mean discrepancies (MMD) with \emph{signature kernels}~\citep{kiraly2019kernels, salvi2021signature, lemercier2024log}, a recently introduced family of kernels on path space which received significant attention due to their efficiency in handling path-dependent problems~\citep{salvi2021higher, lemercier2021distribution, pannier2024path, cirone2025rough}.
While effective, the stochastic calculus underpinning SDEs can be technically cumbersome, especially when developing higher order solvers, deriving and analysing backpropagation algorithms, or extending to rougher noises than Brownian motion. A natural way to bypass these limitations is to view SDEs through the lens of \emph{rough paths}. Rough path theory \citep{lyons1998differential, gubinelli2004controlling, gubinelli2010ramification} provides a deterministic calculus on path space that extends classical It\^o integration beyond semimartingales, enabling one to treat SDEs as a special case of \emph{rough differential equations (RDEs)}
\begin{equation}\label{eqn:rde}
dy(t) \;=\; f_0(t, y_t)\,dt \;+\; f(t, y_t)\,d\mathbf{X}(t),
\qquad y_{t_0} = y_0 \in \mathbb{R}^m,
\end{equation}
where $\mathbf{X}$ is a \emph{rough path} above a driving signal $X = (X^1,...,X^d) : [0,T] \to \mathbb{R}^d$, $f = (f_1,...,f_d)$ and $f_i : \mathbb{R}^m \to \mathbb{R}^m$ are vector fields for $i=0,...,d$ which determine the system dynamics. In fact, there is a large class of stochastic processes which can be “naturally lifted” to rough paths, including Gaussian processes such as fractional Brownian motion with Hurst parameter $H > 1/4$ and Volterra processes \citep{friz2010multidimensional}. Viewing SDEs through the lens of RDEs provides a significant conceptual simplification. In the RDE framework, the driving signal $\mathbf{X}$ is treated as a rough path, which abstracts away the stochastic integral and allows one to work with deterministic calculus on path space. Rough path theory has become a powerful mathematical framework for analyzing modern machine learning models. It has provided the foundation for proving theoretical properties of neural differential equations~\citep{morrill2021neural, arribas2020sigsdes, cirone2023neural, holberg2024exact}, sequence-to-sequence architectures~\citep{kidger2019deep}, and more recently for deep selective state-space models (SSMs)~\citep{cirone2024theoretical, cirone2025parallelflow, walker2025structured} and score-based diffusion models~\citep{barancikovasigdiffusions}. for a survey of recent applications of rough path theory to machine learning, we refer the reader to~\citep{fermanian2023new}.
From an algorithmic perspective, the RDE formulation also plays a central role in training neural SDEs (NSDEs), where one must perform backpropagation through an SDE solver. For example, the derivation of the \emph{adjoint method}---a key tool for backpropagation through differential equation solvers---becomes far more transparent when formulated in terms of RDEs, avoiding much of the technical machinery required in the stochastic setting; see for instance ~\citep{cass_salvi_notes}. More generally, the RDE viewpoint unifies backpropagation across ODEs, SDEs, and controlled differential equations (CDEs) \citep{kidger2020neural}, offering a clean pathwise calculus that is both mathematically rigorous and practically aligned with autodiff frameworks. Several strategies have been proposed. A first approach, known as \emph{discretise-then-optimise}, directly backpropagates through the solver’s internal operations. This yields accurate gradients and is computationally efficient, but requires storing all intermediate states, making it memory-intensive. A second approach, \emph{optimise-then-discretise}, instead derives a backwards-in-time adjoint equation and solves it numerically using another call to the solver. This eliminates the need to store intermediate quantities, resulting in constant memory cost with respect to solver depth. However, it typically produces less accurate gradients and is slower due to the need to recompute forward trajectories during the backward pass.
A third option leverages \emph{algebraically reversible solvers}, which enable the exact reconstruction of the solution trajectory of a differential equation from its terminal point, allowing for accurate and memory-efficient backpropagation. In the setting of an autonomous ODE
\begin{equation} \label{eq:ode}
dy_t = f(y_t) dt,
\end{equation}
a one step method $y_{n+1} = y_n + \Phi_h(y_n)$ is said to be \emph{reversible} or \emph{symmetric} if a step of the method starting from $y_1$ with a negative step size exactly recovers the initial condition $y_0$, that is, $\Phi_{-h} = \Phi^{-1}_h$. Whilst reversible schemes offer an efficient approach to backpropagation through differential equations, such schemes are difficult to construct. It is well known that Runge--Kutta schemes are reversible only if they are implicit, making them unsuitable for applications to Neural ODEs. More generally, symmetric parasitism-free general linear methods cannot be explicit \citep{butcher2016symmetric}. To overcome this problem, existing reversible methods proposed in literature, such as the asynchronous leapfrog integrator (ALF) \citep{zhuang2021mali} and the Reversible Heun method \citep{kidger2021efficient}, track auxiliary states as part of the integration. Whilst this technique allows for explicit reversible schemes, it comes at the price of low stability. Both ALF and Reversible Heun are well-known to be unstable, and fail when integrating complex equations. For example, ~\cite{zhang2021path} find that Reversible Heun is too unstable for their applications, stating:
\begin{displayquote}
\textit{"Regardless the accuracy and memory advantages of Reversible Heun claimed by torchsde, we found this integration approach is less stable compared with simple Euler integration without adjoint and results in numerical issues occasionally. We empirically found that methods without adjoint are more stable and lower loss compared with adjoint ones, even in low dimensional data" ~\citep{zhang2021path}}
\end{displayquote}
A potential solution to the difficulties of reversible solvers comes with the class of \emph{Explicit and Effectively Symmetric (EES)} Runge--Kutta schemes introduced in \cite{shmelev2025explicit}. Instead of considering exactly reversible schemes, the authors propose schemes which are almost reversible, up to an acceptable tolerance level. Such schemes offer stable, explicit integration methods which are virtually indistinguishable from truly symmetric schemes in practice. The authors demonstrate the efficacy of EES schemes on classical ODEs, and show that EES schemes are capable of producing similar results to classical schemes such as RK3 and RK4. In \cite{shmelev2025explicit}, the formulation of EES schemes is limited to ODEs, without an explicit generalisation to the case of SDEs or RDEs.
Derivative-free \emph{Runge--Kutta (RK) methods} for RDEs were introduced by \cite{redmann2022runge}. Similarly to classical RK schemes for ODEs, the study of these methods is conducted through the formalism of B-series ~\citep{hairer2006geometric,mclachlan2015butcher,butcher2021b}. A B-series is an infinite series representation of a method, indexed by (labelled) non-planar rooted trees. A brief overview of B-series is given in Appendix \ref{appendix:trees}. A natural consequence of this analysis is that a general Runge--Kutta method for RDEs is given in terms of tree-iterated integrals of the underlying driving process. In practice, these tree-iterated integrals cannot be simulated directly as their distributions are often intractable. Following~\citep{deya2012milstein}, \cite{redmann2022runge} replaced these tree-iterated integrals with products of increments of the driving path. This substitution simplifies the derivation of Runge--Kutta coefficients and makes it feasible to establish order conditions up to any desired order.
In this paper, we present a formulation of EES Runge--Kutta schemes for RDEs, and demonstrate their efficacy as integrators for Neural SDEs. The paper is structured as follows. Section \ref{sec:rev_solvers} gives an overview of existing reversible methods for neural differential equations. Section \ref{sec:ees} recounts the framework of Runge--Kutta methods for RDEs introduced in \cite{redmann2020runge}. Section \ref{sec:ees} begins by introducing EES schemes for ODEs. Using the framework of \cite{redmann2020runge}, we extend EES schemes to the case of RDEs, and derive results regarding their orders of convergence. Through \emph{mean-square stability} analysis, we show that EES schemes possess a similar stability domain to classical RK3 and RK4 schemes when applied to SDEs, and are significantly more stable than existing reversible schemes designed for Neural SDEs. We end Section \ref{sec:ees} by outlining a backpropagation algorithm for explicit RDE Runge--Kutta solvers such as EES schemes. In Section \ref{sec:experiments} we demonstrate the efficacy of EES schemes as integrators for Neural SDEs with two experiments concerned with the learning of extreme stochastic dynamics where existing reversible solvers fail due to instability, and show that EES successfully overcomes this issue. Section \ref{sec:conclusion} closes this work with a summary of the results, limitations and potential avenues for future work.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Existing reversible SDE solvers}\label{sec:rev_solvers}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The major drawback of classical reversible schemes is their low efficiency. It is well known that Runge--Kutta schemes are reversible only if they are implicit. More generally, symmetric parasitism-free general linear methods cannot be explicit ~\citep{butcher2016symmetric}. A limited number of efficient reversible solvers have been proposed in the literature. The asynchronous leapfrog integrator (ALF) ~\citep{zhuang2021mali} for Neural ODEs overcomes the barrier of implicit schemes by tracking an additional state $v$ as part of the integration. Applied to the ODE \eqref{eq:ode}, the update rule of the ALF scheme can be written as
\begin{align*}
y_{n+2} &= y_n + h f\left(t_n + \frac{h}{2}, y_n + \frac{h}{2} v_n\right),\\
v_{n+2} &=2f\left(t_n + \frac{h}{2}, y_n + \frac{h}{2} v_n\right) - v_n.
\end{align*}
A similar approach is taken for SDEs of the form
\begin{equation}\label{eq:sde}
dy_t = g(t,y_t)dt + f(t,y_t)dW_t
\end{equation}
by the Reversible Heun method ~\citep{kidger2021efficient}, where the integration step reads
\begin{align*}
y_{n+1} &= y_n + \frac{1}{2}(g(t_n, v_n) + g(t_{n+1}, v_{n+1})) \Delta t + \frac{1}{2}(f(t_n, v_n) + f(t_{n+1}, v_{n+1})) \Delta W_n,\\
v_{n+1} &= 2y_n - v_n + g(t_n, v_n) \Delta t + f(t_n, v_n) \Delta W_n.
\end{align*}
The Reversible Heun method is efficient, requiring only one evaluation of the drift $g$ and the diffusion $f$ per step. However, the method is inherently unstable.
\begin{theorem}{\cite[Theorem D.19]{kidger2021efficient}}
Suppose that the Reversible Heun method is used to obtain a solution $\{y_n, v_n\}_{n \geq 0}$ to the linear test ODE $dy = \lambda y\, dt$, where $\lambda \in \mathbb{C}$ and $y_0 \neq 0$. Then $\{y_n, v_n\}_{n \geq 0}$ is bounded if and only if $\lambda h \in [-i, i]$.
\end{theorem}
As remarked in ~\cite{kidger2021efficient}, this domain is also the absolute stability region for the reversible asynchronous leapfrog integrator ~\citep{zhuang2021mali}. This instability has proven to be a significant bottleneck in certain practical applications. In ~\cite{mccallum2024efficient}, a method was proposed for transforming any ODE integration method $y_{n+1} = y_n + \Psi_h(t,y_n)$ into one which is reversible, by taking the update
\begin{align*}
y_{n+1} &= \lambda y_n + (1-\lambda) v_n + \Psi_h(t,v_n),\\
v_{n+1} &= v_n - \Psi_{-h}(t_{n+1}, y_{n+1}),
\end{align*}
for a given coupling parameter $\lambda \in (0,1]$. The method offers a way of constructing reversible schemes with larger stability domains than those of the ALF and Reversible Heun integrators \cite[Theorem 2.3]{mccallum2024efficient}. However, the resulting stability domain of the transformed method is typically much smaller than that of the underlying method $\Psi$, and depends additionally on the coupling parameter $\lambda$.
% \subsection{Runge--Kutta methods for RDEs}\label{sec:rde_rk}
\section{Explicit and Effectively Symmetric (EES) schemes for RDEs}\label{sec:ees}
We will adopt the general framework of RDEs for our analysis of reversible SDE solvers, following the work of \citet{redmann2020runge}. As discussed in the introduction, a generalised Runge--Kutta scheme for RDEs can be formulated in terms of tree-iterated integrals of the underlying driving rough path. For a brief introduction to these generalised methods, we refer the reader to Appendix \ref{appendix:general_rk}. Computation of such tree-iterated integrals is usually not tractable in practice, and so we adopt the simplified scheme given instead in terms of products of increments of the driving rough path.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Simplified Runge--Kutta methods for RDEs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Throughout, we will consider rough differential equations (RDEs) of the form
\begin{equation}\label{eq:rde}
dy_t = f(y_t) d \mathbf{X}_t,
\end{equation}
where $\mathbf{X}$ is an $\alpha$-H\"older branched rough path for some $\alpha \in (0,1]$ and $f$ is sufficiently smooth and bounded with bounded derivatives. For an introduction to branched rough paths, we refer the reader to Appendix \ref{appendix:branched_rp}. Following \citet{redmann2022runge}, we assume that there exist smooth paths $\{X^h\}_{h > 0}$ whose natural lifts to branched rough paths $\{\mathbf{X}^h\}_{h > 0}$ converge (almost surely) to $\mathbf{X}$ under the metric for $\alpha$-H\"older rough paths $\varrho_\alpha$ (see Appendix \ref{appendix:branched_rp}) as $h \to 0$. That is, $\mathbf{X}$ is a geometric rough path. Assume that this Wong-Zakai-type approximation converges at a rate $r_0 > 0$ with respect to the inhomogemeous rough path metric for geometric rough paths $\varrho_\alpha^g$, such that $\varrho_\alpha^g(\mathbf{X}^h, \mathbf{X}) = \mathcal{O}(h^{r_0})$. For examples of such convergence rates for Gaussian processes, see ~\cite{friz2014convergence}. Let $y^h$ denote the solution associated with the driver $\mathbf{X}^h$,
\begin{equation}\label{eq:approx_rde}
dy^h_t = f(y^h_t) d\mathbf{X}^h_t.
\end{equation}
A simplified Runge-Kutta scheme for $y^h$ is defined by
\begin{align}\label{eq:simple_rk}
y^h_{n+1} &= y^h_n + \sum_{m=1}^d\sum_{i=1}^s b_i f_m(k_i) X^{(m)}_{t_n, t_{n+1}},\\
k_i &= y^h_n + \sum_{m=1}^d\sum_{j=1}^s a_{ij} f_m(k_j) X^{(m)}_{t_n, t_{n+1}},\nonumber
\end{align}
where $X_{t_n, t_{n+1}}$ denotes the increment of $X^h$ over $[t_n, t_{n+1}]$. For simplicity, we will assume equidistant grid points $t_{n+1} - t_n = h$ for all $n \geq 0$. The convergence rates derived in \citet{redmann2020runge} for schemes of the form in \eqref{eq:simple_rk} are given in Appendix \ref{appendix:convergence_simplified}.
\begin{notation}
To avoid confusion, we will generally use $\Phi$ to refer to a classical ODE Runge--Kutta method, and $\Upsilon$ to refer to an RDE method of the form given in \eqref{eq:simple_rk}. Given an ODE Runge-Kutta scheme $\Phi$, we will write $\mathcal{R}(\Phi)$ to denote the RDE scheme of the form in \eqref{eq:simple_rk} with the same coefficients $\{a_{ij}\}_{1 \leq i,j\leq s}$ and $\{b_i\}_{1 \leq i \leq s}$ as the ODE scheme.\par\medskip
\end{notation}
% \begin{theorem}[{\cite[Theorem 3.3]{redmann2020runge}}]\label{thm:simple_local}
% Let $\Phi$ be an ODE Runge--Kutta scheme. The Runge-Kutta method $\mathcal{R}(\Phi)$ approximating $y^h$ has a local error of order $(p+1)\alpha$, i.e.
% \begin{equation*}
% y^h(t_0 + h) - y_1^h = \mathcal{O}(h^{(p+1)\alpha}),
% \end{equation*}
% if and only if the ODE Runge--Kutta method $\Phi$ is of order $p$.
% \end{theorem}
% \begin{theorem}[{\cite[Theorem 4.2]{redmann2020runge}}]\label{thm:simple_global}
% Let $\Phi$ be an ODE Runge--Kutta method of order $p$. Suppose that $f$ is $\text{Lip}^\gamma_b$ for some $\gamma > 1/\alpha$. Then $\mathcal{R}(\Phi)$ has a global error rate of $\eta = \min\{r_0, (p+1)\alpha - 1\}$, where $r_0$ is the convergence rate of the Wong-Zakai approximation. That is,
% \begin{equation*}
% \max_{n = 0, \ldots N} |y(t_n) - y^h_n| = \mathcal{O}(h^\eta).
% \end{equation*}
% \end{theorem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{EES schemes for ODEs}\label{sec:ees_rdes_subsec}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
EES schemes ~\citep{shmelev2025explicit} are a class of Runge--Kutta methods which offer an efficient approach to reversible integration without compromising on stability. Given positive integers $m \geq n$, an explicit Runge--Kutta scheme $\Phi_h$ is said to be an $\mathrm{EES}(n,m)$ scheme if $\Phi_h$ is of order $n$ and applying the scheme $\Phi_{-h} \circ \Phi_h$ to an ODE recovers the initial condition up to order $m$. When $m$ is large, such schemes offer near-reversible behaviour, which is often sufficient for practical applications. Butcher tableaux for 3-stage $\mathrm{EES}(2,5)$ and 4-stage $\mathrm{EES}(2,7)$ schemes are derived in ~\cite{shmelev2025explicit}. In ~\cite{shmelev2025explicit}, the authors focus mostly on $\mathrm{EES}(2,7)$ schemes, as these significantly outperform $\mathrm{EES}(2,5)$ schemes as integrators for ODEs. For our applications to Neural SDEs, we will instead restrict ourselves to $\mathrm{EES}(2,5)$, as we do not expect the extra accuracy of $\mathrm{EES}(2,7)$ justifies the extra stage required in this case. Proposition \ref{prop:EES_2_5} gives the general form of the Butcher tableau of a scheme belonging to $\mathrm{EES}(2,5)$ in terms of a parameter $x$, which we will denote by $\mathrm{EES}(2,5;x)$.
\begin{proposition}[{\cite[Proposition 8.4]{shmelev2025explicit}}]
\label{prop:EES_2_5}
3-stage Runge--Kutta schemes belonging to $\mathrm{EES}(2,5)$ have a Butcher tableau of the form:
\begin{equation}
{\renewcommand{\arraystretch}{2.2}
\begin{array}{c|ccc}
0 &&&\\
\displaystyle\frac{1+2x}{4(1-x)} & \displaystyle\frac{1+2x}{4(1-x)} &&\\[7pt]
\displaystyle\frac{3}{4(1-x)} & \displaystyle\frac{(4x-1)^2}{4(x-1)(1-4x^2)} & \displaystyle\frac{1-x}{(1-4x^2)} &\\[7pt]
\hline
& x & \displaystyle\frac{1}{2} & \displaystyle\frac{1}{2} - x
\end{array}
}
\end{equation}
for some $x \in \mathbb{R}$, $x \neq 1, \pm \frac{1}{2}$.
\end{proposition}
\begin{wrapfigure}{r}{5cm}
\vspace{-0.8cm}
\includegraphics[width = 0.4\textwidth]{figures/stability_regions_1.pdf}
\captionsetup{oneside,margin={0.5cm,0cm}}
\caption{\textit{Stability domain for $\mathrm{EES}(2,5;1/10)$ compared to Kutta's RK3 and RK4.}}
\label{fig:ees25_stability}
\vspace{-0.5cm}
\end{wrapfigure}
The stability region for $\mathrm{EES}(2,5)$ schemes is significantly larger than those of the ALF and Reversible Heun integrators, and is comparable to classical methods such as Kutta's RK4, as shown in Figure \ref{fig:ees25_stability}. Theorem \ref{thm:ees25_ode_stability} gives the exact form of this region for $\mathrm{EES}(2,5;x)$.
\begin{theorem}\label{thm:ees25_ode_stability}
Suppose that, for some $x \neq 1, \pm \frac{1}{2}$, $\mathrm{EES}(2,5;x)$ is used to obtain a solution $\{y_n\}_{n \geq 0}$ to the linear test equation $dy = \lambda y\, dt$, where $\lambda \in \mathbb{C}$ and $y_0 \neq 0$. Then $y_n \to 0$ as $n \to \infty$ if and only if
\begin{equation*}
\left\lvert 1 + \rho + \frac{1}{2} \rho^2 + \frac{1}{8} \rho^3 \right\rvert < 1,
\end{equation*}
where $\rho = \lambda h$.
\end{theorem}
The result follows from the fact that Runge--Kutta methods applied to linear test equations admit a linear update rule, $y_{n+1} = R(\rho) y_n$, where $R(\rho)$ is the stability function associated with the scheme. As such, the proof of Theorem \ref{thm:ees25_ode_stability} is simply a direct computation of the stability function $R(\rho) = 1 + \rho + \frac{1}{2} \rho^2 + \frac{1}{8}\rho^3$ for $\mathrm{EES}(2,5;x)$, which we omit here. We note that the stability region of $\mathrm{EES}(2,5;x)$ is completely independent of the choice of the parameter $x$. Motivated by the discussion in \citet[Section 8.1]{shmelev2025explicit}, we choose to fix the parameter $x = 1/10$ and refer to $\mathrm{EES}(2,5;1/10)$ as \textit{the} $\mathrm{EES}(2,5)$ scheme from now on.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{EES schemes for RDEs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We aim to define an analogue of EES schemes for RDEs and give convergence rates for the local and global errors of these schemes. To reason about the reversibility of schemes, it will be convenient to adopt the following notation.
\begin{notation}
Let $\{y^h_n\}_{i=0, \ldots, N}$ denote the solution of a Runge--Kutta scheme $y^h_{n+1} = y^h_n + \Upsilon(y^h_n, X_{t_n, t_{n+1}})$ of the form given in \eqref{eq:simple_rk} applied to \eqref{eq:approx_rde}. Let $\backvec{\Upsilon}$ denote the reverse-time scheme, $\backvec{\Upsilon}(\cdot, X_{t_n, t_{n+1}}) := \Upsilon(\cdot, -X_{t_n, t_{n+1}})$, and let $\backvec{y}^h_n$ denote the result of an n-fold application of $\backvec{\Upsilon}$ to $y^h_n$, such that $\{\backvec{y}^h_n\}_{i=0, \ldots, N}$ is a sequence of approximations to the initial condition $y_0$.
\end{notation}
We may now define a class of $\mathrm{EES}$ schemes for RDEs, which we will denote by $\mathrm{EES}_\mathcal{R}$. In a similar fashion to \citet{shmelev2025explicit}, we will define the schemes in terms of both the local error of the solution and the local error of the recovered initial condition when the scheme is run in reverse.
\begin{definition}
When applied to \eqref{eq:approx_rde}, a Runge--Kutta scheme $y^h_{n+1} = y^h_n + \Upsilon(y^h_n, \mathbf{X}_{t_n, t_{n+1}})$ of the form \eqref{eq:simple_rk} is said to belong to $\mathrm{EES}_\mathcal{R}(n,m)$ for $m \geq n$ if
\begin{equation*}
y^h(t_1) - y^h_1 = \mathcal{O}(h^{(n+1)\alpha}),\quad y_0 - \backvec{y}^h_1 = \mathcal{O}(h^{(m+1)\alpha}).
\end{equation*}
\end{definition}
As a consequence of \citet[Theorem 3.3]{redmann2020runge}, $\mathrm{EES}_\mathcal{R}$ schemes can be constructed directly from $\mathrm{EES}$ schemes by using the same coefficients. A global error rate for the schemes is given below as a corollary of \cite[Theorem 4.2]{redmann2020runge}, noting that the error in the recovery of $y_0$ from $\backvec{y}_n^h$ is independent of the chosen Wong-Zakai approximation, and as such independent of $r_0$. Numerical experiments verifying the global rates on a test RDE driven by a fractional Brownian motion are presented in Appendix \ref{appendix:convergence}.\par\medskip
\begin{theorem}\label{thm:ees_local}
Let $\Phi \in \mathrm{EES}(n,m)$ for some $m \geq n$. Then $\mathcal{R}(\Phi) \in \mathrm{EES}_\mathcal{R}(n,m)$.
\end{theorem}
\begin{proof}
Since $\Phi$ is of order $n$, it follows from \cite[Theorem 3.3]{redmann2020runge} that $\mathcal{R}(\Phi)$ has a local error of order $(n+1)\alpha$. Consider the scheme $\mathcal{R}(\backvec{\Phi}) \circ \mathcal{R}(\Phi) = \mathcal{R}(\backvec{\Phi} \circ \Phi)$. By definition of $\Phi$, the scheme $\backvec{\Phi} \circ \Phi$ admits a B-series expansion $y_0 + \mathcal{O}(h^{m+1})$ ~\citep{shmelev2025explicit}. It follows from ~\citet{redmann2020runge} that the corresponding B-series expansion for $\mathcal{R}(\backvec{\Phi} \circ \Phi)$ is $y_0 + \mathcal{O}(h^{(m+1)\alpha})$.
\end{proof}
\begin{theorem}\label{thm:ees_global}
Let $\Upsilon \in \mathrm{EES}_\mathcal{R}(n,m)$ for some $m \geq n$. Then
\begin{equation*}
\max_{i = 0, \ldots, N}|y(t_i) - y_i| = \mathcal{O}(h^{\eta_1}), \quad \max_{i = 0, \ldots, N}\left|y_0 - \backvec{y}_i\right| = \mathcal{O}(h^{\eta_2}),
\end{equation*}
where $\eta = \min\{r_0, (n+1)\alpha - 1\}$ and $\eta_2 = (m+1)\alpha - 1$.
\end{theorem}
\begin{proof}
Since $\Phi$ is of order $n$, it follows from \cite[Theorem 4.2]{redmann2020runge} that $\mathcal{R}(\Phi)$ has a global error of order $\eta_1 = \min\{r_0, (n+1)\alpha - 1\}$. The global rate for the recovery of the initial condition $y_0$ from $\backvec{y}_i$ follows immediately from \citet[Proposition 4.1]{redmann2020runge}.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Stability of EES schemes for SDEs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As discussed in the introduction and subsequently in Section \ref{sec:ees}, $\mathrm{EES}$ schemes offer a stable alternative to reversible integration. Whilst the stability of $\mathrm{EES}$ in the case of ODEs has been studied in \citet{shmelev2025explicit} and Section \ref{sec:ees}, we are interested in the stability of $\mathrm{EES}_\mathcal{R}$ when applied to stochastic drivers for our applications to Neural SDEs. To evaluate the stability in the context of SDEs, we consider the \textit{mean-square stability}, which has been widely used for the analysis of stochastic integration methods in the literature ~\citep{higham2000mean, drummond1991computer, hernandez1993convergence, komori1995stahle, komori1994some, petersen1998general, saito1993t, saito1996stability, schurz1996asymptotical}. Given the test equations
\begin{equation}\label{eq:sde_test_eq}
dy_t = \lambda y_t dt + \mu y_t dW_t,
\end{equation}
where $\lambda, \mu \in \mathbb{C}$ and $y_0 \neq 0$ almost surely, a solution $\{y_n\}_{n \geq 0}$ derived from a numerical integrator is said to be \textit{mean-square stable} if $\lim_{n \to \infty} \mathbb{E}(|y_n|^2) = 0$. It follows in a similar fashion to Theorem \ref{thm:ees25_ode_stability} that $\mathrm{EES}(2,5;x)$ applied to \ref{eq:sde_test_eq} is mean-square stable if and only if
\begin{equation*}
\mathbb{E} \left[ \left\lvert 1 + \rho + \frac{1}{2} \rho^2 + \frac{1}{8} \rho^3 \right\rvert^2 \right] < 1,
\end{equation*}
where $\rho = \lambda dt + \mu dW_t \sim N(\lambda dt, \mu^2 dt)$. Figure \ref{fig:ees_stoch_stability} shows 4 cross-sections of the stability domain. For comparison, we take the RDE analogues of RK3 and RK4 of the form in \eqref{eq:simple_rk}, $\mathcal{R}(\text{RK3})$ and $\mathcal{R}(\text{RK4})$. Along most cross-sections, $\mathrm{EES}_\mathcal{R}(2,5)$ achieves a similar or greater stability to $\mathcal{R}(\text{RK3})$ and $\mathcal{R}(\text{RK4})$.
\begin{figure}[H]
\centering
\includegraphics[width = \textwidth,trim={0.95cm .3cm 0cm .4cm},clip]{figures/stoch_stability.pdf}
\caption{\textit{Cross sections of the mean-square stability domains of $\mathrm{EES}_\mathcal{R}(2,5)$, $\mathcal{R}(\text{RK3})$ and $\mathcal{R}(\text{RK4})$.}}
\label{fig:ees_stoch_stability}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Backpropagation through explicit Runge--Kutta methods}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The algorithm for backpropagation through an explicit Runge--Kutta scheme $\Upsilon$ of the form in \eqref{eq:simple_rk} is given in Algorithm \ref{alg:rk_backprop}. We assume the solver is applied to a (neural) RDE of the form
\begin{equation}
dy^h_t = f(y^h_t; \theta) d\mathbf{X}^h_t,
\end{equation}
where $\theta$ are learnable parameters requiring backpropagation, trained with respect to a loss $L(\{y^h_n\}_{n=0}^N)$. As with all reversible schemes, a reverse step ${\backvec{\Upsilon}}$ is used to recover $y_n$ from $y_{n+1}$, followed by a backpropagation through the internal operations of the solver $\Upsilon$. The latter step is achieved by defining $z_i = f(k_i; \theta)$ and computing the derivatives $\partial L / \partial z_i$ and $\partial L / \partial k_i$ in reverse through the stages $i = s, s-1, \ldots, 1$. At each stage, a backpropagation algorithm is called to backpropagate the derivative $\partial L / \partial z_i$ through $f$, resulting in the derivative $\partial L / \partial k_i$ and a local derivative with respect to $\theta$, $d_\theta$.
\begin{algorithm}
\caption{Backpropagation through Explicit Runge--Kutta Schemes}\label{alg:rk_backprop}
\textbf{Input:} $y_{n+1}$, $\partial_{y_{n+1}} L$\\
\textbf{Input:} Running derivative with respect to $\theta$, $\partial_\theta L$\\
\textbf{Input:} Explicit RK method $\Upsilon$ of the form in \eqref{eq:simple_rk} with coefficients $\{a_{ij}\}_{1 \leq i,j \leq s}$ and $\{b_i\}_{1 \leq i \leq s}$.
\begin{algorithmic}
\State $y_n = \backvec{\Upsilon}(y_{n+1}, dX)$
\For {$i = s, \ldots 1$}
\State $\partial_{z_i} L = b_i dX \cdot \partial_{y_{n+1}}L + \sum_{j=i+1}^s a_{ji} dX \cdot \partial_{k_j}L$
\State $d_\theta, \partial_{k_i} L = \text{backprop}_f( \partial_{z_i}L)$
\State $\partial_\theta L \mathrel{+}= d_\theta$
\EndFor
\State $\partial_{y_n} L = \partial_{y_{n+1}} L + \sum_{i=1}^s \partial_{k_i} L$\\
\Return $y_n, \partial_{y_n} L, \partial_\theta L$
\end{algorithmic}
\end{algorithm}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}\label{sec:experiments}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We evaluate the performance of $\mathrm{EES}_\mathcal{R}(2,5)$ against the Reversible Heun method on two examples of Neural SDEs with challenging dynamics. In both cases, we encounter stability issues with the Reversible Heun method which $\mathrm{EES}_\mathcal{R}(2,5)$ manages to overcome, resulting in faster training and a lower terminal loss.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{High volatility Ornstein–Uhlenbeck process}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Consider learning the 2-dimensional Ornstein–Uhlenbeck (OU) dynamics
\begin{equation*}
dy_t = \nu(\mu - y_t) dt + \sigma dW_t
\end{equation*}
under a high-volatility regime $\sigma \gg 0$. Motivated by ~\citet{oh2024stable}, we take a Neural Langevin SDE (LSDE) defined by
\begin{equation*}
dz_t = g(z_t; \theta_g) dt + f(t; \theta_f) \circ dW_t, \quad z_0 = h(\mathbf{x}, \theta_h) \in \mathbb{R}^{d_z},
\end{equation*}
\begin{wrapfigure}{r}{5cm}
\vspace{-0.3cm}
\includegraphics[width = 0.4\textwidth]{figures/OU_mse.pdf}
\captionsetup{oneside,margin={0.5cm,0cm}}
\caption{\textit{Training MSE for OU dynamics with $\nu = 0.2$, $\mu = 0.1$ and $\sigma = 2$.}}
\label{fig:lsde}
\vspace{-0.1cm}
\end{wrapfigure}
where $h$ is a learnable affine function of the input data $\mathbf{x} = \{x_n\}_{n\geq 0}$, $x_n \in \mathbb{R}^2$, sampled from the true OU dynamics, and $g,f$ are neural networks parametrised by $\theta_g,\theta_f$ respectively. We choose the dimension of the latent representation $d_z = 32$, and parametrise $f,g$ as 2-layer neural networks of width 32 with LipSwish activations. The SDEs are integrated over $t \in [0,10]$ with a step size of 0.1, and the LSDE is trained for 250 epochs using the Adam optimiser with a fixed learning rate of $10^{-3}$. At each epoch, 50,000 realisations of the trained dynamics are sampled and the MSE loss is computed against the true OU dynamics. Figure \ref{fig:lsde} shows the training loss of the model when Reversible Heun and $\mathrm{EES}_\mathcal{R}(2,5)$ are used as the solver. For the initial $\sim 60$ epochs, the methods perform identically. After this, $\mathrm{EES}_\mathcal{R}(2,5)$ significantly outperforms Reversible Heun, suggesting the model has begun to learn high-volatility dynamics which cause instability in the Reversible Heun method. Whilst $\mathrm{EES}_\mathcal{R}(2,5)$ is generally slower than Reversible Heun per iteration of the solver, its superior stability allow it to achieve a lower MSE in far fewer epochs, resulting in faster training overall.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calibration of a geometric Brownian motion to option prices}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Consider learning the dynamics of a 1-dimensional geometric Brownian motion (GBM)
\begin{equation*}
dS_t = r S_t dt + \sigma S_t dW_t.
\end{equation*}
To simulate a simplified problem of calibration to the risk-neutral dynamics of an asset, suppose that for some fixed unknown parameters $r$ and $\sigma$, we are given a set of undiscounted call prices
\begin{equation*}
C(K, t) = \mathbb{E}[(S_t - K)^+]
\end{equation*}
for $K \in \mathcal{K}$, $t \in \mathcal{T}$, and wish to learn the underlying dynamics using a Neural SDE
\begin{wrapfigure}{r}{5cm}
\vspace{-0.1cm}
\includegraphics[width = 0.4\textwidth]{figures/GBM_mse.pdf}
\captionsetup{oneside,margin={0.5cm,0cm}}
\caption{\textit{Training loss for GBM with $r = 0.5$ and $\sigma = 1.5$.}}
\label{fig:gbm}
\end{wrapfigure}
\begin{equation*}
dS^\theta_t = g(t, S^\theta_t; \theta_g) dt + f(t, S^\theta_t; \theta_f) \circ dW_t,
\end{equation*}
where $g,f$ are neural networks parametrised by $\theta_g,\theta_f$ respectively. For simplicity, we use Stratonovich integration and rely on the Neural SDE to learn the required It\^o correction term. As in the previous example, we increase the difficulty of the integration by considering extreme dynamics with parameters $r = 0.5$ and $\sigma = 1.5$. We choose to parametrise $f,g$ as 3-layer neural networks of width 8 with LipSwish activations. The SDEs are integrated over $t \in [0,25]$ with a step size of 0.25, and the Neural SDE is trained for 250 epochs using the Adam optimiser with an exponentially decaying learning rate initialised at $10^{-2}$ and decayed at a rate of $0.99$. At each epoch, $N=250,000$ realisations of the trained dynamics are sampled and used to compute the (discounted) MSE loss of the call prices,
\begin{align*}
L &:= \frac{1}{|\mathcal{K}|\cdot|\mathcal{T}|} \sum_{k,t \in \mathcal{K} \times \mathcal{T}} e^{-2rt} (C(K,t) - \widehat{C}(K,t))^2\\
\widehat{C}(K,t) &= \frac{1}{N}\sum_{i=1}^N (S^\theta_{i,t} - K)^+
\end{align*}
where $S^\theta_{i,t}$ is the value of the $i^{th}$ sample at time $t$ and we set $\mathcal{K} = \{k \in \mathbb{N} : 90 \leq k \leq 110\}$ and $\mathcal{T} = \{2.5, 5, 7.5, \ldots, 25\}$. Figure \ref{fig:gbm} shows the training loss of the model when Reversible Heun and $\mathrm{EES}_\mathcal{R}(2,5)$ are used as the solver. As with the previous experiment, the methods perform similarly at the start of the training before they encounter extreme dynamics. After this, $\mathrm{EES}_\mathcal{R}(2,5)$ outperforms the Reversible Heun method.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusions, limitations and future work}\label{sec:conclusion}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In this paper, we have introduced the use of Explicit and Effectively Symmetric (EES) schemes as stable integrators for Neural SDEs, and more generally, RDEs. Using the framework of \cite{redmann2020runge}, we have adapted existing EES schemes introduced by~\cite{shmelev2025explicit} for ODEs to the more general setting of RDEs. Through \emph{mean-square stability} analysis, we have shown that EES schemes possess similar stability domains to classical RK3 and RK4 schemes when applied to SDEs. We discussed an efficient algorithm for backpropagation through explicit Runge--Kutta schemes for RDEs, and presented two experiments involving the training of Neural ODEs on extreme SDE dynamics. In both experiments, our $\mathrm{EES}_\mathcal{R}(2,5)$ scheme demonstrated superior stability to the Reversible Heun scheme, resulting in faster training and a lower terminal loss.
EES schemes provide a stable alternative to existing methods such as Reversible Heun, resulting in fast and accurate training when applied to complex dynamics. However, when stability is not an issue, Reversible Heun offers faster integration by requiring only one evaluation of the drift and diffusion per step, as opposed to 3 evaluations required by $\mathrm{EES}_\mathcal{R}$. Addressing this limitation, either through algorithmic changes or the derivation of new EES-type schemes which employ auxiliary variables, is left for future work.
There are several potential extensions to this paper which are left for future research. Applications of EES schemes to more complicated models, including but not limited to Neural Jump SDEs \citep{jia2019neural, herrera2020neural}, Neural CDEs \citep{kidger2020neural} and Neural RDEs \citep{morrill2021neural}, may be of interest. An extension of EES schemes to include partitioned or adaptive step-size schemes would prove valuable for the training of stiff neural differential equations.
% \TODO{Reproducability statement}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%
% \newpage
% %%%%%%%%%%%%%%%%%%%%%%%%%%%
% \begin{figure*}[b!]
% \centering
% \begin{subfigure}[t]{0.85\textwidth}
% \centering
% \includegraphics[width = \textwidth]{figures/rde_example_EES25.pdf}
% \end{subfigure}%
% \caption{EES25 forward, convergence rate \textit{2H - 1/2}}
% \end{figure*}
% \begin{figure*}[b!]
% \centering
% \begin{subfigure}[t]{0.85\textwidth}
% \centering
% \includegraphics[width = \textwidth]{figures/rde_example_EES25_sym.pdf}
% \end{subfigure}%
% \caption{EES25 backward, convergence rate \textit{6H - 1}}
% \end{figure*}
% \begin{figure*}[b!]
% \centering
% \begin{subfigure}[t]{0.85\textwidth}
% \centering
% \includegraphics[width = \textwidth]{figures/rde_example_EES27.pdf}
% \end{subfigure}%
% \caption{EES27 forward, convergence rate \textit{2H - 1/2}}
% \end{figure*}
% \begin{figure*}[b!]
% \centering
% \begin{subfigure}[t]{0.85\textwidth}
% \centering
% \includegraphics[width = \textwidth]{figures/rde_example_EES27_sym.pdf}
% \end{subfigure}%
% \caption{EES25 backward, convergence rate \textit{8H - 1}}
% \end{figure*}
\bibliography{iclr2026_conference}
\bibliographystyle{iclr2026_conference}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Rooted Trees and B-Series}\label{appendix:trees}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{The Connes-Kreimer Hopf Algebra}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We give a brief account of non-planar (labelled) rooted trees and the Connes-Kreimer Hopf algebra. We refer the reader to \cite{hoffman2003hopf} for a comprehensive presentation. A non-planar labelled rooted tree is defined as a graph $\tau = (V, E, r)$ with vertex set $V$, edge set $E$ and a root vertex $r \in V$, together with a set of vertex decorations drawn from $\{1, \ldots, d\}$. We denote the empty tree by $\emptyset$. Given trees $\tau_1, \ldots, \tau_m$, we write $[\tau_1, \ldots, \tau_m]_a$ to denote the tree formed by connecting the root vertices of $\tau_1, \ldots, \tau_m$ to a new root, which receives the label $a \in \{1, \ldots, d\}$. Non-planarity means that the order of the trees in $[\tau_1, \ldots, \tau_m]_a$ is irrelevant. Repeated trees will be denoted using power notation, for instance
\begin{equation*}
[\tau_1,\tau_1, \tau_2, \tau_3, \tau_3, \tau_3]_a = [\tau_1^2, \tau_2, \tau_3^3]_a.
\end{equation*}
We write $|\tau|$ to denote the number of vertices in a tree. Additionally, we define the following combinatorial quantities, defined on unlabelled trees:
\begin{align*}
\emptyset! &= 1, \quad \bullet! = 1, \quad [\tau_1, \ldots, \tau_m]! = |[\tau_1, \ldots, \tau_m]| \prod_{i=1}^m \tau_i!,\\
\sigma(\emptyset) &= 1, \quad \sigma(\bullet) = 1, \quad \sigma([\tau_1^{k_1}, \ldots, \tau_m^{k_m}]) = \prod_{i=1}^m k_i! \sigma(\tau_i)^{k_i},\\
\beta(\emptyset) &= 1, \quad \beta(\bullet) = 1, \quad \beta([\tau_1^{k_1}, \ldots, \tau_m^{k_m}]) = \binom{[\tau_1^{k_1}, \ldots, \tau_m^{k_m}]}{|\tau_1|, \ldots, |\tau_m|}\prod_{i=1}^m \frac{1}{k_i!} \beta(\tau_i)^{k_i}.
\end{align*}
We will refer to the commutative juxtaposition of trees as a forest. We write $\mathcal{T}$ to denote the set of all non-planar labelled rooted trees, and $\mathcal{T}_N \subset \mathcal{T}$ to denote the trees $\tau$ with $|\tau| \leq N$. The free commutative $\mathbb{R}$-algebra generated by $\mathcal{T}$ will be denoted $\mathcal{H}$. The Connes-Kreimer ~\citep{connes1999hopf} Hopf algebra on $\mathcal{H}$ is defined as follows. Multiplication $\mu : \mathcal{H} \otimes \mathcal{H} \to \mathcal{H}$ is defined as the commutative juxtaposition of two forests, extended linearly to $\mathcal{H}$. The multiplicative unit is defined to be the empty forest $\emptyset$. The counit map $\varepsilon: \mathcal{H} \to \mathbb{R}$ is defined by $\varepsilon(\emptyset) = 1$ and $\varepsilon(\tau) = 0$ for all non-empty trees $\tau \in \mathcal{H}$. The coproduct map is defined recursively by
\begin{align*}
\Delta(\emptyset) &= \emptyset \otimes \emptyset,\\
\Delta [\tau_1, \ldots, \tau_m]_a &= [\tau_1, \ldots, \tau_m]_a \otimes \emptyset + (\mathrm{id} \otimes B^a_+)(\Delta \tau_1 \cdots \Delta \tau_m),
\end{align*}
where $B^a_+(\tau_1 \cdots \tau_m) := [\tau_1 \cdots \tau_m]_a$ for a forest $\tau_1 \cdots \tau_m$. The definition is extended to a linear multiplicative map on $\mathcal{H}$. We will occasionally use Sweedler's notation
\begin{equation*}
\Delta \tau = \sum_{(\tau)} \tau^{(1)} \otimes \tau^{(2)}
\end{equation*}
for the coproduct. We omit the definition of the antipode $S$ here, and instead refer the reader to ~\citep{manchon2004hopf, hoffman2003hopf}. We denote the dual of the Connes-Kreimer Hopf algebra by $\mathcal{H}^*$. For $\varphi_1, \varphi_2 \in \mathcal{H}^*$, the convolution product is defined by
\begin{equation*}
\varphi_1 * \varphi_2 = \mu_\mathbb{R} \circ (\varphi_1 \otimes \varphi_2) \circ \Delta,
\end{equation*}
with $\mu_\mathbb{R} : \mathbb{R} \otimes \mathbb{R} \to \mathbb{R}$ denoting multiplication in $\mathbb{R}$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{B-Series Expansions of ODEs}\label{appendix:b_series_ode}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
For any tree $\tau \in \mathcal{T}$, the so-called elementary differential $F(\tau)(y)$ ~\citep{butcher2016numerical} is defined recursively by
\begin{align*}
&F(\emptyset)(y) = y, \quad F(\bullet_i)(y) = f_i(y),\\
&F([\tau_1, \tau_2, \ldots, \tau_m]_i)(y) = f^{(m)}_i(y)(F(\tau_1)(y), F(\tau_2)(y), \ldots, F(\tau_m)(y)).
\end{align*}
Given a map $\varphi: \mathcal{T} \to \mathbb{R}$, the associated B-series is defined
\begin{equation*}
B_h(\varphi, y_0) := \sum_{\tau \in \mathcal{T}} \frac{h^{|\tau|}}{\sigma(\tau)} \varphi(\tau) F(\tau)(y_0).
\end{equation*}
A key property of B-series is their closure under composition. One can show that for two B-series \cite{hairer1974butcher, butcher2024b},
\begin{equation*}
B_h(\varphi_2, B_h(\varphi_1, y_0)) = B_h(\varphi_1 * \varphi_2, y_0),
\end{equation*}
where $\varphi_1 * \varphi_2$ is the convolution product defined above. It can be shown that the exact solution to the ODE in \eqref{eq:ode} admits a B-series representation $y(h) = B_h(e, y_0)$, where $e(\tau) = 1 / \tau!$. Similarly, the solution given by a Runge--Kutta scheme with coefficients $\{a_{ij}\}_{1 \leq i,j \leq s}$, $\{b_i\}_{1 \leq i \leq s}$ admits the B-series representation $B_h(\varphi, y_0)$, where \cite[Lemma 312B]{butcher2016numerical}
\begin{equation*}
\varphi(\tau) := \sum_{i_1, \ldots, i_n} b_{i_1} \prod_{(k,\ell) \in E} a_{i_k, i_\ell}
\end{equation*}
for a tree $\tau = (V,E,r)$ with $|\tau| = n$.\\
We refer the reader to ~\citep{hairer2006geometric,mclachlan2015butcher,butcher2021b} for a detailed account of B-series and the Butcher group.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Branched Rough Paths}\label{appendix:branched_rp}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Let $\mathcal{H}$ be the Connes-Kreimer Hopf algebra of non-planar labelled rooted trees defined above.
\begin{definition}
Let $\alpha \in (0,1]$. An $\alpha$-H\"older branched rough path is a map $\mathbf{X} : [0,T]^2 \to \mathcal{H}^*$ such that
\begin{enumerate}
\item for all $s,t \in [0,T]$ and $\tau_1, \tau_2 \in \mathcal{H}$,
\begin{equation*}
\langle \mathbf{X}_{s,t}, \tau_1 \rangle \langle \mathbf{X}_{s,t}, \tau_2 \rangle = \langle \mathbf{X}_{s,t}, \tau_1 \tau_2 \rangle,
\end{equation*}
\item for all $\tau \in \mathcal{H}$,
\begin{equation*}
\langle \mathbf{X}_{s,t}, \tau \rangle = \sum_{(\tau)} \langle \mathbf{X}_{s,u}, \tau^{(1)} \rangle \langle \mathbf{X}_{u, t}, \tau^{(2)} \rangle,
\end{equation*}
where $\Delta \tau = \sum_{(\tau)} \tau^{(1)} \otimes \tau^{(2)}$.
\item for all $\tau \in \mathcal{H}$,
\begin{equation*}
\sup_{s \neq t} \frac{|\langle \mathbf{X}_{s,t}, \tau \rangle |}{|t - s| ^{\alpha |\tau|}} < \infty.
\end{equation*}
\end{enumerate}
\end{definition}
\begin{remark}
As remarked in ~\citep{hairer2015geometric,gubinelli2010ramification}, the components $\langle \mathbf{X}_{s,t}, \tau \rangle$ with $|\tau| > N$ are determined by those with $|\tau| \leq N$, where $N$ is the largest integer such that $N \alpha \leq 1$.
\end{remark}
The space of $\alpha$-H\"older branched rough paths is a complete metric space under the metric
\begin{equation*}
\varrho_\alpha(\mathbf{X}, \mathbf{Y}) := \sum_{\tau \in \mathcal{T}_N} \sup_{s \neq t} \frac{|\langle \mathbf{X}_{s,t} - \mathbf{Y}_{s,t}, \tau \rangle |}{|t - s|^{\alpha |\tau|}},
\end{equation*}
where $N = \lfloor 1 / \alpha \rfloor$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{B-Series Expansions of RDEs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We recount the results of ~\citep{redmann2020runge} regarding series expansions of the solutions to RDEs of the form \eqref{eq:rde}. Recall the definition of the elementary differential $F(\tau)(y)$ from Appendix \ref{appendix:b_series_ode}.
\begin{theorem}[{\cite[Theorem 2.10]{redmann2020runge}}]
Let $\mathbf{X}$ be an $\alpha$-branched rough path and $h > 0$. Then \eqref{eq:rde} admits the series expansion
\begin{equation*}
y(t_0 + h) = y_0 + \sum_{\tau \in \mathcal{T}_p} \frac{1}{\sigma(\tau)} F(\tau)(y_0) \left<\mathbf{X}_{t_0, t_0 + h}, \tau\right> + \mathcal{O}(h^{(p+1) \alpha})
\end{equation*}
for all $p \geq \lfloor 1 / \alpha \rfloor$.
\end{theorem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{General Runge--Kutta methods for RDEs} \label{appendix:general_rk}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Following ~\cite{burrage1996high, burrage1998general, burrage2000order, redmann2020runge}, consider the class of general Runge-Kutta methods defined by
\begin{align}\label{eq:full_rk}
y_{n+1} &= y_n + \sum_{m=1}^d \sum_{i=1}^s z_i^{(m)} f_m(k_i),\\
k_i &= y_n + \sum_{m=1}^d \sum_{j=1}^s Z^{(m)}_{ij} f_m(k_j),\nonumber
\end{align}
where $Z^{(1)}, \ldots, Z^{(d)} \in \mathbb{R}^{s \times s}$ and $z^{(1)}, \ldots, z^{(d)} \in \mathbb{R}^s$. We briefly recount the local and global error rates for such schemes, as formulated in ~\cite{redmann2020runge}. The results are based on the adaptation of B-series to RDEs presented above.
\begin{definition}
Given $h > 0$, define the maps $a, \varphi$ recursively over non-planar labelled rooted trees $\tau$ by setting $\varphi(\emptyset)(h) := (1,\ldots,1)^T \in \mathbb{R}^s$, where $\emptyset$ denotes the empty tree, and for a tree $\tau = [\tau_1 \cdots \tau_n]_i$ formed by joining $\tau_1, \ldots, \tau_n$ by a new root labelled $i$,
\begin{align*}
\varphi(\tau)(h) &:= \prod_{j=1}^n (Z^{(i)} \varphi(\tau_j)(h)),\\
a(\tau)(h) &:= \left< z^{(i)}, \prod_{j=1}^n \varphi(\tau_j)(h)\right>.
\end{align*}
\end{definition}
\begin{theorem}[{~\citep{redmann2020runge}}]
The general Runge-Kutta method given by \eqref{eq:full_rk} has a local error of order $(p+1)\alpha$ if and only if
\begin{equation*}
\left< \mathbf{X}_{t_0, t_0 + h}, \tau \right> = a(\tau)(h)
\end{equation*}
for all non-planar labelled rooted trees $\tau$ with $p$ or fewer nodes, i.e. $|\tau| \leq p$.
\end{theorem}
\begin{proposition}[{\cite[Proposition 4.1]{redmann2020runge}}]
Let $y(t, y_0)$ denote the solution to \eqref{eq:rde} at time $t$ starting at $y_0$. Suppose the Runge--Kutta method \eqref{eq:full_rk} has a local error of order $(p+1)\alpha$, and there exists a constant $C_1 > 0$ such that
\begin{equation*}
|y(h, y_0) - y(h, \widetilde{y}_0)| \leq C_1 |y_0 - \widetilde{y}_0|,
\end{equation*}
for $h$ sufficiently small. Then there exists $C > 0$ such that
\begin{equation*}
\max_{n = 0, \ldots, N} |y(t_n) - y_n| \leq Ch^{(p+1)\alpha - 1}.
\end{equation*}
\end{proposition}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Convergence of simplified Runge--Kutta methods}\label{appendix:convergence_simplified}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{theorem}[{\cite[Theorem 3.3]{redmann2020runge}}]
Let $\Phi$ be an ODE Runge--Kutta scheme. The Runge-Kutta method $\mathcal{R}(\Phi)$ approximating $y^h$ has a local error of order $(p+1)\alpha$, i.e.
\begin{equation*}
y^h(t_0 + h) - y_1^h = \mathcal{O}(h^{(p+1)\alpha}),
\end{equation*}
if and only if the ODE Runge--Kutta method $\Phi$ is of order $p$.
\end{theorem}
\begin{theorem}[{\cite[Theorem 4.2]{redmann2020runge}}]
Let $\Phi$ be an ODE Runge--Kutta method of order $p$. Suppose that $f$ is $\text{Lip}^\gamma_b$ for some $\gamma > 1/\alpha$. Then $\mathcal{R}(\Phi)$ has a global error rate of $\eta = \min\{r_0, (p+1)\alpha - 1\}$, where $r_0$ is the convergence rate of the Wong-Zakai approximation. That is,
\begin{equation*}
\max_{n = 0, \ldots N} |y(t_n) - y^h_n| = \mathcal{O}(h^\eta).
\end{equation*}
\end{theorem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Convergence of \texorpdfstring{$\mathrm{EES}_\mathcal{R}$ schemes}{TEXT}}\label{appendix:convergence}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% An immediate consequence of the definition is that the map $\mathcal{R}(\cdot)$ is multiplicative.
% \begin{proposition}\label{prop:r_mult}
% Let $\Phi_1, \Phi_2$ be ODE Runge--Kutta methods. Then $\mathcal{R}(\Phi_1 \circ \Phi_2) = \mathcal{R}(\Phi_1) \circ \mathcal{R}(\Phi_2)$.
% \end{proposition}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \subsection{Proof of Theorem \ref{thm:ees_local}}\label{appendix:ees_local_proof}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \TODO{1. Again, the first proof below relies on \cite[Theorem 3.3]{redmann2020runge} holding true for all $p$.}
% \TODO{2. These proofs are quite short, and largely follow from existing results. Do we move them to the main text or keep them here?}
% \begin{proof}
% Since $\Phi$ is of order $n$, it follows from Theorem \ref{thm:simple_local} that $\mathcal{R}(\Phi)$ has a local error of order $(n+1)\alpha$. Consider the scheme $\mathcal{R}(\backvec{\Phi}) \circ \mathcal{R}(\Phi) = \mathcal{R}(\backvec{\Phi} \circ \Phi)$. By definition of $\Phi$, the scheme $\backvec{\Phi} \circ \Phi$ admits a B-series expansion $y_0 + \mathcal{O}(h^{m+1})$ ~\citep{shmelev2025explicit}. It follows from ~\citep{redmann2020runge} that the corresponding B-series expansion for $\mathcal{R}(\backvec{\Phi} \circ \Phi)$ is $y_0 + \mathcal{O}(h^{(m+1)\alpha})$.
% \end{proof}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \subsection{Proof of Theorem \ref{thm:ees_global}}\label{appendix:ees_global_proof}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \begin{proof}
% Since $\Phi$ is of order $n$, it follows from Theorem \ref{thm:simple_global} that $\mathcal{R}(\Phi)$ has a global error of order $\eta_1 = \min\{r_0, (n+1)\alpha - 1\}$. The global error rate for the recovery of the initial condition $y_0$ from $\backvec{y}_i$ follows immediately from \cite[Proposition 4.1]{redmann2020runge}.
% \end{proof}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \subsection{Experiments}\label{appendix:fbm_rates}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We verify the global error rates given in Theorem \ref{thm:ees_global} experimentally by reproducing the example given in ~\citep{redmann2020runge, deya2012milstein} for $\mathrm{EES}_\mathcal{R}(2,5; 1/10)$ and $\mathrm{EES}_\mathcal{R}(2,5; (5 - 3\sqrt{2}) / 14)$. We take the RDE
\begin{equation*}
dy_t = \cos(y_t) d\mathbf{X}^{(1)}_t + \sin(y_t)d\mathbf{X}^{(2)}_t, \quad y_0 = 1
\end{equation*}
for $t \in [0,1]$, where $\mathbf{X}$ is the geometric lift pf a 2-dimensional fractional Brownian motion (fBm) with hurst index $H$. We compute the average of the maximal discretization error over $M = 10$ realisations of the RDE,
\begin{equation*}
\mathcal{E}(h) := \frac{1}{M}\sum_{i=1}^M \max_{n=0,\ldots, N} |y_i(t_n) - y_{i,n}|,
\end{equation*}
where $y_i(t)$ denotes the solution to the $i^{th}$ realisation of the RDE and $y_{i,n}$ denotes the discretisation of the $i^{th}$ solution using an $\mathrm{EES}_\mathcal{R}$ scheme. Additionally, we evaluate the average error when recovering the initial condition,
\begin{equation*}
\backvec{\mathcal{E}}(h) := \frac{1}{M}\sum_{i=1}^M |y_0 - \backvec{y}_{i,n}|.
\end{equation*}
From ~\citep{friz2014convergence}, the rate $r_0$ can be chosen arbitrarily close to $2H - 1/2$ for a fractional Brownian motion with Hurst parameter $H$. It follows that we expect $\eta_1 = 2H - 1/2$ in Theorem \ref{thm:ees_global} for both $\mathrm{EES}(2,5)$ and $\mathrm{EES}(2,7)$, and $\eta_2 = 6H-1$ for $\mathrm{EES}(2,5)$ and $\eta_2 = 8H-1$ for $\mathrm{EES}(2,7)$. These rates are shown in Figures \ref{fig:convergence_4}, \ref{fig:convergence_5} and \ref{fig:convergence_6} for $H = 0.4, 0.5$ and $0.6$ respectively.
\begin{figure}[H]
\centering
\includegraphics[width = 0.7\textwidth]{figures/rde_example_ees40.pdf}
\caption{\textit{Convergence rates for $H = 0.4$}}
\label{fig:convergence_4}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width = 0.7\textwidth]{figures/rde_example_ees50.pdf}
\caption{\textit{Convergence rates for $H = 0.5$}}
\label{fig:convergence_5}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width = 0.7\textwidth]{figures/rde_example_ees60.pdf}
\caption{\textit{Convergence rates for $H = 0.6$}}
\label{fig:convergence_6}
\end{figure}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
%Adapted from:
%https://tex.stackexchange.com/questions/364824/drawing-tree-like-symbols
\usepackage{tikz}
\usetikzlibrary{arrows,calc,positioning}
\makeatletter
\newcommand\RSloop{\@ifnextchar\bgroup\RSloopa\RSloopb}
\makeatother
\newcommand\RSloopa[1]{\bgroup\RSloop#1\relax\egroup\RSloop}
\newcommand\RSloopb[1]%
{\ifx\relax#1%
\else
\ifcsname RS:#1\endcsname
\csname RS:#1\endcsname
\else
\GenericError{(RS)}{RS Error: operator #1 undefined}{}{}%
\fi
\expandafter\RSloop
\fi
}
\newcommand\X{0}
\newcommand\RS[1]%
{\begin{tikzpicture}
[every node/.style=
{circle,draw,fill,minimum size=3pt,inner sep=0pt,outer sep=0pt},
line cap=round
]
\coordinate(\X) at (0,0);
\RSloop{#1}\relax
\end{tikzpicture}
}
\newcommand\RSdef[1]{\expandafter\def\csname RS:#1\endcsname}
\newlength\RSu
\RSu=2ex
\RSdef{n}{\draw (\X) node{};}
\RSdef{i}{\draw (\X) node{} -- + (90:\RSu) node{};}
\RSdef{d}{\draw[dashed] (\X) node{} -- + (90:\RSu) node{};}
\RSdef{l}{\draw (\X) node{} -- +(135:\RSu) node{};}
\RSdef{r}{\draw (\X) node{} -- +(45:\RSu) node{};}
\RSdef{I}{\draw (\X) node{} -- +(90:\RSu) coordinate(\X I);\edef\X{\X I}}
\RSdef{L}{\draw (\X) node{} -- +(135:\RSu) coordinate(\X L);\edef\X{\X L}}
\RSdef{R}{\draw (\X) node{} -- +(45:\RSu) coordinate(\X R);\edef\X{\X R}}
\RSdef{t}<1>{\draw (\X) node{};}
\newcommand{\treeOne}{\begin{tikzpicture}
\coordinate (root) at (0, 0);
\filldraw (root) circle [radius=1.5pt];
\coordinate (left) at ($(root) + (130:2ex)$);
\node[] at ($(left) + (0,1ex)$) {\small $\tau_1$};
\coordinate (right) at ($(root) + (50:2ex)$);
\node[] at ($(right) + (0,1ex)$) {\small $\tau_2$};
\draw [thin] (left) to (root) to (right);
\end{tikzpicture}}
\newcommand{\treeTwo}{\begin{tikzpicture}
\coordinate (root) at (0, 0);
\filldraw (root) circle [radius=1.5pt];
\coordinate (top) at ($(root) + (0,2ex)$);
\node[] at ($(top) + (0,1ex)$) {\small $\tau_1$};
\draw [thin] (root) to (top);
\end{tikzpicture}}
\newcommand{\treeThree}{\begin{tikzpicture}
\coordinate (root) at (0, 0);
\filldraw (root) circle [radius=1.5pt];
\coordinate (node1) at ($(root) + (0,2ex)$);
\filldraw (node1) circle [radius=1.5pt];
\draw [thin] (root) to (node1);
\coordinate (node2) at ($(node1) + (50:2ex)$);
\filldraw (node2) circle [radius=1.5pt];
\draw [thin] (node1) to (node2);
\coordinate (node3) at ($(node2) + (50:2ex)$);
\filldraw (node3) circle [radius=1.5pt];
\draw [thin] (node2) to (node3);
\coordinate (node4) at ($(node3) + (50:4ex)$);
\filldraw (node4) circle [radius=1.5pt];
\draw [densely dotted] (node3) to (node4);
\coordinate (node5) at ($(node4) + (50:2ex)$);
\filldraw (node5) circle [radius=1.5pt];
\draw [thin] (node4) to (node5);
%labels
\coordinate (node11) at ($(node1) + (130:2ex)$);
\draw [thin] (node1) to (node11);
\node[] at ($(node11) + (-1ex,1ex)$) {\small $\tau_1$};
\coordinate (node21) at ($(node2) + (130:2ex)$);
\draw [thin] (node2) to (node21);
\node[] at ($(node21) + (-1ex,1ex)$) {\small $\tau_2$};
\coordinate (node31) at ($(node3) + (130:2ex)$);
\draw [thin] (node3) to (node31);
\node[] at ($(node31) + (-1ex,1ex)$) {\small $\tau_3$};
\coordinate (node41) at ($(node4) + (130:2ex)$);
\draw [thin] (node4) to (node41);
\node[] at ($(node41) + (-1ex,1ex)$) {\small $\tau_s$};
\end{tikzpicture}}
\newcommand{\commonRootTree}{\begin{tikzpicture}
\coordinate (root) at (0, 0);
\filldraw (root) circle [radius=1.5pt];
\coordinate (n1) at ($(root) + (159:4.3ex)$);
\node[] at ($(n1) + (0,1ex)$) {\footnotesize $\tau_1$};
\coordinate (n2) at ($(root) + (130:2ex)$);
\node[] at ($(n2) + (0,1ex)$) {\footnotesize $\tau_2$};
\coordinate (n3) at ($(root) + (50:2ex)$);
\node[] at ($(n3) + (0,1ex)$) {\footnotesize $\cdots$};
\coordinate (n4) at ($(root) + (22:4.1ex)$);
\node[] at ($(n4) + (0,1ex)$) {\footnotesize $\tau_k$};
\draw [thin] (n1) to (root);
\draw [thin] (n2) to (root);
\draw [thin] (n4) to (root);
\end{tikzpicture}}
\newcommand{\treeFour}{
\begin{tikzpicture}
\coordinate (root) at (0, 0);
\filldraw (root) circle [radius=1.5pt];
\coordinate (left) at ($(root) + (130:5ex)$);
\filldraw (left) circle [radius=1.5pt];
\coordinate (right) at ($(root) + (50:5ex)$);
\filldraw (right) circle [radius=1.5pt];
\coordinate (rl) at ($(right) + (130:5ex)$);
\filldraw (rl) circle [radius=1.5pt];
\coordinate (rr) at ($(right) + (50:5ex)$);
\filldraw (rr) circle [radius=1.5pt];
\draw [thin] (left) to (root) to (right);
\draw [thin] (rl) to (right) to (rr);
\draw [thin] (rl) to (right) to (rr);
\node[] at ($(root) + (0ex,-2ex)$) {\small $b_i$};
\node[] at ($(left) + (-0.1ex,-2.5ex)$) {\small $a_{ik}$};
\node[] at ($(right) + (0.1ex,-2.5ex)$) {\small $a_{ij}$};
\node[] at ($(rl) + (-0.1ex,-2.5ex)$) {\small $a_{j\ell}$};
\node[] at ($(rr) + (0.6ex,-2.5ex)$) {\small $a_{jm}$};
\end{tikzpicture}}
\newcommand{\treeFive}{
\begin{tikzpicture}
\coordinate (root) at (0, 0);
\filldraw (root) circle [radius=1.5pt];
\coordinate (left) at ($(root) + (130:5ex)$);
\filldraw (left) circle [radius=1.5pt];
\coordinate (right) at ($(root) + (50:5ex)$);
\filldraw (right) circle [radius=1.5pt];
\coordinate (rl) at ($(right) + (130:5ex)$);
\filldraw (rl) circle [radius=1.5pt];
\coordinate (rr) at ($(right) + (50:5ex)$);
\filldraw (rr) circle [radius=1.5pt];
\draw [thin] (left) to (root);
\draw [thin] (rl) to (right) to (rr);
\draw [thin] (rl) to (right) to (rr);
\node[] at ($(root) + (0ex,-2ex)$) {\small $b_i$};
\node[] at ($(left) + (-0.1ex,-2.5ex)$) {\small $a_{ik}$};
\node[] at ($(right) + (0ex,-2ex)$) {\small $b_j$};
\node[] at ($(rl) + (-0.1ex,-2.5ex)$) {\small $a_{j\ell}$};
\node[] at ($(rr) + (0.6ex,-2.5ex)$) {\small $a_{jm}$};
\end{tikzpicture}}
\newcommand{\treeSix}{\begin{tikzpicture}
\coordinate (right) at (0, 0);
\filldraw (right) circle [radius=1.5pt];
\coordinate (rl) at ($(right) + (130:5ex)$);
\filldraw (rl) circle [radius=1.5pt];
\coordinate (rr) at ($(right) + (50:5ex)$);
\filldraw (rr) circle [radius=1.5pt];
% \draw [thin] (left) to (root);
\draw [thin] (rl) to (right) to (rr);
\draw [thin] (rl) to (right) to (rr);
\node[] at ($(right) + (0ex,-2ex)$) {\small $b_j$};
\node[] at ($(rl) + (-0.1ex,-2.5ex)$) {\small $a_{j\ell}$};
\node[] at ($(rr) + (0.3ex,-2.5ex)$) {\small $a_{jm}$};
\coordinate (newleft) at ($(right) + (160:10ex)$);
\filldraw (newleft) circle [radius=1.5pt];
\draw [thin] (right) to[out=180,in=270] (newleft);
\coordinate (ll) at ($(newleft) + (90:5ex)$);
\filldraw (ll) circle [radius=1.5pt];
\draw [thin] (newleft) to (ll);
\node[] at ($(newleft) + (3ex,-4.2ex)$) {\small $a_{ji}$};
\node[] at ($(ll) + (-2ex,-2.5ex)$) {\small $a_{ji}$};
\end{tikzpicture}}
\begin{thebibliography}{61}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Arribas et~al.(2020)Arribas, Salvi, and Szpruch]{arribas2020sigsdes}
Imanol~Perez Arribas, Cristopher Salvi, and Lukasz Szpruch.
\newblock Sig-sdes model for quantitative finance.
\newblock In \emph{ACM International Conference on AI in Finance}, 2020.
\bibitem[Barancikova et~al.(2024)Barancikova, Huang, and Salvi]{barancikovasigdiffusions}
Barbora Barancikova, Zhuoyue Huang, and Cristopher Salvi.
\newblock Sigdiffusions: Score-based diffusion models for time series via log-signature embeddings.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2024.
\bibitem[Burrage \& Burrage(2000)Burrage and Burrage]{burrage2000order}
Kevin Burrage and Pamela~M Burrage.
\newblock Order conditions of stochastic runge--kutta methods by b-series.
\newblock \emph{SIAM Journal on Numerical Analysis}, 38\penalty0 (5):\penalty0 1626--1646, 2000.
\bibitem[Burrage \& Burrage(1996)Burrage and Burrage]{burrage1996high}
Kevin Burrage and Pamela~Marion Burrage.
\newblock High strong order explicit runge-kutta methods for stochastic ordinary differential equations.
\newblock \emph{Applied Numerical Mathematics}, 22\penalty0 (1-3):\penalty0 81--101, 1996.
\bibitem[Burrage \& Burrage(1998)Burrage and Burrage]{burrage1998general}
Kevin Burrage and PM~Burrage.
\newblock General order conditions for stochastic runge-kutta methods for both commuting and non-commuting stochastic ordinary differential equation systems.
\newblock \emph{Applied Numerical Mathematics}, 28\penalty0 (2-4):\penalty0 161--177, 1998.
\bibitem[Butcher et~al.(2016)Butcher, Hill, and Norton]{butcher2016symmetric}
JC~Butcher, AT~Hill, and TJT Norton.
\newblock Symmetric general linear methods.
\newblock \emph{BIT Numerical Mathematics}, 56:\penalty0 1189--1212, 2016.
\bibitem[Butcher(2021)]{butcher2021b}
John~C Butcher.
\newblock \emph{B-series: algebraic analysis of numerical methods}, volume~55.
\newblock Springer, 2021.
\bibitem[Butcher et~al.(2024)Butcher, Mitsui, Miyatake, and Sato]{butcher2024b}
John~C Butcher, Taketomo Mitsui, Yuto Miyatake, and Shun Sato.
\newblock On the {B}-series composition theorem.
\newblock \emph{arXiv preprint arXiv:2409.08533}, 2024.
\bibitem[Butcher(2016)]{butcher2016numerical}
John~Charles Butcher.
\newblock \emph{Numerical methods for ordinary differential equations}.
\newblock John Wiley \& Sons, 2016.
\bibitem[Cass \& Salvi(2024)Cass and Salvi]{cass_salvi_notes}
Thomas Cass and Cristopher Salvi.
\newblock {Lecture Notes on Rough Paths and Applications to Machine Learning}, 2024.
\bibitem[Cirone et~al.(2023)Cirone, Lemercier, and Salvi]{cirone2023neural}
Nicola~Muca Cirone, Maud Lemercier, and Cristopher Salvi.
\newblock Neural signature kernels as infinite-width-depth-limits of controlled resnets.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 25358--25425. PMLR, 2023.
\bibitem[Connes \& Kreimer(1999)Connes and Kreimer]{connes1999hopf}
Alain Connes and Dirk Kreimer.
\newblock Hopf algebras, renormalization and noncommutative geometry.
\newblock In \emph{Quantum field theory: perspective and prospective}, pp.\ 59--109. Springer, 1999.
\bibitem[Deya et~al.(2012)Deya, Neuenkirch, and Tindel]{deya2012milstein}
Aur{\'e}lien Deya, Andreas Neuenkirch, and Samy Tindel.
\newblock A milstein-type scheme without l{\'e}vy area terms for sdes driven by fractional brownian motion.
\newblock In \emph{Annales de l'IHP Probabilit{\'e}s et statistiques}, volume~48, pp.\ 518--550, 2012.
\bibitem[Drummond \& Mortimer(1991)Drummond and Mortimer]{drummond1991computer}
PD~Drummond and IK~Mortimer.
\newblock Computer simulations of multiplicative stochastic differential equations.
\newblock \emph{Journal of computational physics}, 93\penalty0 (1):\penalty0 144--170, 1991.
\bibitem[Fermanian et~al.(2023)Fermanian, Lyons, Morrill, and Salvi]{fermanian2023new}
Adeline Fermanian, Terry Lyons, James Morrill, and Cristopher Salvi.
\newblock New directions in the applications of rough path theory.
\newblock \emph{IEEE BITS the Information Theory Magazine}, 3\penalty0 (2):\penalty0 41--53, 2023.
\bibitem[Friz \& Riedel(2014)Friz and Riedel]{friz2014convergence}
Peter Friz and Sebastian Riedel.
\newblock Convergence rates for the full gaussian rough paths.
\newblock In \emph{Annales de l'IHP Probabilit{\'e}s et statistiques}, volume~50, pp.\ 154--194, 2014.
\bibitem[Friz \& Victoir(2010)Friz and Victoir]{friz2010multidimensional}
Peter~K Friz and Nicolas~B Victoir.
\newblock \emph{Multidimensional stochastic processes as rough paths: theory and applications}, volume 120.
\newblock Cambridge University Press, 2010.
\bibitem[Gubinelli(2004)]{gubinelli2004controlling}
Massimiliano Gubinelli.
\newblock Controlling rough paths.
\newblock \emph{Journal of Functional Analysis}, 216\penalty0 (1):\penalty0 86--140, 2004.
\bibitem[Gubinelli(2010)]{gubinelli2010ramification}
Massimiliano Gubinelli.
\newblock Ramification of rough paths.
\newblock \emph{Journal of Differential Equations}, 248\penalty0 (4):\penalty0 693--721, 2010.
\bibitem[Hairer \& Wanner(1974)Hairer and Wanner]{hairer1974butcher}
E.~Hairer and G.~Wanner.
\newblock On the {B}utcher group and general multi-value methods.
\newblock \emph{Computing (Arch. Elektron. Rechnen)}, 13\penalty0 (1):\penalty0 1--15, 1974.
\newblock ISSN 0010-485X,1436-5057.
\newblock \doi{10.1007/bf02268387}.
\newblock URL \url{https://doi.org/10.1007/bf02268387}.
\bibitem[Hairer et~al.(2006)Hairer, Hochbruck, Iserles, and Lubich]{hairer2006geometric}
Ernst Hairer, Marlis Hochbruck, Arieh Iserles, and Christian Lubich.
\newblock Geometric numerical integration.
\newblock \emph{Oberwolfach Reports}, 3\penalty0 (1):\penalty0 805--882, 2006.
\bibitem[Hairer \& Kelly(2015)Hairer and Kelly]{hairer2015geometric}
Martin Hairer and David Kelly.
\newblock Geometric versus non-geometric rough paths.
\newblock In \emph{Annales de l'IHP Probabilit{\'e}s et statistiques}, volume~51, pp.\ 207--251, 2015.
\bibitem[Hernandez \& Spigler(1993)Hernandez and Spigler]{hernandez1993convergence}
Diego~Bricio Hernandez and Renato Spigler.
\newblock Convergence and stability of implicit runge-kutta methods for systems with multiplicative noise.
\newblock \emph{BIT Numerical Mathematics}, 33\penalty0 (4):\penalty0 654--669, 1993.
\bibitem[Herrera et~al.(2020)Herrera, Krach, and Teichmann]{herrera2020neural}
Calypso Herrera, Florian Krach, and Josef Teichmann.
\newblock Neural jump ordinary differential equations: Consistent continuous-time prediction and filtering.
\newblock \emph{arXiv preprint arXiv:2006.04727}, 2020.
\bibitem[Higham(2000)]{higham2000mean}
Desmond~J Higham.
\newblock Mean-square and asymptotic stability of the stochastic theta method.
\newblock \emph{SIAM journal on numerical analysis}, 38\penalty0 (3):\penalty0 753--769, 2000.
\bibitem[Hoffman(2003)]{hoffman2003hopf}
Michael~E. Hoffman.
\newblock Combinatorics of rooted trees and {H}opf algebras.
\newblock \emph{Trans. Amer. Math. Soc.}, 355\penalty0 (9):\penalty0 3795--3811, 2003.
\newblock ISSN 0002-9947,1088-6850.
\bibitem[Holberg \& Salvi(2024)Holberg and Salvi]{holberg2024exact}
Christian Holberg and Cristopher Salvi.
\newblock Exact gradients for stochastic spiking neural networks driven by rough signals.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 31907--31939, 2024.
\bibitem[Issa et~al.(2023)Issa, Horvath, Lemercier, and Salvi]{issa2023non}
Zacharia Issa, Blanka Horvath, Maud Lemercier, and Cristopher Salvi.
\newblock Non-adversarial training of neural sdes with signature kernel scores.
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 11102--11126, 2023.
\bibitem[Jia \& Benson(2019)Jia and Benson]{jia2019neural}
Junteng Jia and Austin~R Benson.
\newblock Neural jump stochastic differential equations.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Kidger et~al.(2019)Kidger, Bonnier, Perez~Arribas, Salvi, and Lyons]{kidger2019deep}
Patrick Kidger, Patric Bonnier, Imanol Perez~Arribas, Cristopher Salvi, and Terry Lyons.
\newblock Deep signature transforms.
\newblock \emph{Advances in neural information processing systems}, 32, 2019.
\bibitem[Kidger et~al.(2020)Kidger, Morrill, Foster, and Lyons]{kidger2020neural}
Patrick Kidger, James Morrill, James Foster, and Terry Lyons.
\newblock Neural controlled differential equations for irregular time series.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 6696--6707, 2020.
\bibitem[Kidger et~al.(2021{\natexlab{a}})Kidger, Foster, Li, and Lyons]{kidger2021neural}
Patrick Kidger, James Foster, Xuechen Li, and Terry~J Lyons.
\newblock Neural sdes as infinite-dimensional gans.
\newblock In \emph{International conference on machine learning}, pp.\ 5453--5463. PMLR, 2021{\natexlab{a}}.
\bibitem[Kidger et~al.(2021{\natexlab{b}})Kidger, Foster, Li, and Lyons]{kidger2021efficient}
Patrick Kidger, James Foster, Xuechen~Chen Li, and Terry Lyons.
\newblock Efficient and accurate gradients for neural sdes.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 18747--18761, 2021{\natexlab{b}}.
\bibitem[Kir{\'a}ly \& Oberhauser(2019)Kir{\'a}ly and Oberhauser]{kiraly2019kernels}
Franz~J Kir{\'a}ly and Harald Oberhauser.
\newblock Kernels for sequentially ordered data.
\newblock \emph{Journal of Machine Learning Research}, 20\penalty0 (31):\penalty0 1--45, 2019.
\bibitem[Komori et~al.(1994)Komori, Saito, and Mitsui]{komori1994some}
Y~Komori, Y~Saito, and T~Mitsui.
\newblock Some issues in discrete approximate solution for stochastic differential equations.
\newblock \emph{Computers \& Mathematics with Applications}, 28\penalty0 (10-12):\penalty0 269--278, 1994.
\bibitem[Komori \& Mitsui(1995)Komori and Mitsui]{komori1995stahle}
Yoshio Komori and Taketomo Mitsui.
\newblock Stable row-type weak scheme for stochastic differential equations.
\newblock \emph{Monte Carlo Methods and Applications}, 1:\penalty0 279--300, 01 1995.
\bibitem[Lemercier et~al.(2021)Lemercier, Salvi, Damoulas, Bonilla, and Lyons]{lemercier2021distribution}
Maud Lemercier, Cristopher Salvi, Theodoros Damoulas, Edwin Bonilla, and Terry Lyons.
\newblock Distribution regression for sequential data.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, pp.\ 3754--3762. PMLR, 2021.
\bibitem[Lemercier et~al.(2024)Lemercier, Lyons, and Salvi]{lemercier2024log}
Maud Lemercier, Terry Lyons, and Cristopher Salvi.
\newblock Log-pde methods for rough signature kernels.
\newblock \emph{arXiv preprint arXiv:2404.02926}, 2024.
\bibitem[Li et~al.(2020)Li, Wong, Chen, and Duvenaud]{li2020scalable}
Xuechen Li, Ting-Kam~Leonard Wong, Ricky~TQ Chen, and David~K Duvenaud.
\newblock Scalable gradients and variational inference for stochastic differential equations.
\newblock In \emph{Symposium on Advances in Approximate Bayesian Inference}, pp.\ 1--28. PMLR, 2020.
\bibitem[Lyons(1998)]{lyons1998differential}
Terry~J Lyons.
\newblock Differential equations driven by rough signals.
\newblock \emph{Revista Matem{\'a}tica Iberoamericana}, 14\penalty0 (2):\penalty0 215--310, 1998.
\bibitem[Manchon(2004)]{manchon2004hopf}
Dominique Manchon.
\newblock Hopf algebras, from basics to applications to renormalization.
\newblock \emph{arXiv preprint math/0408405}, 2004.
\bibitem[McCallum \& Foster(2024)McCallum and Foster]{mccallum2024efficient}
Sam McCallum and James Foster.
\newblock Efficient, accurate and stable gradients for neural odes.
\newblock \emph{arXiv preprint arXiv:2410.11648}, 2024.
\bibitem[McLachlan et~al.(2015)McLachlan, Modin, Munthe-Kaas, and Verdier]{mclachlan2015butcher}
Robert~I McLachlan, Klas Modin, Hans Munthe-Kaas, and Olivier Verdier.
\newblock Butcher series: a story of rooted trees and numerical methods for evolution equations.
\newblock \emph{arXiv preprint arXiv:1512.00906}, 2015.
\bibitem[Morrill et~al.(2021)Morrill, Salvi, Kidger, and Foster]{morrill2021neural}
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster.
\newblock Neural rough differential equations for long time series.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 7829--7838. PMLR, 2021.
\bibitem[Mu\c{c}a~Cirone \& Salvi(2025{\natexlab{a}})Mu\c{c}a~Cirone and Salvi]{cirone2025parallelflow}
Nicola Mu\c{c}a~Cirone and Cristopher Salvi.
\newblock Parallelflow: Parallelizing linear transformers via flow discretization, 2025{\natexlab{a}}.
\bibitem[Mu\c{c}a~Cirone \& Salvi(2025{\natexlab{b}})Mu\c{c}a~Cirone and Salvi]{cirone2025rough}
Nicola Mu\c{c}a~Cirone and Cristopher Salvi.
\newblock Rough kernel hedging, 2025{\natexlab{b}}.
\bibitem[Mu\c{c}a~Cirone et~al.(2024)Mu\c{c}a~Cirone, Orvieto, Walker, Salvi, and Lyons]{cirone2024theoretical}
Nicola Mu\c{c}a~Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons.
\newblock Theoretical foundations of deep selective state-space models, 2024.
\bibitem[Oh et~al.(2024)Oh, Lim, and Kim]{oh2024stable}
YongKyung Oh, Dong-Young Lim, and Sungil Kim.
\newblock Stable neural stochastic differential equations in analyzing irregular time series data.
\newblock \emph{arXiv preprint arXiv:2402.14989}, 2024.
\bibitem[Pannier \& Salvi(2024)Pannier and Salvi]{pannier2024path}
Alexandre Pannier and Cristopher Salvi.
\newblock A path-dependent pde solver based on signature kernels, 2024.
\bibitem[Petersen(1998)]{petersen1998general}
WP~Petersen.
\newblock A general implicit splitting for stabilizing numerical simulations of it{\^o} stochastic differential equations.
\newblock \emph{SIAM journal on numerical analysis}, 35\penalty0 (4):\penalty0 1439--1451, 1998.
\bibitem[Redmann \& Riedel(2020)Redmann and Riedel]{redmann2020runge}
Martin Redmann and Sebastian Riedel.
\newblock Runge-kutta methods for rough differential equations.
\newblock \emph{arXiv preprint arXiv:2003.12626}, 2020.
\bibitem[Redmann \& Riedel(2022)Redmann and Riedel]{redmann2022runge}
Martin Redmann and Sebastian Riedel.
\newblock Runge-kutta methods for rough differential equations.
\newblock \emph{Journal of Stochastic Analysis}, 3\penalty0 (4):\penalty0 6, 2022.
\bibitem[Saito \& Mitsui(1993)Saito and Mitsui]{saito1993t}
Yoshihiro Saito and Taketomo Mitsui.
\newblock T-stability of numerial scheme for stochastic differential equations.
\newblock In \emph{Contributions in numerical mathematics}, pp.\ 333--344. World Scientific, 1993.
\bibitem[Saito \& Mitsui(1996)Saito and Mitsui]{saito1996stability}
Yoshihiro Saito and Taketomo Mitsui.
\newblock Stability analysis of numerical schemes for stochastic differential equations.
\newblock \emph{SIAM Journal on Numerical Analysis}, 33\penalty0 (6):\penalty0 2254--2267, 1996.
\bibitem[Salvi et~al.(2021{\natexlab{a}})Salvi, Cass, Foster, Lyons, and Yang]{salvi2021signature}
Cristopher Salvi, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang.
\newblock The signature kernel is the solution of a goursat pde.
\newblock \emph{SIAM Journal on Mathematics of Data Science}, 3\penalty0 (3):\penalty0 873--899, 2021{\natexlab{a}}.
\bibitem[Salvi et~al.(2021{\natexlab{b}})Salvi, Lemercier, Liu, Horvath, Damoulas, and Lyons]{salvi2021higher}
Cristopher Salvi, Maud Lemercier, Chong Liu, Blanka Horvath, Theodoros Damoulas, and Terry Lyons.
\newblock Higher order kernel mean embeddings to capture filtrations of stochastic processes.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 16635--16647, 2021{\natexlab{b}}.
\bibitem[Schurz(1996)]{schurz1996asymptotical}
Henri Schurz.
\newblock Asymptotical mean square stability of an equilibrium point of some linear numerical solutions with multiplicative noise.
\newblock \emph{Stochastic Analysis and Applications}, 14\penalty0 (3):\penalty0 313--353, 1996.
\bibitem[Shmelev et~al.(2025)Shmelev, Ebrahimi-Fard, Tapia, and Salvi]{shmelev2025explicit}
Daniil Shmelev, Kurusch Ebrahimi-Fard, Nikolas Tapia, and Cristopher Salvi.
\newblock Explicit and effectively symmetric runge-kutta methods.
\newblock \emph{arXiv preprint arXiv:2507.21006}, 2025.
\bibitem[Walker et~al.(2025)Walker, Yang, Cirone, Salvi, and Lyons]{walker2025structured}
Benjamin Walker, Lingyi Yang, Nicola~Muca Cirone, Cristopher Salvi, and Terry Lyons.
\newblock Structured linear cdes: Maximally expressive and parallel-in-time sequence models.
\newblock \emph{arXiv preprint arXiv:2505.17761}, 2025.
\bibitem[Zhang \& Chen(2021)Zhang and Chen]{zhang2021path}
Qinsheng Zhang and Yongxin Chen.
\newblock Path integral sampler: a stochastic control approach for sampling.
\newblock \emph{arXiv preprint arXiv:2111.15141}, 2021.
\bibitem[Zhuang et~al.(2021)Zhuang, Dvornek, Tatikonda, and Duncan]{zhuang2021mali}
Juntang Zhuang, Nicha~C Dvornek, Sekhar Tatikonda, and James~S Duncan.
\newblock Mali: A memory efficient and reverse accurate integrator for neural odes.
\newblock \emph{arXiv preprint arXiv:2102.04668}, 2021.
\end{thebibliography}