Estimation-Prediction Tradeoff in Causal Probabilistic Temporal Graphs
Extracted source text ยท arXiv 2606.28225
%%%%%%%% ICML 2023 WORKSHOP SUBMISSION %%%%%%%%%%%%%%%%%
\documentclass{article}
\input{math_commands}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage[none]{hyphenat}
\usepackage[capitalise]{cleveref}
\crefname{equation}{}{}
\Crefname{equation}{}{}
\usepackage[accepted]{icml2023}
\icmltitlerunning{Estimation--Prediction Tradeoff in Causal Probabilistic Temporal Graphs}
\begin{document}
\twocolumn[
\icmltitle{
Estimation--Prediction Tradeoff in Causal Probabilistic Temporal Graphs}
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Aniq Ur Rahman}{ox}
% \icmlauthor{Justin P. Coon}{ox}
\end{icmlauthorlist}
\icmlaffiliation{ox}{Department of Engineering Science, University of Oxford, Oxford, OX1 3PJ, UK.}
\icmlcorrespondingauthor{Aniq Ur Rahman}{aniq.rahman@eng.ox.ac.uk}
\icmlkeywords{binary classification, Fisher information, entropy, hypothesis testing, foundational limits}
\vskip 0.3in
]
\printAffiliationsAndNotice{}
\begin{abstract}
Temporal link prediction is usually evaluated by predictive performance on unseen edges, but in probabilistic temporal graphs this criterion can conflate model error with irreducible uncertainty. We study this issue by characterising an inherent estimation--prediction tradeoff in binary logistic models where regimes that maximise Fisher information and improve parameter recoverability are also those with the highest entropy, making individual predictions intrinsically harder even under perfect parameter recovery.
We propose a probabilistic causal framework for generating temporal graphs with transient edges and known ground-truth causal structure, allowing temporal link prediction to be evaluated jointly with causal parameter recovery. For the proposed binary logistic parametrisation, we derive the Cram\'{e}r--Rao bound and validate the tradeoff between parameter estimation error and irreducible predictive loss. Our results show that predictive accuracy alone may not reflect whether a model has learned the underlying causal mechanism, motivating benchmarks that distinguish reducible model error from intrinsic process uncertainty.
\end{abstract}
\section{Introduction}
In supervised learning, a model is trained on a set of input--output pairs such that the predicted outputs closely match the ground-truth, thereby minimising a prescribed error function, typically referred to as the \textit{loss}. The trained model is then evaluated on previously unseen data, referred to as the test set. The resulting prediction error on the test set is reported as the \textit{predictive performance} of the model, on the basis of which it is benchmarked against competing approaches.
An important aspect, often overlooked by machine learning practitioners, is that
\textit{the training objective is not designed to recover the true mapping, but rather to approximate a model which best fits the given training data.}
In other words, the learned model is inherently sensitive to the sampled training data, and a substantial body of research has focused on mitigating this sensitivity through techniques such as batch training, stochastic gradient descent, batch normalisation, and regularisation, to name a few.
In this work, we revisit the problem of \textit{model sensitivity to the training data}~\cite{murphy2012machine} and instead of accepting low training loss as a proxy for successful learning, we ask the following question:
\begin{myquote}
(Q1) What if the learned model could be compared directly against the true model?
\end{myquote}
To this end, we assume that the true model belongs to a known parametric family, and that minimising the training loss recovers the true parameters asymptotically as the number of training samples increases~\cite{vapnik2013nature}. Parameter recovery performance can then be quantified directly through the difference between the estimated and true parameters. Since, in practice we are limited by a finite number of samples, a question naturally arises:
\begin{myquote}
(Q2) Does the observed data contain sufficient information to accurately recover the parameters?\vspace{-10pt}
\end{myquote}
A refined version of this question is:
\begin{myquote}
(Q3) What is the relation between parameter estimation error and the amount of useful information in the observed data?
\end{myquote}
We have tools from information theory at our disposal to answer such questions, under the additional assumption that the parametric model is probabilistic.
Once the parameters have been estimated as accurately as permitted by the available data, we evaluate the predictive performance of the resulting model on unseen samples, leading to the final question:
\begin{myquote}
(Q4) How is the predictive performance of the estimated model related to the parameter estimation error?
\end{myquote}
More specifically, we study how the lowest achievable parameter estimation error is related to the predictive performance attained by the corresponding estimated model.
We now motivate the problem in the context of temporal link prediction (TLP) \cite{longa_graph_2023}, where the objective is to predict whether an edge exists at a given time based on the past history of the temporal graph. In the literature, this task is typically formulated as a binary classification problem \cite{huang2024temporal}, where the implicit assumption is that the existence or non-existence of edges in the past contains sufficient information to predict the existence of edges in the future \cite{rahman2026generating}.
However, the underlying state space of the problem grows rapidly with the size of the graph. Consider a graph with $n$ nodes and therefore $n(n-1)/2$ possible edges. Suppose the prediction of a target edge depends on the existence history of $m$ past edges. Since each past edge may either exist or not exist, the number of possible historical configurations is $2^m$. Consequently, the size of the state space associated with predicting a single edge grows exponentially in $m$.
Moreover, in practice, the relevant dependencies among edges are generally unknown a priori. As a result, the learning procedure may effectively search over all possible edges in the graph in order to identify the causal dependencies. In such settings, the number of relevant historical edges may itself scale with the size of the graph, i.e. $m = \mathcal{O}(n^2)$, yielding an effective state space of $\mathcal{O}(n^2 2^{n^2})$, which grows exponentially with the number of edges.
\begin{myquote}
For a graph with $5$ nodes, the effective state space exceeds $10^4$, while for a graph with $10$ nodes, it grows to over a quadrillion $(10^{15})$.
\end{myquote}
This \textit{state explosion}~\cite{valmari1996state} raises an important issue for learning and evaluation. Many asymptotic arguments in statistical learning rely on the assumption that the number of available samples grows sufficiently large relative to the size of the underlying state space. However, in temporal link prediction, the combinatorial growth of the state space makes such assumptions difficult to justify.
In the preceding paragraphs, we highlighted an assumption in TLP which has largely remained implicit due to the lack of certifiably causal temporal graph datasets. However, in a recent work, \citep{rahman2026generating}, we proposed a parametric causal model for generating temporal graphs where the causal relation between an edge and its parent edges is deterministic. In this work, we instead require a probabilistic formulation, and therefore extend the deterministic model to a probabilistic causal framework. In this setting, parameter estimation naturally corresponds to causal discovery allowing us to reframe (Q4) as follows:
\begin{myquote}
(Q5) How is the TLP performance achieved by the estimated causal model related to the causal discovery error?
\end{myquote}
With respect to the questions (Q4) and (Q5), we see that the data regime which enables more accurate parameter estimation is simultaneously the regime in which prediction becomes intrinsically more difficult, which follows from the classical relationship between the Fisher information and entropy of binary logistic models. We refer to this relationship as the \textit{estimation--prediction tradeoff} and clarify it analytically in \cref{sec:tradeoff}. Then, we describe the proposed causal framework for temporal graph generation in \cref{sec:rctg}. In \cref{sec:estimation} and \cref{sec:prediction}, we derive the information-theoretic quantities associated with the proposed causal model and use them to analyse the estimation--prediction trade-off. Finally, we present empirical results for TLP in \cref{sec:results} and conclude in \cref{sec:conclusion}.
Therefore, the core contribution of this work is the proposal of a parametric causal model for generating temporal graphs which exhibit the aforementioned estimation--prediction tradeoff.
\section{Estimation--Prediction Tradeoff}
\label{sec:tradeoff}
The following relationship between Fisher information and predictive entropy in binary logistic models is an elementary consequence of both quantities being determined by $p(\theta)(1-p(\theta))$.
\paragraph{Estimation}
Consider a Bernoulli model with success probability $p(\theta)$ parametrised by $\theta \in \mathbb{R}$, such that
\begin{align}
p(\theta) = \sigma(g(\theta)),
\label{blm}
\end{align}
where $\sigma(\cdot)$ denotes the sigmoid function and $g(\theta)$ is differentiable with respect to $\theta$. The log-likelihood of a single observation $x$ is
\begin{align}
\ell(\theta; x) = x \log{p(\theta)} + (1-x)\log{(1 - p(\theta))}.
\end{align}
Differentiating with respect to $\theta$ and applying the chain rule through $\sigma(\cdot)$, the Fisher information is:
\begin{align}
J(\theta) &= \mathbb{E}\left[\left(\frac{\partial \ell(\theta; x)}{\partial \theta}\right)^2\right] \nonumber \\
&= \mathbb{E}\left[(x-p(\theta))^2 \left(\frac{\partial g(\theta)}{\partial \theta}\right)^2\right] \nonumber\\
&= p(\theta)(1-p(\theta)) \left(\frac{\partial g(\theta)}{\partial \theta}\right)^2,
\end{align}
where the last step uses $\mathrm{Var}(X) = p(\theta)(1-p(\theta))$ and the fact that $\frac{\partial g(\theta)}{\partial \theta}$ is deterministic given $\theta$.
Applying the Cram\'{e}r--Rao (CR) inequality \citep[\textsection~11.10]{cover1999elements} for $N$ samples gives a lower bound on the mean squared parameter estimation error,
\begin{align}
\mathbb{E}\left[ ( \hat{\theta} - \theta)^2\right] \geq \frac{1}{N}J(\theta)^{-1}.
\label{eq:crb_final}
\end{align}
\paragraph{Prediction}
Now consider a set of samples from the Bernoulli model, and a prediction model which has recovered the parameter $\hat{\theta}$. Let the true value sampled be $x$ and the prediction be $\hat{x}$.
Under a probabilistic prediction model, the expected binary cross-entropy (BCE) loss is,
\begin{align}
\mathcal{L}_{\text{BCE}}(\theta, \hat{\theta}) &= -\mathbb{E}\left[x \log p(\hat{\theta}) + (1-x)\log(1-p(\hat{\theta}))\right] \nonumber
\\&= h(p(\theta)) + D_{\text{KL}}(p(\theta) \| p(\hat{\theta})),
\end{align}
where $h(p(\theta))$ is the irreducible entropy and $D_{\text{KL}}(p(\theta) \| p(\hat{\theta})) \geq 0$ is the excess loss due to parameter mismatch, vanishing as $\hat{\theta} \to \theta$.
For soft predictions $\hat{x} = p(\hat{\theta})$, the mean squared error (MSE) is
\begin{align}
\mathcal{L}_{\text{MSE}}(\theta, \hat{\theta}) &= \mathbb{E}\left[(x - p(\hat{\theta}))^2 \right] \nonumber \\
&= p(\theta)(1-p(\theta)) + (p(\theta) - p(\hat{\theta}))^2,
\end{align}
where the first term is the irreducible variance of $X$ and the second is the squared bias due to parameter mismatch.
\paragraph{Tradeoff} We are now interested in the relationship between the parameter estimation error $\E[(\theta - \hat{\theta} )^2 ]$ and the prediction error.
In \cref{fig:bce}, we illustrate how the BCE loss varies with parameter estimation error. The BCE loss has two components: the entropy $h(p(\theta))$ of the data-generating process, which cannot be reduced by any estimator, and an excess term $D_{\text{KL}}(p(\theta) \| p(\hat{\theta})) \geq 0$, which reflects how far the estimated parameters are from the truth. By the CR bound, the estimation error cannot be lower than $\frac{1}{N}J(\theta)^{-1}$, meaning there is always a residual parameter mismatch and the KL term cannot vanish entirely with finite samples. As the estimation error grows beyond this bound, the KL term increases further, adding to the total prediction loss. Therefore, the total BCE loss is always strictly greater than the entropy $h(p(\theta))$, with the gap determined by how well the model parameters have been recovered relative to the CR bound.\footnote{Note that the exact shape of the curve depends on the specific form of $p(\theta)$, and the figure is for illustrative purposes only.}
\begin{figure}[t]
\centering
\includegraphics[width=0.8\columnwidth]{figs/bce.pdf}
\caption{BCE loss as a function of parameter estimation error.}
\label{fig:bce}
\vspace{-10pt}
\end{figure}
A similar decomposition holds for the MSE loss, where the total prediction error equals the irreducible variance $p(\theta)(1-p(\theta))$ plus an excess term $(p(\theta) - p(\hat{\theta}))^2$ that grows as the estimation error increases.
We now turn our attention to the relationship between the irreducible prediction error and the CR bound. The irreducible terms of both the BCE loss and the MSE loss, namely $h(p(\theta))$ and $p(\theta)(1-p(\theta))$, respectively, are both functions of $p(\theta)$ alone, as is the CR bound $\frac{1}{N}J(\theta)^{-1}$, which is inversely proportional to $p(\theta)(1-p(\theta))$. As shown in \cref{fig:entropy_p}, both $h(p(\theta))$ and $p(\theta)(1-p(\theta))$ are symmetric about $p(\theta) = \tfrac{1}{2}$ and \textit{monotone equivalent} in $p(\theta) \in [0,1]$.
\begin{figure}[h!]
\centering
\begin{subfigure}[t]{0.48\columnwidth}
\centering
\includegraphics[width=\linewidth]{figs/ent_curve.pdf}
\caption{Prediction error vs. $p$}
\label{fig:entropy_p}
\end{subfigure}
\begin{subfigure}[t]{0.48\columnwidth}
\centering
\includegraphics[width=\linewidth]{figs/ent_p.pdf}
\caption{Entropy vs. the CR bound}
\label{fig:entropy_vs_crb}
\end{subfigure}
\caption{Prediction errors and the CR bound.}
\label{fig:entcurve}
\end{figure}
Furthermore, in \cref{fig:entropy_vs_crb}, we plot the entropy $h(p(\theta))$ against the $p(\theta)$-dependent component of the CR bound, $\frac{1}{p(\theta)(1-p(\theta))}$, revealing a smooth inverse relationship. As the CR bound decreases, the entropy increases. Since a lower CR bound means the parameters can be estimated more accurately, and higher entropy means the lowest achievable prediction error is larger, the two quantities are fundamentally linked through the data-generating process and cannot be controlled.
\begin{myquote}
Therefore, the more we can learn about the parameters from the data, the less we can know about the prediction outcome.
\end{myquote}
This should not be confused with the idea that better parameter estimation leads to higher prediction error. As shown in \cref{fig:bce}, better estimation always leads to lower prediction error for a fixed data-generating process. What we are characterising here\footnote{The relationship is a consequence of definitions. We include it to make the subsequent analysis self-contained.} is the fundamental performance limit of a binary logistic model described in \cref{blm}. Specifically, the regime of the data-generating process that allows parameters to be estimated most accurately is the same regime in which the irreducible prediction error is largest, regardless of how good the estimator is.
\section{Random Causal Temporal Graphs}
\label{sec:rctg}
We now consider the problem of generating causal temporal graphs with transient edges. Unlike ephemeral edges, which exist only instantaneously, transient edges persist for some finite duration before vanishing. Our approach extends our earlier model \citep{rahman2026generating}, originally proposed for ephemeral edges, to the transient case.
\begin{quote}
The key premise is that the presence or absence of certain edges in the \textit{recent past} may cause the activation or deactivation of other edges.
\end{quote}
\subsection{Preliminaries}
We begin by formalising our representation. A temporal graph with transient edges can be viewed as a collection of irregular binary pulse signals, one per edge, that switch between active and inactive states. More precisely:
\begin{definition}[Temporal Graph]
A temporal graph $\gG$ with transient edges and continuous timestamps over a set of nodes $\gV$ is defined as a set of quadruples $(u,v,t,t')$, where $u,v \in \gV$, $t,t' \in \mathbb{R}$ with $t < t'$. Each quadruple indicates that edge $(u,v)$ is active on the interval $[t, t')$.
\end{definition}
To model the timestamps at which edges activate and deactivate, we use one-dimensional homogeneous Poisson point processes~\cite{haenggi2012stochastic}, one process per edge.
\begin{definition}[Poisson Point Process]
A one-dimensional homogeneous Poisson point process (PPP) with rate parameter $\lambda > 0$ is a random countable set $\Phi(\lambda) \subseteq \mathbb{R}$ such that for any interval $\gB \subseteq \mathbb{R}$, the number of points in $\gB$ follows a Poisson distribution with mean $\lambda |\gB|$, where $|\gB|$ denotes the length of the interval.
\end{definition}
Any realization of $\Phi(\lambda)$ can be written as an ordered sequence $\{t_1, t_2, \ldots\}$ where $t_1 < t_2 < \cdots$. The causal model, which we introduce next, determines how these timestamps translate into edge activations and deactivations.
We adopt the causal models introduced by \citet{pearl2009causality}, which formalise how variables causally influence one another through a directed graph and associated structural equations.
The causal model consists of a set of exogenous variables denoted by $\gU= \{U_1, U_2, \ldots\}$, and a set of endogenous variables $\gX = \{X_1, X_2, \ldots\}$, whose values are determined within the model. The causal structure is encoded in a directed acyclic graph $\graph$, where an edge from $A$ to $B$ indicates that $A$ causally influences $B$, and $A$ is called a \textit{parent} of $B$. We denote the set of parents of $X_i$ by $\parents_i$.
\begin{definition}[Causal Model]
A causal model is a pair $( \graph, \Theta )$ consisting of a causal graph $\graph$ and a set of parameters $\Theta$ compatible with $\graph$. The parameters $\Theta$ assign a function $x_i = f_i(\pa_i, u_i)$ to each variable $X_i \in \gX$ and a probability $\mathbb{P}(u_i)$ to each $u_i$.
\end{definition}
In our setting, we use a probabilistic variant where structural equations specify conditional distributions rather than deterministic mappings. To this end, we write the probabilistic structural equation model (SEM) specifying the conditional distribution of $X_i$ given $\pa_i$ and $u_i$ as
\begin{align}
\mathbb{P}(X_i = x_i \mid \parents_i = \pa_i, U_i = u_i) = f_i(x_i \mid \pa_i, u_i).
\end{align}
\subsection{Generation}
We define a causal model that captures the influence of past edge states on future edge activations. For a temporal graph with $n$ nodes, there are $k = \binom{n}{2}$ possible edges. The causal graph $\graph$ is defined over the following sets of binary variables:
\begin{itemize}[noitemsep, topsep=0pt]
\item $\gU = \{U_i(t) : i \in [k]\}$, where $U_i(t)$ indicates whether a Poisson trigger occurs for edge $i$ at time $t$,
\item $\gX' = \{X_i'(t) : i \in [k]\}$, where $X_i'(t)$ denotes the state of edge $i$ immediately before time $t$,
\item $\gX = \{X_i(t) : i \in [k]\}$, where $X_i(t)$ denotes whether edge $i$ is active at time $t$.
\end{itemize}
At time $t$, the variables $\gX$ are treated as endogenous, while $\gU$ and $\gX'$ are treated as exogenous.\footnote{At each time $t$, we treat $\gX'$ and $\gU$ as given inputs to determine $\gX$. Temporally, $\gX'$ represents the edge states from the previous instant.}
We begin by constructing a binomial random intersection graph\footnote{In a binomial random intersection graph, each vertex $i \in [k]$ is associated with a random subset $\gM_i \subseteq [m]$, formed by including each element independently with probability $p$. An undirected edge is placed between $i$ and $j$ if $\gM_i \cap \gM_j \neq \emptyset$.} $G(k,m,p)$, where $k = \binom{n}{2}$ is the number of possible edges in a graph with $n$ nodes~\citep[\textsection~13.1]{frieze2023random}. Each undirected edge in $G$ is then assigned an orientation independently and uniformly at random, yielding a directed graph. Let $\rmA \in \{0,1\}^{k \times k}$ denote its adjacency matrix.
The causal graph $\graph$ is defined over the variable set $\gX' \cup \gX \cup \gU$, where the current state $X_i(t)$ depends on (i) the previous states $X_j'(t)$ of its parent edges, as specified by $\rmA$, and (ii) the trigger variable $U_i(t)$.
Ordering the variables as $(\gX', \gX, \gU)$, with each block arranged by edge index from $1$ to $k$, the adjacency matrix of the causal graph takes the block form
\begin{align}
\rmA^\graph =
\begin{bmatrix}
\bm{0} & \rmA & \bm{0} \\
\bm{0} & \bm{0} & \bm{0} \\
\bm{0} & \rmI & \bm{0}
\end{bmatrix}.
\label{causal_graph}
\end{align}
For each edge $i \in [k]$, we define an independent homogeneous PPP $\Phi_i$ with rate $\lambda_i$ over $[0, T)$, where $T \in \mathbb{R}^+$. We denote the rate parameters collectively as $\bm{\lambda} = \{\lambda_1, \ldots, \lambda_k\}$. At each timestamp $t \in \Phi_i$, the causal model determines whether edge $i$ changes state, as we now specify through the probabilistic structural equations.
% \paragraph{Probabilistic Structural Equation Model}
At each trigger $t \in \Phi_i$, we sample the value of $X_i(t)$ via a Bernoulli trial with success probability
\begin{align}
p_i^t &= \mathbb{P}(X_i(t) = 1 \mid \pa_i, u_i(t) ) \nonumber
\\&= u_i(t) \cdot \sigma \left( \sum_{ X_j'(t) \in \parents_i} \Theta_{j,i} \, x_j'(t) \right),
\end{align}
where $\Theta_{j,i} \in \mathbb{R}$ is the causal parameter matrix satisfying $\rmA_{j,i}=0 \implies \Theta_{j,i} = 0 $. In vector notation, the probabilistic SEM can be written compactly as
\begin{align}
p_i^t = \sigma\left( \Theta_{:,i}^\top \, \rvx'(t) \right),
\end{align}
where $\Theta_{:,i}$ denotes the $i$-th column of the parameter matrix and $\rvx'(t) = [x_1'(t), \ldots, x_k'(t)]^\top \in \{0,1\}^k$ is the vector of all edge states immediately before time $t$.
In this work, we sample the elements of $\Theta$ independently from a zero-mean normal distribution,
\begin{align}
\Theta_{j,i} \sim \gN(0, \sigma^2), \quad \forall i,j \in [k] : \rmA_{j,i} = 1,
\end{align}
and $\Theta_{j,i} = 0$ whenever $\rmA_{j,i} = 0$.
In realistic settings, not all edges are causally influenced by others. To reflect this, we designate a subset of edges as \textit{noisy edges} that evolve independently of the other edges.
For a noise level $\eta \in [0,1)$, we uniformly sample a subset $\gJ \subset [k]$ of size $|\gJ| = \lfloor \eta k \rfloor$ and set
\begin{align}
\rmA_{i,j} = 0 \quad \forall i \in [k], \, j \in \gJ.
\end{align}
This ensures that edges in $\gJ$ have no parents in the causal graph. Consequently, when a trigger occurs for edge $j \in \gJ$, its activation probability becomes $p_j^t = \sigma(0) = \frac{1}{2}$, equivalent to a fair coin flip.
These noisy edges serve two purposes: they provide a more realistic model where not all edges participate in causal relationships, and they present a challenge for causal discovery methods, which must distinguish truly causal edges from noise.
We summarise the proposed generative causal model as a parametric generator $\mathscr{M}$ defined by
\begin{align}
\mathscr{M}(n, m, p, \sigma^2, T, \bm{\lambda}, \eta),
\end{align}
where $n$ is the number of nodes, $(m, p)$ parametrise the binomial random intersection graph used to construct the causal graph $\graph$, $\sigma^2$ is the variance of the Gaussian distribution from which non-zero entries of $\Theta$ are sampled, $T$ defines the time horizon $[0, T)$, $\bm{\lambda} = \{\lambda_1, \ldots, \lambda_k\}$ are the PPP intensities, and $\eta \in [0,1)$ is the noise level.
Sampling from $\mathscr{M}$ proceeds in two stages. First, we sample the causal model structure, parameters, and triggers:
\begin{align}
\mathscr{C}(\rmA^\graph, \Theta, \bm{\Phi}) \sim \mathscr{M}(n, m, p, \sigma^2, T, \bm{\lambda}, \eta),
\end{align}
where $\rmA^\graph$ is the causal graph adjacency matrix \cref{causal_graph}, $\Theta$ is the causal parameter matrix, and $\bm{\Phi} = \{\Phi_1, \ldots, \Phi_k\}$ are the realized Poisson point processes within the time horizon $[0, T)$. Second, we generate the temporal graph by traversing $\cup_{i \in [k]} \Phi_i$ chronologically, and evaluating the probabilistic structural equations for the triggered edge. Sampling a temporal graph $\gG$ from the causal model $\mathscr{C}$ is represented as
\begin{align}
\gG \sim \mathscr{C}(\rmA^\graph, \Theta, \bm{\Phi}).
\end{align}
We present the generative mechanism in \cref{algo1}.
\subsection{Estimation}
\label{sec:estimation}
Given a realisation $\gG \sim \mathscr{C}(\rmA^\graph, \Theta, \bm{\Phi})$, the goal of estimation
is to recover the causal parameter matrix $\Theta$ from the observed edge states. For each
edge $i \in [k]$, we estimate the column $\Theta_{:,i}$ by minimising the binary cross-entropy
loss over the trigger times $\Phi_i$,
\begin{align}
\hat{\Theta}_{:,i} = \argmin_{\bm{\theta} \in \mathbb{R}^k}
- \sum_{t \in \Phi_i} \Big( x_i(t) \ln \sigma\!\left(\bm{\theta}^\top \rvx'(t)\right)
\nonumber \\
+ \left(1 - x_i(t)\right) \ln\left(1 - \sigma\!\left(\bm{\theta}^\top \rvx'(t)\right)\right) \Big).
\label{eq:estimation}
\end{align}
\Eqref{eq:estimation} depicts unregularised logistic regression, solved independently for each edge $i \in [k]$. This formulation ensures consistency:
as $|\Phi_i| \to \infty$, the estimator $\hat{\Theta}_{:,i}$ converges to the true $\Theta_{:,i}$. In \cref{prop:crb} we present the CR bound for our proposed temporal graph model.
\begin{proposition}
\label{prop:crb}
Let $\hat{\Theta}$ be any unbiased estimator of $\Theta$ obtained by solving \cref{eq:estimation} independently for each edge $i \in [k]$. Then the total parameter estimation error $\mathbb{E}[\lVert\hat{\Theta} - \Theta\rVert_F^2] $ is bounded below by
\begin{align}
\sum_{i \in [k]} \mathrm{tr}\!\left(\left(\sum_{t \in \Phi_i}
p_i^t\left(1 - p_i^t\right)
\rvx'(t)\rvx'(t)^\top\right)^{-1}\right).
\end{align}
\end{proposition}
\begin{proof}
See Appendix~\ref{app:proof_crb}.
\end{proof}
\begin{algorithm}[h!]
\caption{Sampling from $\mathcal{M}(n,m,p,\sigma^2,T, \bm{\lambda},\eta)$}
\label{alg:causal-temporal-graph}
\begin{algorithmic}[1]
\REQUIRE Number of nodes $n$, parameters $(m,p)$, variance $\sigma^2$, time horizon $T$, rates $\bm{\lambda}$, noise level $\eta$
\ENSURE Temporal graph $\gG$ and ground-truth causal model $(\rmA^\graph,\Theta, \bm{\Phi})$
\STATE Set $k \gets {n \choose 2}$ and initialize $\gG \gets \emptyset$
\STATE Sample a binomial random intersection graph on $k$ edge variables with $m$ objects and probability $p$
\STATE Orient each edge independently to obtain $\rmA$
\STATE Sample a noise set $\gJ \subset [k]$ with $|\gJ|=\lfloor \eta k \rfloor$
\FOR{$j \in \gJ$}
\STATE Set $\rmA_{i,j} \gets 0$ for all $i \in [k]$
\ENDFOR
\STATE Initialize $\Theta \gets \bm{0}_{k \times k}$
\FOR{$i,j \in [k]$ such that $\rmA_{j,i}=1$}
\STATE Sample $\Theta_{j,i} \sim \mathcal{N}(0,\sigma^2)$
\ENDFOR
\FOR{$i \in [k]$}
\STATE Sample trigger times $\Phi_i \sim \mathrm{PPP}(\lambda_i)$ on $[0,T)$
\ENDFOR
\STATE Initialize $\rvx'(0) \gets \bm{0}$
\STATE Sort all trigger events
$\bigcup_{i \in [k]} \{(i,t): t \in \Phi_i\}$ in increasing time order
\FOR{each trigger event $(i,t)$ in chronological order}
\STATE $\rvx'(t)$ is the edge-state vector immediately before $t$
\STATE Set $p_i^t \gets \sigma(\Theta_{:,i}^{\top}\rvx'(t))$
\STATE Sample $x_i(t) \sim \mathrm{Bernoulli}(p_i^t)$
\IF{$x_i(t)=1$ and $x'_i(t)=0$}
\STATE Record activation time $s_i \gets t$ for edge $i$
\ELSIF{$x_i(t)=0$ and $x'_i(t)=1$}
\STATE Add interval $(e_i,s_i,t)$ to $\gG$
\ENDIF
% \STATE Update $x'_i(t^+) \gets x_i(t)$ and leave all other states unchanged
\ENDFOR
\FOR{each edge $i$ with $x'_i(T)=1$}
\STATE Add interval $(e_i,s_i,T)$ to $\gG$
\ENDFOR
\STATE \textbf{return} $\gG$ and $(\rmA^{\graph},\Theta,\Phi)$
\end{algorithmic}
\label{algo1}
\end{algorithm}
\subsection{Prediction}
\label{sec:prediction}
As the temporal graph $\gG$ is generated in chronological order of trigger events, at each trigger $(i,t)$ a Bernoulli random variable $X_i(t)$ is realized with success probability $p_i^t$.
Given an estimated parameter matrix $\hat{\Theta}$, the corresponding predicted success probability is $\hat{p}_i^t = \sigma\!\left(\hat{\Theta}_{:,i}^\top \rvx'(t)\right).$
To study the fundamental limits of prediction, we consider the expected binary cross-entropy (BCE) loss under the true data-generating model:
\begin{align}
\mathcal{L}_\gG(\Theta, \hat{\Theta})
=
\sum_{i \in [k]} \sum_{t \in \Phi_i}
\mathbb{E}\!\left[
\ell\!\left(X_i(t);\, \hat{p}_i^t\right)
\right],
\end{align}
where $\ell(x; \hat{p}) = -x \ln \hat{p} - (1-x)\ln(1-\hat{p})$.
Taking expectation with respect to $X_i(t) \sim \mathrm{Bernoulli}(p_i^t)$, the loss decomposes as
\begin{align}
\mathcal{L}_\gG(\Theta, \hat{\Theta})
=
\sum_{i \in [k]} \sum_{t \in \Phi_i}
h(p_i^t)
+
\sum_{i \in [k]} \sum_{t \in \Phi_i}
D_{\mathrm{KL}}\!\left(p_i^t \,\|\, \hat{p}_i^t\right),
\end{align}
where $h(p_i^t)$ is the binary entropy of the Bernoulli distribution under the true parameters, and the KL divergence term captures the excess loss due to parameter mismatch, vanishing as $\hat{\Theta} \to \Theta$.
The first term represents the intrinsic uncertainty of the outcome at each trigger and constitutes an irreducible lower bound on predictive performance. The second term measures the additional error introduced by imperfect parameter estimation. Thus, even under perfect recovery of the parameters, the predictive loss is bounded below by the entropy of the underlying process.
\section{Results}
\label{sec:results}
Temporal graphs were generated using $\mathscr{M}$ across multiple parameter regimes. The number of nodes was fixed to $n=5$, yielding $k=10$ possible edges, and the noise level to $\eta=0$, corresponding to a fully causal setting. The causal graph was sampled using a binomial random intersection graph with $m=20$ and
$p \in \{0.1,\ldots,0.9\}$. For each edge, an independent PPP was sampled over $[0, T)$ with $T=10^4$, and the causal parameters were drawn from $\mathcal{N}(0,\sigma^2)$ with $\sigma \in \{1.0, 2.0, 2.5, 3.0, 4.0, 5.0\}$.
\begin{figure}[h!]
\centering
\includegraphics[width=\columnwidth]{figs/est_vs_fisher.pdf}
\caption{Empirical parameter estimation error vs. CR bound.}
\label{fig:est_fisher}
\end{figure}
In \cref{fig:est_fisher} we show the empirical parameter estimation error $\mathbb{E}[\|\hat{\Theta} - \Theta\|_F^2]$
against the CR bound $\sum_{i \in [k]}\mathrm{tr}(\mathbf{J}_i^{-1})$ across parameter configurations of $\mathscr{M}$. The estimation error increases monotonically with the CR bound, consistent with \cref{prop:crb}.
\newpage
In \cref{fig:ent_fisher}, we show the irreducible prediction error $\sum_{i \in [k]}\sum_{t \in \Phi_i} h(p_i^t)$ against the CR bound $\sum_{i \in [k]}\mathrm{tr}(\mathbf{J}_i^{-1})$ across parameter configurations of $\mathscr{M}$. The sharp inverse relationship is consistent with the estimation--prediction tradeoff: the regime in which parameters can be estimated most accurately is precisely the regime in which the irreducible prediction error is largest.
\begin{figure}[h!]
\centering
\includegraphics[width=\columnwidth]{figs/ent_vs_fisher.pdf}
\caption{Irreducible prediction error vs. CR bound.}
\label{fig:ent_fisher}
\end{figure}
Therefore, standard metrics may conflate model performance with irreducible uncertainty and rank models by the entropy of the dataset rather than their ability to recover the data-generating mechanism. This makes the existence of temporal graph models with explicit generative structure important for principled benchmarking.
It must be noted that the estimation--prediction tradeoff is derived here for the binary logistic case, and closed-form results may not generalise directly to more complex temporal graph models. However, one can examine the relationship between dataset entropy and predictive performance empirically to probe the tradeoff without requiring an analytical derivation.
\section{Conclusion}
\label{sec:conclusion}
This work shows that, in the probabilistic temporal graph model proposed here\footnote{The source code will be made available upon request.}, learning and prediction are limited by different aspects of the same data-generating process. In binary logistic models, high Fisher information improves parameter recoverability but coincides with high-entropy regimes where individual outcomes are intrinsically uncertain. The regime in which a causal mechanism is easiest to learn may therefore be precisely the regime in which future edge states are hardest to predict.
To study this in temporal graphs, we proposed a probabilistic causal generator with transient edges and known ground-truth parameters, enabling joint evaluation of predictive loss and causal recovery. The derived Fisher information and Cram\'{e}r--Rao bounds provide information-theoretic reference points, and the experiments confirm the relationship between estimation error and irreducible predictive uncertainty.
The main lesson is that predictive accuracy alone is not a reliable proxy for learning the temporal mechanism. Temporal graph benchmarks should separate reducible model error from irreducible uncertainty and evaluate causal recovery whenever ground truth is available. We hope this work motivates the temporal graph learning community \citep{yitgb} to integrate causality into model evaluation and look beyond standard predictive metrics.
\bibliography{causality}
\bibliographystyle{icml2023}
\appendix
\section{Proof of \cref{prop:crb}}
\label{app:proof_crb}
We proceed in two steps: first deriving the CR bound for a single edge $i \in [k]$, then summing over all edges.
\paragraph{Single edge}
Fix $i \in [k]$. At each trigger $t \in \Phi_i$, the activation $X_i(t) \in \{0,1\}$ is a Bernoulli random variable with success probability $p_i^t = \sigma(\Theta_{:,i}^\top \rvx'(t))$, conditioned on the context $\rvx'(t)$. The
log-likelihood contribution of a single observation is
\begin{align}
\ell\!\left(\Theta_{:,i}; X_i(t)\right) = X_i(t) \ln p_i^t +
\left(1 - X_i(t)\right) \ln\!\left(1 - p_i^t\right).
\end{align}
Differentiating with respect to $\Theta_{:,i}$ and applying the identity
$\sigma'(z) = \sigma(z)(1 - \sigma(z))$, the score is
\begin{align}
\frac{\partial \ell}{\partial \Theta_{:,i}} = \left(X_i(t) - p_i^t\right)\rvx'(t).
\end{align}
The Fisher information matrix for a single observation is
\begin{align}
\rmJ_i^t &= \mathbb{E}\left[\frac{\partial \ell}{\partial \Theta_{:,i}}
\frac{\partial \ell}{\partial \Theta_{:,i}}^\top \right] = \mathbb{E}\left[\left(X_i(t) - p_i^t\right)^2\right] \rvx'(t)\rvx'(t)^\top \nonumber \\
&= p_i^t\left(1 - p_i^t\right) \rvx'(t)\rvx'(t)^\top,
\end{align}
where we have used $\mathrm{Var}(X_i(t)) = p_i^t(1-p_i^t)$ and the fact that $\rvx'(t)$ is deterministic given $\Theta_{:,i}$. Since the $N_i = |\Phi_i|$ observations are independent, the total Fisher information matrix accumulates as
\begin{align}
\rmJ_i = \sum_{t \in \Phi_i} p_i^t\left(1 - p_i^t\right) \rvx'(t)\rvx'(t)^\top.
\end{align}
Applying the multivariate CR bound~\citep[\textsection~11.10]{cover1999elements} to any unbiased estimator $\hat{\Theta}_{:,i}$,
\begin{align}
\mathbb{E}\left[\left\lVert \hat{\Theta}_{:,i} - \Theta_{:,i} \right\rVert^2\right]
\geq \mathrm{tr}\!\left(\rmJ_i^{-1}\right).
\end{align}
\paragraph{All edges}
Since the optimisation in \cref{eq:estimation} is solved independently for each $i \in [k]$, the estimators $\hat{\Theta}_{:,i}$ are independent across edges. The total squared error therefore decomposes as
\begin{align}
\mathbb{E}\left[\left\lVert \hat{\Theta} - \Theta \right\rVert_F^2\right]
= \sum_{i \in [k]} \mathbb{E}\left[\left\lVert \hat{\Theta}_{:,i} -
\Theta_{:,i} \right\rVert^2\right],
\end{align}
where we have used the fact that the Frobenius norm decomposes column-wise. Applying the per-edge CR bound to each term and summing gives the stated lower bound.
% \section{Rough (verify)}
% \begin{proposition}[Sample Complexity for Causal Parameter Recovery]
% \label{prop:sample_complexity}
% Fix edge $i \in [k]$ with parent set $\pa_ii$ of size $d_i$ and PPP rate $\lambda_i$. Assume that the parent state vectors $\rvx'_{\pa_ii}(t)$ at trigger times are i.i.d.\ draws from a distribution $\pi_i$ over $\{0,1\}^{d_i}$, with $\pi_{\min,i} = \min_{\mathbf{s} \in \{0,1\}^{d_i}} \pi_i(\mathbf{s}) > 0$. Define the expected Fisher information per trigger as
% \begin{align}
% \bar{J}_i = \mathbb{E}_{\pi_i}\!\left[ p_i(1-p_i)\, \rvx'_{\pa_ii}\rvx_{\pa_ii}^{\prime\top} \right], \quad p_i = \sigma\!\left(\Theta_{:,i}^\top \rvx'_{\pa_ii}\right).
% \end{align}
% For any unbiased estimator $\hat{\Theta}_{:,i}$ to achieve $\mathbb{E}[\|\hat{\Theta}_{:,i} - \Theta_{:,i}\|^2] \leq \epsilon$, the time horizon must satisfy
% \begin{align}
% T \geq \max\!\left( \frac{1}{\lambda_i \,\pi_{\min,i}},\; \frac{d_i}{\lambda_i\,\epsilon\,\mu_{\min}(\bar{J}_i)} \right).
% \label{eq:T_lower}
% \end{align}
% Under the uniform distribution over $\{0,1\}^{d_i}$, we have $\pi_{\min,i} = 2^{-d_i}$ and the first term gives
% \begin{align}
% T = \Omega\!\left( \frac{2^{d_i}}{\lambda_i} \right),
% \end{align}
% so the required time horizon grows exponentially in the number of causal parents $d_i$. Since $d_i \leq k = \binom{n}{2}$ in the worst case, the time required for consistent recovery of all edge parameters satisfies
% \begin{align}
% T = \Omega\!\left( \frac{2^{n^2/2}}{\min_{i \in [k]} \lambda_i} \right).
% \end{align}
% \end{proposition}
% \begin{proof}
% \textbf{Part 1 (Identifiability threshold).}
% The estimator $\hat{\Theta}_{:,i}$ is obtained by logistic regression over the trigger times $\Phi_i$. The accumulated Fisher information matrix
% \begin{align}
% J_i = \sum_{t \in \Phi_i} p_i^t(1-p_i^t)\, \rvx'_{\pa_ii}(t)\rvx_{\pa_ii}^{\prime\top}(t)
% \end{align}
% is invertible only if the observed parent contexts span $\mathbb{R}^{d_i}$. A necessary condition is that every configuration $\mathbf{s} \in \{0,1\}^{d_i}$ with $\pi_i(\mathbf{s}) > 0$ is observed at least once; in particular the rarest configuration $\mathbf{s}^* = \arg\min_{\mathbf{s}} \pi_i(\mathbf{s})$ must be observed. The number of triggers landing on $\mathbf{s}^*$ in $[0,T)$ follows a Poisson distribution with mean $\lambda_i T \pi_{\min,i}$. For this quantity to be at least $1$ in expectation requires $T \geq 1/(\lambda_i \pi_{\min,i})$, which gives the first term in~\eqref{eq:T_lower}.
% \textbf{Part 2 (Cram\'{e}r--Rao bound).}
% The number of triggers $N_i = |\Phi_i| \sim \mathrm{Poisson}(\lambda_i T)$. Conditionally on $N_i$ i.i.d.\ parent contexts drawn from $\pi_i$, the expected Fisher information satisfies $\mathbb{E}[J_i \mid N_i] = N_i \bar{J}_i$. Applying the multivariate CR bound~\citep[\S~11.10]{cover1999elements} conditionally on $N_i$ and taking expectations over $N_i$ using $\mathbb{E}[N_i] = \lambda_i T$,
% \begin{align}
% \mathbb{E}\!\left[\|\hat{\Theta}_{:,i} - \Theta_{:,i}\|^2\right] \geq \mathbb{E}\!\left[\mathrm{tr}(J_i^{-1})\right] \geq \frac{d_i}{\lambda_i T\, \mu_{\min}(\bar{J}_i)}.
% \end{align}
% For this to be at most $\epsilon$ we require $T \geq d_i / (\lambda_i\,\epsilon\,\mu_{\min}(\bar{J}_i))$, which is the second term in~\eqref{eq:T_lower}.
% \textbf{Part 3 (Exponential scaling).}
% Under the uniform distribution, $\pi_{\min,i} = 2^{-d_i}$, so Part~1 gives $T \geq 2^{d_i}/\lambda_i$. Since $d_i \leq k = \binom{n}{2} \leq n^2/2$ in the worst case, taking the maximum over all $k$ edges and the minimum over all rates gives the stated bound.
% \end{proof}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
\usepackage{pifont}
% \usepackage{ulem}
\newcommand{\ztick}{\ding{51}}
\newcommand{\zcross}{\ding{55}}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
\newcommand{\vep}{\bm{\varepsilon}}
\newcommand{\lnorm}{\left\lVert}
\newcommand{\rnorm}{\right\rVert}
\newcommand{\indep}{\,\perp\!\!\!\!\!\!\perp\,}
% \newtheorem{remark}{Remark}
% For theorems and such
% \usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
% \usepackage{amsthm}
\usepackage{adjustbox}
\usepackage{rotating}
\newcommand{\xrr}[1]{\textcolor[HTML]{e86f13}{#1}}
\newcommand{\xrb}[1]{\textcolor[HTML]{5188e0}{#1}}
\newcommand{\xrm}[1]{\textcolor[HTML]{c21365}{#1}}
\newcommand{\xrc}[1]{\textcolor[HTML]{096947}{#1}}
\newcommand{\xra}[1]{\textcolor[HTML]{13479c}{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \usepackage{amsthm} % for theorem environments
% \theoremstyle{plain}
% \newtheorem{theorem}{Theorem}[section]
% \newtheorem*{theorem*}{Theorem}
% \newtheorem{proposition}[theorem]{Proposition}
% \newtheorem{lemma}[theorem]{Lemma}
% \newtheorem*{lemma*}{Lemma}
% \newtheorem{corollary}{Corollary}[section]
% \theoremstyle{definition}
% \newtheorem{definition}{Definition}[section]
\newtheorem{assumption}{Assumption}[section]
\newtheorem{hypothesis}{Hypothesis}[section]
\newtheorem{problem}{Problem}[section]
\usepackage{mathrsfs}
% \theoremstyle{remark}
% \newtheorem{remark}{Remark}[section]
% \newtheorem{conj}{Conjecture}
% \newtheorem{question}{Question}
% \newtheorem{idea}{Idea}
\RequirePackage{algorithm}
\RequirePackage{algorithmic}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
% \newcommand{\Esample}{\widehat{\mathbb{E}}}
\DeclareMathOperator*{\Esample}{\widehat{\mathbb{E}}}
\DeclareMathOperator*{\Psample}{\widehat{\mathbb{P}}}
\newcommand{\fullnorm}[2][]{\lVert #2 \rVert_{#1}}
% \newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
% \newcommand{\E}{\mathbb{E}}
\DeclareMathOperator*{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathbb{V}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\newcommand{\pa}{pa}
\newcommand{\graph}{\mathscr{G}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\newcommand*{\ldblbrace}{\{\mskip-5mu\{}
\newcommand*{\rdblbrace}{\}\mskip-5mu\}}
\usepackage{amsthm}
\theoremstyle{theorem}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}{Proposition}[section]
\newtheorem{corollary}{Corollary}[section]
\newtheorem{lemma}[theorem]{Lemma}
\theoremstyle{definition}
\newtheorem{definition}{Definition}
\theoremstyle{remark}
\newtheorem*{remark}{Remark}
\newenvironment{myquote}
{\begin{quote}\itshape\vspace{-5pt}}
{\vspace{-5pt}\end{quote}}