PICProp: Physics-Informed Confidence Propagation for Uncertainty Quantification
Extracted source text ยท arXiv 2310.06923
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2023
% ready for submission
% \usepackage{neurips_2023}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2023}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2023}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2023}
\usepackage[final, nonatbib]{neurips_2023}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tablesw
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% Additional packages
\usepackage{subcaption}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{remark2}{Remark}[section]
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% todo
\usepackage{todonotes}
%\setuptodonotes{inline}
\newcommand{\kk}[2][]{\todo[inline,linecolor=orange,backgroundcolor=orange!25,bordercolor=orange,#1]{KK: #2}}
\newcommand{\apostolos}[2][]{\todo[inline,linecolor=orange,backgroundcolor=green!25,bordercolor=orange,#1]{Apostolos: #2}}
\newcommand{\qianli}[2][]{\todo[inline,linecolor=orange,backgroundcolor=blue!25,bordercolor=blue,#1]{Qianli: #2}}
\newcommand{\wh}[2][]{\todo[inline,linecolor=orange,backgroundcolor=yellow!25,bordercolor=orange,#1]{WH: #2}}
\newcommand{\pCI}{{$p$-CI}}
\usepackage{algorithm}
\usepackage{algorithmic}
% \usepackage{natbib}
%\bibliographystyle{plainnat}
% figure label in appendix
\usepackage{chngcntr}
\title{PICProp: Physics-Informed Confidence Propagation for Uncertainty Quantification}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Qianli Shen \\
National University of Singapore \\
Singapore \\
\texttt{shenqianli@u.nus.edu} \\
% examples of more authors
\And
Wai Hoh Tang \\
National University of Singapore \\
Singapore \\
\texttt{waihoh.tang@nus.edu.sg} \\
\AND
Zhun Deng \\
Columbia University \\
USA \\
\texttt{zhun.d@columbia.edu}
\AND
Apostolos Psaros \\
Brown University \\
USA \\
\texttt{a.psaros@brown.edu} \\
\And
Kenji Kawaguchi \\
National University of Singapore \\
Singapore \\
\texttt{kenji@nus.edu.sg} \\
}
\begin{document}
\maketitle
\input{tex/intro}
\input{tex/background}
\input{tex/ci}
\input{tex/method}
\input{tex/experiment}
\input{tex/conclusions}
\input{tex/acknowledgements}
% \input{tex/revision_todos}
\medskip
\small
\input{neurips_2023.bbl}
% \bibliography{neurips_2023.bib}
\bibliographystyle{plain}
\appendix
\counterwithin{figure}{section}
\counterwithin{table}{section}
\newpage
\input{tex/appendix}
\end{document}
\begin{abstract}
Standard approaches for uncertainty quantification in deep learning and physics-informed learning have persistent limitations.
Indicatively, strong assumptions regarding the data likelihood are required, the performance highly depends on the selection of priors, and the posterior can be sampled only approximately, which leads to poor approximations because of the associated computational cost.
This paper introduces and studies confidence interval (CI) estimation for deterministic partial differential equations as a novel problem.
That is, to propagate confidence, in the form of CIs, from data locations to the entire domain with probabilistic guarantees.
We propose a method, termed Physics-Informed Confidence Propagation (PICProp), based on bi-level optimization to compute a valid CI without making heavy assumptions.
We provide a theorem regarding the validity of our method, and computational experiments, where the focus is on physics-informed learning. Code is available at \href{https://github.com/ShenQianli/PICProp}{https://github.com/ShenQianli/PICProp}.
\end{abstract}
\section{Introduction}
Combining data and physics rules using deep learning for solving differential equations \cite{raissi2019physics} and learning operator mappings \cite{lu2019deeponet} has recently attracted increased research interest.
In this context, physics-informed machine learning is transforming the computational science field in an unprecedented manner, by solving ill-posed problems, involving noisy and multi-fidelity data as well as missing functional terms, which could not be tackled before \cite{karniadakis2021physics}.
Nevertheless, uncertainty -- mainly due to data noise and neural network over-parametrization -- must be considered for using such models safely in critical applications \cite{psaros2023uncertainty}.
Despite extensive work on uncertainty quantification (UQ) for deep learning \cite{abdar2021review}, standard UQ has persistent limitations.
Indicatively, although for most practical applications where only confidence intervals (CIs) are required, the harder problem of UQ is often solved first using simplifying assumptions, and the CIs are subsequently computed.
Further, strong assumptions regarding the data likelihood are required, the performance highly depends on the selection of priors, and the posterior can be sampled only approximately, which leads to poor approximations because of the associated computational cost.
In this paper, we introduce and study CI estimation for addressing deterministic problems involving partial differential equations (PDEs) as a novel problem.
That is, to propagate confidence, in the form of CIs, from data locations to the entire domain with probabilistic guarantees.
We name our framework Physics-Informed Confidence Propagation (PICProp) and propose a practical method based on bi-level optimization to identify valid CIs in physics-informed deep learning.
Our method can be applied in diverse practical scenarios, such as propagating data CIs, either estimated or given by experts, to the entire problem domain without distributional assumptions.
Indicative applications are provided in Appendix \ref{appendix:applications}.
PICProp provides distinct advantages over traditional UQ approaches in the context of physics-informed deep learning.
Specifically, PICProp does not rely on strong assumptions regarding the data likelihood and constructs theoretically valid CIs.
Further, PICProp offers a unique capability to construct joint CIs for continuous spatio-temporal domains.
In contrast, UQ methods typically construct marginal intervals for discrete locations.
Furthermore, due to the conservative propagation of uncertainty, PICProp produces conservative CIs, which is preferred in critical and risk-sensitive applications \cite{ciosek2020conservative, he2020bayesian}, considering the significant harm of non-valid confidence intervals. This sets it apart from commonly used UQ methods like deep ensembles \cite{deepensembles} and Monte-Carlo dropout \cite{srivastava2014dropout, gal2016dropout}, which are known to be overconfident in practice \cite{foong2019pathologies, ciosek2020conservative}.
% The contribution of this paper is twofold:
% (i) We introduce and study a new problem, which pertains to propagating randomness, in the form of CIs, from data locations to the entire domain, with probabilistic guarantees;
% (ii) We propose a method based on bi-level optimization to identify a valid CI without making heavy assumptions. The method is accompanied by a theorem regarding its validity.
% To the best of our knowledge, this is the first work on joint CI estimation with theoretical guarantees in the context of physics-informed learning.
The paper is organized as follows:
In Section~\ref{sec:background} (and its Appendices~ \ref{appendix:uq_additional} \& \ref{appendix:why_not_mc}), we introduce the PDE problem setup and discuss related work.
In Section~\ref{sec:ci} (and its Appendices~\ref{appendix:ci_construction_w_data}-\ref{appendix:toy_linear_system}), we introduce the problem of CI estimation for PDEs and extensively describe joint solution predictions.
Next, we formulate this novel problem as a bi-level optimization problem in Section~\ref{sec:method} and Appendices~\ref{appendix:proofs} \& \ref{app:algos}, and adopt various algorithms for computing the required implicit gradients in Appendix~\ref{appendix:bo}.
Finally, we evaluate our method on various forward deterministic PDE problems in Section~\ref{sec:experiment} and Appendices \ref{appendix:implementation}-\ref{app:bpinn}, and discuss our findings in Section~\ref{sec:conclusions}.
\section{Background}\label{sec:background}
\input{tex/background/setup}
\input{tex/background/related_works}
\subsection{Problem Setup}\label{sec:setup}
% \subsection{Data-driven Solutions of Nonlinear Partial Differential Equations}\label{sec:setup}
Consider the following general PDE describing a physical or biological system:
\begin{equation}
\begin{aligned}
&\mathbf{\mathcal{F}}[u(x), \lambda(x)] = 0, x\in\Omega, \quad
&\mathbf{\mathcal{B}}[u(x), \lambda(x)] = b(x), x\in\Gamma,
\end{aligned}
\label{eq:pde}
\end{equation}
where $x$ is the $D_x$-dimensional space-time coordinate, $\Omega$ is a bounded domain with boundary $\Gamma$, and $\mathbf{\mathcal{F}}$ as well as $\mathbf{\mathcal{B}}$ are general differential operators acting on $\Omega$ and $\Gamma$, respectively.
The PDE coefficients are denoted by $\lambda(x)$, the boundary term by $b(x)$, and the solution by $u(x)$.
In this paper, we focus on the data-driven forward deterministic PDE problem, where
(i) operators $\mathcal{F}, \mathcal{B}$ and coefficients $\lambda$ are deterministic and known;
(ii) datasets $\mathcal{D}_f = \{x^i_f\}_{i=1}^{N_f}$ for the force condition and $\mathcal{D}_b = \{(x^j_b, b^j)\}_{j=1}^{N_b}$ for the boundary condition are given;
(iii) uncertainty only occurs in the boundary term due to data noise, i.e., $b^j = b(x^j_b) + \epsilon^j$ for $j=1,..., N_b$, and the rest of the dataset is considered to be deterministic without loss of generality.
We denote the function obtained by utilizing the dataset $z = \{\mathcal{D}_f, \mathcal{D}_b\}$ for approximating the solution $u(x)$ of \cref{eq:pde} with a data-driven solver $\mathcal{A}$ as $u_{\mathcal{A}}(x; z)$.
Clearly, different datasets $z$ contaminated by random noise lead to different solutions $u(x; z)$ and thus, to a distribution of solutions at each $x$.
It follows that the solution $u(x; Z)$ can be construed as stochastic, where $Z$ is used to denote the dataset as a random variable and $z$ to denote an instance of $Z$.
In this regard, this paper can be viewed as an effort towards a universal framework, with minimal assumptions, for constructing CIs for solutions of nonlinear PDEs.
\subsection{Related Work}
\input{tex/background/pinn}
\input{tex/background/related_works/uq}
\input{tex/background/related_works/ipm}
\textbf{Physics-Informed Neural Networks}\ \
%Physics-informed machine learning is currently revolutionizing the computational science field \cite{karniadakis2021physics}.
The physics-informed neural network (PINN) method, developed by \cite{raissi2019physics}, addresses the forward deterministic PDE problem of \cref{eq:pde} by constructing a neural network (NN) approximator $u_\theta(x)$, parameterized by $\theta$, for modeling the approximate solution $u(x; z)$.
The approximator $u_{\theta}(x)$ is substituted into \cref{eq:pde} via automatic differentiation for producing $f_{\theta} = \mathcal{F}[u_{\theta}, \lambda]$ and $b_{\theta} = \mathcal{B}[u_{\theta}, \lambda]$.
Finally, the optimization problem for obtaining $\theta$ in the forward problem scenario given a dataset $z$ is cast as
\begin{equation}\label{eq:pinn_loss}
\begin{gathered}
\hat{\theta} = \ \underset{\theta}{\mathrm{argmin}}~\mathcal{L}_{pinn}(\theta, z), \quad \text{where} \quad {L}_{pinn} := \frac{w_f}{N_f}\sum_{i=1}^{N_f} ||f_{\theta}(x^i_f)||_2^2 + \frac{w_b}{N_b}\sum_{i=1}^{N_b} ||b_{\theta}(x^j_b)-b^j||_2^2,
\end{gathered}
\end{equation}
and $\{w_f, w_b\}$ are objective function weights for balancing the various terms in \cref{eq:pinn_loss}; see \cite{psaros2022meta}.
Following the determination of $\hat{\theta}$, $u_{\hat{\theta}}$ can be evaluated at any $x$ in $\Omega$. PINN can be scalled up for high-dimensional PDEs \cite{hu2023tackling}.
\textbf{Uncertainty Quantification for Physics-Informed Learning}\ \
Although UQ for physics-informed learning is not extensively studied, the field is gaining research attention in recent years.
Indicatively,
\cite{zhang2019quantifying} employed dropout to quantify the uncertainty of NNs in approximating the modal functions of stochastic differential equations;
\cite{yang2019adversarial} presented a model based on generative adversarial networks (GANs) for quantifying and propagating uncertainty in systems governed by PDEs;
\cite{yang2020physics} developed a new class of physics-informed GANs to solve forward, inverse, and mixed stochastic problems;
\cite{gao2021wasserstein} further studied Wasserstein GANs for UQ in solutions of PDEs;
\cite{yang2021b} proposed Bayesian PINNs for solving forward and inverse problems described by PDEs with noisy data.
Although this field is always evolving, we refer to it in this paper as ``standard'' UQ in order to distinguish it from our proposed method in Section \ref{sec:method}.
In Appendices \ref{appendix:uq_additional}-\ref{appendix:why_not_mc}, we summarize the limitations of standard UQ, and we direct interested readers to the review paper \cite{psaros2023uncertainty} for more information.
\textbf{Interval Predictor Model (IPM)}\ \
The Interval Predictor Model (IPM) in regression analysis has a similar motivation as our approach.
A function of interest is approximated by an IPM and the bounds on the function are obtained.
\cite{sadeghi2020robust} proposed a technique using a double-loop Monte Carlo algorithm to propagate a CI using the IPM.
The technique requires sampling epistemic and aleatory parameters from uniform distributions to estimate the upper and lower bounds of the IPM.
Our approach is different for two reasons.
First, we fit a PINN model directly and thus, sampling epistemic parameters may not be feasible.
Second, we do not require a sampling procedure.
Instead, we cast the CI estimation as a bi-level optimization problem and solve it using hyper-gradient methods.
\section{Confidence Intervals for Solutions of Nonlinear Partial Differential Equations}\label{sec:ci}
\input{tex/ci/definition}
\input{tex/ci/assumptions}
In this section, we provide the definition of CI for the exact PDE solution and introduce our main assumption.
\begin{definition}[Joint CI for PDE Solution]\label{def:ci4pde_sol}
A pair of functions $(L_Z, U_Z) \in \mathbb{R}^{\Omega} \times \mathbb{R}^{\Omega}$ is a $p$-CI for the exact PDE solution $u(x)$ if the random event $\mathcal{E}:=\{\{L_Z(x) \le u(x) \le U_Z(x),\ \forall x\in\Omega \}$ holds with probability at least $p$.
\end{definition}
Note that this interval refers to the exact solution $u(x)$.
Therefore, a novel problem is posed in Definition~\ref{def:ci4pde_sol}, pertaining to propagating randomness, in the form of CIs, from the data locations, typically on the boundary, to the rest of the domain.
The subscript $Z$ in $(L_Z, U_Z)$ signifies that the CI is random as it depends on the random dataset $Z$.
The available data can be an instance of $Z$, denoted as $z$, or a collection of instances; see Appendix~\ref{appendix:ci_construction_w_data} for the considered problem scenarios.
Such instances can, for example, be drawn from repeating independent experiments.
The subscript $Z$ in $(L_Z, U_Z)$ is dropped if there is no ambiguity.
We also note that the joint CI (also referred to as confidence sequence \cite{lai1976on_cs, waudby2020cs}) in Definition~\ref{def:ci4pde_sol} corresponds to joint predictions $u(\cdot)$ over the whole domain $\Omega$, rather than to marginal predictions $u(x)$ for a given $x$.
The importance of joint predictions as an essential component in UQ has been emphasized in recent works for active learning~\cite{wang2021beyond}, sequential decision making~\cite{wen2021predictions}, and supervised learning~\cite{osband2022neural}.
Nevertheless, the present paper is the first work to consider joint predictions, and general CI estimation, in the context of physics-informed learning.
In Appendix~\ref{appendix:joint_ci}, we provide several theoretical remarks delineating the differences between marginal and joint predictions, as well as unions of bounds, which are typically used in standard UQ.
In Appendix~\ref{appendix:toy_linear_system}, we highlight these differences with a toy example.
Next, to relate the PINN solution with the exact PDE solution $u(x)$ of Definition~\ref{def:ci4pde_sol}, we make the following assumption regarding the noise at the boundary.
\begin{assumption}[Zero-mean Noise]\label{assumption:zero_mean}
The noise on the boundary is zero-mean, i.e., $\mathbb{E}[\epsilon^j] = 0, \forall j\in \{1, \dots, N_b\}$.
\end{assumption}
Next, a clean dataset (without noise), denoted as $\bar{z}$, can be considered as the expectation of $Z$, i.e., $\bar{z} = \mathbb{E}[Z]$.
In this regard, consider a data-driven PDE solver $\mathcal{A}$, which yields an approximate solution $u_\mathcal{A}(x; z)$ given a dataset $z$.
We define the properness of $\mathcal{A}$ based on the concept of a clean dataset.
\begin{definition}[Proper PDE Solver]\label{def:proper}
A PDE solver $\mathcal{A}$ is $\eta$-proper with respect to a clean dataset $\bar{z}$, if $\max_{x\in\Omega} |u_\mathcal{A}(x; \bar{z}) - u(x)| \le \eta$.
\end{definition}
Henceforth, we only consider proper PDE solvers.
That is, a powerful solver is available such that the absolute error of the solution obtained by the solver with a clean dataset is bounded.
In all cases, we assume that such an informative clean dataset exists, but is not available.
Further, the constant $\eta$, which is only related to $\bar{z}$ and $\mathcal{A}$, is considered known and fixed in the computational experiments.
Note that no assumptions are made about $u(x; z)$ for $z\neq\bar{z}$, i.e., there are no assumptions about the PINN solver robustness against noisy data.
Finally, in all NN training sessions we consider no NN parameter uncertainty, although it is straightforward to account for it by combining our method with Deep Ensembles \cite{deepensembles}, for instance.
The aim of this paper is to propose a principled and theoretically guaranteed solution approach and thus, we refrain from applying such ad hoc solutions in the computational experiments and plan to consider parameter uncertainty rigorously in a future work.
Nevertheless, we provide in Appendix \ref{app:bpinn} a comparison with Bayesian PINNs for completeness.
\section{Method}
\label{sec:method}
In this section, we introduce our PICProp method and provide the accompanying theorem, as well as practical implementation algorithms.
Further, we discuss the computational cost of our method and propose a modification for increasing efficiency.
Specifically, based on Definition~\ref{def:ci4pde_sol}, we address the problem of propagating the uncertainty from the boundary to arbitrary locations $x\in\Omega$ of the domain.
Equivalently, this relates to obtaining the corresponding \pCI{}s, i.e., obtaining the functions $(L, U) \in \mathbb{R}^{\Omega} \times \mathbb{R}^{\Omega}$ in Definition~\ref{def:ci4pde_sol}.
\input{tex/method/picprop.tex}
\input{tex/method/efficient.tex}
\subsection{Physics-Informed Confidence Propagation (PICProp)}
\label{sec:confidence_propagation}
Our method comprises two steps:
(i) a \pCI{} $\tilde{\mathcal{Z}}_p$ for the clean data $\bar{z}$ is constructed. (ii) $\tilde{\mathcal{Z}}_p$ is propagated to any point $x \in \Omega$ of the domain by solving the following problem,
\begin{equation}
\begin{gathered}
L(x) = \min_{z \in \tilde{\mathcal{Z}}_p} u_{\mathcal{A}}(x; z) - \eta, \quad U(x) = \max_{z \in \tilde{\mathcal{Z}}_p} u_{\mathcal{A}}(x; z) + \eta,
\end{gathered}
\label{eq:p-CI}
\end{equation}
which searches for configurations of points $z$ within the interval $\tilde{\mathcal{Z}}_p$ such that the predicted values for $u(x)$ given by the $\eta$-proper solver ${\mathcal{A}}$ are minimized or maximized, respectively.
The CI in the first step can either be given or constructed from data.
The former case relates to various practical scenarios (e.g., the error ranges of measurement sensors with a certain confidence degree are known or given by experts),
whereas the latter serves as a data-driven scheme, where less prior knowledge is required.
Techniques to construct the CI in the first step depend on the data (noise) distribution.
For example, if the noise is assumed to be Gaussian, the corresponding CI can be identified by chi-squared or Hotelling's $T$-squared statistics.
If the noise distribution is not known, a concentration inequality can be used to identify the CI.
Relevant details are provided in \cref{appendix:ci_construction_w_data}.
See also \cref{fig:cr} for an illustration.
The interval constructed via \cref{eq:p-CI} is a joint \pCI{} for the whole domain $\Omega$ of $u$.
This fact is formalized in the next theorem (see \cref{appendix:proofs} for the corresponding proof).
\begin{theorem}
Consider a \pCI{} for $\bar{z}$ denoted as $\tilde{\mathcal{Z}}_p$, constructed such that $\text{Pr}(\bar{z}\in\tilde{\mathcal{Z}}_p) \ge p$.
Then the functions $(L, U)$ obtained by solving \cref{eq:p-CI} define a \pCI{} for $u$.
\label{theorem:valid_general_ci}
\end{theorem}
Although the robustness of the PDE solver does not affect the validity of \cref{theorem:valid_general_ci}, it nevertheless affects the quality of intervals constructed via \cref{eq:p-CI}.
Specifically, although valid CIs can always be constructed with arbitrary proper PDE solvers by solving \cref{eq:p-CI}, a robust PDE solver less sensitive to data noise is expected to yield tighter CIs compared to a less robust solver.
Note that the confidence guarantee of the constructed CIs corresponds to a probability that is greater than, rather than equal to, $p$, which means that the constructed CIs are conservative.
From a design perspective, conservative uncertainty estimation is the preferred route for critical and risk-sensitive applications \cite{ciosek2020conservative, he2020bayesian}, although it can be more costly.
%In contrast, most widely used UQ methods, such as deep ensembles \cite{deepensembles} and Monte-Carlo dropout \cite{srivastava2014dropout, gal2016dropout}, are known to be overconfident in practice \cite{foong2019pathologies, ciosek2020conservative}.
Next, a brute force and computationally expensive approach to solve \cref{eq:p-CI} pertains to
traversing $z\in\tilde{\mathcal{Z}}_p$ in an exhaustive search (ES) manner and invoking a PDE solver for each candidate $z$.
Alternatively, we propose to first employ a NN for obtaining $u_{\hat{\theta}(z)}$ by minimizing the physics-informed loss of \cref{eq:pinn_loss} and approximating $u(x; z)$ for each $z\in\tilde{\mathcal{Z}}_p$.
In this regard, if a PINN is assumed to be an $\eta$-proper PDE solver, \cref{eq:p-CI} can be further extended to a bi-level optimization problem cast as
\begin{equation}\label{eq:p-CI:pinn}
\begin{gathered}
L(x) = \min_{z \in \tilde{\mathcal{Z}}_p} u_{\hat{\theta}(z)}(x) - \eta, \quad U(x) = \max_{z \in \tilde{\mathcal{Z}}_p} u_{\hat{\theta}(z)}(x) + \eta,\\
s.t. \quad \hat{\theta}(z) = \arg\min_\theta \mathcal{L}_{pinn}(\theta, z).
\end{gathered}
\end{equation}
Nevertheless, explicit gradient $\nabla_z u$ calculation is intractable, as multiple steps of gradient descent for the NN are required for a reliable approximation.
Despite this fact, bi-level optimization involving large-scale parameterized models such as NNs, with main applications in meta-learning, has been extensively studied in recent years and has resulted to numerous implicit gradient approximation methods (see \cref{appendix:bo}).
In this regard, \cref{alg:picprop} in Appendix~\ref{app:algos} summarizes our method for solving efficiently \cref{eq:p-CI} by using such approximations.
In passing, note that $\eta = 0$ is considered for all implementations in this work.
\subsection{Efficient Physics-Informed Confidence Propagation (EffiPICProp)}\label{sec:efficient}
If joint CIs are required for multiple query points, denoted by the set
$\mathcal{X}_q$, \cref{alg:picprop} has to be used multiple times to solve \cref{eq:p-CI:pinn} for each and every point $x$ in the set.
This clearly incurs a high computational cost.
To address this issue, a straightforward approach pertains to training an auxiliary model parameterized by $\psi$ that models the CI as $(L_{\psi}(x_q), U_{\psi}(x_q))$ for all query points $x_q \in \mathcal{X}_q$, or more generally of the entire domain $\Omega$.
However, this naive regression using solutions of the bi-level optimization at a set of locations, which we refer to as SimPICProp, does not provide accurate results when the size of the query set is small; see computational results in Section~\ref{sec:poisson}.
For improved generalization properties with limited query points, we propose to train a two-input parameterized meta-model $u_{\psi}(x_q, x)$ that models the resulting PDE solution $u(x)$, via \cref{eq:p-CI:pinn}, for each different query point.
For example, note that $L(x)$ in \cref{eq:p-CI:pinn} evaluates the NN $u_{\hat{\theta}(z)}$ at the location of the query point and $z$ minimizes the corresponding value.
As a result, if a trained model $u_{\psi}(x_q, x)$ minimizes the value of $u$ at $x_q$, the sought \pCI{} can be simply obtained by evaluating $u_{\psi}$ at $(x_q, x_q)$.
More generally, given a query set consisting of $K$ randomly selected query points $\mathcal{X}_q = \{x_q^k\}_{k=1}^K$, the model is trained such that it solves the problem
\begin{equation}
\begin{gathered}
\min_{\psi} \frac{1}{2K} \sum_{k=1}^K \mathcal{L}_{pinn}(u_\psi^L(x_q^k, \cdot), z^L(x_q^k)) + \mathcal{L}_{pinn}(u_\psi^U(x_q^k, \cdot), z^U(x_q^k)),\\
\text{where} \quad \ z^L(x_q^k) = \mathop{\arg\min}_{z\in \tilde{\mathcal{Z}}_p}
u_{\hat{\theta}(z)}(x_q^k), z^U(x_q^k) = \mathop{\arg\max}_{z\in \tilde{\mathcal{Z}}_p}
u_{\hat{\theta}(z)}(x_q^k),\\
s.t. \quad \hat{\theta}(z) = \arg\min_\theta \mathcal{L}_{pinn}(u_\theta(\cdot), z).
\end{gathered}
\label{eq:bo_pinn}
\end{equation}
The estimated \pCI{} is, subsequently, given by
\begin{equation}
\begin{gathered}
L(x) = u_\psi^L(x, x), \quad U(x) = u_\psi^U(x, x).
\end{gathered}
\end{equation}
In fact, we can train a single model for both upper- and lower-bound predictions because the parameters are shared as in \cref{eq:bo_pinn}.
This is achieved by encoding an indicator into the input vector, so that $u^L_{\psi} (x_q, \cdot) = u_{\psi}(x_q, \cdot, -1)$ and $u^U_{\psi} (x_q, \cdot) = u_{\psi}(x_q, \cdot, 1)$.
Furthermore, to balance the weights of the terms in the objective of \cref{eq:bo_pinn} more effectively, PINNs are separately trained with boundary conditions $z^L(x_q^{1:K})$ and $z^U(x_q^{1:K})$.
In this regard, $u_{\psi}$ is obtained by
\begin{equation}\label{eq:effipicprop}
\begin{gathered}
\min_{\psi} \ \mathcal{L}_{\text{Effi}}\ (x_q^{1:K}; \psi), \, \text{where} \quad \mathcal{L}_{\text{Effi}} := \frac{1}{2K} \sum_{k=1}^K (1-\lambda) \mathcal{L}(x_q^k, x_q^k; \psi) + \ \lambda \mathop{\mathbb{E}}_{x\in\Omega} \mathcal{L}(x_q^k, x; \psi), \\
s.t. \quad \mathcal{L}(x_q^k, x; \psi) := (u_\psi(x_q^k, x, -1) - u_{\hat{\theta}(z^L(x_q^k))}(x))^2 + (u_\psi(x_q^k, x, 1) - u_{\hat{\theta}(z^U(x_q^k))}(x))^2,
\end{gathered}
\end{equation}
and $\lambda \in [0, 1]$ is a weight to balance the two loss terms, of which the first aims to enforce close predictions of CIs at query points, and the second to utilize extra information in PDE solutions for better generalization.
% By setting $\lambda=0$, the model becomes equivalent to SimPICProp.
We refer to this model as EffiPICProp and provide the corresponding implementation in Algorithm~\ref{alg:effipicprop} in Appendix~\ref{app:algos}.
Specifically, the EffiPICProp version with $\lambda=0$ is called SimPICProp, as it simply focuses on close predictions of confidence intervals at query points, while disregarding additional information at other locations.
\section{Computational experiments}\label{sec:experiment}
We provide three computational experiments that are relevant to physics-informed machine learning.
We will start with a 1D pedagogical example to illustrate our proposed approaches.
Next, we will move to more complicated problems involving standard representative equations met in many real-world problems including heat transfer, fluid dynamics, flows through porous media, and even epidemiology \cite{meng2021multi, zou2022neuraluq}.
Details regarding implementation, query points, and computational times can be found in Appendix \ref{appendix:implementation}.
% \input{tex/experiment/implementation.tex}
\input{tex/experiment/poisson1d.tex}
\input{tex/experiment/poisson2d.tex}
\input{tex/experiment/burgers.tex}
\subsection{A Pedagogical Example}\label{sec:poisson}
We consider the following equation, which has been adopted from the pedagogical example of \cite{gao2021wasserstein},
\begin{equation}
u_{xx} - u^2u_x = f(x), \quad x\in[-1, 1], \quad (\text{PDE}),
\end{equation}where $f(x) = -\pi^2\sin(\pi x) - \pi \cos(\pi x) \sin^2(\pi x).$
For the standard BCs $u(-1)=u(1)=0$, the exact solution is $u(x) = \sin(\pi x)$.
The purpose of this one-dimensional PDE with only two boundary points, i.e., with a two-dimensional search space in \cref{eq:p-CI:pinn}, is to demonstrate the applicability of our method in different data scenarios (see \cref{appendix:ci_construction_w_data}).
Further, the small size of the problem is ideal for comparing computationally expensive methods, such as exhaustive search (ES) and employing PICProp separately at all domain locations, with the more efficient implementation of Algorithm~\ref{alg:effipicprop}.
\begin{figure}[!ht]
\centering\includegraphics[width=0.9\linewidth,height=0.3\linewidth]{img/pedagogical/cr.png}
\caption{\textbf{Pedagogical example:} Boundary $95 \%$ CIs to be propagated to the rest of the domain.
(a-b) Random samples follow $\mathcal{N}(\mathbf{0}, \sigma^2 \mathbf{I})$, where $\sigma=0.05$.
The black crosses correspond to 5,000 random samples from the distribution.
A $\chi^2$ CI is constructed using the only one available sample and a $T^2$ CI is constructed using the five available samples.
(c) Random samples follow $\mathcal{U}([-1.5, 1.5] \times [-1.5, 1.5])$.
The black crosses correspond to 2,000 random samples from the distribution.
A Hoeffding CI is constructed using the five available samples. All available samples are shown with red crosses.}
\label{fig:cr}
\end{figure}
We consider two types of noisy BCs (\cref{fig:cr}). First, we consider BCs with Gaussian noise given as
\begin{equation}
\begin{gathered}
u(-1) \sim \mathcal{N}(0, \sigma_1^2), \quad u(1) \sim \mathcal{N}(0, \sigma_2^2),
\end{gathered}
\end{equation}
where the standard deviations are $\sigma_1 = \sigma_2 = 0.05$, i.e., the data is assumed to be Gaussian and the standard deviations are known (or estimated if unknown).
%The fact that the data distribution is Gaussian is considered as known and the standard deviations are considered either known or unknown.
We sample one or five datapoint(s) for each side of the boundary, for each case, respectively.
Given the dataset, a $\chi^2$ CI and a $T^2$ CI are constructed according to the corresponding problem scenarios discussed in \cref{appendix:ci_construction_w_data}; see also \cref{fig:cr}(a),(b).
Second, we consider BCs with uniform noise given as
\begin{equation}
\begin{gathered}
u(-1) \sim \mathcal{U}(-D_1, D_1), \quad u(1) \sim \mathcal{U}(-D_2, D_2),
\end{gathered}
\end{equation}
where $D_1 = D_2 = 1.5$ are assumed to be known.
We sample five datapoints for each boundary side.
Given the dataset, a Hoeffding CI is constructed; see also \cref{fig:cr}(c).
For the query set, six points are used for implementing SimPICProp and EffiPICProp in \cref{sec:efficient}, whereas 41 points are used for implementing the brute-force version of PICProp, which amounts to solving the bi-level optimization problem of \cref{eq:p-CI:pinn} for each point.
The implementation details of the PICProp-based methods are summarized in \cref{tab:hyperparameters}.
% Our PINN model is a fully connected neural network with two hidden layers and 32 units in each hidden layer.
% A model of the same size is used for predicting CI according to \cref{sec:efficient}.
Further, 5,000 BCs are sampled for solving \cref{eq:p-CI:pinn} with ES, which can be considered as a reference solution.
% For solving the bi-level optimization problem approximately, the \emph{Adam} optimizer with lr$=0.001$ is used for the inner-loop PINN training.
% For the outer-loop optimization, we consider the combinations of hyper-gradient methods \emph{Neumann Series (NS)}, \emph{Conjugate Gradient (CG)}, \emph{Reverse}, and meta-optimizers \emph{Adam} and \emph{SGD} with learning rate selected from $\{0.1, 0.01, 0.001\}$.
% We conclude that the combination of \emph{Neumann Series (NS)} and \emph{SGD} with learning rate $0.01$ stably and efficiently converges to the reference result of ES; see more details in \cref{appendix:bo}.
% Finally, the \emph{Adam} optimizer with lr$=0.001$ is used for PINN training.
\begin{figure}[h]
\centering
\includegraphics[width=0.9\linewidth]{img/pedagogical/chi2_t2_hoeffding.png}
\caption{\textbf{Pedagogical example:} CI predictions for the entire domain $x\in[-1, 1]$.
Left: Noise is Gaussian with known standard deviation, and a $\chi^2$ CI is constructed at the boundary.
Middle: Noise is Gaussian with unknown standard deviation, and a $T^2$ CI is constructed at the boundary.
Right: Uniform noise with known bounds, and a Hoeffding CI is constructed at the boundary.
% Note that the sizes of the CIs are approximately equal to 0.1 -- 0.2 near the boundaries, which are comparable with the propagated CI sizes of \cref{fig:cr}, and become larger around $x$ of $\pm 0.5$.
}
\label{fig:ci}
\end{figure}
The CI predictions for the three cases are visualized in \cref{fig:ci}, whereas the meta-learning curves can be found in \cref{appendix:pedagogical}.
Note that the sizes of the CIs are approximately equal to 0.1 -- 0.2 near the boundaries, which are comparable with the propagated CI sizes of \cref{fig:cr}, and become larger around $x$ of $\pm 0.5$.
In all cases, the PICProp results match those of ES, despite the fact that PICProp requires solving a challenging bi-level optimization method with approximate implicit gradient methods.
Further, the efficient method EffiPICProp performs accurately as compared to its reference method PICProp, while retaining a significantly lower computational cost (see \cref{fig:bo_14all} for $\chi^2$ results and \cref{fig:bo_14all_t2} \ref{fig:bo_14all_hoeffding} for $T^2$ and Hoeffding results).
On the other hand, the predictions of SimPICProp match the brute-force PICProp results only on the query points, but generalize poorly on other locations.
In contrast, EffiPICProp, which is based on a meta-model that fits the PDE solutions for multiple query points, generalizes well on unseen locations.
Some additional experimental results are summarized in Appendix~\ref{appendix:pedagogical} for the completeness of our empirical study: Firstly, a comparison among PICProp and EffiPICProp with various $\lambda$s is provided in \cref{fig:effipicprop_lambdas} to better demonstrate how the balance between two EffiPICProp loss terms in \cref{eq:effipicprop} affects the CI prediction.
Next, we conduct an additional experiment to empirically verify the validity of EffiPICProp CIs in Appendix~\ref{appendix:validity}.
Finally, a brief comparison with Bayesian PINNs is presented in Appendix \ref{app:bpinn}.
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{img/pedagogical/chi2_14all.png}
\caption{\textbf{Pedagogical example}, $\chi^2$: Although the results of both SimPICProp and EffiPICProp match PICProp results on the query points, EffiPICProp generalizes better on unseen
locations.}
\label{fig:bo_14all}
\end{figure}
\subsection{Two-dimensional Poisson equation}\label{sec:poisson2d}
Next, we consider a two-dimensional Poisson equation given as
\begin{equation}
\begin{gathered}
u_{xx} + u_{yy} = f(x, y), \quad (x, y) \in [-1, 1] \times [-1, 1], \quad (\text{PDE}),\\
u(x, y) = e^x + e^y, \quad x = \pm 1 \text{ or } y = \pm 1, \quad (\text{BCs}).
\end{gathered}
\label{eq:poisson2d}
\end{equation}
The exact solution is $u(x, y) = e^x + e^y$, and the corresponding $f(x, y)$ is obtained by substituting $u(x, y)$ into \cref{eq:poisson2d}.
This example serves to demonstrate the effectiveness of our method for treating cases with large numbers of noisy BC datapoints. In such cases, an exhaustive search approach for solving the bi-level optimization of \cref{alg:picprop} generally fails to identify the global optimum, unless we increase the number of trials and the computational cost at prohibitive levels.
We propagate the following fixed CI from the boundary to the rest of the domain $(x, y) \in (-1, 1) \times (-1, 1)$:
\begin{equation}
\begin{gathered}
u(x, y) \in [e^x + e^y - 0.05, e^x + e^y + 0.05], \quad x = \pm 1 \text{ or } y = \pm 1.
\end{gathered}
\label{eq:poisson2d_bc}
\end{equation}
To solve the problem, 10 boundary points are sampled for each side of the domain, i.e., 40 boundary points in total are used for training and uncertainty is considered at all points.
The search space is thus 40-dimensional.
The implementation details of the PICProp-based methods are summarized in \cref{tab:hyperparameters}, whereas 4,000 steps of standard PINN training are run for each BC for ES.
For a fair comparison, 121 query points for PICProp are selected in the domain, see \cref{fig:query}, and the computational time is approximately the same as of ES with 1,000 trials.
Moreover, we include the ES result with 5,000 trials, which is considered to be exhaustive, as a reference.
\begin{figure}[h]
\centering
\includegraphics[width=1.0\linewidth, height=0.5\linewidth]{img/poisson2d/slice_x.png}
\caption{\textbf{2-D Poisson equation}: Slices of CI predictions for $x\in \{-0.5, 0.0, 0.5, 1.0\}$. EffiPICProp can identify the solution to the bi-level optimization problem more efficiently and accurately, as compared with the computationally expensive, search-based method ES.
% For reference, the size of the CIs obtained with EffiPICProp is approximately 0.1 across the domain, which is equal to the size of the propagated CI of \cref{eq:poisson2d_bc}.
}
\label{fig:poisson2d_slice_x}
\end{figure}
\cref{fig:poisson2d_slice_x} plots the slices of CI predictions for $x\in \{-0.5, 0.0, 0.5, 1.0\}$, whereas \cref{appendix:poisson2d} includes slices along the $y$ axis and the heatmaps over the entire domain.
For reference, the size of the CIs obtained with EffiPICProp is approximately 0.1 across the domain, which is equal to the size of the propagated CI of \cref{eq:poisson2d_bc}. The predicted CI size is also larger in most cases than that of the ES-based CIs.
We conclude that 1,000 trials for ES are clearly not sufficient, as the intervals of ES with 5,000 trials are significantly wider; thus, a global optimum is not reached with 1,000 trials.
In contrast, EffiPICProp converges to similar or wider intervals compared to ES with 5,000 trials.
Overall, the optimization-based method EffiPICProp can identify the solution to the bi-level optimization problem more efficiently and accurately, as compared with the computationally expensive, search-based method ES.
\subsection{One-dimensional, time-dependent Burgers equation}\label{sec:burgers}
In this example, we study the Burgers equation given as
\begin{equation}
\begin{gathered}
u_t + u_x - \nu u_{xx} = 0, \quad (x, t) \in [-1, 1] \times [0, 1], \quad (\text{PDE}),\\
u(x, t=0) = -\sin(\pi x), \quad u(x=-1, t) = u(x=1, t) = 0, \quad (\text{BCs}),
\end{gathered}
\label{eq:burger:pde}
\end{equation}where $\nu = \frac{0.01}{\pi}$ is the viscosity parameter.
We consider clean data for the BCs and noisy data for the initial conditions (ICs). Specifically, for the ICs, we propagate the following fixed CI from $t=0$ to the rest of the time domain $t \in (0, 1]$ and for all $x \in [-1, 1]$:
\begin{equation}\label{eq:burgers:ics}
u(x, t=0) \in [-\sin(\pi x) - 0.2, -\sin(\pi x) + 0.2].
\end{equation}
This problem serves to demonstrate the reliability of our method even in challenging cases with steep gradients or discontinuities of the PDE solution.
For solving the problem using PINNs, we consider 200 boundary points, 256 initial points, and 10,000 samples in the interior domain.
The 256 initial points are evenly spaced along $[-1, 1]$.
The implementation details of the PICProp-based methods are summarized in \cref{tab:hyperparameters}, whereas 2,500 and 5,000 trials are considered for ES.
The 246 PICProp query points are visualized in \cref{fig:query} and the respective computational time is approximately equal to that of ES with 2,500 trials.
Slices of CI predictions for $t\in \{0.0, 0.25, 0.5, 0.75\}$ are plotted in \cref{fig:burgers_es_mc}, whereas \cref{appendix:burgers} includes the respective meta-learning curves.
We conclude that EffiPICProp, although computationally cheaper, in most cases provides wider, i.e., more conservative, CIs, as compared to ES with varying numbers of trials.
It also succeeds in capturing the expected increase in uncertainty as the query points move away from the source of uncertainty, which corresponds to larger $t$ values.
Specifically, the size of the CIs at $t=0$ is approximately 0.4, which is equal to the size of the propagated CI in \cref{eq:burgers:ics}, and becomes approximately 2 at $t=0.75$ and near $x = 0$.
On the other hand, since we consider 256 random initial points, the search space of ES is 256-dimensional. It requires a prohibitively large number of trials to identify the global optima of \cref{eq:p-CI:pinn}.
Finally, note that the results of EffiPICProp in \cref{fig:burgers_es_mc} correspond to $\lambda = 0$ in \cref{eq:effipicprop}, because we found that the additional information provided by the second loss term in \cref{sec:efficient} adversely affects generalization.
This is partly due to the large number of query points in this problem.
\begin{figure}[h]
\centering
\includegraphics[width=1.0\linewidth, height=0.5\linewidth]{img/burgers/slice2.png}
\caption{\textbf{Burgers equation}: Slices of CI predictions for $x\in \{0.0, 0.25, 0.5, 0.75\}$. EffiPICProp, although computationally cheaper, in most cases provides wider, i.e., more conservative CIs, as compared to ES with varying numbers of trials.
% For reference, the size of the CIs at $t=0$ is approximately 0.4, which is equal to the size of the propagated CI in \cref{eq:burgers:ics}, and increases to approximately 2 at $t=0.75$ and near $x = 0$.
To interpret the cost of ES, note that its search space is 256-dimensional and requires a prohibitively large number of trials to identify the global optima of \cref{eq:p-CI:pinn}.
}
\label{fig:burgers_es_mc}
% \vskip -0.5in
\end{figure}
% ----------------------------------------------------------------
% Comparing across x values
% \begin{figure}[H]
% \centering
% \begin{subfigure}[b]{0.4\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/burgers_t0_v0.png}
% \caption{$t = 0.0$}
% \label{fig:burgers_es_mc_t0.0}
% \end{subfigure}
% %
% \begin{subfigure}[b]{0.4\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/burgers_t25_v0.png}
% \caption{$t = 0.25$}
% \label{fig:burgers_es_mc_t0.25}
% \end{subfigure}
% \begin{subfigure}[b]{0.4\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/burgers_t50_v0.png}
% \caption{$t = 0.5$}
% \label{fig:burgers_es_mc_t0.5}
% \end{subfigure}
% %
% \begin{subfigure}[b]{0.4\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/burgers_t75_v0.png}
% \caption{$t = 0.75$}
% \label{fig:burgers_es_mc_t0.75}
% \end{subfigure}
% \caption{\textbf{Burgers equation:} Exhaustive search includes the two extreme cases which all the initial conditions are either all simultaneously set to a maximum or a minimum value, i.e. $-\sin(\pi x) \pm 2\sigma$, as explained in Section~\ref{sec:burgers}. Exhaustive search * (in red) is the 1,000 exhaustive search experiments without the two extreme cases and we can see the boundaries obtained are narrower. We propagate fixed-bounds (i.e. problem scenario 5) of 246 ($x$, $t$) coordinates through our efficient method (PICProp-efficient) in this example. As PICProp-efficient is an approximation by a neural network given multiple (but small) number of solutions of the hyper-gradient \emph{Reverse} method, the curves are not smooth, for example around $x=0$ in Figure (a) and it does not coincide fully with Exhaustive search A in Figure (d) at around $x=-0.2$ and $x=0.2$. Nonetheless, we demonstrate that PICProp-efficient provides good bounds similar to the Exhaustive search with using only a small amount of data.}
% %Interestingly, we also include the results of MC experiments, in which we can observe sharp spikes and dips, especially at high $t$ and at $x$ close to zero. These are due to the skewed and non Gaussian distribution of $u$, which is illustrated in Fig.~\ref{fig:burgers_histogram_t0.75}.}
% \label{fig:burgers_es_mc_}
% \end{figure}
% \iffalse
% In Fig.~\ref{fig:burgers_es_mc} we provide the results for multiple query points.
% We observe that 1) exhaustive search typically produces a larger interval than MC, and 2) a notable spike and dip in the 2 standard deviations from the mean of Monte Carlo results near $x = 0$ as $t$ increases, for instance at $t = 0.5$ and $t = 0.75$, as shown by the red plots in Figs~\ref{fig:burgers_es_mc_t0.5} and \ref{fig:burgers_es_mc_t0.75}, respectively.
% Diving deeper into the Monte Carlo results, we observe from the histograms in Fig.~\ref{fig:burgers_histogram_t0.75} that the predicted values are skewed and/or do not follow a Gaussian distribution.
% This explains the spike and the dip in the vicinity of $ x = 0$.
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.7\linewidth]{img/burgers/burgers_histogram.png}
% \caption{\textbf{Burgers equation:} Distribution of the Monte Carlo data at $x = -0.0039$ and $x = +0.0039$, at $t = 0.75$ do not follow a Gaussian distribution. The figure on the left shows a left-skewed plot while the on the right is bimodal. The standard deviations are large. Consequently, computing 2 standard deviations away from the mean of Monte Carlo data for values that are close to $x = 0$ results in the spike (and the dip) seen in Figs.~\ref{fig:burgers_es_mc_t0.5}-\ref{fig:burgers_es_mc_t0.75}.}
% \label{fig:burgers_histogram_t0.75}
% \end{figure}
% \fi
\section{Conclusion}\label{sec:conclusions}
Physics-informed machine learning is transforming the computational science field in an unprecedented manner, by solving ill-posed problems, involving noisy and multi-fidelity data as well as missing functional terms, which could not be tackled before.
% However, uncertainty quantification is necessary for deploying such methods in critical applications involving physical and biological systems.
However, uncertainty quantification is necessary for deploying such methods in critical applications.
In this work, we considered a novel problem pertaining to propagating randomness, in the form of CIs, from data locations (at the boundary but can easily be extended to inner or collocation data points) to the entire domain, with probabilistic guarantees.
We proposed PICProp, a method based on bi-level optimization, and its efficient version, with a theorem guaranteeing the validity of the obtained CIs (Algorithms~\ref{alg:picprop}-\ref{alg:effipicprop} and \cref{theorem:valid_general_ci}).
To the best of our knowledge, this is the first work on joint CI estimation with theoretical guarantees in the context of physics-informed learning, while our method can be applied in diverse practical scenarios without imposing any distributional assumptions.
%In this regard, Appendices~\ref{appendix:joint_ci} and \ref{appendix:toy_linear_system} include several theoretical and experimental remarks delineating the differences between marginal and joint predictions, as well as unions of bounds, which are typically used in standard UQ.
In the computational experiments of \cref{sec:experiment}, involving various partial differential equations, we demonstrated the effectiveness and efficiency of our proposed methods.
Specifically, the predicted CIs are wider, i.e. more conservative, in most cases, as compared with the more computationally demanding exhaustive search for solving the bi-level optimization.
Although not encountered in the experiments, a potential limitation of our method is that the obtained CIs can be loose in some cases, even when propagating a tight CI from the boundaries, due to the max/min operations in the bi-level optimization of \cref{eq:p-CI}.
Because of the theoretical challenges involved, we plan to study this limitation and perform pertinent experiments in a future work.
\section{Acknowledgements}
This material is based upon work supported by the Google Cloud Research Credit program with the award (6NW8-CF7K-3AG4-1WH1). The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (\href{https://www.nscc.sg}{https://www.nscc.sg}).
% \textbf{Example 1}
% \begin{equation}
% \begin{aligned}
% \theta =& (\theta_1, \theta_2) \in \mathbb{R}^2, \\
% Z =& (Z_1, Z_2) \sim \mathcal{N}(\theta, \sigma^2I),\\
% u_\theta(x) =& (1-x)\theta_1 + x\theta_2, \mathcal{X} = [0, 1]\\
% l_Z(x) =& \begin{cases} Z_1 - z^*\sigma & x = 0\\ -\infty & x \in (0, 1) \\Z_2 - z^*\sigma & x = 1 \end{cases}\\
% u_Z(x) =& \begin{cases} Z_1 + z^*\sigma & x = 0\\ \infty & x \in (0, 1) \\Z_2 + z^*\sigma & x = 1 \end{cases}
% \end{aligned}
% \end{equation}
% where $z^*$ is the upper $(1-p)/2$ critical value of the standard normal distribution for desired confidence level $p < 1$. $(l_Z(x), u_Z(x))$ is a valid $p-$confidence interval for $u_\theta(x)$, while
% \begin{equation}
% \begin{aligned}
% \text{Pr}(\mathcal{E}(Z)) =& \text{Pr}(\{l_Z(x) < u_\theta(x) < u_Z(x), \forall x \in \mathcal{X}\}) \\
% \ge& \text{Pr}(\{l_Z(0) < u_\theta(0) < u_Z(0)\} \cap \{l_Z(1) < u_\theta(1) < u_Z(1)\}) \\
% =& \text{Pr}(\{Z_1 - z^*\sigma < \theta_1 < Z_1 + z^*\sigma\} \cap \{Z_2 - z^*\sigma < \theta_2 < Z_2 + z^*\sigma\})\\
% =& \text{Pr}(\theta_1 - z^*\sigma < Z_1 < \theta_1 - z^*\sigma) \text{Pr}(\theta_2 - z^*\sigma < Z_2 < \theta_2 - z^*\sigma)\\
% =& p^2 < p
% \end{aligned}
% \end{equation}
% Consequently, even though methods for getting a confidence interval for $u_\theta(x)$ at a given $x$ are well studied, there are few works on methods to get a confidence interval for $u_\theta$ as far as we know.
\input{tex/appendix/applications.tex}
\input{tex/appendix/uq_additional.tex}
\input{tex/appendix/why_not_mc.tex}
\input{tex/appendix/ci_with_data.tex}
\input{tex/appendix/joint_ci.tex}
\input{tex/appendix/toy.tex}
\input{tex/appendix/proof.tex}
\input{tex/appendix/algorithms}
\input{tex/appendix/bo.tex}
% \input{tex/appendix/why_not_uq.tex}
\input{tex/appendix/implementation_details}
\input{tex/appendix/experiment.tex}
% \input{tex/appendix/computational_time.tex}
\input{tex/appendix/bpinn.tex}
\section{Indicative applications}\label{appendix:applications}
Our proposed method, delineated in
Algorithms \ref{alg:picprop} and \ref{alg:effipicprop} of Appendix \ref{app:algos}, provides a way to quantify how the data uncertainty affects the obtained solution. In
addition, if the unseen clean data lies within the constructed/given data bounds, the ground-truth solution
lies within our predicted CIs. Some indicative applications include:
\begin{enumerate}
\item In additive manufacturing problems, understanding, through our computed CIs, how the model responds to different material properties arising from the manufacturing process can help in model correction and thus quality control (see \cite{wang2020uncertainty}).
\item Further, in material science, cracks affect the elasticity constants which together with geometric properties affect deformations (see \cite{bharadwaja2022physics}). The range of these deformations can be identified with our method. Accurately predicting the upper bound of such deformations can help designers to adjust cross-sectional properties of structural elements to better account for potential cracks.
\item More generally, our model can be extended to account for PDE model uncertainty. For example, PINNs have been used for short-term predictions of infected cases and deaths due to COVID-19 (see \cite{cai2022fractional}). In such problems, our approach can be used to provide bounds of uncertainty for the predicted variables of interest. Accurately predicting the upper bound of infected cases can help the states better prepare for short-term hospitalization needs.
\end{enumerate}
\section{Additional details regarding standard UQ}\label{appendix:uq_additional}
Standard UQ considers that the data is generated from a distribution $p(u|x, \theta=\theta^*)$, where $\theta^*$ is the NN parameters that are assumed to represent exactly the PDE solution $u(x)$.
Given a dataset $z$, standard UQ approximates the solution $u_{\theta^*}(x)$ with the distribution
\begin{equation}
\begin{aligned}
p(u|x; z) = \int p(u|x, \theta) p(\theta|z) d\theta,
\end{aligned}
\end{equation}
where $p(\theta|z) \propto p(\theta)p(z|\theta)$ according to Bayes' theorem and $p(z|\theta)$ is the data likelihood.
However, as a closed-form $p(\theta|z)$ is typically intractable, UQ draws approximate samples from $p(\theta|z)$, denoted as $\{\hat{\theta}_j\}_{j=1}^{N_{\theta}}$.
Next, it constructs $\hat{p}(u|x; z)$ for each $x$ based on the collected samples to approximate $p(u|x; z)$.
Typically, a Gaussian distribution is fitted to the PINN solutions $\{u_{\hat{\theta}_j}(x)\}_{j=1}^{N_{\theta}}$ and its corresponding statistical intervals are used in lieu of CIs for the PDE solution.
Loosely speaking, statistical intervals of $\{\hat{\theta}_j\}_{j=1}^{N_{\theta}}$ can be viewed as credible intervals from Bayesian statistics, and intervals of $\{u_{\hat{\theta}_j}(x)\}_{j=1}^{N_{\theta}}$
as the corresponding prediction intervals.
Alternatively, one could use a concentration inequality to construct directly a prediction interval using the samples $\{u_{\hat{\theta}_j}(x)\}_{j=1}^{N_{\theta}}$.
Nevertheless, such intervals still correspond to the Bayesian estimate $\mathbb{E}_{\theta\sim p(\theta | \mathcal{D})} u_{\theta}$, rather than to the exact solution $u(x)$, unless further assumptions are made.
Even if the assumption that the Bayes estimate corresponds to $\theta^*$ is made, i.e., $\theta^* = \mathbb{E}_{\theta|\mathcal{D}} \theta$, the typically considered $\mathbb{E}_{\theta|\mathcal{D}} u_\theta(x)$ is not guaranteed to equal $u_{\mathbb{E}_{\theta|\mathcal{D}} \theta}(x)$ as $u_\theta$ can be arbitrary nonlinear (see also \cref{appendix:why_not_mc}).
Standard UQ approaches have several other limitations.
For example, strong assumptions regarding the noise distribution and data likelihood are often required, and depending on the problem it is notoriously difficult to tune Bayesian NNs \cite{brosse2018promises}.
Furthermore, it is not possible to obtain/sample $p(u|x, z)$ exactly in practical problems and thus, additional approximations are required, typically referred to as \emph{approximate posterior inference}.
In addition, UQ approaches for PINNs are not supported by theoretical guarantees regarding the quality of uncertainty propagation due to the physics-informed constraints~\cite{gao2021wasserstein}.
Finally, for the case of multiple available datasets, see Appendices~\ref{appendix:why_not_mc} and \ref{appendix:joint_ci} for the limitations of Monte Carlo-based approaches.
\section{Additional details regarding Monte Carlo Simulation}\label{appendix:why_not_mc}
Assume that we have $m$ i.i.d. datasets denoted as $\{z^{(j)}\}_{j=1}^m$ and a $0$-proper data-driven PDE solver (see Definition~\ref{def:proper}).
It is natural to consider solving the PDE with each dataset in a Monte Carlo manner and then using a concentration inequality to construct CIs.
However, a CI for the exact solution $u(x)$ cannot be identified without further assumptions.
The reason is that we only have access to $\mathbb{E}[u(x; Z)]$, rather than to $u(x)$ given as $u(x) = u(x; \bar{Z}) = u(x; \mathbb{E}[Z])$, considering data-driven solutions from a $0$-proper solver.
Clearly, the difference between the two is in the form of $\mathbb{E}[f(X)] \neq f(\mathbb{E}[X])$, where $f$ in our case is the solver.
In \cref{fig:fex_vs_efx} we provide an illustration of this point as applied to linear/convex/concave functions, as s straightforward consequence of Jensen's inequality.
\begin{figure}[ht]
\centering
\includegraphics[width=0.9\linewidth]{img/fex_vs_efx.pdf}
\caption{Difference between $\mathbb{E}[f(X)]$ and $f(\mathbb{E}[X])$.
(a) Linear function $f(x)=x$, $\mathbb{E}[f(X)] = f(\mathbb{E}[X])$;
(b) Convex function $f(x)=(e^x - 1) / (e - 1)$, $\mathbb{E}[f(X)] > f(\mathbb{E}[X])$;
(c) Concave function $f(x)=\sin(\pi x/2)$, $\mathbb{E}[f(X)] < f(\mathbb{E}[X])$.
}
\label{fig:fex_vs_efx}
\end{figure}
\section{Confidence Interval Construction with Data}
\label{appendix:ci_construction_w_data}
Consider a dataset $\{z^{(j)}\}_{j=1}^m$, which contains $m$ independent samples of an $n$-dimensional random vector $Z$.
Based on the data $\{z^{(j)}\}_{j=1}^m$, we construct a CI for $Z$, denoted as $\tilde{\mathcal{Z}}_p$, where $p$ is the confidence level.
Equivalently, $Pr(\bar{z} \in \tilde{\mathcal{Z}}_p) \geq p$.
In this regard, we consider the following problem scenarios:
\begin{enumerate}
\item The random vector $Z$ follows a multivariate Gaussian distribution with known covariance matrix $\Sigma$ of rank $n$.
Given one observation $z$, i.e., $m=1$, the chi-squared statistic $\chi^2 = (z - \mu)^T \Sigma^{-1} (z - \mu)$ follows a chi-squared distribution with $n$ degrees of freedom.
The corresponding \pCI{} is $\tilde{\mathcal{Z}}_p = \{z' |\ ||z - z'||_{\Sigma^{-1}} \le k^{\chi^2}_{n,p}\}$, where $||z-z'||_{\Sigma^{-1}} = \sqrt{(z - z')^T \Sigma^{-1} (z - z')}$ and $k^{\chi^2}_{n,p}$ is the squared root of the upper $(1-p)/2$ critical value of the chi-squared distribution with $n$ degrees of freedom; see \cref{fig:cr} for an example corresponding to the case of $n=2$.
\item The random vector $Z$ follows a multivariate Gaussian distribution with unknown covariance matrix $\Sigma$ and $m$ observations are available.
The sample mean and covariance are estimated as $\bar{z} = \frac{1}{m} \sum_{i=1}^{m} z_i$ and $\hat{\Sigma} = \frac{1}{m-1}\sum_{i=1}^{m}(z_i - \bar{z})(z_i - \bar{z})^T$, respectively.
The Hotelling's $t$-squared statistic $t^2 = N(\bar{z} - \mu)^T \hat{\Sigma}^{-1} (\bar{z} - \mu)$ follows a Hotelling's $T$-squared distribution with parameters $n$, $m-1$.
The corresponding \pCI{} is $\tilde{\mathcal{Z}}_p = \{z' |\ ||\bar{z}-z'||_{\hat{\Sigma}^{-1}} \le m^{-1/2}k^{T^2}_{n, m-1, p}\}$, where $||\bar{z}-z'||_{\hat{\Sigma}^{-1}} = \sqrt{(\bar{z} - z')^T \hat{\Sigma}^{-1} (\bar{z} - z')}$ and $k^{T^2}_{n, m-1, p}$ denotes the squared root of the upper $(1-p)/2$ critical value of the aforementioned Hotelling's $T$-squared distribution; see \cref{fig:cr} for an example corresponding to the case of $n=2$ and $m=5$.
\item The components of the random vector $Z$ follow an unknown distribution, and $m$ independent observations are available.
For this case, a concentration inequality can be used for each component, i.e., for each data location, and the obtained bounds can be combined using a union of bounds for obtaining a \pCI{} (see also \cref{appendix:joint_ci}).
Note that many popular concentration inequalities require specific assumptions for the unknown distribution besides the fact that independent samples are available.
For example, Hoeffding's inequality and its variants require the random variable to be bounded.
\cite{bentkus2008extension} extends Hoeffding's inequality to unbounded random variables, but it is limited to Pareto-type tails.
Other concentration inequalities without heavy assumptions for the distribution, such as Chebyshev's inequality and Chernoff's inequality, require various true population statistics (e.g., the population mean and variance) for computing the bounds, which are not available in the cases we consider.
In this regard, \cite{saw1984chebyshev} provides a Chebyshev-type bound with estimated mean and variance.
\cite{stellato2017multivariate} further extends the bound of \cite{saw1984chebyshev} to the multivariate case.
\item The CI $\tilde{\mathcal{Z}}_p$ can alternatively be given by domain experts.
\end{enumerate}
\section{Joint Confidence Intervals for PDE Solutions}
\label{appendix:joint_ci}
In this section, we provide the necessary definitions, lemmas, and remarks that are required for introducing and addressing the CI propagation problem in Section~\ref{sec:confidence_propagation}.
That is, how to propagate, with probabilistic guarantees, a CI constructed at the locations of the available data to arbitrary sets of query points of the domain.
To this aim, we first define the marginal CI for a single query point $x_q$.
\begin{definition}[Marginal CI]\label{def:sCI_}
A pair of random variables $(L, U) \in \mathbb{R}^2$ is a \pCI{} for $u(x_q)$ on the single query point $x_q \in \Omega$, if the random event $\mathcal{E}:=\{L < u(x_q) < U\}$ holds with probability at least $p$.
\end{definition}
Intuitively, considering repeating experiments of drawing random datasets $\{z^{(j)}\}_{1:m}$, the proportion of the $(L, U)$ CIs, constructed according to Definition~\ref{def:sCI_}, that contain $u(x_q)$ is at least $p$.
Next, we define the joint CI for multiple query points.
\begin{definition}[Joint CI]\label{def:genCI_}
A pair of functions $(L, U) \in \mathbb{R}^{\mathcal{X}_q} \times \mathbb{R}^{\mathcal{X}_q}$ is a \pCI{} for $u$ on query set $\mathcal{X}_q \subseteq \Omega$, if the random event $\mathcal{E}:=\cap_{x_q\in\mathcal{X}_q}\{L(x_q) < u(x_q) < U(x_q)\}$ holds with probability at least $p$.
\end{definition}
For simplicity, in the rest of the paper we refer to the \pCI{} for $u$ on the whole domain $\Omega$ as \textit{\pCI{} for $u$}, and to the \pCI{} for $u$ on $\mathcal{X}_q\subset\Omega$ as \textit{query set \pCI{}}.
\begin{remark}
Let $(L, U)$ be a \pCI{} for $u$.
For any $\mathcal{X}_q \subseteq \Omega$, $(L, U)$ is a \pCI{} for $u$ on $\mathcal{X}_q$, i.e., the random event $\mathcal{E}(\mathcal{X}_q, Z):=\cap_{x_q\in\mathcal{X}_q}\{L(x_q) < u(x_q) < U(x_q)\}$ holds with probability at least $p$.
\label{remark:general2multiple}
\end{remark}
Remark~\ref{remark:general2multiple} indicates that a \pCI{} for $u$ on an arbitrary query set can be derived from a \pCI{} for $u$ by slicing.
The query set $\mathcal{X}_q$ can be either discrete or continuous including multiple or even infinite queries.
Note that the \pCI{}s of Definition~\ref{def:genCI_} and Remark~\ref{remark:general2multiple} hold with probability at least $p$; i.e., there is no statement about their tightness.
In this regard, the next remark shows that slicing using larger sets $\mathcal{X}_q$ yields tighter \pCI{}s for $u$ on $\mathcal{X}_q$.
Nevertheless, the maximum tightness that can be achieved, for the \pCI{} for $u$ on $\mathcal{X}_q$, is determined by the tightness of the \pCI{} for $u$ on the whole domain $\mathcal{X}_q$.
This is corroborated by the following remark.
\begin{remark}
Let $(L, U) \in \mathbb{R}^{\Omega} \times \mathbb{R}^{\Omega}$ be a \pCI{} for $u$.
For query set $\mathcal{X}_q^1 \subseteq \mathcal{X}_q^2 \subseteq \Omega$, the probabilities of random events $\mathcal{E}_1:=\cap_{x_q\in\mathcal{X}_q^1}\{L(x_q) < u(x_q) < U(x_q)\}$, $\mathcal{E}_2:=\cap_{x_q\in\mathcal{X}_q^2}\{L(x_q) < u(x_q) < U(x_q)\}$ satisfy
\begin{equation}
\begin{aligned}
\text{Pr}(\mathcal{E}_1 \ge \text{Pr}(\mathcal{E}_2).
\end{aligned}
\end{equation}
\label{remark:larger:set}
\end{remark}
Next, we define the union of single-query \pCI{}s.
It is often used in practice as a proxy to the query set \pCI{} and can be used also within the context of the present study for addressing problem scenarios 3 and 4, as described in \cref{appendix:ci_construction_w_data}.
\begin{definition}[Union of CIs]\label{def:unionCI}
A pair of functions $(L, U) \in \mathbb{R}^{\mathcal{X}_q} \times \mathbb{R}^{\mathcal{X}_q}$ constructed based on $\{z^{(j)}\}_{1:m}$ is a union of \pCI{}s for $u$ on query set $\mathcal{X}_q \subseteq \Omega$ if for each $x_q \in \mathcal{X}_q$, $(L(x_q), U(x_q))$ is a \pCI{} for $u(x_q)$, i.e., each random event $\mathcal{E}:=\{L(x_q) < u(x_q) < U(x_q)\}$ holds with at least probability $p$.
\end{definition}
The next lemma provides insight into the connection between the marginal \pCI{} on $\mathcal{X}_q$ and the union of \pCI{}s.
\begin{lemma}
For $u$ and $\mathcal{X}_q$, a \pCI{} is a union of \pCI{}s, whereas a union of \pCI{}s is not necessarily a \pCI{}.
\label{remark:general_vs_union}
\end{lemma}
That is, $u(x_q), \forall x_q \in \mathcal{X}_q,$ belongs to the \pCI{} constructed by slicing of the \pCI{} for $u$ with probability at least $p$.
On the other hand, the probability of the event $\cap_{x_q\in\mathcal{X}_q}\{L(x_q) < u(x_q) < U(x_q)\}$, where $(L, U)$ are constructed based on a union of \pCI{}s, may be less than $p$.
This is supported by the following two remarks as well as the computational experiment of Appendix~\ref{appendix:toy_linear_system}.
\begin{remark}
The probability of the random event $\cap_{x_q\in\mathcal{X}_q} \{L(x_q) < u(x_q) < U(x_q)\}$, where $(L, U)$ are constructed based on a union of \pCI{}s, is upper-bounded by the minimum probability of the events $\{L(x_q) < u(x_q) < U(x_q)\}, \forall x_q \in \mathcal{X}_q$.
That is,
\begin{equation}
\begin{aligned}
&\text{Pr}(\cap_{x_q\in\mathcal{X}_q} \{L(x_q) < u(x_q) < U(x_q)\})\\
\le& \min_{x_q\in\mathcal{X}_q} \text{Pr}(\{L(x_q) < u(x_q) < U(x_q)\}).
\end{aligned}
\end{equation}
\end{remark}
\begin{remark}
If one of the \pCI{}s is tight, i.e. $\exists x'_q \in \mathcal{X}_q \text{ such that } \text{Pr}(\{L(x'_q) < u(x'_q) < U(x'_q)\}) = p$, the union of these \pCI{}s is a \pCI{} for $u$ if and only if it holds $\forall x_q \in \mathcal{X}_q \setminus \{x'_q\}$ that
\begin{equation}
\Pr(\{ L(x_q) < u(x_q) < U(x_q) | L(x'_q) < u(x'_q) < U(x'_q)\}) = 1.
\end{equation}
\end{remark}
Finally, in order to obtain a guarantee that a union of CIs is a \pCI{}, we can equate a lower bound of the probability of the former with $p$.
This lower bound can be obtained by disregarding correlations among the different locations, as formalized in the following lemma.
\begin{lemma}
For $u$ and finite query set $\mathcal{X}_q$, a union of $(1 - (1-p)/|\mathcal{X}_q|)$-CIs is a \pCI{} for $u$ as
\end{lemma}
\begin{equation}
\begin{aligned}
& \text{Pr}(\cap_{x_q\in\mathcal{X}_q}\{L(x_q) \le u(x_q) \le U(x_q)\})\\
=& 1 - \text{Pr}(\cup_{x\in\mathcal{X}_q} \{L(x)\le f(x) \le U(x)\}^C)\\
\ge& 1 - \sum_{x\in\mathcal{X}_q} (1 - \text{Pr}(\{L(x)\le u(x) \le U(x)\}))\\
\ge& 1 - |\mathcal{X}_q|(1 - (1 - \frac{1 - p}{|\mathcal{X}_q|})) = p.
\end{aligned}
\end{equation}
However, as the size of the query set $\mathcal{X}_q$ increases, the required confidence level of each interval must approach 1, which leads to loose union bounds.
Finally, another approach for obtaining a \pCI{} for multiple queries is to use multivariate concentration inequalities~\cite{eaton1991concentration}.
Nevertheless, this approach is limited to a finite number of queries.
\section{Toy Linear System}
\label{appendix:toy_linear_system}
In this section, we highlight the differences between joint CIs and union of CIs, as described in Appendix~\ref{appendix:joint_ci}, with a simple example.
Consider a true function $u$ and a model given as $u_\theta(x) = (1-x)\theta_0 + x\theta_1, x \in [0, 1]$, with $\theta = (\theta_0, \theta_1) \in \mathbb{R}^2$.
We assume that there exists $\theta^*$ such that $u_{\theta^*} = u$, and that we are given measurements $Z = (Z_0, Z_1) \sim \mathcal{N}(\theta^*, I)$.
Thus, for one observation $Z$, we simply estimate $\hat{\theta} = Z$.
As a result, the estimator for $u$ is given as
\begin{equation}
u_{\hat{\theta}} = (1-x) Z_0 + x Z_1 \sim \mathcal{N}((1-x) \theta_0^* + x \theta_1^*, 2x^2 - 2x + 1).
\end{equation}
Next, define as $k$-interval the lower and upper intervals $(L_k, U_k)$ obtained by subtracting and adding $k$ standard deviations from the estimated mean of $u$, respectively, i.e.,
\begin{equation}
\begin{aligned}
L_k(x) :=& (1-x) Z_0 + x Z_1 - k\sqrt{2x^2 - 2x + 1}, \\
U_k(x) :=& (1-x) Z_0 + x Z_1 + k\sqrt{2x^2 - 2x + 1}.
\end{aligned}
\end{equation}
\begin{remark}
We define $k^\mathcal{N}_{p}$-interval as the interval constructed as a union of one-query \pCI{}s for $u_\theta(x)$ and for different values of $x \in \Omega$, similarly to Definition~\ref{def:unionCI}.
The value $k^\mathcal{N}_{p}$ is the upper $(1-p)/2$ critical value of the standard normal distribution.
\end{remark}
\emph{Proof}: For $x\in\Omega$,
\begin{equation}
\begin{aligned}
& \text{Pr}(\{L_{k^\mathcal{N}_{p}} \le u(x) \le U_{k^\mathcal{N}_{p}}\})\\
=& \text{Pr}(\{(1-x) Z_0 + x Z_1 - k^\mathcal{N}_{p}\sqrt{2x^2 - 2x + 1} < f(x) < (1-x) Z_0 + x Z_1 + k^\mathcal{N}_{p}\sqrt{2x^2 - 2x + 1}\}\\
=& \text{Pr}(\{|u_{\hat{\theta}}(x) - \mathbb{E}[ u_{\hat{\theta}}(x)] | < k^\mathcal{N}_{p} \text{std}(u_{\hat{\theta}}(x))\}\\
=& p.
\end{aligned}
\end{equation}
\begin{remark}
We define $k^{\chi^2}_{p}$-interval as the \pCI{} for $u$, constructed as $(L_{k^{\chi^2}_{p}}(x), U_{k^{\chi^2}_{p}}(x))$ for each $x \in \Omega$, where
\begin{equation}
\begin{aligned}
L_{k^{\chi^2}_{p}}(x) = \mathop{\arg\min}_{||z - Z||_2 \le k^{\chi^2}_{p}} (1-x)z_0 + xz_1,\\
U_{k^{\chi^2}_{p}}(x) = \mathop{\arg\max}_{||z - Z||_2 \le k^{\chi^2}_{p}} (1-x)z_0 + xz_1.
\end{aligned}
\end{equation}
The value $k^{\chi^2}_{p}$ is the squared root of the upper $(1-p)/2$ critical value of the chi-squared distribution with 2 degrees of freedom.
\end{remark}
\emph{Proof}: According to Theorem ~\ref{theorem:valid_general_ci},
\begin{equation}
\begin{aligned}
& \text{Pr}(\{L_{k^{\chi^2}_{p}}(x) < u(x) < U_{k^{\chi^2}_{p}(x)}, \forall x\in \Omega\})\\
\ge& \text{Pr}(\{||\theta^* - Z||_2 \le k^{\chi^2}_{p}\})\\
=& p.
\end{aligned}
\end{equation}
The left side of \cref{fig:toy_linear_sys} provides an indicative example of a $k_p^\mathcal{N}$-interval and a $k_p^{\chi^2}$-interval, with $p = 95\%$ and $\theta^* = (0, 0)$.
The results are obtained using only one sample $Z = (Z_0, Z_1)$, which can be visualized by considering that $u_\theta(0) = Z_0$ and $u_\theta(1) = Z_1$.
Further, the empirical confidence level of an interval $(L, U)$ can be estimated as $\bar{p}:= \frac{1}{J} \sum_{j=1}^J \mathbf{1}(\cap_{k=0,1,...,\lfloor1/\epsilon\rfloor}\{L_{Z_j}(k\epsilon) < u(k\epsilon) <
U_{Z_j}(k\epsilon)\})$ with $\epsilon > 0$.
That is, we calculate the proportion of times that such a construction of the interval contains the true values on $\{0, \epsilon,...,\lfloor1/\epsilon\rfloor \epsilon\}$.
According to the law of large numbers, the empirical confidence level converges with increasing $J$, in probability, to the true confidence level $p$.
The right side of \cref{fig:toy_linear_sys} shows that the empirical confidence level curve of the $k_p^{\chi^2}$-interval (or the $k_p^\mathcal{N}$-interval) converges to a value above (or below) $95\%$.
This result implies that the $k_p^{\chi^2}$-interval is a valid $95\%$-CI for $u$ while the $k_p^\mathcal{N}$-interval is not.
\begin{figure}[ht]
\centering
\includegraphics[width=.7\linewidth]{img/toy_linear_sys.pdf}
\caption{Left: An indicative example of a $k_p^\mathcal{N}$-interval and a $k_p^{\chi^2}$-interval, with $p = 95\%$ and $\theta^* = (0, 0)$.
The results are obtained using only one sample $Z = (Z_0, Z_1)$, which can be visualized by considering that $u_\theta(0) = Z_0$ and $u_\theta(1) = Z_1$.
Right: Empirical confidence level on $\Omega$ versus the number of samples $J$.}
\label{fig:toy_linear_sys}
\end{figure}
\section{Proofs}
\label{appendix:proofs}
% \begin{corollary}[Corollary \ref{corol:general2union}]
% Let $(l_Z, u_Z)$ be a $p$-confidence interval for $u_\theta$. For any $x \in \mathcal{X}$, $(l_Z(x), u_Z(x))$ is a $p$-confidence interval for $u_\theta(x)$, i.e. $\text{Pr}(l_Z(x) < u_\theta(x) < u_Z(x)) \ge p$.
% \end{corollary}
% \emph{Proof}: $\mathcal{E}(Z) \Rightarrow l_Z(x) < u_\theta(x) < u_Z(x)$. Thus, $\text{Pr}(l_Z(x) < u_\theta(x) < u_Z(x)) \ge \text{Pr}(\mathcal{E}) \ge p$.
\subsection{Proof of Theorem~\ref{theorem:valid_general_ci}}
If $\bar{z} \in \tilde{\mathcal{Z}}_p$,
then $L(x) = \min_{z\in\tilde{\mathcal{Z}}_p} u(x; \bar{z}) \le u_{\mathcal{A}}(x; \bar{z}) \le \max_{z\in\tilde{\mathcal{Z}}_p} u(x; \bar{z}) = U(x)$, for all $x\in\Omega$.
Further, according to the definition of the proper solver, we have $u_{\mathcal{A}}(x; \bar{z}) - \eta \le u(x) \le u_{\mathcal{A}}(x; \bar{z}) + \eta$, for all $x\in\Omega$, and thus $L(x) - \eta \le u(x) \le U(x) + \eta$, for all $x\in\Omega$.
Finally, we can conclude that $Pr\{L(x) - \eta \le u(x) \le U(x) + \eta, \forall x\in\Omega\} \ge Pr\{\bar{z} \in \tilde{\mathcal{Z}}_p\} \ge p$.
% If we denote $Q_1$ as the event $\{u_{\theta^*} \in \{u_{g(Z)} | Z \in \tilde{\mathcal{Z}}_p\}\}$, then $Pr(Q_1) = Pr(\bar{Z} \in \tilde{\mathcal{Z}}_p) \geq p$.
% Next, denote $Q_2$ as the event $\{\prod_{x \in \mathcal{X}}[L(x) \le u_{\theta^*}(x) \le U(x)]\}$, where $L(x) = \min_{Z \in \tilde{\mathcal{Z}}_p} u_{g(Z)}(x)$ and $U(x) = \max_{Z \in \tilde{\mathcal{Z}}_p} u_{g(Z)}(x)$.
% Then, because for each $x\in \mathcal{X}$ in the definition of $Q_2$ a different $Z \in \tilde{\mathcal{Z}}_p$ is sought for, whereas a single $Z \in \tilde{\mathcal{Z}}_p$ is used in the definition of $Q_1$ that defines a function for all $x \in \mathcal{X}$, we conclude that $Q_1 \subseteq Q_2$. That is, $Pr(Q_2) \geq Pr(Q_1) \geq p$.
\section{Algorithms}\label{app:algos}
The developed methods are summarized in Algorithms \ref{alg:picprop} and \ref{alg:effipicprop}.
\begin{algorithm}[h]
\caption{PICProp: Physics-Informed Confidence Propagation (Gradient-based)}\label{alg:picprop}
\begin{algorithmic}
\STATE {\bfseries Input:} $\tilde{\mathcal{Z}}_p$($p$-CI for $\tilde{z}$), query point $x$, inner-steps schedule $\{N_{k}\}_{k=1}^{K_o}$, learning rates $\alpha, \beta$, outer steps $K_o$.
\STATE Initialize $z_0 \in \tilde{\mathcal{Z}}_p$, $\theta_0$.
\FOR{$k=1,...,K_o$}
\FOR{$n=1,...,N_k$}
\STATE $\theta_n = \theta_{n-1} - \alpha \nabla_{\theta} \mathcal{L}_{pinn}(\theta_{n-1}, z_{k-1})$
\ENDFOR
\STATE $\hat{\theta} = \theta_0 = \theta_{N_k}$
\IF{lower bound is sought}
\STATE $z_{k} = z_{k-1} - \beta \nabla_{z} u_{\hat{\theta}(z_{k-1})}(x)$
\ELSE
\STATE $z_{k} = z_{k-1} + \beta \nabla_{z} u_{\hat{\theta}(z_{k-1})}(x)$
\ENDIF
\ENDFOR
% \RETURN $u_{\hat{\theta}(z_{K})}(x)$
\STATE \textbf{return} $u_{\hat{\theta}(z_{K_o})}(x)$
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[h]
\caption{EffiPICProp: Efficient Physics-Informed Confidence Propagation}\label{alg:effipicprop}
\begin{algorithmic}
\STATE {\bfseries Input:} $\eta$, $p(x_q)$, $\lambda$.
\STATE Randomly sample $x_q^{1:K} \sim p(x_q)$.
\STATE Obtain $\hat{\theta}(z^L(x_q^k)), \hat{\theta}(z^U(x_q^k))$ with Algorithm~\ref{alg:picprop} for $k=1,...,K$.
\STATE Initialize $\psi$.
\WHILE{not done}
\STATE $\psi \leftarrow \psi - \eta \nabla \mathcal{L}(x_q^{1:K}; \psi)$ according to \cref{eq:effipicprop}.
\ENDWHILE
% \RETURN $u_{\psi}$
\STATE \textbf{return} $u_{\psi}$
\end{algorithmic}
\end{algorithm}
\section{Bi-level Optimization}
\label{appendix:bo}
A bi-level optimization problem can be formulated as
\begin{equation}
\begin{aligned}
\mathop{\min}_{\phi}\ \mathcal{J}(\theta^*(\phi), \phi) \quad s.t.\ \theta^*(\phi)=\mathop{\arg\min}_\theta \mathcal{L}(\theta, \phi),
\end{aligned}
\end{equation}
where $\theta$, $\phi$ are the parameters and the hyperparameters, whereas $\mathcal{L}$ and $\mathcal{J}$ are the lower-level objective and the upper-level objective. Many problems like meta-learning, hyperparameter optimization, data poisoning, and auxiliary learning can be naturally formulated in a bi-level optimization way.
The hardest part of bi-level optimization is that $\theta^*(\phi)$ is not available in a closed-form in many scenarios, particularly with an empirical lower-level objective that requires multiple steps of gradient descent. In such cases, the explicit gradient of high-level objective with respect to hyperparameter is intractable.
According to the chain rule,
\begin{equation}
\begin{aligned}
\nabla_\phi \mathcal{J}(\theta^*(\phi), \phi) = \nabla_\theta \mathcal{J} \nabla_\phi \theta^*(\phi) + \nabla_\phi \mathcal{J}.
\end{aligned}
\end{equation}
The most challenging part is the approximation of the implicit gradient $\nabla_\phi \theta^*(\phi)$, for which many advanced methods have been proposed and studied in recent years ~\cite{ji2021bilevel}. In this paper, three methods referred to as \emph{Reverse}, \emph{approximate implicit differentiation with Neumann Series} (AID-NS), and \emph{approximate implicit differentiation with conjugate gradient} (AID-CG) are evaluated on the confidence interval identification tasks.
\subsection{Reverse}
\textbf{Reverse} is an iterative differentiation-based method to approximate the hypergradient by unrolling the inner optimization $\theta_{1:K}$ trajectory in a reverse way. The implicit gradient can be approximated by
\begin{equation}
\begin{aligned}
\nabla_\phi \theta^*(\phi) \approx & \nabla_\phi \theta_K(\phi)
\approx \sum_{k=1}^{K-1} \left( \prod_{i=k+1}^{K-1} \frac{\partial \tilde{\theta}_{i+1}}{\partial \theta_{i}}\right) \frac{\partial \tilde{\theta}_{k+1}}{\phi},
\end{aligned}
\end{equation}
where $\tilde{\theta}_{k+1} = \theta_{k} - \alpha \nabla_\theta l(\theta_k, \phi)$ and $\alpha$ is a pre-defined inner-learning rate.
\subsection{Approximate Implicit Differentiation (AID)}
Another family of methods to approximate the hypergradient is Implicit Differentiation, where the implicit function theorem (IFT) is used to evaluate $\nabla_\phi \theta^*(\phi)$. If $\mathcal{L}$ is second-order derivable and strong convex, we have
\begin{equation}
\begin{aligned}
\nabla_\phi \theta^*(\phi) = -(\nabla^2_\theta \mathcal{L})^{-1}\nabla_\phi \nabla_\theta \mathcal{L}.
\end{aligned}
\end{equation}
Then we can approximate the gradients of $\phi$ using
\begin{equation}
\begin{aligned}
\nabla_\phi \mathcal{J}(\theta^*(\phi), \phi) = - \nabla_\theta \mathcal{J} \cdot (\nabla^2_\theta \mathcal{L})^{-1}\nabla_\phi \nabla_\theta \mathcal{L} + \nabla_\phi \mathcal{J}.
\end{aligned}
\end{equation}
The most challenging part is the approximation of the inverse Hessian vector product (inv-hvp) $A^{-1}p$, where $A \in \mathbb{R}^{d\times d}$ is a given symmetric positive matrix and $p \in \mathbb{R}$ is a given vector.
AID-NS~\cite{lorraine2020optimizing} is a method based on the Neumann Series to approximate the inv-hvp as
\begin{equation}
\begin{aligned}
A^{-1}p = \lambda \sum_{k=0}^\infty (I-\lambda A)^kp \approx \lambda \sum_{k=0}^{K} (I-\lambda A)^kpm
\end{aligned}
\end{equation}
where $\lambda$ is a hyperparameter that ensures convergence and $K$ is a hyperparameter for precision control.
AID-CG~\cite{grazzi2020iteration} approximates the hypergradient by solving the system of homogeneous linear equations $Ax = p$ with conjugate gradient.
\section{Details regarding implementation, hyperparameters, query points, and computational times}\label{appendix:implementation}
\subsection{Implementation and hyperparameters}
For each PDE, we use a multilayer perceptron (MLP) for the PINN model and another MLP of the same architecture for EffiPICProp.
Query points are collected in a grid manner.
For each query point, PICProp starts with standard PINN training with a randomly initialized virtual boundary condition.
We refer to this as the \emph{warmup} phase.
Subsequently, PICProp alternates between performing a number of inner steps to update the PINN model and computing the hyper-gradient for a meta step to update the virtual boundary condition.
The inner/outer learning rate is selected from $\{0.001, 0.01, 0.1\}$.
The inner/outer optimizer is selected from \emph{SGD} and \emph{Adam}.
The hyper-gradient method is selected from \emph{Neumann Series (NS)}, \emph{Conjugate Gradient (CG)}, \emph{Reverse}; see more details in \cref{appendix:bo}.
The numbers of warmup/inner/meta steps are tuned to balance the stability of bi-level training and the computational cost.
For the pedagogical example, as six points are too few for train-validation splitting, we manually set $\lambda = 1$ for EffiPICProp.
In the rest of the examples, we split $10\%$ of the training data as the validation set to select the best $\lambda$ from $\{0.0, 0.25, 0.5, 0.75, 1.0\}$.
The hyperparameters are summarized in \cref{tab:hyperparameters}.
\begin{table}[h]
\caption{Summary of implementation details, and utilized hyperparameters and architectures in the experiments.}
\label{tab:hyperparameters}
\begin{center}
\begin{small}
\begin{tabular}{lcccr}
\toprule
& Pedagogical & Poisson & Burgers \\
\midrule
MLP arch. & 32 $\times$ 2 & 20 $\times$ 8 & 20 $\times$ 8 \\
\# BC points & 2 & 40 & 256\\
\# query points & 6 & 11 $\times$ 11 & 41 $\times$ 6\\
inner optim. & Adam & Adam & Adam \\
inner lr & 0.001 & 0.001 & 0.001 \\
meta optim. & SGD & SGD & SGD \\
meta lr & 0.01 & 0.001 & 0.001 \\
hypergrad & NS & Reverse & Reverse \\
\# warmup steps & 2,000 & 4,000 & 20,000 \\
\# inner steps & 500 & 20 & 100 \\
\# meta steps & 50 & 200 & 500 \\
$\lambda$ & 1.0 & 1.0 & 0.0 \\
\bottomrule
\end{tabular}
\end{small}
\end{center}
\vskip -0.1in
\end{table}
\subsection{Query Points}
The query points are summarized in Figure \ref{fig:query}.
\begin{figure}[h]
\centering
\includegraphics[width=1.0\linewidth]{img/query.png}
\caption{Visualization of the query points on top of the exact solutions of Equations~\ref{eq:poisson2d} and \ref{eq:burger:pde}. (a) 2-D Poisson equation: 121 equally spaced points
on a grid.
(b) Burgers equation: 246 equally spaced points on a grid.}
\label{fig:query}
\end{figure}
\subsection{Computational times}\label{appendix:computational_time}
In our computational experiments, all methods were run serially and thus, the runtimes shown below are much larger than the required ones for using these methods in practice. For this reason, we have also included a ``pseudo-parallel'' runtime that shows the runtimes if parallel processing had been used instead. The runtimes for the examples of Section \ref{sec:experiment} are summarized in Table~\ref{tab:computational_time}.
\begin{table}[h]
\begin{center}
\caption{Computational time for different experiments.}
\label{tab:computational_time}
\begin{tabular}{ |p{0.8in}|p{1in}|p{1in}|p{1in}|p{1in}| }
\hline
Experiments & ES & PICProp & SimPICProp & EffiPICProp \\
\hline
Pedagogical example (CPU) & 42 hrs for 5,000 trials & 1.7 hrs for 41 queries & 16 mins (40s + 6 PICProp queries) & 18 mins (40s per lambda + 6 PICProp queries) \\
\hline
2D Poisson equation (GPU) & 14 hrs for 1,000 trials \newline 70 hrs for 5,000 trials & 400s per query & -- & 14 hrs (180s per lambda + 121 PICProp queries), \newline 10 mins for pseudo-parallel \\
\hline
1D time-dependent Burgers equation (GPU) & 347 hrs for 2,500 trials \newline 694 hrs for 5,000 trials & 5,000s per query & -- & 14 hrs (2000s per lambda + 246 PICProp queries), \newline 2 hrs for pseudo-parallel \\
\hline
\end{tabular}
\end{center}
\end{table}
\section{Additional Experimental Results}\label{appendix:additional_exp}
\subsection{Pedagogical example}
\label{appendix:pedagogical}
\subsubsection{Learning Curves}
\cref{fig:bo_learning_curve} provides the meta-learning curves corresponding to the CI predictions of \cref{fig:ci} (main text).
\cref{fig:bo_hypergrad_metaoptim} provides the results of all considered meta-learning methods and of ES on six query points.
We conclude that the combination of \emph{Neumann Series (NS)} and \emph{SGD} stably converges to the reference result of ES.
\subsubsection{$\lambda$s}
\cref{fig:effipicprop_lambdas} provides comparison among PICProp and EffiPICProp with various $\lambda$s. Notice that EffiPICProp with $\lambda=0$ is referred to as SimPICProp in \cref{fig:bo_14all}.
The mean squared errors (MSE) between PICProp CI and EffiPICProp CI are summarized in \cref{tab:mse_lambdas}.
We conclude that EffiPICProp CI with $\lambda=1.0$ is the best approximation of PICProp CI.
\subsubsection{SimPICProp v.s. EffiPICProp}
\cref{fig:bo_14all_t2} and \cref{fig:bo_14all_hoeffding} provide the comparison between SimPICProp and EffiPICProp on $T^2$ and Hoeffding boundary CI, respectively.
\subsubsection{Validity of EffiPICProp CIs}\label{appendix:validity}
Although it has been already presented in \cref{fig:ci} that PICProp and EffiPICProp predictions well match the ES results, we conduct the following experiment to further verify the validity of EffiPICProp CI predictions.
We independently sampled 500 boundary conditions and propagate the corresponding $\chi^2$ CIs using EffiPICProp to construct CIs for the entire domain.
According to the definition of joint CIs (see Definition~\ref{def:ci4pde_sol}), an instance of CI is valid if it contains the solution $u(x)$ for any $x \in [-1, 1]$.
In this regard, we uniformly sampled 101 data points from $[-1, 1]$ and regard an instance of CI as valid if the solution was enclosed within the interval for these data points.
Further, we run vanilla PINNs with clean boundary data and use the max absolute error among these 101 data points as $\eta$.
As a result, $486$ out of these $500$ ($97.2\% > 95\%$) instances of CI are valid, which empirically indicates that EffiPICProp CI predicitons are valid and conservative.
\begin{figure}[ht]
\centering
\includegraphics[width=0.8\linewidth]{img/pedagogical/learning_curve.pdf}
\caption{\textbf{Pedagogical example:} The learning curve of PICProp with \emph{Neumann Series (NS)} and \emph{SGD} on six query points.
Comparisons with exhaustive search results for solving the bi-level optimization problem for each query point are also provided.}
\label{fig:bo_learning_curve}
\end{figure}
\begin{figure}[ht]
\centering
\includegraphics[width=0.8\linewidth]{img/pedagogical/hypergrad_metaoptim.pdf}
\caption{\textbf{Pedagogical example:} Various combinations of hyper-gradient methods with meta-optimizers and comparisons with exhaustive search results for solving the bi-level optimization problem for each query point.}
\label{fig:bo_hypergrad_metaoptim}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{img/pedagogical/chi2_effipicprop_lambdas.pdf}
\caption{\textbf{Pedagogical example}, $\chi^2$: Comparison among PICProp and EffiPICProp with various $\lambda$s. Notice that EffiPICProp with $\lambda=0$ is referred to as SimPICProp in \cref{fig:bo_14all}}.
\label{fig:effipicprop_lambdas}
\end{figure}
\begin{table}[h!]\label{tab:mse_lambdas}
\centering
\begin{tabular}{c c c c c c}
\hline
$\lambda$ & 0.0 & 0.25 & 0.5 & 0.75 & 1.0 \\
\hline
MSE & 1.4E-3 & 2.2E-4 & 2.2E-4 & 1.7E-4 & \textbf{1.8E-5} \\
\hline
\\
\end{tabular}
\caption{Mean Squared Errors between PICProp CI and EffiPICProp CI. EffiPICProp CI with $\lambda=1.0$ is the best approximation of PICProp CI.}
\end{table}
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{img/pedagogical/t2_14all.pdf}
\caption{\textbf{Pedagogical example}, $T^2$: Although the results of both SimPICProp and EffiPICProp match PICProp results on the query points, EffiPICProp generalizes better on unseen
locations.}
\label{fig:bo_14all_t2}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{img/pedagogical/hoeffding_14all.pdf}
\caption{\textbf{Pedagogical example}, Hoeffding: Although the results of both SimPICProp and EffiPICProp match PICProp results on the query points, EffiPICProp generalizes better on unseen
locations.}
\label{fig:bo_14all_hoeffding}
\end{figure}
\subsection{2-D Poisson equation}\label{appendix:poisson2d}
\cref{fig:poisson2d_slice_y} includes slices along the $y$ axis and accompanies \cref{fig:poisson2d_slice_x} (main text), which includes slices along the $x$ axis.
Heatmaps with CIs over the entire domain are presented in \cref{fig:poisson2d_heatmap}.
\begin{figure}[ht]
\centering
\includegraphics[width=0.8\linewidth]{img/poisson2d/slice_y.png}
\caption{\textbf{2-D Poisson equation}: Slices of CI predictions for $y \in \{-0.5, 0.0, 0.5, 1.0\}$.}
\label{fig:poisson2d_slice_y}
\end{figure}
\begin{figure}[ht]
\centering
\includegraphics[width=1.0\linewidth]{img/poisson2d/heatmap.pdf}
\caption{\textbf{2-D Poisson equation}: Heatmaps of upper and lower CIs predicted by the exhaustive search (ES) and EffiPICProp.}
\label{fig:poisson2d_heatmap}
\end{figure}
\subsection{Burgers equation}\label{appendix:burgers}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{img/burgers/heatmap.pdf}
% \caption{\textbf{Burgers equation}: The heat maps, i.e. $u(x, t)$ values, of upper and lower CI bounds show that EffiPICProp-v1 produces similar patterns as the Exhaustive search method, while EffiPICProp-v2 collapses. Figure \ref{fig:burgers_es_mc} are essentially the cross-section slices of these heatmaps along the $t$ dimension.}
% \label{fig:burgers_es_PICProp_heatmap}
% \end{figure}
% ----------------------------------------------------------------
% Fixed bounds
Figures \ref{fig:burgers_x0.0} \& \ref{fig:burgers_x0.2} provide the meta-learning curves corresponding to the CI predictions of \cref{fig:burgers_es_mc} (main text).
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0_t0.png}
\caption{$t = 0.0$}
\end{subfigure}
%
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0_t0.25.png}
\caption{$t = 0.25$}
\end{subfigure}
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0_t0.5.png}
\caption{$t = 0.5$}
\end{subfigure}
%
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0_t0.75.png}
\caption{$t = 0.75$}
\end{subfigure}
\caption{\textbf{Burgers equation:} Meta-learning curves for $x=0$ demonstrating the stability of our method.}
\label{fig:burgers_x0.0}
\end{figure}
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0.2_t0.png}
\caption{$t = 0.0$}
\end{subfigure}
%
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0.2_t0.25.png}
\caption{$t = 0.25$}
\end{subfigure}
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0.2_t0.5.png}
\caption{$t = 0.5$}
\end{subfigure}
%
\begin{subfigure}[b]{0.24\textwidth}
\centering
\includegraphics[width=\textwidth]{img/burgers/bo_x0.2_t0.75.png}
\caption{$t = 0.75$}
\end{subfigure}
\caption{\textbf{Burgers equation:} Meta-learning curves for $x=0.2$ demonstrating the stability of our method.}
\label{fig:burgers_x0.2}
\end{figure}
% % ----------------------------------------------------------------
% % Chebyshev's bounds
% \begin{figure}[H]
% \centering
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0_t0.png}
% \caption{$t = 0.0$}
% \label{fig:burgers_chebyshev_x0.2_t0.0}
% \end{subfigure}
% %
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0.2_t0.25.png}
% \caption{$t = 0.25$}
% \label{fig:burgers_chebyshev_x0.2_t0.25}
% \end{subfigure}
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0.2_t0.5.png}
% \caption{$t = 0.5$}
% \label{fig:burgers_chebyshev_x0.2_t0.5}
% \end{subfigure}
% %
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0.2_t0.75.png}
% \caption{$t = 0.75$}
% \label{fig:burgers_chebyshev_x0.2_t0.75}
% \end{subfigure}
% \caption{\textbf{Burgers equation:} Initial experiments using hyper-gradient \emph{Reverse} method at $x = 0.2$ and different time instants.
% We propagate Chebyshev's bounds (problem scenario 3) obtained using 1000 datapoints for each of the 256 locations and using equation 2 in \cite{stellato2017multivariate}.
% The method adjusts accordingly when a different bound is propagated. Once again, the upper and lower bounds from the hyper-gradient \emph{Reverse} method converge to the exhaustive search results. In Figure (c), we observe instances where the method reaches an 'intermediate plateau' and it able move out of the plateau as training continues pass 150 epochs. This suggests that having a higher number of epochs when training for Burgers' equation is recommended.}
% \label{fig:burgers_chebyshev_x0.2}
% \end{figure}
% \begin{figure}[H]
% \centering
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0_t0.png}
% \caption{$t = 0.0$}
% \label{fig:burgers_chebyshev_x0.0_t0.0}
% \end{subfigure}
% %
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0_t0.25.png}
% \caption{$t = 0.25$}
% \label{fig:burgers_chebyshev_x0.0_t0.25}
% \end{subfigure}
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0_t0.5.png}
% \caption{$t = 0.5$}
% \label{fig:burgers_chebyshev_x0.0_t0.5}
% \end{subfigure}
% %
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{img/burgers/bo_chebyshev_x0_t0.75.png}
% \caption{$t = 0.75$}
% \label{fig:burgers_chebyshev_x0.0_t0.75}
% \end{subfigure}
% \caption{\textbf{Burgers equation:} Initial experiments using hyper-gradient \emph{Reverse} method at $x = 0.0$ and different time instants.
% We propagate Chebyshev's bounds (problem scenario 3) obtained using 1000 datapoints for each of the 256 locations and using equation 2 in \cite{stellato2017multivariate}.
% The upper and lower bounds converge to the exhaustive search results.}
% \label{fig:burgers_chebyshev_x0.0}
% \end{figure}
\section{Comparison with Bayesian PINNs}\label{app:bpinn}
In this section, we compare PICProp with Bayesian PINNs (BPINNs) \cite{yang2021b} for the problem of Section \ref{sec:poisson}.
For BPINNs, the inference method for posterior estimation is Hamiltonian Monte Carlo (HMC) \cite{2011hmc}.
Although provided here for completeness, we believe that such a comparison can also be misleading because of the many differences between the two methods.
The first difference pertains to the fact that PICProp does not consider parameter uncertainty.
Further, the assumptions for these two methods are different: BPINN requires heavy assumptions about the data likelihood, for instance, while PICProp only requires the noise to be mean-zero.
Furthermore, PICProp produces joint CIs for the whole domain, while the BPINN intervals are marginal.
Finally, although supported by a sound Bayesian justification, BPINNs are not supported by a strict confidence guarantee for the produced intervals, in contrast to \cref{theorem:valid_general_ci} that supports PICProp.
We summarize and contrast the two approaches in the following: BPINNs assume a NN parameter prior, and subsequently approximate and sample the parameter posterior to obtain samples from the solution posterior.
This is done in order to fit a solution posterior distribution and subsequently compute its statistical intervals.
In contrast, the problem of UQ is rethought ab initio in our paper. PICProp first considers either CIs given at the boundaries or data collected at the boundaries and estimates the corresponding CIs. Subsequently, the boundary CIs are propagated to the interior domain with mild assumptions and with the theoretical guarantee that the obtained CIs are indeed CIs according to the definition.
From the results summarized in \ref{fig:pedagogical_bpinn}, we observe from \cref{fig:pedagogical_bpinn} that the PICProp CIs are more conservative than the BPINN.
From a design perspective, conservative uncertainty estimation is the preferred route for critical and risk-sensitive applications \cite{ciosek2020conservative, he2020bayesian}, although it can be more costly.
In contrast, most widely used UQ methods, such as deep ensembles \cite{deepensembles} and Monte-Carlo dropout \cite{srivastava2014dropout, gal2016dropout}, are known to be overconfident in practice \cite{foong2019pathologies, ciosek2020conservative}.
Finally, note that the BPINN results of Figure \ref{fig:pedagogical_bpinn} were obtained by using the software NeuralUQ developed in \cite{zou2022neuraluq}.
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{img/pedagogical/bpinn.pdf}
\caption{\textbf{Pedagogical example}: The comparison between intervals produced by EffiPICProp and BPINNs \cite{yang2021b}+HMC \cite{2011hmc}. Noise on the boundary is Gaussian with known standard deviation $\sigma=0.05$.
}
\label{fig:pedagogical_bpinn}
% \vskip -0.5in
\end{figure}
\begin{thebibliography}{10}
\bibitem{abdar2021review}
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li~Liu,
Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U~Rajendra
Acharya, et~al.
\newblock A review of uncertainty quantification in deep learning: Techniques,
applications and challenges.
\newblock {\em Information Fusion}, 76:243--297, 2021.
\bibitem{bentkus2008extension}
Vidmantas Bentkus.
\newblock An extension of the hoeffding inequality to unbounded random
variables.
\newblock {\em Lithuanian Mathematical Journal}, 48(2):137--157, 2008.
\bibitem{bharadwaja2022physics}
BVSS Bharadwaja, Mohammad~Amin Nabian, Bharatkumar Sharma, Sanjay Choudhry, and
Alankar Alankar.
\newblock Physics-informed machine learning and uncertainty quantification for
mechanics of heterogeneous materials.
\newblock {\em Integrating Materials and Manufacturing Innovation}, pages
1--21, 2022.
\bibitem{2011hmc}
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng, editors.
\newblock {\em Handbook of Markov Chain Monte Carlo}.
\newblock Chapman and Hall/{CRC}, may 2011.
\bibitem{brosse2018promises}
Nicolas Brosse, Alain Durmus, and Eric Moulines.
\newblock The promises and pitfalls of stochastic gradient langevin dynamics.
\newblock {\em Advances in Neural Information Processing Systems}, 31, 2018.
\bibitem{cai2022fractional}
Min Cai, George Em~Karniadakis, and Changpin Li.
\newblock Fractional seir model and data-driven predictions of covid-19
dynamics of omicron variant.
\newblock {\em Chaos: An Interdisciplinary Journal of Nonlinear Science},
32(7):071101, 2022.
\bibitem{ciosek2020conservative}
Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard
Turner.
\newblock Conservative uncertainty estimation by fitting prior networks.
\newblock In {\em International Conference on Learning Representations}, 2020.
\bibitem{eaton1991concentration}
Morris~L Eaton and Michael~D Perlman.
\newblock Concentration inequalities for multivariate distributions: I.
multivariate normal distributions.
\newblock {\em Statistics \& probability letters}, 12(6):487--504, 1991.
\bibitem{foong2019pathologies}
Andrew~YK Foong, David~R Burt, Yingzhen Li, and Richard~E Turner.
\newblock Pathologies of factorised gaussian and mc dropout posteriors in
bayesian neural networks.
\newblock {\em stat}, 1050:2, 2019.
\bibitem{gal2016dropout}
Yarin Gal and Zoubin Ghahramani.
\newblock Dropout as a bayesian approximation: Representing model uncertainty
in deep learning.
\newblock In {\em international conference on machine learning}, pages
1050--1059. PMLR, 2016.
\bibitem{gao2021wasserstein}
Yihang Gao and Michael~K Ng.
\newblock Wasserstein generative adversarial uncertainty quantification in
physics-informed neural networks.
\newblock {\em arXiv preprint arXiv:2108.13054}, 2021.
\bibitem{grazzi2020iteration}
Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo.
\newblock On the iteration complexity of hypergradient computation.
\newblock In {\em International Conference on Machine Learning}, pages
3748--3758. PMLR, 2020.
\bibitem{he2020bayesian}
Bobby He, Balaji Lakshminarayanan, and Yee~Whye Teh.
\newblock Bayesian deep ensembles via the neural tangent kernel.
\newblock {\em Advances in neural information processing systems},
33:1010--1022, 2020.
\bibitem{hu2023tackling}
Zheyuan Hu, Khemraj Shukla, George~Em Karniadakis, and Kenji Kawaguchi.
\newblock Tackling the curse of dimensionality with physics-informed neural
networks.
\newblock {\em arXiv preprint arXiv:2307.12306}, 2023.
\bibitem{ji2021bilevel}
Kaiyi Ji, Junjie Yang, and Yingbin Liang.
\newblock Bilevel optimization: Convergence analysis and enhanced design.
\newblock In {\em International Conference on Machine Learning}, pages
4882--4892. PMLR, 2021.
\bibitem{karniadakis2021physics}
George~Em Karniadakis, Ioannis~G Kevrekidis, Lu~Lu, Paris Perdikaris, Sifan
Wang, and Liu Yang.
\newblock Physics-informed machine learning.
\newblock {\em Nature Reviews Physics}, 3(6):422--440, 2021.
\bibitem{lai1976on_cs}
Tze~Leung Lai.
\newblock On confidence sequences.
\newblock {\em The Annals of Statistics}, pages 265--280, 1976.
\bibitem{deepensembles}
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.
\newblock Simple and scalable predictive uncertainty estimation using deep
ensembles.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{lorraine2020optimizing}
Jonathan Lorraine, Paul Vicol, and David Duvenaud.
\newblock Optimizing millions of hyperparameters by implicit differentiation.
\newblock In {\em International Conference on Artificial Intelligence and
Statistics}, pages 1540--1552. PMLR, 2020.
\bibitem{lu2019deeponet}
Lu~Lu, Pengzhan Jin, and George~Em Karniadakis.
\newblock Deeponet: Learning nonlinear operators for identifying differential
equations based on the universal approximation theorem of operators.
\newblock {\em arXiv preprint arXiv:1910.03193}, 2019.
\bibitem{meng2021multi}
Xuhui Meng, Hessam Babaee, and George~Em Karniadakis.
\newblock Multi-fidelity bayesian neural networks: Algorithms and applications.
\newblock {\em Journal of Computational Physics}, 438:110361, 2021.
\bibitem{osband2022neural}
Ian Osband, Zheng Wen, Seyed~Mohammad Asghari, Vikranth Dwaracherla, Botao Hao,
Morteza Ibrahimi, Dieterich Lawson, Xiuyuan Lu, Brendan O'Donoghue, and
Benjamin Van~Roy.
\newblock The neural testbed: Evaluating joint predictions.
\newblock {\em arXiv preprint arXiv:2110.04629}, 2022.
\bibitem{psaros2022meta}
Apostolos~F Psaros, Kenji Kawaguchi, and George~Em Karniadakis.
\newblock Meta-learning pinn loss functions.
\newblock {\em Journal of Computational Physics}, 458:111121, 2022.
\bibitem{psaros2023uncertainty}
Apostolos~F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George~Em
Karniadakis.
\newblock Uncertainty quantification in scientific machine learning: Methods,
metrics, and comparisons.
\newblock {\em Journal of Computational Physics}, page 111902, 2023.
\bibitem{raissi2019physics}
Maziar Raissi, Paris Perdikaris, and George~E Karniadakis.
\newblock Physics-informed neural networks: A deep learning framework for
solving forward and inverse problems involving nonlinear partial differential
equations.
\newblock {\em Journal of Computational physics}, 378:686--707, 2019.
\bibitem{sadeghi2020robust}
Jonathan Sadeghi, Marco De~Angelis, and Edoardo Patelli.
\newblock Robust propagation of probability boxes by interval predictor models.
\newblock {\em Structural Safety}, 82:101889, 2020.
\bibitem{saw1984chebyshev}
John~G Saw, Mark~CK Yang, and Tse~Chin Mo.
\newblock Chebyshev inequality with estimated mean and variance.
\newblock {\em The American Statistician}, 38(2):130--132, 1984.
\bibitem{srivastava2014dropout}
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Salakhutdinov.
\newblock Dropout: a simple way to prevent neural networks from overfitting.
\newblock {\em The journal of machine learning research}, 15(1):1929--1958,
2014.
\bibitem{stellato2017multivariate}
Bartolomeo Stellato, Bart~PG Van~Parys, and Paul~J Goulart.
\newblock Multivariate chebyshev inequality with estimated mean and variance.
\newblock {\em The American Statistician}, 71(2):123--127, 2017.
\bibitem{wang2021beyond}
Chaoqi Wang, Shengyang Sun, and Roger Grosse.
\newblock Beyond marginal uncertainty: How accurately can bayesian regression
models estimate posterior predictive correlations?
\newblock In {\em International Conference on Artificial Intelligence and
Statistics}, pages 2476--2484. PMLR, 2021.
\bibitem{wang2020uncertainty}
Zhuo Wang, Chen Jiang, Pengwei Liu, Wenhua Yang, Ying Zhao, Mark~F Horstemeyer,
Long-Qing Chen, Zhen Hu, and Lei Chen.
\newblock Uncertainty quantification and reduction in metal additive
manufacturing.
\newblock {\em Npj Computational Materials}, 6(1):175, 2020.
\bibitem{waudby2020cs}
Ian Waudby-Smith and Aaditya Ramdas.
\newblock Confidence sequences for sampling without replacement.
\newblock {\em Advances in Neural Information Processing Systems},
33:20204--20214, 2020.
\bibitem{wen2021predictions}
Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth
Dwaracherla, Mohammad Asghari, and Benjamin Van~Roy.
\newblock From predictions to decisions: The importance of joint predictive
distributions.
\newblock {\em arXiv e-prints}, pages arXiv--2107, 2021.
\bibitem{yang2021b}
Liu Yang, Xuhui Meng, and George~Em Karniadakis.
\newblock B-pinns: Bayesian physics-informed neural networks for forward and
inverse pde problems with noisy data.
\newblock {\em Journal of Computational Physics}, 425:109913, 2021.
\bibitem{yang2020physics}
Liu Yang, Dongkun Zhang, and George~Em Karniadakis.
\newblock Physics-informed generative adversarial networks for stochastic
differential equations.
\newblock {\em SIAM Journal on Scientific Computing}, 42(1):A292--A317, 2020.
\bibitem{yang2019adversarial}
Yibo Yang and Paris Perdikaris.
\newblock Adversarial uncertainty quantification in physics-informed neural
networks.
\newblock {\em Journal of Computational Physics}, 394:136--152, 2019.
\bibitem{zhang2019quantifying}
Dongkun Zhang, Lu~Lu, Ling Guo, and George~Em Karniadakis.
\newblock Quantifying total uncertainty in physics-informed neural networks for
solving forward and inverse stochastic problems.
\newblock {\em Journal of Computational Physics}, 397:108850, 2019.
\bibitem{zou2022neuraluq}
Zongren Zou, Xuhui Meng, Apostolos~F Psaros, and George~Em Karniadakis.
\newblock Neuraluq: A comprehensive library for uncertainty quantification in
neural differential equations and operators.
\newblock {\em arXiv preprint arXiv:2208.11866}, 2022.
\end{thebibliography}