Physics-Informed Neural Operator for Learning Partial Differential Equations
Extracted source text ยท arXiv 2111.03794
\documentclass[letterpaper]{article} % For LaTeX2e
% \usepackage{iclr2021,times}
\usepackage[margin=1in]{geometry}
\usepackage{natbib}
\setcitestyle{square,comma,numbers}
\usepackage{times}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs}
\usepackage{multirow}% professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage[ruled,vlined, linesnumbered]{algorithm2e}
% \usepackage{algorithm}
\usepackage{algorithmic}
%% Self-defined macros
\newcommand{\kamyar}[1]{\textcolor{red}{Kamyar: #1}}
\newcommand{\aacomment}[1]{\textcolor{red}{{\bf Anima:} #1}}
\newcommand{\hongkai}[1]{\textcolor{blue}{Hongkai: #1}}
\newcommand{\swap}[3][-]{#3#1#2} % just an example
\newcommand{\nik}[1]{\textcolor{green}{Nik: #1}}
\newcommand{\A}{A}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\bbZ}{\mathbb{Z}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\C}{\mathbb{C}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\U}{U}
\newcommand{\LL}{\mathsf{L}}
\newcommand{\cR}{\mathcal{R}}
\newcommand{\cK}{\mathcal{K}}
\newcommand{\cP}{\mathcal{P}}
\newcommand{\cQ}{\mathcal{Q}}
\newcommand{\cW}{\mathcal{W}}
% F for the Fourier transform
% G for the target operator
\newcommand{\F}{\mathcal{F}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\Gtrue}{\mathcal{G}^{\dagger}}
% \newcommand{\L}{\mathcal{L}}
\newcommand{\Ldata}{\mathcal{L}_{data}}
\newcommand{\Lpde}{\mathcal{L}_{pde}}
\newcommand{\Lanchor}{\mathcal{L}_{op}}
\newcommand{\Lop}{\mathcal{L}_{op}}
\newcommand{\J}{\mathcal{J}}
\newcommand{\Jdata}{\mathcal{J}_{data}}
\newcommand{\Jpde}{\mathcal{J}_{pde}}
\newcommand{\T}{T}
\newcommand{\Td}{T^{\dagger}}
\newcommand{\Sd}{S^{\dagger}}
\newcommand{\dt}{\mathrm{d}t}
\newcommand{\K}{\mathcal{K}}
\newcommand{\vdown}{\check{v}}
\newcommand{\vup}{\hat{v}}
\newtheorem{definition}{Definition}
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{remark}{Remark}
%Kamyar's
\newcommand{\OpClass}{\mathcal{H}}
\newcommand{\Comp}{\mathcal{C}}
\newcommand{\constraint}{\textit{Conts}}
\newcommand{\Loss}{\mathcal{L}}
\newtheorem{proof}{Proof}[section]
\newtheorem{proof*}{Proof}[section]
\newcommand{\wt}{\widetilde}
\newcommand{\wh}{\widehat}
\newcommand{\wb}{\overline}
\title{Physics-Informed Neural Operator\\
for Learning Partial Differential Equations}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
% \author{%
% Zongyi, Hongkai, Nikola, David, Haoxuan, Burigede, Kamyar, Andrew, Anima.}
\author{
Zongyi Li*,
Hongkai Zheng*,
Nikola Kovachki,
David Jin,
Haoxuan Chen,\\
Burigede Liu,
Kamyar Azizzadenesheli,
Anima Anandkumar
}
\begin{document}
\maketitle
\begin{abstract}
Machine learning methods have recently shown promise in solving partial differential equations (PDEs). They can be classified into two broad categories: approximating the solution function and learning the solution operator. The Physics-Informed Neural Network (PINN) is an example of the former while the Fourier neural operator (FNO) is an example of the latter. Both these approaches have shortcomings. The optimization in PINN is challenging and prone to failure, especially on multi-scale dynamic systems. FNO does not suffer from this optimization issue since it carries out supervised learning on a given dataset, but obtaining such data may be too expensive or infeasible. In this work, we propose the physics-informed neural operator (PINO), where we combine the operating-learning and function-optimization frameworks. This integrated approach improves convergence rates and accuracy over both PINN and FNO models. In the operator-learning phase, PINO learns the solution operator over multiple instances of the parametric PDE family. In the test-time optimization phase, PINO optimizes the pre-trained operator ansatz for the querying instance of the PDE. Experiments show PINO outperforms previous ML methods on many popular PDE families while retaining the extraordinary speed-up of FNO compared to solvers. In particular, PINO accurately solves long temporal transient flows and Kolmogorov flows.
% Typically, the optimization landscape for PINN is very challenging and standard training techniques are prone to failure. FNO requires training data from an existing solver which may be expensive or even non-existent, and generalizing to new problem instances is challenging.
% PINN minimizes the loss given by equations, but the associated optimization problem is challenging. FNO learns the solution operator from data, but it is data-hungry and hard to extrapolate.
% Physics-informed neural networks (PINNs) have shown success in solving partial differential equations (PDEs) by minimizing the loss based on the equations.
% However, the associated optimization problem is usually challenging, since it requires numerous iterations of gradient descent and often fails to converge.
% An alternative framework is the operator learning setting which learns the solution operators of the PDEs. Especially, the Fourier neural operator (FNO) has shown potential in learning highly non-linear and chaotic PDEs.
% To overcome the optimization challenges in PINNs, we propose the physics-informed neural operator (PINO), where we use pre-trained neural operator ansatz in the physics-informed setting.
% It has a novel operating-learning and
% test-time optimization scheme which improves convergence rates and accuracy over both PINN and FNO models.
% In the operator-learning phase, PINO learns the solution operator for a family of equations and in the test-time optimization phase, PINO optimizes the operator to solve for one instance of the equation.
% The combined PINO model resolved both the optimization challenges in PINNs as well as the generalization and data-hunger challenges in operator learning.
% We perform empirical experiments on the 1-d Burgers' equation, the 2-d Darcy flow equation, and a turbulent regime of the 2-d Navier-Stokes equation (Kolmogorov flow). In all cases, PINO outperforms all benchmarks in both accuracy and computation time [xxx].
% Physics-informed neural networks have shown many successes in solving partial differential equations (PDEs). However, the optimization is usually challenging and requires a great number of iterations.
% Recently, the neural operator is proposed to learn the solution operator for PDEs.
% In this work, we propose the physics-informed neural operator (PINO), along with the pre-train and test-time optimization scheme, which bridges the gap between operator-learning and equation-solving settings. During the pre-train phase, PINO learns the solution operator for a family of the equation; During the optimization phase, PINO optimizes the operator to solve for one instance of the equation. Utilizing both the data and equation, PINO generalizes better compared to neural operators and converges faster compared to physics-informed solvers.
% We perform empirical experiments on the Burgers equation and Darcy equation, and Kolmogorov turbulence flow.
% Experiments show PINO requires less computation and achieves better accuracy compared to the benchmarks solvers.
\end{abstract}
% \input{sections/1-introduction}
\section{Introduction}
Machine learning-based methods are starting to show promise in scientific computing and especially in solving partial differential equations (PDEs). They have demonstrated advantages in both efficiency and accuracy compared to conventional solvers. They are even able to tackle previously intractable problems such as higher-dimensional, multi-scale, high-contrast, and chaotic PDE systems \citep{um2020solver, brunton2020machine, fan2018multiscale, long2018pde,han2018solving,bruno2021fcbased}. Broadly, ML-based approaches for PDEs can be divided into two categories: optimizing to solve for a specific solution function of PDE vs. learning the solution operator over a family of PDEs.
\paragraph{Optimization of solution function and PINN.}
Most ML-based methods, as well as the conventional solvers, fall into this category.
Conventional solvers such as FDM and FEM usually discretize the domain into a grid and optimize/approximate the solution function on the grid, which imposes a truncation error.
The Physics-Informed Neural Network (PINN)-type methods are proposed to overcome the discretization issue \citep{raissi2019physics}. They use a neural network as the ansatz of the solution function and take advantage of auto-differentiation to compute the exact, mesh-free derivatives.
Recently, researchers have developed numerous variations of PINN with promising results on inverse problems and partially observed tasks \citep{lu202186, zhu2019physics,smith2021hyposvi}.
However, compared to conventional solvers, PINNs face several optimization issues: (1) the challenging optimization landscape from soft physics or PDE constraints \citep{wang2021understanding}, (2) the difficulty to propagate information from the initial or boundary conditions to unseen parts of the interior or to future times \citep{dwivedi2020physics}, and (3) the sensitivity to hyper-parameters selection \citep{sun2020surrogate}. As a result, PINNs are still unable to compete with conventional solvers in most cases, and they often fail to converge on high-frequency or multi-scale PDEs \citep{wang2020and, fuks2020limitations, raissi2020hidden}. In this work, we propose to overcome these optimization challenges by integrating operator learning with PINN.
% However, PINNs still struggle when solving complicated, nonlinear PDEs.
% little advantage has been shown on solving standard forward problems when compared to conventional solvers.
% PINNs take advantage of the universal approximability of neural networks, but, in return, suffer from the low-frequency induced bias. Empirically, PINNs often fail to solve challenging PDEs when the solution exhibits high-frequency or multi-scale structure \citep{wang2021understanding, wang2020and, fuks2020limitations, raissi2020hidden}.
% Further, as an iterative solver, PINNs have difficulty propagating information from the initial condition or boundary condition to unseen parts of the interior or to future times \citep{dwivedi2020physics}. The optimization is highly sensitive to hyper-parameters selection \citep{sun2020surrogate}, seriously enlarging the cost of the method. Furthermore, PINNs usually require a greater amount of collocation points and iterations compared to conventional solvers, limiting their applications.
% For example, in challenging problems such as turbulence, PINNs are only able to solve the PDE on a relatively small domain \citep{jin2021nsfnets}, or otherwise, require extra observational data which is not always available in practice \citep{raissi2020hidden, cai2021physics}. In this work, we propose to overcome the challenges posed by the optimization by integrating operator learning with PINNs.
% For a time-dependent PDE problem such as the Burgers equation or the Navier-stokes equation,
% conventional solvers usually propagate a very smaller time-steps each iteration, reaching future step-by-step. On the other hand, PINNs usually randomly sample collocation points and optimize the whole time interval altogether.
% In this case, propagating the information from the initial condition to future time can be quite challenging and tricky.
% It usually requires a large number of iterations, and it is very sensitive to hyper-parameters selection. In the end, PINNs usually require a greater amount of collocation points and iterations compared to conventional solvers, limiting their application.
% For the example of challenging problems such as turbulence, PINNs are only able to solve on a relatively small domain \cite{jin2021nsfnets}, or otherwise, it must require observation of extra data \cite{raissi2020hidden, cai2021physics}.
% \paragraph{Utilize data in PINNs.}
% As a neural network-based model, PINN benefits for data significantly. Even given some sparse or noisy data, the optimization procedure in PINNs will be much smoother compared to the no-data setting, which makes PINNs good choices in the data-assimilation, super-resolution, or reconstruction setting.
% However, in most of the scenarios solving equations, we cannot observe any data from the future. The available data are usually coming from the past, or other parallel simulations were done before. This posts a question, is it possible to utilize the data in the past, with different initial conditions, boundary conditions, or equation parameters?
% People with experience know PINN can be painful.
% \paragraph{Machine learning for scientific computing.}
% % Recently, the carbon footprint of deep learning has been increasingly concerning \citep{strubell2019energy}. However, the top source of carbon emission from computation is still conventional science and engineer, especially solving partial differential equations (PDEs).
% % According to the estimate of \citet{lannelongue2020green}, the carbon footprint of a $5000$-particle physics simulations is about $544$kg CO$_2e$, and a weather forecast algorithm can emit $2 312$kg CO$_2e$ per day. Such scientific computing is running on clusters all over the world every day.
% However, much of the computation is indeed similar and repeated. For traditional PDE solvers, we need to start from the beginning if we want to slightly change the condition or coefficient of the equation. For a fluid simulation example, we may need to compute $20$ attacks of angle, and $10$ Reynolds numbers, multiplied by hundreds of shapes of the airfoil.
% Why do we need to solve each instance from scratch, over and over again?
\paragraph{Operator learning and neural operators.}
A recent alternative approach is to learn the solution operator of a family of PDEs, defined by the map from the input--initial conditions and boundary conditions, to the output--solution functions. In this case, usually, a dataset of input-output pairs from an existing solver is given. There are two main aspects to consider (a) model: to design models for learning highly complicated PDE solution operators, and (b) data: to be data-efficient and to improve generalization.
Recent advances in operator learning replace traditional convolutional neural networks and U-Nets from computer vision with operator-based models tailored to PDEs with greatly improved model expressiveness \citep{li2020neural,lu2019deeponet, patel2021physics,wang2020towards,duvall2021nonlinear}. Specifically, the neural operator generalizes the neural network to the operator setting where the input and output spaces are infinite-dimensional. The framework has shown success in learning resolution-invariant solution operators for highly non-linear problems such as turbulence flow \citep{li2020multipole, li2020fourier}. However, the data challenges remain: (1) the need for training data, which assumes an existing solver or experimental setup, (2) the non-negligible generalization error, and (3) extrapolation to unseen conditions.
These issues can be addressed by adding physics or PDE constraints to operator learning \citep{zhu2019physics,wang2021learning, zhang2021data}.
\begin{figure}
\centering
\includegraphics[width=14cm]{fig/pino-diagram4.png}
\caption{PINO: combine operator learning and PINN optimization. }
(a) learn the solution operator from a family of equations. (b) use the learned operator as an ansatz to solve for a specific instance.
\label{fig:diagram}
\end{figure}
% \begin{figure}
% \centering
% \includegraphics[width=4cm]{fig/pino-truth.png}\hspace{1cm}
% \includegraphics[width=4cm]{fig/pino-pred.png}\\
% Kolmogorov flow with $Re=500$ by the PINO solver compared to the ground truth.
% \caption{Use pre-trained neural operator to solve for Kolmogorov flow with $Re=500$}
% \label{fig:my_label}
% \end{figure}
\paragraph{Our contributions.}
To overcome the shortcomings of both physics-informed optimization and data-driven operator learning, we propose the physics-informed neural operator (PINO).
% Inspired by previous physics-constrained and data-informed operator learning settings \citep{zhu2019physics,wang2021learning, zhang2021data}, we propose the physics-informed neural operator (PINO) to overcome the data and generalization challenges in operator learning, as well as the difficult optimization in the physics-informed setting.
It requires fewer or no data points to learn the operator and generalizes better. In PINO, we use the pre-trained operator as the ansatz to optimize for the solution function at test time, which reduces the generalization error.
% This test-time optimization phase relieves the generalization error from the operator-learning setting.
Compared to PINN, PINO has a much better optimization landscape and representation space, and hence, PINO converges faster and more accurately. Our contributions can be summarized as follows:
\begin{itemize}
\item We propose the physics-informed neural operator (PINO), combining the operator-learning and physics-informed settings. We introduce the pre-training and test-time optimization schemes that utilize both the data and equation constraints (whichever are available). We develop an efficient method to compute the exact gradient for neural operators to incorporate the equation constraints.
\item By utilizing pre-trained operator ansatz, PINO overcomes the challenge of propagating information from the initial condition to future time steps with (soft) physics constraints. It can solve the 2d transient flow over an extremely long time period, where PINN and DeepONet~\citep{lu2019deeponet} fail to converge. Even without any pre-training and using only PDE constraints for the given instance, PINO still outperforms PINN by 20x smaller error and 25x speedup on the chaotic Kolmogorov flow, demonstrating superior expressivity of the neural operators over standard neural networks.
\item By utilizing the equation constraints, PINO requires fewer or no training data and generalizes better compared to FNO \citep{li2020neural}. On average it has 7\% smaller error on the transient and Kolmogorov flows, while matching the speedup of FNO (400x) compared to the GPU-based pseudo-spectral solver \citep{he2007stability}, matching FNO. Further, the pre-trained PINO model on the Navier Stokes equation can be easily transferred to different Reynolds numbers ranging from $100$ to $500$ using test-time optimization.
\item We propose the forward and backward PINO models for inverse problems. Our approach accurately recovers the coefficient function in the Darcy flow which is 3000x faster than the conventional solvers using accelerated MCMC \citep{cotter2013mcmc}.
\end{itemize}
\paragraph{PINN vs. PINO: pointwise vs. function-wise optimization.}
The neural operator ansatz in PINO has an easier optimization landscape and a more expressive representation space compared to the neural networks ansatz in PINN. The neural operator parameterizes the solution function as an aggregation of basis functions, and hence, the optimization is in the function space. This is easier than just optimizing a single function as in PINN. Further, we can learn these basis functions in the pre-training phase which makes the test-time optimization on the querying instance even easier. In this case, PINO does not need to solve from scratch. It just fine-tunes the solution function parameterized by the solution operator. Thus, PINO is much faster and more accurate compared to PINN.
% \input{sections/2-problem-setting}
\section{Preliminaries and problem settings}
In this section, we first define the stationary and dynamic PDE systems that we consider. We give an overview of the physics-informed setting and operator-learning setting. In the end, we define the Fourier neural operator as a specific model for operator learning.
\subsection{Problem settings}
We consider two natural class of PDEs. In the first,
we consider the stationary system
\begin{equation}\label{eq:stationary}
\begin{split}
\mathcal{P}(u,a) &= 0, \qquad \text{in } D \subset \R^d \\
u &= g, \qquad \text{in } \partial D
\end{split}
\end{equation}
where \(D\) is a bounded domain, \(a \in \mathcal{A} \subseteq \mathcal{V}\) is a PDE coefficient/parameter, \(u \in \mathcal{U}\) is the unknown, and \(\mathcal{P}: \mathcal{U} \times \mathcal{A} \to \mathcal{F}\) is a possibly non-linear partial differential operator with \((\mathcal{U}, \mathcal{V}, \mathcal{F})\) a triplet of Banach spaces. Usually the function $g$ is a fixed boundary condition (potentially can be entered as a parameter).
This formulation gives rise to the solution operator \(\Gtrue: \mathcal{A} \to \mathcal{U}\) defined by \(a \mapsto u\). A prototypical example is the second-order elliptic equation $ \mathcal{P}(u,a) = -\nabla \cdot (a \nabla u) + f$.
% \begin{align}
% \label{eq:darcy_flow}
% \begin{split}
% -\nabla \cdot (a \nabla u) &= f, \qquad \text{in } D \\
% u &= g, \qquad \text{in } \partial D.
% \end{split}
% \end{align}
% We let \(\mathcal{V} = L^\infty(D)\) and typically choose \(A \subseteq \{a \in L^\infty(D) : a(x) > 0 \text{ Lebesgue-a.e.}\}\) so the equation is uniformly elliptic and has a unique weak solution \(u \in H^1_0(D)\) for any \(f \in H^{-1}(D)\). Hence \(\mathcal{U} = H_0^1(D)\), \(\mathcal{F} = H^{-1}(D)\) and \(\mathcal{P}\) is the weak form of the differential operator \(\mathcal{P}(u,a) = - \nabla \cdot (a \nabla u) - f\).
In the second setting, we consider the dynamical system
\begin{align}\label{eq:dynamic}
\begin{split}
\frac{du}{dt} &= \mathcal{R}(u), \qquad \text{in } D \times (0,\infty) \\
u &= g, \qquad \quad \:\:\: \text{in } \partial D \times (0,\infty) \\
u &= a \qquad \qquad \text{in } \bar{D} \times \{0\}
\end{split}
\end{align}
where \(a = u(0) \in \mathcal{A} \subseteq \mathcal{V}\) is the initial condition, \(u(t) \in \mathcal{U}\) for \(t > 0\) is the unknown, and \(\mathcal{R}\) is a possibly non-linear partial differential operator with \(\mathcal{U}\), and \(\mathcal{V}\) Banach spaces. As before, we take \(g\) to be a known boundary condition. We assume that \(u\) exists and is bounded for all time and for every $u_0 \in \mathcal{U}$.
% \(u_0 \in \mathcal{A} \subseteq \mathcal{V}\) where \(\mathcal{V}\) is a Banach space.
This formulation gives rise to the solution operator \(\Gtrue : \mathcal{A} \to C \big ( (0,T]; \mathcal{U} \big )\) defined by \(a \mapsto u\). Prototypical examples include the Burgers' equation and the Navier-Stokes equation.
\iffalse
we will consider the family of PDEs in the form
\begin{align}
\label{eq:generalpde0}
(\LL_a u)(x, t) &= f(x), &&x \in D, t\in T, \\
u(x, t) &= g(x), &&x \in \partial D, t\in T,\\
u(x, 0) &= u_0(x),&&x \in \partial D,
\end{align}
This includes the Burgers, Darcy, and NS equations we are considering.
\[[...]\]
for some \(a \in \A\), \(f \in \U^*\) and \(D \subset \R^d\) a bounded domain. We assume that the solution \(u: D \to \R\) lives in the Banach space \(\U\) and \(\LL: \A \to \mathcal{L}(\U; \U^*)\) is a mapping from the parameter Banach space \(\A\) to the space of (possibly unbounded) linear operators mapping \(\U\) to its dual \(\U^*\). A natural operator which arises from this PDE is \(\Gtrue : \A \to \U\) defined to map the parameter to the solution \(a \mapsto u\). A simple example that we study further in Section \ref{ssec:darcy} is when \(\LL_a\) is the weak form of the second-order elliptic operator \(\nabla \cdot (a \nabla)\). In this setting, \(\A = L^\infty(D;\R_+)\), \(\U = H^1_0(D;\R)\), and \(\U^* = H^{-1}(D;\R)\).
\fi
\subsection{Solving equation using the physics-informed loss (PINN)}
Given an instance $a$ and a solution operator \(\Gtrue\) defined by equations \eqref{eq:stationary} or \eqref{eq:dynamic} , we denote by \(u^\dagger = \Gtrue (a)\) the unique ground truth. The equation solving task is to approximate $u^{\dagger}$. This setting consists of the ML-enhanced conventional solvers such as learned finite element, finite difference, and multigrid solvers \citep{kochkov2021machine,pathak2021ml, greenfeld2019learning}, as well as purely neural network-based solvers such as the Physics-Informed Neural Networks (PINNs), Deep Galerkin Method, and Deep Ritz Method \citep{raissi2019physics,sirignano2018dgm,weinan2018deep}.
Especially, these PINN-type methods use a neural network ${u_{\theta}}$ with parameters ${\theta}$ as the the ansatz to approximate the solution function $u^{\dagger}$ . The parameters ${\theta}$ are found by minimizing the physics-informed loss with exact derivatives computed using automatic-differentiation (autograd). In the stationary case, the physics-informed loss is defined by minimizing the l.h.s. of equation \eqref{eq:stationary} in the squared norm of \(\mathcal{F}\). A typical choice is \(\mathcal{F} = L^2(D)\), giving the loss function
\begin{align}
\label{eq:pinns-stationary}
\begin{split}
\mathcal{L}_{\text{pde}}(a, u_\theta) &= \Big\|\mathcal{P}(a,u_{\theta})\Big\|^2_{L^2(D)} + \alpha \Big\|u_{\theta}|_{\partial D} - g \Big\|^2_{L^2(\partial D)} \\
&= \int_D |\mathcal{P}(u_{\theta}(x),a(x))|^2\mathrm{d}x +\alpha\int_{\partial D} |u_{\theta}(x) - g(x)|^2 \mathrm{d}x
\end{split}
\end{align}
% The PDE loss consists of the physics loss in the interior and the data loss on the boundary where \(\alpha > 0\) is a hyper-parameter weighting the boundary condition.
In the case of a dynamical system, it minimizes the residual of equation \eqref{eq:dynamic} in some natural norm up to a fixed final time \(T > 0\). A typical choice is the \(L^2 \big( (0,T];L^2(D) \big )\) norm, yielding
\begin{align}
\label{eq:pinns-dynamic}
\begin{split}
\mathcal{L}_{\text{pde}}(a, u_\theta)
&= \Big\|\frac{du_{\theta}}{dt} - \mathcal{R}(u_{\theta})\Big\|^2_{L^2(T;D)} + \alpha \Big\|u_{\theta}|_{\partial D} - g \Big\|^2_{L^2(T; \partial D)} + \beta \Big\|u_{\theta}|_{t=0} - a \Big\|^2_{L^2(D)} \\
&= \int_{0}^T \int_D | \frac{du_{\theta}}{dt}(t,x) -\mathcal{R}(u_{\theta})(t,x)|^2 \mathrm{d}x \mathrm{d}t \\
&\:\:\:+ \alpha \int_{0}^T \int_{\partial D} | u_{\theta}(t,x) - g(t,x)|^2 \mathrm{d}x \mathrm{d}t \\
&\:\:\:+ \beta \int_{D} |u_{\theta}(0,x) - a(x)|^2 \mathrm{d}x
\end{split}
\end{align}
The PDE loss consists of the physics loss in the interior and the data loss on the boundary and initial conditions, with
hyper-parameters \(\alpha, \beta > 0\). It can be generalized to variational form as in \citep{weinan2018deep}.
\paragraph{Challenges of PINN.}
PINNs take advantage of the universal approximability of neural networks, but, in return, suffer from the low-frequency induced bias. Empirically, PINNs often fail to solve challenging PDEs when the solution exhibits high-frequency or multi-scale structure \citep{wang2021understanding, wang2020and, fuks2020limitations, raissi2020hidden}.
Further, as an iterative solver, PINNs have difficulty propagating information from the initial condition or boundary condition to unseen parts of the interior or to future times \citep{dwivedi2020physics}.
For example, in challenging problems such as turbulence, PINNs are only able to solve the PDE on a relatively small domain \citep{jin2021nsfnets}, or otherwise, require extra observational data which is not always available in practice \citep{raissi2020hidden, cai2021physics}. In this work, we propose to overcome the challenges posed by the optimization by integrating operator learning with PINNs.
\subsection{Learning the solution operator (neural operator)}
An alternative setting is to learn the solution operator $\G$.
Given a PDE as defined in \eqref{eq:stationary} or \eqref{eq:dynamic} and the corresponding solution operator \(\Gtrue\), one can use a neural operator ${\G_{\theta}}$ with parameters ${\theta}$ as a surrogate model to approximate \(\Gtrue\).
Usually we assume a dataset $\{a_j, u_j\}_{j=1}^N$ is available, where $\Gtrue(a_j) = u_j$ and $a_j \sim \mu$ are i.i.d. samples from some distribution \(\mu\) supported on \(\mathcal{A}\). In this case, one can optimize the solution operator by minimizing the empirical data loss on a given data pair
\begin{align}
\label{eq:pinn-data}
\mathcal{L}_{\text{data}}(u, \G_\theta(a)) = \|u - \G_\theta(a)\|_{\mathcal{U}}^2= \int_D |u(x) - \G_\theta(a)(x) |^2 \mathrm{d}x
\end{align}
where we assume the setting of \eqref{eq:stationary} for simplicity of the exposition.
The operator data loss is defined as the average error across all possible inputs
\begin{align}
\label{eq:op-data}
\begin{split}
\mathcal{J}_{\text{data}}( \G_\theta) = \|\Gtrue - \G_\theta\|^2_{L^2_\mu(\mathcal{A};\mathcal{U})} = \E_{a \sim \mu}[ \mathcal{L}_{\text{data}}(a,\theta)] \approx \frac{1}{N} \sum_{j=1}^N \int_D |u_j (x) - {\G_\theta}(a_j)(x) |^2 \mathrm{d}x.
\end{split}
\end{align}
Similarly, one can define the operator PDE loss as
\begin{align}
\label{eq:op-pde}
\begin{split}
\mathcal{J}_{\text{pde}}( \G_\theta) = \E_{a \sim \mu}[ \mathcal{L}_{\text{pde}}(a, \G_\theta(a)) ].
\end{split}
\end{align}
In general, it is non-trivial to compute the derivatives $d\G_\theta (a)/dx$ and $d\G_\theta (a)/dt$ for model $\G_\theta$. In the following section, we will discuss how to compute these derivatives for Fourier neural operator.
\iffalse
\paragraph{Remark.} Learning the solution operator $\Gtrue$ is a more general and challenging problem compared to solving for one instance of the solution function $u^{\dagger}$. When the operator $\Gtrue$ is accurately learned, one can directly evaluate $\Gtrue(a) = u^{\dagger}$. Similarly, given a learned operator $\G_\theta$, we can use it as an initial ansatz for a particular $u^{\dagger}$.
\fi
\subsection{Fourier neural operators}
In this work, we will focus on the neural operator model designed for the operator learning problem.
The neural operator, proposed in \citep{li2020neural}, is formulated as an generalization of standard deep neural networks to operator setting. Neural operator compose linear integral operator $\cK$ with pointwise non-linear activation function \(\sigma\) to approximate highly non-linear operators.
\begin{definition}[Neural operator $\G_\theta$] Define the neural operator
\begin{equation}
\label{eq:G}
\G_{\theta} \coloneqq \cQ \circ(W_{L} + \cK_{L}) \circ \cdots \circ \sigma(W_1 + \cK_1) \circ \cP
\end{equation}
where \(\cP: \R^{d_a} \to \R^{d_{1}}\), \(\cQ: \R^{d_{L}} \to \R^{d_u}\) are the pointwise neural networks that encode the lower dimension function into higher dimensional space and decode the higher dimension function back to the lower dimensional space. The model stack $L$ layers of $\sigma(W_{l} + \cK_{l})$ where \(W_l \in \R^{d_{{l+1}} \times d_{l}}\) are pointwise linear operators (matrices), \(\cK_l: \{D \to \R^{d_{l}}\} \to \{D \to \R^{d_{l+1}}\}\) are integral kernel operators, and \(\sigma\) are fixed activation functions. The parameters $\theta$ consists of all the parameters in $\cP, \cQ, W_l, \cK_l$.
% We denote the output of each layer $v_0 \coloneqq \cP a$, $v_{l+1} \coloneqq \sigma(W_{l} + \cK_{l}) v_{l_1}$, and $u = \cQ v_L$. They are functions $v_L \in \{D \to \R^{d_{l}}\}$.
\end{definition}
Recently, \citet{li2020fourier} proposes the Fourier neural operator (FNO) that deploys convolution operator for $\cK$. In this case, it can apply the Fast Fourier Transform (FFT) to efficiently compute $\cK$. This leads to a fast architecture that obtains state-of-the-art results for PDE problems.
\begin{definition}[Fourier convolution operator $\cK$] Define the Fourier convolution operator
\begin{equation}
\label{eq:Fourier}
\bigl(\cK v_t\bigr)(x)=
\F^{-1}\Bigl(R \cdot (\F v_t) \Bigr)(x) \qquad \forall x \in D
\end{equation}
where $R$ is part of the parameter $\theta$ to be learn.
\end{definition}
\paragraph{Challenges of Operator learning.}
Operator learning is similar to supervised learning in computer vision and language where data play a very important role. One needs to assume the training points and testing points follow the same problem setting and the same distribution. Especially, the previous FNO model trained on one coefficient (e.g. Reynolds number) or one geometry cannot be easily generalized to another. Moreover, for more challenging PDEs where the solver is very slow or the solver is even not existent, it is hard to gather a representative dataset.
On the other hand, since FNO doesn't use any knowledge of the equation, it is cannot get arbitrarily close to the ground truth by using the higher resolution as in conventional solvers, leaving a gap of generalization error.
These challenges limit FNO's applications beyond accelerating the solver. In the following section, we will introduce the PINO framework to overcome these problems by using the equation constraints.
% \input{sections/3-method}
\section{Physics-informed neural operator (PINO)}
We propose the PINO framework that uses one neural operator model $\G_\theta$ for solving both operator learning problems and equation solving problems. It consists of two phases
\begin{itemize}
\item \textbf{Pre-train the solution operator}: learn a neural operator $\G_\theta$ to approximate $\Gtrue$ using either/both the data loss $\Jdata$ and/or the PDE loss $\Jpde$ which can be additional to the dataset.
\item \textbf{Test-time optimization}: use $\G_\theta(a)$ as the ansatz to approximate $u^{\dagger}$ with the pde loss $\Lpde$ and/or an additional operator loss $\Lanchor$ obtained from the pre-train phase.
\end{itemize}
\subsection{Pre-train: operator learning with PDE loss}
Given a fixed amount of data $\{a_j, u_j\}$, the data loss $\Jdata$ offers a stronger constraint compared to the PDE loss $\Jpde$. However, the PDE loss does not require a pre-existing dataset. One can sample virtual PDE instances by drawing additional initial conditions or coefficient conditions $a_j \sim \mu$ for training, as shown in Algorithm \ref{algo:pretrain}. In this sense, we have access to the unlimited dataset by sampling new $a_j$ in each iteration.
% [simulated data points are limited and it contains the error of the solver. However, the equation is exact. by minimizing the equation loss computed over virtual instances, the neural operator learns a more accurate mapping from a to u.]
\begin{algorithm}[htbp]
\SetAlgoLined
\KwData{Data: input output function pairs $\{a_j,u_j\}_{j=1}^N$}
\KwResult{Neural operator $\mathcal{G}: \mathcal{A} \rightarrow \mathcal{U}.$}
\For{$i=0, 1, 2 \ldots$}{
Compute $\Ldata$ and $\Lpde$ on $(a_i,u_i)$.
Update neural operator $\mathcal{G}$; \\
\For{$j=1$ \KwTo $K$}
{Sample $a^{\prime}$ from distribution $\mu$;\\
Compute $\Lpde$ on $a^{\prime}$.
Update neural operator $\mathcal{G}$; \\}
}
\caption{Pre-training scheme for Physics-informed neural operator learning}
\label{algo:pretrain}
\end{algorithm}
\subsection{Test-time optimization: solving equation with operator ansatz}
Given a learned operator $\G_\theta$, we use $\G_\theta(a)$ as the ansatz to solve for $u^{\dagger}$. The optimization procedure is similar to PINNs where it computes the PDE loss $\Lpde$ on $a$, except that we propose to use a neural operator instead of a neural network.
Since the PDE loss is a soft constraints and challenging to optimize, we also add an optional operator loss $\Lanchor$ (anchor loss) to bound the further optimized model from the pre-trained model
\[ \Lanchor\big(\G_{\theta_i}(a), \G_{\theta_0}(a)\big) \coloneqq \|\G_{\theta_i}(a) - \G_{\theta_0}(a)\|_{\mathcal{U}}^2\]
where $\G_{\theta_i}(a)$ is the model at $i^{th}$ training epoch.
We update the operator $\G_\theta$ using the loss $\Lpde + \alpha \Lanchor$. It is possible to further apply optimization techniques to fine-tune the last fewer layers of the neural operator and progressive training that gradually increase the grid resolution and use finer resolution in test time.
\paragraph{Optimization landscape.}
Using the operator as the ansatz has two major advantages: (1) PINN does point-wise optimization, while PINO does optimization in the space of functions. In the linear integral operation $\cK$, the operator parameterizes the solution function as a sum of the basis function. Optimization of the set of coefficients and basis is easier than just optimizing a single function as in PINNs. (2) we can learn these basis functions in the operator learning phase which makes the later test-time optimization even easier. In PINO, we do not need to propagate the information from the initial condition and boundary condition to the interior. It just requires fine-tuning the solution function parameterized by the solution operator.
\paragraph{Trade-off.}
(1) complexity and accuracy:
the test-time optimization is an option to spend more computation in exchange for better accuracy. When the speed is most desired, one should use the pre-trained model to directly output the prediction. However, if one wants to solve for a specific instance accurately, then the person can use test-time optimization to correct the generalization error posed in operator learning.
(2) resolution effects on optimization landscape and truncation error: using a higher resolution and finer grid will reduce the truncation error. However, it may make the optimization unstable. Using hard constraints such as the anchor loss $\Lanchor$ relieves such a problem.
\subsection{Derivatives of neural operators}
In order to use the equation loss $\Lpde $, one of the major technical challenge is to efficiently compute the derivatives $D(\G_\theta a) = d(\G_\theta a)(x)/dx$ for neural operators. In this section, we discuss three efficient methods to compute the derivatives of neural operator $\G_\theta$ as defined in \ref{eq:G}.
\paragraph{Numerical differentiation.}
A simple but efficient approach is to use conventional numerical gradients such as finite difference and Fourier gradient \citep{zhu2019physics, gao2021phygeonet}. These numerical gradient methods are fast and memory-efficient: given a $n$-points grid, finite difference requires $O(n)$ and Fourier method requires $O(n\log n)$. However, they face the same challenges as the corresponding numerical solvers: finite difference methods require a fine-resolution uniform grid; spectral methods require smoothness and uniform grids. Especially. These numerical errors on the gradient will be amplified on the output solution.
\paragraph{Autograd.}
Similar to PINN \citep{raissi2019physics}, the most general way to compute the exact gradient is to use the auto-differentiation library of neural networks (autograd). To apply autograd, one needs to use a neural network to parameterize the solution function $\hat{u}: x \mapsto u(x)$. However, it is not straightforward to write out the solution function in the neural operator which directly outputs the numerical solution $u = \G_\theta(a)$ on a grid, especially for FNO which uses FFT. To apply autograd, we design a query function $\hat{u}$ that input $x$ and output $u(x)$. Recall $\G_{\theta} \coloneqq \cQ \circ (W_{L} + \cK_{L}) \circ \cdots \circ \sigma(W_1 + \cK_1) \circ \cP$ and $u = \G_{\theta}a = \cQ v_L = \cQ (W_{L} + \cK_{L}) v_{L-1} \ldots$. Since $\cQ$ is pointwise,
\begin{equation}
\label{eq:autograd}
u(x) = \cQ(v_L(x)) = \cQ ( (W_{L} v_{L-1})(x) + \cK_{L}v_{L-1}(x) )
\end{equation}
To query any location $x$, we need to evaluate the Fourier convolution $\cK_{L}v_{L-1}(x)$ and the bias part $(W_{L} v_{L-1})(x)$. For the first part, we can directly query the Fourier coefficient $R \cdot \F v_t$ at spatial location $x$ without doing the inverse FFT $\F$. For the second part, we can define the query function as an interpolation or a low-rank integral operator as in \citep{kovachki2021neural}. Then we can use autograd to take the exact derivative for \eqref{eq:autograd}.
The autograd method is general and exact, however, it is less efficient. Since the number of parameters $|\theta|$ is usually much greater than the grid size $n$, the numerical methods are indeed significantly faster. Empirically, the autograd method is usually slower and memory-consuming.
\paragraph{Exact gradients.}
We develop an efficient and exact gradient method based on the architecture of the neural operator. The idea is similar to the autograd, but we explicitly write out the gradient on the Fourier space and apply the chain rule.
Given the explicit form \eqref{eq:autograd}, $D(\cK_{L}v_{L-1})$ can be directly compute on the Fourier space. The linear part $D(W_{L} v_{L-1})$ can be interpolated using the Fourier method. Therefore to exactly compute the derivative of $v_L$, one just needs to run the numerical Fourier gradient.
Then we just need to apply chain rule for Jacobian $Ju = J(\cQ \circ v_L) = J(\cQ(v_L)) \circ J(v_L)$ and Hessian $Hu = H(\cQ \circ v_L) = J(v_L) \circ H(\cQ(v_L)) \circ J(v_L) + J(\cQ(v_L)) \circ H(v_L)$. Since $\cQ$ is pointwise, this can be much simplified.
% The detailed implementation is in attachment [].
In the experiments, we mostly use the exact gradient and the numerical Fourier gradient.
% \input{sections/5-experiments}
\section{Experiments}
\label{sec:exp}
In this section, we conduct empirical experiments to examine the efficacy of the proposed PINO. In~\ref{sec:operator-learning}, we show the physics constraints help operator-learning with fewer data and better generalization. Then in~\ref{sec:solve-eqn}, we investigate how PINO uses operator ansatz to solve harder equations with improved speed and accuracy. We study three concrete cases of PDEs on Burgers' Equation, Darcy Equation, and Navier-Stokes equation. The implementation details of PINO and baseline methods are listed in the Appendix \ref{sec:appendix}.
\begin{figure}[t]
\centering
\includegraphics[width=14cm]{fig/pino-acc-cplx.png}\\
(a) The long-temporal transient flow with $Re\sim 20, T=50$. PINO outputs the full trajectory in one step, which leads to 400x speedup compared to the GPU solver. PINN cannot converge to a reasonable error rate due to the long time window. (b) The chaotic Kolmogorov flow with $Re =\sim 500, T=0.5$. PINO converges faster compared to PINN, but their convergence rates with gradient descent are less effective compared to using higher resolutions in the GPU solver.
\caption{The accuracy-complexity trade-off on PINO, PINN, and the GPU-based pseudo-spectral solver.}
\label{fig:kf}
\end{figure}
\paragraph{Burgers' Equation.} The 1-d Burgers' equation is a non-linear PDE
with periodic boundary conditions where $u_0 \in L^2_{\text{per}}((0,1);\R)$ is the initial condition and $\nu = 0.01$ is the viscosity coefficient. We aim to learn the operator mapping the initial condition to the solution, $\Gtrue:u_0 \mapsto u|_{[0,1]}$.
\begin{align}
\label{eq:burgers}
\begin{split}
\partial_t u(x,t) + \partial_x ( u^2(x,t)/2) &= \nu \partial_{xx} u(x,t), \qquad x \in (0,1), t \in (0,1] \\
u(x,0) &= u_0(x), \qquad \qquad \:\: x \in (0,1)
\end{split}
\end{align}
%We want to learn the operator from the initial condition to the final step: $\Gtrue: u(0) \mapsto u(1)$
% The results of our experiments are shown in Figure \ref{fig:error} (a) and Table \ref{table:burgers} (Appendix \ref{app:burgers}). Our proposed method obtains the lowest relative error compared to any of the benchmarks. Further, the error is invariant with the resolution, while the error of convolution neural network-based methods (FCN) grows with the resolution.
% Compared to other neural operator methods such as GNO and MGNO that use Nystr\"om sampling in physical space, the Fourier neural operator is both more accurate and more computationally efficient.
\paragraph{Darcy Flow.} The 2-d steady-state Darcy Flow equation on the unit box which is the second order linear elliptic PDE
with a Dirichlet boundary where $a \in L^\infty((0,1)^2;\R_+)$ is a piecewise constant diffusion coefficient and $f = 1$ is a fixed forcing function. We are interested in learning the operator mapping the diffusion coefficient to the solution,
$\Gtrue: a \mapsto u$. Note that although the PDE is linear, the operator $\Gtrue$ is not.
\begin{align}\label{eq:darcy}
\begin{split}
- \nabla \cdot (a(x) \nabla u(x)) &= f(x) \qquad x \in (0,1)^2 \\
u(x) &= 0 \qquad \quad \:\:x \in \partial (0,1)^2
\end{split}
\end{align}
%We want to learn the operator for the coefficient function $a$ to the solution function $u$:
%$\Gtrue: a \mapsto u$
% The results of our experiments are shown in Figure \ref{fig:error} (b) and Table \ref{table:darcy} (Appendix \ref{app:darcy}). The proposed Fourier neural operator obtains nearly one order of magnitude lower relative error compared to any benchmarks. We again observe the invariance of the error with respect to the resolution.
\paragraph{Navier-Stokes Equation.}
We consider the 2-d Navier-Stokes equation for a viscous, incompressible fluid in vorticity form on the unit torus,
where $u \in C([0,T]; H^r_{\text{per}}((0,l)^2; \R^2))$ for any $r>0$ is the velocity field, $w = \nabla \times u$ is the vorticity, $w_0 \in L^2_{\text{per}}((0,l)^2;\R)$ is the initial vorticity, $\nu \in \R_+$ is the viscosity coefficient, and $f \in L^2_{\text{per}}((0,l)^2;\R)$ is the forcing function.
We want to learn the operator mapping the vorticity from the initial condition to the full solution
$\Gtrue: w_0 \mapsto w|_{[0,T]}$.
% {\bf equations under construction}\\
\begin{align}
\label{eq:ns}
\begin{split}
\partial_t w(x,t) + u(x,t) \cdot \nabla w(x,t) &= \nu \Delta w(x,t) + f(x), \qquad x \in (0,l)^2, t \in (0,T] \\
\nabla \cdot u(x,t) &= 0, \qquad \qquad \qquad \qquad \quad x \in (0,l)^2, t \in [0,T] \\
w(x,0) &= w_0(x), \qquad \qquad \qquad \quad x \in (0,l)^2
\end{split}
\end{align}
Specially, we consider two problem settings:
\begin{itemize}
\item \textbf{Long temporal transient flow}: we study the build-up of the flow from the initial condition $u_0$ near-zero velocity to $u_T$ that reaches the ergodic state. We choose $t \in [0, 50]$, $l=1$, $Re=20$ as in \citet{li2020fourier}. The main challenge is to predict the long time interval.
\item \textbf{Chaotic Kolmogorov flow}: In this case $u$ lies in the attractor where arbitrary starting time $t_0$. We choose $t \in [t_0, t_0 + 0.5] $ or $ [t_0, t_0 + 1]$, $ l=1 $, $Re=500$ similar to \citet{li2021markov}. The main challenge is to capture the small details that evolve chaotically.
\item \textbf{Lid cavity flow}: In this case, we assume the no-slip boundary condition where $u(x,t) = (0, 0)$ at left, bottom, and right walls and $u(x,t) = (1, 0)$ on top, similar to \citet{bruneau20062d}. We choose $t \in [5, 10]$, $ l=1 $, $Re=500$. The main challenge is to address the boundary using the velocity-pressure formulation.
\end{itemize}
\begin{figure}[t]
\centering
\includegraphics[width=14cm]{fig/pino-flow.png}
\caption{PINO on Kolmogorov flow (left) and Lid-cavity flow (right)}
\label{fig:flow}
\end{figure}
\subsection{Operator learning with PDE losses}\label{sec:operator-learning}
We first show the effectiveness of applying the physics constraints $\Jpde$ to learn the solution operator.
\paragraph{Burgers equation and Darcy equation.} PINO can learn the solution operator without any data on simpler problems such as Burgers and Darcy. Compared to other PDE-constrained operators, PINO is more expressive and thereby achieves better accuracy. On Burgers \eqref{eq:burgers}, PINN-DeepONet achieves \textbf{1.38\%} \citep{wang2021learning}; PINO achieves \textbf{0.37\%}.
Similarly, on Darcy flow \eqref{eq:darcy}, PINO outperforms FNO by utilizing physics constraints, as shown in Table \ref{tb:darcy-operator}. For these simpler equations, test-time optimization may not be needed. The implementation detail and the search space of parameters are included in the Appendix \ref{app:burgers} and \ref{app:darcy}.
\subsection{Solve equation using operator ansatz}\label{sec:solve-eqn}
\begin{table}[htbp]
\centering
\begin{tabular}{|ll|l|l|}
\hline
\# data samples & \# additional PDE instances & Solution error & Equation error \\ \hline
\hline
0 & 100k & 74.36\% & 0.3741 \\
\hline
0.4k & 0 & 33.32\% & 1.8779 \\
0.4k & 40k & 31.74\% & 1.8179 \\
0.4k & 160k & \textbf{31.32\%} & 1.7840 \\
\hline
4k & 0 & 25.15\% & 1.8223 \\
4k & 100k & \textbf{24.15\%} & 1.6112 \\
4k & 400k & \textbf{24.22\%} & 1.4596 \\ \hline
\hline
\end{tabular}\\
% Training on additional sampled initial conditions boosts the generalization ability of the operator.
\caption{Operator-learning on Kolmogorov flow $Re=500$. Each relative $L_2$ test error is averaged over 300 instances, which is evaluated with resolution $128\times 128 \times 65$. Complete results of other resolutions are reported in Table \ref{tb:operator-validation-4C}. }
\label{tb:operator-ansatz}
\end{table}
\paragraph{Long temporal transient flow.}
It is extremely challenging to propagate the information from the initial condition to future time steps over such a long interval $T=[0,50]$ just using the soft physics constraint. Neither of the PINN and PINO (from scratch without pre-training) can handle this case (error $>50\%$), no matter solving the full interval at once or solving per smaller steps.
However, when the pre-training data is available for PINO, we can use the learned neural operator ansatz and the anchor loss $\Lanchor$. The anchor loss is a hard constraint that makes the optimization much easier. Providing $N=4800$ training data, the PINO without test-time optimization achieves \textbf{2.87\%} error, lower than FNO \textbf{3.04\%} and it retains a 400x speedup compared to the GPU-based pseudo-spectral solver \citep{he2007stability}, matching FNO.
Further doing test time optimization with the anchor loss and PDE loss, PINO reduces the error to \textbf{1.84\%}.
\paragraph{Chaotic Kolmogorov flow.}
We show an empirical study on how PINO can improve the generalization of neural operators by enforcing more physics. For this experiment, the training set consists of 4000 data points of the initial condition and corresponding solution. Using Algorithm \ref{algo:pretrain}, we can sample unlimited additional initial conditions from a Gaussian random field at any resolution.
% We estimate the generalization error of the operator on a test set that contains 300 instances of Kolmogorov flow and reports the averaged relative $L_2$ error.
Table~\ref{tb:operator-ansatz} and \ref{tb:operator-validation-4C} compare the generalization error of neural operators trained by different schemes and different amounts of simulated data. The result shows that training neural operator with additional PDE instances consistently improves the generalization error on all three resolutions we are evaluating.
Based on the solution operators learned in the above operator-learning section, we continue to do test-time optimization. The results are shown in Figure \label{fig:kf} and Table~\ref{tb:baseline_compare}. Overall, PINO outperforms PINN by \textbf{20x} smaller error and \textbf{25x} speedup. Using a pre-trained model makes PINO converge faster. The implementation detail and the search space of parameters are included in the Appendix \ref{app:kf}.
% Note that we give extra boundary information to PINNs since NSFnet's formulation assumes the value on the boundary is given while PINO is given the initial condition only.
% We grid search the learning rate of Adam from 0.0001 to 0.01, the weights of boundary loss from 10 to 100, and report the best performance within 5000 seconds.
\begin{table}[htbp]
\centering
\begin{tabular}{c|c|c|c|c}
\hline
Method & \# data samples & \# additional PDE instances & Solution error $(w)$ & Time cost \\ \hline
PINNs & - & - & 18.7\% & 4577s \\ \hline
PINO & 0 & 0 &\textbf{0.9\%} & 608s \\
PINO & 4k & 0 & \textbf{0.9\%} & 536s\\
PINO & 4k & 160k & \textbf{0.9\%} & \textbf{473s } \\ \hline
\end{tabular}
\caption{Equation-solving on Kolmogorov flow $Re=500$.}
\label{tb:baseline_compare}
\end{table}
\paragraph{Transfer Reynolds numbers.}
The extrapolation of different parameters and conditions is one of the biggest challenges for ML-based methods. By doing test-time optimization, the pre-trained PINO model on the Kolmogorov flow can be easily transferred to different Reynolds numbers ranging from $100$ to $500$ in test-time optimization as shown in Table \ref{tb:transfer}. Such property envisions broad applications.
\paragraph{Lid cavity flow.}
We demonstrate an addition example using PINO to solve for lid-cavity flow on $T=[5,10]$ with $Re=500$. In this case, we do not have the pre-train phase and directly solve the equation (test-time optimization). We use PINO using the velocity-pressure formulation with resolution $65\times 65\times 50$ and Fourier numerical gradient. It takes $2$ minutes to achieve a relative error of $14.52\%$. Figure \ref{fig:flow} shows the ground truth and prediction of the velocity field at $t=10$ where the PINO accurately predicts the ground truth.
\begin{figure}[t]
\centering
\includegraphics[width=12cm]{fig/pino-ip-threshold.png}
\caption{Darcy inverse problem}
\label{fig:IP}
\end{figure}
\subsection{Inverse problem}
One of the major advantages of the physics-informed method is to solve the inverse problem. In this case, we investigate PINO on the inverse problem of the Darcy equation to recover the coefficient function $a^{\dagger}$ from the given solution function $u^{\dagger}$. We assume a dataset $\{a_j,u_j\}$ is available to pre-train the operator. We propose two formulations of PINO for the inverse problem:
\begin{itemize}
\item \textbf{Forward model}: Learn the forward operator $\G_\theta: a \mapsto u$ with data. Initialize $\hat{a}$ to approximate $ a^{\dagger}$. Optimize $\hat{a}$ using
\[\mathcal{J}_{forward} \coloneqq \Lpde(\hat{a}, u^\dagger) +\Ldata(\G_\theta(\hat{a})) + R(\hat{a}).\]
\item \textbf{Backward model}: Learn the backward operator $\F_\theta: u \mapsto a$ with data. Use $\F_\theta(u^{\dagger})$ to approximate $ a^{\dagger}$. Optimize $\F_\theta$ using
\[\mathcal{J}_{backward} \coloneqq \Lpde(\F_\theta(u^{\dagger}), u^\dagger) + \Lop(\F_\theta(u^{\dagger}), \F_{\theta_0}(u^{\dagger})) + R(\F_\theta(u^{\dagger}))\]
\end{itemize}
Where $R(a)$ is the regularization term. We use the PDE loss $\Lpde$ to deal with the small error in $\G_\theta$ and the ill-defining issue of $\F_\theta$.
We perform a showcase example on the Darcy equation, where we are given the full-field observation of $u$ (clean data) to recover the coefficient function $a$.
The coefficient function $a$ is piecewise constant (representing two types of media), so the inverse problem can be viewed as a classification problem. We define $R(a)$ as the total variance.
As shown in Figure \ref{fig:IP}, the PINO backward model has the best performance: it has \textbf{2.29\%} relative l2 error on the output $u$ and \textbf{97.10\%} classification accuracy on the input $a$; the forward model has 6.43\% error on the output and 95.38\% accuracy on the input.
As a reference, we compare the PINO inverse frameworks with PINN and the conventional solvers using the accelerated MCMC method with 500,000 steps \citep{cotter2013mcmc}.
The posterior mean of the MCMC has 4.52\% error and 90.30\% respectively ( Notice the Bayesian method outputs the posterior distribution, which is beyond obtaining a maximum a posteriori estimation) Meanwhile, PINO methods are 3000x faster compared to MCMC. PINN does not converge in this case.
The major advantage of the PINO backward model compared to the PINO forward model is that it uses a neural operator $\F_\theta(u^{\dagger})$ as the ansatz for the coefficient function. Similar to the forward problem, the operator ansatz has an easier optimization landscape while being expressive.
% \input{sections/6-discussion}
\section{Conclusion and Future works}
In this work, we develop the physics-informed neural operator (PINO) that bridges the gap between physics-informed optimization and data-driven neural operator learning. We introduce the pre-training and test-time optimization schemes for PINO to utilize both the data and physics constraints. In the pre-training phase, PINO learns an operator ansatz over multiple instances of a parametric PDE family. The test-time optimization scheme allows us to take advantage of the learned neural operator ansatz and solve for the solution function on the querying instance faster and more accurately.
There are many exciting future directions. Most of the techniques and analysis of PINN can be transferred to PINO. It is also interesting to ask how to overcome the hard trade-off of accuracy and complexity, and how the PINO model transfers across different geometries. Furthermore, we can develop a software library of pre-trained models. PINO's excellent extrapolation property allows it to be applied on a broad set of conditions, as shown in the Transfer Reynold's number experiments.
% Scaling PINO to 3D turbulence remains a challenging and valuable research topic.
% It would also be interesting to both empirically and theoretically study the phenomenon where the operator trained over instances with small Reynolds number can generalize well to large Reynolds even though the solution manifolds of Navier Stokes equation with different Reynolds number seems entirely different.
\section*{Acknowledgement}
Z. Li gratefully acknowledges the financial support from the Kortschak Scholars Program.
N. Kovachki is partially supported by the Amazon AI4Science Fellowship.
A. Anandkumar is supported in part by Bren endowed chair, Microsoft, Google, Adobe faculty fellowships, and DE Logi grant.
The authors want to thank Sifan Wang for meaningful discussions.
%\bibliographystyle{apalike}
\bibliography{ref}
\bibliographystyle{iclr2022_conference}
\appendix
\input{sections/Appendix}
\end{document}
\newpage
\section{Appendix}\label{sec:appendix}
\subsection{Implementation details}
In this section, we list the detailed experiment setups and parameter searching for each experiment in Section \ref{sec:exp}. Without specification, we use Fourier neural operator backbone with $width = 64$, $mode = 8$, $L=4$ and GeLU activation. The numerical experiments are performed on Nvidia V100 GPUs.
\subsubsection{Burgers Equation}\label{app:burgers}
We use the $1000$ initial conditions \(u_0 \sim \mu\) where \(\mu = \mathcal{N}(0,625(-\Delta + 25I)^{-2})\) to train the solution operator on PINO with $width = 64$, $mode = 20$, and GeLU activation. We use the numerical method to take the gradient. We use Adam optimizer with the learning rate $0.001$ that decays by half every $100$ epochs. $500$ epochs in total. The total training time is about $1250s$ on a single Nvidia 3090 GPU. PINO achieves \textbf{0.37\%} relative l2 error averaged over 200 testing instances. PINN-DeepONet achieves \textbf{1.38\%} which is taken from \citet{wang2021learning} which uses the same problem setting.
\subsubsection{Darcy Flow}\label{app:darcy}
We use the $1000$ coefficient conditions $a$ to train the solution operator where \(a \sim \mu \) where \(\mu = \psi_{\#} \mathcal{N}(0,(-\Delta + 9I)^{-2})\), $\psi(a(x)) = 12$ if $ a(x)\geq 0$; $\psi(a(x)) = 3 $ if $ a(x)<0$. The zero boundary condition is enforced by multiplying a mollifier $m(x) = \sin(\pi x)\sin(\pi y)$ for all methods.
The parameter of PINO on Darcy Flow is the same as in the Burgers equation above.
Regarding the implementation detail of the baselines: as for FNO, we use the same hyperparameters as its paper did \citep{li2020neural}; DeepONet \citep{lu2019deeponet} did not study Darcy flow so we grid search the hyperparameters of DeepONet: depth from 2 to 12, width from 50 to 100. The best result of DeepONet is achieved by depth 8, width 50. The results are shown in Table \ref{tb:darcy-operator}.
\begin{table}[htbp]
\centering
\begin{tabular}{l|c|c}
\hline
Method & Solution error & Equation error \\ \hline
DeepONet with data \citep{lu2019deeponet} & $6.97\pm 0.09\%$ & - \\
FNO with data \citep{li2020neural} & $1.98\pm 0.05\% $ & $1.3645\pm 0.014$ \\
PINO with data & $1.22\pm 0.03\%$ & $0.5740\pm 0.008$ \\ \hline
PINO w/o data & $1.50\pm 0.03 \%$ & $0.4868\pm 0.007$ \\ \hline
\end{tabular}
\caption{Operator learning on Darcy Flow. }
\label{tb:darcy-operator}
\end{table}
\subsubsection{Long temporal transient flow.}\label{app:long}
We study the build-up of the flow from the initial condition $u_0$ near-zero velocity to $u_T$ that reaches the ergodic state. We choose $T = 50, l=1$ as in \citet{li2020fourier}. We choose the weight parameters of error $\alpha=\beta=5$.
The initial condition $ w_0(x)$ is generated according to
\(w_0 \sim \mu\) where \(\mu = \mathcal{N}(0,7^{3/2}(-\Delta + 49I)^{-2.5})\) with periodic boundary conditions. The forcing is kept fixed $f(x) = 0.1(\sin(2\pi(x_1+x_2)) + \cos(2\pi(x_1 + x_2)))$. We compare FNO, PINO (no test-time optimization), and PINO (with test-time optimization). They get $3.04\%$, $2.87\%$, and $1.84\%$ relative l2 error on the last time step $u(50)$ over 5 testing instance.
% \paragraph{Lid Cavity Flow}
% In this experiment, we demonstrate PINO with non-periodic boundary conditions.
\subsubsection{Chaotic Kolmogorov flow.}\label{app:kf}
For this experiment, $u$ lies in the attractor. We choose $T = 0.5 $ or $ 1$, and $ l=1 $, similar to \citet{li2021markov}.
The training set consists of 4000 initial condition functions and corresponding solution functions with a spatial resolution of $64\times 64$ and a temporal resolution of $65$. Extra initial conditions are generated from Gaussian random field $\mathcal{N}\left(0,7^{3/2}(-\Delta+49I)^{-5/2}\right)$.
We estimate the generalization error of the operator on a test set that contains 300 instances of Kolmogorov flow and reports the averaged relative $L_2$ error.
Each neural operator is trained with 4k data points plus a number of extra sampled initial conditions. The Reynolds number in this problem is 500. The reported generalization error is averaged over 300 instances.
\paragraph{Comparison study.}\label{pinn}
The baseline method PINN is implemented using library DeepXDE \citep{lu2021deepxde} with TensorFlow as backend.
We use the two-step optimization strategy (Adam \citep{kingma2014adam} and L-BFGS) following the same practice as NSFNet \citep{jin2021nsfnets}, which applies PINNs to solving Navier Stokes equations. We grid search the hyperparameters: network depth from 4 to 6, width from 50 to 100, learning rate from 0.01 to 0.0001, the weight of boundary loss from 10 to 100 for all experiments of PINNs.
Comparison between PINO and PINNs on test-time optimization. The results are averaged over 20 instances of the Navier Stokes equation with Reynolds number 500. The best result is obtained by PINO using pre-trained operator ansatz and virtual sampling. The neural operator ansatz used here is trained over a set of 400 data points.
The authors acknowledge that there could exist more sophisticated variants of PINN that performs better in our test cases.
% \begin{figure}
% \centering
% \includegraphics[width=8cm]{fig/2or4layer.png}
% \caption{Ablation study: finetuning last two layers versus the whole network.}
% \label{fig:ablation-layers}
% \end{figure}
\begin{table}[htbp]
\centering
\begin{tabular}{l|l|l|l|l}
\hline
\# data samples & \# additional PDE instances & Resolution & Solution error & Equation error \\ \hline
\multirow{3}{*}{400} & \multirow{3}{*}{0} & 128x128x65 & 33.32\% & 1.8779 \\
& & $64\times 64\times 65$ & 33.31\% & 1.8830 \\
& & $32\times 32\times 33$ & 30.61\% & 1.8421 \\ \hline
\multirow{3}{*}{400} & \multirow{3}{*}{40k} & 128x128x65 & 31.74\% & 1.8179 \\
& & $64\times 64\times 65$ & 31.72\% & 1.8227 \\
& & $32\times 32\times 33$ & 29.60\% & 1.8296 \\ \hline
\multirow{3}{*}{400} & \multirow{3}{*}{160k} & 128x128x65 & \textbf{31.32\%} & 1.7840 \\
& & $64\times 64\times 65$ & \textbf{31.29\%} & 1.7864 \\
& & $32\times 32\times 33$ & \textbf{29.28\%} & 1.8524 \\ \hline
\multirow{3}{*}{4k} & \multirow{3}{*}{0} & $128\times 128 \times 65$ & 0.2515 & 1.8223 \\
& & $64\times 64 \times 65$ & 0.2516 & 1.8257 \\
& & $32\times 32\times 33$ & 0.2141 & 1.8468 \\ \hline
\multirow{3}{*}{4k} & \multirow{3}{*}{100k} & $128\times 128 \times 65$ & 0.2415 & 1.6112 \\
& & $64\times 64 \times 65$ & 0.2411 & 1.6159 \\
& & $32\times 32\times 33$ & 0.2085 & 1.8251 \\ \hline
\multirow{3}{*}{4k} & \multirow{3}{*}{400k} & $128\times 128 \times 65$ & 0.2422 & 1.4596 \\
& & $64\times 64 \times 65$ & 0.2395 & 1.4656 \\
& & $32\times 32\times 33$ & 0.2010 & 1.9146 \\ \hline
\multirow{3}{*}{0} & \multirow{3}{*}{100k} & $128\times 128 \times 65$ & 0.7436 & 0.3741 \\
& & $64\times 64 \times 65$ & 0.7438 & 0.3899 \\
& & $32\times 32\times 33$ & 0.7414 & 0.5226 \\ \hline
\end{tabular}
\caption{Each neural operator is trained with 4k data points additionally sampled free initial conditions. The Reynolds number is 500. The reported generalization error is averaged over 300 instances. Training on additional initial conditions boosts the generalization ability of the operator.}
\label{tb:operator-validation-4C}
\end{table}
% \begin{figure}[htbp]
% \centering
% \includegraphics[width=8cm]{fig/Re500-4layer.png}
% \caption{Neural operator with better generalization error boosts the test-time optimization speed.}
% \label{fig:operator-compare}
% \end{figure}
\subsection{Transfer learning across Reynolds numbers}
We study the test-time optimization with different Reynolds numbers on the 1s Kolmogorov flow. For the higher Reynolds number problem $Re=500, 400$, the pre-training operator shows better convergence accuracy. In all cases, the pre-training operator shows better convergence speed as demonstrated in Figure \ref{fig:test_curve_transfer}. The results are shown in Table \ref{tb:transfer} where the error is averaged over 40 instances. Each row is a testing case and each column is a pre-trained operator.
\begin{table}[htbp]
\centering
\begin{tabular}{l|l|lllllll}
\hline
Testing RE & From scratch & 100 & 200 & 250 & 300 & 350 & 400 & 500 \\ \hline
500 & 0.0493 & 0.0383 & 0.0393 & 0.0315 & 0.0477 & 0.0446 & 0.0434 & 0.0436 \\
400 & 0.0296 & 0.0243 & 0.0245 & 0.0244 & 0.0300 & 0.0271 & 0.0273 & 0.0240 \\
350 & 0.0192 & 0.0210 & 0.0211 & 0.0213 & 0.0233 & 0.0222 & 0.0222 & 0.0212 \\
300 & 0.0168 & 0.0161 & 0.0164 & 0.0151 & 0.0177 & 0.0173 & 0.0170 & 0.0160 \\
250 & 0.0151 & 0.0150 & 0.0153 & 0.0151 & 0.016 & 0.0156 & 0.0160 & 0.0151 \\
200 & 0.00921 & 0.00913 & 0.00921 & 0.00915 & 0.00985 & 0.00945 & 0.00923 & 0.00892 \\
100 & 0.00234 & 0.00235 & 0.00236 & 0.00235 & 0.00239 & 0.00239 & 0.00237 & 0.00237 \\ \hline
\end{tabular}
\caption{Reynolds number transfer learning. Each row is a test set of PDEs with corresponding Reynolds number. Each column represents the operator ansatz we use as the starting point of test-time optimization. For example, column header '100' means the operator ansatz is trained over a set of PDEs with Reynolds number 100.
The relative $L_2$ errors is averaged over 40 instances of the corresponding test set. }
\label{tb:transfer}
\end{table}
\begin{figure}
\centering
\includegraphics[width=0.75\textwidth]{fig/Re500-transfer.png}
\caption{Plot of test relative $L_2$ error versus update step for the Kolmogorov flow with Re500, T=1s. We observe that a
ll the operator ansatzs pretrained over PDE instances with different Reynolds number can boost the test-time optimization accuracy and speed compared to training from scratch.}
\label{fig:test_curve_transfer}
\end{figure}
\subsubsection{Lid Cavity flow.}\label{app:kf}
We assume a no-slip boundary where $u(x,t) = (0, 0)$ at left, bottom, and right walls and $u(x,t) = (1, 0)$ on top, similar to \citet{bruneau20062d}. We choose $t \in [5, 10]$, $ l=1 $, $Re=500$. We use the velocity-pressure formulation as in \citet{jin2021nsfnets} where the neural operator output the velocity field in $x$, $y$, and the pressure field. We set $width = 32$, $mode = 20$ with learning rate $0.0005$ which decreases by half every $5000$ iterations, $5000$ iterations in total. We use the Fourier method with Fourier continuation to compute the numerical gradient and minimize the residual error on the velocity, the divergence-free condition, as well as the initial condition and boundary condition. The weight parameters ($\alpha, \beta$) between different error terms are all chosen as 1. Figure \ref{fig:flow} shows the ground truth and prediction of the velocity field at $t=10$ where the PINO accurately predicts the ground truth.
\iffalse
\subsection{Proof of Theorem \ref{thm:generalizaion}}
\begin{proof}\label{proof:generalizaion}
Let $G\in\OpClass$ be an operator such that,
\begin{align}
\Loss_N(G)\leq \inf_{G'\in\OpClass} \Loss_N(G')+\epsilon
\end{align}
and let $\wb G\in\OpClass$ be an operator such that,
\begin{align}
\Loss(\wb G)\leq \Loss^* +\epsilon
\end{align}
Using these definitions, we have,
\begin{align*}
\Loss(G) - \Loss^*&\leq \Loss(G) - \Loss(\wb G) +\epsilon\\
&= \Loss(G) - \Loss_N(G) + \Loss_N(G) - \Loss(\wb G) +\epsilon\\
&= \Loss(G) - \Loss_N(G) + \Loss_N(G) - \Loss_N(\wb G) + \Loss_N(\wb G) - \Loss(\wb G) +\epsilon\\
&\leq \Loss(G) - \Loss_N(G) + \Loss_N(\wb G) - \Loss(\wb G) +2\epsilon
\end{align*}
Therefore, we have,
\begin{align*}
\Loss(G) - \Loss^*&\leq 2\sup_{G'\OpClass}|\Loss(G) - \Loss_N(G)| +2\epsilon
\end{align*}
Following the argument in the classical theory of Rademacher complexity ~\citep{bartlett2002rademacher,wainwright2019high} (e.g. Theorem 4.10 of \citep{wainwright2019high}), we have,
\begin{align*}
\sup_{G'\OpClass}|\Loss(G) - \Loss_N(G)| \leq 2\Comp_\ell(\OpClass,N) + \sqrt{\frac{2\log(1/\delta)}{N}}
\end{align*}
Therefore, we have,
\begin{align*}
\Loss(G) - \Loss^*&\leq 4\Comp_\ell(\OpClass,N) + \sqrt{\frac{2\log(1/\delta)}{N}}+2\epsilon
\end{align*}
\end{proof}
\fi
\begin{thebibliography}{40}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bruneau \& Saad(2006)Bruneau and Saad]{bruneau20062d}
Charles-Henri Bruneau and Mazen Saad.
\newblock The 2d lid-driven cavity problem revisited.
\newblock \emph{Computers \& fluids}, 35\penalty0 (3):\penalty0 326--348, 2006.
\bibitem[Bruno et~al.(2021)Bruno, Hesthaven, and Leibovici]{bruno2021fcbased}
Oscar~P Bruno, Jan~S Hesthaven, and Daniel~V Leibovici.
\newblock Fc-based shock-dynamics solver with neural-network localized
artificial-viscosity assignment.
\newblock \emph{arXiv preprint arXiv:2111.01315}, 2021.
\bibitem[Brunton et~al.(2020)Brunton, Noack, and
Koumoutsakos]{brunton2020machine}
Steven~L Brunton, Bernd~R Noack, and Petros Koumoutsakos.
\newblock Machine learning for fluid mechanics.
\newblock \emph{Annual Review of Fluid Mechanics}, 52:\penalty0 477--508, 2020.
\bibitem[Cai et~al.(2021)Cai, Mao, Wang, Yin, and Karniadakis]{cai2021physics}
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George~Em
Karniadakis.
\newblock Physics-informed neural networks (pinns) for fluid mechanics: A
review.
\newblock \emph{arXiv preprint arXiv:2105.09506}, 2021.
\bibitem[Cotter et~al.(2013)Cotter, Roberts, Stuart, and White]{cotter2013mcmc}
Simon~L Cotter, Gareth~O Roberts, Andrew~M Stuart, and David White.
\newblock Mcmc methods for functions: modifying old algorithms to make them
faster.
\newblock \emph{Statistical Science}, 28\penalty0 (3):\penalty0 424--446, 2013.
\bibitem[Duvall et~al.(2021)Duvall, Duraisamy, and Pan]{duvall2021nonlinear}
James Duvall, Karthik Duraisamy, and Shaowu Pan.
\newblock Non-linear independent dual system (nids) for
discretization-independent surrogate modeling over complex geometries, 2021.
\bibitem[Dwivedi \& Srinivasan(2020)Dwivedi and Srinivasan]{dwivedi2020physics}
Vikas Dwivedi and Balaji Srinivasan.
\newblock Physics informed extreme learning machine (pielm)--a rapid method for
the numerical solution of partial differential equations.
\newblock \emph{Neurocomputing}, 391:\penalty0 96--118, 2020.
\bibitem[Fan et~al.(2018)Fan, Lin, Ying, and
Zepeda-N{\'u}nez]{fan2018multiscale}
Yuwei Fan, Lin Lin, Lexing Ying, and Leonardo Zepeda-N{\'u}nez.
\newblock A multiscale neural network based on hierarchical matrices.
\newblock \emph{arXiv preprint arXiv:1807.01883}, 2018.
\bibitem[Fuks \& Tchelepi(2020)Fuks and Tchelepi]{fuks2020limitations}
Olga Fuks and Hamdi~A Tchelepi.
\newblock Limitations of physics informed machine learning for nonlinear
two-phase transport in porous media.
\newblock \emph{Journal of Machine Learning for Modeling and Computing},
1\penalty0 (1), 2020.
\bibitem[Gao et~al.(2021)Gao, Sun, and Wang]{gao2021phygeonet}
Han Gao, Luning Sun, and Jian-Xun Wang.
\newblock Phygeonet: physics-informed geometry-adaptive convolutional neural
networks for solving parameterized steady-state pdes on irregular domain.
\newblock \emph{Journal of Computational Physics}, 428:\penalty0 110079, 2021.
\bibitem[Greenfeld et~al.(2019)Greenfeld, Galun, Basri, Yavneh, and
Kimmel]{greenfeld2019learning}
Daniel Greenfeld, Meirav Galun, Ronen Basri, Irad Yavneh, and Ron Kimmel.
\newblock Learning to optimize multigrid pde solvers.
\newblock In \emph{International Conference on Machine Learning}, pp.\
2415--2423. PMLR, 2019.
\bibitem[Han et~al.(2018)Han, Jentzen, and Weinan]{han2018solving}
Jiequn Han, Arnulf Jentzen, and E~Weinan.
\newblock Solving high-dimensional partial differential equations using deep
learning.
\newblock \emph{Proceedings of the National Academy of Sciences}, 115\penalty0
(34):\penalty0 8505--8510, 2018.
\bibitem[He \& Sun(2007)He and Sun]{he2007stability}
Yinnian He and Weiwei Sun.
\newblock Stability and convergence of the crank--nicolson/adams--bashforth
scheme for the time-dependent navier--stokes equations.
\newblock \emph{SIAM Journal on Numerical Analysis}, 45\penalty0 (2):\penalty0
837--869, 2007.
\bibitem[Jin et~al.(2021)Jin, Cai, Li, and Karniadakis]{jin2021nsfnets}
Xiaowei Jin, Shengze Cai, Hui Li, and George~Em Karniadakis.
\newblock Nsfnets (navier-stokes flow nets): Physics-informed neural networks
for the incompressible navier-stokes equations.
\newblock \emph{Journal of Computational Physics}, 426:\penalty0 109951, 2021.
\bibitem[Kingma \& Ba(2014)Kingma and Ba]{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{arXiv preprint arXiv:1412.6980}, 2014.
\bibitem[Kochkov et~al.(2021)Kochkov, Smith, Alieva, Wang, Brenner, and
Hoyer]{kochkov2021machine}
Dmitrii Kochkov, Jamie~A Smith, Ayya Alieva, Qing Wang, Michael~P Brenner, and
Stephan Hoyer.
\newblock Machine learning accelerated computational fluid dynamics.
\newblock \emph{arXiv preprint arXiv:2102.01010}, 2021.
\bibitem[Kovachki et~al.(2021)Kovachki, Li, Liu, Azizzadenesheli, Bhattacharya,
Stuart, and Anandkumar]{kovachki2021neural}
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik
Bhattacharya, Andrew Stuart, and Anima Anandkumar.
\newblock Neural operator: Learning maps between function spaces.
\newblock \emph{arXiv preprint arXiv:2108.08481}, 2021.
\bibitem[Li et~al.(2020{\natexlab{a}})Li, Kovachki, Azizzadenesheli, Liu,
Bhattacharya, Stuart, and Anandkumar]{li2020fourier}
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik
Bhattacharya, Andrew Stuart, and Anima Anandkumar.
\newblock Fourier neural operator for parametric partial differential
equations, 2020{\natexlab{a}}.
\bibitem[Li et~al.(2020{\natexlab{b}})Li, Kovachki, Azizzadenesheli, Liu,
Bhattacharya, Stuart, and Anandkumar]{li2020multipole}
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik
Bhattacharya, Andrew Stuart, and Anima Anandkumar.
\newblock Multipole graph neural operator for parametric partial differential
equations, 2020{\natexlab{b}}.
\bibitem[Li et~al.(2020{\natexlab{c}})Li, Kovachki, Azizzadenesheli, Liu,
Bhattacharya, Stuart, and Anandkumar]{li2020neural}
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik
Bhattacharya, Andrew Stuart, and Anima Anandkumar.
\newblock Neural operator: Graph kernel network for partial differential
equations.
\newblock \emph{arXiv preprint arXiv:2003.03485}, 2020{\natexlab{c}}.
\bibitem[Li et~al.(2021)Li, Kovachki, Azizzadenesheli, Liu, Bhattacharya,
Stuart, and Anandkumar]{li2021markov}
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik
Bhattacharya, Andrew Stuart, and Anima Anandkumar.
\newblock Markov neural operators for learning chaotic systems.
\newblock \emph{arXiv preprint arXiv:2106.06898}, 2021.
\bibitem[Long et~al.(2018)Long, Lu, Ma, and Dong]{long2018pde}
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong.
\newblock Pde-net: Learning pdes from data.
\newblock In \emph{International Conference on Machine Learning}, pp.\
3208--3216. PMLR, 2018.
\bibitem[Lu et~al.(2021{\natexlab{a}})Lu, Wang, Chen, Lin, Car, Weinan, Jia,
and Zhang]{lu202186}
Denghui Lu, Han Wang, Mohan Chen, Lin Lin, Roberto Car, E~Weinan, Weile Jia,
and Linfeng Zhang.
\newblock 86 pflops deep potential molecular dynamics simulation of 100 million
atoms with ab initio accuracy.
\newblock \emph{Computer Physics Communications}, 259:\penalty0 107624,
2021{\natexlab{a}}.
\bibitem[Lu et~al.(2019)Lu, Jin, and Karniadakis]{lu2019deeponet}
Lu~Lu, Pengzhan Jin, and George~Em Karniadakis.
\newblock Deeponet: Learning nonlinear operators for identifying differential
equations based on the universal approximation theorem of operators.
\newblock \emph{arXiv preprint arXiv:1910.03193}, 2019.
\bibitem[Lu et~al.(2021{\natexlab{b}})Lu, Meng, Mao, and
Karniadakis]{lu2021deepxde}
Lu~Lu, Xuhui Meng, Zhiping Mao, and George~Em Karniadakis.
\newblock {DeepXDE}: A deep learning library for solving differential
equations.
\newblock \emph{SIAM Review}, 63\penalty0 (1):\penalty0 208--228,
2021{\natexlab{b}}.
\newblock \doi{10.1137/19M1274067}.
\bibitem[Patel et~al.(2021)Patel, Trask, Wood, and Cyr]{patel2021physics}
Ravi~G Patel, Nathaniel~A Trask, Mitchell~A Wood, and Eric~C Cyr.
\newblock A physics-informed operator regression framework for extracting
data-driven continuum models.
\newblock \emph{Computer Methods in Applied Mechanics and Engineering},
373:\penalty0 113500, 2021.
\bibitem[Pathak et~al.(2021)Pathak, Mustafa, Kashinath, Motheau, Kurth, and
Day]{pathak2021ml}
Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten
Kurth, and Marcus Day.
\newblock Ml-pde: A framework for a machine learning enhanced pde solver.
\newblock \emph{Bulletin of the American Physical Society}, 2021.
\bibitem[Raissi et~al.(2019)Raissi, Perdikaris, and
Karniadakis]{raissi2019physics}
Maziar Raissi, Paris Perdikaris, and George~E Karniadakis.
\newblock Physics-informed neural networks: A deep learning framework for
solving forward and inverse problems involving nonlinear partial differential
equations.
\newblock \emph{Journal of Computational Physics}, 378:\penalty0 686--707,
2019.
\bibitem[Raissi et~al.(2020)Raissi, Yazdani, and Karniadakis]{raissi2020hidden}
Maziar Raissi, Alireza Yazdani, and George~Em Karniadakis.
\newblock Hidden fluid mechanics: Learning velocity and pressure fields from
flow visualizations.
\newblock \emph{Science}, 367\penalty0 (6481):\penalty0 1026--1030, 2020.
\bibitem[Sirignano \& Spiliopoulos(2018)Sirignano and
Spiliopoulos]{sirignano2018dgm}
Justin Sirignano and Konstantinos Spiliopoulos.
\newblock Dgm: A deep learning algorithm for solving partial differential
equations.
\newblock \emph{Journal of computational physics}, 375:\penalty0 1339--1364,
2018.
\bibitem[Smith et~al.(2021)Smith, Ross, Azizzadenesheli, and
Muir]{smith2021hyposvi}
Jonathan~D Smith, Zachary~E Ross, Kamyar Azizzadenesheli, and Jack~B Muir.
\newblock Hyposvi: Hypocenter inversion with stein variational inference and
physics informed neural networks.
\newblock \emph{arXiv}, 2021.
\bibitem[Sun et~al.(2020)Sun, Gao, Pan, and Wang]{sun2020surrogate}
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang.
\newblock Surrogate modeling for fluid flows based on physics-constrained deep
learning without simulation data.
\newblock \emph{Computer Methods in Applied Mechanics and Engineering},
361:\penalty0 112732, 2020.
\bibitem[Um et~al.(2020)Um, Holl, Brand, Thuerey, et~al.]{um2020solver}
Kiwon Um, Philipp Holl, Robert Brand, Nils Thuerey, et~al.
\newblock Solver-in-the-loop: Learning from differentiable physics to interact
with iterative pde-solvers.
\newblock \emph{arXiv preprint arXiv:2007.00016}, 2020.
\bibitem[Wang et~al.(2020{\natexlab{a}})Wang, Kashinath, Mustafa, Albert, and
Yu]{wang2020towards}
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu.
\newblock Towards physics-informed deep learning for turbulent flow prediction.
\newblock In \emph{Proceedings of the 26th ACM SIGKDD International Conference
on Knowledge Discovery \& Data Mining}, pp.\ 1457--1466, 2020{\natexlab{a}}.
\bibitem[Wang et~al.(2020{\natexlab{b}})Wang, Yu, and Perdikaris]{wang2020and}
Sifan Wang, Xinling Yu, and Paris Perdikaris.
\newblock When and why pinns fail to train: A neural tangent kernel
perspective.
\newblock \emph{arXiv preprint arXiv:2007.14527}, 2020{\natexlab{b}}.
\bibitem[Wang et~al.(2021{\natexlab{a}})Wang, Teng, and
Perdikaris]{wang2021understanding}
Sifan Wang, Yujun Teng, and Paris Perdikaris.
\newblock Understanding and mitigating gradient flow pathologies in
physics-informed neural networks.
\newblock \emph{SIAM Journal on Scientific Computing}, 43\penalty0
(5):\penalty0 A3055--A3081, 2021{\natexlab{a}}.
\bibitem[Wang et~al.(2021{\natexlab{b}})Wang, Wang, and
Perdikaris]{wang2021learning}
Sifan Wang, Hanwen Wang, and Paris Perdikaris.
\newblock Learning the solution operator of parametric partial differential
equations with physics-informed deeponets.
\newblock \emph{arXiv preprint arXiv:2103.10974}, 2021{\natexlab{b}}.
\bibitem[Weinan \& Yu(2018)Weinan and Yu]{weinan2018deep}
E~Weinan and Bing Yu.
\newblock The deep ritz method: a deep learning-based numerical algorithm for
solving variational problems.
\newblock \emph{Communications in Mathematics and Statistics}, 6\penalty0
(1):\penalty0 1--12, 2018.
\bibitem[Zhang et~al.(2021)Zhang, Xu, and Zhang]{zhang2021data}
Lulu Zhang, Zhi-Qin~John Xu, and Yaoyu Zhang.
\newblock Data-informed deep optimization.
\newblock \emph{arXiv preprint arXiv:2107.08166}, 2021.
\bibitem[Zhu et~al.(2019)Zhu, Zabaras, Koutsourelakis, and
Perdikaris]{zhu2019physics}
Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris
Perdikaris.
\newblock Physics-constrained deep learning for high-dimensional surrogate
modeling and uncertainty quantification without labeled data.
\newblock \emph{Journal of Computational Physics}, 394:\penalty0 56--81, 2019.
\end{thebibliography}