RANG: A Residual-based Adaptive Node Generation Method for Physics-Informed Neural Networks
Extracted source text · arXiv 2205.01051
%%
%% Copyright 2007-2020 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version. The latest version of this license is in
%% http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%% SP 2008/03/01
%%
%%
%%
%% $Id: elsarticle-template-num.tex 190 2020-11-23 11:12:32Z rishi $
%%
%%
\documentclass[preprint,12pt]{elsarticle}
%% Use the option review to obtain double line spacing
% \documentclass[authoryear,preprint,review,12pt]{elsarticle}
%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}
%% For including figures, graphicx.sty has been loaded in
%% elsarticle.cls. If you prefer to use the old commands
%% please give \usepackage{epsfig}
%% The amssymb package provides various useful mathematical symbols
\usepackage{lineno,hyperref}
\usepackage{amssymb,amsmath, amsthm}
\usepackage{multirow}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{subcaption}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}
%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers.
%% \usepackage{lineno}
%% \usepackage[fontset=ubuntu]{ctex}
\journal{xxxx}
\begin{document}
\begin{frontmatter}
%% Title, authors and addresses
%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for theassociated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for theassociated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for theassociated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \affiliation{organization={},
%% addressline={},
%% city={},
%% postcode={},
%% state={},
%% country={}}
%% \fntext[label3]{}
\title{RANG: A Residual-based Adaptive Node Generation Method for Physics-Informed Neural Networks}
%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{}
%% \affiliation[label1]{organization={},
%% addressline={},
%% city={},
%% postcode={},
%% state={},
%% country={}}
%%
%% \affiliation[label2]{organization={},
%% addressline={},
%% city={},
%% postcode={},
%% state={},
%% country={}}
\author[inst1]{Wei Peng}
\ead{weipeng0098@126.com}
\affiliation[inst1]{organization={Defense Innovation Institute, Chinese Academy of Military Science},%Department and Organization
% addressline={Address One},
city={Beijing},
postcode={100071},
% state={State One},
country={China}}
\author[inst1]{Weien Zhou}\ead{weienzhou@nudt.edu.cn}
\author[inst1]{Xiaoya Zhang}
\author[inst1]{Wen Yao\corref{cor}}
\ead{wendy0782@126.com}
\cortext[cor]{Corresponding Author}
\author[inst2]{Zheliang Liu}
\affiliation[inst2]{organization={Human Resource Center},%Department and Organization
% addressline={},
city={Beijing},
postcode={100031},
% state={State Two},
country={China}}
\begin{abstract}
Learning solutions of partial differential equations (PDEs) with Physics-Informed Neural Networks (PINNs) is an attractive alternative approach to traditional solvers due to its flexibility and ease of incorporating observed data.
Despite the success of PINNs in accurately solving a wide variety of PDEs, the method still requires improvements in terms of computational efficiency.
One possible improvement idea is to optimize the generation of training point sets.
Residual-based adaptive sampling and quasi-uniform sampling approaches have been each applied to improve the training effects of PINNs, respectively.
To benefit from both methods,
we propose the Residual-based Adaptive Node Generation (RANG) approach for efficient training of PINNs, which is based on a variable density nodal distribution method for RBF-FD.
The method is also enhanced by a memory mechanism to further improve training stability.
We conduct experiments on three linear PDEs and three nonlinear PDEs with various node generation methods, through which the accuracy and efficiency of the proposed method compared to the predominant uniform sampling approach is verified numerically.
\end{abstract}
%%Graphical abstract
% \begin{graphicalabstract}
% \includegraphics{grabs}
% \end{graphicalabstract}
%Research highlights
% \begin{highlights}
% \item We observed that training Physics-Informed Neural Networks (PINN) could benefit from adaptability and local regularity of node generation and tried to benefit from both techniques.
% \item We regard PINN as an extension of the traditional strong form meshless method and propose the Residual-based Adaptive Node Generation method (RANG) for PINN using a variable density node generation method from the traditional meshless methods.
% \item We introduced a memory mechanism to RANG to make PINN training more stable.
% \item We compared several methods to solve linear and nonlinear equations and provided open-source code for reproducing the results.
% \end{highlights}
\begin{keyword}
%% keywords here, in the form: keyword \sep keyword
Meshless method \sep Neural network \sep Node generation \sep PDEs
%% PACS codes here, in the form: \PACS code \sep code
% \PACS 0000 \sep 1111
%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)
% \MSC 0000 \sep 1111
\end{keyword}
\end{frontmatter}
%% \linenumbers
%% main text
\section{Introduction}
When solving PDEs, generating a high-quality mesh can be computationally expensive. Therefore, to avoid generating any mesh, many meshless methods have been developed to solve partial differential equations numerically in recent years. Meshless methods usually generate the computational nodes to fit the boundary and meet the spatially variable resolution requirement, thus avoiding generating any mesh. Due to the high freedom of geometry, meshless methods have great potential in dealing with data-driven related or inverse PDE problems even with complex domains, especially in recent methods incorporating deep neural networks.
Deep Neural Networks (DNN) have achieved remarkable performance in many scientific computing tasks, including recent studies using DNN models to solve PDEs numerically. This paper focuses on one of these methods, the Physics-Informed Neural Network (PINN) \cite{raissiPhysicsinformedNeuralNetworks2019}, where physical conditions are imposed on a feedforward neural network. Though PINN usually appears as a deep learning method, it is also a strong-form meshless PDE solver.
PINN has arisen as a promising framework for combining observed data and physical laws for various applications in science and engineering.
Raissi et al. proposed PINN to identify unknown parameters in partial differential equations (PDEs), such as the Korteweg-de Vries equation \cite{raissiPhysicsinformedNeuralNetworks2019}, and used the Navier-Stokes equations to reveal hidden fluid dynamics \cite{raissiHiddenFluidMechanics2020}.
Jin et al. \cite{jinNSFnetsNavierStokesFlow2021} employed PINN to model incompressible flow from laminar to turbulent.
Sun et al. \cite{sunSurrogateModelingFluid2020a} proposed PINN-based surrogates of fluids without relying on any external data.
Haghighat et al. \cite{haghighatPhysicsinformedDeepLearning2021} used PINN for inversion and surrogate modeling in the field of solid mechanics.
Cai et al. \cite{caiPhysicsInformedNeuralNetworks2021} investigated solutions to various prototype heat transfer problems.
He et al. \cite{hePhysicsinformedDeepLearning2021} studied the problem of direct and inverse heat conduction in materials.
Lu et al. \cite{luPhysicsInformedNeuralNetworks2021} used PINN in reverse design to solve the holographic problem in optics and the fluid problem of Stokes flow.
Schiassi et al. \cite{schiassiPhysicsInformedNeuralNetworks2022} used PINN to solve the optimal planar orbital transfer problem.
PINN is also used to generate grid meshes for traditional methods \cite{chenMGNetNovelDifferential2022a}.
Since a DNN is a nonlinear function containing many parameters, PINN discretizes a PDE by modeling it as an unconstrained nonlinear least-squares optimization problem. However, PINN often faces serious computational difficulties and fails to solve the PDE due to the high nonlinearity. Therefore, a series of extensions to the vanilla PINN has been proposed to improve the accuracy and robustness of handling increasingly challenging problems. Such extensions include, but are not limited to, tuning of loss weights \cite{wangUnderstandingMitigatingGradient2021, mcclennySelfAdaptivePhysicsInformedNeural2020, xiang2021selfadaptive, liuDualDimerMethodTraining2021}, novel network architectures and activation functions \cite{ramabathiranSPINNSparsePhysicsbased2021a, wangRespectingCausalityAll2022, yuGradientenhancedPhysicsinformedNeural2022, cuiEfficientNeuralNetwork2022}, domain decomposition methods \cite{mengPPINNPararealPhysicsinformed2020}, and mini-batch collocation generation algorithms \cite{dasStateoftheArtReviewDesign2022, nabianEfficientTrainingPhysics2021a, luDeepXDEDeepLearning2021, zhaoSolvingAllenCahnCahnHilliard2021}.
From another perspective, PINN is a meshless method with DNNs replacing RBFs, which is a direct extension of the Kansa method \cite{kansaMultiquadricsScatteredData1990}. For traditional meshless approaches, especially RBF-FD methods, the choice of collocation points requires constraints on node sets to ensure solution accuracy and stability \cite{fornbergSolvingPDEsRadial2015}. This paper will draw on extensive research on node generation approaches in the traditional meshless field. Current methods of node generation can be categorized broadly into iterative methods, sphere packing methods, and advancing front methods. Iterative methods begin with initializing a node set and updating their positions \cite{zamoloTwoAlgorithmsFast2018, vlasiukFastHighdimensionalNode2018}, which are computationally expensive and require an initial distribution. Sphere packing methods are often constrained to a constant radius, which leads to constant node density. One of the most usual methods is Poisson disk sampling \cite{bridsonFastPoissonDisk2007}. Advancing front methods are computationally more efficient and relatively simple to implement. The proposed method in this paper is based on the original advancing front type method presented by Fornberg and Flyer \cite{fornbergFastGeneration2D2015}. Other similar works include \cite{vandersandeFastVariableDensity2021, duhFastVariableDensity2021, slakGenerationNodeDistributions2019a,liBitingAdvancingFront2000, lhnerProgressGridGeneration1996}. The meshless method RBF-FD can also be combined with adaptive node generation strategies. For example, Li et al. \cite{liAdaptiveRBFFDMethod2017} and Slack et al. \cite{slakAdaptiveRadialBasis2019} proposed locally adaptive sampling methods based on the estimation of PDE residuals.
\textbf{Related work}. There are relatively few studies on the generation of PINN collocation points. Lu et al. \cite{luDeepXDEDeepLearning2021} proposed the RAR method, which gradually adds and retains the collocation points with more significant residuals in the collocation point set. \cite{nabianEfficientTrainingPhysics2021a} employed the mode of subdomain training to strengthen the training of regions with large residuals. Another subdomain adaptive sampling method is proposed by Zeng et al. \cite{zengAdaptiveDeepNeural2022}. \cite{zhaoSolvingAllenCahnCahnHilliard2021} found it challenging for the vanilla PINN to solve the Allen-Cahn and Hilliard equations and improved results after adopting a similar RAR strategy.
Tang et al. \cite{tang2021} introduced an additional neural network acting as an error indicator to guide the refinement, which is aimed at avoiding the large variance in uniform random sampling. Hanna et al. \cite{hannaResidualbasedAdaptivityTwophase2022} estimated probalistic density functions to adaptively generate addtitional nodes based on the PDE residuals.
From the experimental design perspective, Das and Tesfamariam \cite{dasStateoftheArtReviewDesign2022} discussed the effects of various sampling methods and compared several node generation methods for PINN. In their experimental results, the quasi-uniform method, Hammersley sampling, seems very simple but has a significant advantage over other methods.
\textbf{Motivation}. These studies inspire us that the combination of quasi-uniform sampling and residual-based adaptive sampling may further improve PINN training.
However, the adaptability and quasi-uniformity of a sampling method seem to be two contradictory goals, and this paper is trying to reconcile the contradiction by introducing variable density node generation methods with local regularity. Based on the rapid node generation technology in two-dimensional space proposed by \cite{fornbergFastGeneration2D2015}, it is possible to generate locally quasi-uniform nodes with variable density based on the PDE residuals adaptively. The Residual-based Adaptive Node Generation method (RANG) is then proposed via combining the residual of PINN with the method that can dynamically change the sampling radius in a computational domain. Therefore, on the one hand, we expect to take advantage of the quasi-uniform sampling method that may bring to PINN and, on the other hand, combines the advantages brought by residual-based adaptive refinement methods.
The structure of this paper is organized as follows. Section 2 introduces PINNs and two basic sampling techniques. Section 3 proposes RANG for PINNs. Section 4 conducts six numerical experiments, including three linear and three nonlinear PDEs. The final section discusses the experimental results, points out the limitations of this paper and possible improvements, and concludes the paper.
\section{Background}
This section will introduce the necessary elements for solving PINN with RANG. First, the vanilla PINN is briefly introduced. Then two sampling methods that generate nodes in 2D rectangles are introduced, where the Hammersley sampling method is introduced for reference, and an advancing front sampling method is introduced to be the basis of RANG.
\subsection{Physics Informed Neural Networks}
PINN employs a feedforward neural networks $\hat u$ to approximate the solution $u$ for the following generic differential equation:
\begin{align*}
\mathcal{N}(t,x,u)=0, &\quad t\in[0,T],x\in D,\\
\mathcal{I}(x,u)=0, &\quad x\in D,\\
\mathcal{B}(t,x,u)=0, &\quad t\in[0,T],x\in\partial D,
\end{align*}
where $\mathcal{N}$ is a general differential operator that may consist of derivatives, linear terms, and nonlinear terms. $x$ usually denotes a spatial vector, while $t$ is a scalar about time. The operators $\mathcal{I}$ and $\mathcal{B}$ denote the initial and boundary value conditions, respectively, which may also consist of differential, linear, and nonlinear terms.
Following the work of Raissi et al. \cite{raissiPhysicsinformedNeuralNetworks2019}, we consider $\hat u_\theta$ to be a classic fully-connected neural network which is parameterized by $\theta$. Define the residual networks, which share the same network parameters $\theta$ and satisfy
\begin{align*}
r_{pde}(t,x;\theta)&:=\mathcal{N}(t,x,\hat u(t,x;\theta)),\\
r_{0}(t,x;\theta)&:=\mathcal{I}(x,\hat u(t,x;\theta)),\\
r_{b}(t,x;\theta)&:=\mathcal{B}(t,x,\hat u(t,x;\theta)),
\end{align*}
where all partial derivatives can be computed by automatic differentiation methods. The shared network parameters $\theta$ are trained by minimizing loss terms that penalizes the residuals for not equaling zero:
\begin{align}
L_{pde}(\theta)&=\frac{1}{\#\mathbf{N}_{pde}}\sum_{(x,t)\in \mathbf{N}_{pde}} |r_{pde}(t,x,\theta)|^2,\label{eq:pde_l}\\
L_{0}(\theta)&=\frac{1}{\#\mathbf{N}_{0}}\sum_{(x,t)\in \mathbf{N}_{0}} |r_{0}(t,x,\theta)|^2,\nonumber\\
L_{b}(\theta)&=\frac{1}{\#\mathbf{N}_{b}}\sum_{(x,t)\in \mathbf{N}_{b}} |r_{b}(t,x,\theta)|^2, \nonumber
\end{align}
where $\mathbf{N}_{pde}$ is the collocation point set for the governing PDE, $\mathbf{N}_0$ represents the initial condition point set, and $\mathbf{N}_b$ represents a boundary point set. The notation $\#$ denotes the number of elements in a set. The total loss is defined by summing up the weighted loss terms:
\begin{align}\label{eq:total_loss}
L(\theta)=w_{pde}L_{pde}(\theta)+w_{0}L_{0}(\theta)+w_{b}L_b(\theta),
\end{align}
where $w_{pde}$, $w_{0}$, and $w_b$ are positive weights for each loss term. In the case of an inverse problem, the loss term for observed data $L_D(\theta)$ is appended to Eq.\eqref{eq:total_loss}. Finally, the solution $\hat u_\theta$ can be approximated by minimizing the total loss $L(\theta)$:
\begin{align}\label{eq:minimize}
\theta^\ast=\arg\min_\theta L(\theta).
\end{align}
In the vanilla PINN implementation, the PDE loss collocation node set $\mathbf{N}_{pde}$ is often generated by one-time sampling, and then the set is used during the whole training stage. Subsequent studies have pointed out that changing the point set $\mathbf{N}_{pde}$ during training may improve results.
%Some of these methods include PDE residual-based adaptive resampling during training.
The following numerical experiments also show that resampling the collocation node set $\mathbf{N}_{pde}$ after each iteration of a certain number of steps usually leads to a better result than the one-time sampling strategy.
\subsection{Hammersley Sampling}
The selection of $\mathbf{N_{pde}}$ seems to play an important role in the training of PINN.
An i.i.d. uniform random sampling strategy is usually employed to generate the collocation point set to solve Eq.\eqref{eq:minimize}. Existing software packages and implementations provided by previous studies also include the sampling strategies of quasi-uniform low-discrepancy sampling. The original implementation of PINN \cite{raissiPhysicsinformedNeuralNetworks2019} used Latin Hypercubic Sampling (LHS). IDRLnet \cite{Peng2021IDRLnetAP} provides jittered grid sampling. The authors of \cite{dasStateoftheArtReviewDesign2022} explored the construction of point sets from the perspective of experimental design, compared various methods, and found that Hammersley sampling works well. In the PINN library SimNet \cite{hennighNVIDIASimNetAIAccelerated2021}, Halton pseudo-random sequence is provided as an alternative algorithm for generating low-discrepancy points. Since Hammersley sampling is one of the motivations of this paper and for the subsequent experiments, we briefly introduce Hammersley sampling.
Hammersley sampling is a quasi-uniform sampling method with lower discrepancy than i.i.d. uniform random sampling. Given the number of samples $N$ and an arbitrary prime $p$, we can use Hammersley sampling to generate a point set in a 2D square $[0,1]\times[0,1]$:
\begin{align*}
\mathbf{N}=\left\{\left(\frac{k}{N}, \Phi_p(k)\right) | k=1,2,\cdots,N\right\}
\end{align*}
where $\Phi_p$ is the Van de Corput sequence which maps an integer
\begin{align*}
a_0+a_1p+a_2p^2+\cdots+a_rp^r
\end{align*}
to a decimal
\begin{align*}
a_0p^{-1}+a_1p^{-2}+a_2p^{-3}+\cdots+a_rp^{-r-1}.
\end{align*}
Sampling points in an arbitrary rectangle can be achieved by scaling the square.
This simple approach seems quite effective for PINN training. We conjecture that the low-discrepancy property or local-regularity makes the points evenly distributed on the computational domain, which increases the quality of the collocation point set.
On the other hand, the residual-based sampling method locally refines points on the high residual region and can also improve the efficiency of PINN training.
However, it is difficult for Hammersley sampling to vary the spatial density and thus hard to implement residual-based refinement directly.
Therefore, in the following subsection, we introduce a variable density high-quality node generation method used in meshless methods to combine these two aspects of potential advantages.
\subsection{Node Generation by Fornberg and Flyer}
Although Hammersley sampling seems effective in PINN, the quality of the generated collocation points is not suitable for traditional meshless solvers such as RBF-FD. Instead, Fornberg and Flyer proposed a node positioning algorithm in 2015 \cite{fornbergFastGeneration2D2015}. The foundamental form of the algorithm constructs
a point set in 2D rectangles and is described by Algorithm \ref{alg:ff}.
In the following discussion, we use the first letters of the two authors’ surnames (FF) to refer to the algorithm for simplicity.
FF is an advancing node generation method with variable density. Although there is no guarantee for the lower bound of the minimum distance of generated nodes, it still has substantially local regularity.
\begin{figure}[ht]
\begin{subfigure}{.3\textwidth}
\centering
% include first image
\includegraphics[width=1.\linewidth]{p1.png}
\caption{}
\label{fig:ff1}
\end{subfigure}
\begin{subfigure}{.3\textwidth}
\centering
% include second image
\includegraphics[width=1.\linewidth]{p2.png}
\caption{}
\label{fig:ff2}
\end{subfigure}
\begin{subfigure}{.3\textwidth}
\centering
% include second image
\includegraphics[width=1.\linewidth]{p3.png}
\caption{}
\label{fig:ff3}
\end{subfigure}
\caption{Illustration of Algorithm \ref{alg:ff}. The potential nodes are marked as solid red circles. The generated nodes are marked black diamonds. (a) Identify the lowest potential node in the $y$-axis direction. (b) Delete potential nodes within distance $r$. (c) Generate new potential nodes.}
\label{fig:ff}
\end{figure}
The core part of the FF method is sampling points in a 2D rectangle. Two sets of nodes are maintained. One set is the potential node set, and the other is the selected node set. An overview of the method is shown in Figure \ref{fig:ff}. The main steps are listed as follows.
\begin{enumerate}
\item Randomly initialize some nodes at the bottom of the rectangular as the potential point set $A$, and initialize the selected node set $B$ to be an empty set.
\item Find the lowest node in the $y$-axis direction from the potential node set $A$, and add the point to $B$. (Figure \ref{fig:ff1})
\item Remove all the potential points from $A$ with a distance less than $h(\hat x,\hat y)$ to the selected potential point. The radius can vary across spatial locations. (Figure \ref{fig:ff2})
\item The two neighboring points on the left and right sides of the node determine an arc with a radius of $h(\hat x,\hat y)$. Generate $5$ new equispaced nodes across the arc and add them to $A$. (Figure \ref{fig:ff3})
\item Repeat the above the 2-4 steps until the selected potential node exceeds the upper bound.
\end{enumerate}
The pseudo-code is given in Algorithm \ref{alg:ff}.
\begin{algorithm}
\caption{Node positioning algorithm by Fornberg and Flyer.}\label{alg:ff}
\begin{algorithmic}[1]
\Require Box $[x_{min}, x_{max}]\times[y_{min},y_{max}]$,
\Statex Radius function $h: [x_{min}, x_{max}]\times[y_{min}, y_{max}]\rightarrow (0,\infty)$
\Statex \textbf{Output} A set of points in the box distributed according to $h$.
\Function{FF}{$x_{min}, x_{max}, y_{min}, y_{max}, h$}
\State $B \gets $ an empty set of points;
\State $A\gets $ points spaced according to $h$ from $x_{min}$ to $x_{max}$ at $y$ coordinate $y_{min}$ with small positive perturbation for each point;
\State $y_{idx}, idx\gets \min(A.y)$;
\While{$y_{idx}\leq y_{max}$}
\State $p \gets A[idx]$;
\State Add $p$ to $B$;
\State Remove points closer than $h(p)$ from $A$;
\State Find nearest remaining points in candidates to the left and to the right of $p$;
\State Add $5$ new points to the potential node set $A$, equispaced on the circular sector with center $p$ and a radius of $h(p)$, spanning from the nearest left to the nearest right point;
\State $y_{idx}, idx\gets \min(A.y)$;
\EndWhile
\State\Return $B$
\EndFunction
\end{algorithmic}
\end{algorithm}
The original text did not provide the time complexity analysis, and the analysis was presented in a subsequent study \cite{slakGenerationNodeDistributions2019a}. According to the analysis, the algorithm has a complexity of $O(NS)$, where $S$ is the maximum size of the potential node set, and $N$ denotes the number of generated nodes.
%In practice, it is almost negligible for running time, which can be ignored compared to the expensive computational cost of training PINN.
In practice, the running time of FF can be ignored compared to the expensive computational cost of training PINN.
Since our subsequent experiments focus on time-dependent one-dimensional problems and a time-independent two-dimensional problem, where computational domains of the problems are essentially rectangles, FF is competent enough as the primary node generation method. More complex situations can also be handled by similar node generation frameworks. Due to the limited scope of the paper, we only consider 2D rectangular domains. We will discuss several potential improvements in the final section for higher-dimensional and more complex geometries.
\section{Methods}
We present a node generation method based on any given error map in the first subsection. In the second subsection, the method is then combined with the PINN, where PDE residuals are regarded as the error maps during the PINN training process.
\subsection{Error-based Adaptive Node Generation with Memory}
Suppose that a posteriori error map $e$ is given, where $e(x,y)$ represents an error indicator between an actual and computed value of some desired quantity at a point $(x,y)$. A larger absolute value of $e(x,y)$ indicates that the region near $(x,y)$ needs refinement. %另一方面，我们也希望能够构建一种记忆机制，在当前步采样中仍然对之前存在过较大误差的区域增加一些采样密度。
On the other hand, we hope to incorporate a prior error map, through which we can build a memory mechanism that refines the regions where large error values have previously existed. Our motivation for the step will be illustrated in the numerical experiment part (Section \ref{sec:ac}). By introducing a prior error map $\tilde e$, we implement Algorithm \ref{alg:ff-ar} based on Algorithm \ref{alg:ff}.
\begin{algorithm}
\caption{Adaptive Refinement Node Positioning algorithm with FF}\label{alg:ff-ar}
\begin{algorithmic}[1]
\Require Box $[x_{min}, x_{max}]\times[y_{min},y_{max}]$,
\Statex Error map $e: [x_{min}, x_{max}]\times[y_{min}, y_{max}]\rightarrow (-\infty,\infty)$,
\Statex Prior error map $\tilde e: [x_{min}, x_{max}]\times[y_{min}, y_{max}]\rightarrow [0,1]$,
\Statex Memory coefficient $\beta\in[0,1]$,
\Statex Density ratio $r\in[1,\infty)$,
\Statex Radius scaling parameter $s$.
\Statex \textbf{Output} A set of points in the box and a prior error map for the next call.
\Function{ARFF}{$x_{min}, x_{max}, y_{min}, y_{max}, e, \tilde e, \beta, r, s$}
\State $\bar e\gets \max\left\{\frac{|e|-\inf_B |e|}{\sup_B|e|-\inf_B|e|+\varepsilon}, \beta \tilde e\right\}$ \Comment{$\varepsilon$ is a small positive number}
\State $h\gets \left((1-\bar e)\times(1-1/\sqrt r)+1/\sqrt{r}\right)s$
\State $pts\gets FF(x_{min}, x_{max}, y_{min}, y_{max}, h)$
\State\Return $pts$, $\bar e$
\EndFunction
\end{algorithmic}
\end{algorithm}
%Algorithm \ref{alg:ff-ar}利用2个函数参数和三个实数参数来控制点的生成。在后面的使用中，我们并没有保存函数变量，而是利用均匀网格最近邻插值对函数进行了逼近。
Algorithm \ref{alg:ff-ar} requires two function parameters ($e, \tilde e$) and three real hyperparameters ($\beta, r, s$) as the inputs to control node generation. In subsequent uses, function variables are not directly kept. Instead, we use the nearest neighbor interpolation with values on a uniform grid to approximate the error map. Another possible option is to interpolate directly on the collocation point for approximation, replacing the uniform grid.
%\textbf{standardize the error function}：$e$是输入的误差函数，对于不同的应用其值域的变化范围非常大。因此，首先利用仿射变换将其值标准化到$[0,1)$区间上。结合前面计算步骤中给出的标准化的误差函数$\tilde e$，利用$\max$这个非线性算子，保持对误差较大区域的持续注意力：
\textbf{Standardize the error function}. Suppose $e$ is the error function as an input, the range of which can vary a lot in different applications. Therefore, values of the range are first standardized to $[0, 1)$. Combined with the standardized prior error function $\tilde e$, the nonlinear operator $\max$ is used to maintain sustained memory of large residual regions:
\begin{align*}
\bar e\gets \max\left\{\frac{|e|-\inf_B |e|}{\sup_B|e|-\inf_B|e|+\varepsilon}, \beta \tilde e\right\}
\end{align*}
The standardized error map $\bar e$ used in the current sampling depends on both the current error map $e$ and the prior error map $\tilde e$, where the memory coefficient $\beta$ controls the impact of $\tilde e$.
%当前的标准化误差函数$\bar e$由两方面决定，一部分是当前误差，一部分是提供的之前的标准化误差函数$\hat e$,由记忆参数$\beta$控制了前一次标准化误差函数$\hat e$的影响.
%特别地，当$\beta=0$时，不会受到之前的影响；而当$\beta=1$时，算法将不会遗忘之前残差较大的区域，持续对前面的大残差区域进行采样。在实验中，我们并没有特别优化，而是选取了$\beta=0.9$，相对$\beta=0.0$取得了较好的效果。
In particular, the memory coefficient $\beta=0.0$ implies that the memory mechanism will not work. While $\beta=1.0$, the algorithm will never forget the areas where large residuals have existed before.
Throughout all the experiments in this paper, the memory coefficient is not specially finetuned, and $\beta=0.9$ is applied to all cases.
%\textbf{Sampling radius function}：The sampling radius function $h$ controls the radius of FF, which controls the local density.
\textbf{Sampling radius function}. The sampling radius function $h$ directly controls the generation radius of the FF algorithm in each position, thereby controlling the local point density.
The current standardized error map $\bar e$ partly determines the radius function $h$.
The sampling radius at the large error position is small, thereby higher density; the sampling radius at the small error position is large, thereby lower density.
The two sampling parameters ($r,s$) determine the eventual maximum and minimum generation radius in the rectangle. The ratio parameter $r$ controls the maximum to minimum radius ratio, thereby determining the ratio of the maximum to minimum density. The scaling parameter $s$ controls the overall scaling of radius. In the 2D plane, the maximum and minimum sampling radius is estimated by
\begin{align*}
\max{h} = s,\quad \min{h}=s/\sqrt{r}.
\end{align*}
When the radius parameter $s$ is doubled, the density will reduce to a quarter of the original.
For an intuitive warm-up, we use the signed distance function (SDF) of an L-shape to construct an error map to illustrate the effects of parameters $r$ and $s$.
%为了直观介绍算法 \ref{alg:ff-ar}，我们使用SDF作为残差函数来展示参数$r$和$s$的效果。
As is shown in Figure \ref{Lshape}, considering sampling on the L-shape inside a box domain $[0,1]\times[0,1]$, define $e:=1-SDF$, which implies high density near the boundary.
%考虑$[0,1]\times[0,1]$区域上的L形上的采样，定义$e:=1-sdf$，也就是会对边界处进行加密的节点生成。采样后再使用SDF排除区域外的点.
Since Algorithm \ref{alg:ff} is designed for rectangles, we will sample nodes in the box domain and then remove nodes that lie outside of the L-shape.
Figure \ref{Lshape} shows the sampling results under different parameters. We can see that $s$ determines the radius in the lowest density area. Keep $s$ unchanged, and then $r$ controls the maximum to minimum density ratio.
%各个参数的结果如下图\ref{Lshape}所示。s决定了最低密度区域的采样半径。在s保持不变的情况下，r控制了最低密度与最高密度的比例。
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{Lshapescatter.png}
\caption{Nodal distribution in an L-shape domain. Sample nodes inside the L-shape according to $e(x,y):=1-SDF(x,y)$ under different scaling parameters $s$ and ratio parameters $r$. Each row has the same density distribution up to a constant factor, and the minimum distribution density in each column is the same.}
\label{Lshape}
\end{figure}
The sampling radius function $h$ uses only two necessary hyperparameters ($r, s$), and is implemented in a simple weighted linear model (Line 3 in Algorithm \ref{alg:ff-ar}). Throughout all the experiments in this paper, we fix the ratio parameters $r$ to be $100$. The selection of $s$ will be discussed in the following subsection.
\subsection{RANG for PINN}
%这种灵活的采样方式也带来了困难，即我们难以控制点的总数。因此为了控制输出点数来进行算法比较，我们会在PINN的求解中利用对参数$s$松弛的二分法来实现将点的数量控制在一个目标范围内。
\textbf{Resample the collocation node set.}
We define the resampling interval $I$, then the set will be re-generated every $I$ iterations. The collocation node set $\mathbf{N}_{pde}$ in Eq.\eqref{eq:pde_l} no longer stays unchanged. The PDE loss term Eq.\eqref{eq:pde_l} in the $i$-th iteration becomes
%The collocation node set $\mathbf{M}$ in Eq.\eqref{eq:pde_l} no longer stays unchanged. Instead, define the resampling interval $I$, then the set will be re-generated every $I$ iterations:
\begin{align}
L_{pde}(\theta)&=\frac{1}{\#\mathbf{M}_{\lfloor i/I\rfloor}}\sum_{(x,t)\in \mathbf{M}_{\lfloor i/I\rfloor}} |r_{pde}(t,x,\theta)|^2,
\end{align}
where $\mathbf{M}_{\lfloor i/I\rfloor}$ denotes the collocation point set used in the $i$-th iteration.
The PDE residual $r_{pde}$ is employed as the error map in Algorithm \ref{alg:ff-ar}, which is applied to generate collocation nodes every $I$ iterations. Then Algorithm \ref{alg:ff-ar} is incorporated into the training process, and we obtain the Residual-based Adaptive Node Generation method (RANG) for PINN described in Algorithm \ref{alg:PINN-arff}.
\textbf{Determine the scaling coefficient}.
Drastically varying the size of data sets may lead to a waste of storage resources such as RAM or video memory since we have to allocate enough space for the possible maximum size of the node set. Therefore we hope the number keeps roughly the same during training. However, when Algorithm \ref{alg:ff-ar} is applied to generate nodes, it is difficult to control the total number directly.
To keep the size of collocation point sets nearly constant, we employ a loose bisection search for the scale parameter $s$ to generate a designated amount of points during training PINN, as shown in lines 10-16 in Algorithm \ref{alg:PINN-arff}.
% \textbf{Memory mechanism}. We note that increasing sampling training in particular regions will decrease the PDE residual of the region within several iterations. However, once the $residual$ is decreased and collocation nodes in the region is reduced, the PDE residual will again increase and more nodes will again be sampled in this region. Therefore, the oscillating phenomenon of the sampling nodes will hamper the training effect. After the memorization is introduced, the training on the region can be strengthened lasting a period of time even after the residual is reduced, so that training becomes more robust. An intuitive example is given in Section 4.1. Combining Algorithm \ref{alg:ff-ar} and PINN, we obtain Algorithm \ref{alg:PINN-arff}.
\textbf{Memory mechanism}. We note that refining a local region will decrease the PDE residual of the local region within several iterations.
However, once the local residual is decreased and the node density of the local region is reduced adaptively, the local PDE residual will increase again in a few iterations, and more nodes will be sampled in the region again.
The periodic increase and decrease of the local density will be repeated several times, and the oscillation might hamper the training efficiency.
After the memory mechanism is introduced, the training on the region will continue strengthening for a period of time even after the local residual is reduced.
An intuitive example is given in Section \ref{sec:ac}.
%Combining the algorithm and PINN, we obtain the algorithm.
\begin{algorithm}
% \def\NoNumber#1{{\def\alglinenumber##1{}\State #1}\addtocounter{ALG@line}{-1}}
\caption{Residual-based Adaptive Node Generation for PINN (RANG)}\label{alg:PINN-arff}
\begin{algorithmic}[1]
\Require $s_{low}$, $s_{up}$ for bisection search of $s$,
\Statex Maximal iterations $Maxiter$,
\Statex Designated number of collocation points for PDE loss $N_{pde}$,
\Statex Resampling interval $I$.
\State Generate nodes for initial loss, boundary loss;
\State Generate nodes for PDE loss with FF with a constant radius;
\State $i\gets 0$
\While{$i<Maxiter$}
\If{$i\mod I=0$}
\If{$i=0$}
\State $e\gets 0$, $\tilde e\gets 0$
\Else
\State $e\gets\left|\mathcal{N}(t,x,\hat u(t,x;\theta))\right|$
% \State $e\gets\left|r_{pde}\left(x,y,t; \frac{\partial \hat{u}}{\partial x}, \frac{\partial \hat{u}}{\partial y} ; \frac{\partial^{2} \hat{u}}{\partial x \partial x}, \ldots, \frac{\partial^{2} \hat{u}}{\partial x\partial y} ; \ldots\right)\right|$
\EndIf
\State $\bar s_{low}\gets s_{low}$, $\bar s_{up}\gets s_{up}$
\State $s\gets (\bar s_{low}+\bar s_{up})/2$
\While{$|\#\mathbf{M}_{\lfloor i/I\rfloor}-N_{pde}|/|N_{pde}|>0.05$ or $\bar s_{up}-\bar s_{low}<0.003$}
\If{$\#\mathbf{M}_{\lfloor i/I\rfloor}<N_{pde}$}
\State $\bar s_{up}\gets s$
\Else
\State $\bar s_{low}\gets s$
\EndIf
\State $s\gets (\bar s_{low}+\bar s_{up})/2$
% \State $\mathbf{M}_0, \tilde e\gets ARFF(x_{min}, x_{max}, y_{min}, y_{max}, 1, 0, 0, 1, s)$
\State $\mathbf{M}_{\lfloor i/I\rfloor}, \bar e\gets ARFF(x_{min}, x_{max}, y_{min}, y_{max}, e, \tilde e, \beta, r, s)$
\EndWhile
\State $\tilde e\gets \bar e$
\EndIf
\State Compute the total loss $L$ in Eq.\eqref{eq:total_loss}.
\State $\hat u\gets Adam(L)$\Comment{or other optimizers such as L-BFGS}
\State $i\gets i+1$
\EndWhile
\end{algorithmic}
\end{algorithm}
\section{Results}
In order to distinguish memoryles ($\beta=0.0$) and memorable ($\beta=0.9$) sampling methods in the numerical experiments,
let RANG represent Algorithm \ref{alg:PINN-arff} with $\beta=0.0$, and denote RANG-m for the case where $\beta=0.9$ holds, which is short for \textit{RANG with Memory mechanism}.
This section employs nine sampling methods to generate collocation node sets.
We consider solving six different PDE equations, where the nonlinear equation includes a 1D Allen-Cahn equation, a 1D Schr\"odinger equation, and a 1D Korteweg-de Vries (KdV) equation, the linear equation includes a 1D wave equation, a 2D Poisson equation, and a 1D convection-diffusion equation. We start with the Allen-Cahn equation to illustrate the primary motivation of the memory mechanism. Then we observe the time-marching property of the RANG-m method in the wave equation. The remaining four PDEs will be tested in turn.
Since the performance of PINNs may heavily depend on hyperparameters, the optimal choices for these hyperparameters are problem-dependent.
However, we focus on discussing the effects of the sampling methods on the results, and we use the following settings throughout the section without finetuning.
For all problems, fully connected networks are used and contain four hidden layers, each with $64$ neurons.
The hyperbolic tangent $\tanh$ is used as the activation function for every layer except for the output layer.
The Adam method is used to train PINNs, with the learning rate fixed to $0.001$.
% However, we focus on discussing the effects of the sampling methods on the results, so the default settings above are fixed for all problems.
%FF方法本身是面向二维问题提出的，我们基于此的RANG算法也是针对二维问题。考虑时间维度的话，那么六组实验本质上都是二维问题。因此，下面会考虑矩形上的训练点采样方法。对于所有的例子，会同时考虑9种采样方法。
Due to the FF method being proposed for generating nodes in a two-dimensional rectangle, RANG based on the FF method is only suitable for two-dimensional problems.
Note that the domains of time-dependent one-dimensional PDEs are essentially two-dimensional.
Therefore, the following six experiments consider the collocation point sampling methods in rectangles, where nodes are generated in a square of $[0,1]\times[0,1]$ and mapped to the rectangle with an affine transformation.
For all examples, the following nine different sampling methods are taken into consideration:
\begin{enumerate}
\item \textbf{Random}. Nodes are independent and identically distributed samples on the domain, obeying the uniform distribution.
\item \textbf{Random-R}. Resample nodes using the Random method every $I$ iterations.
\item \textbf{Hammersley}. The Hammersely method with $p=2$ is used for node generation.
\item \textbf{LHS}. Latin hypercubic sampling is employed.
\item \textbf{LHS-R}. Resample nodes using LHS every $I$ iterations.
\item \textbf{FF}. Sample nodes with FF where the radius function is constant on the rectangular domain, and the radius constant is determined by a bi-section search.
\item \textbf{FF-R}. Resample nodes using FF every $I$ iterations.
\item \textbf{RANG}. Residual-based Adaptive Node Generation with the memory coefficient $\beta=0.0$ is employed. The method resamples nodes every $I$ iterations based on PDE residuals.
\item \textbf{RANG-m}. Residual Adaptive Node Generation with the memory coefficient $\beta=0.9$ is employed. The method also resamples nodes every $I$ iterations.
\end{enumerate}
The above nine sampling methods can be divided into two categories: the first category is the one-time sampling method, including Random, Hammersley, FF, and LHS, of which the sampling nodes are generated before training and will be used throughout the entire training process;
the second is the resampling method including the remaining five approaches.
In order to reduce the effects of other stochastic factors on the results, we only consider sampling methods applied to PDE-informed loss terms. For boundaries and initial conditions, by default, equispaced nodes are used.
Of course, for more complex boundaries, adaptive sampling should also be considered, and we will extend this method in future research.
For all examples, the accuracy of $\hat u$ is evaluated by the mean squared error (MSE) after training for a fixed number of iterations:
\begin{align*}
MSE(\hat u)=\frac{1}{\#\mathbf{G}}\sum_{i\in \mathbf{G}} \|u(t_i, x_i)-\hat u(t_i, x_i)\|^2
\end{align*}
where $u$ is the reference solution, and $\mathbf{G}$ is the testing set generated by a uniform grid.
%其中$G$为设置为均匀网格点集合。
All the source code is implemented with python and PyTorch, available online (\url{https://github.com/weipengOO98/rang_pinn}).
%代码由Python+pytorch实现，部分例子附带了tensorflow版本，可以在github上下载。
\subsection{Allen-Cahn Equation}\label{sec:ac}
The first nonlinear equation we solve is the Allen-Cahn equation. Consider the following form:
\begin{align*}
\frac{\partial u}{\partial t} - 0.0001\frac{\partial^2 u}{\partial x^2} + 5u^3 - 5u=0, \\
\quad(t,x)\in[0,1]\times[-1,1],\\
u(t,-1)=u(t,1)=-1,\\
u(0,x)=x^2\cos(\pi x).
\end{align*}
The neuron numbers for each layer are $[2,64,64,64,64,1]$.
The maximum iteration number is set to $50,000$, and the resampling interval $I$ is set to $1,000$. The number of replicates is $30$.
%初值条件损失项为：
The initial condition loss term is
\begin{align*}
L_0 = \frac{1}{\#\mathbf{N}_0}\sum_{x_0\in\mathbf{N}_0}\left(|u(0,x_0)-x_0^2\cos(\pi x_0)|^2+|u_x(0,x_0)|^2\right).
\end{align*}
The boudnary condition loss term is
\begin{align*}
L_b = \frac{1}{\# \mathbf{N}_b}\sum_{t_b\in\mathbf{N}_b}\left(|u(t_b,-1)+1|^2+|u(t_b,1)+1|^2\right).
\end{align*}
The designated number of collocation points $N_{pde}$ for the PDE loss term is set to $1,000$. The PDE loss term becomes
\begin{align*}
L_{pde} = \frac{1}{\#\mathbf{M}_{j}}\sum_{(\tilde x,\tilde t)\in \mathbf{M}_{j}}\left(\frac{\partial u(\tilde t,\tilde x)}{\partial t} - 0.0001\frac{\partial^2 u(\tilde t, \tilde x)}{\partial x^2} + 5u^3(\tilde t, \tilde x) - 5u(\tilde t, \tilde x)\right)^2,
\end{align*}
where $\mathbf{M}_j$ denotes the $j$-th node set employed, which is used from the $j\times10^3$-th iteration to the $(j+1)\times 10^3$-th iteration. For the resampling methods, the collocation point set $\mathbf{M}_j$ varies every $I$ iterations ($j=0,1,2,\cdots$). For the non-resampling methods, the PDE loss collocation point set $\mathbf{M}_j=\mathbf{M}_0$ holds, which stays unchanged. $\#\mathbf{M}_j=N_{pde}$ holds for Random, Random-R, LHS, LHS-R and Hammersley, and $\#\mathbf{M}_j\approx N_{pde}$ holds for FF, FF-R, RANG and RANG-m.
The total loss is a weighted sum of the initial value loss term, the boundary value loss term, and the PDE loss term.
We choose $w_0=2.0$, $w_b=2.0$, $w_{pde}=0.2$ in Eq.\eqref{eq:total_loss} for better enforcing the initial and boundary conditions:
\begin{align*}
L=2L_0+2L_b+0.2L_{pde}.
\end{align*}
Variants of PINNs have solved the Allen-Cahn equation in several studies.
However, as reported by Zhao \cite{zhaoSolvingAllenCahnCahnHilliard2021}, the vanilla PINN fails to solve the equation.
Figure \ref{fig:essemble_ac} depicts the reference solution and the results of nine different sampling methods.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{essemble_ac.png}
\caption{Solutions for the Allen-Cahn equation. The top left image exhibits the reference solution of the Allen-Cahn equation. The remaining nine images show the prediction results of nine sampling methods in one experiment. The results might be different in another run due to the randomness of sampling and training.}
\label{fig:essemble_ac}
\end{figure}
As shown in Figure \ref{fig:ac_it}, it tends to saturate for an extended period in the training process of PINNs. After a lengthy computation, the MSE may fall drastically, where RANG-m has the fastest decline.
After $50,000$ iteration, statistics of MSEs obtained by nine sampling methods are shown in Figure \ref{fig:ac_error} and Table \ref{tab:ac}. All statistics indicate that RANG-m achieves the best performance. Consistent with the previous result \cite{dasStateoftheArtReviewDesign2022}, the Hammersley sampling performs best among the four non-resampling methods. In this example, we can also see that RANG-m is more stable than RANG.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{ac_iteration.png}
\caption{MSE values in iteration for the Allen-Cahn equation. This figure shows the curve of the median of MSE corresponding to the nine different methods. Note that four methods (Random/LHS/Random-R/LHS-R) are almost overlapped, and there is nearly no decrease in MSE. MSE values of the remaining methods decline, which might benefit from resampling and the local quasi-uniform properties.}
\label{fig:ac_it}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{ac_errorplot.png}
\caption{Box plot of MSEs for Allan-Cahn equation considering different node generation strategies. The plot represents median and 25, 75\% interquartile and extreme values. The mean values are marked with triangles.}
\label{fig:ac_error}
\end{figure}
\begin{table}
\begin{tabular}{l|rrrrrr}
\hline
{} & mean & min & 25\% & 50\% & 75\% & max \\
\hline
Hammersley & 3.814E-03 & 1.135E-03 & 1.571E-03 & 1.959E-03 & 3.922E-03 & 1.788E-02 \\
Random & 1.382E-01 & 2.110E-03 & 1.403E-01 & 1.480E-01 & 1.589E-01 & 1.891E-01 \\
LHS & 1.310E-01 & 1.317E-03 & 1.356E-01 & 1.425E-01 & 1.495E-01 & 1.755E-01 \\
FF & 7.559E-02 & 9.563E-04 & 1.718E-03 & 4.085E-02 & 1.443E-01 & 1.910E-01 \\
Random-R & 1.366E-01 & 7.051E-03 & 1.376E-01 & 1.410E-01 & 1.501E-01 & 1.730E-01 \\
LHS-R & 1.265E-01 & 1.491E-03 & 1.362E-01 & 1.390E-01 & 1.486E-01 & 1.777E-01 \\
FF-R & 5.175E-02 & 1.097E-03 & 1.741E-03 & 4.086E-03 & 1.274E-01 & 1.688E-01 \\
RANG & 2.094E-02 & 9.378E-04 & 1.416E-03 & 1.869E-03 & 2.526E-03 & 1.574E-01 \\
RANG-m & \textbf{1.537E-03} & \textbf{8.052E-04} & \textbf{1.266E-03} & \textbf{1.448E-03} & \textbf{1.690E-03} & \textbf{2.702E-03} \\
\hline
\end{tabular}
\caption{The statistics of MSE for the Allen-Cahn equation considering different node generation strategies over 30 replicates. The optimal value is marked in bold for each statistic. The RANG-m method performs best in various statistics.}
\label{tab:ac}
\end{table}
Next we will show the motivation for introducing the memory mechanism.
%图\ref{fig:jump0}展示了RANG方法在迭代过程中对局部区域进行refine。
Figure \ref{fig:jump0} shows that the RANG method refines two local regions during training (from 15,000-th to 23,000-th iteration).
The collocation point set is resampled every $1,000$ iterations. It can be seen that RANG repeatedly refines two regions in turn. When region A is refined, the local PDE residual on region B will again increase, such that refinement is again required on region B. We propose the memory mechanism, which continues refining both A and B such that the periodic oscillation is prevented. As is shown in Figure \ref{fig:jump9}, the refinement seems steady during the training, which might benefit the stability of the final results of RANG-m.
%每一千代进行了重新采样。可以发现RANG对两个区域进行反复检查，对另一个区域进行加密之后，之前训练的区域的残差又会重复增大。由于这种残差重复增大的现象会在一定迭代内反复出现，因此我们提出了记忆化的变种技术，如图\ref{fig:jump9}所示，训练过程中，加密区域比较稳定，这可能也是RANG-m在多个算例中结果稳定性更优的原因。
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{jump0.png}
\caption{Adaptive nodal distribution via RANG. The top left image is a reference solution for the Allen-Cahn equation. The remaining nine images depict the RANG collocation nodes from 15,000-th to 23,000-th iteration. An interesting phenomenon is the periodic refinement of two different regions. This oscillation may hamper convergence speed.}
\label{fig:jump0}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{jump9.png}
\caption{Adaptive nodal distribution via RANG-m. The top left image is a reference solution for the Allen-Cahn equation. The remaining nine images depict the RANG-m collocation nodes from 15,000-th to 23,000-th iteration. The memory mechanism can continuously refine the region that previously had a large residual for a period of time.}
\label{fig:jump9}
\end{figure}
\subsection{1D Wave Equation}
The first linear PDE equation we discuss is a one-dimensional wave equation that describes the deflection of a linear string over time and space.
Given an interval $[-l,l]$ with fixed endpoints ($l=4$), the governing PDE equation with the initial and boundary conditions for the wave equation is given by
%考虑固定端点情况下，$[-L,L]$ 区间上的波传播(其中L=5)和反射。其PDE形式如下：
\begin{align*}
\frac{\partial^2 u}{\partial t^2}&-3\frac{\partial^2 u}{\partial x^2}=0,\\
u(0,x)&=\frac{1}{\cosh(2x )}-\frac{0.5}{\cosh(2(x-2l))}-\frac{0.5}{\cosh(2(x+2l))},\\
\frac{\partial u}{\partial t}(0,x)&=0,\\
u(t,-l)&=u(t,l)=0,
\end{align*}
where the exact solution within the range $t\in[0,6]$ is
%其在时间域$[0,6]$上确切解为
\begin{align*}
u^{ref}(x,t)=&\frac{0.5}{\cosh(2(x+\sqrt{3}t))}-\frac{0.5}{\cosh(2(x-2l+\sqrt{3}t))}\\
&+\frac{0.5}{\cosh(2(x-\sqrt{3}t))}-\frac{0.5}{\cosh(2(x+2l-\sqrt{3}t))},
\end{align*}
which describes a pulse propagating on both sides and reflects at the boundary. %设置神经网络每一层的单元数为[2,64,64,64,64,1]。
The neuron number for each layer is set to $[2,64,64,64,64,1]$, respectively.
%总迭代次数为15k，对于重采样方法，每1k次迭代进行一次重采样，每次采样点数设置为1000.
%初值条件损失项为初始位置损失与初始速度损失项之和。
The initial condition loss consists of the initial position and velocity loss:
\begin{align*}
L_0 = \frac{1}{\#\mathbf{N}_0}\sum_{x_0\in\mathbf{N}_0}\left(|u(0,x_0)-u^{ref}(0,x_0)|^2+|u_x(0,x_0)|^2\right).
\end{align*}
The Dirichlet boundary loss term is
\begin{align*}
L_b = \frac{1}{\#\mathbf{N}_b}\sum_{t_b\in\mathbf{N}_b}\left(|u(t_b,-l)|^2+|u(t_b,l)|^2\right)
\end{align*}
%PDE损失为
The PDE loss term is given by
\begin{align*}
L_{pde} = \frac{1}{\# \mathbf{M}_{j}}\sum_{(\tilde x,\tilde t)\in \mathbf{M}_{j}}\left( \frac{\partial^2 u(\tilde t,\tilde x)}{\partial t^2}-3\frac{\partial^2 u(\tilde t,\tilde x)}{\partial x^2}\right)^2
\end{align*}
where $\mathbf{M}_j$ denotes the $j$-th node set employed. Since we set the resampling interval $I=1,000$, the collocation point set $M_j$ changes every $1,000$ iterations for the five resampling methods ($j=0,1,2,\cdots$). For the four non-resampling methods, the PDE loss collocation point set $\mathbf{M}_j=\mathbf{M}_0$ holds, which stays unchanged during the training.
The total loss is a weighted sum of the initial value loss term, the boundary value loss term, and the PDE loss term. We also choose $w_0=2.0, w_b=2.0, w_{pde}=0.2$ for better enforcing the initial and boundary conditions:
\begin{align*}
L=2L_0+2L_b+0.2L_{pde}.
\end{align*}
Set the maximum iterations to be $15,000$, and the number of replicates is $50$. The designated capacity of collocation point set is $N_{pde}=1000$.
Figure \ref{fig:scatter_exp} displays the collocation point set $\mathbf{M}_3$ which is used from 3,000-th to 4,000-th iteration. Hammersley, FF, and FF-R exhibit low-discrepancy and their nodes are more evenly spread on the domain than Random. The nodes generated by RANG and RANG-m look locally regular, and their local density varies adaptivly according to the current PDE residual.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{scatters_3000.png}
\caption{Nodal distribution for the wave equation at the 3,000-th iteration. The top left image shows the exact solution. The remaining nine images depict $\mathbf{M}_3$ at iteration 3,000 for each approach. The four node sets on the first row remain unchanged in the entire iteration process. The second row displays the five resampling methods that resample nodes every 1,000 iterations. RANG and RANG-m in the bottom right images refine the region of the fluctuation propagation trajectory in early iterations.}
\label{fig:scatter_exp}
\end{figure}
Figure \ref{fig:time_marching} depicts the collocation nodes generated by RANG-m in different iterations.
Figure \ref{fig:time_marching_hist} shows the frequency histogram of the sample point along the time axis.
It is shown that the sampling nodes are concentrated near $t=0$ in the early iterations.
As the iteration goes on, the number of samples in subsequent regions gradually increases.
After the training ends, the distribution of sampling points in the time axis also becomes uniform. The RANG method automatically increases the weight of the initial area, which may contribute to the training of PINNs.
In this problem, the RANG method has a certain time-marching feature, which conforms to the causal rules of evolutionary equations.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{time_marching.png}
\caption{
%每一列描述了RANG-m方法在对应迭代数的采样结果与预测解。随着迭代进行，采样点逐渐变得均匀。
Nodal distributions and corresponding predicted solutions at different iterations. Each column displays the collocation set node and prediction of the RANG-m method at the same iteration. As the iteration goes on, the sampling nodes are gradually distributed evenly along the time axis.}
\label{fig:time_marching}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{time_marching_hist.png}
\caption{
%数量关于时间轴的统计直方图。在迭代的初始阶段，采样点也集中在时间轴的初始处，随着迭代的进行，采样点逐渐在全区域上分布均匀。
Histogram of nodes along the time axis. In the initial iteration stage, the sampling point is concentrated near $t=0$. The collocation points are gradually distributed evenly on the whole region with iteration going on.}
\label{fig:time_marching_hist}
\end{figure}
%图\ref{fig:essemble_wave}展示了经过15,000次迭代，不同采样方法对应的一次结果。
Figure \ref{fig:essemble_wave} depicts the reference solution and the results of nine different sampling methods after $15,000$ iterations.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{essemble_wave.png}
\caption{Solutions for the wave equation. The top left image exhibits the reference solution of the wave equation. The remaining nine images show the prediction results of nine sampling methods in one experiment. The results might be different in another run due to the randomness of sampling and training.}
\label{fig:essemble_wave}
\end{figure}
Figure \ref{fig:wave_it} shows that FF sampling has the best performance among the non-resampling methods, even better than Hammersley. In this example, the four FF based methods (FF, FF-R, RANG, RANG-m) all perform well.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{wave_iteration.png}
\caption{MSE values in iteration for the wave equation. This figure shows the curve of the median of MSE corresponding to the nine different methods.}
\label{fig:wave_it}
\end{figure}
%图\ref{fig:wave_error}与表格\ref{tab:wave}展示了经过15000次迭代后的MSE的多个统计量。尽管RANG-m/FF/FF-R/RANG方法在中位数上较为接近，但是RANG-m的平均值要小得多，这是来源于RANG-m方法最为稳健，结果更加集中，通常都能保证一个不差的收敛。
Figure \ref{fig:wave_error} and Table \ref{tab:wave} display several statistics about MSE after 15,000 iterations with 50 replicates. Although the medians of RANF, FF, FF-R, and RANG-m are relatively close, the average value of RANG-m is much smaller, which implies the method may avoid poor convergence. The stability of RANG-m might benefit from the memory mechanism.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{wave_errorplot.png}
\caption{Box plot of MSEs for the wave equation considering different node generation strategies. The plot represents median and 25, 75\% interquartile and extreme values. The mean values are marked with triangles.}
\label{fig:wave_error}
\end{figure}
\begin{table}
\begin{tabular}{l|rrrrrr}
\hline
{} & mean & min & 25\% & 50\% & 75\% & max \\
\hline
Hammersley & 3.793E-02 & 2.872E-05 & 2.066E-04 & 2.547E-02 & 6.058E-02 & 1.884E-01 \\
Random & 7.248E-02 & 9.773E-03 & 4.889E-02 & 5.897E-02 & 7.826E-02 & 2.988E-01 \\
LHS & 5.765E-02 & 1.101E-02 & 4.159E-02 & 5.274E-02 & 6.480E-02 & 1.535E-01 \\
FF & 9.878E-03 & 3.009E-05 & 6.422E-05 & 1.412E-04 & 9.143E-04 & 1.876E-01 \\
Random-R & 5.155E-02 & 6.457E-05 & 2.958E-02 & 4.719E-02 & 6.474E-02 & 1.560E-01 \\
LHS-R & 4.947E-02 & 6.648E-04 & 2.801E-02 & 4.744E-02 & 6.340E-02 & 1.220E-01 \\
FF-R & 1.910E-03 & 3.302E-05 & 7.295E-05 & 1.290E-04 & 6.270E-04 & 5.092E-02 \\
RANG & 4.790E-04 & \textbf{2.289E-05} & \textbf{5.153E-05} & \textbf{6.919E-05} & \textbf{1.289E-04} & 5.683E-03 \\
RANG-m & \textbf{1.508E-04} & 3.479E-05 & 5.878E-05 & 8.051E-05 & 1.485E-04 & \textbf{1.069E-03} \\
\hline
\end{tabular}
\caption{
The statistics of MSE for the wave equation considering different node generation strategies over 50 replicates. The optimal value is marked in bold for each statistic. RANG-m performs best in the \textit{mean} column.}
\label{tab:wave}
\end{table}
\subsection{1D Sch\"odinger Euqation}
The 1D Sch\"odinger equation is the second nonlinear equation in our experiments. Consider the following complex-valued PDE:
\begin{align*}
\mathrm{i} \frac{\partial u}{\partial t}+\frac{1}{2}\frac{\partial^2 u}{\partial x^2}+|u|^2\cdot u=0,\\
u(0,x)=2\mathrm{sech}(x),\\
u(t,-5)=u(t,5),\\
\frac{\partial u}{\partial x}(t,-5)=\frac{\partial u}{\partial x}(t,5),
\end{align*}
where
\begin{align*}
(x,t)\in \Omega = [-5,5]\times[0,\pi/2].
\end{align*}
The solution $u$ is complex-valued, which can be equivalently represented by two real-valued functions, so the number of output layer neurons of the neural network $\hat u$ is $2$.
The number of neurons in each layer is $[2, 64, 64, 64, 64, 2]$, respectively.
The initial condition loss term is
\begin{align*}
L_0 = \frac{1}{\#\mathbf{N}_0}\sum_{x_0\in\mathbf{N}_0}|u(0,x_0)-2\cdot\mathrm{sech}(x_0)|^2.
\end{align*}
The boundary condition loss term is
\begin{align*}
L_b = \frac{1}{\#\mathbf{N}_b}\sum_{t_b\in\mathbf{N}_0}\left(|u(t_b,-5)-u(t_b, 5)|^2+|u_x(t_b,-5)-u_x(t_b, 5)|^2\right),
\end{align*}
where $u(t,x)=v(t,x)+\mathrm{i}\cdot w(t,x)$, $v,w$ are two real-valued functions. The complex derivative $u_x$ is defined as
\begin{align*}
u_x:=\frac{\partial v}{\partial x}+\mathrm{i}\frac{\partial w}{\partial x}.
\end{align*}
The governing PDE loss is
\begin{align*}
L_{pde} = \frac{1}{\# \mathbf{M}_j}\sum_{(\tilde x,\tilde t)\in \mathbf{M}_{j}}\left(\left|\mathrm{i} \frac{\partial u}{\partial t}+\frac{1}{2}\frac{\partial^2 u}{\partial x^2}+|u|^2\cdot u\right|^2\right).
\end{align*}
For the non-resampling methods, the PDE loss collocation point set $\mathbf{M}_j=\mathbf{M}_0$ holds, which stays unchanged. The total loss is a weighted sum of the initial value loss term, the boundary value loss term and the PDE loss term.
We also choose $w_0=2.0, w_b=2.0, w_{pde}=0.2$ for better enforcing the initial and boundary conditions:
\begin{align*}
L=2L_0+2L_b+0.2L_{pde}.
\end{align*}
The numerical reference solution could be obtained via Fourier spectral semi-discrete scheme with Runge-Kutta integration. We refer to the numerical solution provided by \cite{raissiPhysicsinformedNeuralNetworks2019}.
%以上方程数值解可由傅立叶谱方法结合Runge-Kutta格式得到，这里参考Raissi提供的一个数值解。
%设置总迭代次数为50000，重复次数为30.
The maximum number of iterations is set to $50,000$, and the number of replicates is $30$. The designated collocation point number $N_{pde}$ is set to $1,000$.
%图\ref{fig:essemble_schodinger}展示了参考真值和一次计算后9种方法得到的结果。
Figure \ref{fig:essemble_schodinger} depicts the reference solution and results of the nine different sampling methods.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{essemble_schodinger.png}
\caption{Solutions for the sch\"odinger equation. The top left image exhibits the reference solution. The remaining nine images show the prediction results of $9$ sampling methods in one experiment. The results might be different in another run due to the randomness of sampling and training.
}
\label{fig:essemble_schodinger}
\end{figure}
As is shown in Figure \ref{fig:schodinger_it}, MSE curves of the four non-resampling methods decrease in the early iterations but increase later. The U-shape might be caused by the limited training collocation points, which leads to over-fitting after several iterations.
The resampling methods seem to avoid this issue, but Random-R, FF-R, and LHS-R quickly fall into a plateau period, and the MSE no longer declines.
The two adaptive sampling methods, RANG and RANG-m, converge fast, and RANG-m endowed with the memory mechanism is better than RANG.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{schodinger_iteration.png}
\caption{MSE values in iteration for the Sch\"odinger equation. This figure shows the curve of the median of MSE corresponding to nine different methods for the Sch\"odinger equation.}
\label{fig:schodinger_it}
\end{figure}
To compare the methods, the MSE obtained by various sampling methods is shown in Figure \ref{fig:schodinger_error} and Table \ref{tab:schodinger} after training ends. All statistics indicate that RANG-m achieves the best performance.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{schodinger_errorplot.png}
\caption{Box plot of MSEs for the Sch\"odinger equation considering different node generation strategies. The box plot represents median and 25, 75\% interquartile and extreme values of MSE for the sch\"odinger equation. The mean values are marked with triangles.}
\label{fig:schodinger_error}
\end{figure}
\begin{table}
\begin{tabular}{l|rrrrrr}
\hline
{} & mean & min & 25\% & 50\% & 75\% & max \\
\hline
Hammersley & 4.038E-01 & 2.166E-01 & 3.639E-01 & 4.053E-01 & 4.591E-01 & 5.248E-01 \\
Random & 8.748E-01 & 5.907E-01 & 7.978E-01 & 8.672E-01 & 9.631E-01 & 1.167E+00 \\
LHS & 8.303E-01 & 5.554E-01 & 7.201E-01 & 8.488E-01 & 9.318E-01 & 1.080E+00 \\
FF & 5.760E-01 & 2.782E-01 & 4.678E-01 & 5.675E-01 & 6.523E-01 & 8.354E-01 \\
Random-R & 3.376E-01 & 1.864E-01 & 2.663E-01 & 3.074E-01 & 4.106E-01 & 7.143E-01 \\
LHS-R & 2.648E-01 & 9.134E-02 & 1.957E-01 & 2.525E-01 & 3.181E-01 & 4.969E-01 \\
FF-R & 1.225E-01 & 3.058E-02 & 7.762E-02 & 1.018E-01 & 1.713E-01 & 3.098E-01 \\
RANG & 3.475E-02 & 1.756E-02 & 2.473E-02 & 2.986E-02 & 4.477E-02 & 7.993E-02 \\
RANG-m & \textbf{1.415E-02} & \textbf{9.669E-03} & \textbf{1.256E-02} & \textbf{1.422E-02} & \textbf{1.520E-02} & \textbf{2.173E-02} \\
\hline
\end{tabular}
\caption{The statistics of MSE for the Sch\"odinger equation considering different node generation strategies over 30 replicates. The optimal value is marked in bold for each statistic. The RANG-m method performs best in various statistics.}
\label{tab:schodinger}
\end{table}
\subsection{Korteweg-de Vries Equation}
The last nonlinear PDE we consider is the 1D Korteweg-de Vries (KdV) Equation.
Its governing equation includes high-order derivatives. We consider the KdV equation in the following form:
\begin{align*}
\frac{\partial u}{\partial t} + 6u \frac{\partial u}{\partial x} + \frac{\partial^3u}{\partial x^3}=0,
\end{align*}
where the solitary wave solution is
\begin{align*}
u(t,x)=\frac{c}{2} \mathrm{sech}\left(\frac{\sqrt{c}}{2}(x - ct)\right)^2.
\end{align*}
Take $c=7.0$ and define the initial value condition:
\begin{align*}
u(0,x)=\frac{c}{2} \mathrm{sech}\left(\frac{\sqrt{c}}{2}(x+7)\right)^2.
\end{align*}
To handle the boundary value condition, we define a wide spatial domain $[-4\pi, 4\pi]$, where $u$ is nearly $0$ at two ends:
\begin{align*}
u(t,-4\pi)&=0,\\
u(t,4\pi)&=0, \quad t\in[0,2].
\end{align*}
Define the initial and boundary value condition loss terms like the previous subsection, and we obtain the weighted total loss:
\begin{align*}
L=2L_0+2L_b+0.2L_{pde}.
\end{align*}
%设置总迭代次数为50000，重复次数为20.
The maximum iteration number is $50,000$, and the number of replicates is $50$. The capacity of collocation point set is $N_{pde}=1000$. Figure \ref{fig:essemble_kdv} depicts the reference solution and the results of nine different sampling methods after training ends.
%图\ref{fig:essemble_schodinger}展示了经过50k迭代，参考解与9种不同采样算法下的结果。
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{essemble_kdv.png}
\caption{Solutions for the KdV equation. The top left image exhibits the reference solution of the KdV equation. The remaining nine images show the prediction results of nine sampling methods in one experiment. The results might be different in another run due to the randomness of sampling and training. The four non-resampling methods in the first row have significant errors.}
\label{fig:essemble_kdv}
\end{figure}
%图\ref{fig:kdv_it}展示了几种方法随迭代的进行，MSE（中位数）的变化。
As is shown in Figure \ref{fig:kdv_it}, it is hard for the four non-resampling methods to converge to the exact solution, which all fail to capture the motion of the solitary wave solution in Figure \ref{fig:essemble_kdv}. The two adaptive methods, RANG and RANG-m, perform best among the nine methods.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{kdv_iteration.png}
\caption{%本图展示了对于KDV方程，随迭代的进行，关于9种不同方法对应的MSE中位数的曲线，四种非重采样方法重合，MSE几乎没有下降。非自适应的三种重采样方法（Random-R/LHS-R/FF-R）有下降，效果比较好的是两种自适应重采样方法（RANG和RANG-m）.
MSE values in iteration for the KdV equation. This figure shows the curve of the median of MSE corresponding to nine different methods for the KdV equation. Note that the four methods (Random/LHS/Random-R/LHS-R) are almost overlapped, and there is no decrease in their MSE values. MSE of the remaining methods decline, which might benefit from the local quasi-uniform properties.
}
\label{fig:kdv_it}
\end{figure}
MSE values obtained by nine sampling methods are shown in Figure \ref{fig:kdv_error} and Table \ref{tab:kdv}. All statistics indicate that RANG-m achieves the best performance.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{kdv_errorplot.png}
\caption{Box plot of MSEs for the KdV equation considering different node generation strategies. The plot represents median and 25, 75\% interquartile and extreme values. The mean values are marked with triangles.}
\label{fig:kdv_error}
\end{figure}
\begin{table}
\begin{tabular}{l|rrrrrr}
\hline
{} & mean & min & 25\% & 50\% & 75\% & max \\
\hline
Hammersley & 4.913E-01 & 3.567E-01 & 4.679E-01 & 5.043E-01 & 5.280E-01 & 5.471E-01 \\
Random & 4.699E-01 & 3.617E-01 & 4.486E-01 & 4.775E-01 & 4.976E-01 & 5.313E-01 \\
LHS & 4.712E-01 & 3.316E-01 & 4.457E-01 & 4.816E-01 & 5.073E-01 & 5.408E-01 \\
FF & 4.664E-01 & 3.690E-01 & 4.410E-01 & 4.665E-01 & 5.026E-01 & 5.342E-01 \\
Random-R & 2.679E-01 & 1.444E-02 & 1.293E-01 & 2.820E-01 & 4.155E-01 & 5.136E-01 \\
LHS-R & 2.346E-01 & 6.175E-03 & 1.138E-01 & 2.268E-01 & 3.423E-01 & 5.039E-01 \\
FF-R & 1.926E-01 & 5.381E-03 & 6.253E-02 & 1.422E-01 & 3.206E-01 & 4.792E-01 \\
RANG & 7.387E-02 & 2.642E-04 & 1.137E-02 & 3.888E-02 & 1.049E-01 & 3.380E-01 \\
RANG-m & \textbf{3.050E-02} & \textbf{9.484E-06} & \textbf{3.574E-03} & \textbf{1.309E-02} & \textbf{2.979E-02} & \textbf{3.171E-01} \\
\hline
\end{tabular}
\caption{The statistics of MSE for the KdV equation considering different node generation strategies over $50$ replicates. The optimal value is marked in bold for each statistic. The RANG-m method performs best in various statistics.}
\label{tab:kdv}
\end{table}
\subsection{2D Poisson Equation}
The second linear PDE we consider is a 2D Poisson equation in a square:
%讨论的第二类线性方程是二维泊松方程，泊松方程被广泛应用在固体力学/流体力学/热力学等领域。在二维正方形区域中考虑如下形式的泊松方程：
\begin{align*}
\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2}&=w_{\sigma}(x,y),\quad (x,y)\in\Omega=[-1,1]\times[-1,1]\\
u|_{\partial \Omega}&=0,
\end{align*}
where
\begin{align*}
w_\sigma(x,y)=\frac{-2\sigma^2 + (x-0.3)^2+(y-0.3)^2}{\sigma^4}\exp\left(-\frac{(x-0.3)^2 + (y-0.3)^2}{2\sigma^2}\right)\\-\frac{-2\sigma^2 + (x+0.3)^2+(y+0.3)^2}{\sigma^4}\exp\left(-\frac{(x+0.3)^2 + (y+0.3)^2}{2\sigma^2}\right).
\end{align*}
The ground truth is the superposition of two opposite Gaussian pulses:
%其真解为两个异号的高斯脉冲的叠加:
\begin{align*}
u(x,y) = g_\sigma(x-0.3,y-0.3)- g_\sigma(x+0.3,y+0.3)
\end{align*}
where the Gaussian pulse function $g_\sigma(x,y)$ is defined by
\begin{align*}
g_\sigma(x,y)=\exp\left(-\frac{x^2 + y ^2}{2\sigma^2}\right).
\end{align*}
Due to the lack of the initial value conditions, only the boundary condition loss $L_b$ and the PDE loss $L_{pde}$ are taken into consideration. Take $\sigma=0.1$, and the solution $u$ restricted on the boundary is nearly zero. Then we define the boundary value loss term:
\begin{align*}
L_b = \frac{1}{4\#\mathbf{N}_b}\sum_{x_b\in\mathbf{N}_b}\left(|u(x_b,-1)|^2+|u(x_b,1)|^2+|u(-1,x_b)|^2+|u(1,x_b)|^2\right).
\end{align*}
The PDE loss term is defined as
\begin{align*}
L_{pde} = \frac{1}{\#\mathbf{M}_j}\sum_{(\tilde x,\tilde y)\in \mathbf{M}_{j}}\left|\frac{\partial^2 u}{\partial x^2}(\tilde x,\tilde y)+\frac{\partial^2 u}{\partial y^2}(\tilde x,\tilde y)-g_\sigma(\tilde x,\tilde y)\right|^2.
\end{align*}
The weighted total loss is
\begin{align*}
L=2\cdot L_b+0.2\cdot L_{pde}.
\end{align*}
The neuron number for each layer is $[2,64,64,64,64,1]$.
The maximum iteration number is set to 3,000, and the resampling interval $I$ is set to $100$. The number of replicates is $100$.
The designated capacity of collocation point set is $N_{pde}=400$.
Figure \ref{fig:essemble_poisson} depicts the reference solution and the results of nine different sampling methods.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{essemble_poisson.png}
\caption{Solutions for the Poisson equation. The top left image exhibits the reference solution of the Poisson equation. The remaining nine images show the prediction results of nine sampling methods in one experiment. The results might be different in another run due to the randomness of sampling and training.
}
\label{fig:essemble_poisson}
\end{figure}
%其各代的MSE的中值结果如图\ref{fig:poisson_it}所示。几种非重采样方法随着迭代进行，误差不再较缓慢。几类重采样方法有更高的精度，其中RANG-m的效果优于其他几类重采样方法：
As shown in Figure \ref{fig:poisson_it}, the error curves of the four non-resampling methods decline slowly. RANG and RANG-m perform better than the others.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{poisson_iteration.png}
\caption{MSE values in iteration for the Poisson equation. This figure shows the curve of the median of MSE corresponding to the nine different methods for the Poisson equation. The four non-resampling methods (Hammersley/Random/FF/LHS) are obviously slower than the five resampling methods.}
\label{fig:poisson_it}
\end{figure}
After 3,000 iteration, the MSE obtained by various sampling methods is shown in Figure \ref{fig:poisson_error} and Table \ref{tab:poisson}.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{poisson_errorplot.png}
\caption{Box plot of MSEs for the Poisson equation considering different node generation strategies. The box represents median and 25, 75\% interquartile and extreme values. The mean values are marked with triangles.}
\label{fig:poisson_error}
\end{figure}
\begin{table}
\begin{tabular}{lrrrrrr}
\hline
{} & mean & min & 25\% & 50\% & 75\% & max \\
\hline
Hammersley & 4.805E-03 & 8.556E-05 & 2.034E-03 & 3.850E-03 & 6.379E-03 & 2.874E-02 \\
Random & 1.517E-02 & 9.231E-04 & 3.428E-03 & 8.514E-03 & 1.868E-02 & 1.448E-01 \\
LHS & 1.225E-02 & 3.490E-04 & 4.204E-03 & 6.928E-03 & 1.606E-02 & 9.605E-02 \\
FF & 1.076E-02 & 4.498E-05 & 2.041E-03 & 5.562E-03 & 1.280E-02 & 7.144E-02 \\
Random-R & 3.185E-04 & 2.899E-05 & 1.137E-04 & 2.325E-04 & 4.086E-04 & 1.302E-03 \\
LHS-R & 3.420E-04 & 2.383E-05 & 1.417E-04 & 2.679E-04 & 4.482E-04 & 1.624E-03 \\
FF-R & 2.528E-04 & 1.889E-05 & 6.966E-05 & 1.760E-04 & 3.225E-04 & 1.667E-03 \\
RANG & 1.158E-04 & \textbf{7.146E-06} & 3.417E-05 & 6.461E-05 & 1.474E-04 & 1.102E-03 \\
RANG-m & \textbf{7.970E-05} & 7.286E-06 & \textbf{1.818E-05} & \textbf{4.645E-05} & \textbf{1.192E-04} & \textbf{6.610E-04} \\
\hline
\end{tabular}
\caption{The statistics of MSE for the Poisson equation considering different node generation strategies over $100$ replicates. The optimal value is marked in bold for each statistic. The RANG-m method performs best in various statistics except for the extreme minimum, which is slightly larger than RANG.}
\label{tab:poisson}
\end{table}
\subsection{1D convection–diffusion equation Equation}
The last linear PDE in the section is a 1D convection-diffusion equations containing an advection term and a diffusion term. The governing PDE is shown as follows:
%我们讨论的第三类线性方程是一维对流扩散方程，该方程融合了对流和扩散两项，考虑的具体形式如下：
\begin{align*}
\frac{\partial u}{\partial t}+c\cdot\frac{\partial u}{\partial x}&=\mu\cdot\frac{\partial^2 u}{\partial x^2},\\
u(0,x)&=u_0(x),\\
(t,x)&\in[0,T]\times(-\infty, \infty).
\end{align*}
Take $T=1.0$, $c=4.0$ and $\mu=0.05$, the initial value is a Gaussian pulse function
\begin{align*}
u_0(x)=\frac{0.1}{\sqrt{0.1\mu}}\cdot\exp\left(-\frac{(x+2)^2}{4\cdot 0.1\cdot \mu}\right).
\end{align*}
And the exact solution is
\begin{align*}
u_0(x)=\frac{0.1}{\sqrt{\mu(t + 0.1)}} \cdot \exp\left(-\frac{(x+2-4t)^2}{4 \mu (t + 0.1))}\right).
\end{align*}
We truncate the spatial domain to $[-l, l]$ such that the solution is almost $0$ at the two ends. The boundary value condition is defined as
\begin{align*}
u(t,-l)=u(t,l)=0, t\in[0,T].
\end{align*}
Then we solve the equation in domain $[0,T]\times[-l,l]$.
%类似前述实验的做法，我们得到加权的总loss：
Following the steps of previous subsections, we get the weighted total loss:
\begin{align*}
L=2L_0+2L_b+0.2L_{pde}.
\end{align*}
%where $L_0$ is the initial value loss term, $L_b$ is the boundary value loss term and $L_{pde}$ is the loss term for the governing PDE.
%其中$L_0$是初值损失，$L_b$是边值损失，$L_{pde}$是控制方程的损失。
The neuron number for each layer is $[2,64,64,64,64,1]$.
The maximum iteration number is set to 10,000, and the resample interval $I$ is set to 1,000. The number of replicates is $60$. The designated capacity of collocation point set is $N_{pde}=1000$.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{essemble_heat.png}
\caption{Solutions for the convection–diffusion equation. The top left image exhibits the reference solution. The remaining nine images show the prediction results of nine sampling methods in one experiment. The results might be different in another run due to the randomness of sampling and training.}
\label{fig:essemble_heat}
\end{figure}
%迭代的MSE的中值结果如图\ref{fig:advection_it}所示。几种非重采样方法随着迭代进行，误差不再下降。几类重采样方法有更高的精度，其中RANG-m与RANG方法效果明显优于其他几类重采样方法。
As shown in Figure \ref{fig:advection_it}, the error curves of the four non-resampling methods decline slowly. Three non-adaptive resampling methods are faster. RANG and RANG-m are still the fastest.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{advection_iteration.png}
\caption{MSE values in iteration for the convection–diffusion equation. This figure shows the curve of the median of MSE corresponding to nine different methods.}
\label{fig:advection_it}
\end{figure}
After training ends, the MSE obtained by various sampling methods is shown in Figure \ref{fig:advection_error} and Table \ref{tab:adv}. All statistics imply that RANG-m achieves the best performance.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{advection_errorplot.png}
\caption{Box plot of MSEs for the convection–diffusion equation considering different node generation strategies. The plot represents median and 25, 75\% interquartile and extreme values. The mean values are marked with triangles.}
\label{fig:advection_error}
\end{figure}
\begin{table}
\begin{tabular}{l|rrrrrr}
\hline
{} & mean & min & 25\% & 50\% & 75\% & max \\
\hline
Hammersley & 1.916E-04 & 3.582E-06 & 5.633E-05 & 1.109E-04 & 2.392E-04 & 1.368E-03 \\
Random & 2.842E-04 & 1.839E-06 & 1.965E-05 & 6.302E-05 & 1.817E-04 & 4.317E-03 \\
LHS & 4.286E-04 & 1.654E-06 & 2.785E-05 & 1.085E-04 & 2.728E-04 & 6.958E-03 \\
FF & 7.311E-05 & 8.566E-07 & 1.408E-05 & 3.535E-05 & 7.370E-05 & 6.284E-04 \\
Random-R & 3.789E-05 & 1.007E-06 & 4.266E-06 & 1.126E-05 & 3.638E-05 & 4.377E-04 \\
LHS-R & 4.412E-05 & 1.773E-06 & 6.903E-06 & 1.677E-05 & 3.779E-05 & 4.216E-04 \\
FF-R & 5.003E-05 & 6.388E-07 & 4.961E-06 & 1.447E-05 & 4.875E-05 & 3.592E-04 \\
RANG & 2.671E-05 & 1.510E-07 & 1.899E-06 & 3.671E-06 & 1.528E-05 & 5.183E-04 \\
RANG-m & \textbf{1.459E-05} & \textbf{1.093E-07} & \textbf{8.959E-07} & \textbf{2.912E-06} & \textbf{1.422E-05} & \textbf{1.453E-04} \\
\hline
\end{tabular}
\caption{%本表展示了对于对流扩散方程，经过重复实验关于MSE的统计结果，对于每种指标，最优值被加粗标记。RANG-m方法在各项指标中都最优，与之接近的是RANG方法。
The statistics of MSE for the convection–diffusion equation considering different node generation strategies over $60$ replicates. The optimal value is marked in bold for each statistic. The RANG-m method performs best in various statistics, close to RANG.}
\label{tab:adv}
\end{table}
\section{Conclusive Remarks}
% 本文提出了一种基于残差的可变密度节点生成方法，同时考虑了局部均匀性和自适应采样能力.并基于记忆给出了变种。本文在多个time-dependent 1D or time-inpendent 2D的 线性和非线性问题上对多种采样方式进行了测试。结果显示了提出的方法在一定样本数量情形下，PINN能够更好地得到收敛结果。
We propose a residual-based adaptive node generation method for training PINN on a two-dimensional domain, enjoying local regularity and adaptivity simultaneously. The memory mechanism is included to enhance the training stability.
A variety of sampling methods on linear and nonlinear PDEs are taken into comparison. The results show that PINN with the proposed method can improve convergence.
% This paper tests a variety of sampling methods on a linear and nonlinear issues of multiple Time-Dependent 1D or Time-Inpend 2D. The results show that the proposed method is in case of a certain amount of sample, PINN can better result in convergence.
We observed that configurations of collocation nodes significantly impact the results, just like meshless methods such as RBF-FD. PINN has similar or different requirements from RBF-FD for evaluating the quality of collocation points.
Several aspects are compared:
%在本文的实验中，我们观察到了采样点不同配置对结果有较大的影响。但是不同于RBF-FD这种无网格方法，PINN对配点生成的质量有相似或者不同的需求，下面从几个方面进行比较：
%1. 最小距离，极小的距离。由于神经网络通常有大量参数，通常为欠拟合状态，两个重合的点并不会影响PINN的稳定性。但是对于RBF-FD方法，
\begin{enumerate}
\item \textbf{Minimum distance}. Neural networks usually have a large number of parameters and are iteratively trained by gradient descent with small learning rate. It does not affect the stability of PINN even in the case of two training points coinciding. However, collocation nodes that are positioned too closely can severely impact the stability of some meshfree methods \cite{liuMeshFreeMethods2002}. Therefore, we can choose FF as the basic framework, even though it does not guarantee the minimum distance.
\item \textbf{Local regularity}. For RBF-FD, nodal distributions should be locally regular throughout the computational domain, i.e., the distances between neighbor nodes should be roughly equal. The requirement derives from the fact that local strong form meshless methods are often sensitive to node positions, and large differences in distances to the nearest neighbors or other irregularities can cause ill-conditioned approximations, making the distribution improper for solving PDEs \cite{slakGenerationNodeDistributions2019a}.
Though the local regularity does not affect the stability of the minimization problem introduced by PINN, it is usually more efficient with the similar numbers of collocation nodes in our experiments. Random and LHS methods do not generate nodes with local regularity, while Hammersley and FF can generate locally regular nodes, which might be the reason why Hammersley and FF usually have better results than the other non-resampling methods in our experiments.
\item \textbf{Adaptive Node Generation}. Similar adaptive node generation methods have been proposed in the RBF-FD studies \cite{slakAdaptiveRadialBasis2019,liAdaptiveRBFFDMethod2017}, which illustrate that PINN may benefit from traditional meshless studies and vice versa.
\end{enumerate}
RBF-FD is a more mature meshless method, and many techniques used in RBF-FD might be transferred and applied to PINN studies. This method provides a preliminary attempt at node generation inspired by traditional meshless methods for PINNs.
Limited to the scope, this paper is aimed at a two-dimensional rectangular domain, which corresponds to a steady two-dimensional PDE on a rectangular space region or a time-dependent one-dimensional PDE. For high-dimensional or more complex regions, the following extensions should be considered:
\begin{enumerate}
%\item 对于二维复杂区域，考虑在足够大的区域中生成节点，并且过滤掉区域外的点，如同本文图L对应的实验，在正方形区域上采样，并且筛选L型内部的点。
\item For a two-dimensional complex domain, consider generating nodes in a large enough rectangular region containing the domain and filtering out the points outside, as is shown in Figure \ref{Lshape}, sampling on the square region, and removing the points outside the $L$-shape. Note that the method might be highly inefficient.
\item Using subsequent methods of FF, the collocation points can be generated based on any given boundary \cite{slakGenerationNodeDistributions2019a}. For high-dimensional PDEs or PDEs on manifolds, the following methods of FF can replace the basis (FF) in RANG for implementation.
\item Combined with the domain decomposition methods in PINN, complex domains are converted into several simple subdomains and then solved \cite{mengPPINNPararealPhysicsinformed2020,shuklaParallelPhysicsinformedNeural2021}.
%由于PINN对节点相对一般无网格方法更加鲁棒，并没有最小距离之类的限制，在子区域的边界处并不额外需要诸如“repel algorithm”的步骤。
The nodes can also be generated for each simple subdomain seperately. Since points that are very close do not undermine the stability of PINNs, and then no minimum distance guarantees are needed. Therefore, nodes generated near the interfaces and boundaries do not need the repel algorithm for post-processing.
%\item 对于高维的处理方法或者流形上的PDE，利用FF的后续算法，替换本文步骤进行实现。
\item Adaptive sampling methods for boundary and initial condition residuals are not considered. The boundaries used in the experiments are all rectangular. Furthermore, in order to reduce the influence of random factors, the nodes for boundary or initial value loss terms are equispaced artificially in the experiments. However, for high-dimensional problems with complex regions, adaptively learning boundary conditions may also benefit from the proposed point generation methods combined with variable density manifold sampling methods \cite{duhFastVariableDensity2021}.
\end{enumerate}
\section*{Acknowledgement}
This work was supported by National Natural Science Foundation of China under Grant No.52005505 and 11725211.
\bibliographystyle{elsarticle-num}
\bibliography{cas-refs}
%% else use the following coding to input the bibitems directly in the
%% TeX file.
% \begin{thebibliography}{00}
% %% \bibitem{label}
% %% Text of bibliographic item
% \bibitem{}
% \end{thebibliography}
\end{document}
\endinput
%%
%% End of file `elsarticle-template-num.tex'.
\begin{thebibliography}{10}
\expandafter\ifx\csname url\endcsname\relax
\def\url#1{\texttt{#1}}\fi
\expandafter\ifx\csname urlprefix\endcsname\relax\def\urlprefix{URL }\fi
\expandafter\ifx\csname href\endcsname\relax
\def\href#1#2{#2} \def\path#1{#1}\fi
\bibitem{raissiPhysicsinformedNeuralNetworks2019}
M.~Raissi, P.~Perdikaris, G.~E. Karniadakis,
\href{http://www.sciencedirect.com/science/article/pii/S0021999118307125}{Physics-informed
neural networks: {A} deep learning framework for solving forward and inverse
problems involving nonlinear partial differential equations}, Journal of
Computational Physics 378 (2019) 686 -- 707.
\newblock \href {https://doi.org/https://doi.org/10.1016/j.jcp.2018.10.045}
{\path{doi:https://doi.org/10.1016/j.jcp.2018.10.045}}.
\newline\urlprefix\url{http://www.sciencedirect.com/science/article/pii/S0021999118307125}
\bibitem{raissiHiddenFluidMechanics2020}
M.~Raissi, A.~Yazdani, G.~E. Karniadakis,
\href{https://www.sciencemag.org/lookup/doi/10.1126/science.aaw4741}{Hidden
fluid mechanics: {Learning} velocity and pressure fields from flow
visualizations}, Science 367~(6481) (2020) 1026--1030.
\newblock \href {https://doi.org/10.1126/science.aaw4741}
{\path{doi:10.1126/science.aaw4741}}.
\newline\urlprefix\url{https://www.sciencemag.org/lookup/doi/10.1126/science.aaw4741}
\bibitem{jinNSFnetsNavierStokesFlow2021}
X.~Jin, S.~Cai, H.~Li, G.~E. Karniadakis,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0021999120307257}{{NSFnets}
({Navier}-{Stokes} flow nets): {Physics}-informed neural networks for the
incompressible {Navier}-{Stokes} equations}, Journal of Computational Physics
426 (2021) 109951.
\newblock \href {https://doi.org/10.1016/j.jcp.2020.109951}
{\path{doi:10.1016/j.jcp.2020.109951}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0021999120307257}
\bibitem{sunSurrogateModelingFluid2020a}
L.~Sun, H.~Gao, S.~Pan, J.-X. Wang,
\href{https://linkinghub.elsevier.com/retrieve/pii/S004578251930622X}{Surrogate
modeling for fluid flows based on physics-constrained deep learning without
simulation data}, Computer Methods in Applied Mechanics and Engineering 361
(2020) 112732.
\newblock \href {https://doi.org/10.1016/j.cma.2019.112732}
{\path{doi:10.1016/j.cma.2019.112732}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S004578251930622X}
\bibitem{haghighatPhysicsinformedDeepLearning2021}
E.~Haghighat, M.~Raissi, A.~Moure, H.~Gomez, R.~Juanes,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0045782521000773}{A
physics-informed deep learning framework for inversion and surrogate modeling
in solid mechanics}, Computer Methods in Applied Mechanics and Engineering
379 (2021) 113741.
\newblock \href {https://doi.org/10.1016/j.cma.2021.113741}
{\path{doi:10.1016/j.cma.2021.113741}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0045782521000773}
\bibitem{caiPhysicsInformedNeuralNetworks2021}
S.~Cai, Z.~Wang, S.~Wang, P.~Perdikaris, G.~E. Karniadakis,
\href{https://asmedigitalcollection.asme.org/heattransfer/article/143/6/060801/1104439/Physics-Informed-Neural-Networks-for-Heat-Transfer}{Physics-{Informed}
{Neural} {Networks} for {Heat} {Transfer} {Problems}}, Journal of Heat
Transfer 143~(6) (2021) 060801.
\newblock \href {https://doi.org/10.1115/1.4050542}
{\path{doi:10.1115/1.4050542}}.
\newline\urlprefix\url{https://asmedigitalcollection.asme.org/heattransfer/article/143/6/060801/1104439/Physics-Informed-Neural-Networks-for-Heat-Transfer}
\bibitem{hePhysicsinformedDeepLearning2021}
Z.~He, F.~Ni, W.~Wang, J.~Zhang,
\href{https://linkinghub.elsevier.com/retrieve/pii/S235249282100711X}{A
physics-informed deep learning method for solving direct and inverse heat
conduction problems of materials}, Materials Today Communications 28 (2021)
102719.
\newblock \href {https://doi.org/10.1016/j.mtcomm.2021.102719}
{\path{doi:10.1016/j.mtcomm.2021.102719}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S235249282100711X}
\bibitem{luPhysicsInformedNeuralNetworks2021}
L.~Lu, R.~Pestourie, W.~Yao, Z.~Wang, F.~Verdugo, S.~G. Johnson,
\href{https://epubs.siam.org/doi/10.1137/21M1397908}{Physics-{Informed}
{Neural} {Networks} with {Hard} {Constraints} for {Inverse} {Design}}, SIAM
Journal on Scientific Computing 43~(6) (2021) B1105--B1132.
\newblock \href {https://doi.org/10.1137/21M1397908}
{\path{doi:10.1137/21M1397908}}.
\newline\urlprefix\url{https://epubs.siam.org/doi/10.1137/21M1397908}
\bibitem{schiassiPhysicsInformedNeuralNetworks2022}
E.~Schiassi, A.~D’Ambrosio, K.~Drozd, F.~Curti, R.~Furfaro,
\href{https://arc.aiaa.org/doi/10.2514/1.A35138}{Physics-{Informed} {Neural}
{Networks} for {Optimal} {Planar} {Orbit} {Transfers}}, Journal of Spacecraft
and Rockets (2022) 1--16\href {https://doi.org/10.2514/1.A35138}
{\path{doi:10.2514/1.A35138}}.
\newline\urlprefix\url{https://arc.aiaa.org/doi/10.2514/1.A35138}
\bibitem{chenMGNetNovelDifferential2022a}
X.~Chen, T.~Li, Q.~Wan, X.~He, C.~Gong, Y.~Pang, J.~Liu,
\href{https://link.springer.com/10.1007/s00366-022-01632-7}{{MGNet}: a novel
differential mesh generation method based on unsupervised neural networks},
Engineering with Computers (Mar. 2022).
\newblock \href {https://doi.org/10.1007/s00366-022-01632-7}
{\path{doi:10.1007/s00366-022-01632-7}}.
\newline\urlprefix\url{https://link.springer.com/10.1007/s00366-022-01632-7}
\bibitem{wangUnderstandingMitigatingGradient2021}
S.~Wang, Y.~Teng, P.~Perdikaris,
\href{https://epubs.siam.org/doi/10.1137/20M1318043}{Understanding and
{Mitigating} {Gradient} {Flow} {Pathologies} in {Physics}-{Informed} {Neural}
{Networks}}, SIAM Journal on Scientific Computing 43~(5) (2021) A3055--A3081.
\newblock \href {https://doi.org/10.1137/20M1318043}
{\path{doi:10.1137/20M1318043}}.
\newline\urlprefix\url{https://epubs.siam.org/doi/10.1137/20M1318043}
\bibitem{mcclennySelfAdaptivePhysicsInformedNeural2020}
L.~McClenny, U.~Braga-Neto,
\href{http://arxiv.org/abs/2009.04544}{Self-{Adaptive} {Physics}-{Informed}
{Neural} {Networks} using a {Soft} {Attention} {Mechanism}}, arXiv:2009.04544
[cs, stat]6 citations (Semantic Scholar/arXiv) [2021-06-20] arXiv: 2009.04544
(Sep. 2020).
\newline\urlprefix\url{http://arxiv.org/abs/2009.04544}
\bibitem{xiang2021selfadaptive}
Z.~Xiang, W.~Peng, X.~Zheng, X.~Zhao, W.~Yao, Self-adaptive loss balanced
{Physics}-informed neural networks for the incompressible {Navier}-{Stokes}
equations, arXiv: 2104.06217 [physics.flu-dyn] (2021).
\bibitem{liuDualDimerMethodTraining2021}
D.~Liu, Y.~Wang,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0893608020304536}{A
{Dual}-{Dimer} method for training physics-constrained neural networks with
minimax architecture}, Neural Networks 136 (2021) 112--125.
\newblock \href {https://doi.org/10.1016/j.neunet.2020.12.028}
{\path{doi:10.1016/j.neunet.2020.12.028}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0893608020304536}
\bibitem{ramabathiranSPINNSparsePhysicsbased2021a}
A.~A. Ramabathiran, P.~Ramachandran,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0021999121004952}{{SPINN}:
{Sparse}, {Physics}-based, and partially {Interpretable} {Neural} {Networks}
for {PDEs}}, Journal of Computational Physics 445 (2021) 110600.
\newblock \href {https://doi.org/10.1016/j.jcp.2021.110600}
{\path{doi:10.1016/j.jcp.2021.110600}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0021999121004952}
\bibitem{wangRespectingCausalityAll2022}
S.~Wang, S.~Sankaran, P.~Perdikaris,
\href{https://arxiv.org/abs/2203.07404}{Respecting causality is all you need
for training physics-informed neural networks}, ArXiv (2022).
\newblock \href {https://doi.org/10.48550/ARXIV.2203.07404}
{\path{doi:10.48550/ARXIV.2203.07404}}.
\newline\urlprefix\url{https://arxiv.org/abs/2203.07404}
\bibitem{yuGradientenhancedPhysicsinformedNeural2022}
J.~Yu, L.~Lu, X.~Meng, G.~E. Karniadakis,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0045782522001438}{Gradient-enhanced
physics-informed neural networks for forward and inverse {PDE} problems},
Computer Methods in Applied Mechanics and Engineering 393 (2022) 114823.
\newblock \href {https://doi.org/10.1016/j.cma.2022.114823}
{\path{doi:10.1016/j.cma.2022.114823}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0045782522001438}
\bibitem{cuiEfficientNeuralNetwork2022}
T.~Cui, Z.~Wang, X.~Xiang,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0898122122000542}{An
efficient neural network method with plane wave activation functions for
solving {Helmholtz} equation}, Computers \& Mathematics with Applications 111
(2022) 34--49.
\newblock \href {https://doi.org/10.1016/j.camwa.2022.02.004}
{\path{doi:10.1016/j.camwa.2022.02.004}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0898122122000542}
\bibitem{mengPPINNPararealPhysicsinformed2020}
X.~Meng, Z.~Li, D.~Zhang, G.~E. Karniadakis,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0045782520304357}{{PPINN}:
{Parareal} physics-informed neural network for time-dependent {PDEs}},
Computer Methods in Applied Mechanics and Engineering 370 (2020) 113250, 41
citations (Semantic Scholar/DOI) [2021-06-24].
\newblock \href {https://doi.org/10.1016/j.cma.2020.113250}
{\path{doi:10.1016/j.cma.2020.113250}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0045782520304357}
\bibitem{dasStateoftheArtReviewDesign2022}
S.~Das, S.~Tesfamariam,
\href{http://arxiv.org/abs/2202.06416}{State-of-the-{Art} {Review} of
{Design} of {Experiments} for {Physics}-{Informed} {Deep} {Learning}},
arXiv:2202.06416 [cs, stat]ArXiv: 2202.06416 (Feb. 2022).
\newline\urlprefix\url{http://arxiv.org/abs/2202.06416}
\bibitem{nabianEfficientTrainingPhysics2021a}
M.~A. Nabian, R.~J. Gladstone, H.~Meidani,
\href{https://onlinelibrary.wiley.com/doi/10.1111/mice.12685}{Efficient
training of physics‐informed neural networks via importance sampling},
Computer-Aided Civil and Infrastructure Engineering 36~(8) (2021) 962--977.
\newblock \href {https://doi.org/10.1111/mice.12685}
{\path{doi:10.1111/mice.12685}}.
\newline\urlprefix\url{https://onlinelibrary.wiley.com/doi/10.1111/mice.12685}
\bibitem{luDeepXDEDeepLearning2021}
L.~Lu, X.~Meng, Z.~Mao, G.~E. Karniadakis,
\href{https://epubs.siam.org/doi/10.1137/19M1274067}{{DeepXDE}: {A} {Deep}
{Learning} {Library} for {Solving} {Differential} {Equations}}, SIAM Review
63~(1) (2021) 208--228, 147 citations (Semantic Scholar/DOI) [2021-06-20].
\newblock \href {https://doi.org/10.1137/19M1274067}
{\path{doi:10.1137/19M1274067}}.
\newline\urlprefix\url{https://epubs.siam.org/doi/10.1137/19M1274067}
\bibitem{zhaoSolvingAllenCahnCahnHilliard2021}
C.~L. W. .~J. Zhao,
\href{http://global-sci.org/intro/article_detail/cicp/18571.html}{Solving
{Allen}-{Cahn} and {Cahn}-{Hilliard} {Equations} using the {Adaptive}
{Physics} {Informed} {Neural} {Networks}}, Communications in Computational
Physics 29~(3) (2021) 930--954.
\newblock \href {https://doi.org/10.4208/cicp.OA-2020-0086}
{\path{doi:10.4208/cicp.OA-2020-0086}}.
\newline\urlprefix\url{http://global-sci.org/intro/article_detail/cicp/18571.html}
\bibitem{kansaMultiquadricsScatteredData1990}
E.~Kansa,
\href{https://linkinghub.elsevier.com/retrieve/pii/089812219090270T}{Multiquadrics—{A}
scattered data approximation scheme with applications to computational
fluid-dynamics—{I} surface approximations and partial derivative
estimates}, Computers \& Mathematics with Applications 19~(8-9) (1990)
127--145.
\newblock \href {https://doi.org/10.1016/0898-1221(90)90270-T}
{\path{doi:10.1016/0898-1221(90)90270-T}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/089812219090270T}
\bibitem{fornbergSolvingPDEsRadial2015}
B.~Fornberg, N.~Flyer,
\href{https://www.cambridge.org/core/product/identifier/S0962492914000130/type/journal_article}{Solving
{PDEs} with radial basis functions}, Acta Numerica 24 (2015) 215--258.
\newblock \href {https://doi.org/10.1017/S0962492914000130}
{\path{doi:10.1017/S0962492914000130}}.
\newline\urlprefix\url{https://www.cambridge.org/core/product/identifier/S0962492914000130/type/journal_article}
\bibitem{zamoloTwoAlgorithmsFast2018}
R.~Zamolo, E.~Nobile,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0898122118301676}{Two
algorithms for fast {2D} node generation: {Application} to {RBF} meshless
discretization of diffusion problems and image halftoning}, Computers \&
Mathematics with Applications 75~(12) (2018) 4305--4321.
\newblock \href {https://doi.org/10.1016/j.camwa.2018.03.031}
{\path{doi:10.1016/j.camwa.2018.03.031}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0898122118301676}
\bibitem{vlasiukFastHighdimensionalNode2018}
O.~Vlasiuk, T.~Michaels, N.~Flyer, B.~Fornberg,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0898122118303912}{Fast
high-dimensional node generation with variable density}, Computers \&
Mathematics with Applications 76~(7) (2018) 1739--1757.
\newblock \href {https://doi.org/10.1016/j.camwa.2018.07.026}
{\path{doi:10.1016/j.camwa.2018.07.026}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0898122118303912}
\bibitem{bridsonFastPoissonDisk2007}
R.~Bridson, \href{http://dl.acm.org/citation.cfm?doid=1278780.1278807}{Fast
{Poisson} disk sampling in arbitrary dimensions}, in: {ACM} {SIGGRAPH} 2007
sketches on - {SIGGRAPH} '07, ACM Press, San Diego, California, 2007, pp.
22--es.
\newblock \href {https://doi.org/10.1145/1278780.1278807}
{\path{doi:10.1145/1278780.1278807}}.
\newline\urlprefix\url{http://dl.acm.org/citation.cfm?doid=1278780.1278807}
\bibitem{fornbergFastGeneration2D2015}
B.~Fornberg, N.~Flyer,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0898122115000334}{Fast
generation of 2-{D} node distributions for mesh-free {PDE} discretizations},
Computers \& Mathematics with Applications 69~(7) (2015) 531--544.
\newblock \href {https://doi.org/10.1016/j.camwa.2015.01.009}
{\path{doi:10.1016/j.camwa.2015.01.009}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0898122115000334}
\bibitem{vandersandeFastVariableDensity2021}
K.~van~der Sande, B.~Fornberg,
\href{https://epubs.siam.org/doi/10.1137/20M1337016}{Fast {Variable}
{Density} 3-{D} {Node} {Generation}}, SIAM Journal on Scientific Computing
43~(1) (2021) A242--A257.
\newblock \href {https://doi.org/10.1137/20M1337016}
{\path{doi:10.1137/20M1337016}}.
\newline\urlprefix\url{https://epubs.siam.org/doi/10.1137/20M1337016}
\bibitem{duhFastVariableDensity2021}
U.~Duh, G.~Kosec, J.~Slak,
\href{https://epubs.siam.org/doi/10.1137/20M1325642}{Fast {Variable}
{Density} {Node} {Generation} on {Parametric} {Surfaces} with {Application}
to {Mesh}-{Free} {Methods}}, SIAM Journal on Scientific Computing 43~(2)
(2021) A980--A1000.
\newblock \href {https://doi.org/10.1137/20M1325642}
{\path{doi:10.1137/20M1325642}}.
\newline\urlprefix\url{https://epubs.siam.org/doi/10.1137/20M1325642}
\bibitem{slakGenerationNodeDistributions2019a}
J.~Slak, G.~Kosec, \href{https://epubs.siam.org/doi/10.1137/18M1231456}{On
{Generation} of {Node} {Distributions} for {Meshless} {PDE}
{Discretizations}}, SIAM Journal on Scientific Computing 41~(5) (2019)
A3202--A3229.
\newblock \href {https://doi.org/10.1137/18M1231456}
{\path{doi:10.1137/18M1231456}}.
\newline\urlprefix\url{https://epubs.siam.org/doi/10.1137/18M1231456}
\bibitem{liBitingAdvancingFront2000}
X.-Y. Li, S.-H. Teng, A.~{\"U}ng{\"o}r,
\href{https://onlinelibrary.wiley.com/doi/10.1002/1097-0207(20000910/20)49:1/2<61::AID-NME923>3.0.CO;2-Y}{Biting:
advancing front meets sphere packing}, International Journal for Numerical
Methods in Engineering 49~(1-2) (2000) 61--81.
\newblock \href
{https://doi.org/10.1002/1097-0207(20000910/20)49:1/2<61::AID-NME923>3.0.CO;2-Y}
{\path{doi:10.1002/1097-0207(20000910/20)49:1/2<61::AID-NME923>3.0.CO;2-Y}}.
\newline\urlprefix\url{https://onlinelibrary.wiley.com/doi/10.1002/1097-0207(20000910/20)49:1/2<61::AID-NME923>3.0.CO;2-Y}
\bibitem{lhnerProgressGridGeneration1996}
R.~L{\"o}hner, \href{http://link.springer.com/10.1007/BF01198734}{Progress in
grid generation via the advancing front technique}, Engineering with
Computers 12~(3-4) (1996) 186--210.
\newblock \href {https://doi.org/10.1007/BF01198734}
{\path{doi:10.1007/BF01198734}}.
\newline\urlprefix\url{http://link.springer.com/10.1007/BF01198734}
\bibitem{liAdaptiveRBFFDMethod2017}
J.~Li, S.~Zhai, Z.~Weng, X.~Feng,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0735193317301215}{H-adaptive
{RBF}-{FD} method for the high-dimensional convection-diffusion equation},
International Communications in Heat and Mass Transfer 89 (2017) 139--146.
\newblock \href {https://doi.org/10.1016/j.icheatmasstransfer.2017.06.001}
{\path{doi:10.1016/j.icheatmasstransfer.2017.06.001}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0735193317301215}
\bibitem{slakAdaptiveRadialBasis2019}
J.~Slak, G.~Kosec,
\href{https://onlinelibrary.wiley.com/doi/10.1002/nme.6067}{Adaptive radial
basis function–generated finite differences method for contact problems},
International Journal for Numerical Methods in Engineering 119~(7) (2019)
661--686.
\newblock \href {https://doi.org/10.1002/nme.6067}
{\path{doi:10.1002/nme.6067}}.
\newline\urlprefix\url{https://onlinelibrary.wiley.com/doi/10.1002/nme.6067}
\bibitem{zengAdaptiveDeepNeural2022}
S.~Zeng, Z.~Zhang, Q.~Zou,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0021999122002947}{Adaptive
deep neural networks methods for high-dimensional partial differential
equations}, Journal of Computational Physics (2022) 111232\href
{https://doi.org/10.1016/j.jcp.2022.111232}
{\path{doi:10.1016/j.jcp.2022.111232}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0021999122002947}
\bibitem{tang2021}
K.~Tang, X.~Wan, C.~Yang, \href{http://arxiv.org/abs/2112.14038}{{DAS}: {A}
deep adaptive sampling method for solving partial differential equations},
arXiv:2112.14038 [cs, math, stat]ArXiv: 2112.14038 (Dec. 2021).
\newline\urlprefix\url{http://arxiv.org/abs/2112.14038}
\bibitem{hannaResidualbasedAdaptivityTwophase2022}
J.~M. Hanna, J.~V. Aguado, S.~Comas-Cardona, R.~Askri, D.~Borzacchiello,
\href{http://arxiv.org/abs/2109.14290}{Residual-based adaptivity for
two-phase flow simulation in porous media using {Physics}-informed {Neural}
{Networks}}, arXiv:2109.14290 [physics]ArXiv: 2109.14290 (Feb. 2022).
\newline\urlprefix\url{http://arxiv.org/abs/2109.14290}
\bibitem{Peng2021IDRLnetAP}
W.~Peng, J.~Zhang, W.~Zhou, X.~Zhao, W.~Yao, X.~Chen, Idrlnet: A
physics-informed neural network library, ArXiv abs/2107.04320 (2021).
\bibitem{hennighNVIDIASimNetAIAccelerated2021}
O.~Hennigh, S.~Narasimhan, M.~A. Nabian, A.~Subramaniam, K.~Tangsali, Z.~Fang,
M.~Rietmann, W.~Byeon, S.~Choudhry, {NVIDIA} {SimNet}™: {An}
{AI}-{Accelerated} {Multi}-{Physics} {Simulation} {Framework}, in:
M.~Paszynski, D.~Kranzlmüller, V.~V. Krzhizhanovskaya, J.~J. Dongarra, P.~M.
Sloot (Eds.), Computational {Science} – {ICCS} 2021, Springer International
Publishing, Cham, 2021, pp. 447--461.
\bibitem{liuMeshFreeMethods2002}
G.~Liu, \href{https://www.taylorfrancis.com/books/9781420040586}{Mesh {Free}
{Methods}: {Moving} {Beyond} the {Finite} {Element} {Method}}, 0th Edition,
CRC Press, 2002.
\newblock \href {https://doi.org/10.1201/9781420040586}
{\path{doi:10.1201/9781420040586}}.
\newline\urlprefix\url{https://www.taylorfrancis.com/books/9781420040586}
\bibitem{shuklaParallelPhysicsinformedNeural2021}
K.~Shukla, A.~D. Jagtap, G.~E. Karniadakis,
\href{https://linkinghub.elsevier.com/retrieve/pii/S0021999121005787}{Parallel
physics-informed neural networks via domain decomposition}, Journal of
Computational Physics 447 (2021) 110683.
\newblock \href {https://doi.org/10.1016/j.jcp.2021.110683}
{\path{doi:10.1016/j.jcp.2021.110683}}.
\newline\urlprefix\url{https://linkinghub.elsevier.com/retrieve/pii/S0021999121005787}
\end{thebibliography}