Machine learning accelerated particle-in-cell plasma simulations
Extracted source text ยท arXiv 2110.12444
\documentclass[preprint]{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2021
% ready for submission
\usepackage[nonatbib]{neurips_2021}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2021}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2021}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2021}
\usepackage[square, numbers]{natbib}
\bibliographystyle{unsrtnat}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\Eqnref}[1]{Eq.~\ref{eq:#1}}
\newcommand{\Eqsref}[1]{Eqs.~\ref{eq:#1}}
\newcommand{\Figref}[1]{Fig.~\ref{fig:#1}}
\newcommand{\Tabref}[1]{Tab.~\ref{tab:#1}}
\newcommand{\Secref}[1]{Sec.~\ref{sec:#1}}
\newcommand{\secref}[1]{sec.~\ref{sec:#1}}
\newcommand{\naug}{\ensuremath{n_\mathrm{aug}}\,}
\newcommand{\nee}{\ensuremath{n_\mathrm{e}}\,}
\title{Machine learning accelerated particle-in-cell plasma simulations}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
R. Kube %\thanks{Use footnote for providing further information
%about author (webpage, alternative address)---\emph{not} for acknowledging
%funding agencies.}
\\
Princeton Plasma Physics Laboratory\\
Princeton, NJ 08540 USA \\
\texttt{rkube@pppl.gov} \\
% examples of more authors
\And
R.M. Churchill \\
Princeton Plasma Physics Laboratory\\
Princeton, NJ 08540 USA \\
\texttt{rchurchi@pppl.gov} \\
\AND
B. Sturdevant \\
Princeton Plasma Physics Laboratory \\
Princeton, NJ 08540 USA \\
\texttt{bsturdev@pppl.gov} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
Particle-In-Cell (PIC) methods are frequently used for kinetic, high-fidelity simulations of plasmas.
Implicit formulations of PIC algorithms feature strong conservation properties, up to numerical round-off errors,
and are not subject to time-step limitations which make them an attractive candidate to use in simulations
fusion plasmas. Currently they remain prohibitively expensive for high-fidelity simulation of macroscopic plasmas.
We investigate how amortized solvers can be incorporated with PIC methods for simulations of plasmas. Incorporated into the
amortized solver, a neural network predicts a vector space that entails an approximate solution of the PIC system.
The network uses only fluid moments and the electric field as input and its output is used to augment the vector space
of an iterative linear solver. We find that this approach reduces the average number of
required solver iterations by about $25\%$ when simulating electron plasma oscillations. This novel approach may
allow to accelerate implicit PIC simulations while retaining all conservation laws and may also be appropriate for multi-scale systems.
\end{abstract}
\section{Background}
The dynamics of plasmas are governed by the Vlasov-Maxwell equations, which describe the self-consistent time-evolution of the plasmas
distribution function in phase space and the electromagnetic field. While approximations can be made to derive reduced
fluid models from this equation set, many situations require to resolve kinetic effects in simulations. Particle-in-cell (PIC) algorithms
power predictive high-fidelity kinetic simulations of fusion plasmas, implemented for example in the XGC code \cite{ku-2016}. Such
simulations resolve complicated multi-scale interactions on a kinetic level in order to uncover the physics that drive turbulent transport
in nuclear fusion experiments \cite{chang-2017, ku-2018, hager-2020}.
Simulations of macroscopic systems which employ kinetic formulations of the physics model quickly become prohibitively expensive,
even for leadership class, pre-exascale compute facilities. Researchers are therefore constantly implementing advanced parallelization
and optimization methods, often tailored to specific target hardware architectures, in order to speed up such codes
\cite{wang-2019, chien-2020, bird-2021, ohana-2021, Mniszewski-2021}.
In this contribution we are reporting on the development of a machine learning aided amortized solver that accelerates PIC simulations.
In particular, we focus on the method described by \citet{chen-2011, chen-2014}, which has been adapted for the XGC code to study
electromagnetic phenomena in fusion plasmas \cite{sturdevant-2021}.
%PIC algorithms treat the plasma kinetically and evolve position and velocity of a set of marker particles
%in time. The evolution of the electromagnetic field is described by Maxwell's equations.
PIC algorithms couple the evolution of plasma distribution function and electromagnetic fields in a self-consistent manner.
An ensemble of marker particles represents the plasmas distribution function. Their coordinates typically assume continous
values, i.e. a Lagrangian description is used. The electromagnetic field on the other hand is commonly discretized in a spatial grid.
Evolving this system in time happens is a four-step process.
First, the particles are pushed by integrating their equations of motion. Second, the electric current and mass densities
are evaluated on the grid, using the updated particle coordinates. Third, the electromagnetic field is updated using
the new source terms. And finally, electromagnetic forces on the individual particles are calculated from the updated field.
Implicit formulations of this algorithm can be formulated as to conserve energy and charge up to numerical round-off errors
\cite{chen-2011, chen-2014}.
%electromagnetic forces acting on the particles are evaluated and the particles are pushed accordingly.
%Second, current and mass densities are calculated from the updated particle distribution updated.
%Third, the mass and electric current densities are integrated in order to evaluate the electric and magnetic field
%on the grid. And finally, electromagnetic forces on the individual particles are calculated from the updated fields.
%Explicit implementation of this time integration loops are straightforward, but are subject to the Courant-Friedrichs-Lewy
%(CFL) condition, which requires to resolve the fastest frequencies in the system. Implicit formulations of this system
%are not subject to the CFL condition and can be formulated so that they conserve energy and charge up to numerical round-off errors
%\cite{chen-2011, chen-2014}. These conservation properties make Chen's formulation particularly attractive for downstream
%applications, such as high-fidelity modeling of fusion plasmas \cite{sturdevant-2021}.
Time-stepping of implicit PIC algorithms requires to solve a system of non-linear equations. Jacobian-free Newton Krylov methods,
such as GMRES, are known to effectively solve such systems \cite{chen-2011}. However, to evaluate the effect of the systems Jacobian
acting on a vector requires to evaluate the systems time-stepping loop. This is the most expensive part of implicit PIC algorithms
and it is desirable to minimize the number of GMRES iterations required to integrate the system in time.
In this contribution we explore how machine learning can be used to minimize the number of GMRES iterations required to
advance the PIC system in time. In particular, Neural Networks are trained to suggest vectors which augment the initial Krylov space of GMRES iterations
\cite{morgan-1995, chapman-1997, gaul-2014}. Interfacing a predictive model with a robust iterative solver has several
advantages when aiming to accelerate PIC simulations of plasmas.
First, the simulation still obeys all conservation laws inherent to the used discretization. Meeting this condition is crucial to ensure
physically correct simulations. Second, by predicting a set of augmentation vectors, the model may predict only values of order unity.
PIC algorithms are used to solve multi-scale problems where no single characteristic scale can be defined for the physical quantities in the system.
If a model were to predict physical quantities it would eventually be required to make predictions over multiple orders of magnitude,
which is a hard problem.
Other contemporary approaches that aim to accelerate numerical simulations mostly target systems that are described by a set of partial
differential equations on an eulerian grid, such as the Navier-Stokes fluid equations. One thread of research aims to develop surrogate models,
for example physics-informed neural networks \cite{raissi-2019} while other approaches use machine learned sub-grid models to accelerate
simulations \cite{pathak-2020, kochkov-2021}. Properly trained, such models allow to run coarse-grid simulations that include the same
level of detail as fine-grid models and result in speed-ups up to 100x. However, the hybrid Lagrangian-Eulerian structure of PIC methods
is not readily amenable to such approaches.
%To the best of our knowledge, this contribution describes the first approach to accelerate
%the solution of implicit PIC algorithms using machine learning.
\section{Interfacing an implicit PIC solver and predictive machine learning models}
Considering a collisionless, electrostatic plasma, an implicit discretization of the Vlasov-Ampere system can be
formulated as
%
\begin{subequations}
\begin{align}
\frac{x_\mathrm{p}^{n+1} - x_\mathrm{p}^{n}} {\triangle t} & = v_\mathrm{p}^{n+1/2} \label{eq:PIC_timestep_x}\\
\frac{v_\mathrm{p}^{n+1} - v_\mathrm{p}^{n}}{\triangle t} & = \frac{q_\mathrm{p}}{m_\mathrm{p}} \mathrm{SM}\left[ E^{n+1/2} \right] \left( x_\mathrm{p}^{n+1/2} \right) \label{eq:PIC_timestep_v}\\
\epsilon_0 \frac{E^{n+1}_{i} - E^{n}_{i}}{\triangle t} + \frac{q_\mathrm{p}}{m_\mathrm{p}} \mathrm{SM}\left[ j_{i}^{n+1/2} \right] & = \langle j \rangle^{n+1/2} \label{eq:PIC_timestep_E}.
\end{align}
\label{eq:PIC_timestep}
\end{subequations}
%
Here $x_\mathrm{p}$ and $v_\mathrm{p}$ denote position and velocity of particle $\mathrm{p}$,
$\triangle t$ denotes the time step length and
$q_\mathrm{p}$ and $m_\mathrm{p}$ denote the particles charge and mass respectively.
Superscript indices denote the time step and half-step quantities are given by an arithmetic mean, $x^{n+1/2} = \left(x^{n+1} + x^{n} \right) / 2$.
The electric field $E$ is discretized on a periodic domain $z_i = i\, \triangle z \in L_z$, where $i=0\ldots N_z-1$ and $\triangle z = L_z / N_z$. The vaccuum permittivity is denoted as $\epsilon_0$ and
$\mathrm{SM}[\cdot]$ denotes a binomial smoothing operator. The electric current is denoted as $j$ and $\langle \cdot \rangle$ denotes
spatial averaging.
Given particle coordinates and the electric field at time step $n$, $x^{n}$, $v^{n}$, $E^{n}$, \Eqsref{PIC_timestep} guides the time
evolution of the system. Only for the physically correct $x^{n+1}$, $v^{n+1}$ and $E^{n+1}$ do these equations hold.
Re-casting \Eqsref{PIC_timestep} as a set of non-linear equations $G(E(\{x_\mathrm{p}\},\{v_\mathrm{p}\})) = 0$ allows to solve
this system for the correct values at $t= (n+1) \triangle t$ using Newton's method. That approach requires to solve the linear system of equations
%
\begin{align}
\left. \frac{\partial G}{\partial E} \right|^{k} \delta E^{k} & = -G(E^{k}) \label{eq:Newton_Axb}
\end{align}
%
at each Newton iteration $k$. Here we denote the iterative solution as $\delta E^{k} = E^{k+1} - E^{k}$.
Probing Jacobian-vector products $\partial G / \partial E |^{k} v$ by evaluating Gateaux derivative, \Eqnref{Newton_Axb} is solved
using GMRES, an iterative Krylov subspace method.
%Such methods are implemented using only Matrix-vector products and in particular don't require explicit knowledge of the coefficient matrix.
Each evaluation of $\partial G / \partial E |^{k} v$ evaluates the entire four-step PIC loop. It is thus desirable to reduce the number
of iterations to converge the residual below the level of tolerance.
Neglecting preconditioning, the number of GMRES iterations required to solve a system $Ax=b$ within a prescribed residual tolerance may be reduced
by starting with a good initial guess. We therefore train a machine learning model to suggest vectors $\left\{ v_1, \ldots, v_{\naug} \right\}$
such that an initial guess $A^{-1}b$ lies closely in $\mathcal{V}$. In particular, the solution vector $x$ is decomposed as
%
\begin{align}
x = \sum_{j=1}^{\naug} \alpha_j v_j + \widetilde{x},
\end{align}
%
where $x_v = \sum_j \alpha_j v_j \in \mathcal{V}$ and $\widetilde{x} \perp \mathcal{V}$ and
$\mathcal{V} = \mathrm{span} \left( v_1, \ldots, v_{\naug} \right)$. The coefficients $\alpha_j$ are
recovered by inserting the decomposition $x = x_v + \widetilde{x}$ into $Ax=b$, forming the scalar products between
this equation and the vectors $\left\{ Av_j \right\}_{j=1}^{\naug}$, and then solving the resulting system for $\alpha_j$.
Forming $\widetilde{b} = b - \sum_{j=1}^{\naug} \alpha_j A v_j$ we then proceed to solve
%
\begin{align}
A \widetilde{x} = \widetilde{b} \label{eq:modified_system}
\end{align}
%
and reconstruct $x = \widetilde{x} + \sum_{j=1}^{\naug} \alpha_j A v_j$. This projection into
a space perpendicular to $\mathcal{V}$, the solution of \Eqnref{modified_system}, and the subsequent back-projection taken together
constitute the amortized solver.
% Accounting for the total
% number of operations, this amortized solver is computationally more expensive than simply solving \Eqnref{Newton_Axb}.
% However, here we focus only on the performance of a predictive model to suggest suitable vectors $\left\{ v_j \right\}_{j=1}^{\naug}$
% which
% performance of GMRES to solve \Eqsref{eq:Newton_Axb} and \ref{eq:modified_system} are compared.
% But here we only
% compare the GMRES iterations to required solve \Eqnref{Newton_Axb} and \Eqnref{modified_system}. A well-performing model predicts
% so that the modified system \Eqnref{modified_system} converges in fewer
% iterations than the original system. Here we also note that the elements of the augmentation vectors $v_j$, as provided
% by any predictive model, may be of order unity. Any transformation of the vectors $v_j$ into quantities such as $\delta E^{k}$ is done by
% robust linear algebra routines. This approach makes future applications in multi scale problems feasible, where physical
% quantities may not have a single characteristic scale.
%In the following we explore the performance of the amortized solver, characterized by the number of GMRES iterations required to
%solve \Eqnref{modified_system}.
\begin{figure}
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth]{figure01.png}
\caption{Convolutional neural network architecture used in the amortized solver that maps simulation profiles onto a set of
$\naug$ augmentation vectors $v_1, \ldots\, v_{\naug}$.}
\label{fig:nnv6_architecture}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth]{figure02.png}
\caption{Loss over the test set, grouped by the number of augmentation vectors $\naug$ returned by the model.}
\label{fig:test_loss}
\end{subfigure}
\caption{Network architecture (left) and test losses (right)}
\end{figure}
A convolutional neural network, shown in \Figref{nnv6_architecture}, maps the electron density profile $\nee(z)$,
the Electric field $E(z)$ profile and a one-hot encoding of simulation parameters onto $\left\{ v_j \right\}_{j=1}^{\naug}$. The profiles are concatenated and feed into
two convolutional layers with $4$ and $16$ channels respectively. The one-hot encoded simulation parameters feed into
two fully-connected MLP layers with $32$ neurons each. The output of these channels are flattened and merge into two
fully-connected layers which are fed into the output layer of size $N_z \times \naug$.
We train this network on data from a series of simulations where the initial positions of $32,768$ electrons is sampled
from profiles of the form $\nee(z, t=0) = A \cdot f(k z)$ with
$A \in \{1, 2, 5, 10\} \times 10^{-3}$, $f \in \left\{ \sin, \cos \right\}$, $k \in \{1, 2\}$. The initial positions of $32,768$ singly
charged ions is sampled from a uniform distributions and both electrons and ions are initially cold. These initial conditions
on the particles are chosen as to
excite electron plasma oscillations. The fields are evaluated on a grid with $N_z=64$ and $L_z = 4\pi$. These 16 reference
simulations are advanced with a timestep of $\triangle t=1$ from $T=0$ to $T=30$. This yields $480$ training profiles.
Targeting only the first Newton iteration, the neural network is trained by minimizing the orthogonal projection of
$\left\{ \delta E^{k=1} \right\}$ over the vector space spanned by the model output, as described by \citet{trivedi-2019}
%
\begin{align}
\mathcal{L} = \argmin_{\widehat{y} \in \mathrm{span}(v_1\, \ldots, v_{\naug})} \frac{\| \widehat{y} - \delta E^{k=1} \|_2 }{\| \delta E^{k=1} \|_2} + \lambda \, \| \mathrm{triu} \left( V*V^{\mathrm{tr}} \right) \|_2. \label{eq:lossfun}
\end{align}
%
To calculate the projection, the output of the model $V$ is subject to a $\mathrm{QR}$ factorization, $Q,R = \mathrm{qr}\left(V\right)$
\cite{seeger-2017} and the projection is calculated as $\widehat{y} = Q \cdot Q' \cdot \delta E^{k=1}$.
Additionally, the last term penalizes collinearity of output vectors. Here $\mathrm{triu}(A) = \left\{ a_{r,s} \right\}_{r<s}$
denotes the upper triangular part of a square matrix and $V = \left[ v_1 | \ldots | v_{\naug}\right]$.
% Splitting the $480$ profiles in the reference simulation set into a training and test set, consisting of $384$ and
% $96$ elements respectively, we optimize the networks' weights using the ADAM optimizer \cite{kingma-2017-adam} with
% an initial learning rate of $0.001$, and use mini-batch training with a batch-size of $32$. All layers except the final layer use ReLU
% activation functions and both convolutional layers use a filter width of $3$. During training the fully-connected
% layers additionally use Dropout \cite{srivastava-2014}, where we chose the probability of a node dropping out from
% $\{0.1, 0.2, 0.3\}$. Training losses for this hyperparameter scan, grouped by $\lambda$ are plotted in \Figref{test_loss}.
To identify optimal hyperparameters of the network, the $480$ profiles are split $80\%/20\%$ into a training set and a test set.
The network is trained for 200 epochs to minimize \Eqnref{lossfun} over the training set. We use
ReLU, tanh, and swish \cite{ramachandran-2017} activation functions,
vary the size of the convolution filters used in both convolutional layers, choosing from $\{ 3, 5, 7, 9, 13\}$,
vary dropout probabilities in fully connected layers \cite{srivastava-2014} over $\{0.1, 0.2, 0.3\}$,
vary $\lambda \in \{10^{-3}, 10^{-4}, 10^{-5}\}$
vary the initial learning rate from $\{ 10^{-2}, 10^{-3}, 5 \times 10^{-4}\}$,
and use ADAM \cite{kingma-2017-adam}, RMSProp and SGD optimizers.
Average loss over the test set serves as the metric to evaluate optimal hyperparameters.
Figure \ref{fig:test_loss} shows a sub-set of the training losses for various $\naug$, $\lambda$ and dropout probabilities.
We observe that more augmentation vectors results in lower per-sample loss. Additionally we observe only
little over-fitting for $\naug = 8, 16$. From this scan we identify
ReLU activation functions,
convolutional filters of width $3$,
a Dropout probability of $0.2$,
$\lambda=10^{-4}$,
as ideal parameters
and optimize using the ADAM algorithm with an initial learning rate of $0.001$.
Training was performed on nVidia V100 GPUs, in less than 2 hours of total computation time, using the Flux library \cite{Flux.jl-2018, innes-2018}.
Simulation data for training and inference was produced using this code \footnote{\url{https://github.com/rkube/picfun}}. Instructions
on how to generate the data are available from the author upon request.
\begin{figure}[h!tb]
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{figure03.png}
\caption{$n_{\mathrm{e},t=0}(z)=10^{-2} \left[ 0.1 \cos(2z) + 0.9 \sin(z)\right]$.}
\label{fig:avgres_case1}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{figure04.png}
\caption{$n_{\mathrm{e},t=0}(z)=3\times10^{-3} \left[ 0.5 \sin(z) + 0.5 \cos(2z)\right]$.}
\label{fig:avgres_case2}
\end{subfigure}
\caption{Residuals at various GMRES iterations of the original and amortized solver using $\naug \in \left\{ 8, 16, 32, 48\right\}$,
averaged over an entire simulation.}
\end{figure}
The performance of the trained amortized solver is evaluated on simulations where the initial
positions of the electrons are sampled from
$n_{\mathrm{e}, t=0}(z) = A\left[ a_1 f_1(k_1 z) + a_2 f_2 (k_2 z)\right]$. Ions are again sampled from
a uniform distribution and both particles are cold.
Initial profile parameters are chosen as
$f_1, f_2 \in \left\{ \sin, \cos \right\}$, $k_1, k_2 \in \left\{1,2\right\}$. Figures
\ref{fig:avgres_case1} and \ref{fig:avgres_case2} show the residual at each GMRES iteration in
the these simulation, averaged over all 30 timesteps. The thick blue lines denote residuals
observed in simulations using a plain GMRES solver and the thin, colored lines show residuals
observed when using the trained amortized for $\naug = 8, 16, 32, 48$. Using $\naug=8$
the average residuals of the amortized solver are only slightly smaller than those of
plain GMRES. For $\naug=16$, the average residuals of the amortized solver are of
the same magnitude as those of plain GMRES but after its following iteration ahead. That is,
the amortized solver requires one less iteration to converge to the same tolerance as plain GMRES.
For $\naug=32$ and $\naug=48$, the amortized solver requires two fewer iterations to converge the
residual to the same tolerance as the plain GMRES.
In conclusion, we demonstrate the feasibility to accelerate plasma simulations using
implicit PIC methods with a machine-learning based amortized solver. For simulations of
electron plasma oscillations the number of required GMRES iterations per first Newton iteration
may be reduced from about 7 to about 5. The fidelity of the accelerated simulations is
unmodified and they retain all conservation properties.
Future work will focus on re-formulating the amortized solver to make use of more direct
Krylov space augmentation techniques as discussed for example in \cite{morgan-1995, chapman-1997, trivedi-2019}.
We hope to achieve a net speed-up of the simulations using such techniques. Finally, we aim to investigate
how the performance of the amortized solver depends on the training set
and explore performance for other types of simulations such as ion acoustic waves.
\section{Acknowledgements}
The simulations presented in this article were performed on computational resources managed and supported by Princeton Research Computing,
a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and the Office of Information
Technology's High Performance Computing Center and Visualization Laboratory at Princeton University.
\bibliography{myrefs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section*{Checklist}
% %%% BEGIN INSTRUCTIONS %%%
% The checklist follows the references. Please
% read the checklist guidelines carefully for information on how to answer these
% questions. For each question, change the default \answerTODO{} to \answerYes{},
% \answerNo{}, or \answerNA{}. You are strongly encouraged to include a {\bf
% justification to your answer}, either by referencing the appropriate section of
% your paper or providing a brief inline description. For example:
% \begin{itemize}
% \item Did you include the license to the code and datasets? \answerYes{See ...}
% \item Did you include the license to the code and datasets? \answerNo{The code and the data are proprietary.}
% \item Did you include the license to the code and datasets? \answerNA{}
% \end{itemize}
% Please do not modify the questions and only use the provided macros for your
% answers. Note that the Checklist section does not count towards the page
% limit. In your paper, please delete this instructions block and only keep the
% Checklist section heading above along with the questions/answers below.
% %%% END INSTRUCTIONS %%%
%
\end{document}
\begin{thebibliography}{25}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ku et~al.(2016)Ku, Hager, Chang, Kwon, and Parker]{ku-2016}
S.~Ku, R.~Hager, C.S. Chang, J.M. Kwon, and S.E. Parker.
\newblock A new hybrid-lagrangian numerical scheme for gyrokinetic simulation
of tokamak edge plasma.
\newblock \emph{Journal of Computational Physics}, 315:\penalty0 467--475,
2016.
\newblock ISSN 0021-9991.
\newblock \doi{https://doi.org/10.1016/j.jcp.2016.03.062}.
\newblock URL
\url{https://www.sciencedirect.com/science/article/pii/S0021999116300274}.
\bibitem[Chang et~al.(2017)Chang, Ku, Tynan, Hager, Churchill, Cziegler,
Greenwald, Hubbard, and Hughes]{chang-2017}
C.~S. Chang, S.~Ku, G.~R. Tynan, R.~Hager, R.~M. Churchill, I.~Cziegler,
M.~Greenwald, A.~E. Hubbard, and J.~W. Hughes.
\newblock Fast low-to-high confinement mode bifurcation dynamics in a tokamak
edge plasma gyrokinetic simulation.
\newblock \emph{Phys. Rev. Lett.}, 118:\penalty0 175001, Apr 2017.
\newblock \doi{10.1103/PhysRevLett.118.175001}.
\newblock URL \url{https://link.aps.org/doi/10.1103/PhysRevLett.118.175001}.
\bibitem[Ku et~al.(2018)Ku, Chang, Hager, Churchill, Tynan, Cziegler,
Greenwald, Hughes, Parker, Adams, D'Azevedo, and Worley]{ku-2018}
S.~Ku, C.~S. Chang, R.~Hager, R.~M. Churchill, G.~R. Tynan, I.~Cziegler,
M.~Greenwald, J.~Hughes, S.~E. Parker, M.~F. Adams, E.~D'Azevedo, and
P.~Worley.
\newblock A fast low-to-high confinement mode bifurcation dynamics in the
boundary-plasma gyrokinetic code xgc1.
\newblock \emph{Physics of Plasmas}, 25\penalty0 (5):\penalty0 056107, 2018.
\newblock \doi{10.1063/1.5020792}.
\newblock URL \url{https://doi.org/10.1063/1.5020792}.
\bibitem[Hager et~al.(2020)Hager, Chang, Ferraro, and Nazikian]{hager-2020}
R.~Hager, C.~S. Chang, N.~M. Ferraro, and R.~Nazikian.
\newblock Gyrokinetic understanding of the edge pedestal transport driven by
resonant magnetic perturbations in a realistic divertor geometry.
\newblock \emph{Physics of Plasmas}, 27\penalty0 (6):\penalty0 062301, 2020.
\newblock \doi{10.1063/1.5144445}.
\newblock URL \url{https://doi.org/10.1063/1.5144445}.
\bibitem[Wang et~al.(2019)Wang, Ethier, Tang, Ibrahim, Madduri, Williams, and
Oliker]{wang-2019}
Bei Wang, Stephane Ethier, William Tang, Khaled~Z Ibrahim, Kamesh Madduri,
Samuel Williams, and Leonid Oliker.
\newblock Modern gyrokinetic particle-in-cell simulation of fusion plasmas on
top supercomputers.
\newblock \emph{The International Journal of High Performance Computing
Applications}, 33\penalty0 (1):\penalty0 169--188, 2019.
\newblock \doi{10.1177/1094342017712059}.
\newblock URL \url{https://doi.org/10.1177/1094342017712059}.
\bibitem[Chien et~al.(2020)Chien, Nylund, Bengtsson, Peng, Podobas, and
Markidis]{chien-2020}
Steven W.~D. Chien, Jonas Nylund, Gabriel Bengtsson, Ivy~B. Peng, Artur
Podobas, and Stefano Markidis.
\newblock sputnipic: An implicit particle-in-cell code for multi-gpu systems.
\newblock In \emph{2020 IEEE 32nd International Symposium on Computer
Architecture and High Performance Computing (SBAC-PAD)}, pages 149--156,
2020.
\newblock \doi{10.1109/SBAC-PAD49847.2020.00030}.
\bibitem[Bird et~al.(2021)Bird, Tan, Luedtke, Harrell, Taufer, and
Albright]{bird-2021}
Robert~F. Bird, Nigel Tan, Scott~V. Luedtke, Stephen~Lien Harrell, Michela
Taufer, and Brian~J. Albright.
\newblock {VPIC} 2.0: Next generation particle-in-cell simulations.
\newblock \emph{CoRR}, abs/2102.13133, 2021.
\newblock URL \url{https://arxiv.org/abs/2102.13133}.
\bibitem[Ohana et~al.(2021)Ohana, Gheller, Lanti, Jocksch, Brunner, and
Villard]{ohana-2021}
No{\'e} Ohana, Claudio Gheller, Emmanuel Lanti, Andreas Jocksch, Stephan
Brunner, and Laurent Villard.
\newblock Gyrokinetic simulations on many- and multi-core architectures with
the global electromagnetic particle-in-cell code orb5.
\newblock \emph{Computer Physics Communications}, 262:\penalty0 107208, 2021.
\newblock ISSN 0010-4655.
\newblock \doi{https://doi.org/10.1016/j.cpc.2020.107208}.
\newblock URL
\url{https://www.sciencedirect.com/science/article/pii/S0010465520300461}.
\bibitem[Mniszewski et~al.(2021)Mniszewski, Belak, Fattebert, Negre, Slattery,
Adedoyin, Bird, Chang, Chen, Ethier, Fogerty, Habib, Junghans,
Lebrun-Grandi{\'e}, Mohd-Yusof, Moore, Osei-Kuffuor, Plimpton, Pope, Reeve,
Ricketson, Scheinberg, Sharma, and Wall]{Mniszewski-2021}
Susan~M Mniszewski, James Belak, Jean-Luc Fattebert, Christian~FA Negre,
Stuart~R Slattery, Adetokunbo~A Adedoyin, Robert~F Bird, Choongseok Chang,
Guangye Chen, St{\'e}phane Ethier, Shane Fogerty, Salman Habib, Christoph
Junghans, Damien Lebrun-Grandi{\'e}, Jamaludin Mohd-Yusof, Stan~G Moore,
Daniel Osei-Kuffuor, Steven~J Plimpton, Adrian Pope, Samuel~Temple Reeve, Lee
Ricketson, Aaron Scheinberg, Amil~Y Sharma, and Michael~E Wall.
\newblock Enabling particle applications for exascale computing platforms.
\newblock \emph{The International Journal of High Performance Computing
Applications}, 0\penalty0 (0):\penalty0 10943420211022829, 2021.
\newblock \doi{10.1177/10943420211022829}.
\newblock URL \url{https://doi.org/10.1177/10943420211022829}.
\bibitem[Chen et~al.(2011)Chen, Chac{\'o}n, and Barnes]{chen-2011}
G.~Chen, L.~Chac{\'o}n, and D.C. Barnes.
\newblock An energy- and charge-conserving, implicit, electrostatic
particle-in-cell algorithm.
\newblock \emph{Journal of Computational Physics}, 230\penalty0 (18):\penalty0
7018--7036, 2011.
\newblock ISSN 0021-9991.
\newblock \doi{https://doi.org/10.1016/j.jcp.2011.05.031}.
\newblock URL
\url{https://www.sciencedirect.com/science/article/pii/S0021999111003421}.
\bibitem[Chen et~al.(2014)Chen, Chac{\'o}n, Leibs, Knoll, and
Taitano]{chen-2014}
G.~Chen, L.~Chac{\'o}n, C.A. Leibs, D.A. Knoll, and W.~Taitano.
\newblock Fluid preconditioning for newton--krylov-based, fully implicit,
electrostatic particle-in-cell simulations.
\newblock \emph{Journal of Computational Physics}, 258:\penalty0 555--567,
2014.
\newblock ISSN 0021-9991.
\newblock \doi{https://doi.org/10.1016/j.jcp.2013.10.052}.
\newblock URL
\url{https://www.sciencedirect.com/science/article/pii/S0021999113007286}.
\bibitem[Sturdevant et~al.(2021)Sturdevant, Ku, Chac{\'o}n, Chen, Hatch, Cole,
Sharma, Adams, Chang, Parker, and Hager]{sturdevant-2021}
Benjamin~J. Sturdevant, S.~Ku, L.~Chac{\'o}n, Y.~Chen, D.~Hatch, M.~D.~J. Cole,
A.~Y. Sharma, M.~F. Adams, C.~S. Chang, S.~E. Parker, and R.~Hager.
\newblock Verification of a fully implicit particle-in-cell method for the
v||-formalism of electromagnetic gyrokinetics in the xgc code.
\newblock \emph{Physics of Plasmas}, 28\penalty0 (7):\penalty0 072505, 2021.
\newblock \doi{10.1063/5.0047842}.
\newblock URL \url{https://doi.org/10.1063/5.0047842}.
\bibitem[Morgan(1995)]{morgan-1995}
Ronald~B. Morgan.
\newblock A restarted gmres method augmented with eigenvectors.
\newblock \emph{SIAM Journal on Matrix Analysis and Applications}, 16\penalty0
(4):\penalty0 1154--1171, 1995.
\newblock \doi{10.1137/S0895479893253975}.
\newblock URL \url{https://doi.org/10.1137/S0895479893253975}.
\bibitem[Chapman and Saad(1997)]{chapman-1997}
Andrew Chapman and Yousef Saad.
\newblock Deflated and augmented krylov subspace techniques.
\newblock In \emph{Numerical Linear Algebra with Applications}, volume~4, pages
43--66. John Wiley \& Sons, Ltd, 2021/09/20 1997.
\newblock ISBN 1070-5325.
\newblock
\doi{https://doi.org/10.1002/(SICI)1099-1506(199701/02)4:1<43::AID-NLA99>3.0.CO;2-Z}.
\newblock URL
\url{https://doi.org/10.1002/(SICI)1099-1506(199701/02)4:1<43::AID-NLA99>3.0.CO;2-Z}.
\bibitem[Gaul(2014)]{gaul-2014}
Andr{\'e} Gaul.
\newblock \emph{Recycling Krylov subspace methods for sequences of linear
systems : Analysis and applications}.
\newblock Doctoral thesis, Technische Universit{\"a}t Berlin, Fakult{\"a}t II -
Mathematik und Naturwissenschaften, Berlin, 2014.
\newblock URL \url{http://dx.doi.org/10.14279/depositonce-4147}.
\bibitem[Raissi et~al.(2019)Raissi, Perdikaris, and Karniadakis]{raissi-2019}
M.~Raissi, P.~Perdikaris, and G.E. Karniadakis.
\newblock Physics-informed neural networks: A deep learning framework for
solving forward and inverse problems involving nonlinear partial differential
equations.
\newblock \emph{Journal of Computational Physics}, 378:\penalty0 686--707,
2019.
\newblock ISSN 0021-9991.
\newblock \doi{https://doi.org/10.1016/j.jcp.2018.10.045}.
\newblock URL
\url{https://www.sciencedirect.com/science/article/pii/S0021999118307125}.
\bibitem[Pathak et~al.(2020)Pathak, Mustafa, Kashinath, Motheau, Kurth, and
Day]{pathak-2020}
Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten
Kurth, and Marcus Day.
\newblock Using machine learning to augment coarse-grid computational fluid
dynamics simulations, 2020.
\bibitem[Kochkov et~al.(2021)Kochkov, Smith, Alieva, Wang, Brenner, and
Hoyer]{kochkov-2021}
Dmitrii Kochkov, Jamie~A. Smith, Ayya Alieva, Qing Wang, Michael~P. Brenner,
and Stephan Hoyer.
\newblock Machine learning{\textendash}accelerated computational fluid
dynamics.
\newblock \emph{Proceedings of the National Academy of Sciences}, 118\penalty0
(21), 2021.
\newblock ISSN 0027-8424.
\newblock \doi{10.1073/pnas.2101784118}.
\newblock URL \url{https://www.pnas.org/content/118/21/e2101784118}.
\bibitem[Trivedi et~al.(2019)Trivedi, Su, Lu, Schubert, and
Vuckovic]{trivedi-2019}
Rahul Trivedi, Logan Su, Jesse Lu, Martin~F. Schubert, and Jelena Vuckovic.
\newblock Data-driven acceleration of photonic simulations.
\newblock \emph{Scientific Reports}, 9\penalty0 (1):\penalty0 19728, 2019.
\newblock \doi{10.1038/s41598-019-56212-5}.
\newblock URL \url{https://doi.org/10.1038/s41598-019-56212-5}.
\bibitem[Seeger et~al.(2017)Seeger, Hetzel, Dai, and Lawrence]{seeger-2017}
Matthias~W. Seeger, Asmus Hetzel, Zhenwen Dai, and Neil~D. Lawrence.
\newblock Auto-differentiating linear algebra.
\newblock \emph{CoRR}, abs/1710.08717, 2017.
\newblock URL \url{http://arxiv.org/abs/1710.08717}.
\bibitem[Ramachandran et~al.(2017)Ramachandran, Zoph, and
Le]{ramachandran-2017}
Prajit Ramachandran, Barret Zoph, and Quoc~V. Le.
\newblock Searching for activation functions.
\newblock \emph{CoRR}, abs/1710.05941, 2017.
\newblock URL \url{http://arxiv.org/abs/1710.05941}.
\bibitem[Srivastava et~al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever, and
Salakhutdinov]{srivastava-2014}
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Salakhutdinov.
\newblock Dropout: A simple way to prevent neural networks from overfitting.
\newblock \emph{Journal of Machine Learning Research}, 15\penalty0
(56):\penalty0 1929--1958, 2014.
\newblock URL \url{http://jmlr.org/papers/v15/srivastava14a.html}.
\bibitem[Kingma and Ba(2017)]{kingma-2017-adam}
Diederik~P. Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization, 2017.
\bibitem[Innes et~al.(2018)Innes, Saba, Fischer, Gandhi, Rudilosso, Joy,
Karmali, Pal, and Shah]{Flux.jl-2018}
Michael Innes, Elliot Saba, Keno Fischer, Dhairya Gandhi, Marco~Concetto
Rudilosso, Neethu~Mariya Joy, Tejan Karmali, Avik Pal, and Viral Shah.
\newblock Fashionable modelling with flux.
\newblock \emph{CoRR}, abs/1811.01457, 2018.
\newblock URL \url{https://arxiv.org/abs/1811.01457}.
\bibitem[Innes(2018)]{innes-2018}
Mike Innes.
\newblock Flux: Elegant machine learning with julia.
\newblock \emph{Journal of Open Source Software}, 2018.
\newblock \doi{10.21105/joss.00602}.
\end{thebibliography}