Penalized Ensemble Kalman Filters for High Dimensional Non-linear Systems
Extracted source text · arXiv 1610.00195
%% Version 4.3.2, 25 August 2014
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Template.tex -- LaTeX-based template for submissions to the
% American Meteorological Society
%
% Template developed by Amy Hendrickson, 2013, TeXnology Inc.,
% amyh@texnology.com, http://www.texnology.com
% following earlier work by Brian Papa, American Meteorological Society
%
% Email questions to latex@ametsoc.org.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% PREAMBLE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Start with one of the following:
% DOUBLE-SPACED VERSION FOR SUBMISSION TO THE AMS
\documentclass{ametsoc}
% TWO-COLUMN JOURNAL PAGE LAYOUT---FOR AUTHOR USE ONLY
%\documentclass[twocol]{ametsoc}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% To be entered only if twocol option is used
\journal{mwr}
% Please choose a journal abbreviation to use above from the following list:
%
% jamc (Journal of Applied Meteorology and Climatology)
% jtech (Journal of Atmospheric and Oceanic Technology)
% jhm (Journal of Hydrometeorology)
% jpo (Journal of Physical Oceanography)
% jas (Journal of Atmospheric Sciences)
% jcli (Journal of Climate)
% mwr (Monthly Weather Review)
% wcas (Weather, Climate, and Society)
% waf (Weather and Forecasting)
% bams (Bulletin of the American Meteorological Society)
% ei (Earth Interactions)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Citations should be of the form ``author year'' not ``author, year''
\bibpunct{(}{)}{;}{a}{}{,}
\usepackage{amsthm}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}{Lemma}[theorem]
\newtheorem*{definition*}{Definition}
\newtheorem*{assumptions*}{Assumptions}
\newcommand{\mat}[1]{\pmb{\mathsf{#1}}}
\renewcommand{\v}[1]{\mathbf{#1}}
\newcommand{\E}{\text{E}}
\newcommand{\Cov}{\text{Cov}}
\newcommand{\tr}{\text{tr}}
\newcommand{\B}{\text{B}}
\usepackage{algorithm, algorithmicx, algpseudocode}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}
\newcommand{\algorithmiclcomment}{}
\newcommand{\LineComment}{\item[\algorithmiclcomment]}
\usepackage{enumitem, subcaption, xcolor}
\newcommand{\arxiv}{}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% For arxiv
\usepackage{setspace}
\singlespacing
\nolinenumbers
\def\abstract#1{\def\theabstract{%
\centerline{\vtop{\ifdraft\normalsize\else \bigfootnotesize\fi\hsize 5.125in
\ifdraft\fi
\noindent\hskip8pt\relax #1\vskip1sp}}}}
\def\correspondingauthor#1{%
\thanks{\ifdraft\normalsize\fi
\noindent{\it Corresponding author address:}\ #1
\vskip1pt
\noindent\theemail
}}
\renewcommand{\arxiv}{\nolinenumbers}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% To be entered by author:
%% May use \\ to break lines in title:
\title{Penalized Ensemble Kalman Filters for High Dimensional Non-linear Systems}
%%% Enter authors' names, as you see in this example:
%%% Use \correspondingauthor{} and \thanks{Current Affiliation:...}
%%% immediately following the appropriate author.
%%%
%%% Note that the \correspondingauthor{} command is NECESSARY.
%%% The \thanks{} commands are OPTIONAL.
%\authors{Author One\correspondingauthor{Author One,
% American Meteorological Society,
% 45 Beacon St., Boston, MA 02108.}
% and Author Two\thanks{Current affiliation: American Meteorological Society,
% 45 Beacon St., Boston, MA 02108.}}
\authors{Elizabeth Hou\correspondingauthor{University of Michigan, Ann Arbor, MI.} And Alfred O. Hero}
%% Follow this form:
% \affiliation{American Meteorological Society,
% Boston, Massachusetts.}
\affiliation{\arxiv University of Michigan, Ann Arbor, Michigan}
%% Follow this form:
%\email{latex@ametsoc.org}
\email{emhou@umich.edu}
% If appropriate, add additional authors, different affiliations:
\extraauthor{\arxiv Earl Lawrence}
\extraaffil{\arxiv Los Alamos National Laboratory, Los Alamos, New Mexico}
%\extraauthor{}
%\extraaffil{}
%% May repeat for a additional authors/affiliations:
%\extraauthor{}
%\extraaffil{}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% ABSTRACT
%
% Enter your abstract here
% Abstracts should not exceed 250 words in length!
%
% For BAMS authors only: If your article requires a Capsule Summary, please place the capsule text at the end of your abstract
% and identify it as the capsule. Example: This is the end of the abstract. (Capsule Summary) This is the capsule summary.
\abstract{The ensemble Kalman filter (EnKF) is a data assimilation technique that uses an ensemble of models, updated with data, to track the time evolution of a usually non-linear system. It does so by using an empirical approximation to the well-known Kalman filter. However, its performance can suffer when the ensemble size is smaller than the state space, as is often necessary for computationally burdensome models. This scenario means that the empirical estimate of the state covariance is not full rank and possibly quite noisy. To solve this problem in this high dimensional regime, we propose a computationally fast and easy to implement algorithm called the penalized ensemble Kalman filter (PEnKF). Under certain conditions, it can be theoretically proven that the PEnKF will have good performance (the error will converge to zero) despite having fewer ensemble members than state dimensions. Further, the proposed approach makes fewer assumptions about the structure of the system than localization methods. These theoretical results are supported with simulations of several non-linear and high dimensional systems.}
\begin{document}
%% Necessary!
\maketitle
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% MAIN BODY OF PAPER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%% In all cases, if there is only one entry of this type within
%% the higher level heading, use the star form:
%%
% \section{Section title}
% \subsection*{subsection}
% text...
% \section{Section title}
%vs
% \section{Section title}
% \subsection{subsection one}
% text...
% \subsection{subsection two}
% \section{Section title}
%%%
% \section{First primary heading}
% \subsection{First secondary heading}
% \subsubsection{First tertiary heading}
% \paragraph{First quaternary heading}
\section{Introduction}
The Kalman filter is a well-known technique to track a linear system over time, and many variants based on the extended and ensemble Kalman filters have been proposed to deal with non-linear systems. The ensemble Kalman filter (EnKF) \cite{evensen1994sequential, burgers1998analysis} is particularly popular when the non-linear system is extremely complicated and its gradient is infeasible to calculate, which is often the case in geophysical systems. However, these systems are often high dimensional and forecasting each ensemble member forward through the system is computationally expensive. Thus, the filtering often operates in the high dimensional regime where the number of ensemble members, $n$, is much less than the size of the state, $p$. It is well known that even when $p / n \rightarrow const. $ and the samples are from a Gaussian distribution, the eigenvalues and the eigenvectors of the sample covariance matrix do not converge to their population equivalents, \cite{johnstone2001, johnstone2004}. Since our ensemble is both non-Gaussian and high dimensional ($n << p$), the sample covariance matrix of the forecast ensemble will be extremely noisy. In this paper, we propose a variant of the EnKF specifically designed to handle covariance estimation in this difficult regime, but with weaker assumptions and less prior information than competing approaches.
\subsection*{Other Work}
To deal with the sampling errors, many schemes have been developed to de-noise the forecast sample covariance matrix. These schemes ``tune" the matrix with variance inflation and localization, \cite{ hamill2001distance, Houtekamer2001sequential, ott2004local, Houtekamer2005, wang2007, anderson2007, anderson2009, li2009, bishop2009a, bishop2009b, Houtekamer2009, campbell2010, Greybush2011, Miyoshi2011}. However, these schemes are often not trivial to implement because they require carefully choosing the inflation factor and using expert knowledge of the true system to set up the localization. Additionally, the EnKF with perturbed observations introduces additional sampling errors due to the perturbation noise's lack of orthogonality with the ensemble. Methods have been devised that construct perturbation matrices that are orthogonal, \cite{evensen2003ensemble}; however these methods are computationally expensive, \cite{evensen2004sampling}. This has lead to the development of matrix factorization versions of the EnKF such as the square root and transform filters, \cite{bishop2001, Whitaker2002, Tippett2003, evensen2004sampling, hunt2007efficient, Godinez2012, Nerger2012, Todter2015}, which do not perturb the observations and are designed to avoid these additional sampling errors.
The ensemble Kalman filter is closely related to the particle filter \cite{Papadakis}, although it uses a Gaussian approximation of the conditional state distribution in order to get an update that is a closed form expression for the analysis ensemble (as opposed to one that requires numerical integration). While the particle filter does not use this approximation, it also requires an exponential number of particles to avoid filter collapse, \cite{snyder}. Recently, there has been significant effort to apply the particle filter to larger scale systems using equal weights, \cite{vanLeeuwen, Ades}, and merging it with the ensemble Kalman filter to form hybrid filters, \cite{Papadakis, lei2011, frei2013bridging, Nakano, Robert}. EnKF is also related to the unscented Kalman filter, \cite{julier1997new, wan2000unscented}, which handles nonlinearity by propagating a carefully selected set of ``sigma points'' (as opposed to the randomly sampled points of the EnKF) through the nonlinear forecast equations. The results are then used to reconstruct the forecasted mean and covariance.
Most similar to our proposed work are \cite{ueno2009covariance} and \cite{Nino-Ruiz:2015:PEK:2832080.2832084}, which also propose methods that use sparse inverse covariance matrices. Both methods justify the appropriateness of using the inverse space with large scale simulations or real weather data. The former reports that their computational complexity is polynomial in the state dimension and requires the stronger assumptions of Gaussianity and structural knowledge. The latter algorithm can be implement in parallel making it very efficient, however, the paper still makes the \textit{much} stronger assumptions of Gaussianity and conditional independence.
\subsection*{Proposed Method}
We propose a penalized ensemble Kalman filter (PEnKF), which uses an estimator of the forecast covariance whose inverse is sparsity regularized. While the localization approaches effectively dampen or zero out entires in the covariance, {\em our approach zeros out entries in the inverse covariance, resulting in a sparse inverse covariance}. This provides two advantages. First, it makes a weaker assumption about the relationship between state variables. Second, our approach does not require anything like localization's detailed knowledge of which covariances to fix at zero or how much to dampen. Instead, it merely favors sparsity in the inverse covariance. Additionally, our method is very easy to implement because it just requires using a different estimator for the covariance matrix in the EnKF. We can explicitly show the improvement of our estimator through theoretical guarantees.
\subsubsection*{Outline}
In Section 2, we explain the assumptions in our high-dimensional system and we give background on the EnKF and $\ell_1$ penalized inverse covariance matrices. In Section 3, we give details on how to modify the EnKF to our proposed PEnKF and provide theoretical guarantees on the filter. Section 4 contains the simulation results of the classical Lorenz 96 system and a more complicated system based on modified shallow water equations.
\section{Background} \label{background}
In this paper, we consider the scenario of a noisy, non-linear dynamics model $ f(\cdot) $, which evolves a vector of unobserved states $ \v{x}_t \in \mathbb{R}^p $ through time. We observe a noisy vector $ \v{y}_{t} \in \mathbb{R}^r $, which is a transformation of $ \v{x}_t $ by a function $ h(\cdot) $. Both the process noise $ \bm{\omega}_t$ and the observation noise $ \bm{\epsilon}_t $ are independent of the states $ \v{x}_t $. We assume both noises are zero mean Gaussian distributed with known diagonal covariance matrices, $\mat{Q}$ and $\mat{R}$. Often, it is assumed that the dynamics model does not have noise making $ \bm{\omega}_t $ a zero vector, but for generality we allow $ \bm{\omega}_t $ to be a random vector.
\begin{flalign*}
& \v{x}_{t} = f (\v{x}_{t-1}) + \bm{\omega}_t \tag*{Dynamics Model} \\
& \v{y}_{t} = h(\v{x}_{t}) + \bm{\epsilon}_{t} \tag*{Observation Model}
\end{flalign*}
As with localization methods, we make an assumption about the correlation structure of the state vector in order to handle the high dimensionality of the state. In particular, we assume that relatively few pairs of state variables have non-zero conditional correlation, $ \Cov(x_i, x_j | x_{-(i,j)}) \neq 0 $ where $ x_{-(i,j)} $ represents all state variables except $ x_i $ and $x_j $. This means that, conditioning on all of the rest of the state, $x_i$ and $x_j$ are uncorrelated. They may have a dependency, meaning that the correlation between them is non-zero, but that correlation is entirely explained by dependence on other parts of the state. A sample example is given by a one-dimensional spatial field with the three locations $x_1$, $x_2$, and $x_3$ where $x_1$ and $x_3$ are both connected to $x_2$, but not each other. In this case, it might be reasonable to model $x_1$ and $x_3$ as uncorrelated conditional on $x_2$ although not necessarily unconditionally uncorrelated. Their simple correlation might not be zero, but their conditional or partial correlation is zero. This is similar to our assumption, although we do not assume any particular pattern of the conditional dependencies as you might in a spatial field.
We assume that the set of non-zero conditional correlations is sparse. This is equivalent to assuming that the inverse correlation matrix of the model state is sparse.
%Since our state space is high-dimensional, we assume each state only interacts with a few other states. For any pair of state variables $ x_i $ and $x_j $, we define their interaction as having non-zero conditional correlation, $ \Cov(x_i, x_j | x_{-i,j}) \neq 0 $ where $ x_{-i,j} $ represents all state variables except $ x_i $ and $x_j $. So if we denote the set, $\mathcal{E}$, as containing all pairs $i,j$ such that $ \Cov(x_i, x_j | x_{-i,j}) \neq 0 $, its cardinality $ s=|\mathcal{E}|$ is sufficiently small such that $s << p^2$. Consequently, the number of pairs of state variables that are conditionally uncorrelated is large. Note that this is a different assumption than the one made by most localization approaches which assume that there are a small number of {\em unconditionally} correlated pairs, $ \Cov(x_i, x_j) \neq 0 $. Our assumption is much weaker because (xxx what is best way to say?). This conditional uncorrelation condition manifests as zeros in the inverse covariance matrix of $\v{x}$, as opposed to zeros in the covariance matrix.
In other words, the inverse covariance matrix will have $s$ non-zero off-diagonal entries. We can also quantify the sparsity level as $d$, which is the maximum number of non-zero off-diagonals in any row, so $ d^2 << p^2 $. Note that our assumption is on the conditional \textit{correlation} or lack of it, and we do not make any claims on the independence between states. This is because $ \v{x}_t $ is not Gaussian when $ f(\cdot) $ is non-linear so uncorrelation does not imply independence thus the zeros in the inverse covariance matrix do not imply conditional independence.
This assumption is weaker than the one made using localization. That assumption is equivalent to assuming that the covariance matrix itself is sparse whereas our assumption admits a dense covariance.
%The only knowledge required about the numerical model is the assumption that the number of conditionally correlated pairs of variables is small compared to the number of variables squared ($s << p^2$), which is commonplace when $p$ is large.
Finally, because we do not assume that the state variable interactions are the same for different time points, we allow the set $\mathcal{E}_t$ and its size $s_t$ to change over time.
\subsection{Ensemble Kalman Filter}
The standard EnKF algorithm of \cite{evensen2003ensemble} is shown in Algorithm 1. At time $ t = 0 $, $n$ samples are drawn from some distribution, which is often chosen as the standard multivariate normal distribution, if the true initial distribution is unknown, to form an initial ensemble $ \mat{A} \in \mathbb{R}^{p \times n} $. And, at every time point $t$, the observations $\v{y}_t$ are perturbed $n$ times with Gaussian white noise, $ \v{\eta}^{j} \sim N(\v{0}, \mat{R}) $, to form a perturbed observation matrix $\mat{D}_t \in \mathbb{R}^{p \times n}$, where $\v{d}^j_t = \v{y}_t + \v{\eta}^j$.
\begin{algorithm}[H]
\caption{Ensemble Kalman Filter}
\begin{algorithmic}[H]
\Require $ \mat{A}, \mat{H}, \mat{Q}, \mat{R}, $ and $\mat{D}_t$ \\
where $ \mat{H} $ is the the measurement operator%gradient of $h(\cdot)$
\For{ $ t \in \{ 1, ..., T\} $ }
\LineComment $\triangleright$ Evolve each ensemble member forward in time
\State $ \v{a}^{j}_0 = f(\v{a}^{j}) + \v{w}^{j} \quad \forall j \in \{1, ... , n \} $ \\
where $ \v{w}^{j} \sim N(\v{0}, \mat{Q}) $
\LineComment $\triangleright$ Correct the ensemble with the observations
\State $ \mat{A} = \mat{A}_0 + \hat{\mat{K}} ( \mat{D}_t - \mat{H A}_0 ) $ \\
where $ \hat{\mat{K}} = \hat{\mat{P}}^f \mat{H}^T ( \mat{H} \hat{\mat{P}}^f \mat{H}^T + \mat{R} )^{-1} $
\LineComment $\triangleright$ Predict using the analysis ensemble mean
\State $ \hat{\v{x}}_t = \frac{1}{n} \sum_{j=1}^n \v{a}^{j} $
\EndFor
\Ensure $ \hat{\v{x}}_t $
\end{algorithmic}
\end{algorithm}
The forecast covariance estimator $\hat{\mat{P}}^f$ is typically the sample covariance of the forecast ensemble, defined as $ \widehat{\Cov}(\mat{A}_0) = \frac{1}{n-1} (\mat{A}_0 - \bar{\mat{A}}_0) (\mat{A}_0 - \bar{\mat{A}}_0) ^T $, where $ \bar{\mat{A}}_0 $ is the sample mean vector, but it can be another estimator such as a localized estimator (one that is localized with a taper matrix), or a penalized estimator as proposed in this paper.
\subsection{Bregman Divergence and the $\ell_1$ Penalty} \label{ell_1}
Below, we give a brief overview of the $\ell_1$ penalized log-determinant Bregman divergence and some properties of its minimizing estimator, as described in \cite{ravikumar2011high}.
We denote $ \mat{S} $ to be any arbitrary sample covariance matrix, and $ \bm{\Sigma} = \E(\mat{S}) $ to be its true covariance matrix, where $ \E(\cdot) $ is the expectation function.
The Bregman divergence is a very general method to measure the difference between two functions. Here the functions to be compared are covariance matrices. Since we are interested in finding a sparse positive definite estimator for the inverse covariance matrix, a natural choice of Bregman function is $ -\log\det ( \cdot ) $, which has a domain restricted to positive definite matrices. Thus $\bm{\Theta}$, our optimal estimator for the inverse covariance matrix $\bm{\Sigma}^{-1}$, will minimize
\begin{flalign*}
& \underset{\bm{\Theta} \in \mathbb{S}^{p \times p}_{++}}{\arg\min} \, - \log \det (\bm{\Theta}) - \log \det (\bm{\Sigma}) + \tr \left( \bm{\Sigma} (\bm{\Theta} - \bm{\Sigma}^{-1} ) \right)
\end{flalign*}
where $ \mathbb{S}^{p \times p}_{++} $ is the set of all symmetric positive definite $ p \times p $ matrices. This loss function requires the covariance matrix $ \bm{\Sigma} $ to be known, but it can approximated by an empirical loss, which replaces $ \bm{\Sigma} $ with its empirical equivalent $ \mat{S} $ and adds a penalty term to ensure strict convexity.
The empirical Bregman divergence with function $ -\log\det ( \cdot ) $ and an $\ell_1$ penalty term essentially reduces (by dropping the constants) to
\begin{flalign} \label{objective}
& \underset{\bm{\Theta} \in \mathbb{S}^{p \times p}_{++}}{\arg\min} \, - \log \det (\bm{\Theta}) + \tr (\bm{\Theta} \mat{S}) + \lambda ||\Theta||_1
\end{flalign}
where $\lambda \geq 0$ is a penalty parameter, and $|| \cdot ||_1$ denotes an element-wise $\ell_1$ norm. This can be generalized so that each entry of $\bm{\Theta}$ can be penalized differently if $\lambda$ is a matrix and using a element-wise product with the norm.
This objective has a unique solution, $ \bm{\Theta} = (\tilde{\mat{S}})^{-1} $, which satisfies
\begin{flalign*}
\frac{\partial}{\partial \bm{\Theta}} \B^{\lambda}( \bm{\Theta} || \mat{S}^{-1}) = \mat{S} -\bm{\Theta}^{-1} + \lambda \partial || \bm{\Theta}||_1 = 0
\end{flalign*}
where $ \partial || \bm{\Theta}||_1 $ is a subdifferential of the $\ell_1$ norm defined in \eqref{subdiff} in the appendix. The solution $(\tilde{\mat{S}})^{-1}$ is a sparse positive definite estimator of the inverse covariance matrix $\bm{\Sigma}^{-1}$, and we can write its inverse explicitly as $ \tilde{\mat{S}} = \mat{S} + \lambda \tilde{\mat{Z}} $, where $ \tilde{\mat{Z}} $ is the unique subdifferential matrix that makes the gradient zero. See \cite{ravikumar2011high} or \cite{jankova2015confidence} for a more thorough explanation of $ \tilde{\mat{Z}} $.
\cite{ravikumar2011high} show that for well-conditioned covariances and certain minimum sample sizes, the estimator $ (\tilde{\mat{S}})^{-1} $ has many nice properties including having, with high probability, the correct zero and signed non-zero entries and a sum of squared error that converges to 0 as $n, p, s \rightarrow \infty$. These properties will allow our method, described in the next section, to attain superior performance over the EnKF.
\section{$\bm{\ell_1}$ Penalized Ensemble Kalman Filter}
Our penalized ensemble Kalman filter modifies the EnKF, by using a penalized forecast covariance estimator $ \tilde{\mat{P}}^f $. This penalized estimator is derived from its inverse, which is the minimizer of \eqref{objective}. Thus from Section 2.\ref{ell_1}, it can be explicitly written as $\tilde{\mat{P}}^f = \hat{\mat{P}}^f + \lambda \tilde{\mat{Z}} $, implying that we essentially learn a matrix $ \tilde{\mat{Z}} $, and use it to modify our sample covariance $ \hat{\mat{P}}^f $ so that $(\hat{\mat{P}}^f + \lambda \tilde{\mat{Z}} )^{-1}$ is sparse. From this, our modified Kalman gain matrix is
\begin{flalign*}
\tilde{\mat{K}} = ( \hat{\mat{P}}^f + \lambda \tilde{\mat{Z}} ) \mat{H}^T \left( \mat{H} ( \hat{\mat{P}}^f + \lambda \tilde{\mat{Z}} ) \mat{H}^T + \mat{R} \right)^{-1} .
\end{flalign*}
The intuition behind this estimator is that since only a small number of the state variables in the state vector $\v{x}_t$ are conditionally correlated with each other, {\em the forecast inverse covariance matrix $ (\mat{P}^f)^{-1} $ will be sparse with many zeros in the off-diagonal entries.} Furthermore, since minimizing \eqref{objective} gives a sparse estimator for $ (\mat{P}^f)^{-1} $, this sparse estimator will accurately capture the conditional correlations and uncorrelations of the state variables. Thus $ \tilde{\mat{P}}^f $ will be a much better estimator of the true forecast covariance matrix $ \mat{P}^f $ because the $ \ell_1 $ penalty will depress spurious noise in order to make $ (\tilde{\mat{P}}^f)^{-1} $ sparse, while the inverse of the sample forecast covariance $(\hat{\mat{P}}^f )^{-1} $, when it exists, will be non-sparse. As in most penalized estimators, the $ \tilde{\mat{P}}^f $ is a biased estimator of the forecast covariance, while the sample forecast covariance is not. But because the forecast distribution is corrected for in the analysis step, it is acceptable to take this bias as a trade-off for less variance (sampling errors). A more in-depth study of the consequences of this bias in $\ell_1$ penalized inverse covariance matrices and their inverses is described in \cite{jankova2015confidence}. Additionally, this bias due to penalization in the inverse covariance behaves in a similar way as variance inflation because the bias on the diagonal of $ (\tilde{\mat{P}}^f)^{-1} $ is due to it being increased by $\lambda$, so having a biased estimator is not necessarily disadvantageous. And finally, since we do not assume the state variables interact in the same way over all time, we re-learn the matrix $ \tilde{\mat{Z}} $ every time the ensemble is evolved forward.
We can choose the penalty parameter $ \lambda $ in a systematic fashion by calculating a regularization path, solving \eqref{objective} for a list of decreasing $\lambda$s, and evaluating each solution with an information criterion such as an extended or generalized Akaike information criterion (AIC) or Bayesian information criterion (BIC), \cite{foygel2010extended, lv2014model}. Additionally, if we have knowledge or make assumptions about the moments of the ensemble's distribution, we know the optimal proportionality of the penalty parameter (see proof of Theorem \ref{SSE_K}. Thus, we can refine the penalty parameter by calculating a regularization path for the constant of the optimal order. In Section \ref{sec:sims}, we describe a practical approach to choosing $\lambda$ using a free forecast model run like in \cite{Robert} and the BIC.
\subsection{Implications on the Kalman Gain Matrix}
The only estimated randomness in the EnKF occurs in the ensemble update step, which is a linear function of the Kalman gain matrix. So, having an accurate estimator of the true Kalman gain matrix $\mat{K}$ will ensure that the algorithm performs well. And, because the true Kalman gain matrix inherits many of the properties of the forecast covariance matrix $\mat{P}^f $, our modified Kalman gain matrix $ \tilde{\mat{K}} $ will benefit from many of the nice properties of our forecast covariance estimator $ \tilde{\mat{P}}^f $.
How good of an estimator we can get for the forecast covariance matrix $\mat{P}^f$ will of course depend on its structure. If it is close to singular or contains lots of entries with magnitudes smaller than the noise level, it will be always be difficult to estimate. So for the following theorem, we assume that the forecast covariance matrix is well-behaved. This means that it satisfies the standard regularity conditions (incoherence, bounded eigenvalue, sparsity, sign consistency and monotonicity of the tail function) found in many places including \cite{ravikumar2011high, jankova2015confidence} and also defined in the appendix.
\begin{theorem}\label{SSE_K}
Under regularity conditions and for the system described in Section \ref{background}, when $\lambda \asymp \sqrt{3 \log(p) / n}$ for sub-Gaussian ensembles and $ \lambda \asymp \sqrt{p^{3/m}/n} $ for ensembles with bounded $ 4m^\mathrm{th} $ moments,
$$ \text{Sum of Squared Errors of } \tilde{\mat{K}} \lesssim \text{ Sum of Squared Errors of } \hat{\mat{K}} $$
and as long as the sample size is at least $ o(n) = 3 d^2 \log(p) $ for sub-Gaussian ensembles and $o(n) = d^2 p^{3/m} $ for ensembles with bounded $ 4m^\mathrm{th} $ moments,
$$ \text{Sum of Squared Errors of } \tilde{\mat{K}} \rightarrow 0 \text{ with high probability as } n, p, s \rightarrow \infty .$$
\end{theorem}
The above theorem gives us a sense of the performance of the modified Kalman gain matrix in comparison to the sample Kalman gain matrix. It shows that with high probability, the modified Kalman gain matrix will have an asymptotically smaller sum of squared error (SSE) than a Kalman gain matrix formed using the sample forecast covariance matrix. Also, for a given number of states $p$, the theorem tell us the minimum ensemble size $n$ required for our modified Kalman gain matrix to be a good estimate of the true Kalman gain matrix. The sub-Gaussian criterion, where all moments are bounded, is actually very broad and includes any state vectors with a strictly log-concave density and any finite mixture of sub-Gaussian distributions. However even if not all moments are bounded, the larger the number of bounded fourth-order moments $m$, the smaller the necessary sample size. In comparison, the sample Kalman gain matrix requires $ o(n) = p^2 $ samples in the sub-Gaussian case, and also significantly more in the other case (see appendix for exact details). When the minimum sample size for a estimator is not met, good performance cannot be guaranteed because the asymptotic error will diverge to infinity instead of converge to zero. This is why when the number of ensembles $n$ is smaller than the number of states $p$, just using the sample forecast covariance matrix is not sufficient.
\subsection{Implications on the Analysis Ensemble}
It is well known that due to the additional stochastic noise used to perturb the observations, the covariance of the EnKF's analysis ensemble, $ \widehat{\Cov}(\mat{A}) $ is not equivalent to its analysis covariance calculated by the Gaussian update $ \hat{\mat{P}}^a = (\mat{I} - \hat{\mat{K}} \mat{H}) \hat{\mat{P}}^f $. This has led to the development of deterministic variants such as the square root and transform filters, which do have $ \widehat{\Cov}(\mat{A}) = \hat{\mat{P}}^a $. However, in a non-linear system, this update is sub-optimal because it uses a Gaussian approximation of $ \Pr(\v{x}_t | \v{y}_{t-1}) $, the actual distribution of forecast ensemble $ \mat{A}_0 $. Thus let us denote $ \mat{P}^a $ as the true analysis covariance defined as
\begin{flalign*}
\int (\v{x}_t)^2 \Pr(\v{x}_t | \v{y}_t) d\v{x}_t - \left( \int \v{x}_t \Pr(\v{x}_t | \v{y}_t) \, d\v{x}_t \right)^2
\end{flalign*}
where $ \Pr(\v{x}_t | \v{y}_t) = \Pr(\v{y}_t | \v{x}_t) \Pr(\v{x}_t | \v{y}_{t-1}) / \Pr(\v{y}_t) $ is not Gaussian. Then, $\E(\hat{\mat{P}}^a ) \neq \mat{P}^a $ and there will always be this analysis spread error regardless of whether $ \widehat{\Cov}(\mat{A}) = \hat{\mat{P}}^a $ or not.
As also mentioned in \cite{lei2011}, actually none of the analysis moments of the EnKF are consistent with the true moments including the analysis mean. However this analysis error is present in all methods that do not introduce particle filter properties to the EnKF, and thus is not the focus of our paper. We are primarily concerned with the sampling errors in high-dimensional systems and simply wanted to address that the lack of equivalence to the Gaussian update is irrelevant in our case of a non-linear system.
\subsection{Computational Time and Storage Issues}
The computational complexity of solving for the minimizer of \eqref{objective} with the GLASSO algorithm from \cite{friedman2008sparse} is $O(s p^2)$ because it is a coordinate descent algorithm. Although the final estimator $(\tilde{\mat{P}^f})^{-1}$ is sparse and only requires storing $s+p$ values, the algorithm requires storing $ p \times p $ matrices in memory. However, by using iterative quadratic approximations to \eqref{objective}, block coordinate descent, and parallelization, the BIGQUIC algorithm of \cite{hsieh2013big} has computational complexity $O(s (p/k))$ and only requires storing $ (p/k) \times (p/k) $ matrices, where $k$ is the number of parallel subproblems or blocks.
The matrix operations for the analysis update $ \mat{A} = \mat{A}_0 + up $ can also be linear in $p$ if $\mat{R}$ is diagonal and $\mat{H}$ is sparse (like in banded interpolation matrices) with at most $h << q$ non-zero entries in a row. Then $( (\tilde{\mat{P}^f})^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H})$ has at most $(s + p + qh^2) << p^2$ non-zero entries and can be computed with $ O(s+p+qh^2) $ matrix operations. And, solving for $up$ only takes $O(n(s+p+qh^2))$ matrix operations because it is made from the solutions to the sparse linear systems $ ((\tilde{\mat{P}^f})^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H}) up = \mat{H}^T \mat{R}^{-1} (\mat{D}_t - \mat{H} \mat{A}_0)$ where the right-hand side takes $ O(p q^2 + q p n) $ matrix operations to form.
\section{Simulations} \label{sec:sims}
In all simulations, we compare to an ensemble Kalman filter where the forecast covariance matrix is localized with a taper matrix generated from equation (4.10) in \cite{gaspari1999}. The taper matrix parameter $c$ is chosen using the true interactions of the system, so the localization should be close to optimal for simple systems. We use this TAPER-EnKF as the baseline because if the PEnKF can do as well as this filter, it implies that the PEnKF can learn a close to optimal covariance matrix, even without the need to impose a known neighborhood structure. If the PEnKF can do better than this filter, it implies that the PEnKF is learning some structure that is not captured by localization with a taper matrix.
In order to choose the penalty parameter for PEnKF, we assume that the state variables in our examples are sub-Gaussian. In this case, we can set $\lambda = c_{\lambda} \sqrt{ R \log(p) / n }$ for some appropriate choice of $c_{\lambda}$ (see the proof of Theorem \ref{SSE_K}), where $R$ is the observation noise's variance. To estimate $c_{\lambda}$, we generate a representative ensemble (which may also be our initial ensemble) using a free forecast run like in \cite{Robert} in which a state vector is drawn at random (e.g. from $N(\v{0}, \mat{I}) $) and evolved forward. The representative ensemble is produced by taking a set of equally spaced points (e.g. every 100th state vector) from the evolution. This ensemble is used to choose $c_{\lambda}$ from some predefined interval by minimizing the extended Bayesian information criterion (eBIC) of \cite{foygel2010extended} if $p > n$ or the BIC of \cite{schwarz1978} if $p < n$.
Of course \eqref{objective} is not a likelihood unless the states are Gaussian. So, we have a misspecified model where we are treating the states as having a Gaussian likelihood when evaluating a potential penalty parameter using an information criterion. In this case, we should correct our information criterion for the misspecification as in \cite{lv2014model}. However, this can be quite difficult and we leave an in-depth exploration of this problem for future work. In the meantime, we assume the misspecified information criterion is close to the correct information criterion, and it does seem to perform well despite its lack of optimality.
We define the root mean squared error (RMSE) used to evaluate a filters performance by
\begin{flalign*}
\text{RMSE}_t = \sqrt{ (|| \hat{\v{x}}_t - \v{x}_t ||_2)^2 / p}
\end{flalign*}
where $\text{RMSE}_t$ is an element of a vector indicating the RMSE at time point $t$, $ \v{x}_t $ is a vector of the true hidden state variables, $ \hat{\v{x}}_t $ is a filter's estimators for the true state vector, and $ || \cdot ||_2 $ is the $ \ell_2 $ norm. We will refer to quantiles such as the mean or median RMSE to be the mean or median of the elements of the RMSE vector.
\subsection{Lorenz 96 System}
The 40-state Lorenz 96 model is one of the most common systems used to evaluate ensemble Kalman filters. The state variables are governed by the following differential equations
\begin{flalign*}
\frac{dx^{i}_{t}}{dt} = \left( x^{i+1}_t - x^{i-2}_t \right) x^{i-1}_t - x^i_t + 8 \qquad \forall i=1, \dots, 40
\end{flalign*}
where $ x^{41}_t = x^{1}_t, x^{0}_t = x^{40}_t, \text{ and } x^{-1}_t = x^{39}_t$.
We use the following simulation settings. We have observations for the odd state variables, so $ \v{y}_t = \mat{H} \v{x}_t + \bm{\epsilon}_t$ where $\mat{H}$ is a $ 20 \times 40$ matrix with ones at entries $ \{ i, j=2i-1 \} $ and zeros everywhere else and $ \bm{\epsilon}_t $ is a $ 20 \times 1 $ vector drawn from a $N(\v{0}, 0.5 \, \mat{I}) $. We initialize the true state vector from a $N(\v{0}, \mat{I}) $ and we assimilate at every $ 0.4 t $ time steps, where $ t = 1, \dots, 2000 $. The system is numerically integrated with a 4th order Runge-Kutta method and a step size of 0.01. The main difficulties of this system are the large assimilation time step of $0.4$, which makes it significantly non-linear, and the lack of observations for the even state variables.
Since the exact equations of the Lorenz 96 model are fairly simple, it is clear how the state variables interact with each other. This makes it possible to localize with a taper matrix that is almost optimal by using the Lorenz 96 equations to choose a half-length parameter $c$. However, we do not incorporate this information in the PEnKF algorithm, which instead learns interactions by essentially extracting it from the sample covariance matrix. We set the penalty parameter $ \lambda = c_\lambda \sqrt{0.5 \log(p) / n}$ by using an offline free forecast run to search for the constant $ c_\lambda $ in the range $ [0.1, 10] $ as described at the beginning of this section.
We average the PEnKF estimator of the forecast inverse covariance matrix at the time points 500, 1000, 1500, and 2000 for 50 trials with 25 ensembles members, and we compare it to the ``true" inverse covariance matrix, which is calculated by moving an ensemble of size 2000 through time. In Figure \ref{iCovEntries}, each line represents the averaged normalized rows of an inverse covariance matrix and the lines are centered at the diagonal. The penalized inverse covariance matrix does a qualitatively good job of capturing the neighborhood information and successfully identifies that any state variables far away from state variable $i$, do not interact with it.
Because the PEnKF is successful at estimating the structure of the inverse covariance matrix and thus the forecast covariance matrix, we expect it will have good performance for estimating the true state variables. We compare the PEnKF to the TAPER-EnKF and other estimators from \cite{bengtsson2003, lei2011, frei2013bridging, frei2013mixture} by looking at statistics of the RMSE. Note that in order to have comparable statistics to as many other papers as possible, we do not add variance inflation to the TAPER-EnKF (like in \cite{bengtsson2003, frei2013bridging, frei2013mixture} and unlike in \cite{lei2011} ). Also, like in those papers, we initialize the ensemble from a $N(\v{0}, \mat{I}) $, and we use this ensemble to start the filters. Note that in this case, the initial ensemble is different than the offline ensemble that we use to estimate the PEnKF's penalty parameter. This is because the initial ensemble is not representative of the system and its sample covariance is an estimator for the identity matrix. The TAPER-EnKF, which is simply called the EnKF in the other papers, is localize by applying a taper matrix where $c = 10$ to the sample covariance matrix.
We show the mean, median, 10\%, and 90\% quantiles of the RMSE averaged over 50 independent trials for ensembles of size 400, 100, 25, and 10 in Table \ref{RMSE_lorenz}. For 400 ensemble members, the PEnKF does considerably better than the TAPER-EnKF and its relative improvement is larger than that of the XEnKF reported in \cite{bengtsson2003} and similar to those of the NLEAF, EnKPF, and XEnKF reported in \cite{lei2011, frei2013bridging, frei2013mixture} respectively. For 100 ensemble members, the PEnKF does do worse than the TAPER-EnKF and EnKPF of \cite{frei2013bridging}; this we suspect may be do to the bias-variance trade-off when estimating the forecast covariance matrix. The PEnKF has the most significant improvement over the TAPER-EnKF in the most realistic regime where we have fewer ensemble members than state variables. For both 25 and 10 ensemble members, the PEnKF does considerably better than the TAPER-EnKF and it does not suffer from filter divergence, which \cite{frei2013bridging} report occurs for the EnKPF at 50 ensemble members.
While it is clear the PEnKF does well even when there are fewer ensemble members than state variables, 40 variables is not enough for the problem to be considered truly high-dimensional. We now consider simulation settings where we increase the dimension of the state space $p$ while holding the number of ensemble members $n$ constant. We initialize the ensemble from the free forecast run and set $ \lambda $ and the taper matrix in the same way as in the previous simulations. We examine the mean RMSE averaged over 50 trials and its approximate 95\% confidence intervals in the Figure \ref{Lorenz_RMSE}. The mean RMSE of the PEnKF is significantly smaller than the mean RMSE of the TAPER-EnKF for all $p$. Additionally the confidence intervals of the mean RMSE are much narrower than the ones for the TAPER-EnKF. This suggest that there is little variability in the PEnKF's performance, while the TAPER-EnKF's performance is more dependent on the trial, with some trials being ``easier" for the TAPER-EnKF than others.
\subsection{Modified Shallow Water Equations System}
While the Lorenz 96 system shows that the PEnKF has strong performance because it is successful at reducing the sampling errors and capable of learning the interactions between state variables, the system is not very realistic in that all state variables are identical and the relationship between state variables is very simplistic. We now consider a system based on the modified shallow water equations of \cite{würsch2014}, which models cloud convection with fluid dynamics equations, but is substantially computationally less expensive than actual numerical weather prediction models. The system has three types of state variables: fluid height, rain content, and horizontal wind speed.
To generate this system we use the R package ``modifiedSWEQ" created by \cite{Rpackage}, and the same simulation settings as in \cite{Robert}. So we always observe the rain content, but wind speed is only observed at locations where it is raining and fluid height is never observed. Explicitly for the R function $generate.xy()$, we use $h_c = 90.02, h_r = 90.4 $ for the cloud and rainwater thresholds, a 0.005 rain threshold, $ \sigma_r = 0.1, \sigma_u = 0.0025 $ to be the standard deviation of the observation noise for rain and wind respectively, and $ \mat{R} = \text{diag}([R^2_r=0.025^2 \,\, R^2_u=\sigma_u^2] )$ to be the estimated diagonal noise covariance matrix. All other parameters are just the default ones in the function. The initial ensemble is drawn from a free forecast run with 10000/60 time-steps between each ensemble member. We give a snapshot of the system at a random time point in Figure \ref{SWEQsystem}. There are $ p = 300 $ state variables for each type, making the state space have 900 dimensions and we assimilate the system every 5 seconds for a total time period of 6 hours. Like in \cite{Robert}, we choose to use only 50 ensemble members and we do not perturb rain observations that are 0, because at these points there is no measurement noise.
The TAPER-EnKF uses a $ 3p \times 3p $ taper matrix with $c = 5$, however the entries off the $ p \times p $ block diagonals are depressed (they are multiplied by 0.9). The NAIVE-LEnKPF uses the same settings as in \cite{Robert}, so a localization parameter of 5km, which gives the same taper matrix as the one used in the TAPER-EnKF, and an adaptive $\gamma$ parameter. For the PEnKF, we set the penalty parameter to be a $ 3p \times 3p $ matrix, $ \bm{\Lambda} = c_\lambda \sqrt{ \bm{\lambda}_R \bm{\lambda}_R^T \log(3p) / n } $, where the first $p$ entries of the vector $ \bm{\lambda}_R $ are reference units and the rest are to scale for the perturbation noise of the different state types. So the first $p$ are 1 (reference) for fluid height, the second $p$ are $ R_u $ for wind, and the last $p$ are $ R_r $ for rain. We choose the constant $ c_\lambda $ with eBIC like before and search in the range $ [.005, 1] $.
Figure \ref{SWEQ_RMSE} shows the mean and approximate 95\% confidence intervals of the RMSE for fluid height, wind speed, and rain content over 6 hours of time using 50 trials. The mean RMSE for all three filters are well within each others' confidence intervals for the fluid height and wind variables. For the rain variables, the mean RMSE of neither the TAPER-EnKF nor the NAIVE-LEnKPF are in the PEnKF's confidence intervals and the mean RMSE of the PEnKF is on the boundary of the other two models' confidence intervals. This strongly suggests that the PEnKF's rain error is statistically smaller than the rain errors of the other two filters. Since this simulation is not as simple as the previous ones, the interactions between the state variables are most likely not as effectively captured by the taper matrix or other localization methods, and the results from this simulation suggest that the PEnKF is learning more accurate interactions for the rain variables. We do not show the results of the BLOCK-LEnKPF of \cite{Robert} because the algorithm suffered from filter divergence in 27 of the 50 trials, and in the trials where it did not fail, it performed very similar to the NAIVE-LEnKPF.
\section{Discussion}
We propose a new algorithm based on the ensemble Kalman filter that is designed for superior performance in non-linear high dimensional systems. This algorithm we call the penalized ensemble Kalman filter because it uses the popular statistical concept of penalization/regularization in order to make the problem of estimating the forecast covariance matrix well-defined (strictly convex). This in turn both decreases the sampling errors in the forecast covariance estimator by trading it off for bias and prevents filter divergence by ensuring that the estimator is positive definite. The PEnKF is computationally efficient in that it is not significantly slower than the standard EnKF algorithms and easy to implement since it only adds one additional step, and this step uses the well-established GLASSO algorithm available in almost any scientific computing language. We give theoretical results that prove that the Kalman gain matrix constructed from this estimator will converge to the population Kalman gain matrix under the non-simplistic asymptotic case of high-dimensional scaling, where the sample size and the dimensionality increase to infinity.
Through simulations, we show that the PEnKF can do at least as well as, and sometimes better than, localized filters that use much more prior information. We emphasize that by doing just as well as the TAPER-EnKF which has a close to optimal taper matrix, the PEnKF is effectively correctly learning the structure of interactions between the state variables. In a non-simulation setting where there is no ground-truth knowledge of the interactions between state variables, correct localization is much more difficult, making any localized filter's performance likely sub-optimal. In contrast, since the PEnKF does not use any of this ``oracle" information, its performance will not differ in this way between simulations and real-life situations. The more complicated simulation, based on the modified shallow water equations, highlights this advantage of the PEnKF through its substantial superior performance in estimating the hidden states of the rain variables. Another feature of the approach is that it seems to require less variance inflation. None was applied to any algorithm in our comparison, but the PEnKF approach never collapsed. The penalization of the inverse covariance actually produces a slight inflation on the diagonal of the covariance, which seems to help in this regard.
While we display a very naive way of searching for a good penalty parameter for the PEnKF in the simulations, it is theoretically incorrect and thus not a way to chose the truly optimal penalty parameter. We do believe deriving a specific information criterion for our PEnKF with correct theoretical properties is very important since the PEnKF can be sensitive to the penalty parameter. However, this model selection in misspecified models problem is not trivial to solve and an active topic in current statistical research. Therefore, we will leave deriving a theoretically correct information criterion for future work.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% ACKNOWLEDGMENTS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
\acknowledgments
This work was partially supported by the Consortium for Verification Technology under Department of Energy National Nuclear Security Administration award number DE-NA0002534 and partially supported by the Laboratory Directed Research and Development program at Los Alamos National Laboratory under project 20150033DR, SHIELDS: Space Hazards Induced near Earth by Large Dynamic Storms - Understanding, Modeling, and Predicting.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% TABLES
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Enter tables at the end of the document, before figures.
\begin{table}[t]
\caption{Mean, median, 10\% and 90\% quantile of RMSE averaged over 50 trials. The number in the parentheses is the summary statistics' corresponding standard deviation. }
\begin{center}
\begin{tabular}{ccccrrcrc}
\hline\hline
$n = 400$ & 10\% & 50 \% & Mean & 90\% \\
\hline
TAPER-EnKF & 0.580 (.01) & 0.815 (.01) & 0.878 (.02) & 1.240 (.03) \\
PEnKF & 0.538 (.02) & 0.757 (.03) & 0.827 (.03) & 1.180 (.05) \\
\hline
\end{tabular}
\medskip
\noindent
\begin{tabular}{ccccrrcrc}
\hline\hline
$n = 100$ & 10\% & 50 \% & Mean & 90\% \\
\hline
TAPER-EnKF & 0.582 (.01) & 0.839 (.02) & 0.937 (.03) & 1.390 (.06) \\
PEnKF & 0.717 (.04) & 0.988 (.04) & 1.067 (.04) & 1.508 (.05) \\
\hline
\end{tabular}
\medskip
\noindent
\begin{tabular}{ccccrrcrc}
\hline\hline
$ \ n = 25 \ $ & 10\% & 50 \% & Mean & 90\% \\
\hline
TAPER-EnKF & 0.769 (.04) & 1.668 (.13) & 1.882 (.09) & 3.315 (.11) \\
PEnKF & 0.971 (.03) & 1.361 (.03) & 1.442 (.03) & 2.026 (.04) \\
\hline
\end{tabular}
\medskip
\noindent
\begin{tabular}{ccccrrcrc}
\hline\hline
$ \ n = 10 \ $ & 10\% & 50 \% & Mean & 90\% \\
\hline
TAPER-EnKF & 2.659 (.07) & 3.909 (.06) & 3.961 (.05) & 5.312 (.06) \\
PEnKF & 1.147 (.02) & 1.656 (.02) & 1.735 (.02) & 2.437 (.04) \\
\hline
\end{tabular}
\end{center}
\label{RMSE_lorenz}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% FIGURES
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Enter figures at the end of the document, after tables.
\begin{figure}[t]
\centering
\noindent\includegraphics[width=3.25in, height=3in]{iCovEntries.png}\\
\caption{Each line represents the normalized values of entries of row $i$ of the inverse covariance matrix, ordered from $i-20$ to $i+20$, where the values are averaged over 50 trials. The PEnKF algorithm is successful at identifying that the state variables far away from variable $i$ have no effect on it, even though there are fewer ensemble members than state variables. }\label{iCovEntries}
\end{figure}
\begin{figure}[t]
\centering
\noindent\includegraphics[width=3.25in, height=2.5in]{RMSE_col.png}\\
\caption{The RMSE of the TAPER-EnKF and PEnKF over 50 trials. The darker lines of each linetype are the mean and the colored areas are the 95\% confidence intervals. There is clear separation between the RMSE of the two filters with the PEnKF's error as significantly smaller.} \label{Lorenz_RMSE}
\end{figure}
\begin{figure}[t]
\centering
\noindent\includegraphics[width=3.25in, height=3.5in]{SWEQsystem.png}\\
\caption{Fluid height, rain, and wind at 300 different locations at an instance of time. The blue dots are observations; rain is always observed, wind is only observed when the rain is non-zero, fluid height is never observed. The dashed lines in fluid height are the cloud and rainwater thresholds.}\label{SWEQsystem}
\end{figure}
\begin{figure}[t]
\centering
\noindent\includegraphics[width=3.25in, height=4.75in]{SWEQ_RMSE.png}\\
\caption{The RMSE of the TAPER-EnKF, NAIVE-LEnKPF, and PEnKF over 50 trials. The darker lines of each linetype are the mean and the colored areas are the 95\% confidence intervals. All three filters are pretty indistinguishable except for the PEnKF's rain error, which is statistically smaller than the others.} \label{SWEQ_RMSE}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIXES
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
\appendix
\begin{definition*} \leavevmode
\begin{enumerate}[label= (D\arabic*)]
\item $\sigma_{\min}(\mat{A})$ and $\sigma_{\max}(\mat{A})$ denote the minimum and maximum singular values of any matrix $\mat{A}$.
\item The spectral $||\cdot||_2$ and Frobenius $||\cdot||_F$ norms are submultiplicative $\lVert \mat{A}\mat{B} \rVert \leq \lVert \mat{A} \rVert \lVert \mat{B} \rVert $ and unitary invariant $\lVert\mat{A} \mat{U} \rVert = \lVert \mat{U}^T \mat{A} \rVert = \lVert \mat{A}^T \rVert $ where $\mat{U} \mat{U}^T = \mat{I}$. So $ \lVert \mat{A} \mat{B}\rVert_F = \lVert \mat{A} \mat{U}\mat{D}\mat{V}^T||_F = \lVert\mat{A} \mat{U} \mat{D}\rVert_F \leq \lVert \mat{A} \mat{U} \sigma_{\max}(\mat{D})\rVert_F = \lVert\mat{A} \mat{U} \rVert_F \lVert \mat{D}\rVert_2 = \lVert \mat{A}||_F \lVert\mat{B}\rVert_2 $
\item $||\mat{A}^{-1}||_2 = \sigma_{\max}(\mat{A}^{-1}) = 1/\sigma_{\min}(\mat{A}) $
\item $\mat{K}$ and $\tilde{\mat{K}}$ can be decomposed like
\vspace{-8pt}
\begin{flalign*}
& \mat{K} = \mat{P}^f \mat{H}^T ( \mat{H} \mat{P}^f \mat{H}^T +\mat{R} )^{-1} \\
& \hspace{8pt} = \mat{P}^f \mat{H}^T \left( \mat{R}^{-1} - \mat{R}^{-1} \mat{H} ((\mat{P}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H} )^{-1} \mat{H}^T \mat{R}^{-1} \right) \\
& \hspace{8pt} = \left( \mat{P}^f - \mat{P}^f \left((\mat{P}^f)^{-1} (\mat{H}^T \mat{R}^{-1} \mat{H})^{-1} + \mat{I} \right)^{-1} \right) \mat{H}^T \mat{R}^{-1} \\
& \hspace{8pt} = \left( \mat{P}^f - \mat{P}^f \left((\mat{H}^T \mat{R}^{-1} \mat{H})^{-1} + \mat{P}^f \right)^{-1} \mat{P}^f \right) \mat{H}^T \mat{R}^{-1} \\
& \hspace{8pt} = \left((\mat{P}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H}\right)^{-1} \mat{H}^T \mat{R}^{-1}.
\end{flalign*}
\item $\mathcal{E}$ is the edge set corresponding to non-zeros in $ ( \mat{P}^f )^{-1} $ and $\mathcal{E}^c$ is its complement. $ \Gamma $ is the Hessian of \eqref{objective}.
\item $\partial \lVert \bm{\Theta}\rVert _1 $ can be any number between -1 and 1 where $\bm{\Theta}$ is 0 because the derivative of an absolute value is undefined at zero. Thus, it is the set of all matrices $ \mat{Z} \in \mathbb{S}^{p \times p} $ such that
\vspace{-8pt}
\begin{flalign} \label{subdiff}
& Z_{ij} =
\begin{cases}
& \text{sign} (\Theta_{ij}) \quad\quad \text{if} \quad \Theta_{ij} \neq 0 \\
& \in [-1, 1] \, \quad\quad \text{if} \quad \Theta_{ij} = 0 \, .
\end{cases}
\end{flalign}
\item A bounded $ 4m^\mathrm{th} $ moment is the highest fourth-order moment of a random variable that is finite, where $m$ is the number of fourth-order moments.
\end{enumerate}
\end{definition*}
\begin{lemma} \label{constantH}
Because $\mat{H}$ is a constant matrix, it does not affect the asymptotic magnitude of the modified or sample Kalman gain matricies under any norm.
\end{lemma}
\begin{proof}[Proof of Lemma \ref{constantH}]
$ ||\tilde{\mat{K}}|| \asymp ||\mat{H}|| \,\, ||\tilde{\mat{K}}|| \asymp ||\mat{H} \tilde{\mat{K}}||$ under any norm where $\mat{H} \tilde{\mat{K}}$
\begin{flalign*}
& = \mat{H} \tilde{\mat{P}}^f \mat{H}^T ( \mat{H} \tilde{\mat{P}}^f \mat{H}^T +\mat{R} )^{-1} = \left( \mat{I} + \mat{R} (\mat{H} \tilde{\mat{P}}^f \mat{H}^T)^{-1} \right)^{-1} \\
& = \mat{R} \mat{R}^{-1} \left( \mat{R}^{-1} + (\mat{H} \tilde{\mat{P}}^f \mat{H}^T)^{-1} \right)^{-1} \mat{R}^{-1} \\
& = \mat{I} - \mat{R} (\mat{H} \tilde{\mat{P}}^f \mat{H}^T +\mat{R} )^{-1}
\end{flalign*}
The same argument holds for $\hat{\mat{K}}$, where $(\mat{H} \hat{\mat{P}}^f \mat{H}^T)^{-1}$ is the pseudoinverse if the inverse does not exist.
\end{proof}
\begin{assumptions*} \leavevmode
The following assumptions are necessary for the minimizer of \eqref{objective} to have good theoretical properties, \cite{ravikumar2011high}. Thus we assume they are true for the theorem.
\begin{enumerate}[label= (A\arabic*)]
\item \label{a1} There exists some $ \alpha \in (0, 1] $ such that $ \underset{e \in \mathcal{E}^c}{\max} \, || \Gamma_{e\mathcal{E}} (\Gamma_{\mathcal{E}\mathcal{E}})^{-1} ||_1\le (1-\alpha) $.
\item \label{a2} The ratio between the maximum and minimum eigenvalues of $\mat{P}^f$ is bounded.
\item \label{a3} The maximum $ \ell_1 $ norms of the rows of $ \mat{P}^f $ and $(\Gamma_{\mathcal{E}\mathcal{E}})^{-1} $ are bounded.
\item \label{a4} The minimum non-zero value of $ ( \mat{P}^f )^{-1} $ is $ \Omega( \sqrt{\log(p) / n} ) $ for a sub-Gaussian state vector and $ \Omega(\sqrt{p^{3/m}/n}) $ for state vectors with bounded $ 4m^\mathrm{th} $ moments.
\end{enumerate}
Our assumptions are stronger than necessary, and it is common to allow the error rates to depend on the bounding constants above, but for simplicity we give the error rates only as a function of the dimensionality $n, p$ and sparsity $s, d$ parameters.
\end{assumptions*}
\begin{proof}[Proof of Theorem \ref{SSE_K}]
From \cite{vershynin2012close} and \cite{ravikumar2011high}, we know that for sub-Gaussian random variables and those with bounded 4$m^\mathrm{th}$ moments respectively, the SSE of the sample covariance matrix are
\begin{equation} \label{SSE_SCM}
\begin{cases}
& O \left( p^2 / n \right) \\
& O \left( \left(\log_2\log_2(p) \right)^4 p (p/n)^{1-1/m} \right)
\end{cases}
\end{equation}
and with high probability and the SSE of $(\tilde{\mat{P}}^f)^{-1}$ are
\begin{equation} \label{SSE_glasso}
\begin{cases}
& O \left( 3 (s + p) \log(p) / n \right) \text{ for } \quad \lambda \asymp \sqrt{3 \log(p) / n} \\
& O \left( (s + p) p^{3/m} / n \right) \text{ for } \quad \lambda \asymp \sqrt{p^{3/m}/n}
\end{cases}
\end{equation}
with probability 1 - 1/p.
\begin{flalign*}
& \lVert \mat{H} \hat{\mat{K}} - \mat{H} \mat{K}\rVert_F^2 = \lVert \mat{R} (\mat{H} \mat{P}^f \mat{H}^T +\mat{R} )^{-1} - \mat{R} (\mat{H} \hat{\mat{P}}^f \mat{H}^T +\mat{R} )^{-1} \rVert_F^2 \\
& = \lVert \mat{R} \left( (\mat{H} \hat{\mat{P}}^f \mat{H}^T +\mat{R} )^{-1} \left( (\mat{H} \hat{\mat{P}}^f \mat{H}^T +\mat{R} )- (\mat{H} \mat{P}^f \mat{H}^T +\mat{R} ) \right) (\mat{H} \mat{P}^f \mat{H}^T +\mat{R} )^{-1} \right) \rVert_F^2 \\
& = \lVert \mat{R} (\mat{H} \hat{\mat{P}}^f \mat{H}^T +\mat{R} )^{-1} \mat{H} (\hat{\mat{P}}^f - \mat{P}^f) \mat{H}^T (\mat{H} \mat{P}^f \mat{H}^T +\mat{R} )^{-1} \rVert_F^2 \\
& \leq \lVert \mat{R} (\mat{H} \hat{\mat{P}}^f \mat{H}^T +\mat{R} )^{-1} \mat{H} \rVert_2^2 \lVert (\hat{\mat{P}}^f - \mat{P}^f) \rVert_F^2 \lVert \mat{H}^T (\mat{H} \mat{P}^f \mat{H}^T +\mat{R} )^{-1} \rVert_F^2
\end{flalign*}
So, the second term has the rates in \eqref{SSE_SCM} and the final term is a constant. The first term is also a constant because
\begin{flalign*}
& \lVert \mat{R} (\mat{H} \hat{\mat{P}}^f \mat{H}^T +\mat{R} )^{-1} \mat{H} \rVert_2^2 \leq \lVert (\mat{H} \hat{\mat{P}}^f \mat{H}^T \mat{R}^{-1} + \mat{I})^{-1} \rVert_2^2 \lVert\mat{H}\rVert_2^2 \\
& = \lVert \mat{H}\rVert_2^2 / (\sigma_{\min}(\mat{H} \hat{\mat{P}}^f \mat{H}^T \mat{R}^{-1} + \mat{I}))^2 \leq \lVert\mat{H}\rVert_2^2 .
\end{flalign*}
Thus $\lVert\mat{H} \hat{\mat{K}} - \mat{H} \mat{K}\rVert_F^2$ also has the rates in \eqref{SSE_SCM} and from Lemma \ref{constantH}, $ \lVert\hat{\mat{K}} - \mat{K}\rVert_F^2 $ does too.
\begin{flalign*}
& \lVert\tilde{\mat{K}} - \mat{K}\rVert_F^2 \\
& = \lVert\left( ((\tilde{\mat{P}}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H} )^{-1} - ((\mat{P}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H})^{-1} \right) \mat{H}^T \mat{R}^{-1}\rVert_F^2 \\
& = \lVert\left( ((\mat{P}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H})^{-1} \Big( ((\mat{P}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H}) \right. \\
& \hspace{12pt} \left. - ((\tilde{\mat{P}}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H}) \Big) ((\tilde{\mat{P}}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H})^{-1} \right) \mat{H}^T \mat{R}^{-1} \rVert_F^2 \\
& \leq \lVert ((\mat{P}^f)^{-1} + \mat{H}^T \mat{R}^{-1} \mat{H})^{-1} \rVert_F^2 \lVert (\mat{P}^f)^{-1} - (\tilde{\mat{P}}^f)^{-1} \rVert_F^2 \lVert \tilde{\mat{K}} \rVert_2^2
\end{flalign*}
The first term is a constant and the second term has the rates in \eqref{SSE_glasso}. The final term is also a constant because $ || \tilde{\mat{K}} ||_2^2 \asymp || \mat{H} \tilde{\mat{K}} ||_2^2 = 1/ \sigma_{\min}( \mat{I} + \mat{R} (\mat{H} \tilde{\mat{P}}^f \mat{H}^T)^{-1}) \leq 1$ . Thus $ \lVert\tilde{\mat{K}} - \mat{K}\rVert_F^2 $ also has the rates in \eqref{SSE_glasso}.
\end{proof}
\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% REFERENCES
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Make your BibTeX bibliography by using these commands:
\bibliographystyle{ametsoc2014}
\bibliography{references}
\end{document}
\begin{thebibliography}{52}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\renewcommand{\UrlFont}{\rmfamily}
\providecommand{\urlprefix}{URL }
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi:\discretionary{}{}{}#1}\else
\providecommand{\doi}{doi:\discretionary{}{}{}\begingroup
\urlstyle{rm}\Url}\fi
\providecommand{\eprint}[2][]{\url{#2}}
\bibitem[{Ades and van Leeuwen(2013)Ades, and van Leeuwen}]{Ades}
Ades, M., and P.~J. van Leeuwen, 2013: An exploration of the equivalent weights
particle filter. \textit{Quarterly Journal of the Royal Meteorological
Society}, \textbf{139~(672)}, 820--840, \doi{10.1002/qj.1995}.
\bibitem[{Anderson(2007)}]{anderson2007}
Anderson, J.~L., 2007: An adaptive covariance inflation error correction
algorithm for ensemble filters. \textit{Tellus A}, \textbf{59~(2)}, 210--224,
\doi{10.1111/j.1600-0870.2006.00216.x},
\urlprefix\url{http://dx.doi.org/10.1111/j.1600-0870.2006.00216.x}.
\bibitem[{Anderson(2009)}]{anderson2009}
Anderson, J.~L., 2009: Spatially and temporally varying adaptive covariance
inflation for ensemble filters. \textit{Tellus A}, \textbf{61~(1)}, 72--83,
\doi{10.1111/j.1600-0870.2008.00361.x},
\urlprefix\url{http://dx.doi.org/10.1111/j.1600-0870.2008.00361.x}.
\bibitem[{Bengtsson et~al.(2003)Bengtsson, Snyder,, and Nychka}]{bengtsson2003}
Bengtsson, T., C.~Snyder, and D.~Nychka, 2003: Toward a nonlinear ensemble
filter for high-dimensional systems. \textit{Journal of Geophysical Research:
Atmospheres}, \textbf{108~(D24)}.
\bibitem[{Bishop and Hodyss(2009{\natexlab{a}})Bishop, and
Hodyss}]{bishop2009a}
Bishop, C., and D.~Hodyss, 2009{\natexlab{a}}: Ensemble covariances adaptively
localized with eco-rap. part 1: tests on simple error models. \textit{Tellus
A}, \textbf{61~(1)}.
\bibitem[{Bishop et~al.(2001)Bishop, Etherton,, and Majumdar}]{bishop2001}
Bishop, C.~H., B.~J. Etherton, and S.~J. Majumdar, 2001: Adaptive sampling with
the ensemble transform kalman filter. part i: Theoretical aspects.
\textit{Monthly Weather Review}, \textbf{129~(3)}, 420--436,
\doi{10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2}.
\bibitem[{Bishop and Hodyss(2009{\natexlab{b}})Bishop, and
Hodyss}]{bishop2009b}
Bishop, C.~H., and D.~Hodyss, 2009{\natexlab{b}}: Ensemble covariances
adaptively localized with eco-rap. part 2: a strategy for the atmosphere.
\textit{Tellus A}, \textbf{61~(1)}, 97--111,
\doi{10.1111/j.1600-0870.2008.00372.x}.
\bibitem[{Burgers et~al.(1998)Burgers, Jan~van Leeuwen,, and
Evensen}]{burgers1998analysis}
Burgers, G., P.~Jan~van Leeuwen, and G.~Evensen, 1998: Analysis scheme in the
ensemble kalman filter. \textit{Monthly weather review}, \textbf{126~(6)},
1719--1724.
\bibitem[{Campbell et~al.(2010)Campbell, Bishop,, and Hodyss}]{campbell2010}
Campbell, W.~F., C.~H. Bishop, and D.~Hodyss, 2010: Vertical covariance
localization for satellite radiances in ensemble kalman filters.
\textit{Monthly Weather Review}, \textbf{138~(1)}, 282--290,
\doi{10.1175/2009MWR3017.1}.
\bibitem[{Evensen(1994)}]{evensen1994sequential}
Evensen, G., 1994: Sequential data assimilation with a nonlinear
quasi-geostrophic model using monte carlo methods to forecast error
statistics. \textit{Journal of Geophysical Research: Oceans},
\textbf{99~(C5)}, 10\,143--10\,162.
\bibitem[{Evensen(2003)}]{evensen2003ensemble}
Evensen, G., 2003: The ensemble kalman filter: Theoretical formulation and
practical implementation. \textit{Ocean dynamics}, \textbf{53~(4)}, 343--367.
\bibitem[{Evensen(2004)}]{evensen2004sampling}
Evensen, G., 2004: Sampling strategies and square root analysis schemes for the
enkf. \textit{Ocean dynamics}, \textbf{54~(6)}, 539--560.
\bibitem[{Foygel and Drton(2010)Foygel, and Drton}]{foygel2010extended}
Foygel, R., and M.~Drton, 2010: Extended bayesian information criteria for
gaussian graphical models. \textit{Advances in neural information processing
systems}, 604--612.
\bibitem[{Frei and K{\"u}nsch(2013{\natexlab{a}})Frei, and
K{\"u}nsch}]{frei2013bridging}
Frei, M., and H.~R. K{\"u}nsch, 2013{\natexlab{a}}: Bridging the ensemble
kalman and particle filters. \textit{Biometrika}, \textbf{100~(4)}, 781--800.
\bibitem[{Frei and K{\"u}nsch(2013{\natexlab{b}})Frei, and
K{\"u}nsch}]{frei2013mixture}
Frei, M., and H.~R. K{\"u}nsch, 2013{\natexlab{b}}: Mixture ensemble kalman
filters. \textit{Computational Statistics \& Data Analysis}, \textbf{58},
127--138.
\bibitem[{Friedman et~al.(2008)Friedman, Hastie,, and
Tibshirani}]{friedman2008sparse}
Friedman, J., T.~Hastie, and R.~Tibshirani, 2008: Sparse inverse covariance
estimation with the graphical lasso. \textit{Biostatistics}, \textbf{9~(3)},
432--441.
\bibitem[{Gaspari and Cohn(1999)Gaspari, and Cohn}]{gaspari1999}
Gaspari, G., and S.~E. Cohn, 1999: Construction of correlation functions in two
and three dimensions. \textit{Quarterly Journal of the Royal Meteorological
Society}, \textbf{125~(554)}, 723--757.
\bibitem[{Godinez and Moulton(2012)Godinez, and Moulton}]{Godinez2012}
Godinez, H.~C., and J.~D. Moulton, 2012: An efficient matrix-free algorithm for
the ensemble kalman filter. \textit{Computational Geosciences},
\textbf{16~(3)}, 565--575, \doi{10.1007/s10596-011-9268-9}.
\bibitem[{Greybush et~al.(2011)Greybush, Kalnay, Miyoshi, Ide,, and
Hunt}]{Greybush2011}
Greybush, S.~J., E.~Kalnay, T.~Miyoshi, K.~Ide, and B.~R. Hunt, 2011: Balance
and ensemble kalman filter localization techniques. \textit{Monthly Weather
Review}, \textbf{139~(2)}, 511--522, \doi{10.1175/2010MWR3328.1}.
\bibitem[{Hamill et~al.(2001)Hamill, Whitaker,, and
Snyder}]{hamill2001distance}
Hamill, T.~M., J.~S. Whitaker, and C.~Snyder, 2001: Distance-dependent
filtering of background error covariance estimates in an ensemble kalman
filter. \textit{Monthly Weather Review}, \textbf{129~(11)}, 2776--2790.
\bibitem[{Houtekamer and Mitchell(2001)Houtekamer, and
Mitchell}]{Houtekamer2001sequential}
Houtekamer, P.~L., and H.~L. Mitchell, 2001: A sequential ensemble kalman
filter for atmospheric data assimilation. \textit{Monthly Weather Review},
\textbf{129~(1)}, 123--137.
\bibitem[{Houtekamer et~al.(2009)Houtekamer, Mitchell,, and
Deng}]{Houtekamer2009}
Houtekamer, P.~L., H.~L. Mitchell, and X.~Deng, 2009: Model error
representation in an operational ensemble kalman filter. \textit{Monthly
Weather Review}, \textbf{137~(7)}, 2126--2143, \doi{10.1175/2008MWR2737.1}.
\bibitem[{Houtekamer et~al.(2005)Houtekamer, Mitchell, Pellerin, Buehner,
Charron, Spacek,, and Hansen}]{Houtekamer2005}
Houtekamer, P.~L., H.~L. Mitchell, G.~Pellerin, M.~Buehner, M.~Charron,
L.~Spacek, and B.~Hansen, 2005: Atmospheric data assimilation with an
ensemble kalman filter: Results with real observations. \textit{Monthly
Weather Review}, \textbf{133~(3)}, 604--620, \doi{10.1175/MWR-2864.1}.
\bibitem[{Hsieh et~al.(2013)Hsieh, Sustik, Dhillon, Ravikumar,, and
Poldrack}]{hsieh2013big}
Hsieh, C.-J., M.~A. Sustik, I.~S. Dhillon, P.~K. Ravikumar, and R.~Poldrack,
2013: Big \& quic: Sparse inverse covariance estimation for a million
variables. \textit{Advances in Neural Information Processing Systems},
3165--3173.
\bibitem[{Hunt et~al.(2007)Hunt, Kostelich,, and Szunyogh}]{hunt2007efficient}
Hunt, B.~R., E.~J. Kostelich, and I.~Szunyogh, 2007: Efficient data
assimilation for spatiotemporal chaos: a local ensemble transform kalman
filter. \textit{Physica D: Nonlinear Phenomena}, \textbf{230~(1-2)},
112--126.
\bibitem[{Jankov{\'a} et~al.(2015)Jankov{\'a}, Van De~Geer
et~al.}]{jankova2015confidence}
Jankov{\'a}, J., S.~Van De~Geer, and Coauthors, 2015: Confidence intervals for
high-dimensional inverse covariance estimation. \textit{Electronic Journal of
Statistics}, \textbf{9~(1)}, 1205--1229.
\bibitem[{Johnstone(2001)}]{johnstone2001}
Johnstone, I.~M., 2001: On the distribution of the largest eigenvalue in
principal components analysis. \textit{The Annals of Statistics},
\textbf{29~(2)}, 295--327.
\bibitem[{Johnstone and Yu~Lu(2004)Johnstone, and Yu~Lu}]{johnstone2004}
Johnstone, I.~M., and A.~Yu~Lu, 2004: Sparse principle component analysis.
\textit{Unpublished Manuscript}.
\bibitem[{Julier and Uhlmann(1997)Julier, and Uhlmann}]{julier1997new}
Julier, S.~J., and J.~K. Uhlmann, 1997: New extension of the kalman filter to
nonlinear systems. \textit{AeroSense'97}, International Society for Optics
and Photonics, 182--193.
\bibitem[{Lei and Bickel(2011)Lei, and Bickel}]{lei2011}
Lei, J., and P.~Bickel, 2011: A moment matching ensemble filter for nonlinear
non-gaussian data assimilation. \textit{Monthly Weather Review},
\textbf{139~(12)}, 3964--3973.
\bibitem[{Li et~al.(2009)Li, Kalnay,, and Miyoshi}]{li2009}
Li, H., E.~Kalnay, and T.~Miyoshi, 2009: Simultaneous estimation of covariance
inflation and observation errors within an ensemble kalman filter.
\textit{Quarterly Journal of the Royal Meteorological Society},
\textbf{135~(639)}, 523--533, \doi{10.1002/qj.371}.
\bibitem[{Lv and Liu(2014)Lv, and Liu}]{lv2014model}
Lv, J., and J.~S. Liu, 2014: Model selection principles in misspecified models.
\textit{Journal of the Royal Statistical Society: Series B (Statistical
Methodology)}, \textbf{76~(1)}, 141--167.
\bibitem[{Miyoshi(2011)}]{Miyoshi2011}
Miyoshi, T., 2011: The gaussian approach to adaptive covariance inflation and
its implementation with the local ensemble transform kalman filter.
\textit{Monthly Weather Review}, \textbf{139~(5)}, 1519--1535,
\doi{10.1175/2010MWR3570.1}.
\bibitem[{Nakano(2014)}]{Nakano}
Nakano, S., 2014: Hybrid algorithm of ensemble transform and importance
sampling for assimilation of non-gaussian observations. \textit{Tellus A},
\textbf{66~(0)}.
\bibitem[{Nerger et~al.(2012)Nerger, Janjić, Schröter,, and
Hiller}]{Nerger2012}
Nerger, L., T.~Janjić, J.~Schröter, and W.~Hiller, 2012: A unification of
ensemble square root kalman filters. \textit{Monthly Weather Review},
\textbf{140~(7)}, 2335--2345, \doi{10.1175/MWR-D-11-00102.1}.
\bibitem[{Nino-Ruiz et~al.(2015)Nino-Ruiz, Sandu,, and
Deng}]{Nino-Ruiz:2015:PEK:2832080.2832084}
Nino-Ruiz, E.~D., A.~Sandu, and X.~Deng, 2015: A parallel ensemble kalman
filter implementation based on modified cholesky decomposition.
\textit{Proceedings of the 6th Workshop on Latest Advances in Scalable
Algorithms for Large-Scale Systems}, 4:1--4:8, ScalA '15,
\doi{10.1145/2832080.2832084}.
\bibitem[{Ott et~al.(2004)}]{ott2004local}
Ott, E., and Coauthors, 2004: A local ensemble kalman filter for atmospheric
data assimilation. \textit{Tellus A}, \textbf{56~(5)}, 415--428.
\bibitem[{Papadakis et~al.(2010)Papadakis, M{\'e}min, Cuzol,, and
Gengembre}]{Papadakis}
Papadakis, N., E.~M{\'e}min, A.~Cuzol, and N.~Gengembre, 2010: Data
assimilation with the weighted ensemble kalman filter. \textit{Tellus A},
\textbf{62~(5)}, 673--697, \doi{10.1111/j.1600-0870.2010.00461.x}.
\bibitem[{Ravikumar et~al.(2011)Ravikumar, Wainwright, Raskutti, Yu
et~al.}]{ravikumar2011high}
Ravikumar, P., M.~J. Wainwright, G.~Raskutti, B.~Yu, and Coauthors, 2011:
High-dimensional covariance estimation by minimizing ℓ1-penalized
log-determinant divergence. \textit{Electronic Journal of Statistics},
\textbf{5}, 935--980.
\bibitem[{Robert(2014)}]{Rpackage}
Robert, S., 2014: \textit{modifiedSWEQ: Simplified cumulus convection with the
modified SWEQ}. R package version 0.1,
\url{https://github.com/robertsy/modifiedSWEQ}.
\bibitem[{{Robert} and {K{\"u}nsch}(2016){Robert}, and {K{\"u}nsch}}]{Robert}
{Robert}, S., and H.~R. {K{\"u}nsch}, 2016: {Local Ensemble Kalman Particle
Filters for efficient data assimilation}. \textit{ArXiv e-prints},
\eprint{1605.05476}.
\bibitem[{Schwarz(1978)}]{schwarz1978}
Schwarz, G., 1978: Estimating the dimension of a model. \textit{Ann. Statist.},
\textbf{6~(2)}, 461--464, \doi{10.1214/aos/1176344136}.
\bibitem[{Snyder et~al.(2008)Snyder, Bengtsson, Bickel,, and Anderson}]{snyder}
Snyder, C., T.~Bengtsson, P.~Bickel, and J.~Anderson, 2008: Obstacles to
high-dimensional particle filtering. \textit{Monthly Weather Review},
\textbf{136~(12)}, 4629--4640, \doi{10.1175/2008MWR2529.1}.
\bibitem[{Tippett et~al.(2003)Tippett, Anderson, Bishop, Hamill,, and
Whitaker}]{Tippett2003}
Tippett, M.~K., J.~L. Anderson, C.~H. Bishop, T.~M. Hamill, and J.~S. Whitaker,
2003: Ensemble square root filters. \textit{Monthly Weather Review},
\textbf{131~(7)}, 1485--1490,
\doi{10.1175/1520-0493(2003)131<1485:ESRF>2.0.CO;2}.
\bibitem[{T{\"o}dter and Ahrens(2015)T{\"o}dter, and Ahrens}]{Todter2015}
T{\"o}dter, J., and B.~Ahrens, 2015: A second-order exact ensemble square root
filter for nonlinear data assimilation. \textit{Monthly Weather Review},
\textbf{143~(4)}, 1347--1367, \doi{10.1175/MWR-D-14-00108.1}.
\bibitem[{Ueno and Tsuchiya(2009)Ueno, and Tsuchiya}]{ueno2009covariance}
Ueno, G., and T.~Tsuchiya, 2009: Covariance regularization in inverse space.
\textit{Quarterly Journal of the Royal Meteorological Society},
\textbf{135~(642)}, 1133--1156.
\bibitem[{van Leeuwen(2010)}]{vanLeeuwen}
van Leeuwen, P.~J., 2010: Nonlinear data assimilation in geosciences: an
extremely efficient particle filter. \textit{Quarterly Journal of the Royal
Meteorological Society}, \textbf{136~(653)}, 1991--1999,
\doi{10.1002/qj.699}.
\bibitem[{Vershynin(2012)}]{vershynin2012close}
Vershynin, R., 2012: How close is the sample covariance matrix to the actual
covariance matrix? \textit{Journal of Theoretical Probability},
\textbf{25~(3)}, 655--686.
\bibitem[{Wan and Van Der~Merwe(2000)Wan, and Van Der~Merwe}]{wan2000unscented}
Wan, E.~A., and R.~Van Der~Merwe, 2000: The unscented kalman filter for
nonlinear estimation. \textit{Adaptive Systems for Signal Processing,
Communications, and Control Symposium 2000. AS-SPCC. The IEEE 2000}, Ieee,
153--158.
\bibitem[{Wang et~al.(2007)Wang, Hamill, Whitaker,, and Bishop}]{wang2007}
Wang, X., T.~M. Hamill, J.~S. Whitaker, and C.~H. Bishop, 2007: A comparison of
hybrid ensemble transform kalman filter–optimum interpolation and ensemble
square root filter analysis schemes. \textit{Monthly Weather Review},
\textbf{135~(3)}, 1055--1076, \doi{10.1175/MWR3307.1}.
\bibitem[{Whitaker and Hamill(2002)Whitaker, and Hamill}]{Whitaker2002}
Whitaker, J.~S., and T.~M. Hamill, 2002: Ensemble data assimilation without
perturbed observations. \textit{Monthly Weather Review}, \textbf{130~(7)},
1913--1924, \doi{10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2}.
\bibitem[{W{\"u}rsch and Craig(2014)W{\"u}rsch, and Craig}]{würsch2014}
W{\"u}rsch, M., and G.~C. Craig, 2014: A simple dynamical model of cumulus
convection for data assimilation research. \textit{Meteorologische
Zeitschrift}, \textbf{23~(5)}, 483--490, \doi{10.1127/0941-2948/2014/0492}.
\end{thebibliography}