Estimating Time-Varying Graphical Models
Extracted source text · arXiv 1804.03811
\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{graphicx,psfrag,epsf}
\usepackage{enumerate}
\usepackage{natbib}
\usepackage{url} % not crucial - just used below for the URL
%\pdfminorversion=4
% NOTE: To produce blinded version, replace "0" with "1" below.
\newcommand{\blind}{0}
% DON'T change margins - should be 1 inch all around.
\addtolength{\oddsidemargin}{-.5in}%
\addtolength{\evensidemargin}{-.5in}%
\addtolength{\textwidth}{1in}%
\addtolength{\textheight}{1.3in}%
\addtolength{\topmargin}{-.8in}%
\usepackage{graphics}
\usepackage{mathbbol}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage{bm}
\usepackage{color}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{subfig}
\usepackage{float}
\usepackage{comment}
\usepackage{srctex}
\newtheorem{lemma}{Lemma}
\newtheorem{assumption}{Assumption}
\newtheorem{theorem}{Theorem}
\newtheorem{proposition}{Proposition}
\newtheorem{corollary}{Corollary}
\newtheorem{remark}{Remark}
\definecolor{JP}{RGB}{255,0,0}
\newcommand{\JP}[1]{\textcolor{JP}{[J: #1]}}
\definecolor{JY}{RGB}{255,0,0}
\newcommand{\JY}[1]{\textcolor{JY}{[Y: #1]}}
\graphicspath{{images/}}
\pdfminorversion=4
\pdfobjcompresslevel=0
\begin{document}
\def\spacingset#1{\renewcommand{\baselinestretch}%
{#1}\small\normalsize} \spacingset{1}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\if0\blind
{
\title{\bf Estimating Time-Varying Graphical Models}
\author{Jilei Yang \thanks{The authors gratefully acknowledge the following support:
UCD Dissertation Year Fellowship (JLY), NIH 1R01EB021707 (JLY and JP) and NSF-DMS-1148643 (JP).
}\\
Department of Statistics, University of California, Davis\\
and \\
Jie Peng\footnote{Correspondence author: jiepeng@ucdavis.edu}\\
Department of Statistics, University of California, Davis}
\maketitle
} \fi
\if1\blind
{
\bigskip
\bigskip
\bigskip
\begin{center}
{\LARGE\bf Estimating Time-Varying Graphical Models}
\end{center}
\medskip
} \fi
\bigskip
\begin{abstract}
In this paper, we study time-varying graphical models based on data measured over a temporal grid.
%under smoothness assumptions on both the covariance matrix (referred to as parameter smoothness) and graph structure (referred to as structural smoothness).
Such models are motivated by the needs to describe and understand evolving interacting relationships among a set of random variables in many real applications, for instance the study of how stocks interact with each other and how such interactions change over time.
%Most methods for Gaussian graphical models assume i.i.d. observations. However, when data are measured over time or a spatial grid, the conditional independence relationships may evolve over time or location such that the observations are not identically distributed. In order to learn time varying graphs, it often assumed that the covariance matrix is smoothly changing over time (e.g., \cite{zhou2010time}). Moreover, in \cite{wang2014inference}, it is assumed that the graph structure is time-invariant.
%However, the former method does not utilize potential structural smoothness of the graphs and the latter method assumes a time-invariant graph structure which is often too restrictive in practice.
We propose a new model, \textit{LOcal Group Graphical Lasso Estimation} (\texttt{loggle}), under the assumption that the graph topology changes gradually over time. Specifically, \texttt{loggle} uses a novel local group-lasso type penalty to efficiently incorporate information from neighboring time points and to impose structural smoothness of the graphs. We implement an ADMM based algorithm to fit the \texttt{loggle} model. This algorithm utilizes blockwise fast computation and pseudo-likelihood approximation to improve computational efficiency. An R package \texttt{loggle} has also been developed.
% and use cross validation for tuning parameters selection.
%Moreover, a pseudo-likelihood approximation is proposed, where local group-lasso regression is performed separately for each variable on the rest of variables.
We evaluate the performance of \texttt{loggle} by simulation experiments. We also apply \texttt{loggle} to S\&P 500 stock price data and demonstrate that \texttt{loggle} is able to reveal the interacting relationships among stocks and among industrial sectors in a time period that covers the recent global financial crisis.
\end{abstract}
\noindent%
{\it Keywords:} ADMM algorithm, Gaussian graphical model, group-lasso, pseudo-likelihood approximation, S\&P 500
\vfill
\newpage
\spacingset{1.45} % DON'T change the spacing!
\section{Introduction}
In recent years, there are many problems where the study of the interacting relationships among a large number of variables is of interest. One popular approach is to characterize interactions as conditional dependencies: Two variables are interacting with each other if they are conditionally dependent given the rest of the variables. An advantage of using conditional dependency instead of marginal dependency (e.g., through correlation) is that we are aiming for more direct interactions after taking out the effects of the rest of the variables. Moreover, if the random variables follow a multivariate normal distribution, then the elements of the inverse covariance matrix $\bm{\Sigma}^{-1}$ (a.k.a. precision matrix) would indicate the presence/absence of such interactions. This is because under normality, two variables are conditionally dependent given the rest of the variables if and only if the corresponding element of the precision matrix is nonzero. Furthermore, we can represent such interactions by a graph $G=(V,E)$, where the node set $V$ represents the random variables of interest and the edge set $E$ consists of pairs $\{i,j\}$ where the $(i,j)$th element of $\bm{\Sigma}^{-1}$ is nonzero. Such models are referred to as \textit{Gaussian graphical models (GGM)}.
%such as estimating gene regulatory network based on high throughput genomics data, or the studying the relationships a set of stocks based on the price profiles \JP{cite}.
%In many of these problems, for $p$ variables of interest, we have $n$ observations $\bm{x}_1,\cdots,\bm{x}_n\in\mathbb{R}^p$ assumed to be independent and identically distributed with multinomial Gaussian distribution $\mathcal{N}_p(\bm{\mu},\bm{\Sigma})$, where $\bm{\mu}\in\mathbb{R}^p$ and $\bm{\Sigma}$ is a positive definite $p\times p$ matrix. To investigate the interactions among these $p$ variables, one approach is to focus on the inverse covariance matrix (also known as precision matrix) $\bm{\Sigma}^{-1}$, where each zero in $\bm{\Sigma}^{-1}$, say in row $i$ and column $j$, indicates the conditional independence between the $i$th and the $j$th variables given the rest of the variables. In other words, these two variables have no direct interaction between each other if $(\bm{\Sigma}^{-1})_{ij}=0$. In a Gaussian graphical model, the relationships among the variables can be represented by an undirected graph $G=(V,E)$, where the node set $V$ represents the $p$ variables, and the edge set $E$ consists of pairs $(i,j)$ where the corresponding $i$th and $j$th variables are conditionally dependent.\\
Many methods have been proposed to learn GGMs when the number of variables is large (relative to the sample size), including \cite{meinshausen2006high}, \cite{yuan2007model}, \cite{friedman2008sparse}, \cite{banerjee2008model}, \cite{rothman2008sparse}, \cite{peng2009partial}, \cite{lam2009sparsistency}, \cite{ravikumar2011high}, \cite{cai2011constrained}. These methods rely on the sparsity assumption, i.e., only a small subset of elements in the precision matrix is nonzero, to deal with challenges posed by high-dimension-low-sample-size.
%One trivial way is to calculate the inverse of the sample covariance matrix $\hat{\bm{\Sigma}}$. However, in contemporary applications the number of variables $p$ is usually larger than the sample size $n$, which means $\hat{\bm{\Sigma}}$ is a poor estimate of the covariance matrix $\bm{\Sigma}$ and is even not invertible.\\
%In many practical problems it is believed that only a few interactions between variables are significant, which leads to the sparsity assumption of the precision matrix, i.e., only a few elements in the precision matrix are nonzero. Based on this assumption, one popular method for estimating precision matrix called \texttt{graphical lasso} (\cite{friedman2008sparse}) is proposed. In \texttt{graphical lasso}, we use the minimizer of the penalized (negative) log-likelihood $\min_{\bm{\Omega}\succ0}\left\{\text{tr}(\hat{\bm{\Sigma}}\bm{\Omega})-\log\det\bm{\Omega}+\lambda||\bm{\Omega}||_1\right\}$ as an estimate of the precision matrix $\bm{\Sigma}^{-1}$, where $\lambda\geq0$ is the tuning parameter, and $||\bm{\Omega}||_1$ is the $l_1$ norm, i.e., the sum of the absolute values of the elements in $\bm{\Omega}$. The $l_1$ penalty leads to the sparsity of the estimated precision matrix when $\lambda$ is large. Consequently, the corresponding graph is easy to interpret since it contains only a few edges. Moreover, it can handle the cases when $p\gg n$ due to its intrinsically low degrees of freedom. More methods on estimating the precision matrix include likelihood-based methods \cite{yuan2007model}, \cite{banerjee2008model}, \cite{rothman2008sparse}, \cite{lam2009sparsistency}, \cite{ravikumar2011high}, and pseudo-likelihood-based methods \cite{meinshausen2006high}, \cite{peng2009partial}, \cite{friedman2010applications}, \cite{cai2011constrained}.\\
The aforementioned methods learn a single graph based on the observed data. However, when data are observed over a temporal or spatial grid, the underlying graphs might change over time/space. For example, the relationships among stocks could evolve over time as illustrated by Figure \ref{fig:graph_loggle}. If we had described them by a single graph, the results would be misleading. This necessitates the
study of time-varying graphical models.
When the graphs/covariance matrices change over time, the observations would not be identically distributed anymore. To deal with this challenge, one approach is to assume the covariance matrices change smoothly over time. For example, in \cite{zhou2010time}, \cite{song2009keller}, \cite{kolar2010estimating}, \cite{kolar2012estimating}, \cite{wang2014inference}, \cite{monti2014estimating}, \cite{gibberd2014high}, \cite{gibberd2017regularized}, kernel estimates of the covariance matrices are used in the objective functions. However, smoothness of the covariance matrix alone does not tell us how the graph topology would evolve over time, despite in
practice this is often of more interests than estimating the covariance matrices. Moreover, imposing certain assumption on how the graph topology changes over time could greatly facilitate interpretation and consequently provide insights about the interacting relationships.
%Recently, \cite{zhou2010time} and \cite{wang2014inference} proposed methods to deal with the time-varying graphical model, under the assumption that the observed data are independent and normally distributed with the covariance matrix changes smoothly over time. To incorporate this parameter smoothness assumption, both Zhou's and Wang's methods utilize the kernel estimate of the covariance matrix in their likelihood-based objective functions. Kernel method in time-varying graph estimation is also proposed and implemented in \cite{song2009keller}, \cite{kolar2010estimating}, \cite{kolar2012estimating}, \cite{monti2014estimating}, \cite{gibberd2014high}, \cite{gibberd2017regularized}. However, using the kernel method only does not take the smoothness assumption on graph structure into consideration. The structural smoothness refers to the consistency of presence/absence of graph edges across the time. It is easy to see that the smoothness in graph structure cannot be guaranteed by the smoothness in covariance matrix. In practice, structure smoothness is usually even more important than parameter smoothness, since a smoothly-changing graph structure can greatly facilitate people with domain knowledge to understand, interpret and get insights from the model. Here in Zhou's method, the structural smoothness assumption is totally ignored, while in Wang's method, this assumption is overemphasized and make the estimated graph structure identical across time. As a result, both Zhou's and Wang's methods have some drawbacks in real world applications.\\
One type of time-varying graphical models utilize fused-lasso type penalties \citep{ahmed2009recovering, kolar2010estimating, kolar2012estimating, monti2014estimating, gibberd2014high, wit2015inferring, gibberd2017regularized, hallac2017network}, such that the estimated graph topology would be piecewise constant. This is particularly convenient when we are primarily interested in detecting jump points and abrupt changes.
%There also exist some methods which can construct graphs for data obtained from different categories that share some common structure, for example, in \cite{guo2011joint}, \cite{chiquet2011inferring}, \cite{danaher2014joint}, \cite{mohan2014node}, \cite{yang2015fused}, \cite{tao2016multiple}. Despite their good performance in estimating separate graphs for each category, it is hard for them to be extended to the more challenging time grid case where the time points are ordered and on each time point there exists only one observation.\\
In this paper, we consider the \textit{structural smoothness} assumption, which assumes that the graph topology is gradually changing over time. For this purpose, we propose \textit{LOcal Group Graphical Lasso Estimation} (\texttt{loggle}), a novel time-varying graphical model that imposes structural smoothness through a local group-lasso type penalty.
%\texttt{loggle} provides a flexible and data driven way to capture structural smoothness of the graphs without imposing strong assumptions (such as piecewise constancy).
The \texttt{loggle} method is able to efficiently utilize neighborhood information and is also able to adapt to the local degree of smoothness in a data driven fashion. Consequently, the \texttt{loggle} method is flexible and effective for a wide range of scenarios including models with both time-varying and time-invariant graphs. Moreover, we implement an ADMM based algorithm that utilizes blockwise fast computation and pseudo-likelihood approximation to achieve computational efficiency and we use cross-validation to select tuning parameters. We demonstrate the performance of \texttt{loggle} through a simulation study. Finally, we apply \texttt{loggle} to the S\&P 500 stock price data to reveal how interactions among stocks and among industrial sectors evolve during the recent global financial crisis. An R package \texttt{loggle} has also been developed.
%Moreover, \texttt{Loggle} encompasses two previous methods, namely, \cite{zhou2010time} and \cite{wang2014inference} \JP{check updated reference}, as special cases.
%In \cite{zhou2010time}, structural smoothness is being ignored and one graph is independently estimated for each time point. At another extreme, \cite{wang2014inference} \JP{check updated reference} assumes that the graph structure is time-invariant, making it impossible to capture potential evolving patterns of the interactions.
The rest of the paper is organized as follows. In Section \ref{sec:method}, we introduce the \texttt{loggle} model, model fitting algorithms and strategies for model tuning. In Section \ref{sec:simulation}, we present simulation results to demonstrate the performance of \texttt{loggle} and compare it with two existing methods. We report the application on S\&P 500 stock price data in Section \ref{sec:real_data}, followed by conclusions in Section \ref{sec:discussion}. Technical details are in an Appendix. Additional details are deferred to a Supplementary Material.
\section{Methods}\label{sec:method}
\subsection{Local Group Graphical Lasso Estimation}\label{subsec:loggle}
In this section, we introduce \texttt{loggle} \textit{(LOcal Group Graphical Lasso Estimation)} for time-varying graphical models.
%\subsubsection{Preliminaries}
Let $\bm{X}(t)=(X^1(t),\ldots,X^p(t))^T\sim \mathcal{N}_p(\bm{\mu}(t),\bm{\Sigma}(t))$ be a $p$-dimensional Gaussian random vector indexed by $t \in [0,1]$.
We assume $\bm{X}(t)$'s are independent across $t$. We also assume the mean function $\bm{\mu}(t)$ and the covariance function $\bm{\Sigma}(t)$ are smooth in $t$. We denote the observations by $\{\bm{x}_k\}_{k \in \mathcal{I}}$ where $\mathcal{I}=\{1,\cdots, N\}$, $\bm{x}_k$ is a realization of $\bm{X}({t_k})$ ($k \in \mathcal{I}$) and $0\leq t_1\leq\ldots\leq t_N\leq1$.
For simplicity, we assume that the observations are centered so that $\bm{x}_k$ is drawn from $\mathcal{N}_p(\bm{0},\bm{\Sigma}(t_k))$. In practice, we can achieve this by subtracting the estimated mean $\hat{\bm{\mu}}(t_k)$ from $\bm{x}_k$. See Section \ref{sec:detrend} of the Supplementary Material for details.
Our goal is to estimate the precision matrix $\bm{\Omega}(t):=\bm{\Sigma}^{-1}(t)$ based on the observed data $\{\bm{x}_k\}_{k \in \mathcal{I}}$ and then construct the edge set (equiv. the graph topology) $E(t)$ based on the sparsity pattern of the estimated precision matrix $\hat{\bm{\Omega}}(t)$.
We further assume that the edge set (equiv. the graph topology) changes gradually over time.
%\subsubsection{Model}
To estimate the precision matrix $\bm{\Omega}(t_k)$ at the $k$th observed time point, we propose to minimize a locally weighted negative log-likelihood function with a local group-lasso penalty:
\begin{equation}
\label{eq:loggle}
L(\mathbb{\Omega}_k):=\frac{1}{\sqrt{|\mathcal{N}_{k,d}|}}\sum_{i\in \mathcal{N}_{k,d}}\left[\text{tr}\left(\bm{\Omega}(t_i)\hat{\bm{\Sigma}}(t_i)\right)-\log|\bm{\Omega}(t_i)|\right]
+\lambda\sum_{u\neq v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\Omega_{uv}(t_i)^2},
\end{equation}
where $\mathcal{N}_{k,d}=\{i \in \mathcal{I}:|t_i-t_k|\leq d\}$ denotes the indices of the time points centered around $t_k$ with neighborhood width $d$ and $|\mathcal{N}_{k,d}|$ denotes the number of elements in $\mathcal{N}_{k,d}$; $\mathbb{\Omega}_k=\{\bm{\Omega}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ denotes the set of precision matrices within this neighborhood and $\Omega_{uv}(t_i)$ denotes the $(u,v)$-th element of $\bm{\Omega}(t_i)$; $\hat{\bm{\Sigma}}(t)=\sum_{j=1}^N\omega_h^{t_j}(t)\bm{x}_j\bm{x}_j^T$ is the kernel estimate of the covariance matrix at time $t$, where the weights $\omega_h^{t_j}(t)=\frac{K_h(t_j-t)}{\sum_{j=1}^NK_h(t_j-t)}$, $K_h(\cdot)=K(\cdot/h)$ is a symmetric nonnegative kernel function and $h(>0)$ is the bandwidth.
We obtain:
$$\hat{\mathbb{\Omega}}_k=\{\hat{\bm{\Omega}}(t_i)\}_{i\in \mathcal{N}_{k,d}}=\arg\min_{\bm{\Omega}(t_i)\succ\bm{0},\;i\in \mathcal{N}_{k,d}} L(\mathbb{\Omega}_k),$$
and set $\hat{\bm{\Omega}}(t_k)$ as the estimated precision matrix at time $t_k$.
%We apply this procedure to each $t_k (k=1,\ldots,N)$ and obtain the sequence of estimated precision matrices $\{\hat{\bm{\Omega}}(t_k)\}_{k=1,\ldots,N}$.
Since the group-lasso type penalty is likely to over-shrink the elements in the precision matrix, we further perform model refitting by maximizing the weighted log-likelihood function under the constraint of the estimated edge set (equiv. sparsity pattern). We denote the refitted estimate by
$\hat{\bm{\Omega}}^\text{rf}(t_k)$.
Note that, the precision matrix may be estimated at any time point $t\in[0,1]$: If $t \notin \{t_k: k \in \mathcal{I}\}$, then choose an integer $\tilde{k} \notin \mathcal{I}$ and define $t_{\tilde{k}}=t$, $\mathcal{\tilde{I}}=\mathcal{I} \cup \{\tilde{k}\}$ and $\mathcal{N}_{\tilde{k},d}=\{i \in \mathcal{\tilde{I}}:|t_i-t_{\tilde{k}}|\leq d\}$. For simplicity of exposition, throughout we describe the \texttt{loggle} fits at observed time points.
%In practice, we would often fit the \texttt{loggle} model on a subset of the observed time points.
The use of the kernel estimate $\hat{\bm{\Sigma}}(t)$
%in place of the sample covariance matrix as in conventional GGMs such as \texttt{glasso} \citep{friedman2008sparse}
is justified by the assumption that the covariance matrix $\bm{\Sigma}(t)$ is smooth in $t$. This allows us to borrow information from neighboring time points. In practice, we often replace the kernel smoothed covariance matrices by kernel smoothed correlation matrices which amounts to data standardization.
The penalty term $\lambda\sum_{u\neq v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\Omega_{uv}(t_i)^2}$ is a group-lasso type sparse regularizer \citep{yuan2006model, danaher2014joint} that makes the graph topology change smoothly over time. The degree of such smoothness is controlled by the tuning parameter $d (>0)$, the larger the neighborhood width $d$, the more gradually the graph topology would change. The overall sparsity of the graphs is controlled by the tuning parameter $\lambda (>0)$, the larger the $\lambda$, the sparser the graphs tend to be. The factor $\frac{1}{\sqrt{|\mathcal{N}_{k,d}|}}$ in equation (\ref{eq:loggle}) is to make $\lambda$ comparable for different $d$.
The \texttt{loggle} model includes two existing time-varying graphical models as special cases. Specifically, in \cite{zhou2010time}, $\bm{\Omega}(t_k)$ is estimated by minimizing a weighted negative log-likelihood function with a lasso penalty:
$$\min_{\bm{\Omega}(t_k)\succ\bm{0}}\text{tr}\left(\bm{\Omega}(t_k)\hat{\bm{\Sigma}}(t_k)\right)-\log|\bm{\Omega}(t_k)|+\lambda\sum_{u\neq v}|\Omega_{uv}(t_k)|,$$ which is a special case of \texttt{loggle} by setting $d=0$. This method utilizes the smoothness of the covariance matrix by introducing the kernel estimate $\hat{\bm{\Sigma}}(t)$ in the likelihood function. However, it ignores potential structural smoothness of the graphs and thus might not utilize the data most efficiently. Hereafter, we refer to this method as \texttt{kernel}.
%Although Zhou's method utilizes the smoothness of the parameters in covariance matrix by introducing the kernel estimate $\hat{\bm{\Sigma}}(t)$, it ignores another important fact that the change in the graphical structure across time is also likely to be smooth.
On the other hand, \cite{wang2014inference} propose to use a (global) group-lasso penalty to estimate $\bm{\Omega}(t_k)$'s simultaneously:
$$\min_{\{\bm{\Omega}(t_k)\succ\bm{0}\}_{k=1,\ldots,N}}\sum_{k=1}^N\left[\text{tr}\left(\bm{\Omega}(t_k)\hat{\bm{\Sigma}}(t_k)\right)-\log|\bm{\Omega}(t_k)|\right]
+\lambda\sum_{u\neq v}\sqrt{\sum_{k=1}^N\Omega_{uv}(t_k)^2}.$$ This is another special case of \texttt{loggle} by setting $d$ large enough to cover the entire time interval $[0, 1]$ (e.g., $d=1$). The (global) group-lasso penalty makes the estimated precision matrices have the same sparsity pattern (equiv. same graph topology) across the entire time domain. This could be too restrictive for many applications where the graph topology is expected to change over time. Hereafter, we refer to this method as \texttt{invar}.
%In \texttt{loggle}, we utilize not only the smoothness assumption on the covariance matrix as Zhou's and Wang's methods did, but also incorporate the smoothness assumption (not as extreme as time-invariance assumption as in Wang's method) on the graphical structure, thus to overcome the limitations in both Zhou's and Wang's methods.
\subsection{Model Fitting}\label{subsec:algorithm}
Minimizing the objective function in equation (\ref{eq:loggle}) with respect to $\mathbb{\Omega}_k$ is a convex optimization problem. This can be solved by an ADMM (\textit{alternating directions method of multipliers}) algorithm (See details in Section \ref{sec:ADMM} of the Supplementary Material). ADMM algorithms can converge to the global optimum for convex optimization problems under very mild conditions. A comprehensive introduction can be found in \cite{boyd2011distributed}. However, this ADMM algorithm involves $|\mathcal{N}_{k,d}|$ eigen-decompositions of $p\times p$ matrices (each corresponding to a time point in the neighborhood) in every iteration, which is computationally very expensive when $p$ is large. In the following, we propose a fast blockwise algorithm as well as pseudo-likelihood type approximation of the objective function to speed up the computation.
%In addition, the problem of model refitting can be directly solved by using the function \texttt{glasso} in the ``\texttt{glasso}" package (\cite{friedman2008sparse}), where the edge set obtained from the ADMM algorithm will be one of the inputs to this function.
%\subsubsection*{Fast blockwise computations}\label{sec:fast_computation}
\subsubsection*{Fast blockwise algorithm}
If the solution is block diagonal (after suitable permutation of the variables), then we can apply the ADMM algorithm to each block separately, and consequently reduce the computational complexity from $O(p^3)$ to $\sum_{l=1}^LO(p_l^3)$, where $p_l$'s are the block sizes and $\sum_{l=1}^Lp_l=p$.
We establish the following theorems when there are two blocks. These results follow similar results in \cite{witten2011new} and \cite{danaher2014joint} and can be easily extended to an arbitrary number of blocks.
\begin{theorem}\label{thm:block_diag}
Suppose the solution of minimization of (\ref{eq:loggle}) with respect to $\mathbb{\Omega}_k$ has the following form (after appropriate variable permutation):
$$\hat{\bm{\Omega}}(t_i)=\left(\begin{array}{c c}\hat{\bm{\Omega}}_1(t_i) & \bm{0} \\ \bm{0} &\hat{\bm{\Omega}}_2(t_i)\end{array}\right), ~~~ i\in \mathcal{N}_{k,d}, $$
where all $\hat{\bm{\Omega}}_1(t_i)$'s have the same dimension. Then $\{\hat{\bm{\Omega}}_1(t_i)\}_{i\in \mathcal{N}_{k,d}}$ and $\{\hat{\bm{\Omega}}_2(t_i)\}_{i\in \mathcal{N}_{k,d}}$ can be obtained by minimizing (\ref{eq:loggle}) on the respective sets of variables separately.
\end{theorem}
\begin{theorem}\label{thm:disconnect}
Let $\{G_1,G_2\}$ be a non-overlapping partition of the $p$ variables. A necessary and sufficient condition for the variables in $G_1$ to be completely disconnected from those in $G_2$ in all estimated precision matrices $\{\hat{\bm{\Omega}}(t_i)\}_{i \in \mathcal{N}_{k,d}}$ through minimizing (\ref{eq:loggle}) is:
$$\frac{1}{|\mathcal{N}_{k,d}|}\sum_{i\in \mathcal{N}_{k,d}}\hat{\Sigma}_{uv}(t_i)^2\leq\lambda^2, ~~~~ \text{for all} ~~ u\in G_1,\;v\in G_2.$$
\end{theorem}
The proof of Theorem \ref{thm:block_diag} is straightforward through inspecting the Karush-Kuhn-Tucker (KKT) condition of the optimization problem of minimizing (\ref{eq:loggle}). The proof of Theorem \ref{thm:disconnect} is given in Appendix \ref{sec:disconnect}.
%Theorem \ref{thm:disconnect} gives the necessary and sufficient condition for the presence of block diagonal structure in the estimated precision matrices.
Based on Theorem \ref{thm:disconnect}, we propose the following \textit{fast blockwise ADMM algorithm}:
\begin{enumerate}[(i)]
\item
Create a $p\times p$ adjacency matrix $\bm{A}$. For $1\leq u\neq v\leq p$, set the off-diagonal elements $A_{uv}=0$ if $\frac{1}{|\mathcal{N}_{k,d}|}\sum_{i\in \mathcal{N}_{k,d}}\hat{\Sigma}_{uv}(t_i)^2\leq\lambda^2$; and $A_{uv}=1$, if otherwise.
\item
Identify the connected components, $G_1,\cdots,G_L$, given the adjancency matrix $\bm{A}$. Denote their sizes by $p_1,\cdots,p_L$ ($\sum_{l=1}^Lp_l=p$).
\item For $l=1,\cdots,L$, if $p_l=1$, i.e., $G_l$ contains only one variable, say the $u$th variable, then set $\hat{\Omega}_{uu}(t_i)=(\hat{\Sigma}_{uu}(t_i))^{-1}$ for $i\in \mathcal{N}_{k,d}$; If $p_l>1$, then apply the ADMM algorithm to the $p_l$ variables in $G_l$ to obtain the corresponding $\{\hat{\bm{\Omega}}_l(t_i)\}_{i \in \mathcal{N}_{k,d}}$.
\end{enumerate}
\subsubsection*{Pseudo-likelihood approximation}\label{sec:pseudo_likelihood}
Even with the fast blockwise algorithm, the computational cost could still be high due to the eigen-decompositions. In the following, we propose a \textit{pseudo-likelihood approximation} to speed up step (iii) of this algorithm. In practice, this approximation has been able to reduce computational cost by as much as 90\%. For simplicity of exposition, the description is based on the entire set of the variables.
The proposed approximation is based on the following well known fact that relates the elements of the precision matrix to the coefficients of regressing one variable to the rest of the variables \citep{meinshausen2006high, peng2009partial}. Suppose a random vector $(X^1,\ldots,X^p)^T$ has mean zero and covariance matrix $\bm{\Sigma}$. Denote the precision matrix by $\bm{\Omega}=((\Omega_{uv})):=\bm{\Sigma}^{-1}$. If we write $X^u=\sum_{v\neq u}\beta_{uv}X^v+\epsilon_u$, where the residual $\epsilon_u$ is uncorrelated with $\{X^v:v\neq u\}$, then $\beta_{uv}=-\frac{\Omega_{uv}}{\Omega_{uu}}$.
%$=\rho_{uv}\sqrt{\frac{\Omega_{vv}}{\Omega_{uu}}}$
%, where $\rho_{uv}$ is the partial correlation between $\bm{x}_u$ and $\bm{x}_v$.
Note that, $\beta_{uv}=0$ if and only if $\Omega_{uv}=0$. Therefore identifying the sparsity pattern of the precision matrix is equivalent to identifying sparsity pattern of the regression coefficients.
%i.e., $u\nsim v$ in the graph induced by $\bm{\Omega}$.
%\begin{theorem}\label{thm:pseudo}
%Consider the $p$ linear equations $\bm{x}_u=\sum_{v\neq u}\beta_{uv}\bm{x}_v+\bm{\epsilon}_u$. if %and only if $\beta_{uv}=-\frac{\Omega_{uv}}{\Omega_{uu}}=\rho_{uv}\sqrt{\frac{\Omega_{vv}}{\Omega_{uu}}}$, where $\rho_{uv}$ is the partial correlation between $\bm{x}_u$ and $\bm{x}_v$. Moreover, for such defined $\beta_{uv}$, $Var(\bm{\epsilon}_u)=\frac{1}{\Omega_{uu}}$, $Cov(\bm{\epsilon}_u, \bm{\epsilon}_v)=\frac{\Omega_{uv}}{\Omega_{uu}\Omega_{vv}}$.
%\end{theorem}
%Based on Theorem \ref{thm:pseudo} and the \textit{neighborhood selection approach} (\cite{meinshausen2006high}),
We consider minimizing the following local group-lasso penalized weighted $L_2$ loss function for estimating $\bm{\beta}(t_k)=(\beta_{uv}(t_k))_{u\neq v}$:
\begin{equation}\label{eq:pseudo_naive}
L_{PL}(\mathbb{B}_k):=\frac{1}{\sqrt{|\mathcal{N}_{k,d}|}}\sum_{i\in \mathcal{N}_{k,d}}\left[\frac{1}{2}\sum_{u=1}^p||\bm{X}_u-\sum_{v\neq u}\beta_{uv}(t_i)\bm{X}_v||_{\bm{W}_h(t_i)}^2\right]
+\lambda\sum_{u\neq v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\beta_{uv}(t_i)^2},
\end{equation}
where $\mathbb{B}_k=\{\bm{\beta}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ is the set of $\bm{\beta}(t_i)$'s within the neighborhood centered around $t_k$ with neighborhood width $d$; $\bm{X}_u=(x^u_1,\ldots,x^u_N)^T$ is the sequence of the $u$th variable in observations $\{\bm{x}_j\}_{1\leq j\leq N}$ and $\bm{W}_h(t_i)={\rm diag}\{\omega_h^{t_i}(t_j)\}_{1\leq j\leq N}$ is a weight matrix. The $\bm{W}$-norm of a vector $\bm{z}$ is defined as $||\bm{z}||_{\bm{W}}=\sqrt{\bm{z}^T\bm{W}\bm{z}}$. Once $\hat{\bm{\beta}}(t_k)$ is obtained through minimizing (\ref{eq:pseudo_naive}) with respect to $\mathbb{B}_k$, we can derive the estimated edge set at $t_k$: $\widehat{E}(t_k)=\{\{u,v\}: \hat{\beta}_{uv}(t_k)\neq0, ~~ u < v\}$.
The objective function (\ref{eq:pseudo_naive}) may be viewed as an approximation of the likelihood based objective function (\ref{eq:loggle}) through the aforementioned regression connection by ignoring the correlation among the residuals $\epsilon_u$'s. We refer to this approximation as the \textit{pseudo-likelihood approximation}.
However, minimizing (\ref{eq:pseudo_naive}) cannot guarantee symmetry of edge selection, i.e., $\hat{\beta}_{uv}(t)$ and $\hat{\beta}_{vu}(t)$ being simultaneously zero or nonzero. To achieve this, we modify (\ref{eq:pseudo_naive}) by using a \textit{paired group-lasso penalty} \citep{friedman2010applications}:
\begin{equation}\label{eq:pseudo}
\begin{split}
\tilde{L}_{PL}(\mathbb{B}_k)=&\frac{1}{\sqrt{|\mathcal{N}_{k,d}|}}\sum_{i\in \mathcal{N}_{k,d}}\left[\frac{1}{2}\sum_{u=1}^p||\bm{X}_u-\sum_{v\neq u}\beta_{uv}(t_i)\bm{X}_v||_{\bm{W}_h(t_i)}^2\right]\\
&+\lambda\sum_{u<v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\left[\beta_{uv}(t_i)^2+\beta_{vu}(t_i)^2\right]}.
\end{split}
\end{equation}
The paired group-lasso penalty guarantees simultaneous selection of $\beta_{uv}(t)$ and $\beta_{vu}(t)$.
The objective function (\ref{eq:pseudo}) can be rewritten as:
\begin{equation}\label{eq:pseudo_concat}
\tilde{L}_{PL}(\mathbb{B}_k)=\frac{1}{\sqrt{|\mathcal{N}_{k,d}|}}\sum_{i\in \mathcal{N}_{k,d}}\frac{1}{2}||\mathbf{Y}(t_i)-\mathbf{X}(t_i)\bm{\beta}(t_i)||_2^2+\lambda\sum_{u<v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\left[\beta_{uv}(t_i)^2+\beta_{vu}(t_i)^2\right]},
\end{equation}
where $\mathbf{Y}(t_i)=(\tilde{\bm{X}}_1(t_i)^T,\ldots,\tilde{\bm{X}}_p(t_i)^T)^T$ is an $Np\times1$ vector with $\tilde{\bm{X}}_u(t_i)=\sqrt{\bm{W}_h(t_i)}\bm{X}_u$ being an $N\times1$ vector ($u=1,\cdots, p$); $\mathbf{X}(t_i)=(\tilde{\bm{X}}_{(1,2)}(t_i),\ldots,\tilde{\bm{X}}_{(p,p-1)}(t_i))$ is an $Np\times p(p-1)$ matrix, with $\tilde{\bm{X}}_{(u,v)}(t_i)=(\bm{0}_N^T,\ldots,\bm{0}_N^T,\tilde{\bm{X}}_v(t_i)^T,\bm{0}_N^T,\ldots,\bm{0}_N^T)^T$ being an $Np\times1$ vector, where $\tilde{\bm{X}}_v(t_i)$ is in the $u$th block ($1\leq u\neq v \leq p$); and $\bm{\beta}(t_i)=(\beta_{12}(t_i),\ldots,\beta_{p,p-1}(t_i))^T$ is a $p(p-1)\times1$ vector.
We implement an ADMM algorithm to minimize (\ref{eq:pseudo_concat}), which does not involve eigen-decomposition and thus is much faster than the ADMM algorithm for minimizing the original likelihood based objective function (\ref{eq:loggle}). This is because, the $L_2$ loss used in (\ref{eq:pseudo}) and (\ref{eq:pseudo_concat}) is quadratic in the parameters $\mathbb{B}_k$ as opposed to the negative log-likelihood loss used in (\ref{eq:loggle}) which has a log-determinant term. Moreover, $\mathbf{X}(t_i)$ is actually a block diagonal matrix: $\mathbf{X}(t_i)={\rm diag}\{\tilde{\mathbf{X}}_{(-u)}(t_i)\}_{1\leq u\leq p}$, where $\tilde{\mathbf{X}}_{(-u)}(t_i)=(\tilde{\bm{X}}_1(t_i),\ldots,\tilde{\bm{X}}_{u-1}(t_i), \tilde{\bm{X}}_{u+1}(t_i),\ldots,\tilde{\bm{X}}_p(t_i))$ is an $N\times(p-1)$ matrix. Therefore, computations can be done in a blockwise fashion and potentially can be parallelized. The detailed algorithm is given in Appendix \ref{sec:ADMM_PL}.
\begin{comment}
\cite{peng2009partial} proposed the \textit{sparse partial correlation estimation} (\texttt{space}) method, which improves the \textit{neighborhood selection approach} by ensuring the sign consistency between $\beta_{uv}$ and $\beta_{vu}$. However, the running time of \texttt{space} method is usually 2-3 times longer than that of \textit{neighborhood selection approach} in practice. The details of implementing \texttt{space} method in pseudo-likelihood approximation are discussed in \ref{sec:space} in Supplementary Material. Extensive simulation studies show that the performance of pseudo-likelihood approximation based on \texttt{space} method is similar to that based on \textit{neighborhood selection approach} (results not reported). In Section \ref{sec:simulation} and \ref{sec:real_data}, when referring to pseudo-likelihood approximation, only the results based on \textit{neighborhood selection approach} will be presented.
\end{comment}
\subsection{Model Tuning}\label{subsec:tuning_parameter}
In the \texttt{loggle} model, there are three tuning parameters, namely, $h$ -- the kernel bandwidth (for $\hat{\bm \Sigma}(t)$'s), $d$ -- the neighborhood width (for $\mathcal{N}_{k,d}$'s) and $\lambda$ -- the sparsity parameter. In the following, we describe $V$-fold cross-validation (CV) to choose these parameters.
Recall that observations are made on a temporal grid. So we create the validation sets by including every $V$th data point and the corresponding training set would be the rest of the data points. E.g., for $V=5$, the 1st validation set would include observations at $t_1,t_6,t_{11}, \cdots$, the 2nd validation set would include those at $t_2, t_7, t_{12}, \cdots$, etc. In the following, let $\mathcal{I}_{(v)}$ denote the indices of the time points in the $v$th validation set and $\mathcal{I}_{-(v)}$ denote those in the $v$th training set ($v=1,\cdots, V$).
Let $h_{\rm grid}, d_{\rm grid}, \lambda_{\rm grid}$ denote the tuning grids from which $h$, $d$ and $\lambda$, respectively, are chosen. See Section \ref{sec:simulation} for an example of the tuning grids. We recommend to choose $d$ and $\lambda$ separately for each $t_k$ as the degrees of sparsity and smoothness of the graph topology may vary over time. On the other hand, we recommend to choose a common $h$ for all time points. %\JP{In Section \ref{sec:simulation}, this tuning strategy is shown to outperform the approach where all three tuning parameters are fixed across time.}
%For the choice of $\lambda$, we can choose either a single $\lambda$ or separate $\lambda_k$'s for each objective function $L(\bm{\Omega}_k)$. Choosing separate $\lambda_k$'s may make our model more flexible and reliable, since in different time points the extent of sparsity of the precision matrices may vary a lot, making a single $\lambda$ perform poorly. We can also choose either a single $d$ or separate $d_k$'s for each $L(\bm{\Omega}_k)$, since the consistency of the sparsity pattern may vary over time as well. Now suppose we want to choose separate $\lambda_k$'s and $d_k$'s, and a single $h$.\\
Given time $t_k$ and $h$, for $(d_k,\lambda_k)$, we obtain the refitted estimate $\hat{\bm{\Omega}}^\text{rf}_{-(v)}(t_k; d_k,\lambda_k,h)$ by applying \texttt{loggle} to the $v$th training set $\{\bm{x}_i\}_{i\in \mathcal{I}_{-(v)}}$ ($v=1,\cdots, V$). As mentioned in Section \ref{subsec:loggle}, this can be done even if $t_k \notin \{t_i: i\in\mathcal{I}_{-(v)}\}$. We then derive the validation score on the $v$th validation set:
$$\text{CV}_v(t_k;\lambda_k,d_k,h)=\text{tr}\left(\hat{\bm{\Omega}}^\text{rf}_{-(v)}(t_k;d_k,\lambda_k,h)\hat{\bm{\Sigma}}_{(v)}(t_k)\right)-\log|\hat{\bm{\Omega}}^\text{rf}_{-(v)}(t_k;d_k,\lambda_k,h)|,$$
where $\hat{\bm{\Sigma}}_{(v)}(t_k):=\sum_{i\in \mathcal{I}_{(v)}}\omega_{h_V}^{t_i}(t_k)\bm{x}_i\bm{x}_i^T$ is the kernel estimate of the covariance matrix $\bm{\Sigma}(t_k)$ based on the $v$th validation set $\{\bm{x}_i\}_{i\in \mathcal{I}_{(v)}}$. Here, the bandwidth $h_V$ is set to be $h\cdot(\frac{1}{V-1})^{-1/5}$ to reflect the difference in sample sizes between the validation and training sets. Finally, the $V$-fold cross-validation score at time $t_k$ is defined as: $\text{CV}(t_k;\lambda_k,d_k,h)=\sum_{v=1}^V\text{CV}_v(t_k;\lambda_k,d_k,h)$. The ``optimal" tuning parameters at $t_k$ given $h$, $(\hat{d}_k(h), \hat{\lambda}_k(h))$, is the pair that minimizes the CV score. Finally, the ``optimal" $h$ is chosen by minimizing the sum of $\text{CV}(t_k;\hat{\lambda}_k(h),\hat{d}_k(h),h)$ over those time points where a \texttt{loggle} model is fitted.
We also adopt the \texttt{cv.vote} procedure proposed in \cite{peng2010regularized} which has been shown to be able to significantly reduce the false discovery rate while sacrifice only modestly in power. Specifically, given the CV selected tuning parameters, we examine the fitted model on each training set and only retain those edges that appear in at least T\% of these models. In practice, we recommend 80\% as the cut off value for edge retention.
Moreover, we implement efficient grid search strategies including early stopping and coarse search followed by refined search to further speed up the computation. Details can be found in Section \ref{sec:supp_tuning} of the Supplementary Material.
%\JP{What is the cut off $V_0$ we are using?; default 0.8; simulation: 0.6,0.6, 1.0; real data: 0.8}
% \JP{ To speed up computation, we also implement threshold cut in grid search?? and coarse grid search??. The details are given in Supplementary Materials \ref{sec:thresh_grid} and \ref{sec:coarse_grid}. }
\begin{comment}
To further reduce the possibility of overfitting, we introduce \texttt{cv.vote} (\cite{peng2010regularized}). The idea is that the graph edges being selected by a majority of cross-validation folds have a larger chance to appear in the true model than those selected by only few cross-validation folds. Specifically, for $1\leq u<v\leq p$, $k=1,\ldots,N$, define
\begin{align*}
s_{uv}(\lambda_k,d_k,h)=\begin{cases}
1 & \text{if}\ \sum_{v=1}^VI\left((\hat{\bm{\Omega}}^\text{rf}_{-(v)}(t_k))_{uv}\neq0\right)/V\geq V_0,\\
0 & \text{otherwise},
\end{cases}
\end{align*}
where $I(\cdot)$ is the indicator function, and $V_0\in[0, 1]$ is a pre-specified proportion. We then select the edge $(u, v)$ in the graph at time point $t_k$ if and only if $s_{uv}(\lambda_k,d_k,h)=1$. \cite{peng2010regularized} shows that \texttt{cv.vote} can significantly reduce the false discovery rate while only sacrifice a little power.
\end{comment}
\section{Simulation}\label{sec:simulation}
In this section, we evaluate the performance of \texttt{loggle} and compare it with \texttt{kernel} and \texttt{invar} by simulation experiments.
\subsection{Setting}\label{sec:simulation_setting}
We consider models with both time-varying graphs and time-invariant graphs:
\begin{itemize}
\item
\textit{Time-varying graphs}: (i) Generate four lower triangular matrices $\bm{B}_1,\;\bm{B}_2,\;\bm{B}_3,\;\bm{B}_4\in\mathbb{R}^{p\times p}$ with elements independently drawn from $\mathcal{N}(0,1/2)$. (ii) Let $\phi_1(t)=\sin(\pi t/2)$, $\phi_2(t)=\cos(\pi t/2)$, $\phi_3(t)=\sin(\pi t/4)$ and $\phi_4(t)=\cos(\pi t/4)$, $t\in[0,1]$, and set $\bm{G}(t)=\left(\bm{B}_1\phi_1(t)+\bm{B}_2\phi_2(t)+\bm{B}_3\phi_3(t)+\bm{B}_4\phi_4(t)\right)/2$.
(iii) Define $\bm{\Omega}^o(t)=\bm{G}(t)\bm{G}^T(t)$ and ``soft threshold" its off-diagonal elements to obtain $\bm{\Omega}(t)$: $\Omega_{uv}(t)=\text{sign}(1-\frac{0.28}{|\Omega^o_{uv}(t)|})\cdot(1-\frac{0.14}{|\Omega^o_{uv}(t)|})_+\Omega^o_{uv}(t)$, where $(x)_+=\max\{x, 0\}$. (iv) Add $\log_{10}(p)/4$ to the diagonal elements of $\bm{\Omega}(t)$ to ensure positive definiteness.
%In this case, the graph structure and the graph coefficients are both changing smoothly across time.
\item
\textit{Time-invariant graphs}: (i) Generate an Erdos-Renyi graph \citep{erdos1959random} where each pair of nodes is connected independently with probability $2/p$ (so the total number of edges is around $p$). Denote the edge set of this graph by $\mathbb{E}$. (ii) For off-diagonal elements ($1 \leq u \not= v \leq p$ ), if $\{u,v\}\notin \mathbb{E}$, set $\Omega_{uv}(t) \equiv 0$ for $t\in[0, 1]$; If $\{u,v\}\in \mathbb{E}$, set $\Omega_{uv}(t)=\sin(2\pi t-c_{uv})$, where $c_{uv}\sim\text{uniform}(0, 1)$ is a random offset. (iii) For diagonal elements ($1\leq u \leq p$), set $\Omega_{uu}(t)=|\sin(2\pi t-c_{uu})|+\log_{10}(p)$, where $c_{uu}\sim \text{uniform}(0, 1)$ is a random offset.
%The graphs have identical structures across time with smoothly fluctuating coefficients.
\end{itemize}
We construct three models following the above descriptions. Specifically, two models have time-varying graphs with $p=100$ and $p=500$ nodes, respectively. In these two models, the graphs change smoothly over time with the average number of edges being 51.6 (standard deviation 6.0) for $p=100$ model and 203.0 (standard deviation 66.8) for $p=500$ model. The plots depicting the number of edges vs. time are given in Figure \ref{fig:ed_num} of the Supplementary Material. In the third model, the graphs are time-invariant (even though the precision matrices change over time) with $p=100$ nodes and 93 fixed edges.
For each model, we generate $\bm{x}_k\sim\mathcal{N}_p(\bm{0},\bm{\Omega}^{-1}(t_k))$, with $t_k=\frac{k-1}{N}$ ($k=1,\cdots, N+1$).
%We considered cases with $p=100, 500$ and $N=1000,2000$.
%where time points $\{t_k\}_{k=1,\ldots,N}$ take values $\{0, 0.001, \ldots, 0.999, 1\}$.
We use the Epanechnikov kernel $K_h(x)=\frac{3}{4}\left(1-(x/h)^2\right)I_{\{|x|\leq h\}}$ to obtain smoothed estimates of the correlation matrices. In the following, we consider $N=1000$ observations and conduct model fitting on $K=49$ time points at $\tilde{t}_k \in \{0.02, 0.04, \ldots, 0.96, 0.98\}$.
%We implement pseudo-likelihood approximation in all three methods \texttt{loggle}, \texttt{kernel} and \texttt{invariant}. Extensive simulation studies (not shown here) show that the model selection results are slightly better than those obtained using original likelihood, and computation time can be reduced by as many as $90\%$.
%\subsection{Model selection}\label{sec:model_selection}
We use 5-fold cross-validation for tuning parameters selection from $h_{\rm grid}=\{$0.1, 0.15, $\ldots$, 0.3$\}$, $d_{\rm grid}=\{$0, 0.001, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.25, 0.3, 1$\}$ and $\lambda_{\rm grid}=\{$0.15, 0.17, $\ldots$, 0.35$\}$.
%Recall that for \texttt{kernel} and \texttt{invar}, $d$ is fixed at $0,1$, respectively.
%In method \texttt{loggle}, we can choose separate $d_k$ and $\lambda_k$ at each time point $t_k$, as mentioned in Section \ref{subsec:tuning_parameter}. Here to make our comparison complete, we also add a method \texttt{loggle*}, a variant of \texttt{loggle} where $d$ and $\lambda$ are fixed across all time points. Moreover, we have also examined the variants of \texttt{kernel} and \texttt{invariant} where $\lambda$ is allowed to vary at different time points, and the results (see Supplementary Materials \ref{sec:simu_add}) show that it makes no qualitative difference despite slightly better performance.\\
The metrics used for performance evaluation include false discovery rate: $\text{FDR}:=1-\frac{1}{K}\sum_{k=1}^K|\hat{S}_k\cap S_k|/|\hat{S}_k|$ and
$\text{power}:=\frac{1}{K}\sum_{k=1}^K|\hat{S}_k\cap S_k|/|S_k|$ for edge detection (averaged over the $K$ time points where graphs are estimated), where $S_k$ and $\hat{S}_k$ are the true edge set and the estimated edge set at time point $\tilde{t}_k$, respectively. We also consider $F_1:=2\cdot\frac{(1-\text{FDR})\cdot\text{power}}{(1-\text{FDR})+\text{power}}$ as an overall metric for model selection performance which strikes a balance between FDR and power: The larger $F_1$ is, the better a method performs in terms of edge selection. In addition, we calculate the Kullback-Leibler (K-L) divergence (relative entropy) between the true models and the estimated models: $\delta_{KL}:=\frac{1}{K}\sum_{k=1}^K[\text{tr}(\hat{\bm{\Omega}}(\tilde{t}_k)\bm{\Omega}^{-1}(\tilde{t}_k))-\log|\hat{\bm{\Omega}}(\tilde{t}_k)\bm{\Omega}^{-1}(\tilde{t}_k)|-p]$.
%Smaller K-L divergence means better performance. K-L divergence is defined by
\subsection{Results}\label{sec:eval_result}
%\JP{need updates: add N=2000 results; add K-L divergence column.}
%\subsubsection{Model selection results}
%The simulation results are summarized in Table \ref{table:model_selection}.
%When $\lambda$ and/or $d$ are selected separately for each time point, we report their mean (and standard deviation in parenthesis) across the time points.
\begin{table}[H]
\begin{center}
\caption{\textbf{Simulation Results.} \label{table:model_selection}}
\begin{tabular}{c|ccc|c}
\hline\hline
\multicolumn{5}{c}{$p=100$ time-varying graphs model}\\
Method & FDR & power & $F_1$&$\delta_{KL}$\\
\hline
\texttt{loggle} & 0.196 & 0.702 & 0.747 & 2.284\\
\texttt{kernel} & 0.063 & 0.571 & 0.703 & 2.690\\
\texttt{invar} & 0.583 & 0.678 & 0.514 & 2.565\\
\hline\hline
\multicolumn{5}{c}{$p=500$ time-varying graphs model}\\
Method & FDR & power & $F_1$&$\delta_{KL}$\\
\hline
\texttt{loggle} & 0.215 & 0.613 & 0.678 & 9.564\\
\texttt{kernel} & 0.035 & 0.399 & 0.561 & 11.818\\
\texttt{invar} & 0.590 & 0.597 & 0.478 & 10.608\\
\hline\hline
\multicolumn{5}{c}{$p=100$ time-invariant graphs model}\\
Method & FDR & power & $F_1$&$\delta_{KL}$\\
\hline
\texttt{loggle} & 0.000 & 0.978 & 0.988 & 1.559\\
\texttt{kernel} & 0.042 & 0.509 & 0.598 & 3.168\\
\texttt{invar} & 0.000 & 1.000 & 1.000 & 1.531\\
\hline\hline
\end{tabular}
\end{center}
\end{table}
\begin{comment}
\begin{table}[H]
\begin{center}
\caption{\textbf{Simulation Results.} \label{table:model_selection}}
\begin{tabular}{c|ccc|ccc|c}
\hline\hline
\multicolumn{8}{c}{$p=100$ time-varying graphs, $N=1000$}\\
Method & $h$ & $d$ & $\lambda$ & FDR & power & $F_1$&K-L\\
\hline
\texttt{loggle} & 0.15 & 0.27(0.03) & 0.19(0.02) & 0.196 & 0.702 & 0.747 & 2.284\\
\texttt{kernel} & 0.20 & 0.00 & 0.21 & 0.063 & 0.571 & 0.703 & 2.690\\
\texttt{invar} & 0.15 & 1.00 & 0.15 & 0.583 & 0.678 & 0.514 & 2.565\\
\hline\hline
\multicolumn{8}{c}{$p=500$ time-varying graphs, $N=1000$}\\
Method & $h$ & $d$ & $\lambda$ & FDR & power & $F_1$&K-L\\
\hline
\texttt{loggle} & 0.15 & 0.29(0.02) & 0.21(0.02) & 0.215 & 0.613 & 0.678 & 9.564\\
\texttt{kernel} & 0.20 & 0.00 & 0.27 & 0.035 & 0.399 & 0.561 & 11.818\\
\texttt{invar} & 0.15 & 1.00 & 0.17 & 0.590 & 0.597 & 0.478 & 10.608\\
\hline\hline
\multicolumn{8}{c}{$p=100$ time-invariant graphs, $N=1000$}\\
Method & $h$ & $d$ & $\lambda$ & FDR & power & $F_1$&K-L\\
\hline
\texttt{loggle} & 0.10 & 0.96(0.17) & 0.20(0.02) & 0.000 & 0.978 & 0.988 & 1.559\\
\texttt{kernel} & 0.15 & 0.00 & 0.23 & 0.042 & 0.509 & 0.598 & 3.168\\
\texttt{invar} & 0.10 & 1.00 & 0.19 & 0.000 & 1.000 & 1.000 & 1.531\\
\hline\hline
\end{tabular}
\end{center}
\end{table}
\begin{table}[H]
\begin{center}
\caption{\textbf{Simulation Results.} Numbers in parenthesis are standard deviations. \label{table:model_selection}}
\begin{tabular}{c|ccc|ccc|c}
\hline\hline
\multicolumn{8}{c}{$p=100$ time-varying graphs, $N=1000$}\\
Method & $h$ & $d$ & $\lambda$ & FDR & power & $F_1$&K-L\\
\hline
\texttt{loggle} & 0.15 & 0.27(0.03) & 0.19(0.02) & 0.196(0.057) & 0.702(0.070) & 0.747 & 2.284(0.774)\\
\texttt{loggle*} & 0.15 & 0.30 & 0.19 & 0.223(0.048) & 0.664(0.071) & 0.715 & 2.463(0.688)\\
\texttt{kernel} & 0.20 & 0.00 & 0.21 & 0.063(0.071) & 0.571(0.077) & 0.703 & 2.690(0.522)\\
\texttt{invar} & 0.15 & 1.00 & 0.15 & 0.583(0.060) & 0.678(0.044) & 0.514 & 2.565(0.842)\\
\hline\hline
\multicolumn{8}{c}{$p=100$ time-varying graphs, $N=2000$}\\
Method & $h$ & $d$ & $\lambda$ & FDR & power & $F_1$&K-L\\
\hline
\texttt{loggle} & 0.10 & 0.22(0.05) & 0.16(0.01) & 0.192(0.068) & 0.860(0.059) & 0.832 & 1.326(0.541)\\
\texttt{loggle*} & 0.10 & 0.25 & 0.15 & 0.237(0.050) & 0.855(0.062) & 0.805 & 1.372(0.496)\\
\texttt{kernel} & 0.15 & 0.00 & 0.17 & 0.049(0.056) & 0.749(0.071) & 0.834 & 1.610(0.358)\\
\texttt{invar} & 0.10 & 1.00 & 0.15 & 0.576(0.042) & 0.711(0.055) & 0.530 & 2.078(0.714)\\
\hline\hline
\multicolumn{8}{c}{$p=500$ time-varying graphs, $N=1000$}\\
Method & $h$ & $d$ & $\lambda$ & FDR & power & $F_1$&K-L\\
\hline
\texttt{loggle} & 0.15 & 0.29(0.02) & 0.21(0.02) & 0.215(0.080) & 0.613(0.102) & 0.678 & 9.564(6.881)\\
\texttt{loggle*} & 0.15 & 0.30 & 0.23 & 0.175(0.058) & 0.543(0.057) & 0.652 & 10.159(6.125)\\
\texttt{kernel} & 0.20 & 0.00 & 0.27 & 0.035(0.060) & 0.399(0.059) & 0.561 & 11.818(4.978)\\
\texttt{invar} & 0.15 & 1.00 & 0.17 & 0.590(0.062) & 0.597(0.120) & 0.478 & 10.608(7.698)\\
\hline\hline
\multicolumn{8}{c}{$p=100$ time-invariant graphs, $N=1000$}\\
Method & $h$ & $d$ & $\lambda$ & FDR & power & $F_1$&K-L\\
\hline
\texttt{loggle} & 0.10 & 0.96(0.17) & 0.20(0.02) & 0.000(0.000) & 0.978(0.060) & 0.988 & 1.559(0.696)\\
\texttt{loggle*} & 0.10 & 1.00 & 0.19 & 0.000(0.000) & 1.000(0.000) & 1.000 & 1.531(0.677)\\
\texttt{kernel} & 0.15 & 0.00 & 0.23 & 0.042(0.096) & 0.509(0.336) & 0.598 & 3.168(0.676)\\
\texttt{invar} & 0.10 & 1.00 & 0.19 & 0.000(0.000) & 1.000(0.000) & 1.000 & 1.531(0.677)\\
\hline\hline
\end{tabular}
\end{center}
\end{table}
\end{comment}
Table \ref{table:model_selection} shows that under the time-varying graphs models,
\texttt{loggle} outperforms \texttt{kernel} according to $F_1$ score and K-L divergence.
%\texttt{loggle} performs better than \texttt{loggle*} due to the flexibility of selecting (possibly) different $d$ and $\lambda$ at different time points. This is expected as both the degrees of smoothness and sparsity vary across time.
Not surprisingly, \texttt{invar} performs very poorly for time-varying graphs models. On the other hand, under the time-invariant graphs model, \texttt{loggle} performs similarly as \texttt{invar}, whereas \texttt{kernel} performs very poorly.
%On the other side, in time-invariant case (scenario (c)), \texttt{invariant} is believed to perform well in this setting as it assumes the time-invariance nature of graph structure. Table \ref{table:model_selection} shows that our method \texttt{loggle} has similar great performance as \texttt{invariant} in model selection. Moreover, method \texttt{loggle*} where $d$ and $\lambda$ are fixed across time can recover exactly the same model selection result as \texttt{invariant}. Meanwhile, \texttt{kernel} performs very poorly in capturing the time-invariant graph structure, which is reasonable since \texttt{kernel} totally ignores the consistency of graph structure across time.\\
These results demonstrate that \texttt{loggle} can adapt to different degrees of smoothness of the graph topology in a data driven fashion and has generally good performance across a wide range of scenarios including both time-varying and time-invariant graphs.
The \texttt{loggle} procedure is more computationally intensive than \texttt{kernel} and \texttt{invar} as it fits many more models.
For the $p=100$ time-varying graphs model, \texttt{loggle} took 3750 seconds using 25 cores on a linux server with 72 cores, 256GB RAM and two Intel Xeon E5-2699 v3 @ 2.30GHz processors. At the same time, \texttt{kernel} took 226 seconds and \texttt{invar} took 777 seconds.
%\JY{Compared with \texttt{kernel} and \texttt{invar}, \texttt{loggle} searches on a larger grid with 161,700 models to fit, while \texttt{kernel} and \texttt{invar} have 13,475 and 275 models to fit respectively.}
On average, per \texttt{loggle} model fit took 23.2 milliseconds (ms), per \texttt{kernel} model fit took 16.8ms and per \texttt{invar} model fit took 2825.5 ms. The additional computational cost of \texttt{loggle} is justified by its superior performance and should become less of a burden with fast growth of computational power.
\begin{comment}
For example, for the $p=100$ nodes and $N=1000$ time-varying graphs model, a total of 161,700 \texttt{loggle} models were fitted for 49 time points with 5-fold CV using the tuning grids described earlier. This was done in 3750 seconds using 25 cores on a linux server with 72 cores, 256GB RAM and two Intel Xeon E5-2699 v3 @ 2.30GHz processors. At the same time, \texttt{kernel} fits 13,475 models using 226 seconds, whereas \texttt{invar} fits 275 models using 777 seconds. On average, per \texttt{loggle} fit takes 23.2 milliseconds (ms), per \texttt{kernel} fit takes 16.8ms and per \texttt{invar} fit takes 2825.5 ms. The additional computational cost of the \texttt{loggle} procedure is justified by its superior performance and should become less a burden with fast growth of computational power.
\end{comment}
\begin{comment}
\subsection{Computation cost}
We experiment using 25 cores on the server \texttt{phall}, a 2-year-old linux server with 72 cores, 256GB RAM and two Intel Xeon E5-2699 v3 @ 2.30GHz processors provided by Department of Statistics, UC Davis. We implement grid search for $h$, $d$ and $\lambda$ by 5-fold cross-validation on 49 time points, as described in Section \ref{sec:model_selection}, where the length of $h$ grid is 5 in scenario (a) and (c), and 2 in scenario (b) (pre-selected using coarse grid search), the length of $d$ grid is 12 and the length of $\lambda$ grid is 11, thus a total of 161,700 models to fit in scenario (a) and (c) and 64,680 models to fit in scenario (b). The computation costs are shown in Table \ref{table:computation_cost}, where the mean and standard deviation of edge numbers can help us get insights of intrinsic dimension of graphs:
\begin{table}[H]
\begin{center}
\caption{Computation cost\label{table:computation_cost}}
\begin{tabular}{c|cc|cc}
\hline\hline
Scenario & $\#$ Variable & $\#$ Edge & Time in total (s) & Time per model (ms)\\
\hline
(a) & 100 & 51.6(6.0) & 3750 & 23.2 \\
(b) & 500 & 203.0(66.8) & 85390 & 1320.2 \\
(c) & 100 & 93(0) & 9669 & 59.8 \\
\hline\hline
\end{tabular}
\end{center}
\end{table}
For scenario (a), we have also recorded computation costs for methods \texttt{kernel} and \texttt{invariant} shown in Table \ref{table:computation_cost_a}. Note that for both methods we only implement grid search on $h$ and $\lambda$, since $d$ is fixed at 0 for \texttt{kernel} and 1 for \texttt{invariant}. Moreover, for \texttt{invariant}, model only need to be fitted once on all time points when $h$ and $\lambda$ are given. There are two reasons why \texttt{loggle} costs more time than \texttt{kernel} or \texttt{invariant}. Firstly, \texttt{loggle} searches on a larger grid with more models to fit. Secondly, models become more complex and time-consuming to fit as $d$ increases in \texttt{loggle}.
\begin{table}[H]
\begin{center}
\caption{Computation cost\label{table:computation_cost_a}}
\begin{tabular}{c|c|cc}
\hline\hline
Method & $\#$ Model & Time in total (s) & Time per model (ms)\\
\hline
\texttt{loggle} & 161,700 & 3750 & 23.2 \\
\texttt{kernel} & 13,475 & 226 & 16.8 \\
\texttt{invar} & 275 & 777 & 2825.5 \\
\hline\hline
\end{tabular}
\end{center}
\end{table}
\end{comment}
\section{S\&P 500 Stock Price} \label{sec:real_data}
In this section, we apply \texttt{loggle} to the S\&P 500 stock price dataset obtained via R package \texttt{quantmod} from \textit{www.yahoo.com}. We focus on 283 stocks from 5 Global Industry Classification Standard (GICS) sectors: 58 stocks from Information Technology, 72 stocks from Consumer Discretionary, 32 stocks from Consumer Staples, 59 stocks from Financials, and 62 stocks from Industrials. We are interested in elucidating how direct interactions (characterized by conditional dependencies) among these stocks are evolving over time and particularly how such interactions are affected by the recent global financial crisis.
For this purpose, we consider a 4-year time period from January 1st, 2007 to January 1st, 2011, which covers the recent global financial crisis: ``According to the U.S. National Bureau of Economic Research, the recession, as experienced in that country, began in December 2007 and ended in June 2009, thus extending over 19 months. The Great Recession was related to the financial crisis of 2007-2008 and U.S. subprime mortgage crisis of 2007-2009 (Source: wikipedia)". Each stock has 1008 closing prices during this period, denoted by $\{y_k\}_{k=1}^{1008}$. We use the logarithm of the ratio between two adjacent prices, i.e., $\log \frac{y_{k+1}}{y_{k}} (k=1,\cdots, 1007)$ for the subsequent analysis. We also convert the time points onto [0, 1] by $t_k=\frac{k-1}{1006}$ for $k=1,\cdots, 1007$. By examining the autocorrelation (Figure \ref{fig:acf} of the Supplementary Material), the independence assumption appears to hold reasonably well.
%See Section \ref{sec:real_data_add} of the Supplementary Materials for more details.
We use the Epanechnikov kernel to obtain the kernel estimates of the correlation matrices.
\begin{comment}
At each time point, we first examined a correlation-based graph constructed by keeping an edge between two stocks if the respective (estimated) pairwise correlation is above 0.7 (in absolute value). As can be seen from
Figure \ref{fig:marcor} of the Supplementary Materials, the degree of connectivity is increased (indicated by a large number of edges in the aforementioned graphs) during the financial crisis and peaks around the end of 2008. In contrast, before and after the crisis, the degree of connectivity is much less.
\end{comment}
%The pairwise marginal correlation density plot (Figure \ref{fig:marcor} in Supplementary Materials \ref{sec:real_data_add}) shows that during the global financial crisis period (from the beginning of 2008 to the end of 2009), a large amount of stocks are marginally highly correlated, while before or after the global financial crisis, the number of stock pairs with high marginal correlations drops significantly. The marginal correlation plot seems reasonable at the first glance, however, it may sometimes provide misleading insights. For example, during financial crisis, the high marginal correlation between two stocks can be easily induced by another stock which these two stocks are directly interacted with. Thus, the effect of high stock correlation during financial crisis may be exaggerated. In this case, graphical models which can detect conditional correlations rather than marginal correlations between stocks becomes a better choice.\\
We then fit three models, namely, \texttt{loggle}, \texttt{kernel} and \texttt{invar}, at $K=201$ time points $\{0.005, 0.010, \ldots, 0.995\}$ using 5-fold cross-validation for model tuning. We use the tuning grids $h_{\rm grid}=\{0.1, 0.15\}$, $\lambda_{\rm grid}=\{10^{-2}, 10^{-1.9}, \ldots, 10^{-0.1}, 1\}$ and
$d_{\rm grid}=\{0$, 0.001, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.25, 0.3, 1$\}$, where $h_{\rm grid}$ is pre-selected by using coarse search described in Section \ref{sec:supp_tuning} of the Supplementary Material. Table \ref{table:model_selection_real} reports the average number of edges across the fitted graphs (and standard deviations in parenthesis) as well as the CV scores. We can see that \texttt{loggle} has a significantly smaller CV score than those of \texttt{kernel} and \texttt{invar}. Moreover, on average, \texttt{loggle} and \texttt{invar} models have similar number of edges, whereas \texttt{kernel} models have more edges.
%and the result of \texttt{loggle*} coincides with the result of \texttt{invariant}.
\begin{table}[h]
\begin{center}
\caption{\textbf{Stock price:} Number of edges and CV score}\label{table:model_selection_real}
\begin{tabular}{c|cc}
\hline\hline
Method & Average edge \# (s.d.) & CV score\\
\hline
\texttt{loggle} & 819.4 (331.0) & 123.06\\
%\texttt{loggle*} & 0.10 & 1 & 0.50 &...& 130.68\\
\texttt{kernel} & 1103.5 (487.1) & 160.14\\
\texttt{invar} & 811.0 (0.0) &130.68\\
\hline\hline
\end{tabular}
\end{center}
\end{table}
\begin{comment}
\begin{table}[H]
\begin{center}
\caption{\textbf{Stock price:} Results}\label{table:model_selection_real}
\JP{add average edge number ( sd.) to this table; }
\begin{tabular}{c|ccc|c|c}
\hline\hline
Method & $h$ & $d$ & $\lambda$ & edge \# & CV score\\
\hline
\texttt{loggle} & 0.10 & 0.56(0.43) & 0.52(0.16)& 819.4(331.0) & 123.06\\
%\texttt{loggle*} & 0.10 & 1 & 0.50 &...& 130.68\\
\texttt{kernel} & 0.15 & 0 & 0.16 & 1103.5(487.1) & 160.14\\
\texttt{invar} & 0.10 & 1 & 0.50 & 811.0(0.0) &130.68\\
\hline\hline
\end{tabular}
\end{center}
\end{table}
\end{comment}
\begin{figure}[h]
\begin{center}
\begin{tabular}{cc}
\includegraphics[width = 2.8in]{ednum_stock.pdf} & \includegraphics[width = 2.8in]{proped_stock.pdf}\\
(a) & (b)
\end{tabular}
\end{center}
\caption{\textbf{Stock Price.} (a) Number of edges vs. time (b) Proportion of within-sector edges vs. time}\label{fig:ed}
\end{figure}
%We then investigate the ability of capturing signals from financial crisis and recovering sector structure for methods \texttt{loggle}, \texttt{kernel} and \texttt{invariant} based on model selection results.\\
Figure \ref{fig:ed}(a) shows the number of edges in the fitted graph over time.
%(smoothed by a weighted moving average over one month windows).
The \texttt{invar} fitted graphs have an identical topology, which is unable to reflect the evolving relationships among these stocks. On the other hand, both \texttt{loggle} and \texttt{kernel} are able to capture the changing relationships by fitting graphs with time-varying topologies. More specifically, both methods detect an increased amount of interaction (characterized by larger number of edges) during the financial crisis. The amount of interaction peaked around the end of 2008 and then went down to a level still higher than that of the pre-crisis period. As can be seen from the figure, the \texttt{kernel} graphs show rather drastic changes, whereas the \texttt{loggle} graphs change more gradually. The \texttt{loggle} method in addition detects a period with increased interaction in the early stage of the financial crisis, indicated by the smaller peak around October 2007 in Figure \ref{fig:ed}(a). This is likely due to the subprime mortgage crisis which acted as a precursor of the financial crisis \citep{amadeo_2017_07}. In the period after the financial crisis, the \texttt{loggle} fits are similar to those of \texttt{invar} with a nearly constant graph topology after March 2010, indicating that the relationships among the stocks had stabilized. In contrast, \texttt{kernel} fits show a small bump in edge number around the middle of 2010 and decreasing amount of interaction afterwards.
\begin{comment}
Compared to the marginal correlation based graphs (Figure \ref{fig:marcor}), the \texttt{loggle} method in addition detects a period with increased degree of interactions in the early stage of the financial crisis, indicated by the smaller peak around October 2007 in Figure \ref{fig:ed}(a). This is likely due to the subprime mortgage crisis which acts as a precursor of the following global financial crisis \citep{amadeo_2017_07}. Moreover, unlike the marginal-correlation based graphs, the
\texttt{loggle} fits also show that the degree of strong interactions among stocks after financial the crisis remains much higher than that before the financial crisis.
\end{comment}
%We can see that \texttt{kernel} captures some signals during financial crisis (from the beginning of 2008 to the end of 2009), but the number of edges seems to change too drastically, for example, 8-time increment of edge number from the beginning of 2007 to the end of 2008, which is uncommon in real life., which is too restrictive and cannot reflect any pattern of evolution in relationship between stocks. \texttt{loggle} takes the advantages of both methods and selects smoothly-changing graphs across time. Specifically, during financial crisis, the evolution pattern of graph structure is similar to the one detected by \texttt{kernel}, which displays consistent changes of direct interactions between stocks. After financial crisis, the evolution pattern of graph structure coincides with the constant one detected by \texttt{invariant}, indicating that there exist no changes of direct interactions between stocks anymore after crisis.\\
%Now we compare the curve corresponding to \texttt{loggle} in Figure \ref{fig:ed}(a) with the curve in marginal correlation plot Figure \ref{fig:marcor}. We can see that both curves have successfully detected peaks during financial crisis. However, \texttt{loggle} in Figure \ref{fig:ed}(a) has also detected a small peak right before 2008 global financial crisis. \cite{amadeo_2017_07} points out that it should depict 2007 financial crisis, which is highly related with subprime mortgage crisis and acts as a precursor of the following global financial crisis. Moreover, different from the marginal correlation plot, \texttt{loggle} in Figure \ref{fig:ed}(a) shows that the number of strong interactions between stocks after financial crisis is much larger than that before financial crisis, implying the serious consequence of financial crisis.\\
Figure \ref{fig:ed}(b) displays the proportion of within-sector edges among the total number of detected edges. During the entire time period, \texttt{loggle} fitted graphs consistently have higher proportion of within-sector edges than that of the \texttt{kernel} fitted graph. For both methods, this proportion decreased during the financial crisis due to increased amount of cross-sector interaction. For \texttt{loggle}, the within-sector edge proportion eventually increased and stabilized after March 2010, although at a level lower than that of the pre-crisis period. In contrast, for \texttt{kernel}, the within-sector proportion took a downturn again after October 2009.
%and in practice, the direct interactions within sectors should occur more frequently than those between sectors.
In summary, the \texttt{loggle} fitted graphs are easier to interpret in terms of describing the evolving interacting relationships among the stocks and identifying the underlying sector structure of the stocks. Hereafter, we focus our discussion on \texttt{loggle} fitted graphs.
%greatly outperforms both \texttt{kernel} and \texttt{invariant} in this real data application. Compared with \texttt{kernel}, \texttt{loggle} can do better job in identifying the intrinsic sector structure of stocks, and the change of graph structure is more realistic. Compared with \texttt{invariant}, \texttt{loggle} can select more reasonable graphs in real life with smoothly changing graph structure. We will focus on \texttt{loggle} hereafter.
\begin{figure}[h]
\begin{center}
\begin{tabular}{ccc}
\includegraphics[width = 2.1in]{stock_1.pdf} & \includegraphics[width = 2.1in]{stock_2.pdf} & \includegraphics[width = 2.1in]{stock_3.pdf}\\
(a) & (b) & (c)\\
\includegraphics[width = 2.1in]{stock_4.pdf} & \includegraphics[width = 2.1in]{stock_5.pdf} & \includegraphics[width = 2.1in]{sepsector_stock.pdf}\\
(d) & (e) & (f)
\end{tabular}
\end{center}
\caption{\textbf{Stock Price.} (a)-(e) \texttt{loggle} fitted graphs at 5 time points. (f) The sector-wise percentage of presence of within-sector edges and the percentage of presence of cross-sector edges of \texttt{loggle} fitted graphs vs. time.}\label{fig:graph_loggle}
\end{figure}
Figure \ref{fig:graph_loggle}(a)-(e) show the \texttt{loggle} fitted graphs at 5 different time points, namely, before, at the early stage, around the peak, towards the end and after the financial crisis. These graphs show clear evolving interacting patterns among the stocks. The amount of interaction increased with the deepened crisis and decreased and eventually stabilized with the passing of the crisis. Moreover, the stocks have more interactions after the crisis compared to the pre-crisis era, indicating fundamental change of the financial landscape.
In addition, these graphs show clear sector-wise clusters (nodes with the same color corresponding to stocks from the same sector).
Figure \ref{fig:graph_loggle}(f) shows the sector-wise \textit{percentage of presence of within-sector edges}, defined as the ratio between the number of detected within-sector edges and the total number of possible within-sector edges for a given sector; and the \textit{percentage of presence of cross-sector edges}, defined as the ratio between the number of detected cross-sector edges and the total number of possible cross-sector edges. As can be seen from this figure, the within-sector percentages are much higher than the cross-sector percentage, reaffirming the observation that \texttt{loggle} is able to identify the underlying sector structure. Moreover, the within-Financials sector percentage is among the highest across the entire time period, indicating that the stocks in this sector have been consistently highly interacting with each other. Finally, all percentages increased after the financial crisis began and leveled off afterwards, reflecting the increased amount of interaction during the financial crisis.
%which reflects the important role of the sector Financials during the period containing financial crisis. It is reasonable since the stocks in that sector are more sensitive to the financial environment changes than the other stocks.
\begin{comment}
\begin{figure}[t]
\begin{center}
\begin{tabular}{cc}
\includegraphics[width = 2.5in]{financials_1.pdf} & \includegraphics[width = 2.5in]{financials_2.pdf}\\
\includegraphics[width = 2.5in]{financials_3.pdf} & \includegraphics[width = 2.5in]{financials_4.pdf}
\end{tabular}
\end{center}
\caption{\textbf{Stock Price.} \texttt{loggle} fitted (sub)graphs of $32$ stocks from Financials sector at 4 different time points in the second half of 2018. Node size is proportional to node degree. }\label{fig:financials_loggle}
\end{figure}
We further look into the Financials sector and focus on 32 stocks from investment banks, investment management companies, financial services companies and bank holding companies. These companies were highly affected by the financial crisis. Figure \ref{fig:financials_loggle} shows the \texttt{loggle} fitted subgraphs of these stocks at 4 different time points from July 2008 to December 2008 (around the peak of the financial crisis). In these graphs, the node size is proportional to node degree (i.e., the number of edges of a node). There are some interesting observations. For instance, there is an increase in degree for JPMorgan Chase (JPM) between September 2018 and November 2018. % (\JP{from 7 to 10})
We hypothesize that this is due to the acquisition of Washington Mutual by JPMorgan Chase on September 25th, 2008,
%which was closed by federal regulators (\cite{usatoday_2013}, \cite{amadeo_2017_08}),
leading to the increase of direct interactions between JPMorgan Chase and other financial companies. Besides node degrees, the edges also reveal the dynamic nature of company relationships during financial crisis. For instance, direct interaction between Goldman Sachs Group (GS) and Bank of America (BAC) disappeared after September 2018.
%which might be due to the agreement of Bank of America to purchase Merrill Lynch on September 15th, 2008.
%(\cite{usatoday_2013}, \cite{amadeo_2017_08}).
On the other hand, some interactions appeared to be more persistent. For instance, Goldman Sachs Group (GS) maintained an interacting relationship with Morgan Stanley (MS), another top investment bank, throughout these six months.
\end{comment}
\begin{figure}[h]
\begin{center}
\begin{tabular}{ccc}
\includegraphics[width = 2.1in]{intersector_1.pdf} & \includegraphics[width = 2.1in]{intersector_2.pdf} & \includegraphics[width = 2.1in]{intersector_3.pdf}\\
\includegraphics[width = 2.1in]{intersector_4.pdf} & \includegraphics[width = 2.1in]{intersector_5.pdf} & \
\end{tabular}
\end{center}
\caption{\textbf{Stock Price.} Cross-sector interaction plots at 5 different time points based on \texttt{loggle} fitted graphs. Each node represents a sector and the edge width is proportional to the percentage of presence of the corresponding cross-sector edges.}\label{fig:intersector_loggle}
\end{figure}
%We also examine the cross-sector interactions.
In Figure \ref{fig:intersector_loggle}, the graphs describe cross-sector interactions among the 5 GICS sectors at five different time points (before, at early stage, around the peak, at late stage and after the financial crisis). In these graphs, each node represents a sector and edge width is proportional to the respective percentage of presence of cross-sector edges (defined as the detected number of edges between two sectors divided by the total number of possible edges between these two sectors). Moreover, edges with cross-sector percentage less than $0.2\%$ are not displayed. We can see that there are more cross-sector interactions during the financial crisis, indicating higher degree of dependency among different sectors in that period. There are also some interesting observations with regard to how these sectors interact with one another and how such interactions change over time.
For example, strong cross-sector interactions between the Financials sector and the Consumer Staples sector arose during the financial crisis despite of their weak relationship before and after the crisis. This is probably due to strong influence of the financial industry on the entire economy during financial crisis. Take the Consumer Discretionary sector and the Industrials sector as another example. These two sectors maintained a persistent relationship throughout the four years, indicating intrinsic connections between them irrespective of the financial landscape.
%For example, some products from some companies in sector Consumer Discretionary is always highly related to engineering and manufacturing provided by some companies in sector Industrials no matter how economic situation evolves.\\
\section{Conclusion}\label{sec:discussion}
In this paper, we propose \textit{LOcal Group Graphical Lasso Estimation} -- \texttt{loggle}, a novel model for estimating a sequence of time-varying graphs based on temporal observations. By using a local group-lasso type penalty, \texttt{loggle} imposes structural smoothness on the estimated graphs and consequently leads to more efficient use of the data as well as more interpretable graphs. Moreover, \texttt{loggle} can adapt to the local degrees of smoothness and sparsity of the underlying graphs in a data driven fashion and thus is effective under a wide range of scenarios.
We develop a computationally efficient algorithm for \texttt{loggle} that utilizes the block-diagonal structure and pseudo-likelihood approximation.
The effectiveness of \texttt{loggle} is demonstrated through simulation experiments. Moreover, by applying \texttt{loggle} to the S\&P 500 stock price data, we obtain interpretable and insightful graphs about the dynamic interacting relationships among the stocks, particularly on how such relationships change in response to the recent global financial crisis.
%Future directions include building the model which allows for time dependence. For example, consider a process $\{\bm{X}^t\in\mathbb{R}^p:t=0,\frac{1}{N},\ldots,1\}$ generated according to the following model with serially independent Gaussian errors: $\bm{X}^t=\bm{A}_1\bm{X}^{t-\frac{1}{N}}+\ldots+\bm{A}_q\bm{X}^{t-\frac{q}{N}}+\bm{\epsilon}^t,\;\bm{\epsilon}^t\sim \mathcal{N}_p(\bm{0},\bm{\Sigma}(t)),\;\text{Cov}(\bm{\epsilon}^t,\bm{\epsilon}^s)=\bm{0}\ \text{if}\ t\neq s,$
%where $\bm{A}_1,\ldots,\bm{A}_q$ are $p\times p$ matrices. When $q=0$, this model is exactly the time varying graphical model we have discussed. Now for this generalized model, besides estimating $\bm{\Sigma}(t)$ (or $\bm{\Sigma}^{-1}(t)$), we also need to determine the order $q$ and estimate the coefficient matrices $\bm{A}_1,\ldots,\bm{A}_q$. \cite{basu2013estimation} proposed a method for estimation in high-dimensional vector autoregressive models. Their model is actually a special case of the above model where $\bm{\Sigma}(t)$ is set to be time-invariant.\\
An R package \texttt{loggle} has been developed and will be available on \url{http://cran.r-project.org/}.
%%%%%%%%% appendix
\section*{Appendix}
\setcounter{subsection}{0}
\renewcommand{\thesubsection}{A.\arabic{subsection}}
\setcounter{equation}{0}
\renewcommand{\theequation}{A.\arabic{equation}}
\subsection{Proof of Theorem \ref{thm:disconnect}}\label{sec:disconnect}
By the KKT conditions, a necessary and sufficient set of conditions for $\hat{\mathbb{\Omega}}_k=\{\hat{\bm{\Omega}}(t_i)\}_{i \in \mathcal{N}_{k,d}}$ being the minimizer of $L(\mathbb{\Omega}_k)$ in (\ref{eq:loggle}) is:
\begin{equation}\label{eq:disconnect}
\frac{1}{\sqrt{|\mathcal{N}_{k,d}|}}\left(\hat{\bm{\Sigma}}(t_i)-\hat{\bm{\Omega}}(t_i)^{-1}\right)+\lambda\bm{\Gamma}(t_i)=\bm{0},\ \forall i\in \mathcal{N}_{k,d},
\end{equation}
where $\bm{\Gamma}(t_i)=(\Gamma_{uv}(t_i))_{p\times p}$, and $(\Gamma_{uv}(t_i))_{i\in \mathcal{N}_{k,d}}$ is a subgradient of $\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\Omega_{uv}(t_i)^2}$:
\begin{equation*}
(\Gamma_{uv}(t_i))_{i\in \mathcal{N}_{k,d}}
\begin{cases}
=\left(\frac{\Omega_{uv}(t_i)}{\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\Omega_{uv}(t_i)^2}}\right)_{i\in \mathcal{N}_{k,d}} & \text{if}\ \sum_{i\in \mathcal{N}_{k,d}}\Omega_{uv}(t_i)^2>0\\
\text{such that}\ \sum_{i\in \mathcal{N}_{k,d}}\Gamma_{uv}(t_i)^2\leq1 & \text{if}\ \sum_{i\in \mathcal{N}_{k,d}}\Omega_{uv}(t_i)^2=0.
\end{cases}
\end{equation*}
If for $\forall i \in \mathcal{N}_{k,d}$, $\hat{\bm{\Omega}}(t_i)=\left(\begin{array}{c c}\hat{\bm{\Omega}}_1(t_i) & \bm{0} \\ \bm{0} &\hat{\bm{\Omega}}_2(t_i)\end{array}\right)$, where $\hat{\bm{\Omega}}_1(t_i)$ and $\hat{\bm{\Omega}}_2(t_i)$ consist of the variables in $G_1$ and $G_2$ respectively, then $\hat{\mathbb{\Omega}}_k=\{\hat{\bm{\Omega}}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ satisfies (\ref{eq:disconnect}) iff $\forall u\in G_1$, $\forall v\in G_2$, $\exists$ $(\Gamma_{uv}(t_i))_{i\in \mathcal{N}_{k,d}}$ satisfying $\sum_{i\in \mathcal{N}_{k,d}}\Gamma_{uv}(t_i)^2\leq1$ such that $$\frac{1}{\sqrt{|\mathcal{N}_{k,d}|}}\hat{\Sigma}_{uv}(t_i)+\lambda\Gamma_{uv}(t_i)=0,\ \forall i\in \mathcal{N}_{k,d}.$$
This is equivalent to
$$\forall u\in G_1,\ \forall v\in G_2,\ \frac{1}{|\mathcal{N}_{k,d}|}\sum_{i\in \mathcal{N}_{k,d}}\hat{\Sigma}_{uv}(t_i)^2\leq\lambda^2.$$
\subsection{ADMM algorithm under pseudo-likelihood approximation}\label{sec:ADMM_PL}
To solve the optimization problem in (\ref{eq:pseudo_concat}) using ADMM algorithm, we notice that the problem can be written as
\begin{align*}
&\textrm{minimize }_{\mathbb{B}_k,\mathbb{Z}_k}\sum_{i\in \mathcal{N}_{k,d}}\frac{1}{2}||\mathbf{Y}(t_i)-\mathbf{X}(t_i)\bm{\beta}(t_i)||_2^2
+\lambda\sum_{u<v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\left[Z_{uv}(t_i)^2+Z_{vu}(t_i)^2\right]},\\
&\textrm{subject to }\bm{\beta}(t_i)-\bm{Z}(t_i)=\bm{0},\;i\in \mathcal{N}_{k,d},
\end{align*}
where $\mathcal{N}_{k,d}=\{i:|t_i-t_k|\leq d\}$, $\mathbb{B}_k=\{\bm{\beta}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ and $\mathbb{Z}_k=\{\bm{Z}(t_i)\}_{i\in \mathcal{N}_{k,d}}$. Note $\bm{\beta}(t_i)=(\beta_{uv}(t_i))_{u\neq v}$ and $\bm{Z}(t_i)=(Z_{uv}(t_i))_{u\neq v}$ are $\mathbb{R}^{p(p-1)}$ vectors.
The scaled augmented Lagrangian is
\begin{align*}
L_\rho(\mathbb{B}_k,\mathbb{Z}_k,\mathbb{U}_k)=&\sum_{i\in \mathcal{N}_{k,d}}\frac{1}{2}||\mathbf{Y}(t_i)-\mathbf{X}(t_i)\bm{\beta}(t_i)||_2^2
+\lambda\sum_{u<v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\left[Z_{uv}(t_i)^2+Z_{vu}(t_i)^2\right]}\\
&+\frac{\rho}{2}\sum_{i\in \mathcal{N}_{k,d}}||\bm{\beta}(t_i)-\bm{Z}(t_i)+\bm{U}(t_i)||_2^2,
\end{align*}
where $\mathbb{U}_k=\{\bm{U}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ are dual variables ($\bm{U}(t_i)=(U_{uv}(t_i))_{u\neq v}\in\mathbb{R}^{p(p-1)}$).
The ADMM algorithm is as follows. We first initialize $\bm{Z}^{(0)}(t_i)=\bm{0}$, $\bm{U}^{(0)}(t_i)=\bm{0}$, $i\in \mathcal{N}_{k,d}$. We also need to specify $\rho (>0)$, which in practice is recommended to be $\approx\lambda$ \citep{wahlberg2012admm}. For step $s=1,2,\ldots$ until convergence:
\begin{enumerate}[(i)]
\item
For $i\in \mathcal{N}_{k,d}$,
\begin{align*}
\bm{\beta}^{s}(t_i)=\arg\min_{\bm{\beta}(t_i)}\frac{1}{2}||\mathbf{Y}(t_i)-\mathbf{X}(t_i)\bm{\beta}(t_i)||_2^2+\frac{\rho}{2}||\bm{\beta}(t_i)-\bm{Z}^{s-1}(t_i)+\bm{U}^{s-1}(t_i)||_2^2.
\end{align*}
The solution $\bm{\beta}^{s}(t_i)$ sets the derivative of the objective function to 0:
$$
(\mathbf{X}(t_i)^T\mathbf{X}(t_i)+\rho\bm{I})\bm{\beta}^s(t_i)=\mathbf{X}(t_i)^T\mathbf{Y}(t_i)+\rho(\bm{Z}^{s-1}(t_i)-\bm{U}^{s-1}(t_i)).
$$
It is easy to see that
$$
\mathbf{X}(t_i)^T\mathbf{X}(t_i)+\rho\bm{I}={\rm diag}\{\tilde{\mathbf{X}}_{(-u)}(t_i)^T\tilde{\mathbf{X}}_{(-u)}(t_i)+\rho\bm{I}\}_{1\leq u\leq p}={\rm diag}\{(\hat{\bm{\Sigma}}(t_i)+\rho\bm{I})_{(-u,-u)}\}_{1\leq u\leq p},
$$
where $\hat{\bm{\Sigma}}(t_i)$ is the kernel estimate of the covariance matrix as in Section \ref{subsec:loggle}. That is, $\mathbf{X}(t_i)^T\mathbf{X}(t_i)+\rho\bm{I}$ is a block diagonal matrix with $p$ blocks, where the $u$th block, the $(p-1)\times(p-1)$ matrix $(\hat{\bm{\Sigma}}(t_i)+\rho\bm{I})_{(-u,-u)}$, is the matrix $\hat{\bm{\Sigma}}(t_i)+\rho\bm{I}$ with the $u$th row and the $u$th column deleted.
Moreover,
\begin{align*}
\mathbf{X}(t_i)^T\mathbf{Y}(t_i)=&((\tilde{\mathbf{X}}_{(-1)}(t_i)^T\tilde{\bm{X}}_1(t_i))^T,\ldots, (\tilde{\mathbf{X}}_{(-p)}(t_i)^T\tilde{\bm{X}}_p(t_i))^T)^T\\
=&((\hat{\bm{\Sigma}}(t_i)_{(-1,1)})^T,\ldots,(\hat{\bm{\Sigma}}(t_i)_{(-p,p)})^T)^T.
\end{align*}
That is, $\mathbf{X}(t_i)^T\mathbf{Y}(t_i)$ is a $p(p-1)\times1$ column vector consisting of $p$ sub-vectors, where the $u$th sub-vector is the $u$th column of $\hat{\bm{\Sigma}}(t_i)$ with the $u$th element (i.e., the diagonal element) deleted.
Since $\mathbf{X}(t_i)^T\mathbf{X}(t_i)+\rho\bm{I}$ and $\mathbf{X}(t_i)^T\mathbf{Y}(t_i)$ can be decomposed into blocks, $\bm{\beta}^s(t_i)$ can be solved block-wisely:
$$
\bm{\beta}_u^s(t_i)=((\hat{\bm{\Sigma}}(t_i)+\rho\bm{I})_{(-u,-u)})^{-1}(\hat{\bm{\Sigma}}(t_i)_{(-u,u)}+\rho(\bm{Z}_u^{s-1}(t_i)-\bm{U}_u^{s-1}(t_i))),\ u=1,\ldots,p,
$$
where $\bm{\beta}_u^s(t_i)=(\beta_{u1}^s(t_i),\ldots,\beta_{u,u-1}^s(t_i),\ldots,\beta_{u,u+1}^s(t_i),\ldots,\beta_{up}^s(t_i))$ is a $(p-1)\times1$ column vector, $\bm{\beta}^{s}(t_i)=(\bm{\beta}_1^s(t_i)^T,\ldots,\bm{\beta}_p^s(t_i)^T)^T$, and $\bm{Z}_u^{s-1}(t_i)=(Z_{u1}^{s-1}(t_i),\ldots,Z_{up}^{s-1}(t_i))$ and $\bm{U}_u^{s-1}(t_i)=(U_{u1}^{s-1}(t_i),\ldots,U_{up}^{s-1}(t_i))$ contain the corresponding elements in $\bm{Z}^{s-1}(t_i)$ and $\bm{U}^{s-1}(t_i)$, respectively.
Here, we need to solve $p$ linear systems, each with $p$ equations. One way is to conduct Cholesky decompositions of the matrices $(\hat{\bm{\Sigma}}(t_i)+\rho\bm{I})_{(-u,-u)}$, $u=1,\ldots,p$ in advance and use Gaussian elimination to solve the corresponding triangular linear systems. To do this, we apply Cholesky decomposition to $\hat{\bm{\Sigma}}(t_i)+\rho\bm{I}$ followed by $p$ Givens rotations. This has overall time complexity $O(p^3)$, the same as the time complexity of the subsequent $p$ applications of Gaussian eliminations. Note that, if we had performed Cholesky decomposition on each of the $(p-1)\times (p-1)$ matrix directly, the total time complexity would have been $O(p^4)$.
The details of conducting Cholesky decompositions of the matrices $(\hat{\bm{\Sigma}}(t_i)+\rho\bm{I})_{(-u,-u)} (u=1,\ldots,p)$ through Givens rotations are given in \ref{sec:givens_rotation} of the Supplementary Material.
\item
$$\mathbb{Z}_k^{s}=\arg\min_{\mathbb{Z}_k}\left[\frac{\rho}{2}\sum_{i\in \mathcal{N}_{k,d}}||\bm{Z}(t_i)-\bm{\beta}^{s}(t_i)-\bm{U}^{s-1}(t_i)||_2^2+\lambda\sum_{u<v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}\left[Z_{uv}(t_i)^2+Z_{vu}(t_i)^2\right]}\right].$$
For $i\in \mathcal{N}_{k,d}$, $1\leq u\neq v\leq p$, it is easy to see that
\begin{align*}
Z_{uv}^{s}(t_i)=&\left(1-\frac{\lambda}{\rho\sqrt{\sum_{j\in \mathcal{N}_{k,d}}\left[(\beta_{uv}^{s}(t_j)+U_{uv}^{s-1}(t_j))^2+(\beta_{vu}^{s}(t_j)+U_{vu}^{s-1}(t_j))^2\right]}}\right)_+\\
&\cdot(\beta_{uv}^{s}(t_i)+U_{uv}^{s-1}(t_i)).
\end{align*}
\item
For $i\in \mathcal{N}_{k,d}$,
$$\bm{U}^{s}(t_i)=\bm{U}^{s-1}(t_i)+\bm{\beta}^{s}(t_i)-\bm{Z}^{s}(t_i).$$
\end{enumerate}
\subsubsection*{Over-relaxation}
In steps (ii) and (iii), we replace $\bm{\beta}^s(t_i)$ by $\alpha\bm{\beta}^s(t_i)+(1-\alpha)\bm{Z}^{s-1}(t_i)$, where the relaxation parameter $\alpha$ is set to be 1.5. It is suggested in \cite{boyd2011distributed} that over-relaxation with $\alpha\in[1.5, 1.8]$ can improve convergence.
\subsubsection*{Stopping criterion}
The norm of the primal residual at step $s$ is
$||\bm{r}^s||_2=\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{\beta}^s(t_i)-\bm{Z}^s(t_i)||_2^2}$,
and the norm of the dual residual at step $s$ is
$||\bm{d}^s||_2=\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{Z}^s(t_i)-\bm{Z}^{s-1}(t_i)||_2^2}$. Define the feasibility tolerance for the primal as $\epsilon^{pri}=\epsilon^{abs}\sqrt{p|\mathcal{N}_{k,d}|}+\epsilon^{rel}\max\{\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{\beta}^s(t_i)||_2^2},$ $\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{Z}^s(t_i)||_2^2}\}$,
and the feasibility tolerance for the dual as $\epsilon^{dual}=\epsilon^{abs}\sqrt{p|\mathcal{N}_{k,d}|}+\epsilon^{rel}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{U}^s(t_i)||_2^2}$.
Here $\epsilon^{abs}$ is the absolute tolerance and in practice is often set as $10^{-5}$ or $10^{-4}$, and $\epsilon^{rel}$ is the relative tolerance and in practice is often set as $10^{-3}$ or $10^{-2}$.
The stopping criterion is that the algorithm stops if and only if $||\bm{r}^s||_2\leq\epsilon^{pri}$ and $||\bm{d}^s||_2\leq\epsilon^{dual}$.
\begin{comment}
\bigskip
\begin{center}
{\large\bf SUPPLEMENTARY MATERIAL}
\end{center}
\begin{description}
\item[Title:] Brief description. (file type)
\item[R-package for MYNEW routine:] R-package ÒMYNEWÓ containing code to perform the diagnostic methods described in the article. The package also contains all datasets used as examples in the article. (GNU zipped tar file)
\item[HIV data set:] Data set used in the illustration of MYNEW method in Section~ 3.2. (.txt file)
\end{description}
\end{comment}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%----------------- bibliography -------------------
\bibliographystyle{agsm}
\bibliography{loggle_reference}
%%%%%%% supp.
\clearpage
\newpage
\setcounter{page}{1}
\setcounter{section}{0}
\renewcommand{\thesection}{S.\arabic{section}}
\setcounter{subsection}{0}
\renewcommand{\thesubsection}{S.\arabic{section}.\arabic{subsection}}
\setcounter{equation}{0}
\renewcommand{\theequation}{S.\arabic{equation}}
\setcounter{figure}{0}
\renewcommand{\thefigure}{S.\arabic{figure}}
\setcounter{table}{0}
\renewcommand{\thetable}{S.\arabic{table}}
\setcounter{proposition}{0}
\renewcommand{\theproposition}{S.\arabic{proposition}}
\setcounter{lemma}{0}
\renewcommand{\thelemma}{S.\arabic{lemma}}
\setcounter{corollary}{0}
\renewcommand{\thecorollary}{S.\arabic{corollary}}
\begin{center}
{\large\bf Estimating Time-Varying Graphical Models\\
SUPPLEMENTARY MATERIAL}
\end{center}
\section{Algorithm details}
\subsection{ADMM algorithm in likelihood-based \texttt{loggle}}\label{sec:ADMM}
To solve the optimization problem in (\ref{eq:loggle}) using ADMM algorithm, we notice that the problem can be written as
\begin{align*}
&\textrm{minimize }_{\mathbb{\Omega}_k,\mathbb{Z}_k}\sum_{i\in \mathcal{N}_{k,d}}\left[\text{tr}\left(\bm{\Omega}(t_i)\hat{\bm{\Sigma}}(t_i)\right)-\log|\bm{\Omega}(t_i)|\right]
+\lambda\sum_{u\neq v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}Z_{uv}(t_i)^2},\\
&\textrm{subject to }\bm{\Omega}(t_i)-\bm{Z}(t_i)=\bm{0},\;\bm{\Omega}(t_i)\succ\bm{0},\;i\in \mathcal{N}_{k,d},
\end{align*}
where $\mathcal{N}_{k,d}=\{i:|t_i-t_k|\leq d\}$, $\mathbb{\Omega}_k=\{\bm{\Omega}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ and $\mathbb{Z}_k=\{\bm{Z}(t_i)\}_{i\in \mathcal{N}_{k,d}}$. Note $\bm{\Omega}(t_i)=(\Omega_{uv}(t_i))$ and $\bm{Z}(t_i)=(Z_{uv}(t_i))$ are $\mathbb{R}^{p\times p}$ matrices.
The scaled augmented Lagrangian is
\begin{align*}
L_\rho(\mathbb{\Omega}_k,\mathbb{Z}_k,\mathbb{U}_k)=&\sum_{i\in \mathcal{N}_{k,d}}\left[\text{tr}\left(\bm{\Omega}(t_i)\hat{\bm{\Sigma}}(t_i)\right)-\log|\bm{\Omega}(t_i)|\right]
+\lambda\sum_{u\neq v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}Z_{uv}(t_i)^2}\\
&+\frac{\rho}{2}\sum_{i\in \mathcal{N}_{k,d}}||\bm{\Omega}(t_i)-\bm{Z}(t_i)+\bm{U}(t_i)||_F^2,
\end{align*}
where $\mathbb{U}_k=\{\bm{U}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ are dual variables ($\bm{U}(t_i)=(U_{uv}(t_i))\in\mathbb{R}^{p\times p}$).
The ADMM algorithm is as follows. We first initialize $\bm{Z}^{(0)}(t_i)=\bm{0}$, $\bm{U}^{(0)}(t_i)=\bm{0}$, $i\in \mathcal{N}_{k,d}$. We also need to specify $\rho (>0)$, which in practice is recommended to be $\approx\lambda$ \citep{wahlberg2012admm}. For step $s=1,2,\ldots$ until convergence:
\begin{enumerate}[(i)]
\item
For $i\in \mathcal{N}_{k,d}$,
\begin{align*}
\bm{\Omega}^{s}(t_i)=\arg\min_{\bm{\Omega}(t_i)\succ\bm{0}}\left[\text{tr}\left(\bm{\Omega}(t_i)\hat{\bm{\Sigma}}(t_i)\right)-\log|\bm{\Omega}(t_i)|+\frac{\rho}{2}||\bm{\Omega}(t_i)-\bm{Z}^{s-1}(t_i)+\bm{U}^{s-1}(t_i)||_F^2\right].
\end{align*}
Set the derivative to be 0, we have
$$\hat{\bm{\Sigma}}(t_i)-\rho(\bm{Z}^{s-1}(t_i)-\bm{U}^{s-1}(t_i))=\bm{\Omega}^{-1}(t_i)-\rho\bm{\Omega}(t_i).$$
Let $\bm{Q\Lambda Q}^T$ denote the eigen-decomposition of $\hat{\bm{\Sigma}}(t_i)-\rho(\bm{Z}^{s-1}(t_i)-\bm{U}^{s-1}(t_i))$, where $\bm{\Lambda}={\rm diag}\{\lambda_1,\ldots,\lambda_p\}$, then
$$\bm{\Omega}^{s}(t_i)=\bm{Q}\tilde{\bm{\Lambda}}\bm{Q}^T,$$ where $\tilde{\bm{\Lambda}}$ is the diagonal matrix with $j$th diagonal element $\frac{-\lambda_j+\sqrt{\lambda_j^2+4\rho}}{2\rho}$.
\item
$$\mathbb{Z}_k^{s}=\arg\min_{\mathbb{Z}_k}\left[\frac{\rho}{2}\sum_{i\in \mathcal{N}_{k,d}}||\bm{Z}(t_i)-\bm{\Omega}^{s}(t_i)-\bm{U}^{s-1}(t_i)||_F^2+\lambda\sum_{u\neq v}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}Z_{uv}(t_i)^2}\right].$$
For $i\in \mathcal{N}_{k,d}$, it is easy to see that the diagonal elements
$$Z_{uu}^{s}(t_i)=\Omega_{uu}^{s}(t_i)+U_{uu}^{s-1}(t_i),\ u=1,\ldots,p.$$
For the off-diagonal elements, one can show that they should take the form
$$Z_{uv}^{s}(t_i)=\left(1-\frac{\lambda}{\rho\sqrt{\sum_{j\in \mathcal{N}_{k,d}}(\Omega_{uv}^{s}(t_j)+U_{uv}^{s-1}(t_j))^2}}\right)_+(\Omega_{uv}^{s}(t_i)+U_{uv}^{s-1}(t_i)).$$
\item
For $i\in \mathcal{N}_{k,d}$,
$$\bm{U}^{s}(t_i)=\bm{U}^{s-1}(t_i)+\bm{\Omega}^{s}(t_i)-\bm{Z}^{s}(t_i).$$
\end{enumerate}
Note that in Step (i), the positive-definiteness constraint on $\{\bm{\Omega}(t_i)\}_{i\in \mathcal{N}_{k,d}}$ is automatically satisfied by implementing eigen-decomposition.
\subsubsection*{Over-relaxation}
In step (ii) and (iii), we replace $\bm{\Omega}^s(t_i)$ by $\alpha\bm{\Omega}^s(t_i)+(1-\alpha)\bm{Z}^{s-1}(t_i)$, where the relaxation parameter $\alpha$ is set to be 1.5.
%It is suggested in \cite{boyd2011distributed} that over-relaxation with $\alpha\in[1.5, 1.8]$ can improve convergence.
\subsubsection*{Stopping criterion}
The norm of the primal residual at step $s$ is $||\bm{r}^s||_F=\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{\Omega}^s(t_i)-\bm{Z}^s(t_i)||_F^2}$, and the norm of the dual residual at step $s$ is $||\bm{d}^s||_F=\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{Z}^s(t_i)-\bm{Z}^{s-1}(t_i)||_F^2}$. Define the feasibility tolerance for the primal as $\epsilon^{pri}=\epsilon^{abs}\sqrt{p|\mathcal{N}_{k,d}|}+\epsilon^{rel}\max\{\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{\Omega}^s(t_i)||_F^2},$ $\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{Z}^s(t_i)||_F^2}\}$, and the feasibility tolerance for the dual as $\epsilon^{dual}=\epsilon^{abs}\sqrt{p|\mathcal{N}_{k,d}|}+\epsilon^{rel}\sqrt{\sum_{i\in \mathcal{N}_{k,d}}||\bm{U}^s(t_i)||_F^2}$.
Here $\epsilon^{abs}$ is the absolute tolerance and in practice is often set as $10^{-5}$ or $10^{-4}$, and $\epsilon^{rel}$ is the relative tolerance and in practice is often set as $10^{-3}$ or $10^{-2}$. The stopping criterion is that the algorithm stops if and only if $||\bm{r}^s||_F\leq\epsilon^{pri}$ and $||\bm{d}^s||_F\leq\epsilon^{dual}$.
\subsection{ADMM algorithm in pseudo-likelihood-based \texttt{loggle}: additional details}
\subsubsection*{Givens rotation}\label{sec:givens_rotation}
Let $\bm{A}=\hat{\bm{\Sigma}}(t_i)+\rho\bm{I}$, a $p\times p$ positive definite matrix. We aim to efficiently obtain the Cholesky decomposition of $\bm{A}_j$: $\bm{A}_j=\bm{U}_j^T\bm{U}_j$, where $\bm{A}_j$ is a $(p-1)\times(p-1)$ matrix obtained by deleting the $j$th row and the $j$th column of $\bm{A}$, and $\bm{U}_j$ is an upper triangular matrix.
We first apply the Cholesky decomposition to $\bm{A}$: $\bm{A}=\bm{U}^T\bm{U}$, where $\bm{U}$ is an upper triangular matrix. It is easy to see that $\bm{A}_j=\tilde{\bm{U}}_j^T\tilde{\bm{U}}_j$, where $\tilde{\bm{U}}_j$ is a $p\times(p-1)$ matrix from deleting the $j$th column of $\bm{U}$. We then apply the Givens rotation to $\tilde{\bm{U}}_j$ to get its QR decomposition: $\tilde{\bm{U}}_j=\bm{Q}_j\bm{R}_j$, where $\bm{Q}_j$ is a $p\times p$ orthogonal matrix and $\bm{R}_j^T=[\bm{U}_j^T, \bm{0}]$ where $\bm{U}_j$ is a $(p-1)\times(p-1)$ upper triangular matrix. Thus $\bm{A}_j=(\bm{Q}_j\bm{R}_j)^T\bm{Q}_j\bm{R}_j=\bm{R}_j^T\bm{R}_j=\bm{U}_j^T\bm{U}_j$ is exactly the Cholesky decomposition of $\bm{A}_j$.
Note that the cost of Cholesky decomposition is $O(p^3)$ and the cost of QR decomposition by Givens rotation is $O(p^2)$, hence by using the Givens rotation, the total computational complexity for implementing $p$ Cholesky decompositions is $O(p^3)$.
\subsection{De-trend}
\label{sec:detrend}
Our observations $\{\bm{x}_k\}_{k=1,\ldots,N}$ are drawn from $\mathcal{N}_p(\bm{\mu}(t_k),\bm{\Sigma}(t_k))$ ($k=1,\ldots,N$) independently. For simplicity, we can assume that these observations are centered so that each $\bm{x}_k$ is drawn independently from $\mathcal{N}_p(\bm{0},\bm{\Sigma}(t_k))$ ($k=1,\ldots,N$). To achieve this, we first obtain a kernel estimate of the mean function $\hat{\bm{\mu}}(t)=\sum_{j=1}^N\omega_h^{t_j}(t)\bm{x}_j$ by using R package \texttt{sm}, where $\omega_h^{t_j}(t)=\frac{K_h(t_j-t)}{\sum_{j=1}^NK_h(t_j-t)}$ and $K_h(\cdot)$ is a normal kernel function with $h$ being its standard deviation. We then subtract the kernel estimated mean $\hat{\bm{\mu}}(t_k)$ from $\bm{x}_k$ for each $k=1,\ldots,N$.
\section{Model tuning: additional details}
\label{sec:supp_tuning}
\subsection{Early stopping in grid search}\label{sec:thresh_grid}
When the number of nodes $p$ is large, it is often very time consuming to estimate a dense graph. Moreover, when sample size is relatively small, dense models seldom lead to good cross-validation scores (so they will not be selected anyway). Thus, we will fit models for a decreasing sequence of
$\lambda$ and when the edge number of the fitted graph at a $\lambda$ exceeds a prespecified threshold (e.g, 5 times of $p$), we will stop the grid search as smaller values of $\lambda$ usually lead to even denser fitted graphs.
\subsection{Coarse grid search}\label{sec:coarse_grid}
To further reduce the computational cost, we implement a coarse grid search followed by a fine grid search for model tuning. Specifically, we first apply cross-validation on coarse grids to search for appropriate values of $h$, and then we use finer grids to search for good values of $d$ and $\lambda$.
%For example, in $p=500$ time-varying model in simulation experiments, we first apply 5-fold cross-validation on a coarse grid, say, $h=0.15$, 0.2, 0.25, 0.3, $d=0$, 0.01, 0.05, 0.15, 0.25, 1, $\lambda=0.17$, 0.21, 0.25, 0.29, and $K=9$ time points $\{0.02, 0.14, \ldots, 0.86, 0.98\}$, and find appropriate values of $h$ to be 0.25 or 0.3 by calculating cross-validation scores. We then apply 5-fold cross-validation on $h=0.25$ and 0.3 only, with finer grid listed in Section \ref{sec:simulation_setting} to do model selection.
\section{Simulation: additional details}\label{sec:simu_add}
\begin{figure}[H]
\begin{center}
\begin{tabular}{cc}
\includegraphics[width = 2.8in]{ednum_100.pdf} & \includegraphics[width = 2.8in]{ednum_500.pdf}\\
\end{tabular}
\end{center}
\caption{\textbf{Simulation.} Number of edges versus time. Left: $p=100$ time-varying graphs model; Right: $p=500$ time-varying graphs model.}\label{fig:ed_num}
\end{figure}
\section{S\&P 500 Stock Price: additional details}\label{sec:real_data_add}
%\subsection*{Autocorrelation}\label{sec:indep_check}
%To justify the independence of the stock price data between adjacent time points, which is one of the assumptions in time-varying graphical model, we calculate the autocorrelations at time lag 1, 2 and 3 for each stock. The corresponding histograms are shown in Figure \ref{fig:acf}. It is clear that the stock price data are (nearly) independent across time, with almost all the autocorrelation values close to 0.
\begin{comment}
\begin{figure}[H]
\begin{center}
\begin{tabular}{ccc}
\includegraphics[width = 1.8in]{acf_lag1.pdf} & \includegraphics[width = 1.8in]{acf_lag2.pdf} & \includegraphics[width = 1.8in]{acf_lag3.pdf}\\
\end{tabular}
\end{center}
\caption{Histograms of autocorrelations at time lag 1, 2 and 3.}\label{fig:acf}
\end{figure}
\end{comment}
\begin{figure}[H]
\begin{center}
\includegraphics[width = 3in]{acf.pdf}\\
\end{center}
\caption{Boxplots of autocorrelations at time lag 1, 2 and 3.}\label{fig:acf}
\end{figure}
\begin{comment}
\subsection{Marginal correlation plot}
Figure \ref{fig:marcor} is the marginal correlation plot for the stock price dataset. At each time point, the number of stock pairs with absolute marginal correlation values no less than 0.7 is counted and plotted.
\begin{figure}[H]
\begin{center}
\includegraphics[width = 2.5in]{marcor.pdf}\\
\end{center}
\caption{Marginal correlation plot.}\label{fig:marcor}
\end{figure}
\subsection{Model selection results}
We consider three types of tuning parameter configurations as described in Section \ref{sec:simu_add} to make a comprehensive comparison between methods \texttt{loggle}, \texttt{kernel} and \texttt{invariant}. The model selection results are shown in Table \ref{table:real_cv}.
\begin{table}[H]
\begin{center}
\caption{Results of model selection for stock dataset}\label{table:real_cv}
\begin{tabular}{c|ccc|c}
\hline\hline
Method & $h$ & $d$ & $\lambda$ & CV score\\
\hline
\texttt{loggle(I)} & 0.10 & 1 & 0.50 & 130.68\\
\texttt{loggle(II)} & 0.10 & 1 & 0.50(0.14) & 125.35\\
\texttt{loggle(III)} & 0.10 & 0.56(0.43) & 0.52(0.16) & 123.06\\
\texttt{kernel(I)} & 0.15 & 0 & 0.16 & 160.14\\
\texttt{kernel(II)} & 0.15 & 0 & 0.19(0.07) & 154.25\\
\texttt{invar(I)} & 0.10 & 1 & 0.50 & 130.68\\
\texttt{invar(II)} & 0.10 & 1 & 0.50(0.14) & 125.35\\
\hline\hline
\end{tabular}
\end{center}
\end{table}
\end{comment}
\end{document}
\begin{thebibliography}{xx}
\harvarditem{Ahmed \harvardand\ Xing}{2009}{ahmed2009recovering}
Ahmed, A. \harvardand\ Xing, E.~P. \harvardyearleft 2009\harvardyearright ,
`Recovering time-varying networks of dependencies in social and biological
studies', {\em Proceedings of the National Academy of Sciences} {\bf
106}(29),~11878--11883.
\harvarditem{Amadeo}{2017}{amadeo_2017_07}
Amadeo, K. \harvardyearleft 2017\harvardyearright , `Here's how they missed
the early clues of the financial crisis'.
\newline\harvardurl{\url{https://www.thebalance.com/2007-financial-crisis-overview-3306138}}
\harvarditem[Banerjee et~al.]{Banerjee, Ghaoui \harvardand\
d’Aspremont}{2008}{banerjee2008model}
Banerjee, O., Ghaoui, L.~E. \harvardand\ d’Aspremont, A. \harvardyearleft
2008\harvardyearright , `Model selection through sparse maximum likelihood
estimation for multivariate gaussian or binary data', {\em Journal of Machine
learning research} {\bf 9}(Mar),~485--516.
\harvarditem[Boyd et~al.]{Boyd, Parikh, Chu, Peleato \harvardand\
Eckstein}{2011}{boyd2011distributed}
Boyd, S., Parikh, N., Chu, E., Peleato, B. \harvardand\ Eckstein, J.
\harvardyearleft 2011\harvardyearright , `Distributed optimization and
statistical learning via the alternating direction method of multipliers',
{\em Foundations and Trends{\textregistered} in Machine Learning} {\bf
3}(1),~1--122.
\harvarditem[Cai et~al.]{Cai, Liu \harvardand\ Luo}{2011}{cai2011constrained}
Cai, T., Liu, W. \harvardand\ Luo, X. \harvardyearleft 2011\harvardyearright ,
`A constrained ℓ 1 minimization approach to sparse precision matrix
estimation', {\em Journal of the American Statistical Association} {\bf
106}(494),~594--607.
\harvarditem[Danaher et~al.]{Danaher, Wang \harvardand\
Witten}{2014}{danaher2014joint}
Danaher, P., Wang, P. \harvardand\ Witten, D.~M. \harvardyearleft
2014\harvardyearright , `The joint graphical lasso for inverse covariance
estimation across multiple classes', {\em Journal of the Royal Statistical
Society: Series B (Statistical Methodology)} {\bf 76}(2),~373--397.
\harvarditem{Erd{\"o}s \harvardand\ R{\'e}nyi}{1959}{erdos1959random}
Erd{\"o}s, P. \harvardand\ R{\'e}nyi, A. \harvardyearleft
1959\harvardyearright , `On random graphs, i', {\em Publicationes
Mathematicae (Debrecen)} {\bf 6},~290--297.
\harvarditem[Friedman et~al.]{Friedman, Hastie \harvardand\
Tibshirani}{2008}{friedman2008sparse}
Friedman, J., Hastie, T. \harvardand\ Tibshirani, R. \harvardyearleft
2008\harvardyearright , `Sparse inverse covariance estimation with the
graphical lasso', {\em Biostatistics} {\bf 9}(3),~432--441.
\harvarditem[Friedman et~al.]{Friedman, Hastie \harvardand\
Tibshirani}{2010}{friedman2010applications}
Friedman, J., Hastie, T. \harvardand\ Tibshirani, R. \harvardyearleft
2010\harvardyearright , Applications of the lasso and grouped lasso to the
estimation of sparse graphical models, Technical report, Technical report,
Stanford University.
\harvarditem{Gibberd \harvardand\ Nelson}{2014}{gibberd2014high}
Gibberd, A.~J. \harvardand\ Nelson, J.~D. \harvardyearleft
2014\harvardyearright , High dimensional changepoint detection with a dynamic
graphical lasso, {\em in} `Acoustics, Speech and Signal Processing (ICASSP),
2014 IEEE International Conference on', IEEE, pp.~2684--2688.
\harvarditem{Gibberd \harvardand\ Nelson}{2017}{gibberd2017regularized}
Gibberd, A.~J. \harvardand\ Nelson, J.~D. \harvardyearleft
2017\harvardyearright , `Regularized estimation of piecewise constant
gaussian graphical models: The group-fused graphical lasso', {\em Journal of
Computational and Graphical Statistics} (just-accepted).
\harvarditem[Hallac et~al.]{Hallac, Park, Boyd \harvardand\
Leskovec}{2017}{hallac2017network}
Hallac, D., Park, Y., Boyd, S. \harvardand\ Leskovec, J. \harvardyearleft
2017\harvardyearright , `Network inference via the time-varying graphical
lasso', {\em arXiv preprint arXiv:1703.01958} .
\harvarditem[Kolar et~al.]{Kolar, Song, Ahmed \harvardand\
Xing}{2010}{kolar2010estimating}
Kolar, M., Song, L., Ahmed, A. \harvardand\ Xing, E.~P. \harvardyearleft
2010\harvardyearright , `Estimating time-varying networks', {\em The Annals
of Applied Statistics} pp.~94--123.
\harvarditem{Kolar \harvardand\ Xing}{2012}{kolar2012estimating}
Kolar, M. \harvardand\ Xing, E.~P. \harvardyearleft 2012\harvardyearright ,
`Estimating networks with jumps', {\em Electronic journal of statistics} {\bf
6},~2069.
\harvarditem{Lam \harvardand\ Fan}{2009}{lam2009sparsistency}
Lam, C. \harvardand\ Fan, J. \harvardyearleft 2009\harvardyearright ,
`Sparsistency and rates of convergence in large covariance matrix
estimation', {\em Annals of statistics} {\bf 37}(6B),~4254.
\harvarditem{Meinshausen \harvardand\ B{\"u}hlmann}{2006}{meinshausen2006high}
Meinshausen, N. \harvardand\ B{\"u}hlmann, P. \harvardyearleft
2006\harvardyearright , `High-dimensional graphs and variable selection with
the lasso', {\em The annals of statistics} pp.~1436--1462.
\harvarditem[Monti et~al.]{Monti, Hellyer, Sharp, Leech, Anagnostopoulos
\harvardand\ Montana}{2014}{monti2014estimating}
Monti, R.~P., Hellyer, P., Sharp, D., Leech, R., Anagnostopoulos, C.
\harvardand\ Montana, G. \harvardyearleft 2014\harvardyearright ,
`Estimating time-varying brain connectivity networks from functional mri time
series', {\em NeuroImage} {\bf 103},~427--443.
\harvarditem[Peng et~al.]{Peng, Wang, Zhou \harvardand\
Zhu}{2009}{peng2009partial}
Peng, J., Wang, P., Zhou, N. \harvardand\ Zhu, J. \harvardyearleft
2009\harvardyearright , `Partial correlation estimation by joint sparse
regression models', {\em Journal of the American Statistical Association}
{\bf 104}(486),~735--746.
\harvarditem[Peng et~al.]{Peng, Zhu, Bergamaschi, Han, Noh, Pollack
\harvardand\ Wang}{2010}{peng2010regularized}
Peng, J., Zhu, J., Bergamaschi, A., Han, W., Noh, D.-Y., Pollack, J.~R.
\harvardand\ Wang, P. \harvardyearleft 2010\harvardyearright , `Regularized
multivariate regression for identifying master predictors with application to
integrative genomics study of breast cancer', {\em The annals of applied
statistics} {\bf 4}(1),~53.
\harvarditem[Ravikumar et~al.]{Ravikumar, Wainwright, Raskutti, Yu
et~al.}{2011}{ravikumar2011high}
Ravikumar, P., Wainwright, M.~J., Raskutti, G., Yu, B. et~al. \harvardyearleft
2011\harvardyearright , `High-dimensional covariance estimation by minimizing
ℓ1-penalized log-determinant divergence', {\em Electronic Journal of
Statistics} {\bf 5},~935--980.
\harvarditem[Rothman et~al.]{Rothman, Bickel, Levina, Zhu
et~al.}{2008}{rothman2008sparse}
Rothman, A.~J., Bickel, P.~J., Levina, E., Zhu, J. et~al. \harvardyearleft
2008\harvardyearright , `Sparse permutation invariant covariance estimation',
{\em Electronic Journal of Statistics} {\bf 2},~494--515.
\harvarditem[Song et~al.]{Song, Kolar \harvardand\ Xing}{2009}{song2009keller}
Song, L., Kolar, M. \harvardand\ Xing, E.~P. \harvardyearleft
2009\harvardyearright , `Keller: estimating time-varying interactions between
genes', {\em Bioinformatics} {\bf 25}(12),~i128--i136.
\harvarditem[Wahlberg et~al.]{Wahlberg, Boyd, Annergren \harvardand\
Wang}{2012}{wahlberg2012admm}
Wahlberg, B., Boyd, S., Annergren, M. \harvardand\ Wang, Y. \harvardyearleft
2012\harvardyearright , `An admm algorithm for a class of total variation
regularized estimation problems', {\em IFAC Proceedings Volumes} {\bf
45}(16),~83--88.
\harvarditem{Wang \harvardand\ Kolar}{2014}{wang2014inference}
Wang, J. \harvardand\ Kolar, M. \harvardyearleft 2014\harvardyearright ,
`Inference for sparse conditional precision matrices', {\em arXiv preprint
arXiv:1412.7638} .
\harvarditem{Wit \harvardand\ Abbruzzo}{2015}{wit2015inferring}
Wit, E.~C. \harvardand\ Abbruzzo, A. \harvardyearleft 2015\harvardyearright ,
`Inferring slowly-changing dynamic gene-regulatory networks', {\em BMC
bioinformatics} {\bf 16}(6),~S5.
\harvarditem[Witten et~al.]{Witten, Friedman \harvardand\
Simon}{2011}{witten2011new}
Witten, D.~M., Friedman, J.~H. \harvardand\ Simon, N. \harvardyearleft
2011\harvardyearright , `New insights and faster computations for the
graphical lasso', {\em Journal of Computational and Graphical Statistics}
{\bf 20}(4),~892--900.
\harvarditem{Yuan \harvardand\ Lin}{2006}{yuan2006model}
Yuan, M. \harvardand\ Lin, Y. \harvardyearleft 2006\harvardyearright , `Model
selection and estimation in regression with grouped variables', {\em Journal
of the Royal Statistical Society: Series B (Statistical Methodology)} {\bf
68}(1),~49--67.
\harvarditem{Yuan \harvardand\ Lin}{2007}{yuan2007model}
Yuan, M. \harvardand\ Lin, Y. \harvardyearleft 2007\harvardyearright , `Model
selection and estimation in the gaussian graphical model', {\em Biometrika}
{\bf 94}(1),~19--35.
\harvarditem[Zhou et~al.]{Zhou, Lafferty \harvardand\
Wasserman}{2010}{zhou2010time}
Zhou, S., Lafferty, J. \harvardand\ Wasserman, L. \harvardyearleft
2010\harvardyearright , `Time varying undirected graphs', {\em Machine
Learning} {\bf 80}(2-3),~295--319.
\end{thebibliography}