Empirical priors and posterior concentration in a piecewise polynomial sequence model
Extracted source text ยท arXiv 1712.03848
\documentclass[a4paper, 12pt]{article}
\usepackage[sort&compress]{natbib}
\bibpunct{(}{)}{;}{a}{}{,}
\usepackage{amsthm, amsmath, amssymb, mathrsfs, multirow, url, subfigure}
\usepackage{graphicx}
\usepackage{ifthen}
\usepackage{amsfonts}
\usepackage[usenames]{color}
\usepackage{fullpage}
%\usepackage{setspace}
%\RequirePackage[OT1]{fontenc}
%\RequirePackage[colorlinks]{hyperref}
%\RequirePackage{hypernat}
%\numberwithin{equation}{section}
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}
\newtheorem{cor}{Corollary}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\theoremstyle{definition}
\newtheorem{defn}{Definition}
\theoremstyle{remark}
\newtheorem{remark}{Remark}
\newtheorem{example}{Example}
\newcommand{\FF}{\mathbb{F}}
\newcommand{\KK}{\mathbb{K}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\XX}{\mathbb{X}}
\newcommand{\YY}{\mathbb{Y}}
\newcommand{\A}{\mathscr{A}}
\newcommand{\F}{\mathscr{F}}
\newcommand{\X}{\mathscr{X}}
\newcommand{\Y}{\mathscr{Y}}
\newcommand{\M}{\mathcal{M}}
\newcommand{\E}{\mathsf{E}}
\newcommand{\V}{\mathsf{V}}
\newcommand{\prob}{\mathsf{P}}
\newcommand{\aff}{\mathsf{A}}
\newcommand{\eps}{\varepsilon}
\renewcommand{\phi}{\varphi}
\newcommand{\supp}{\mathrm{supp}}
\newcommand{\iid}{\overset{\text{\tiny iid}}{\sim}}
\newcommand{\ind}{\overset{\text{\tiny ind}}{\sim}}
\newcommand{\Bigmid}{\; \Bigl\vert \;}
\newcommand{\bigmid}{\; \bigl\vert \;}
\renewcommand{\L}{\mathcal{L}}
\newcommand{\nm}{\mathsf{N}}
\newcommand{\bet}{\mathsf{Beta}}
\newcommand{\ber}{\mathsf{Ber}}
\newcommand{\bin}{\mathsf{Bin}}
\newcommand{\dpp}{\mathsf{DP}}
\newcommand{\dpm}{\mathsf{DPM}}
\newcommand{\chisq}{\mathsf{ChiSq}}
\newcommand{\unif}{\mathsf{Unif}}
\newcommand{\dir}{\mathsf{Dir}}
\newcommand{\pareto}{\mathsf{Par}}
\newcommand{\Xbar}{\overline{X}}
%\pdfminorversion=4
%\title{Empirical priors and posterior concentration rates for a piecewise constant sequence model}
\title{Asymptotically optimal empirical Bayes inference in a piecewise constant sequence model}
\author{
Ryan Martin\footnote{Department of Statistics, North Carolina State University, {\tt rgmarti3@ncsu.edu}} \quad and \quad Weining Shen\footnote{Department of Statistics, University of California Irvine, {\tt weinings@uci.edu}}
}
\date{\today}
\begin{document}
\maketitle
%\doublspacing
\begin{abstract}
Inference on high-dimensional parameters in structured linear models is an important statistical problem. This paper focuses on the piecewise constant Gaussian sequence model, and we develop a new empirical Bayes solution that enjoys adaptive minimax posterior concentration rates and, thanks to the conjugate form of the empirical prior, relatively simple posterior computations.
\smallskip
\emph{Keywords and phrases:} Adaptive estimation; change-point; high-dimensional; minimax; posterior concentration rate.
\end{abstract}
\section{Introduction}
\label{S:intro}
Consider a Gaussian sequence model
\begin{equation}
\label{eq:model}
Y_i \sim \nm(\theta_i, \sigma^2), \quad i=1,\ldots,n,
\end{equation}
where $Y=(Y_1,\ldots,Y_n)$ are independent, the variance $\sigma^2 > 0$ is known, and inference on the unknown mean vector $\theta = (\theta_1,\ldots,\theta_n)$ is desired. It is common to assume that $\theta$ satisfies a {\em sparsity structure}, i.e., most $\theta_i$'s are zero, and work on these problems goes back at least to \citet{donohojohnstone1994b}, and more recently in \citet{johnstonesilverman2004}, \citet{jiang.zhang.2009}, \citet{castillo.vaart.2012}, \citet{martin.walker.eb}, and \citet{pas.szabo.vaart.rate}.
There has also been recent interest in the {\em piecewise constant structure}, in which there is a simple partition $B$ of the set $\{1,2,\ldots,n\}$ into consecutive blocks $B(s) \subseteq \{1,2,\ldots,n\}$, $s=1,\ldots,|B|$, where
\[ \theta_i \equiv \theta_{B(s)}, \quad i \in B(s), \quad s=1,\ldots,|B|. \]
This structure is parametrized by $(B, \theta_B)$, where $B$ is the blocking configuration and $\theta_B$ is the blocking configuration-specific mean parameters.
%In other words, the blocks $B$ are the pieces or segments where the means are constant, and $\theta_B$ is the $|B|$-vector of constant values.
This piecewise constant sequence model is the canonical example in the change-point literature, e.g., \citet{frick.munk.sieling.2014} and \citet{fryzlewicz2014}. The former paper describes applications of this piecewise constant sequence model in DNA copy number profiling and photo emission spectroscopy, and the latter in finance. A related model is piecewise constant regression, investigated by \citet{hutter2007} and \citet{pas.rockova.2017}, the former cites applications in seismology, tomography, biology, and economics, while the latter focuses primarily on machine learning applications.
Our goal in this paper is to develop a Bayesian or, more precisely, empirical Bayesian approach for posterior inference on $\theta$ in the piecewise constant sequence model. Our proposed empirical prior makes use of genuine prior information that says the vector $\theta$ is not too complex, i.e., that the block configuration $B$ has size $|B|$ relatively small, but is data-dependent, or non-informative, on the actual values $\theta_B$ corresponding to the block configuration. Section~\ref{S:model} describes how data is incorporated in the prior in two ways: the first is via a prior centering and the second is a mild regularization. Our theoretical results in Section~\ref{S:rate} demonstrate that the corresponding empirical Bayes posterior distribution enjoys adaptive concentration at the minimax rate, recently worked out in \citet{gao.etal.piece}, even adjusting to phase transitions. To our knowledge, this is the first adaptive minimax posterior concentration rate result for the piecewise Gaussian sequence model. For the related piecewise constant regression problem, similar results are obtained by \citet{pas.rockova.2017} but ours are stronger and more comprehensive; see Remark~\ref{re:pas.rockova}. Moreover, since the proposed empirical prior for $\theta_B$ are conjugate, the posterior is relatively easy to compute, and numerical examples in Section~\ref{S:examples} show the remarkable accuracy of our posterior inference even in finite samples.
\section{Empirical Bayes model}
\label{S:model}
\subsection{Prior}
\label{SS:prior}
In light of the representation $(B,\theta_B)$ of the mean vector $\theta$ in terms of a block configuration and block-specific parameters, a hierarchical prior is appealing. That is, first specify a prior for $B$, then a conditional prior for $\theta_B$, given $B$. Here we follow this general prior specification strategy, but with a twist to make it an {\em empirical prior}.
Intuitively, there is no reason to introduce a piecewise constant structure if not for the belief that there are not too many constant blocks, i.e., that $|B|$ is relatively small compared to $n$; see Section~\ref{S:rate}. This belief can be incorporated into the prior for $B$ in the following way. Set $b=|B|$, and introduce a marginal prior
\begin{equation}
\label{eq:S.prior}
f_n(b) \propto n^{-\lambda (b-1)}, \quad b=1,\ldots,n,
\end{equation}
where $\lambda > 0$ is a constant to be specified. Note that this is effectively a truncated geometric distribution with parameter $p=n^{-\lambda}$, which puts most of its mass on small values of the block configuration size, hence incorporating the prior information that $\theta$ is not too complex. Next, if the configuration size $b$ is given, the blocks correspond to a simple partition of $\{1,2,\ldots,n\}$ into $b$ consecutive chunks, and there are $\binom{n-1}{b-1}$ such partitions. So, for the conditional prior distribution of $B$, given $|B|$, we can simply take a discrete uniform distribution. Therefore, the prior distribution for $B$ is given by
\begin{equation}
\label{eq:B.prior}
\pi_n(B) = f_n(|B|) \textstyle\binom{n-1}{|B|-1}^{-1},
\end{equation}
where $B$ ranges over all simple partitions of $\{1,2,\ldots,n\}$ into consecutive blocks.
It remains to propose a conditional prior for $\theta_B$, given $B$. Here we take a prior which assigns independent normal distributions to each $\theta_{B(s)}$, $s=1,\ldots,|B|$, but we let the data inform the prior location. According to \citet{martin.walker.eb, martin.walker.deb} and \citet{martin.mess.walker.eb}, this data-driven prior centering reduces the potentially-problematic influence of the prior tails on the asymptotic behavior of the posterior. In this case, define the $B$-specific maximum likelihood estimator $\hat\theta_B = \{\hat\theta_{B(s)}: s=1,\ldots,|B|\}$, where $\hat\theta_{B(s)}$ is the average of $\{Y_i: i \in B(s)\}$.
%given by
%\[ \hat\theta_{B(s)} = \text{average of $\{Y_i: i \in B(s)\}$}, \quad s=1,\ldots,|B|. \]
Then we center the normal prior on $\hat\theta_B$, i.e., the conditional prior for $\theta_B$, given $B$, is given by
\[ \theta_{B(s)} \sim \nm(\hat\theta_{B(s)}, v_{B(s)}), \quad s=1,\ldots,|B|, \quad \text{independent}, \]
where the variance $v_{B(s)} = v|B(s)|^{-1}$ and $v > 0$ is a constant to be specified. Denote the density function, with respect to Lebesgue measure on $\RR^{|B|}$, of this proposed prior distribution for $\theta_B$ as $\pi_n(\cdot \mid B)$. A related empirical prior construction for a monotone density was given in \citet{ebmono}.
As usual, the prior distribution for the mean vector $\theta$ under this hierarchical formulation is the finite mixture of $\pi_n(\theta_B \mid B)$ with respect to the prior distribution $\pi_n(B)$ for the configuration $B$. That mixture prior distribution will be denoted as $\Pi_n$.
The reader may be anticipating that the combination of an empirical prior with the likelihood amounts to double-use of data. To avoid potentially over-fitting, we propose the following very mild additional regularization. Let $L_n(\theta)$ denote the likelihood function based on the model \eqref{eq:model}, i.e., $L_n(\theta) \propto \exp\{-\tfrac{1}{2\sigma^2} \|Y-\theta\|^2\}$, where $\|\cdot\|$ denotes the $\ell_2$-norm on $\RR^n$. For a fixed $\alpha \in (0,1)$, define a regularized empirical prior
\[ \Pi_n^\text{reg}(d\theta) \propto L_n(\theta)^{-(1-\alpha)} \, \Pi_n(d\theta). \]
Dividing by a fractional power of the likelihood effectively down-weights those parameter values that are too consistent with the data, hence discouraging over-fitting. Typically, one would take $\alpha$ to be close to 1---e.g., we take $\alpha=0.99$ in our simulation examples in Section~\ref{S:examples}---so this additional regularization is very mild indeed.
\subsection{Posterior}
\label{SS:posterior}
For the posterior distribution, we proposed to combine the regularized empirical prior $\Pi_n^\text{reg}$ with the likelihood $L_n$ according to Bayes's formula:
\begin{equation}
\label{eq:post0}
\Pi^n(A) \propto \int_A L_n(\theta) \, \Pi_n^\text{reg}(d\theta), \quad A \subseteq \RR^n.
\end{equation}
The following sections investigate the theoretical properties and practical performance of this empirical Bayes posterior distribution.
Of course, the above posterior can be rewritten as
\[ \Pi^n(A) \propto \int_A L_n(\theta)^\alpha \, \Pi_n(d\theta), \]
which is particular well-suited for theoretical analysis; see Appendix~\ref{S:proofs}. This sort of {\em generalized Bayes posterior} has received considerable attention recently, e.g., \citet{grunwald.ommen.scaling}, \citet{miller.dunson.power}, \citet{holmes.walker.scaling}, \citet{syring.martin.scaling}, and \citet{bhat.pati.yang.fractional}, though not specifically for regularization purposes. One might ask if $\alpha=1$ is a valid choice, since this makes the above display look more like the familiar Bayesian update, but the answer is unclear. Likely, plugging in $\alpha=1$, would result in a posterior distribution with the desired theoretical properties, but this does not follow from the analysis presented here, which takes advantages of having $\alpha < 1$. This is not a shortcoming of our approach, we are actually able to improve upon the existing Bayesian rate results for this problem; see Remark~\ref{re:pas.rockova}. While it should be possible to prove results analogous to those in Section~\ref{S:rate} below for the $\alpha=1$ case, such an extension is of little practical consequence. Indeed, there is virtually no effect on numerical results since $\alpha$ can be arbitrarily close to 1,there can be no improvement to the theory since the rates in Theorem~\ref{thm:rate} are optimal, and, finally, the empirical Bayes posterior $\Pi^n$ in \eqref{eq:post0} is already derived by combining the likelihood with an empirical prior that is no more or less justifiable than that with $\alpha=1$.
%Since the power $\alpha$ can be arbitrarily close to 1, there is no practical difference between $\Pi^n$ above and that with $\alpha=1$. %However, the theoretical analysis presented herein does not apply to the $\alpha=1$ case; see below.
%For an empirical Bayes perspective on this, consider rewriting the $\Pi^n$ above as
%\[ \Pi^n(A) \propto \int_A L_n(\theta) \, \frac{\Pi_n(d\theta)}{L_n(\theta)^{1-\alpha}}. \]
%This expression suggests interpreting $\Pi^n$ as the posterior obtained by Bayes's formula applied to the likelihood and a modified empirical prior $\widetilde \Pi_n(d\theta) = \Pi_n(d\theta) / L_n(\theta)^{1-\alpha}$. %Since $\Pi_n$ is an empirical prior anyway, incorporating some additional data dependence into the prior should not be a serious concern.
%The primary motivation for the posterior $\Pi^n$ defined above is the theoretical concentration rate properties it achieves; see Section~\ref{S:rate}. The power $\alpha \in (0,1)$ also makes for simple and elegant proofs. We expect that the same properties would hold for the more traditional $\alpha=1$ model, but this cannot be established using the arguments here. A likely more difficult analysis is needed to get optimal results for the $\alpha=1$ case; compare, e.g., the proof of Theorem~1 in \citet{martin.mess.walker.eb} to that of Theorem~1 in \citet{belitser.ghosal.ebuq}. But the rates presented in Theorem~\ref{thm:rate} below are optimal---in particular, the rates with $\alpha=1$ can be no better---and, since $\alpha$ can be arbitrarily close to 1, there is no practical difference between the $\alpha = 1$ and $\alpha < 1$ cases. Therefore, we see no obvious reason to prefer the former case over the latter.
\section{Posterior concentration rates}
\label{S:rate}
Consider a true piecewise constant mean vector $\theta^\star \in \RR^n$ having block configuration $B_{\theta^\star}$; here and throughout, $B_\theta$ will denote the block configuration of a vector $\theta \in \RR^n$. Define the target rate
\begin{equation}
\label{eq:target}
\eps_n(\theta^\star) = \begin{cases} 1 & \text{if $|B_{\theta^\star}|=1$} \\ |B_{\theta^\star}| \log \frac{en}{|B_{\theta^\star}|} & \text{if $|B_{\theta^\star}| \geq 2$}. \end{cases}
\end{equation}
Note that, in the case $|B_{\theta^\star}|=1$, the model is iid, and the best estimator of the constant vector $\theta^\star$ would be the $n$-vector with each entry equal to $\bar Y$, the sample mean, and its risk is constant, as in \eqref{eq:target}. More generally, according to Corollary~3.1 in \citet{gao.etal.piece}, the rate \eqref{eq:target} is close to the minimax optimal; see Remark~\ref{re:phase} below. Theorem~\ref{thm:rate} says that $\Pi^n$ attains the target rate in \eqref{eq:target}. Proofs are deferred to the appendix.
%In this case, the target rate is unknown because it depends on the unknown $|B_{\theta^\star}|$.
\begin{theorem}
\label{thm:rate}
Under the model \eqref{eq:model}, with known $\sigma^2 > 0$, let $\Pi^n$ be the corresponding empirical Bayes posterior distribution for $\theta \in \RR^n$ described above. If $\eps_n(\theta^\star)$ is the target rate in \eqref{eq:target} and $M_n$ is any sequence satisfying $M_n \to \infty$, then
\[ \sup_{\theta^\star} \E_{\theta^\star} \Pi^n(\{\theta \in \RR^n: \|\theta-\theta^\star\|^2 > M_n \eps_n(\theta^\star)\}) \to 0, \quad n \to \infty, \]
with $\|\cdot\|$ the $\ell_2$-norm on $\RR^n$. If $\theta^\star$ is such that $|B_{\theta^\star}| \geq 2$, the above convergence property holds with $M_n$ replaced by a sufficiently large constant $M > 0$.
\end{theorem}
%\begin{proof}
%See Appendix~\ref{SS:proofs.rate}.
%\end{proof}
The prior described above does not have knowledge of the block configuration size $|B_{\theta^\star}|$, hence the rate is adaptive to the unknown complexity level. %To our knowledge, this is the first result on adaptive posterior concentration for the piecewise constant sequence model.
\begin{remark}
\label{re:pas.rockova}
A result very similar to that in Theorem~\ref{thm:rate} is presented in \citet{pas.rockova.2017}, with a rate of $|B_{\theta^\star}| \log(n / |B_{\theta^\star}|)$. However, translating their notation to ours, they assume bounds on both $\theta^\star$ and on $|B_{\theta^\star}|$, which we do not require. Also, their rates do not detect any phase transitions; see Remark~\ref{re:phase}. It turns out that results similar to that in Theorem~\ref{thm:rate} can be proved with $\alpha=1$ if we assume suitable bounds on $\theta^\star$ but, since we prefer not to make uncheckable assumptions, we opt to take $\alpha < 1$, which is entirely within our control.
\end{remark}
\begin{remark}
\label{re:phase}
The target rate \eqref{eq:target} includes a phase transition, but the optimal rate presented in \citet{gao.etal.piece} actually has two. Their target rate has $\eps_n(\theta^\star) = \log\log n$ for all $\theta^\star$ with $|B_{\theta^\star}| = 2$. Following the proof of Theorem~\ref{thm:rate}, accommodating this phase transition would require $f_n(2) \asymp n (\log n)^{-c}$ for some constant $c > 0$. However, $f_n$ is a probability mass function, so it cannot blow up with $n$ like this. Since the concentration rate proofs for other posterior distributions would follow along similar lines, we expect that our inability to handle this additional phase transition is a general phenomenon for Bayesian approaches to this problem, not unique to our approach here.
\end{remark}
Next, we show that the posterior mean $\hat\theta = \int \theta \, \Pi^n(d\theta)$ is an adaptive, asymptotically minimax estimator.
\begin{theorem}
\label{thm:mean}
Under the setup in Theorem~\ref{thm:rate}, $\sup_{\theta^\star} \eps_n(\theta^\star)^{-1} \E_{\theta^\star}\|\hat\theta - \theta^\star\|^2 \lesssim 1$.
\end{theorem}
%\begin{proof}
%See Appendix~\ref{SS:proofs.mean}
%\end{proof}
In addition to recovery rate results, the complexity of the posterior itself is interesting. Theorem~\ref{thm:dim} says that
%One such result concerns its effective dimension, i.e., that $\Pi^n$ will assign most of its mass to a space of dimension not much larger than $|B_{\theta^\star}|$. In other words,
the effective dimension of the posterior is no larger than a multiple of the true block configuration size, i.e., the posterior is of roughly the correct complexity. \citet{pas.rockova.2017} do not have an analogous complexity result. %comment on the complexity of their posterior.
\begin{theorem}
\label{thm:dim}
Under the setup in Theorem~\ref{thm:rate}, there exists a constant $C > 1$ such that
\[ \sup_{\theta^\star} \E_{\theta^\star} \Pi^n(\{\theta: |B_\theta| > C |B_{\theta^\star}|\}) \to 0, \quad n \to \infty. \]
\end{theorem}
%\begin{proof}
%See Appendix~\ref{SS:proofs.dim}
%\end{proof}
\section{Practical considerations}
\label{S:practical}
This section details several practical considerations pertaining to the proposed empirical Bayes solution. This includes posterior computation, tuning parameter selection, etc.
Genuine Bayesian solutions to high-dimensional problems, ones for which optimal posterior rates are available, tend to be based on non-conjugate priors, making computation non-trivial. Our empirical Bayes solution, on the other hand, is based on a conjugate prior, making computations relatively simple. Indeed, the marginal posterior for $B$ is available in closed-form,
\[ \pi^n(B) \propto \pi_n(B) e^{-\frac{\alpha}{2\sigma^2} \|Y - \hat\theta_B\|^2} \bigl( 1 + \tfrac{v \alpha}{\sigma^2} \bigr)^{-|B|/2}. \]
Sampling from $\pi^n(B)$ via Metropolis--Hastings is easy, as is sampling from the conditional posterior of $\theta_B$, given $B$. The result is posterior samples of $(B, \theta_B)$ from which all sorts of relevant quantities can be evaluated, including the posterior mean $\hat\theta$ for estimation and the marginal posterior for $|B|$ which is relevant for assessing model complexity. We can also extract various types of credible regions for the vector $\theta$. For example, we can easily get marginal credible intervals for each $\theta_i$ based on quantiles of its marginal posterior distribution. R code for carrying out this posterior sampling is available at \url{http://www4.stat.ncsu.edu/~rmartin}.
Next, we need to set the tuning parameters $\alpha$, $v$, and $\lambda$. As mentioned before, it is reasonable to take $\alpha$ close to 1 and our simulations suggest that $\alpha=0.99$ is a reasonable choice. Next, it makes sense to take $v$ to be larger than $\sigma^2$ and, for the examples below, with relatively small $\sigma$, we have found that $v=1$ works well. Finally, $\lambda$ controls the penalty against large $|B|$ and, in the examples considered here, $\lambda=1$ is satisfactory.
As for the model variance, the theory in Section~\ref{S:rate} assumes $\sigma^2$ to be known. However, in real applications, this may not be known and, therefore, must be estimated. Of course, one can take a prior for $\sigma^2$ and get a corresponding joint posterior for $(\theta, \sigma^2)$. Here, in keeping with the empirical Bayes approach, we opt for a plug-in estimator. We consider the estimator
\begin{equation}
\label{eq:variance}
\hat\sigma^2 = \frac{1}{2(n-1)} \sum_{i=1}^{n-1} (Y_{i+1} - Y_i)^2,
\end{equation}
which, as \citet{hutter2007} argues, will work well when $|B|$ is relatively small. Indeed, in the simulated data examples below, this estimator is shown to be quite accurate.
\section{Numerical results}
\label{S:examples}
\begin{example}
\label{ex:one}
First is a standard test example described in \citet{frick.munk.sieling.2014}, p.~561; see, also, \citet{fryzlewicz2014}, Appendix~B(2). This one has $n=497$ mean parameters, but only $|B^\star|=7$ unique values. This true signal is depicted by the thin black line in Figure~\ref{fig:case4}(a). Data $Y$ is sampled from a normal distribution, centered at the true signal, with standard deviation $\sigma=0.2$. Here we use the known variance to construct our posterior, but it is worth pointing out that the estimate \eqref{eq:variance} of the variance in this case is $\hat\sigma^2 = 0.05$, which is very close to the true $\sigma^2=0.04$, so the results would be virtually the same had we used the estimated variance. The empirical Bayes model described above is fit to these data, using $v=1$ and $\lambda=1$, and the posterior mean based on 10,000 Monte Carlo samples from $\Pi^n$ is depicted by the black dots in the figure. The posterior mean does an excellent job estimating the true signal, and the marginal 95\% credible intervals capture the true $\theta$. Note also that the wider blocks have narrower credible intervals, so a sort of ``borrowing information'' is achieved. Figure~\ref{fig:case4}(b) shows the posterior distribution for the block configuration size $|B|$ and, in this case, all the mass is assigned to the true $|B^\star|=7$.
\end{example}
\begin{figure}[t]
\begin{center}
\subfigure[Posterior mean and credible intervals]{\scalebox{0.6}{\includegraphics{case4_mean}}}
\subfigure[Posterior distribution for $|B|$]{\scalebox{0.6}{\includegraphics{case4_size}}}
\end{center}
\caption{Output for Example~\ref{ex:one}. Panel~(a) shows the data (gray dots), true signal (green line), posterior mean (black line), and marginal 95\% credible intervals (red line). Panel~(b) shows the posterior distribution of the block configuration size $|B|$.}
\label{fig:case4}
\end{figure}
\begin{example}
\label{ex:two}
Next, we follow \citet{fan.guan.piece}, Section~6.3, and consider a mean vector of size $n=1000$ with $|B^\star|=20$ unique mean parameters, equally spaced. The specific mean values are randomly sampled from a uniform distribution on $(-2,2)$. Data are simulated from a normal distribution centered at the signal with $\sigma=0.5$. Again, we use the known variance in our construction, but the results using estimated variance, $\hat\sigma^2 = 0.26$, would be indistinguishable from those presented here. Figure~\ref{fig:case4}(b) summarizes the output of the empirical Bayes posterior, again based on $v=1$ and $\lambda=1$. Again, we see that the posterior mean does a good job recovering the true signal, and the marginal 95\% credible intervals cover $\theta$; the credible intervals are all roughly the same width here because of the equally spaced structure. Finally, according to Panel~(b), the posterior for $|B|$ concentrates just below the true $|B^\star|=20$.
\end{example}
\begin{figure}[t]
\begin{center}
\subfigure[Posterior mean and credible intervals]{\scalebox{0.6}{\includegraphics{case1_mean}}}
\subfigure[Posterior distribution for $|B|$]{\scalebox{0.6}{\includegraphics{case1_size}}}
\end{center}
\caption{Output for Example~\ref{ex:two}. Panel~(a) shows the data (gray dots), true signal (green line), posterior mean (black line), and marginal 95\% credible intervals (red line). Panel~(b) shows the posterior distribution of the block configuration size $|B|$.}
\label{fig:case1}
\end{figure}
\begin{example}
\label{ex:copy}
Finally, we consider a real data example following up on the DNA copy number data analysis in \citet{hutter2007}. In these applications, it is of biological importance to identify the change points, hence the proposed method would be of practical use. Data on the copy number for a particular gene are displayed in Figure~\ref{fig:copy}(a). We fit the proposed empirical Bayes model to these data, using the plug-in estimator for the model variance, which in this case is $\hat\sigma^2 = 0.093$, just like in Table~2 of \citet{hutter2007}. Plots of the posterior mean estimate and marginal 95\% credible intervals are also shown. The fit here appears to be quite good, perhaps with the exception of around $\text{Index}=600$, and arguably the reason for this is the within-group variance seems to be much larger here than in other regions. Interestingly, the distribution of $|B|$ in Panel~(B) is concentrated on much smaller values than in \citet{hutter2007}, who estimates about 15 piecewise constant blocks. But a visual inspection of the data suggests much fewer blocks, and roughly 6--7 seems much more reasonable than 15.
\end{example}
\begin{figure}[t]
\begin{center}
\subfigure[Posterior mean and credible intervals]{\scalebox{0.6}{\includegraphics{copynum_mean}}}
\subfigure[Posterior distribution for $|B|$]{\scalebox{0.6}{\includegraphics{copynum_size}}}
\end{center}
\caption{Output for Example~\ref{ex:copy}. Panel~(a) shows the data (gray dots), posterior mean (black line), and marginal 95\% credible intervals (red line). Panel~(b) shows the posterior distribution of the block configuration size $|B|$.}
\label{fig:copy}
\end{figure}
%\begin{figure}[t]
%\begin{center}
%\subfigure[Data and mean]{\scalebox{0.55}{\includegraphics{case1_mean}}}
%\subfigure[Block configuration size]{\scalebox{0.55}{\includegraphics{case1_size}}}
%\end{center}
%\caption{Summary of the posterior output for the $n=1000$ example. Panel~(a) shows the data (gray dots), true signal (thin black line), and posterior mean (black dots). Panel~(b) shows the posterior distribution of the block configuration size $|B|$.}
%\label{fig:case1}
%\end{figure}
\section{Discussion}
\label{S:discuss}
This paper presents an empirical Bayes approach for inference on a high-dimensional normal mean vector under a piecewise constant constraint. We show that the posterior adaptively achieves the asymptotic minimax concentration rate under weaker conditions than those assumed in \citet{pas.rockova.2017} for a related piecewise constant regression model. Moreover, by centering the prior distribution on the data in a suitable way, we can take its shape to be a conjugate normal, which makes computation relatively simple compared to that with a fixed-center but heavy-tailed prior.
An interesting possible extension of the work here is related to the formulation in \citet{fan.guan.piece}. Consider a graph $G=(V,E)$ and, at each vertex $i \in V$, there is a response $Y_i \sim \nm(\theta_i^\star, \sigma^2)$, but only a small number of edges $(i,j) \in E$ have $\theta_i^\star \neq \theta_j^\star$. That paper derives minimax bounds on the recovery rate that are analogous to those achieved here in the sequence model. The only obstacle preventing us from extending our analysis to this more general graph setting is the need to assign a prior distribution for the block structure $B$ in this more complex setting. For example, in a two-dimensional lattice graph, as might be used in imaging applications, one would need a prior on all possible ways that the lattice can be carved up into connected chunks, which is non-trivial. But given such a prior, the theoretical results described here would carry over directly.
Another interesting and practically important extension is to the case where the mean vector is monotone in addition to piecewise constant. In such a case, our proposed prior for the block configuration $B$ is fine, but a prior for $\theta_B$ that respects the monotonicity is not so straightforward. A natural idea is to project the prior presented in Section~\ref{S:model} onto the cone of monotone sequences, but this projection forces positive prior mass on the boundary of the cone which introduces some additional technicalities that affect the convergence rate proofs. We hope to present results on this elsewhere.
\section*{Acknowledgment}
This work is partially supported by the National Science Foundation, DMS--1737933, and the Simons Foundation Award 512620. The authors thank Marcus Hutter for sharing the DNA copy number data analyzed in Example~\ref{ex:copy} above.
\appendix
\section*{A.\quad Proofs}
\label{S:proofs}
\subsection*{A.0.\quad Preliminary results}
For our theoretical analysis, it will help to rewrite the posterior distribution $\Pi^n$ as the ratio $\Pi^n(A) = N_n(A) / D_n$, where the numerator and denominator are
\begin{align*}
N_n(A) & = \sum_B \pi_n(B) \int_{A \cap \Theta_B} R_n(\theta_B^+)^\alpha \, \pi_n(\theta_B \mid B) \,d\theta_B, \quad A \subseteq \RR^n, \\
D_n & = \sum_B \pi_n(B) \int_{\Theta_B} R_n(\theta_B^+)^\alpha \, \pi_n(\theta_B \mid B) \,d\theta_B,
\end{align*}
$R_n(\theta) = L_n(\theta)/L_n(\theta^\star)$ is the likelihood ratio, $\Theta_B \subseteq \RR^{|B|}$ consists of all $|B|$-vectors $\theta_B=\{\theta_{B(s)}: s=1,\ldots,|B|\}$ corresponding to block configuration $B$, and $\theta_B^+$ is the $n$-vector that satisfies $(\theta_B^+)_i = \theta_{B(s)}$ for $i \in B(s)$, with $s=1,\ldots,|B|$. Since $\theta^\star$ is fixed in our calculations, we will abbreviate $B_{\theta^\star}$ by $B^\star$.
%The first step is to suitably lower-bound the denominator $D_n$, and the following lemma takes care of this.
\begin{lemma}
\label{lem:den}
There exists $c > 0$ such that $D_n \gtrsim \pi_n(B^\star) e^{-c|B^\star|}$ for all large $n$.
\end{lemma}
\begin{proof}
Define the set $\L_{n, B^\star} = \{\theta_{B^\star} \in \Theta_{B^\star}: L_n(\theta_{B^\star}^+) \geq e^{-|B^\star|} L_n(\hat\theta_{B^\star}^+)\}$. Then
\begin{align*}
D_n %& > \pi_n(B^\star) \int_{\Theta_{B^\star}} R_n(\theta_{B^\star}^+)^\alpha \pi_n(\theta_{B^\star} \mid B^\star) \,d\theta_{B^\star} \\
%& \geq \pi_n(B^\star) \int_{\Theta_{B^\star}} \Bigl\{ \frac{L_n(\theta_{B^\star}^+)}{L_n(\hat\theta_{B^\star}^+)} \Bigr\}^\alpha \pi_n(\theta_{B^\star} \mid B^\star) \,d\theta_{B^\star} \\
& > \pi_n(B^\star) \int_{\L_{n,B^\star}} \Bigl\{ \frac{L_n(\theta_{B^\star}^+)}{L_n(\hat\theta_{B^\star}^+)} \Bigr\}^\alpha \pi_n(\theta_{B^\star} \mid B^\star) \,d\theta_{B^\star} \\
& \geq \pi_n(B^\star) e^{-\alpha |B^\star|} \pi_n(\L_{n,B^\star} \mid B^\star).
\end{align*}
Under the empirical prior for $\theta_{B^\star}$, $Z := \frac1v \sum_{s=1}^{|B^\star|} |B^\star(s)| (\theta_{B^\star(s)} - \hat\theta_{B^\star(s)})^2$ has a chi-square distribution with $|B^\star|$ degrees of freedom, and the event $\{\theta_B \in \L_{n,B^\star}\}$ is precisely $\{Z \leq 2|B^\star|\}$. Using standard bounds on the chi-square distribution function, and Stirling's approximation of the gamma function, the claim follows, with $c = \alpha + \frac12 - \log 2$.
\end{proof}
Next, for the numerator, we consider a particular sequence of subsets, namely,
\[ A_n = A_{M\eps_n} = \{\theta \in \RR^n: \|\theta-\theta^\star\|^2 > M\eps_n\}, \]
where $M > 0$ is a sufficiently large constant, and $\eps_n$ is the target rate.
%Then the following lemma upper-bounds the numerator along this sequence of subsets.
%The argument here is different from that in \citet{martin.walker.deb} in the sense that, although we can borrow their general method of proof, we do not require a fractional power on the likelihood.
\begin{lemma}
\label{lem:num}
Take $q > 1$ such that $\alpha q < 1$. Then $\E_{\theta^\star}\{N_n(A_n)\} \lesssim e^{-M k \eps_n}$, for all large $n$, where $k = \alpha(1-\alpha q) / 2\sigma^2$
\end{lemma}
\begin{proof}
Towards an upper bound, we interchange expectation with the finite sum over $B$ and the integral over $\theta_B$, the latter step justified by Tonelli's theorem, so that
\begin{equation}
\label{eq:num.expectation}
\E_{\theta^\star}\{N_n(A_n)\} = \sum_B \pi_n(B) \int_{\Theta_B} \E_{\theta^\star}\{ R_n(\theta_B^+)^\alpha \pi_n(\theta_B \mid B)\} \, d\theta_B.
\end{equation}
Next, we work with each of the $B$-dependent integrands separately. Take an arbitrary $p > 1$ and set $q=p/(p-1)$ to be its H\"older conjugate. Then H\"older's inequality gives
\[ \E_{\theta^\star}\{R_n(\theta_B^+)^\alpha \, \pi_n(\theta_B \mid B)\} \leq \E_{\theta^\star}^{1/q}\{R_n(\theta_B^+)^{\alpha q}\} \, \E_{\theta^\star}^{1/p}\{ \pi_n(\theta_B \mid B)^p \}. \]
On the set $A_n$, since $\alpha q < 1$, the first term above is uniformly bounded by $e^{-M k \eps_n}$. To see this, note that, for a general $\theta \in A_n$, if $p_\theta^n$ denotes the joint density of $Y$ under \eqref{eq:model}, and $D_{\alpha q}$ the R\'enyi $\alpha q$-divergence of one normal distribution from another \citep[e.g.,][p.~3800]{vanerven.renyi}, then
\begin{align*}
\E_{\theta^\star}\{R_n(\theta)^{\alpha q}\} & = \int \{ p_\theta^n(y) \}^{\alpha q} \, \{ p_{\theta^\star}^n(y) \}^{1-\alpha q} \,dy
%& = e^{-(1-\alpha q) D_{\alpha q}(p_\theta^n, p_{\theta^\star}^n)} \\
= e^{-\frac{\alpha q(1-\alpha q)}{2\sigma^2} \|\theta-\theta^\star\|^2}.
\end{align*}
For the second term, we show that it simplifies to a suitable constant times a normal density function in $\theta_B$. Using the fact that $\hat\theta_{B(s)}$ is normally distributed, a simple-but-tedious moment generating function calculation gives
\[ \E_{\theta^\star}^{1/p}\{\pi_n(\theta_B \mid B)^p\} = \Bigl( \frac{v}{v + p\sigma^2} \Bigr)^{\frac{|B|}{2}(\frac1p - 1)} \prod_{s=1}^{|B|} \nm(\theta_{B(s)} \mid \E_{\theta^\star} \hat\theta_{B(s)}, (v + p\sigma^2)|B(s)|^{-1}), \]
and the latter product is a normal density in $\theta_B$. Integrating the upper bound with respect to $\theta_B$ gives
\[ \E_{\theta^\star}\{N_n(A_n)\} \leq e^{-M k \eps_n} \sum_B \zeta^{|B|} \pi_n(B) = e^{-M k \eps_n} \sum_{b=1}^n \zeta^b f_n(b), \]
where $\zeta = (1 + p\sigma^2/v)^{(p-1)/2p} > 1$. Using the formula \eqref{eq:S.prior} for $f_n$ and standard properties of a geometric series, it is easy to see that, for all large $n$, the summation term in the above upper bound is uniformly bounded in $n$, proving the claim.
\end{proof}
\subsection*{A.1.\quad Proof of Theorem~\ref{thm:rate}}
\label{SS:proofs.rate}
Then the proof of Theorem~\ref{thm:rate} follows by simply combining the bounds in Lemmas~\ref{lem:den}--\ref{lem:num}. Since the lower bound on $D_n$ in Lemma~\ref{lem:den} is non-stochastic, we have
\[ \E_{\theta^\star}\{\Pi^n(A_n)\} \leq \frac{e^{c|B^\star|}}{\pi_n(B^\star)} \E_{\theta^\star}\{N_n(A_n)\}. \]
Plug in the bound from Lemma~\ref{lem:num}, with $M=M_n$ to get
\[ \E_{\theta^\star}\{\Pi^n(A_n)\} \lesssim e^{-M_n k \eps_n + c|B^\star|} \frac{\binom{n-1}{|B^\star|-1}}{f_n(|B^\star|)}. \]
If $|B_{\theta^\star}|=1$, then both $\eps_n$ and the ratio in the above display are constant. Therefore, the upper bound vanishes if $M_n \to \infty$. On the other hand, if $|B_{\theta^\star}| \geq 2$, then $\eps_n$ is diverging. Also, using the formula for $f_n(|B^\star|)$ in \eqref{eq:S.prior} and the standard bound, $\binom{n}{b} \leq e^{b \log(en/b)}$, on the binomial coefficient, we get
\[ \E_{\theta^\star}\{\Pi^n(A_n)\} \lesssim e^{-M_n k \eps_n + \eps_n + \lambda |B^\star| \log n + c|B^\star|}. \]
Since $|B^\star| \log n$ is roughly $\eps_n + |B^\star| \log |B^\star| = O(\eps_n)$, it follows that, for fixed $M_n$ sufficiently large, the upper bound above vanishes as $n \to \infty$, as was to be proved.
\subsection*{A.2.\quad Proof of Theorem~\ref{thm:mean}}
\label{SS:proofs.mean}
Start with $\int \|\theta-\theta^\star\|^2 \, \Pi^n(d\theta)$. Write $\RR^n$ as $A \cup A^c$, where $A=A_{M\eps_n}$ is as defined above. Then
\begin{equation}
\label{eq:mean.bound}
\E_{\theta^\star} \int \|\theta-\theta^\star\|^2 \, \Pi^n(d\theta) \leq M \eps_n + \E_{\theta^\star} \int_A \|\theta-\theta^\star\|^2 \, \Pi^n(d\theta).
\end{equation}
That remaining integral can be expressed as a ratio of numerator to denominator, where the denominator $D_n$ is just as in Lemma~\ref{lem:den} and the numerator $\widetilde N_n(A)$ is
\begin{align*}
\widetilde N_n(A) & = \int_A \|\theta-\theta^\star\|^2 \, R_n(\theta)^\alpha \, \Pi_n(d\theta) \\
& = \sum_B \pi_n(B) \int_{A \cap \Theta_B} \|\theta_B^+-\theta^\star\|^2 \, R_n(\theta_B^+)^\alpha \, \pi_n(\theta_B \mid B) \,d\theta_B.
\end{align*}
Take expectation of the numerator to the inside of the integral and apply H\"older's inequality just like in the proof of Lemma~\ref{lem:num}. This gives the following upper bound on each $B$-specific integral:
\[ \int_{A \cap \Theta_B} \|\theta-\theta^\star\|^2 e^{-h\|\theta-\theta^\star\|^2} \E_{\theta^\star}^{1/p} \{ \pi_n(\theta_B \mid B)^p\} \, d\theta_B, \]
where $h > 0$ is a constant that depends only on $\alpha$, $\sigma^2$, and the H\"older constant $q > 1$. Since the function $x \mapsto x e^{-h x}$ is eventually monotone decreasing, for sufficiently large $M$ we get a trivial upper bound on the above display, i.e.,
\[ M \eps_n e^{-Mh \eps_n} \int_{\Theta_B} \E_{\theta^\star}^{1/p} \{ \pi_n(\theta_B \mid B)^p\} \, d\theta_B. \]
The same argument as above bounds the remaining integral by $\zeta^{|B|}$, and the prior $\pi_n(B)$ takes care of its contribution. Making $M$ large enough in $e^{-Mh\eps_n}$ will also take care of the bound on $D_n$ from Lemma~\ref{lem:den}. Therefore, the second term on the right-hand side of \eqref{eq:mean.bound} is also bounded by a multiple of $\eps_n$. Finally, Jensen's inequality gives $\|\hat\theta - \theta^\star\|^2 \leq \int \|\theta-\theta^\star\|^2\, \Pi^n(d\theta)$, and the claim follows.
\subsection*{A.3.\quad Proof of Theorem~\ref{thm:dim}}
\label{SS:proofs.dim}
The lower bound on the denominator in Lemma~\ref{lem:den} holds here, and we can employ a similar argument to that in Lemma~\ref{lem:num} to upper-bound the numerator. Indeed, we need to upper-bound
\[ \sum_{B: |B| > C|B^\star|} \pi_n(B) \int_{\Theta_B} \E_{\theta^\star} \{R_n(\theta_B^+)^\alpha \pi_n(\theta_B \mid B)\} \,d\theta_B. \]
The same H\"older's inequality approach leads to an upper bound like $\zeta^{|B|}$. Then
\[ \E_{\theta^\star} \Pi^n(\{\theta: |B_\theta| > C |B^\star|\}) \lesssim \frac{e^{c|B^\star|}}{\pi_n(B^\star)} \sum_{b=C|B^\star|+1}^n \zeta^b f_n(b). \]
From \eqref{eq:S.prior}, factor out a common $(\zeta n^{-\lambda})^{C|B^\star|}$ from the summation, which will be the dominant term. Indeed, like in the proof of Theorem~\ref{thm:rate}, the ratio in the above display is of order $e^{\eps_n + \lambda |B^\star|\log n}$. Then the right-hand side above is of order
$\exp\{\eps_n + \lambda |B^\star|\log n + C|B^\star|\log \zeta - C\lambda |B^\star|\log n\}$.
For sufficiently large $C > 1$, the negative term dominates, so the upper bound vanishes, proving the claim.
\ifthenelse{1=1}{
\bibliographystyle{apalike}
\bibliography{/Users/rgmarti3/Dropbox/Research/mybib}
}{
\begin{thebibliography}{}
\bibitem[Abramovich et~al., 2006]{abramovich2006}
Abramovich, F., Benjamini, Y., Donoho, D.~L., and Johnstone, I.~M. (2006).
\newblock Adapting to unknown sparsity by controlling the false discovery rate.
\newblock {\em Ann. Statist.}, 34(2):584--653.
\bibitem[Belitser and Ghosal, 2017]{belitser.ghosal.ebuq}
Belitser, E. and Ghosal, S. (2017).
\newblock Empirical {B}ayes oracle uncertainty quantification.
\newblock Unpublished manuscript,
\url{http://www4.stat.ncsu.edu/~ghoshal/papers/oracle_regression.pdf}.
\bibitem[Bhattacharya et~al., 2017]{bhat.pati.yang.fractional}
Bhattacharya, A., Pati, D., and Yang, Y. (2017).
\newblock {B}ayesian fractional posteriors.
\newblock Unpublished manuscript, {\tt arXiv:1611.01125}.
\bibitem[Castillo and van~der Vaart, 2012]{castillo.vaart.2012}
Castillo, I. and van~der Vaart, A. (2012).
\newblock Needles and straw in a haystack: posterior concentration for possibly
sparse sequences.
\newblock {\em Ann. Statist.}, 40(4):2069--2101.
\bibitem[Donoho and Johnstone, 1994]{donohojohnstone1994b}
Donoho, D.~L. and Johnstone, I.~M. (1994).
\newblock Minimax risk over {$l\sb p$}-balls for {$l\sb q$}-error.
\newblock {\em Probab. Theory Related Fields}, 99(2):277--303.
\bibitem[Fan and Guan, 2017]{fan.guan.piece}
Fan, Z. and Guan, L. (2017).
\newblock Approximate $\ell_0$-penalized estimation of piecewise constant
signals on graphs.
\newblock {\em Ann. Statist.}, to appear; {\tt arXiv:1703.01421}.
\bibitem[Frick et~al., 2014]{frick.munk.sieling.2014}
Frick, K., Munk, A., and Sieling, H. (2014).
\newblock Multiscale change point inference.
\newblock {\em J. R. Stat. Soc. Ser. B. Stat. Methodol.}, 76(3):495--580.
\newblock With 32 discussions by 47 authors and a rejoinder by the authors.
\bibitem[Fryzlewicz, 2014]{fryzlewicz2014}
Fryzlewicz, P. (2014).
\newblock Wild binary segmentation for multiple change-point detection.
\newblock {\em Ann. Statist.}, 42(6):2243--2281.
\bibitem[Gao et~al., 2017]{gao.etal.piece}
Gao, C., Han, F., and Zhang, C.-H. (2017).
\newblock Minimax risk bounds for piecewise constant models.
\newblock Unpublished manuscript, {\tt arXiv:1705.06386}.
\bibitem[Gr\"unwald and van Ommen, 2017]{grunwald.ommen.scaling}
Gr\"unwald, P. and van Ommen, T. (2017).
\newblock Inconsistency of {B}ayesian inference for misspecified linear models,
and a proposal for repairing it.
\newblock {\em Bayesian Anal.}, to appear; {\tt arXiv:1412.3730}.
\bibitem[Holmes and Walker, 2017]{holmes.walker.scaling}
Holmes, C.~C. and Walker, S.~G. (2017).
\newblock Assigning a value to a power likelihood in a general {B}ayesian
model.
\newblock {\em Biometrika}, 104(2):497--503.
\bibitem[Jiang and Zhang, 2009]{jiang.zhang.2009}
Jiang, W. and Zhang, C.-H. (2009).
\newblock General maximum likelihood empirical {B}ayes estimation of normal
means.
\newblock {\em Ann. Statist.}, 37(4):1647--1684.
\bibitem[Johnstone and Silverman, 2004]{johnstonesilverman2004}
Johnstone, I.~M. and Silverman, B.~W. (2004).
\newblock Needles and straw in haystacks: empirical {B}ayes estimates of
possibly sparse sequences.
\newblock {\em Ann. Statist.}, 32(4):1594--1649.
\bibitem[Martin, 2017]{ebmono}
Martin, R. (2017).
\newblock Empirical priors and posterior concentration rates for a monotone
density.
\newblock Unpublished manuscript, {\tt arXiv:1706.08567}.
\bibitem[Martin et~al., 2017]{martin.mess.walker.eb}
Martin, R., Mess, R., and Walker, S.~G. (2017).
\newblock Empirical {B}ayes posterior concentration in sparse high-dimensional
linear models.
\newblock {\em Bernoulli}, 23(3):1822--1847.
\bibitem[Martin and Walker, 2014]{martin.walker.eb}
Martin, R. and Walker, S.~G. (2014).
\newblock Asymptotically minimax empirical {B}ayes estimation of a sparse
normal mean vector.
\newblock {\em Electron. J. Stat.}, 8(2):2188--2206.
\bibitem[Martin and Walker, 2017]{martin.walker.deb}
Martin, R. and Walker, S.~G. (2017).
\newblock Empirical priors for target posterior concentration rates.
\newblock Unpublished manuscript, {\tt arXiv:1604.05734}.
\bibitem[Miller and Dunson, 2015]{miller.dunson.power}
Miller, J.~W. and Dunson, D.~B. (2015).
\newblock Robust {B}ayesian inference via coarsening.
\newblock Unpublished manuscript, {\tt arXiv:1506.06101}.
\bibitem[van~der Pas et~al., 2017]{pas.szabo.vaart.rate}
van~der Pas, S., Szab\'o, B., and van~der Vaart, A. (2017).
\newblock Adaptive posterior contraction rates for the horseshoe.
\newblock {\em Electron. J. Stat.}, 11(2):3196--3225.
\bibitem[van Erven and Harremo\"es, 2014]{vanerven.renyi}
van Erven, T. and Harremo\"es, P. (2014).
\newblock R\'enyi divergence and {K}ullback-{L}eibler divergence.
\newblock {\em IEEE Trans. Inform. Theory}, 60(7):3797--3820.
\end{thebibliography}
}
\end{document}
\begin{lem}
\label{lem:num}
Suppose that the prior and model variance satisfy $v < 2\sigma^2$. Then there exists a constant $p > 1$, such that
\[ \E_{\theta^\star}\{N_n(A_n)\} \lesssim e^{-(M/q) \eps_n + (G/p) \|\theta^\star\|^2}, \]
where $G$ depends only on $(p,\sigma^2)$ and $q=p/(p-1)$ is the H\"older conjugate of $p$.
\end{lem}
\begin{proof}
The goal is to upper-bound $\E_{\theta^\star}\{N_n(A_n)\}$. Towards this, we interchange expectation with the finite sum over $B$ and the integral over $\theta_B$, the latter step justified by Tonelli's theorem, so that
\begin{equation}
\label{eq:num.expectation}
\E_{\theta^\star}\{N_n(A_n)\} = \sum_B \pi_n(B) \int_{\Theta_B} \E_{\theta^\star}\{ R_n(\theta_B^+) \pi_n(\theta_B \mid B)\} \, d\theta_B.
\end{equation}
Next, we work with each of the $B$-dependent integrands separately. For a $p > 1$ indicated in the lemma statement, set $\alpha=(p-1)/2p \in (0,1)$, and write
\[ \E_{\theta^\star}\{R_n(\theta_B^+) \pi_n(\theta_B \mid B)\} = \E_{\theta^\star} \Bigl\{ R_n(\theta_B^+)^\alpha \, \frac{\pi_n(\theta_B \mid B)}{R_n(\theta_B^+)^{1-\alpha}} \Bigr\}. \]
For $q=p/(p-1)$ the H\"older conjugate of $p$, it follows by definition of $\alpha$ that $\alpha q = \frac12$. Applying H\"older's inequality gives
\[ \E_{\theta^\star}\{R_n(\theta_B^+) \, \pi_n(\theta_B \mid B)\} \leq \E_{\theta^\star}^{1/q}\{R_n(\theta_B^+)^{1/2}\} \, \E_{\theta^\star}^{1/p}\Bigl\{\frac{\pi_n(\theta_B \mid B)^p}{R_n(\theta_B^+)^{p(1-\alpha)}} \Bigr\}. \]
On the set $A_n$ above, the first term above is uniformly bounded by $e^{-(M/8q) \eps_n}$. To see this, note that, for a general $\theta \in A_n$, if $p_\theta^n$ denotes the joint density of $Y$ under \eqref{eq:model}, and $H$ denotes Hellinger distance, then
\[ \E_{\theta^\star}\{R_n(\theta)^{1/2}\} = \int \{ p_\theta^n(y) \, p_{\theta^\star}^n(y) \}^{1/2} \,dy = 1 - H^2(p_\theta^n, p_{\theta^\star}^n) = e^{-\frac18 \|\theta-\theta^\star\|^2}. \]
For the second term, we have
\[ R_n(\theta_B^+) = e^{\frac{1}{2\sigma^2} \{ \|\hat\theta_B^+ - \theta_B^+\|^2 - \|\hat\theta_B^+ - \theta^\star\|^2 + 2 \langle Y-\hat\theta_B^+, \theta^\star - \theta_B^+ \rangle\}} \]
and, therefore,
\begin{align*}
\frac{\pi_n(\theta_B)^p}{R_n(\theta_B^+)^{p(1-\alpha)}} & = e^{-\frac{p(1-\alpha)}{2\sigma^2} \{\|\hat\theta_B^+ - \theta_B^+\|^2 - \|\hat\theta_B^+ - \theta^\star\|^2 + 2 \langle Y-\hat\theta_B^+, \theta^\star - \theta_B^+ \rangle\}} \pi_n(\theta_B \mid B)^p \\
& \leq \Bigl( \frac{1}{2\pi v} \Bigr)^{\frac{p|B|}{2}} e^{\frac{p(1-\alpha)}{\sigma^2} \langle Y-\hat\theta_B^+, \theta^\star - \theta_B^+ \rangle } e^{-\frac{p}{2} \{ \frac{1}{v} - \frac{1-\alpha}{\sigma^2}\}\|\theta_B^+ - \hat\theta_B^+\|^2},
\end{align*}
where the inequality follows since $\|\hat\theta_B^+ - \theta^\star\|^2 \geq 0$. The key point is that the two exponential terms, the latter a function of a $B$-specific mean, $\hat\theta_B^+$ and the former a function of the corresponding residuals, $Y-\hat\theta_B^+$, are {\em independent}. Therefore, the expectation of the product is the product of expectations, i.e.,
\[ \E_{\theta^\star}\Bigl\{\frac{\pi_n(\theta_B \mid B)^p}{R_n(\theta_B^+)^{p(1-\alpha)}} \Bigr\} \leq \Bigl( \frac{1}{2\pi v} \Bigr)^{\frac{p|B|}{2}} \E_{\theta^\star}\bigl\{ e^{\frac{p(1-\alpha)}{\sigma^2} \langle Y-\hat\theta_B^+, \theta^\star - \theta_B^+ \rangle} \bigr\} \, \E_{\theta^\star} \bigl\{ e^{-\frac{p}{2} \{ \frac{1}{v} - \frac{1-\alpha}{\sigma^2}\}\|\theta_B^+ - \hat\theta_B^+\|^2} \bigr\}. \]
Since $Y-\hat\theta_B^+ = (I-P_B) Y$ is a linear function of $Y$, where $P_B$ is a projection, the first expectation is just a moment-generating function for a multivariate normal, in particular,
\[ \E_{\theta^\star}\bigl\{ e^{\frac{p(1-\alpha)}{\sigma^2} \langle Y-\hat\theta_B^+, \theta^\star - \theta_B^+ \rangle} \bigr\} = e^{G \, \theta^{\star\top} (I-P_B) \theta^\star}, \]
where $G = \frac{p(1-\alpha)}{\sigma^2} \{1 + \frac{p(1-\alpha)}{2} \} = \frac{p+1}{2\sigma^2}(1 + \frac{p+1}{4})$ is a positive constant.
For the second expectation, use the distribution theory for $\hat\theta_B^+ = P_B Y$ to see that this boils down to a moment-generating function in terms of a non-central chi-square, i.e.,
\[ \E_{\theta^\star} \bigl\{ e^{-\frac{p}{2} \{ \frac{1}{v} - \frac{1-\alpha}{\sigma^2}\}\|\theta_B^+ - \hat\theta_B^+\|^2} \bigr\} = \Bigl( \frac{1}{1 + z} \Bigr)^{\frac{|B|}{2}} e^{-\frac{1}{2\sigma^2}\frac{z}{1+z} \|\theta_B^+ - P_B \theta^\star\|^2}, \]
where $z = p\sigma^2/v - p(1-\alpha) = p\sigma^2/v - (p+1)/2$. In order for the moment-generating function calculations above to be valid, and for the exponent to be negative, we need
\[ z > 0 \iff \frac{\sigma^2}{v} > \frac12 + \frac{1}{2p}. \]
That is, we require a choice of $p > 1$ such that the inequality in the right-hand side above is satisfied. This is possible under the stated condition, i.e., if $v < 2\sigma^2$.
The above the product of expectations is upper-bounded by a constant (in $\theta_B$) that depends on $B$ times a normal density in $\theta_B$. Taking $p^{-1}$ power and integrating over $\Theta_B$ as in \eqref{eq:num.expectation} gives
\[ \E_{\theta^\star}\{N_n(A_n)\} \leq e^{-(M/8q) \eps_n} \sum_B \pi_n(B) \zeta^{|B|} e^{(G/p) \theta^{\star \top}(I-P_B)\theta^\star}, \]
where $\zeta = (\sigma^2/vz)^{1/2}$ is a positive constant. Since the largest eigenvalue of $I-P_B$ is unity, we get a further upper bound
\[ \E_{\theta^\star}\{N_n(A_n)\} \leq e^{-(M/8q) \eps_n} e^{(G/p)\|\theta^\star\|^2} \sum_{b=1}^n \zeta^b f_n(b), \]
involving only an average with respect to the marginal distribution of block sizes, not the full block distribution. By definition of the block size distribution $f_n$ in \eqref{eq:S.prior}, the sum in the above display is bounded and, hence, the claim follows.
\end{proof}
\begin{thebibliography}{}
\bibitem[Bhattacharya et~al., 2017]{bhat.pati.yang.fractional}
Bhattacharya, A., Pati, D., and Yang, Y. (2017).
\newblock {B}ayesian fractional posteriors.
\newblock Unpublished manuscript, {\tt arXiv:1611.01125}.
\bibitem[Castillo and van~der Vaart, 2012]{castillo.vaart.2012}
Castillo, I. and van~der Vaart, A. (2012).
\newblock Needles and straw in a haystack: posterior concentration for possibly
sparse sequences.
\newblock {\em Ann. Statist.}, 40(4):2069--2101.
\bibitem[Donoho and Johnstone, 1994]{donohojohnstone1994b}
Donoho, D.~L. and Johnstone, I.~M. (1994).
\newblock Minimax risk over {$l\sb p$}-balls for {$l\sb q$}-error.
\newblock {\em Probab. Theory Related Fields}, 99(2):277--303.
\bibitem[Fan and Guan, 2017]{fan.guan.piece}
Fan, Z. and Guan, L. (2017).
\newblock Approximate $\ell_0$-penalized estimation of piecewise constant
signals on graphs.
\newblock {\em Ann. Statist.}, to appear; {\tt arXiv:1703.01421}.
\bibitem[Frick et~al., 2014]{frick.munk.sieling.2014}
Frick, K., Munk, A., and Sieling, H. (2014).
\newblock Multiscale change point inference.
\newblock {\em J. R. Stat. Soc. Ser. B. Stat. Methodol.}, 76(3):495--580.
\newblock With 32 discussions by 47 authors and a rejoinder by the authors.
\bibitem[Fryzlewicz, 2014]{fryzlewicz2014}
Fryzlewicz, P. (2014).
\newblock Wild binary segmentation for multiple change-point detection.
\newblock {\em Ann. Statist.}, 42(6):2243--2281.
\bibitem[Gao et~al., 2017]{gao.etal.piece}
Gao, C., Han, F., and Zhang, C.-H. (2017).
\newblock Minimax risk bounds for piecewise constant models.
\newblock Unpublished manuscript, {\tt arXiv:1705.06386}.
\bibitem[Gr\"unwald and van Ommen, 2017]{grunwald.ommen.scaling}
Gr\"unwald, P. and van Ommen, T. (2017).
\newblock Inconsistency of {B}ayesian inference for misspecified linear models,
and a proposal for repairing it.
\newblock {\em Bayesian Anal.}, 12(4):1069--1103.
\bibitem[Holmes and Walker, 2017]{holmes.walker.scaling}
Holmes, C.~C. and Walker, S.~G. (2017).
\newblock Assigning a value to a power likelihood in a general {B}ayesian
model.
\newblock {\em Biometrika}, 104(2):497--503.
\bibitem[Hutter, 2007]{hutter2007}
Hutter, M. (2007).
\newblock Exact {B}ayesian regression of piecewise constant functions.
\newblock {\em Bayesian Anal.}, 2(4):635--664.
\bibitem[Jiang and Zhang, 2009]{jiang.zhang.2009}
Jiang, W. and Zhang, C.-H. (2009).
\newblock General maximum likelihood empirical {B}ayes estimation of normal
means.
\newblock {\em Ann. Statist.}, 37(4):1647--1684.
\bibitem[Johnstone and Silverman, 2004]{johnstonesilverman2004}
Johnstone, I.~M. and Silverman, B.~W. (2004).
\newblock Needles and straw in haystacks: empirical {B}ayes estimates of
possibly sparse sequences.
\newblock {\em Ann. Statist.}, 32(4):1594--1649.
\bibitem[Martin, 2017]{ebmono}
Martin, R. (2017).
\newblock Empirical priors and posterior concentration rates for a monotone
density.
\newblock Unpublished manuscript, {\tt arXiv:1706.08567}.
\bibitem[Martin et~al., 2017]{martin.mess.walker.eb}
Martin, R., Mess, R., and Walker, S.~G. (2017).
\newblock Empirical {B}ayes posterior concentration in sparse high-dimensional
linear models.
\newblock {\em Bernoulli}, 23(3):1822--1847.
\bibitem[Martin and Walker, 2014]{martin.walker.eb}
Martin, R. and Walker, S.~G. (2014).
\newblock Asymptotically minimax empirical {B}ayes estimation of a sparse
normal mean vector.
\newblock {\em Electron. J. Stat.}, 8(2):2188--2206.
\bibitem[Martin and Walker, 2017]{martin.walker.deb}
Martin, R. and Walker, S.~G. (2017).
\newblock Empirical priors for target posterior concentration rates.
\newblock Unpublished manuscript, {\tt arXiv:1604.05734}.
\bibitem[Miller and Dunson, 2015]{miller.dunson.power}
Miller, J.~W. and Dunson, D.~B. (2015).
\newblock Robust {B}ayesian inference via coarsening.
\newblock Unpublished manuscript, {\tt arXiv:1506.06101}.
\bibitem[Syring and Martin, 2018]{syring.martin.scaling}
Syring, N. and Martin, R. (2018).
\newblock Calibrating general posterior credible regions.
\newblock Unpublished manuscript, {\tt arXiv:1509.00922}.
\bibitem[van~der Pas and Rockova, 2017]{pas.rockova.2017}
van~der Pas, S. and Rockova, V. (2017).
\newblock {B}ayesian dyadic trees and histograms for regression.
\newblock In Guyon, I., Luxburg, U.~V., Bengio, S., Wallach, H., Fergus, R.,
Vishwanathan, S., and Garnett, R., editors, {\em {A}dvances in {N}eural
{I}nformation {P}rocessing {S}ystems 30}, pages 2089--2099. Curran
Associates, Inc.
\bibitem[van~der Pas et~al., 2017]{pas.szabo.vaart.rate}
van~der Pas, S., Szab\'o, B., and van~der Vaart, A. (2017).
\newblock Adaptive posterior contraction rates for the horseshoe.
\newblock {\em Electron. J. Stat.}, 11(2):3196--3225.
\bibitem[van Erven and Harremo\"es, 2014]{vanerven.renyi}
van Erven, T. and Harremo\"es, P. (2014).
\newblock R\'enyi divergence and {K}ullback-{L}eibler divergence.
\newblock {\em IEEE Trans. Inform. Theory}, 60(7):3797--3820.
\end{thebibliography}